ARROW-16887: [R][Docs] Update Filesystem Vignette for GCS by wjones127 · Pull Request #13601 · apache/arrow · GitHub

This PR:

Replaces all references to the Ursa Labs bucket with the new voltrondata-labs-datasets bucket.
Adds a new gs_bucket() function to R, which parallels the design of the s3_bucket() function.
Updates the fs.Rmd vignette to also discuss GCS. Left discussion about authentication for a follow-up, partly because I am still confused how GCP auth works for non-application use.

https://issues.apache.org/jira/browse/ARROW-16887

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

IIRC we recently changed how the Linux installation works. Do these lines still apply?

@nealrichardson do you know the answer to this?

Oof, the answer is "it depends", it's not on by default in the pure source build, but if your system meets the requirements (which are too fussy to expand on here), you may get a prebuilt libarrow binary, which (may) have support for S3 and GCS.

Is there also a reason for removing the partitioning option?

This is a newer version of the dataset that is Hive partitioned, so doesn't require explicitly passing partitioning.

What happens on Windows exactly? These bare FIXMEs are not explanatory.

This was copied over from the S3 tests. I'm not sure what happens, but I can try and see.

I deleted the skips and seems to run fine locally and in CI.

I added this because it got a lot of files in it while building the R doc site; lmk if there's a good reason to not add it.

No objection; what creates this?

When I run pkgdown::build_site() within RStudio, during the "Installing package into temporary library" it creates this directory.

This isn't intentional. I just ran make doc locally. Is something about my R out of date, or does that command need an update?

Upgrading to roxygen2 7.2 fixed this.

cc @thisisnic would you be wiling to review?

I haven't read in detail, but this looks fine to me overall.

A few suggestions here and there but overall LGTM, thanks for taking this on.

No objection; what creates this?

Not in scope here (and probably not relevant for this exact line, this just made me think of it): we should show an example of glimpse() in this vignette somewhere.

Oof, the answer is "it depends", it's not on by default in the pure source build, but if your system meets the requirements (which are too fussy to expand on here), you may get a prebuilt libarrow binary, which (may) have support for S3 and GCS.

! Why didn't we do this before then?!

It might be worth showing how this is equivalent to GcsFileSystem$create(anonymous = TRUE, retry_limit_seconds = 10)$path("voltrondata-labs-datasets/nyc-taxi/") or whatever it is. As in, URI query params are useful but not the only way to provide these options.

Oh, since the FS objects are introduced below, maybe this note about equivalence should be introduced down there.

Yeah I wasn't sure I wanted to do this refactor, but it did feel like it would almost be easier to introduce GcsFileSystem$create() and then explain options in URIs.

I've reordered them so we introduce the filesystems and that made it more natural to make the statement you suggested.

cc @thisisnic would you be wiling to review?

With combined vacation + rstudio::conf I won't have time to look at this til 2nd August, so probably best not wait for me!

@jonkeane @nealrichardson can we go ahead and merge it?

@jonkeane @nealrichardson can we go ahead and merge it?

I already approved and don't feel the need to re-review, I'm sure @wjones127 took care of everything, please merge.

Merging, thanks @wjones127!

Benchmark runs are scheduled for baseline = ab8c92c and contender = 42647dc. 42647dc is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.07% ⬆️0.03%] test-mac-arm
[Finished ⬇️0.54% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.25% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Failed] 42647dcd ec2-t3-xlarge-us-east-2
[Finished] 42647dcd test-mac-arm
[Finished] 42647dcd ursa-i9-9960x
[Finished] 42647dcd ursa-thinkcentre-m75q
[Failed] ab8c92cf ec2-t3-xlarge-us-east-2
[Finished] ab8c92cf test-mac-arm
[Finished] ab8c92cf ursa-i9-9960x
[Finished] ab8c92cf ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

This PR: * Moves minio integration tests into a generic suite that is now run on minio (S3 emulator) and GCS testbench (GCS emulator). This is run in CI. * Move Minio and GCS test server initialization to within the tests. This makes it easier to setup the background processes in a cross-platform way. * MinIO and GCS tests are now run on R Ubuntu CI. MinIO is now run on Windows CI. I couldn't get GCS to run on Windows CI yet, due to some issue where the tests hang (I believe this is an issue with the test setup and not the functionality). See follow up at: [ARROW-17149: [R] Enable GCS tests for Windows](https://issues.apache.org/jira/browse/ARROW-17149) * Sets the default retry timeout to 15 seconds to mitigate issue described by ARROW-17020. This affects explicitly-created fs with `GcsFileSystem$create()` (and `gs_bucket()` introducted in #13601), but not URIs. Authored-by: Will Jones <willjones127@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

…Hub issue numbers (#34260) Rewrite the Jira issue numbers to the GitHub issue numbers, so that the GitHub issue numbers are automatically linked to the issues by pkgdown's auto-linking feature. Issue numbers have been rewritten based on the following correspondence. Also, the pkgdown settings have been changed and updated to link to GitHub. I generated the Changelog page using the `pkgdown::build_news()` function and verified that the links work correctly. --- ARROW-6338 #5198 ARROW-6364 #5201 ARROW-6323 #5169 ARROW-6278 #5141 ARROW-6360 #5329 ARROW-6533 #5450 ARROW-6348 #5223 ARROW-6337 #5399 ARROW-10850 #9128 ARROW-10624 #9092 ARROW-10386 #8549 ARROW-6994 #23308 ARROW-12774 #10320 ARROW-12670 #10287 ARROW-16828 #13484 ARROW-14989 #13482 ARROW-16977 #13514 ARROW-13404 #10999 ARROW-16887 #13601 ARROW-15906 #13206 ARROW-15280 #13171 ARROW-16144 #13183 ARROW-16511 #13105 ARROW-16085 #13088 ARROW-16715 #13555 ARROW-16268 #13550 ARROW-16700 #13518 ARROW-16807 #13583 ARROW-16871 #13517 ARROW-16415 #13190 ARROW-14821 #12154 ARROW-16439 #13174 ARROW-16394 #13118 ARROW-16516 #13163 ARROW-16395 #13627 ARROW-14848 #12589 ARROW-16407 #13196 ARROW-16653 #13506 ARROW-14575 #13160 ARROW-15271 #13170 ARROW-16703 #13650 ARROW-16444 #13397 ARROW-15016 #13541 ARROW-16776 #13563 ARROW-15622 #13090 ARROW-18131 #14484 ARROW-18305 #14581 ARROW-18285 #14615 * Closes: #33631 Authored-by: SHIMA Tatsuya <ts1s1andn@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

github-actions Bot added Component: Documentation Component: C++ Component: Python Component: R labels Jul 13, 2022

wjones127 commented Jul 18, 2022

View reviewed changes

pitrou reviewed Jul 19, 2022

View reviewed changes

wjones127 commented Jul 19, 2022

View reviewed changes

wjones127 marked this pull request as ready for review July 19, 2022 17:55

wjones127 added 10 commits July 20, 2022 11:43

Start drafting todos

0d1d538

chore: replace ursa bucket with voltrondata-labs-datasets

79281c3

feat: add gs_bucket() function

91d3211

fix: address failures in gs_bucket()

5725e8c

chore: update dataset.Rmd fake output

c4ffa74

doc: add GCS to fs.Rmd vignette

6fd2e3f

docs: leave auth for another PR

909b84c

doc: fix regression in docs format

cd659af

fix: lint python

2114969

test: try not skipping unnecessarily on Windows

176a7f5

wjones127 force-pushed the ARROW-16887-doc-gcs branch from 96f7897 to 176a7f5 Compare July 20, 2022 18:46

wjones127 mentioned this pull request Jul 20, 2022

ARROW-16879: [R][CI] Test R GCS bindings with testbench #13542

Merged

wjones127 requested a review from pitrou July 21, 2022 15:08

pitrou approved these changes Jul 21, 2022

View reviewed changes

doc: clarify use of gcloud to get credentials

72a64d0

nealrichardson approved these changes Jul 21, 2022

View reviewed changes

fix: minor PR feedback

3aae2e9

docs: refactor fs vignette and add description of GcsFileSystem params

7bd2f8a

wjones127 requested a review from nealrichardson July 22, 2022 20:06

kszucs approved these changes Jul 25, 2022

View reviewed changes

kszucs merged commit 42647dc into apache:master Jul 25, 2022

asfimport mentioned this pull request Nov 23, 2022

[Doc][R] Document GCSFileSystem for R package #32209

Closed

eitsupi mentioned this pull request Feb 19, 2023

GH-33631: [R] Rewrite Jira ticket numbers in pkgdown documents to GitHub issue numbers #34260

Merged

+               $ tip_amount     : num [1:10918] 3 0 4 1 1 6 0 1.35 0 5.9 ...
+               $ total_amount   : num [1:10918] 18.8 13.3 20.3 15.8 13.3 ...
+               $ passenger_count: int [1:10918] 3 2 1 1 1 1 1 1 1 3 ...
+               $ tip_pct        : num [1:10918] 0.1596 0 0.197 0.0633 0.0752 ...

+              ```
+              tells the `S3FileSystem` that it should allow the creation of new buckets and to
+              talk to Google Storage instead of S3. The latter works because GCS implements an

+              ```
+              gs://anonymous@voltrondata-labs-datasets/nyc-taxi/?retry_limit_seconds=10
+              ```

ARROW-16887: [R][Docs] Update Filesystem Vignette for GCS by wjones127 · Pull Request #13601 · apache/arrow · GitHub

Uh oh!

ARROW-16887: [R][Docs] Update Filesystem Vignette for GCS - #13601

ARROW-16887: [R][Docs] Update Filesystem Vignette for GCS#13601kszucs merged 13 commits intoapache:masterapache/arrow:masterfrom wjones127:ARROW-16887-doc-gcswjones127/arrow:ARROW-16887-doc-gcsCopy head branch name to clipboard

Conversation

wjones127 commented Jul 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2022

Uh oh!

github-actions Bot commented Jul 13, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wjones127 Jul 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjones127 Jul 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjones127 commented Jul 21, 2022

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

nealrichardson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thisisnic commented Jul 22, 2022

Uh oh!

kszucs commented Jul 25, 2022

Uh oh!

nealrichardson commented Jul 25, 2022

Uh oh!

kszucs left a comment

Choose a reason for hiding this comment

Uh oh!

ursabot commented Jul 26, 2022

Uh oh!

ARROW-16887: [R][Docs] Update Filesystem Vignette for GCS#13601
kszucs merged 13 commits into
apache:masterfrom
wjones127:ARROW-16887-doc-gcs

wjones127 commented Jul 13, 2022 •
edited
Loading

wjones127 Jul 19, 2022 •
edited
Loading

wjones127 Jul 19, 2022 •
edited
Loading