GH-42173: [R][C++] Writing partitioned dataset on S3 fails if ListBucket is not allowed for the user by simonelbaz · Pull Request #47599 · apache/arrow · GitHub

Rationale for this change

This PR gives the user to choose not to create directory in the bucket before writing dataset.
In case the create_directory option is set to FALSE, no verification will be made by R arrow.
The S3 storage will itself verify if the directory exists and if the users has the rigth to modify it.
This way no ListBucket or HeadBucket are necessary to achieve the write operation.

df |> arrow::write_dataset(
  minio$path(paste0("smartsla-bucket/rarrow/")),
  partitioning = "qualitative",
  create_directory = FALSE,
  format = "parquet"
)

What changes are included in this PR?

create_directory is now available to the user in the write_dataset function.
Before this PR, this option was automatically set to TRUE (by default).

Are these changes tested?

Yes

Are there any user-facing changes?

No, the default value for create_directory is still TRUE.

GitHub Issue: [R] Writing partitionned dataset with Rarrow on S3 failed if ListBucket is not allowed for the user #42173

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

⚠️ GitHub issue #42173 has been automatically assigned in GitHub to PR creator.

Hi,

@jonkeane @thisisnic thanks for any comment || review

⚠️ GitHub issue #42173 has been automatically assigned in GitHub to PR creator.

Thanks for making this PR @simonelbaz! I've given it a look over, and it looks like there's an existing option for this scenario that might be a better solution, though I haven't tried it out myself - let me know what you think.

⚠️ GitHub issue #42173 has been automatically assigned in GitHub to PR creator.

…d if ListBucket is not allowed for the user

…3 failed if ListBucket is not allowed for the user

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

+1 from me. @thisisnic Are you ok with the new argument name and docstring?

Hi @thisisnic, thanks for any feedback or comment on the PR.

Great, thanks!

⚠️ GitHub issue #42173 has been automatically assigned in GitHub to PR creator.

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit b808fbb.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

…istBucket is not allowed for the user (apache#47599) ### Rationale for this change This PR gives the user to choose not to create directory in the bucket before writing dataset. In case the `create_directory` option is set to FALSE, no verification will be made by R arrow. The S3 storage will itself verify if the directory exists and if the users has the rigth to modify it. This way no `ListBucket` or ` HeadBucket` are necessary to achieve the write operation. ``` df |> arrow::write_dataset( minio$path(paste0("smartsla-bucket/rarrow/")), partitioning = "qualitative", create_directory = FALSE, format = "parquet" ) ``` ### What changes are included in this PR? `create_directory` is now available to the user in the `write_dataset` function. Before this PR, this option was automatically set to TRUE (by default). ### Are these changes tested? Yes ### Are there any user-facing changes? No, the default value for `create_directory` is still TRUE. * GitHub Issue: apache#42173 Lead-authored-by: Simon ELBAZ <elbazsimon9@gmail.com> Co-authored-by: Simon Elbaz <elbazsimon9@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr> Signed-off-by: Antoine Pitrou <antoine@python.org>

simonelbaz requested review from jonkeane and thisisnic as code owners September 18, 2025 20:53

github-actions Bot added Component: R Component: C++ awaiting review Awaiting review labels Sep 18, 2025

simonelbaz changed the title [ISSUE 42173][R] Writing partitionned dataset with Rarrow on S3 failed if ListBucket is not allowed for the user [GH-42173][R] Writing partitionned dataset with Rarrow on S3 failed if ListBucket is not allowed for the user Sep 18, 2025

simonelbaz changed the title [GH-42173][R] Writing partitionned dataset with Rarrow on S3 failed if ListBucket is not allowed for the user GH-42173: [R] Writing partitionned dataset with Rarrow on S3 failed if ListBucket is not allowed for the user Sep 18, 2025

simonelbaz changed the title GH-42173: [R] Writing partitionned dataset with Rarrow on S3 failed if ListBucket is not allowed for the user GH-42173: [R][S3] Writing partitionned dataset with Rarrow on S3 failed if ListBucket is not allowed for the user Sep 18, 2025

simonelbaz marked this pull request as draft September 18, 2025 21:08

simonelbaz marked this pull request as ready for review September 25, 2025 08:07

simonelbaz force-pushed the issue42173 branch from f7cb2a3 to 558fc5a Compare October 10, 2025 08:54

thisisnic changed the title GH-42173: [R][S3] Writing partitionned dataset with Rarrow on S3 failed if ListBucket is not allowed for the user GH-42173: [R][C++] Writing partitioned dataset with S3 fails if ListBucket is not allowed for the user Oct 13, 2025

thisisnic requested changes Oct 13, 2025

View reviewed changes

Comment thread cpp/src/arrow/dataset/dataset_writer.cc Outdated

thisisnic changed the title GH-42173: [R][C++] Writing partitioned dataset with S3 fails if ListBucket is not allowed for the user GH-42173: [R][C++] Writing partitioned dataset on S3 fails if ListBucket is not allowed for the user Oct 13, 2025

simonelbaz commented Oct 13, 2025

View reviewed changes

Comment thread r/R/dataset-write.R

pitrou reviewed Oct 13, 2025

View reviewed changes

Comment thread cpp/src/arrow/dataset/dataset_writer.cc Outdated

simonelbaz added 2 commits October 15, 2025 20:08

[ISSUE 42173][R] Writing partitionned dataset with Rarrow on S3 faile…

33774ea

…d if ListBucket is not allowed for the user

fixup! [ISSUE 42173][R] Writing partitionned dataset with Rarrow on S…

4dea170

…3 failed if ListBucket is not allowed for the user

simonelbaz force-pushed the issue42173 branch from 2facd98 to 2258358 Compare October 15, 2025 21:44

github-actions Bot removed the Component: C++ label Oct 15, 2025

fixup! [ISSUE 42173][R] Writing partitionned dataset with Rarrow on S…

0a064ca

…3 failed if ListBucket is not allowed for the user

simonelbaz force-pushed the issue42173 branch from 2258358 to 0a064ca Compare October 16, 2025 06:24

simonelbaz requested review from pitrou and thisisnic October 16, 2025 08:29

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Oct 16, 2025

pitrou reviewed Oct 16, 2025

View reviewed changes

Comment thread r/R/dataset-write.R Outdated

Update r/R/dataset-write.R

af71e52

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

simonelbaz requested a review from pitrou October 16, 2025 09:32

pitrou approved these changes Oct 16, 2025

View reviewed changes

thisisnic approved these changes Oct 21, 2025

View reviewed changes

github-actions Bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Oct 21, 2025

pitrou merged commit b808fbb into apache:main Oct 21, 2025
11 of 12 checks passed

pitrou removed the awaiting merge Awaiting merge label Oct 21, 2025

pitrou mentioned this pull request Oct 21, 2025

[R] Writing partitionned dataset with Rarrow on S3 failed if ListBucket is not allowed for the user #42173

Closed

GH-42173: [R][C++] Writing partitioned dataset on S3 fails if ListBucket is not allowed for the user by simonelbaz · Pull Request #47599 · apache/arrow · GitHub

Uh oh!

GH-42173: [R][C++] Writing partitioned dataset on S3 fails if ListBucket is not allowed for the user - #47599

GH-42173: [R][C++] Writing partitioned dataset on S3 fails if ListBucket is not allowed for the user#47599pitrou merged 4 commits intoapache:mainapache/arrow:mainfrom simonelbaz:issue42173simonelbaz/arrow:issue42173Copy head branch name to clipboard

Conversation

simonelbaz commented Sep 18, 2025 • edited by pitrou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Sep 18, 2025

Uh oh!

github-actions Bot commented Sep 18, 2025

Uh oh!

github-actions Bot commented Sep 18, 2025

Uh oh!

simonelbaz commented Oct 10, 2025

Uh oh!

github-actions Bot commented Oct 13, 2025

Uh oh!

thisisnic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

simonelbaz commented Oct 20, 2025

Uh oh!

thisisnic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 21, 2025

Uh oh!

conbench-apache-arrow Bot commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GH-42173: [R][C++] Writing partitioned dataset on S3 fails if ListBucket is not allowed for the user#47599
pitrou merged 4 commits into
apache:mainfrom
simonelbaz:issue42173

simonelbaz commented Sep 18, 2025 •
edited by pitrou
Loading