FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Implement all CSV reader options by timsaucer · Pull Request #1361 · apache/datafusion-python · GitHub

/ datafusion-python Public

Implement all CSV reader options - #1361

Merged
timsaucer merged 4 commits into
apache:mainfrom
timsaucer:feat/all-csv-options
Feb 5, 2026
ConversationCommits (4)ChecksFiles changed
Merged

Implement all CSV reader options#1361
timsaucer merged 4 commits into
apache:mainfrom
timsaucer:feat/all-csv-options

Conversation

timsaucer commented Feb 1, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #1358

Rationale for this change

This change allows access to ALL csv reader options available in the upstream DataFusion repository. It adds a builder pattern on setting options for a more pleasant user experience, or they can specify each value individually in the constructor.

What changes are included in this PR?

  • Adds a CsvReaderOptions class and underlying struct
  • Adds parameter for these options in read_csv and register_csv
  • Adds user example and unit tests
  • Update documentation site

Are there any user-facing changes?

This change is fully backwards compatible.

timsaucer commented Feb 1, 2026

Copy link
Copy Markdown
Member Author

FYI @djouallah

This was referenced Feb 1, 2026
Closed
Closed
nuno-faria approved these changes Feb 3, 2026
View reviewed changes

nuno-faria left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM

Comment thread docs/source/user-guide/io/csv.rst
Comment on lines +40 to +44
If you require additional control over how to read the CSV file, you can use
:py:class:`~datafusion.options.CsvReadOptions` to set a variety of options.

.. code-block:: python

nuno-faria Feb 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think it would be nice to have a link to docs.rs or similar that points to all available options for CsvReadOptions.

timsaucer Feb 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Added

Comment thread python/datafusion/options.py
Comment on lines +71 to +72
terminator: Optional line terminator character. If ``None``, uses CRLF.
Must be a single ASCII character.

nuno-faria Feb 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'm surprised the default is CRLF.

timsaucer Feb 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

🤷

Comment thread python/tests/test_context.py Outdated
Comment on lines +752 to +760
csv_content = "name;age;city\nAlice;30;New York\nBob;25\n#Charlie;35;Paris"
csv_path.write_text(csv_content)

ctx = SessionContext()

# Test with CsvReadOptions
options = CsvReadOptions(
has_header=True, delimiter=";", comment="#", truncated_rows=True
)

nuno-faria Feb 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Should more parameters be tested? Like "quote", "truncated_rows", "compression", "null_regex", ...

timsaucer Feb 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Added unit tests to cover all parameters. Good suggestion!

timsaucer merged commit 1da08a2 into apache:main Feb 5, 2026
17 checks passed
timsaucer deleted the feat/all-csv-options branch February 5, 2026 21:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

nuno-faria nuno-faria approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Expose all CSV read options

2 participants

Back | FazBrowse Home | New Git URL