bpo-40059: tomllib by hukkin · Pull Request #31498 · python/cpython · GitHub

This adds a new standard library module, tomllib, for parsing TOML. The recently accepted PEP 680 -- tomllib is relevant here.

This PR has already seen some review in a PR under my personal fork: hukkin#2 (thanks to @encukou, @merwok, @hauntsaninja, @JelleZijlstra (I hope I'm not forgetting anyone)).

The implementation is based on Tomli which I plan to keep maintaining as a backport for Python versions 3.7, 3.8, 3.9 and 3.10, until finally Python 3.10 goes EOL.

Steps taken (converting tomli to tomllib)

Move everything in tomli:src/tomli to Lib/tomllib. Exclude py.typed.
Remove __version__ = ... line from Lib/tomllib/__init__.py
Move everything in tomli:tests to Lib/test/test_tomllib. Exclude the following test data dirs recursively:
- tomli:tests/data/invalid/_external/
- tomli:tests/data/valid/_external/

Create Lib/test/test_tomllib/__main__.py:

import unittest

from . import load_tests


unittest.main()

Add the following to Lib/test/test_tomllib/__init__.py:

import os
from test.support import load_package_tests

def load_tests(*args):
    return load_package_tests(os.path.dirname(__file__), *args)

Also change import tomli as tomllib to import tomllib.

In cpython/Lib/tomllib/_parser.py replace __fp with fp and __s with s. Add the / to load and loads function signatures.
Run make regen-stdlib-module-names
Create Doc/library/tomllib.rst and reference it in Doc/library/fileformats.rst

edit: For reference, there's one more step – Add tomllib to Makefile.pre.in

https://bugs.python.org/issue40059

A question: upstream (Tomli) is formatted with Black, using Black's defaults. This means a line length of 88. Should I reformat with line length at 79?

This will need NEWS and "What's new" entries:

https://devguide.python.org/committing/#updating-news-and-what-s-new-in-python

You can add this to https://github.com/python/cpython/blob/main/.github/CODEOWNERS

Perhaps it'd make sense to include the tomli version used (or even the git commit hash), to make future syncing easier.

Perhaps it'd make sense to include the tomli version used (or even the git commit hash), to make future syncing easier.

It was previously suggested that I remove Tomli version from the standard library port: hukkin#2 (comment)

I plan to include a migration guide in Tomli repository, also including commit hash that tracks what is currently in the stdlib.

Would it be good to mention in the docs why load() takes only binary files? The encoding requirement probably isn't obvious for first-time users.

I should get to the review this week or the next.

A question: upstream (Tomli) is formatted with Black, using Black's defaults. This means a line length of 88. Should I reformat with line length at 79?

No, I don't think that's worth it. And anyway, line length is not the only point where Black disagrees with PEP8 (starting with, like, the whole philosophy).
But future edits probably won't use Black.

Would it be good to mention in the docs why load() takes only binary files? The encoding requirement probably isn't obvious for first-time users.

Let's leave docs improvements to future PRs, so they get a proper discussion and don't delay this PR?

Co-authored-by: Petr Viktorin <encukou@gmail.com>

I have a few test nitpicks, but I'm happy to merge this!

MissingFile looks unnecessary. Does tomli need it?

Ah, I see it does.
For a poor man's xfail, you could you assert that p.stem is one of the expected failing cases.

MissingFile looks unnecessary. Does tomli need it?

Yeah one of the two external test suites has a couple test cases where the expected data is missing.

For a poor man's xfail, you could you assert that p.stem is one of the expected failing cases.

I'm not sure I understand what you mean. Maybe you can show with the "Add a suggestion" feature?

I think something like:

if isinstance(expected, MissingFile):
    assert valid.stem in ("xfail_test_case1, "xfail_test_case2", ...)
    continue

Oh yes, of course, thanks. Committed that.

For peace of mind, could you assert len(VALID_FILES) > 0, and same for INVALID_FILES?

👍 Makes sense. Added the asserts.

🤖 New build scheduled with the buildbot fleet by @encukou for commit 2898cc3 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

It seems the failing CI job (AMD64 Arch Linux Usan PR) errors in other PRs too so should be unrelated to this PR.

The alpha 6 release is a bit bumpy and I don't want to destabilize it, so I'm holding off the merge until it's out.
I tested with all buildbots to see if there's an unforeseen platform-specific issue. It's common to see a few buildbots fail.

Let's get it in!

hukkin added 7 commits February 2, 2022 01:02

Add tomllib and tests

2f5edf2

Run make regen-stdlib-module-names

026a48b

Add tomllib docs

1cdda7c

Document parse_float limitations

eafc4e0

Add conversion table to docs

1c9b341

Sync with Tomli

876a43c

Remove commented out test code

ab5e488

hukkin requested review from abalkin and pganssle as code owners February 22, 2022 13:24

bedevere-bot added the awaiting review label Feb 22, 2022

the-knights-who-say-ni added the CLA signed label Feb 22, 2022

hukkin mentioned this pull request Feb 22, 2022

Please consider pushing tomli into stdlib hukkin/tomli#141

Closed

hugovk added the type-feature A feature request or enhancement label Feb 22, 2022

hukkin added 3 commits February 23, 2022 01:12

Add NEWS

4316ae6

Add whatsnew

13edc69

Add CODEOWNERS

6cd95a5

hukkin mentioned this pull request Feb 22, 2022

bpo-40059: tomllib hukkin/cpython#2

Closed

astrojuanlu mentioned this pull request Feb 23, 2022

ENH: BLD: enable building SciPy with Meson scipy/scipy#14847

Merged

Contextualist mentioned this pull request Feb 27, 2022

Migrate Python dependency uiri/toml to tomllib / hukkin/tomli PyO3/maturin#821

Merged

encukou reviewed Mar 2, 2022

View reviewed changes

Comment thread .github/CODEOWNERS Outdated

Update .github/CODEOWNERS

d5e8053

Co-authored-by: Petr Viktorin <encukou@gmail.com>

encukou approved these changes Mar 2, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Mar 2, 2022

hukkin added 2 commits March 3, 2022 00:00

Add asserts

ea27f36

Add a poor man's xfail

2898cc3

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 3, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 3, 2022

brandtbucher mentioned this pull request Mar 3, 2022

bpo-46841: Use inline caching for attribute accesses #31640

Merged

hukkin mentioned this pull request Mar 3, 2022

Add type hints for tomllib python/typeshed#7432

Merged

encukou merged commit 591f675 into python:main Mar 8, 2022

bedevere-bot removed the awaiting merge label Mar 8, 2022

hukkin deleted the tomllib branch March 8, 2022 08:29

mkniewallner mentioned this pull request Mar 8, 2022

Replace toml with tomli PyCQA/bandit#829

Merged

messense mentioned this pull request Mar 15, 2022

No TOML module installed for Python 3.11 PyO3/maturin#850

Closed

2 tasks

DanielNoord mentioned this pull request Mar 15, 2022

Can't install on the CPython 3.11 (broken indirect wrapt deps) pylint-dev/pylint#5919

Closed

Pierre-Sassoulas mentioned this pull request Mar 15, 2022

Add Python 3.11-dev to Github actions pylint-dev/pylint#5920

Closed

mgorny mannequin mentioned this pull request Apr 10, 2022

Provide a toml module in the standard library #84240

Closed

coderabbitai Bot mentioned this pull request Jul 26, 2026

Add reproducible Linux release packaging and compatibility CI JoshCLWren/cdisplayagain#49

Merged

coderabbitai Bot mentioned this pull request Aug 9, 2026

feat(labels): one ordered axis, as data, with a sync script tinyhumansai/tinysweeper#70

Merged

		INVALID_FILES = tuple((DATA_DIR / "invalid").glob("*/.toml"))


		class TestData(unittest.TestCase):

bpo-40059: tomllib by hukkin · Pull Request #31498 · python/cpython · GitHub

Uh oh!

Uh oh!

bpo-40059: tomllib - #31498

bpo-40059: tomllib#31498encukou merged 13 commits intopython:mainpython/cpython:mainfrom hukkin:tomllibhukkin/cpython:tomllibCopy head branch name to clipboard

Conversation

hukkin commented Feb 22, 2022 • edited by encukou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps taken (converting tomli to tomllib)

Uh oh!

hukkin commented Feb 22, 2022

Uh oh!

hugovk commented Feb 22, 2022

Uh oh!

mgorny commented Feb 22, 2022

Uh oh!

hukkin commented Feb 22, 2022

Uh oh!

TeamSpen210 commented Feb 22, 2022

Uh oh!

encukou commented Feb 23, 2022

Uh oh!

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

encukou Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

encukou Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

hukkin Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

hauntsaninja Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

hukkin Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

encukou Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

hukkin Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Mar 3, 2022

Uh oh!

hukkin commented Mar 4, 2022

Uh oh!

encukou commented Mar 4, 2022

Uh oh!

encukou commented Mar 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

bpo-40059: tomllib#31498
encukou merged 13 commits into
python:mainfrom
hukkin:tomllib

hukkin commented Feb 22, 2022 •
edited by encukou
Loading