FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Scaling regression in `Tools/ftscalingbench/ftscalingbench.py` generator · Issue #144438 · python/cpython · GitHub

  1. python
  2. cpython

Repository navigation

Scaling regression in Tools/ftscalingbench/ftscalingbench.py generator #144438

Closed
Closed
Scaling regression in Tools/ftscalingbench/ftscalingbench.py generator#144438
Labels
performancePerformance or resource usagetopic-free-threadingtype-bugAn unexpected behavior, bug, or error

Description

colesbury
opened on Feb 3, 2026

Bug report

I'm having trouble pinpointing the commit, but there seems to be a substantial regression:

Machine 1 (Intel(R) Core(TM) Ultra 7 155H)

~/cpython$ uv run -p 3.14.2t python Tools/ftscalingbench/ftscalingbench.py  -t 10 --scale 10000 generator --parallel-only
Running benchmarks with 10 threads
generator          407.3 ms
~/cpython$ uv run -p 3.15.0a3t python Tools/ftscalingbench/ftscalingbench.py  -t 10 --scale 10000 generator --parallel-only
Running benchmarks with 10 threads
generator          587.8 ms

Machine 2 (Intel(R) Xeon(R) Gold 6230 CPU)

~/cpython$ uv run -p 3.14.2t python Tools/ftscalingbench/ftscalingbench.py  -t 10 --scale 10000 generator --parallel-only
Running benchmarks with 10 threads
generator          608.5 ms
~/cpython$ uv run -p 3.15.0a3t python Tools/ftscalingbench/ftscalingbench.py  -t 10 --scale 10000 generator --parallel-only
Running benchmarks with 10 threads
generator          1328.7 ms

Linked PRs

Activity

  1. colesbury
    added
    type-bugAn unexpected behavior, bug, or error
    performancePerformance or resource usage
    topic-free-threading
    on Feb 3, 2026

  2. kumaraditya303 commented on Feb 6, 2026

    kumaraditya303
    on Feb 6, 2026
    Contributor

    Timings on my machine:

    cpython on  immortal [$] 
❯ uv run -p 3.14.2t python Tools/ftscalingbench/ftscalingbench.py  -t 10 --scale 10000 generator --parallel-only
Running benchmarks with 10 threads
generator          675.9 ms

cpython on  immortal [$]
❯ uv run -p 3.15.0a3t python Tools/ftscalingbench/ftscalingbench.py  -t 10 --scale 10000 generator --parallel-only
Running benchmarks with 10 threads
generator          626.5 ms

    I am not seeing any regression on mac, is this linux specific?

  3. colesbury commented on Feb 6, 2026

    colesbury
    on Feb 6, 2026
    ContributorAuthor

    It might be specific to certain hardware. I'm also not seeing it on my Mac, but I can reliably reproduce the biggest gap on the Xeon Gold 6230 (Cascade Lake / 2019).

    I don't think it's the typical contention on reference count fields. It might be contention on some other data structure.

  4. colesbury commented on Feb 6, 2026

    colesbury
    on Feb 6, 2026
    ContributorAuthor

    Using Claude to measure and bisect. Here's a summary so far

    Perf on various release tags:

      ┌───────────┬───────────┬────────────────────────────┐
  │    Tag    │ Time (ms) │           Notes            │
  ├───────────┼───────────┼────────────────────────────┤
  │ v3.14.0   │ ~840      │ Baseline                   │
  ├───────────┼───────────┼────────────────────────────┤
  │ v3.14.2   │ ~850      │ Same as baseline           │
  ├───────────┼───────────┼────────────────────────────┤
  │ v3.15.0a1 │ ~1210     │ Already regressed          │
  ├───────────┼───────────┼────────────────────────────┤
  │ v3.15.0a3 │ ~970      │ Better than a1 (some fix?) │
  ├───────────┼───────────┼────────────────────────────┤
  │ v3.15.0a4 │ ~1530     │ Big regression             │
  ├───────────┼───────────┼────────────────────────────┤
  │ v3.15.0a5 │ ~1890     │ Worse still                │
  └───────────┴───────────┴────────────────────────────┘

    Bisecting between v3.15.0a3 and v3.15.0a4 gives 9e51301. Particularly, it seems like it's related to the added static strings. If I apply just the static strings to the previous commit, it introduces the regression.

    Per stat shows:

      ┌───────────────────────┬──────────────┬─────────────────┬───────┐
  │        Counter        │ GOOD (clean) │ BAD (+ strings) │ Ratio │
  ├───────────────────────┼──────────────┼─────────────────┼───────┤
  │ cycles                │ 24.2B        │ 42.6B           │ 1.76x │
  ├───────────────────────┼──────────────┼─────────────────┼───────┤
  │ instructions          │ 70.6B        │ 71.6B           │ 1.01x │
  ├───────────────────────┼──────────────┼─────────────────┼───────┤
  │ IPC                   │ 2.91         │ 1.68            │ 0.58x │
  ├───────────────────────┼──────────────┼─────────────────┼───────┤
  │ L1-dcache-load-misses │ 7.2M         │ 109.0M          │ 15.1x │
  ├───────────────────────┼──────────────┼─────────────────┼───────┤
  │ LLC-loads             │ 1.6M         │ 44.1M           │ 27.8x │
  ├───────────────────────┼──────────────┼─────────────────┼───────┤
  │ cache-references      │ 2.9M         │ 262.5M          │ 91.9x │
  ├───────────────────────┼──────────────┼─────────────────┼───────┤
  │ branch-misses         │ 12.2M        │ 12.1M           │ 1.0x  │
  ├───────────────────────┼──────────────┼─────────────────┼───────┤
  │ dTLB-load-misses      │ 116K         │ 168K            │ 1.4x  │
  └───────────────────────┴──────────────┴─────────────────┴───────┘

    And some other events:

      ┌──────────────────────────────┬───────┬───────┬───────┐
  │           Counter            │ GOOD  │  BAD  │ Ratio │
  ├──────────────────────────────┼───────┼───────┼───────┤
  │ L1-dcache-load-misses        │ 7.4M  │ 41.8M │ 5.7x  │
  ├──────────────────────────────┼───────┼───────┼───────┤
  │ l2_rqsts.all_demand_data_rd  │ 4.6M  │ 20.8M │ 4.5x  │
  ├──────────────────────────────┼───────┼───────┼───────┤
  │ l2_rqsts.demand_data_rd_miss │ 1.2M  │ 18.5M │ 15.9x │
  ├──────────────────────────────┼───────┼───────┼───────┤
  │ l2_rqsts.all_rfo             │ 1.1M  │ 19.4M │ 17.6x │
  ├──────────────────────────────┼───────┼───────┼───────┤
  │ l2_rqsts.rfo_miss            │ 0.35M │ 19.0M │ 54.3x │
  ├──────────────────────────────┼───────┼───────┼───────┤
  │ offcore_response.demand_rfo  │ 0.16M │ 18.8M │ 115x  │
  └──────────────────────────────┴───────┴───────┴───────┘

    RFO is "request for ownership". Basically, when a CPU needs to modify some memory (execute a store instruction), it must have exclusive ownership of the cache line. I'm not sure these are necessarily due to actually executed store instruction. I think it might be either issued speculatively due to mispredicting stores or a prefetcher.

  5. colesbury commented on Feb 6, 2026

    colesbury
    on Feb 6, 2026
    ContributorAuthor

    Ok, it's false sharing between tlbc_index in _PyThreadStateImpl and the _qsbr_thread_state from a different thread. If you're unlucky, the tail end of _PyThreadStateImpl ends up in the same cache line as the beginning of the _qsbr_thread_state array.

    cpython/Include/internal/pycore_tstate.h

    Lines 86 to 87 in 895e837

    // Index to use to retrieve thread-local bytecode for this thread
    int32_t tlbc_index;

    cpython/Include/internal/pycore_qsbr.h

    Lines 72 to 76 in 895e837

    // Padding to avoid false sharing
    struct _qsbr_pad {
    struct _qsbr_thread_state qsbr;
    char __padding[64 - sizeof(struct _qsbr_thread_state)];
    };

    We have padding at the end of each struct _qsbr_pad to avoid false sharing with other entries within the array, but that doesn't prevent false sharing between the first entry in the array and other data.

    I think we want to do two things:

    1. Align the beginning of the array to 64 bytes. Since we use the PyMem_Raw allocator, I think we just over allocate by 64 bytes and then align.

    2. Pad the end of _PyThreadStateImpl so that we avoid potential false sharing with other data structures.

    Helpful commands:

    PYTHONHASHSEED=0 perf c2c record -o /tmp/c2c.data -- setarch -R taskset -c 10-19 ./python Tools/ftscalingbench/ftscalingbench.py -t 10 --scale 10000 generator --parallel-only
perf c2c report -i /tmp/c2c.data --stdio -g none
  6. colesbury
    added 3 commits that reference this issue on Feb 6, 2026

    pythongh-144438: Fix false sharing between QSBR and tlbc_index

    2d34deb

    pythongh-144438: Fix false sharing between QSBR and tlbc_index

    91bf6d2

    pythongh-144438: Fix false sharing between QSBR and tlbc_index

    ded8366
  7. colesbury
    added 2 commits that reference this issue on Feb 17, 2026

    gh-144438: Fix false sharing between QSBR and tlbc_index (gh-144554)

    6577d87

    [3.14] pythongh-144438: Fix false sharing between QSBR and tlbc_index (…

    5e2993f
  8. brijkapadia
    added a commit that references this issue on Feb 28, 2026

    pythongh-144438: Fix false sharing between QSBR and tlbc_index (pytho…

    3f3bd45
  9. colesbury
    added a commit that references this issue on Mar 31, 2026

    [3.14] gh-144438: Fix false sharing between QSBR and tlbc_index (gh-1…

    6ea4f84
  10. colesbury
    closed this as completedon Apr 8, 2026
  11. ljfp
    added a commit that references this issue on Apr 25, 2026

    pythongh-144438: Fix false sharing between QSBR and tlbc_index (pytho…

    072b288
  12. gr4ph0s
    added a commit that references this issue on Oct 20, 2026

    pythongh-159860: Pad QSBR thread states to 128 bytes on AArch64

    4ab0962
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagetopic-free-threadingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      Back | FazBrowse Home | New Git URL