FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Close guard-scan and reused-buffer gaps in dictionary reclamation by rkennke · Pull Request #845 · DataDog/java-profiler · GitHub

Repository navigation

Close guard-scan and reused-buffer gaps in dictionary reclamation - #845

Merged
rkennke merged 3 commits into
mainfrom
fix/prof-16136-guard-scan-torn-read
Oct 9, 2026
Merged

rkennke merged 3 commits into
mainfrom
fix/prof-16136-guard-scan-torn-read

Conversation

rkennke commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?:

Fixes two gaps in the dictionary reclamation from #839, both found by Bits Code Review on #840.

  1. Guard scan could miss a protected resource (use-after-free window). A nested guard used to install its resource in active_ptr and park the displaced outer resource in outer_stack, then move it back when it ended. The scanner reads active_ptr, then the stack, so it could read each location while the resource was in the other one:

    • after one nested guard ended;
    • or across one ending and the next starting. Re-reading active_ptr does not close this case (see the review thread).

    Nested guards now record their own resource in nested[depth - 1] (formerly outer_stack) and never touch active_ptr, which keeps the root guard's resource. Every protected resource stays in one place from its guard's construction to its destruction, so one pass over active_ptr and nested[] cannot miss it.

    • Nested guards no longer save and restore active_ptr.
    • On nesting deeper than NESTED_DEPTH (3), the innermost guard's resource goes unrecorded (logged once); the old protocol instead lost track of a suspended outer one. Both need four nested signal deliveries on one thread.
    • The nested[] store is SEQ_CST, CallTraceStorage::put()'s re-check load is SEQ_CST, and each scan pass starts with a SEQ_CST fence, so a guard's publication is ordered before its re-check and a drainer's unpublication before its scan.
    • Adds RefCountGuard::isReferenced(), a single non-waiting scan.
  2. Reused uncleared buffer could resurrect a stale id. When clearStandby() skips its target because a guard is still held, the next rotate() makes that buffer active again.

    • How the stale id gets there: a straggler that outlived an earlier rotate()'s drain can have inserted a key into that buffer after both copies, with an id the active buffer has since assigned differently. Phase 1's copyFrom() keeps the existing entry's id, so the stale id won and cached ids could go missing from later constant pools.
    • The fix: StringDictionary now remembers the skipped buffer, and rotate() checks it again with a short series of non-waiting scans, clearing it if no guard remains, before reusing it. rotate() returns false if the buffer is still in use, which rotateDictsAndRun() reports (dictionary_drain_timeouts and a warning).

Motivation:

PROF-16136. Gap 1 is the same use-after-free class as the original ticket.

  • When it happens: a JNI dictionary lookup (for example registerConstant0() → bounded_lookup()) is interrupted by a profiling signal whose handler's put() takes a nested RefCountGuard on a call-trace table. A clearAll() running at that moment could report a clean drain and free storage the lookup was still using.
  • Why Never reclaim dictionary storage after a timed-out drain #839 exposed it: Never reclaim dictionary storage after a timed-out drain #839 moved clearAll() from the global drain, which saw count > 0 and waited, to the targeted drain. The single-target waitForRefCountToClear() (used by CallTraceStorage and rotate()) already had the gap.

Gap 2 needs a straggler insert that stalls across a whole dump cycle to matter. The retry covers everything short of a thread stuck that long, and that remaining case is now reported.

Additional Notes:

rotate() now returns bool. It is not [[nodiscard]], so existing test and fuzz callers are unchanged.

How to test the change?:

  • RefCountGuardScanTest.SeesOuterResourceAcrossNestedGuards: a thread holds an outer guard and repeatedly opens and closes a nested one, while the test scans with isReferenced() for 500 ms.
    • On main: 0.1–1.5% of scans miss the outer resource (34–217 misses per run, debug and release).
    • With the fix: 0 misses.
  • RefCountGuardScanTest.NestedGuardsNeverMoveProtectedResources: checks deterministically the property that rules out torn scans. Across repeated two-level nesting, active_ptr stays the outer resource and each nested resource stays in its own nested[] entry.
  • StringDictionaryReclamationTest.ReusedUnclearedBufferKeepsCurrentIds: reproduces the straggler and skipped-clear sequence.
    • Without the fix: the reused buffer returns the stale id (999999 instead of 2).
    • With the fix: it returns the current id.
  • Ran: full gtestDebug; refCountGuard_ut in debug, release, ASan and TSan (×3); stringDictionary_ut and test_callTraceStorage under ASan and TSan; compileFuzzer; testDebug (225 tests, 0 failures).

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16136

🤖 Generated with Claude Code

rkennke requested a review from a team as a code owner October 8, 2026 15:32

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Successive nested guards can evade both active-pointer reads, allowing the new retry-clear path to free dictionary storage while an outer accessor still holds its guard.

Open Bits AI session

🤖 Bits Code Review · Commit 4724376 · @DataDog review to ask questions

dd-octo-sts Bot commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #37912237452 | Commit: 45e2bd7 | Duration: 17m 14s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-10-09 09:56:12 UTC

dd-octo-sts Bot commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 2d4139ae

This comment has been minimized.

rkennke commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Self-review. I did a thorough review of this PR. Findings and how each is resolved, fixes in 7c24b1c:

  1. Nested guard publication had no store→load ordering (refCountGuard.cpp). A nested guard stored its nested[] entry with RELEASE after count++, and the caller's re-check load could complete before that store was visible (x86 store buffer; on arm64 an acquire load may pass a release store). A drainer that cleared the re-checked pointer and then scanned could miss both. This gap pre-dates the PR (the old code stored active_ptr the same way). It is not reachable today, because nested guards come from signal-handler put()s, which lockAll() keeps away from reclamation.
    Fixed: the nested[] store and CallTraceStorage::put()'s re-check load are now SEQ_CST, and each scan pass starts with a SEQ_CST fence. The root path relies on its count++ RMW after the active_ptr store, and a comment now says so.
  2. A stuck straggler cost two full drains per cycle. rotate()'s retry ran a second ~500 ms wait right after clearStandby()'s, on the dump thread.
    Fixed: the retry is now a short series of non-waiting isReferenced() scans. The straggler has had a whole dump cycle to finish, and no new guard can reach a non-active buffer. New test RotateGivesUpQuicklyOnBufferStillInUse: it fails with the full-drain retry (the whole test took 1.9 s) and passes with the fix.
  3. Stale rotateDictsAndRun comment claiming rotation needs no external lock. Fixed: it now says rotate(), clearStandby() and clearAll() share the skipped-buffer record and are serialised by _state_lock.
  4. Duplicated drain-and-clear logic in clearStandby() and the retry. Fixed: both use finishClear().
  5. A reused uncleared buffer keeps the stale id until clearAll(). This is a deliberate trade-off, now stated in rotate() and in StringDictionary.md: overwriting the stale id would orphan whatever the straggler recorded under it. Either way it takes a thread stuck in a guarded insert for a whole dump cycle, and that case is reported.
  6. isReferenced() is public but only tests used it. It is now the retry's check, and its doc says that a single scan is not a drain.
  7. On nesting overflow, the unrecorded guard is the innermost (running) one, where the old protocol lost track of a suspended outer one. I kept this, and documented it accurately instead of calling it "equivalent": recording the innermost guard by overwriting an entry would move a protected resource again, which is what reintroduced torn scans.

Ran: full gtestDebug; refCountGuard_ut in release, ASan and TSan; stringDictionary_ut and test_callTraceStorage under ASan and TSan; compileFuzzer; testDebug (228 tests, 0 failures).

rkennke and others added 3 commits October 9, 2026 09:24
slotReferences() read a slot's active_ptr, then its outer_stack. A
reentrant guard destructor moves the outer resource from outer_stack
back to active_ptr, storing it before clearing the stack entry, so a
scan that read active_ptr before the store and outer_stack after the
clear saw the resource in neither place. A dictionary lookup interrupted
by a signal handler's put() could therefore be missed by the targeted
drain, and clearAll() could free storage the lookup was still using.
Read active_ptr again after the stack; the ACQUIRE load of the cleared
entry guarantees the second read sees the restored pointer. The same
scan backs waitForRefCountToClear(), so its callers benefit too. Add
RefCountGuard::isReferenced(), a single non-waiting scan.

A buffer whose clearStandby() was skipped became the active buffer on
the next rotate() with its old entries. A straggler can have inserted a
key there after both copies of an earlier rotate(), with an id the
active buffer has since assigned differently, and Phase 1's copyFrom()
kept the stale id. StringDictionary now remembers the skipped buffer
and rotate() retries the drain and clear before reusing it; it returns
false if the buffer is still in use, which rotateDictsAndRun() reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The re-read added to slotReferences() closed the miss across one nested
guard ending, but not across one ending and the next starting: the outer
resource moved outer_stack -> active_ptr -> outer_stack between the
scanner's reads, so it could be absent from every location the scanner
read although the outer guard held it throughout.

Stop moving it. A reentrant guard now records its own resource in
nested[depth - 1] (formerly outer_stack) and leaves active_ptr, which
keeps the root guard's resource. Every protected resource stays in one
location from its guard's construction to its destruction, so a single
pass over active_ptr and nested[] cannot miss it, and the re-read is
dropped. Nested guards no longer save and restore active_ptr, and the
destructor and move assignment share release().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Self-review of the guard-scan fix.

A nested guard stored its nested[] entry with RELEASE after count++, and
the caller's re-check load could complete before that store was visible
(x86 store buffer; arm64 lets an acquire load pass a release store). A
drainer that cleared the re-checked pointer and then scanned could see
neither the entry nor a changed pointer. Make the nested[] store and
CallTraceStorage::put()'s re-check load SEQ_CST, and start each scan
pass with a SEQ_CST fence for the drainer side. The root path already
gets the ordering from the count++ RMW after its active_ptr store.

rotate()'s retry of a buffer whose clear was skipped now makes a short
series of non-waiting isReferenced() scans instead of a full drain, so a
stuck straggler no longer costs a second ~500ms timeout per cycle on the
dump thread. It shares the clear-or-remember step with clearStandby().

Say that rotate() relies on _state_lock now that it shares the skipped
buffer record with clearStandby() and clearAll(), that isReferenced() is
not a drain, which guard goes unrecorded on nesting overflow, and that
reusing an uncleared buffer keeps the stale id by choice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rkennke force-pushed the fix/prof-16136-guard-scan-torn-read branch from 7c24b1c to 2d4139a Compare October 9, 2026 09:35
rkennke merged commit 148bae7 into main Oct 9, 2026
116 checks passed
rkennke deleted the fix/prof-16136-guard-scan-torn-read branch October 9, 2026 10:37
github-actions Bot added this to the 1.52.0 milestone Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant


Back | FazBrowse Home | New Git URL