Fix recurring nightly sanitized failures (UBSan null-pc, fuzz harness compile, jmethodID churn test flake) by jbachorik · Pull Request #833 · DataDog/java-profiler · GitHub

What does this PR do?:
Fixes the three failure classes recurring in the Nightly Sanitized Run (e.g. https://github.com/DataDog/java-profiler/actions/runs/36663570752):

UBSan: applying non-zero offset to null pointer — attributionPC() (stackWalker.inline.h) does (char*)pc - 1 for return-address pcs. An optimistic unwind can read a zeroed return-address slot, so pc == nullptr with pc_is_return_address == true reaches it, UBSan (asan config) reports the error and the test JVM exits 1, killing every run-slow-test-asan job. Fix: pass a null pc through unchanged (behavior-identical to pre-PROF-15955 walkers — findLibraryByAddress(nullptr) fails either way).
Fuzz harness no longer compiles — 9010c4c3a switched CallTraceSet to CountingAllocator, but the fuzz_callTraceStorage.cpp lambda still declared const std::unordered_set<CallTrace*>& (default allocator), which is not convertible to std::function<void(const CallTraceSet&)>. compileFuzz_callTraceStorage has failed every nightly since. Fix: use const CallTraceSet&.
JMethodIDInvalidationStressTest flake (graal/musl/glibc, JDK 21/25) — jmethodid_skipped_count accumulates over the whole churn window (dumps and background JFR flushes), but the <unloaded> label assertion read only the last dump file; the stale trace can be evicted from the call-trace storage before the final dump. Fix: snapshot the dump whose counter window fired (the increment and the <unloaded> label are emitted in the same fillJavaMethodInfo call, so that recording is guaranteed to carry the label); if the counter only fired between dumps, take one extra dump before stop().

The cache-jdks / cache-amd64-musl failure in the same run is an Alpine CDN TLS infra flake, untouched.

Motivation:
Nightly Sanitized Run failing regularly for the last few weeks, masking real regressions. The fuzz failure also showed that a job failure can coexist with a run-level "success" conclusion, so per-job status is the reliable signal.

Additional Notes:

Verified locally (macOS): UBSan mini-repro reproduces the exact CI error unguarded and is clean with the guard; testSlowDebug JMethodID test passes; full debug gtest suite (530 tests) green; spotless clean.
Verified on Linux (workspace-jb): :ddprof-lib:fuzz:compileFuzz_callTraceStorage BUILD SUCCESSFUL; asan-config gtests (walkVmAttribution_ut, returnAddressAttribution_ut, stackWalker_ut, test_callTraceStorage) green; JMethodID test 1 + 4 --rerun runs all green (test executes, not skipped).

How to test the change?:

./gradlew :ddprof-lib:fuzz:compileFuzz_callTraceStorage (was failing on Linux CI)
./gradlew :ddprof-lib:gtestAsan_walkVmAttribution_ut :ddprof-lib:gtestAsan_returnAddressAttribution_ut :ddprof-lib:gtestAsan_stackWalker_ut :ddprof-lib:gtestAsan_test_callTraceStorage
./gradlew :ddprof-test:testSlowDebug -Ptests=JMethodIDInvalidationStressTest (repeat with --rerun)

For Datadog employees:

If this PR touches code that signs or publishes builds or packages, or handles
credentials of any kind, I've requested a security review (run the dd:platform-security-review
skill, or file a request via the PSEC review form).
bewaire also runs automatically on every PR.
This PR doesn't touch any of that.
JIRA: [JIRA-XXXX]

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 7fe81d9f

CI Test Results

Run: #37610327351 | Commit: f5a97ff | Duration: 17m 33s (longest job)

✅ All 76 test jobs passed

Status Overview

JDK	glibc-aarch64/asan	glibc-aarch64/debug	glibc-aarch64/tsan	glibc-amd64/asan	glibc-amd64/debug	glibc-amd64/tsan	musl-aarch64/debug	musl-amd64/debug
8	-	-	-	✅	✅	✅	-	-
8-ibm	-	-	-	✅	✅	✅	-	-
8-j9	✅	✅	✅	✅	✅	✅	-	-
8-librca	-	-	-	-	-	-	✅	✅
8-orcl	-	-	-	✅	✅	✅	-	-
11	-	-	-	✅	✅	✅	-	-
11-j9	✅	✅	✅	✅	✅	✅	-	-
11-librca	-	-	-	-	-	-	✅	✅
17	✅	✅	✅	✅	✅	✅	-	-
17-graal	✅	✅	✅	✅	✅	✅	-	-
17-j9	✅	✅	✅	✅	✅	✅	-	-
17-librca	-	-	-	-	-	-	✅	✅
21	✅	✅	✅	✅	✅	✅	-	-
21-graal	✅	✅	✅	✅	✅	✅	-	-
21-librca	-	-	-	-	-	-	✅	✅
25	✅	✅	✅	✅	✅	✅	-	-
25-graal	✅	✅	✅	✅	✅	✅	-	-
25-librca	-	-	-	-	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 76 | Passed: 76 | Failed: 0

Updated: 2026-10-07 11:13:19 UTC

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-30T15:07:37.351599Z	fbfa879	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@rkennke Not sure if these changes are not redoing some things from your currently open PRs, let's wait until they are merged to see if this still makes sense.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fbfa879569

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The retained first counter-firing dump can contain the stale frame only in datadog.ObjectSample, but the assertion ignores that event type, so the nightly stress test can still fail even when the intended <unloaded> label was emitted.

Open Bits AI session

🤖 Bits Code Review · Commit fbfa879 · @DataDog review to ask questions

Inspect ObjectSample in the selected snapshot

When the first counter increase comes from an allocation trace, the retained dump may contain the stale frame only in datadog.ObjectSample. The assertion instead scans nonexistent datadog.AllocationSample, causing a false failure even though the expected <unloaded> label was emitted.

Was this helpful? React 👍 or 👎
🤖 Bits Code Review · @DataDog review to ask questions · Open Bits AI session

looks good to me; thanks for the fix!

…d_count The counter delta spans the whole churn window (dumps and background JFR flushes) while the label assertion read only the last dump file; a stale trace can be evicted from the call-trace storage before the final dump, making the test flaky across JDKs/platforms. Snapshot the dump whose window observed the counter crossing and assert on it; if the counter only fired between dumps, take one more dump before stop.

jbachorik added AI test:asan Run CI tests with AddressSanitizer configuration test:tsan Run CI tests with ThreadSanitizer configuration test:fuzz labels Sep 30, 2026

jbachorik marked this pull request as ready for review September 30, 2026 15:01

jbachorik requested a review from a team as a code owner September 30, 2026 15:01

jbachorik requested a review from rkennke September 30, 2026 15:01

chatgpt-codex-connector Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread ddprof-lib/src/main/cpp/stackWalker.inline.h Outdated

datadog-prod-us1-6 Bot reviewed Sep 30, 2026

View reviewed changes

kaahos approved these changes Sep 30, 2026

View reviewed changes

jbachorik force-pushed the fix/nightlies_sanitized branch from fbfa879 to 7fe81d9 Compare October 5, 2026 08:55

jbachorik force-pushed the fix/nightlies_sanitized branch from 7fe81d9 to d37807a Compare October 7, 2026 10:52

jbachorik merged commit eb93874 into main Oct 7, 2026
139 of 152 checks passed

jbachorik deleted the fix/nightlies_sanitized branch October 7, 2026 11:00

github-actions Bot added this to the 1.52.0 milestone Oct 7, 2026

+                    // With skippedDelta > 0, firedDumpFile is always set: either the dump whose window
+                    // observed the counter crossing (label emitted in that same fillJavaMethodInfo call)
+                    // or the extra post-churn dump taken above.
+                    assertUnloadedFrameLabel(firedDumpFile);

Fix recurring nightly sanitized failures (UBSan null-pc, fuzz harness compile, jmethodID churn test flake) by jbachorik · Pull Request #833 · DataDog/java-profiler · GitHub

Repository navigation

Fix recurring nightly sanitized failures (UBSan null-pc, fuzz harness compile, jmethodID churn test flake) - #833

Fix recurring nightly sanitized failures (UBSan null-pc, fuzz harness compile, jmethodID churn test flake)#833
jbachorik merged 1 commit into
mainfrom
fix/nightlies_sanitized

jbachorik commented Sep 30, 2026

Uh oh!

dd-octo-sts Bot commented Sep 30, 2026 •
edited
Loading

Uh oh!

Uh oh!

dd-octo-sts Bot commented Sep 30, 2026 •
edited
Loading

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Sep 30, 2026 •
edited
Loading

Uh oh!

Uh oh!

jbachorik commented Sep 30, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

datadog-prod-us1-6 Bot left a comment

Uh oh!

datadog-prod-us1-6 Bot Sep 30, 2026

Uh oh!

kaahos left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Fix recurring nightly sanitized failures (UBSan null-pc, fuzz harness compile, jmethodID churn test flake) by jbachorik · Pull Request #833 · DataDog/java-profiler · GitHub

Repository navigation

Fix recurring nightly sanitized failures (UBSan null-pc, fuzz harness compile, jmethodID churn test flake) - #833

Fix recurring nightly sanitized failures (UBSan null-pc, fuzz harness compile, jmethodID churn test flake)#833jbachorik merged 1 commit intomainDataDog/java-profiler:mainfrom fix/nightlies_sanitizedDataDog/java-profiler:fix/nightlies_sanitizedCopy head branch name to clipboard

Conversation

jbachorik commented Sep 30, 2026

Uh oh!

dd-octo-sts Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI Test Results

Status Overview

Uh oh!

chatgpt-codex-connector Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

jbachorik commented Sep 30, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

datadog-prod-us1-6 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-prod-us1-6 Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

kaahos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Fix recurring nightly sanitized failures (UBSan null-pc, fuzz harness compile, jmethodID churn test flake)#833
jbachorik merged 1 commit into
mainfrom
fix/nightlies_sanitized

dd-octo-sts Bot commented Sep 30, 2026 •
edited
Loading

dd-octo-sts Bot commented Sep 30, 2026 •
edited
Loading

chatgpt-codex-connector Bot commented Sep 30, 2026 •
edited
Loading