FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Preventing fault-injection and sampler perf counting from leaking into production build by zhengyu123 · Pull Request #832 · DataDog/java-profiler · GitHub

Repository navigation

Preventing fault-injection and sampler perf counting from leaking into production build - #832

Open
zhengyu123 wants to merge 4 commits into
mainfrom
zgu/check_release_build
Open

zhengyu123 wants to merge 4 commits into
mainfrom
zgu/check_release_build

Conversation

zhengyu123 commented Sep 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?:
Adds a release-build guard, .gitlab/scripts/check-release-flags.sh. It fails the build if libjavaProfiler.so was compiled with either of the opt-in flags -PenableFaultInjection or -PenableSamplerPerf. build.sh runs it on every target, right after copying the native libs into libs/ and before the ABI floor check.

The script looks for markers that only exist when one of the flags is on:

  • Symbols (from nm, mangled names, so every overload, constructor and destructor matches):
    • _ZN8faultinj (the faultinj:: namespace)
    • _Z8crashNowv (crashNow())
    • _ZN16SamplerPerfProbe (the SamplerPerfProbe class)
  • Strings (from strings): the faults_injected, sampler_ticks.* and sampler_count.* counter names. Counters::describeCounters() keeps these in .rodata, so the check still works when the symbols get inlined away.

If nm finds no symbols, or fails, the script reports an error rather than a pass, so a wrong path or an unreadable file can't slip through.

Motivation:
Both flags are additive: they add -D__FAULT_INJECTION__ / -D__SAMPLER_PERF__ on top of the normal release config. A build made with either one still links, runs and passes every existing check. Fault injection deliberately corrupts memory reads on a random sample of calls, and sampler-perf adds probes and a timing report at Profiler::stop(). Shipping either to customers would be a silent correctness or performance regression, and nothing in the pipeline caught it before this PR.

Additional Notes:

  • I chose marker matching over diffing against a golden symbol list. Because the flags are additive, a golden list would break on every unrelated symbol change.
  • SamplerPerf (without Probe) is in every build, since its disabled variant ships by default. The clean test fixture includes it to show the check doesn't over-match.
  • The check relies on the release .so keeping its symbol table. The Gradle link task (NativeLinkTask.kt) only runs strip --strip-debug, so the table survives. (The script comment says build.sh does the stripping; it's actually the Gradle task. That's a small comment fix.)
  • crashNow() is also compiled into DEBUG builds, but release builds don't define DEBUG.
  • NM / STRINGS can be overridden, which lets the unit tests feed in canned output.

How to test the change?:

  • New unit tests in .gitlab/scripts/tests/check_release_flags_test.sh use stub nm/strings tools, so they don't need a compiler or a real flagged build. They cover:
    • a clean build passes
    • each symbol and string marker fails, and the message names the marker and the flag
    • an empty or unreadable symbol table is rejected
    • a missing file or missing argument is rejected
  • These tests now run in the GitHub Actions script-lint job (bash -n + shellcheck + run), alongside the ABI floor tests.
  • Locally: bash .gitlab/scripts/tests/check_release_flags_test.sh
  • End to end: build with -PenableFaultInjection (or -PenableSamplerPerf) and run .gitlab/scripts/check-release-flags.sh ddprof-lib/build/.../libjavaProfiler.so. It should fail, and a normal release build should pass.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16102

Unsure? Have a question? Request a review!

dd-octo-sts Bot commented Sep 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 7a98b77b

dd-octo-sts Bot commented Sep 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #37464127402 | Commit: 008fa1b | Duration: 17m 53s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-10-06 12:55:31 UTC

zhengyu123 changed the title Preventing fault-injection and sampler perf counting in production build Preventing fault-injection and sampler perf counting from leaking into production build Oct 1, 2026
zhengyu123 marked this pull request as ready for review October 1, 2026 19:58
zhengyu123 requested a review from a team as a code owner October 1, 2026 19:58

chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca11b8383a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

More details

The release guard runs for every target and fails closed when symbol inspection is unavailable; the completed static review found no reportable defect.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit ca11b83 · @DataDog review to ask questions

This comment has been minimized.

chatgpt-codex-connector Bot commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-06T12:37:31.589147Z 7a98b77 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

rkennke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Thanks, this is a useful guard. I checked the markers against the source: faultinj:: and the counter names exist only under the flag defines, crashNow() also only under DEBUG (which release doesn't define), and --strip-debug keeps .symtab. I didn't find a way for a flagged build to slip through today. My inline comments are mostly about the misleading strip comment, how much detection there really is, and test gaps.

One related point that isn't in the diff, so I can't comment on it inline: ConfigurationPresets.kt:46 and :56 use project.hasProperty("enableFaultInjection") / hasProperty("enableSamplerPerf"). That is true even for -PenableFaultInjection=false, or for enableFaultInjection=false in gradle.properties or ORG_GRADLE_PROJECT_* env vars. So someone who explicitly turns the flag off still gets -D__FAULT_INJECTION__. That trap is a likely way for these flags to end up on in a release by accident. I'd suggest (maybe as a follow-up) parsing the value as a boolean, and failing the release build at Gradle configuration time if either flag is set. This script would stay as the last line of defence after the full compile.

# e.g. check-release-flags.sh libs/linux-x64/libjavaProfiler.so
#
# Relies on the release .so still carrying its symbol table: build.sh strips
# only debug sections (--strip-debug), not the symbol table itself, before

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This (and the error text at line 68-69, "see build.sh") says build.sh does the --strip-debug, but build.sh doesn't strip anything. The strip comes from Gradle's NativeLinkTask.stripLibrary (strip --strip-debug on Linux, strip -S on macOS). The whole guard depends on .symtab surviving. So whoever later changes that task to --strip-all/--strip-unneeded needs to find this dependency, and this comment sends them to the wrong file. Could you point it at NativeLinkTask.kt instead? Ideally also add a back-reference comment next to stripLibrary saying this check relies on the symtab.

SYMBOL_MARKERS=(
'_ZN8faultinj::-PenableFaultInjection::the faultinj:: namespace (faultInjection.h)'
'_Z8crashNowv::-PenableFaultInjection::crashNow() (faultInjection.h)'
'_ZN16SamplerPerfProbe::-PenableSamplerPerf::the SamplerPerfProbe class (samplerPerf.h)'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I doubt this marker ever fires on a real -O3 release build. SamplerPerfProbe is header-only, with an inline ctor/dtor, so I wouldn't expect any out-of-line _ZN16SamplerPerfProbe* symbol to be emitted. The test fixture's _ZN16SamplerPerfProbeD2Ev (t) models something a real build probably doesn't produce. In practice that leaves the sampler_ticks./sampler_count. strings as the only real detection for sampler-perf, while the tests suggest two independent layers. Could you check nm on an actual -PenableSamplerPerf release .so? If the symbol isn't there, either drop the marker or replace it with something that's guaranteed to survive.

# constructor and destructor without needing c++filt.
SYMBOL_MARKERS=(
'_ZN8faultinj::-PenableFaultInjection::the faultinj:: namespace (faultInjection.h)'
'_Z8crashNowv::-PenableFaultInjection::crashNow() (faultInjection.h)'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This marker isn't tested on its own. The nm-faultinj fixture has crashNow and three _ZN8faultinj symbols, so deleting or mistyping this entry leaves the suite green. Yet it's exactly the marker that would catch a build where the faultinj:: helpers got inlined or gc-section'd away. Could you add a fixture with only _Z8crashNowv?

'sampler_count.::-PenableSamplerPerf::the "sampler_count.*" counters (counters.h)'
)

SYMTAB=$("${NM}" "${SO}" 2>/dev/null) || die "${NM} failed on ${SO}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Sending nm's stderr to /dev/null drops the useful part when it fails, e.g. "file format not recognized" from a host-only nm on an aarch64 .so, or a truncated file. CI would then show just "nm failed on …". I'd let stderr through, or capture it and include it in the die message.

exit 1
fi

STRTAB=$("${STRINGS}" "${SO}") || die "${STRINGS} failed on ${SO}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This path isn't tested. The stub defines FAKE_STRINGS_FAIL (test line 33), but no test case sets it. The command -v "tool not found" guards (lines 48-49) aren't covered either. Since the string markers are effectively the only reliable sampler-perf detection (see my comment on line 56), a regression that turned a strings failure into an empty STRTAB would silently disable them. Worth adding a test case for both.

fi
done

for entry in "${STRING_MARKERS[@]}"; do

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Nit: this loop is a copy of the one above. The parsing, grep and FAIL message are the same; only the array and the text searched differ. A small check_markers "$haystack" "${ARR[@]}" helper would keep the two from drifting apart, e.g. if one ever loses the SIGPIPE-safe here-string. Minor, too: each here-string re-copies the multi-MB nm/strings output, six times in total. Writing each output to a file once and running a single grep -F -f would avoid that. Not a big deal at CI scale.

kaahos left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

thanks for the work! Apart from the comments left by Roman, I have nothing else to report. Please address them first, but otherwise this looks good to me!

Comment on lines +53 to +57
SYMBOL_MARKERS=(
'_ZN8faultinj::-PenableFaultInjection::the faultinj:: namespace (faultInjection.h)'
'_Z8crashNowv::-PenableFaultInjection::crashNow() (faultInjection.h)'
'_ZN16SamplerPerfProbe::-PenableSamplerPerf::the SamplerPerfProbe class (samplerPerf.h)'
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Related to @rkennke's check of the markers against the source: right now that check only happens by hand. Nothing keeps these arrays in sync with the code they detect, or with the list of flags:

  • New flags aren't covered: if someone adds a 3rd hasProperty("enable…") next to ConfigurationPresets.kt:56, this script never checks for it, and a release build with that flag passes.
  • Renames are unnoticed: the test fixtures (check_release_flags_test.sh:52-83) are copies of these markers; the suite never reads the source. Example: if "sampler_ticks.cpu" in counters.h is renamed, the suite stays green while sampler-perf detection falls back to the compiler-dependent SamplerPerfProbe symbol, or to nothing.

This branch has not been deployed

No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants


Back | FazBrowse Home | New Git URL