| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
Go services shipped as bare binaries in distroless/scratch images are invisible to 'fossa container analyze': container analysis is static-only, Go strategies are excluded, and no binary inspection exists. Go binaries embed their full module list (what 'go version -m' reads), so the data is sitting in the artifact. - millhone: new go_buildinfo parser (Go >= 1.18 inline format only; older and packed binaries skip gracefully). Wired into the existing analyze-container layer walk behind a cheap magic-byte sniff; output gains an additive 'discovered_go_binaries' field. - millhone: transparently gunzip layer blobs. Modern 'docker save' emits OCI layout with gzipped layers, which also silently broke JAR-in-container analysis when run against such archives directly. - CLI: convert discovered modules into real 'go+' locators in per-layer srcUnits (new App.Fossa.Container.Sources.GoBinary), normalizing pseudo-versions to commit hashes exactly like go.mod scans (Strategy.Go.Gomod.parsePackageVersion). No backend changes required. Verified end-to-end on scratch and chainguard/static images containing a Go binary with known modules: previously empty scans now report the full module list; apk system deps and layer placement unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XzWtfpTqJtLyL48WfB35gG
Go binary discovery in 'fossa container analyze' is now opt-in via '--experimental-enable-go-binary-discovery', following the precedent of 'fossa analyze --experimental-enable-binary-discovery'. - New GoBinaryDiscovery flag threaded from ContainerAnalyzeConfig through scanImage and all four container sources into analyzeFromDockerArchive, where discovered Go binary modules are only merged into layer source units when the flag is set. Millhone still runs either way (it is already invoked for JAR analysis and the Go sniff is a cheap magic-byte check); when it finds Go binaries with the flag off, the CLI logs a hint naming the flag. - Docs: new "Container Go binary analysis (experimental)" section in the container scanner reference (mechanism, usage, Go >= 1.18 and main-module limitations), flag usage in the container subcommand reference, support-table row, and a changelog entry. Verified: 'cabal build lib:spectrometer' clean; container unit suite passes 80/80 examples including the 7 GoBinarySpec examples and the JAR observation tests that exercise the gated code path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJWNhwmzxVM6kiZZGNFmTQ
…nary-analysis # Conflicts: # Changelog.md # src/App/Fossa/Container/AnalyzeNative.hs # src/App/Fossa/Container/Scan.hs # src/App/Fossa/Container/Sources/DockerArchive.hs # src/App/Fossa/Container/Sources/DockerEngine.hs # src/App/Fossa/Container/Sources/Podman.hs # src/App/Fossa/Container/Sources/Registry.hs # test/App/Fossa/Container/AnalyzeNativeSpec.hs
fill_buf performs a single read and only guarantees one byte, so a short read could misclassify a gzipped layer as plain tar. Read exactly two bytes and chain them back onto the stream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XRURX4RiBVsFYgeDCLbvWN
Remove the rebuild-trigger note in EmbeddedBinary.hs and a comment restating the loop below it; trim doc comments that enumerated their function bodies; fix a doubled-word typo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XRURX4RiBVsFYgeDCLbvWN
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XRURX4RiBVsFYgeDCLbvWN
- scan_layer: gzip and plain layers produce identical results; empty and zero-byte layers scan cleanly (covers the magic-bytes stitch-back) - maybe_go_binary: size gates, non-file entries, non-binary files, and binaries without buildinfo all skip without failing the scan - parse_go_buildinfo: parse a buildinfo region carved from a released binary, since synthetic fixtures share their encoder with the parser - GoBinary.hs: pin +incompatible normalization, the go-install main module case, and locator dedup Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XRURX4RiBVsFYgeDCLbvWN
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XRURX4RiBVsFYgeDCLbvWN
WalkthroughThis change adds experimental Go binary discovery to container analysis. The scanner detects supported binaries in image layers, parses embedded Go buildinfo, and returns module metadata per layer. The CLI flag propagates through container analysis paths. Discovered modules convert to gobinary source units with normalized locators. Tests cover parsing, scanning, compatibility, conversion, and filtering. Documentation and changelog entries describe the feature and its limitations. 🚥 Pre-merge checks | ✅ 5 ✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. ❤️ ShareComment @coderabbitai help to get the list of available commands. |
Sorry, something went wrong.
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agentsVerify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Inline comments: In `@extlib/millhone/src/cmd/analyze_container.rs`: - Around line 171-174: Update the scan_layer_tar flow to enforce a cumulative decompressed-byte budget for both gzip and uncompressed layers, propagating an error when the limit is exceeded. Replace full buffering via entry.read_to_end before parse_go_buildinfo with bounded, streaming build-info parsing that reads only the required data and preserves MAX_GO_BINARY_SIZE safeguards. In `@extlib/millhone/src/cmd/go_buildinfo.rs`: - Around line 55-57: Update is_candidate_binary() to recognize the 64-bit Mach-O universal magic values 0xcafebabf and 0xbebaface in both big-endian and little-endian byte orders, alongside the existing 32-bit Mach-O constants. - Around line 127-136: Update read_uvarint to validate the tenth byte before applying its shifted value: when i == 9, reject only byte values greater than 1, while allowing the valid terminating 0x00 and 0x01 cases. Preserve normal termination and byte-count reporting, and add coverage for valid ten-byte values plus overflowing tenth-byte values. In `@src/App/Fossa/Container/Sources/DockerArchive.hs`: - Around line 20-22: Qualify the listed Haskell imports and update all matching references: in src/App/Fossa/Container/Sources/DockerArchive.hs lines 20-22 qualify App.Fossa.Config.Container.Analyze, App.Fossa.Container.Sources.Discovery, and App.Fossa.Container.Sources.GoBinary, including GoBinaryDiscovery, layerAnalyzers, renderLayerTarget, and goBinariesToSourceUnits; apply the same qualified-import and reference updates for App.Fossa.Config.Container.Analyze in src/App/Fossa/Container/Scan.hs line 14, src/App/Fossa/Container/Sources/DockerEngine.hs line 10, src/App/Fossa/Container/Sources/Podman.hs line 11, and src/App/Fossa/Container/Sources/Registry.hs line 11. In `@src/App/Fossa/Container/Sources/GoBinary.hs`: - Around line 10-21: Use qualified imports for the newly added Haskell modules in GoBinary.hs, Container/Types.hs, and GoBinarySpec.hs, updating references as needed while preserving behavior. Run Fourmolu on all three affected files after the import changes. - Around line 79-86: Rewrite normalizeVersion without guarded equations for the empty and "(devel)" cases, using pattern matching or conditional expressions instead. Preserve the existing parseMaybe (parsePackageVersion id) branches and their current outputs unchanged.
Fix all unresolved CodeRabbit comments on this PR:
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: e1768e99-1ce6-447f-803a-c9e3bc5de1e9
📥 CommitsReviewing files that changed from the base of the PR and between d693e0d and 36689ec.
⛔ Files ignored due to path filters (2)
Sorry, something went wrong.
| #[tracing::instrument] | ||
| fn jars_in_container(image_path: &PathBuf) -> Result<JarAnalysis> { | ||
| // Visit each layer and fingerprint the JARs within. | ||
| fn jars_in_container(image_path: &PathBuf) -> Result<ContainerAnalysis> { |
There was a problem hiding this comment.
This name and its doc string is no longer accurate.
Sorry, something went wrong.
There was a problem hiding this comment.
Renamed to binaries_in_container in 62709d3.
Sorry, something went wrong.
| Err(e) => return Err(e).context("detect layer compression"), | ||
| } | ||
| } | ||
| let is_gzip = filled == 2 && magic == [0x1f, 0x8b]; |
There was a problem hiding this comment.
It'd be best to put these in a const/static array with a descriptive name.
Sorry, something went wrong.
There was a problem hiding this comment.
Done in 62709d3 — GZIP_MAGIC.
Sorry, something went wrong.
| // Read exactly two bytes rather than trusting a single fill_buf to return | ||
| // that many, then stitch them back onto the front of the stream. |
There was a problem hiding this comment.
Why wouldn't you trust it? The BufReader is likely reading more than two already.
Sorry, something went wrong.
There was a problem hiding this comment.
The comment was misleading — reworded in 62709d3. It's not about BufReader's internal buffer: a single Read::read may legally return fewer than 2 bytes even when more remain, so the loop guarantees both magic bytes (or EOF) before deciding.
Sorry, something went wrong.
| const MIN_GO_BINARY_SIZE: u64 = 4096; | ||
|
|
||
| /// Skip candidates larger than this rather than buffering them in memory. | ||
| const MAX_GO_BINARY_SIZE: u64 = 512 * 1024 * 1024; |
There was a problem hiding this comment.
[nit] Typically we use usize for sizes. If u64 is what some API requires you can leave these as is.
Sorry, something went wrong.
There was a problem hiding this comment.
Keeping u64 per your parenthetical — it's what tar's Header::size() returns. (The MAX cap is gone entirely as of 62709d3 now that the scan streams; only MIN_GO_BINARY_SIZE remains.)
Sorry, something went wrong.
| if size > MAX_GO_BINARY_SIZE { | ||
| warn!(?path, size, "skipped: candidate binary exceeds size limit"); | ||
| return None; | ||
| } |
There was a problem hiding this comment.
[Question] Is there a way to do this in a streaming fashion where we don't need to keep the whole entry in memory at once?
Sorry, something went wrong.
There was a problem hiding this comment.
Implemented in 62709d3 — single pass, no seeking: scan_go_buildinfo keeps a rolling 64KiB window to find the aligned magic, then reads just the two varint-prefixed strings. Whole-entry buffering for Go binaries is gone, and the 512MiB cap went with it since it only existed to bound buffering.
Sorry, something went wrong.
| warn!(?path, "failed to read candidate binary: {e:?}"); | ||
| return None; | ||
| } | ||
| parse_go_buildinfo(&contents).map(|info| DiscoveredGoBinary::new(path.to_path_buf(), info)) |
There was a problem hiding this comment.
Does parse_go_buildinfo expect to take a file without its header info? Does Entry::read_exact actually advance some internal value? If it does we should do one of the following:
Sorry, something went wrong.
There was a problem hiding this comment.
Good catch — it was a real trap: read_exact does advance the Entry (forward-only reader), and correctness depended on the caller stitching the prefix back on. The streaming rewrite in 62709d3 absorbs that: maybe_go_binary chains the sniffed prefix back onto the stream in one expression, scan_go_buildinfo documents that it needs bytes from file offset 0, and parse_go_buildinfo is no longer pub.
Sorry, something went wrong.
| let mut info = GoBuildInfo { | ||
| go_version: String::from_utf8_lossy(go_version).into_owned(), | ||
| main_module: None, | ||
| modules: Vec::new(), | ||
| }; | ||
| if info.go_version.is_empty() { | ||
| return None; | ||
| } | ||
|
|
||
| if let Some(modinfo) = modinfo { | ||
| parse_modinfo(&String::from_utf8_lossy(modinfo), &mut info); | ||
| } |
There was a problem hiding this comment.
It's generally better not to use mutability when it's so easy to avoid. Make parse_modinfo return its result and then:
let info = {
let version = String::from_utf8_lossy(go_version)
.is_some_and(|s| ! s.is_empty() )?;
let info = parse_modinfo(&String::from_utf8_lossy(modinfo);
GoBuildInfo { ... }
};The reason that I say that is declaring something mut basically means that you have to consider everything that might happen to it after you declare it. That's a lot of mental overhead as opposed to knowing "it's this value and always will be."
In this case, it's perhaps not so bad since the function is small but it's a good habit to exercise everywhere IMO.
Sorry, something went wrong.
There was a problem hiding this comment.
Done in 62709d3 — parse_modinfo returns (main_module, modules) and GoBuildInfo is built in one expression.
Sorry, something went wrong.
| let mut header = [0u8; 64]; | ||
| if let Err(e) = entry.read_exact(&mut header) { | ||
| debug!(?path, "skipped: failed to read header: {e:?}"); | ||
| return None; | ||
| } | ||
| if !is_candidate_binary(&header) { | ||
| return None; | ||
| } |
There was a problem hiding this comment.
What's the difference between this header and entry.header()?
Sorry, something went wrong.
There was a problem hiding this comment.
Just a badly named local — the file's first 64 bytes, nothing to do with entry.header(). Renamed to prefix in 62709d3.
Sorry, something went wrong.
|
|
||
| ## Unreleased | ||
|
|
||
| - Container scanning: Add `--experimental-enable-go-binary-discovery` flag to `fossa container analyze`. When enabled, Go module dependencies embedded in Go binaries (built with Go >= 1.18) found in container image layers are reported as regular Go dependencies, supporting images without package manager metadata such as `scratch` and distroless images ([#1740](https://github.com/fossas/fossa-cli/pull/1740)) |
There was a problem hiding this comment.
Bump this for release.
Sorry, something went wrong.
There was a problem hiding this comment.
Leaving the bump to you since you're stacking your changes before merge — or say the word and I'll push it.
Sorry, something went wrong.
There was a problem hiding this comment.
Done in f6c8aad — bumped to 3.18.0, following the 3.16.0 precedent of minting a minor version for new analysis support.
Sorry, something went wrong.
- scan_go_buildinfo streams with a rolling 64KiB window and reads only the two inline strings once the magic is found, replacing whole-file read_to_end buffering; parse_go_buildinfo is no longer pub - drop the 512MiB candidate size cap: it existed to bound buffering, and streaming made it unnecessary - reject uvarint tenth bytes that would silently overflow - recognize 64-bit Mach-O universal (fat) magics - rename jars_in_container -> binaries_in_container and a stale test name; rename the sniffed `header` buffer to `prefix` - name the gzip magic bytes, make BUILDINFO_MAGIC static, reference the Go buildinfo format's reference implementation, and make parse_modinfo return its result instead of mutating Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XuweLq6YqCxfHE9tZjg21G
The millhone output type now carries Go binaries as well as jars, so the old name undersold it. normalizeVersion loses its guarded equation per the style guide's "Do not use match guards". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XuweLq6YqCxfHE9tZjg21G
Conflicts: Changelog.md (kept both Unreleased entries), extlib/millhone/src/cmd.rs (master's millhone slim-down kept, plus our go_buildinfo module)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XuweLq6YqCxfHE9tZjg21G
New capability (container Go binary discovery) follows the 3.16.0 precedent of minting a minor version for new analysis support. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XuweLq6YqCxfHE9tZjg21G
Per review discussion: the buildinfo scan already runs on every container scan (millhone walks the layers for JAR analysis either way), so the --experimental-enable-go-binary-discovery flag only gated reporting of the discovered deps. Remove the flag and report Go binary deps unconditionally; opting out remains possible via target filtering. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UDWH3uc2N2NdhNJeb1mhXL
|
Note on opting out (correcting a claim from the review discussion): During review we said that anyone not wanting Go binary deps could filter them out by target. Having checked the code, that's not actually true today:
So there is currently no scan-side opt-out. This has direct precedent — JAR-in-container observations are also always-on and bypass filters the same way (analyzeContainerJars takes no filters) — but it's worth stating plainly since the review discussion assumed otherwise. If an opt-out is ever needed, wiring gobinary units through the filter engine (or adding a dedicated exclusion) would be a follow-up change; deliberately not built speculatively here. |
Sorry, something went wrong.
| | Dart (pub) | :warning: | [Dart](./../../strategies/languages/dart/pub.md) | | ||
| | Maven | :warning: | [Maven](./../../strategies/languages/maven/maven.md) | | ||
| | Java Jar Files | :white_check_mark: | [Container Jar Analysis](#container-jar-analysis) | | ||
| | Go Binaries | :white_check_mark: | [Container Go Binary Analysis](#container-go-binary-analysis) | |
There was a problem hiding this comment.
| | Go Binaries | :white_check_mark: | [Container Go Binary Analysis](#container-go-binary-analysis) | | |
| | Golang (Binaries) | :white_check_mark: | [Container Go Binary Analysis](#container-go-binary-analysis) | |
Sorry, something went wrong.
| let discovered = sniff_first(&[("usr/bin/big", &buf)]).expect("should discover"); | ||
| assert_eq!(discovered.go_version, "go1.25.6"); | ||
| assert_eq!(discovered.modules.len(), 161); | ||
| } |
There was a problem hiding this comment.
This all seems good - if it's easy I'd recommend maybe adding an on-disk fixture so you can test it end to end.
Sorry, something went wrong.
|
|
||
| -- | Analyze a container for Jar fingerprints using Millhone. | ||
| analyzeContainerJars :: (Has Logger sig m, Has Exec sig m, Has (Lift IO) sig m, Has Diagnostics sig m) => Path Abs File -> m DiscoveredJars | ||
| analyzeContainerJars :: (Has Logger sig m, Has Exec sig m, Has (Lift IO) sig m, Has Diagnostics sig m) => Path Abs File -> m DiscoveredBinaries |
There was a problem hiding this comment.
[nit] A small comment explaining that this is also where we kick of the Go analysis would be good. Ideally we'd just remove the Jar specific nomenclature altogether. If you want to take a stab at that you can but I can also look at it some other time.
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Summary
Go services shipped as bare binaries in distroless/scratch/chainguard images are currently invisible to fossa container analyze: container analysis is static-only, Go strategies are excluded from the container discovery list, and no binary inspection exists. But Go binaries embed their complete module list (the data go version -m reads), so the dependency data is sitting in the artifact — syft already reads it.
This PR closes that gap end-to-end, prompted by an enterprise customer for whom this is their top-ranked ask (support has a validated repro + escalation context; ask @saramaebee for details):
PoC scope, deliberately deferred: committed container-tar test fixtures, fourmolu pass, and the pre-1.18 pointer-format encoding.
Test plan
Reproducing it yourself
1. Get the PoC binary
Every push to this PR builds release binaries; download the artifact for your platform from the PR's Build workflow run (Artifacts section at the bottom of the run summary), or:
(Other artifact names: Linux-binaries, Linux-arm-binaries, macOS-intel-binaries, Windows-binaries.)
2. Build a repro image
Any image containing a Go >= 1.18 module-built binary works. A minimal one:
(Swap FROM scratch for cgr.dev/chainguard/static to also see system deps alongside the Go modules.)
3. Run the PR binary
Expected: a gobinary source unit for the app binary whose imports include go+github.com/google/uuid$v1.6.0.
4. Controls
🤖 Generated with Claude Code
https://claude.ai/code/session_01XzWtfpTqJtLyL48WfB35gG