FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Protect v2 journal populate walk from SIGBUS by stelfrag · Pull Request #22514 · netdata/netdata · GitHub

Protect v2 journal populate walk from SIGBUS - #22514

Merged
stelfrag merged 15 commits into
netdata:masterfrom
stelfrag:journalfile_v2_fixes
Jun 9, 2026
Merged

Protect v2 journal populate walk from SIGBUS#22514
stelfrag merged 15 commits into
netdata:masterfrom
stelfrag:journalfile_v2_fixes

Conversation

stelfrag commented May 21, 2026
edited by cubic-dev-ai Bot
Loading

Copy link
Copy Markdown
Collaborator
Summary

The startup walk of v2 journal files reads through mmap. If a page is unreadable, the kernel raises SIGBUS.

  • Apply the project's SIGBUS-safe wrapper to the default-on CRC check that runs first, so one bad journal does not crash the agent on every restart.
  • Fix a refcount leak on the CRC-failure early-return.

Summary by cubic

Hardened v2 journal startup, retention, and size-stats walks against SIGBUS by guarding mmap reads, validating header/trailer offsets, copying UUIDs before mrg calls, and tearing down failed journals immediately. Adds clearer error logs for bad headers, keeps protected regions tightly scoped, and prevents crashes on unreadable/corrupt pages while avoiding fd/mmap/index duplication by forcing a clean rebuild.

  • Bug Fixes
    • Wrapped both the CRC scan and MRG update in journalfile_v2_populate_retention_to_mrg() with PROTECTED_ACCESS_SETUP; on signal/CRC/bounds failure, clear JOURNALFILE_FLAG_IS_AVAILABLE under the data spinlock and journalfile_v2_data_unmap_permanently() to rebuild without fd/mmap/index leaks.
    • Tightened v2 header checks: validate metric/trailer offsets and placement against mmap.size, order checks to avoid underflow/32-bit wrap, and pass mmap.size to journalfile_check_v2_metric_list; fixed CRC early-return refcount leak; improved error logging for out-of-range headers.
    • Protected retention updates (update_metrics_first_time_s) with bounds/overflow checks and PROTECTED_ACCESS_SETUP, scoped tightly to the mmap walk; on failure or SIGBUS, release partially acquired metrics and exit cleanly.
    • Copied UUIDs from the mmap to the stack before mrg_* calls to avoid SIGBUS inside mrg and prevent lock-state corruption.
    • Marked uuid_first_entry_list, count, added, and journal_access_failed as volatile to keep setjmp/longjmp recovery defined.
    • Guarded size-stats walk (populate_v2_statistics) with PROTECTED_ACCESS_SETUP and bounds checks for extent/metric lists and per-metric page headers; keep best-effort totals and always release the journal; switched data_start to uint8_t* and clarified the bounded descr->type index for analyzers.
    • Explicitly set has_references = false during v2 journal cleanup to avoid stale reference tracking after failures.

Written for commit c44849e. Summary will update on new commits. Review in cubic

  journalfile_v2_populate_retention_to_mrg() walked the mmap'd v2 journal
  in two phases: an optional CRC check (when JOURNALFILE_FLAG_METRIC_CRC_CHECK
  is set -- the default cheap-load path on every startup) followed by the
  mrg update. Only the mrg update ran inside PROTECTED_ACCESS_SETUP. If a
  backing page could not be paged in (file truncated, sparse hole,
  transient I/O error) the CRC walk hit SIGBUS and the process aborted
  with no recovery handler armed.

  Move PROTECTED_ACCESS_SETUP above the CRC check so one region covers
  both phases. On signal recovery, clear JOURNALFILE_FLAG_IS_AVAILABLE so
  the next pass rebuilds the journal -- same effect as a CRC failure. Drop
  the bare early-return on CRC failure, which leaked the acquire refcount,
  and fall through to the single journalfile_v2_data_release at the end.
  Mirrors the pattern in journalfile_v2_validate() at the load site.

cubic-dev-ai Bot left a comment
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

1 issue found across 1 file

Confidence score: 2/5

  • There is a high-risk teardown bug in src/database/engine/journalfile.c: the SIGBUS recovery path clears IS_AVAILABLE before permanent cleanup, which can prevent journalfile_v2_data_unmap_permanently() from running.
  • This can leave a dangling njfv2idx entry for a destroyed datafile, creating a concrete regression risk in cleanup/state consistency rather than a minor code-quality concern.
  • Given the reported severity (8/10) and solid confidence (7/10), this is likely merge-blocking until the recovery/teardown ordering is fixed.
  • Pay close attention to src/database/engine/journalfile.c - ensure SIGBUS recovery does not bypass permanent unmap and index cleanup.
Architecture diagram
sequenceDiagram
    participant Agent as Netdata Agent
    participant MRG as Main MRG (Metrics Registry)
    participant V2Load as journalfile_v2_populate_retention_to_mrg()
    participant Mmap as mmap'd v2 Journal File
    participant OS as Kernel / SIGBUS Handler
    participant Log as Error Log (Rate-Limited)

    Note over Agent,Log: V2 Journal Startup Walk (SIGBUS-Safe)

    Agent->>V2Load: populate retention from v2 journal
    V2Load->>Mmap: PROTECTED_ACCESS_SETUP (mmap region)
    V2Load->>V2Load: Check no_signal_received flag

    alt no_signal_received = true
        alt CRC check enabled (JOURNALFILE_FLAG_METRIC_CRC_CHECK)
            V2Load->>Mmap: journalfile_check_v2_metric_list()
            alt CRC failure
                V2Load->>V2Load: Clear IS_AVAILABLE flag
                V2Load->>V2Load: Set failed = true
            else CRC success
                V2Load->>V2Load: Continue to MRG update
            end
        end

        alt no_signal_received AND NOT failed
            loop For each metric entry
                V2Load->>Mmap: Read metric UUID, start/end time, update_every
                V2Load->>MRG: mrg_update_metric_retention_and_granularity_by_uuid()
                MRG-->>V2Load: Retention updated
            end
        end
    else no_signal_received = false (SIGBUS/SIGSEGV caught)
        V2Load->>V2Load: Clear IS_AVAILABLE flag
        V2Load->>Log: Rate-limited error (SIGBUS during mmap walk)
        V2Load->>V2Load: Set failed = true
    end

    V2Load->>V2Load: journalfile_v2_data_release() (unified cleanup)
    V2Load-->>Agent: Return (rebuild on next pass if IS_AVAILABLE cleared)
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

  Add immediate unmap of fd/mmap and cleanup of njfv2idx entry when v2 journals fail during critical paths. Prevents leaving journals marked unavailable but partially initialized, which could lead to fd/mmap overwrite and duplicate index entries during rebuilds.
stelfrag marked this pull request as ready for review May 21, 2026 06:15
Copilot AI review requested due to automatic review settings May 21, 2026 06:15
stelfrag marked this pull request as draft May 21, 2026 06:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Make the v2 journal startup mmap-walk resilient to SIGBUS/SIGSEGV so corrupt/unreadable journal pages don’t crash the agent during startup, and ensure failed journals are torn down cleanly to avoid leaks/duplication.

Changes:

  • Wrap the CRC scan + MRG retention walk in PROTECTED_ACCESS_SETUP and mark journals unavailable on SIGBUS/SIGSEGV.
  • Replace the CRC-failure early-return with unified cleanup via journalfile_v2_data_release().
  • When a journal fails validation/walk, immediately journalfile_v2_data_unmap_permanently() to force a clean rebuild and prevent duplicate fd/mmap + index entries.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/database/engine/journalfile.c Outdated
Comment thread src/database/engine/journalfile.c Outdated
…d and ensure clean failure handling

Add bounds checks for header-controlled offsets against the mmap size to prevent corruption-driven over-reads. Centralize teardown logic by clearing IS_AVAILABLE under a spinlock and unmapping permanently, avoiding dangling index entries or fd/mmap leaks during rebuild.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment thread src/database/engine/journalfile.c Outdated
…size wrapping

Refine bounds checks for metric offsets and metric count in v2 journal headers. Avoid underflow and prevent size_t wrapping on 32-bit builds by reordering checks and using division instead of multiplication for capacity validation.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

Comment thread src/database/engine/journalfile.c Outdated
Comment thread src/database/engine/journalfile.c Outdated
Add stricter checks for trailer offsets to detect malformed v2 journal headers and prevent corrupted metric lists. Adjust header-controlled size bounds in CRC checks for future-proofing. Refactor IS_AVAILABLE clearing and refcount release sequence for robust teardown, avoiding dangling mappings or index entries.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

stelfrag marked this pull request as ready for review May 21, 2026 07:54
stelfrag marked this pull request as draft May 21, 2026 09:55
Add PROTECTED_ACCESS_SETUP to safely handle mmap walks during metric retention updates. Ensure clean failure handling and resource release in case of journal access errors.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment thread src/database/engine/rrdengine.c Outdated

cubic-dev-ai Bot left a comment
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

1 issue found across 1 file (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread src/database/engine/rrdengine.c Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread src/database/engine/rrdengine.c Outdated
Comment thread src/database/engine/rrdengine.c Outdated
stelfrag marked this pull request as draft May 21, 2026 16:31
…unds checks

Declare `journal_access_failed` as volatile to ensure safety in recovery scenarios and prevent future invariance breaks. Add checks to avoid overflow in uuid entry list size computations during v2 journal setup.

cubic-dev-ai Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

You're iterating quickly on this pull request. To help protect your rate limits, cubic has paused automatic reviews on new pushes for now—when you're ready for another review, comment @cubic-dev-ai review.

stelfrag requested a review from Copilot May 21, 2026 16:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

Comment thread src/database/engine/journalfile.c Outdated
Comment thread src/database/engine/rrdengine.c Outdated
Comment thread src/database/engine/journalfile.c Outdated
… mmap access

Safeguard `journalfile` and `rrdengine` operations by copying UUIDs to the stack before invoking mrg functions. Ensures clean recovery in protected regions on unreadable backing pages and avoids lock state corruption during SIGBUS errors.
stelfrag requested a review from Copilot May 21, 2026 17:31

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

cubic-dev-ai Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@stelfrag I have started the AI code review. It will take a few minutes to complete.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

…is intent

Enhance error logging for out-of-range v2 journal header offsets, enabling clearer diagnostics during rebuild scenarios. Add a comment clarifying tautological bounds for `descr->type` to assist static analyzers.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread src/database/engine/rrdengine.c Outdated
…ion updates

Reduce nesting depth and mask of unrelated faults by tightly scoping the PROTECTED_ACCESS_SETUP frame to mmap walks only. Ensure proper cleanup and avoid inflated nesting in subsequent operations.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Copy link
Copy Markdown

stelfrag marked this pull request as ready for review May 22, 2026 07:06
stelfrag merged commit b5e92cb into netdata:master Jun 9, 2026
161 checks passed
stelfrag deleted the journalfile_v2_fixes branch June 9, 2026 10:20
stelfrag mentioned this pull request Jun 22, 2026
Ferroin pushed a commit that referenced this pull request Jul 15, 2026
* fix(dbengine): protect v2 journal populate walk from SIGBUS

  journalfile_v2_populate_retention_to_mrg() walked the mmap'd v2 journal
  in two phases: an optional CRC check (when JOURNALFILE_FLAG_METRIC_CRC_CHECK
  is set -- the default cheap-load path on every startup) followed by the
  mrg update. Only the mrg update ran inside PROTECTED_ACCESS_SETUP. If a
  backing page could not be paged in (file truncated, sparse hole,
  transient I/O error) the CRC walk hit SIGBUS and the process aborted
  with no recovery handler armed.

  Move PROTECTED_ACCESS_SETUP above the CRC check so one region covers
  both phases. On signal recovery, clear JOURNALFILE_FLAG_IS_AVAILABLE so
  the next pass rebuilds the journal -- same effect as a CRC failure. Drop
  the bare early-return on CRC failure, which leaked the acquire refcount,
  and fall through to the single journalfile_v2_data_release at the end.
  Mirrors the pattern in journalfile_v2_validate() at the load site.

* fix(dbengine): ensure clean teardown of v2 journal on failure

  Add immediate unmap of fd/mmap and cleanup of njfv2idx entry when v2 journals fail during critical paths. Prevents leaving journals marked unavailable but partially initialized, which could lead to fd/mmap overwrite and duplicate index entries during rebuilds.

* fix(dbengine): validate v2 journal header offsets to prevent over-read and ensure clean failure handling

Add bounds checks for header-controlled offsets against the mmap size to prevent corruption-driven over-reads. Centralize teardown logic by clearing IS_AVAILABLE under a spinlock and unmapping permanently, avoiding dangling index entries or fd/mmap leaks during rebuild.

* fix(dbengine): improve v2 header validation to prevent underflow and size wrapping

Refine bounds checks for metric offsets and metric count in v2 journal headers. Avoid underflow and prevent size_t wrapping on 32-bit builds by reordering checks and using division instead of multiplication for capacity validation.

* fix(dbengine): enhance v2 journal validation and clarify teardown logic

Add stricter checks for trailer offsets to detect malformed v2 journal headers and prevent corrupted metric lists. Adjust header-controlled size bounds in CRC checks for future-proofing. Refactor IS_AVAILABLE clearing and refcount release sequence for robust teardown, avoiding dangling mappings or index entries.

* protect(dbengine): safeguard v2 journal retention updates from SIGBUS

Add PROTECTED_ACCESS_SETUP to safely handle mmap walks during metric retention updates. Ensure clean failure handling and resource release in case of journal access errors.

* fix(dbengine): mark volatile variables in v2 journal access to ensure safety after recovery

Declare uuid_first_entry_list, count, and added as volatile to maintain well-defined behavior across setjmp/longjmp in PROTECTED_ACCESS_SETUP regions.

* protect(dbengine): safeguard v2 journal size statistics walk from SIGBUS

Add PROTECTED_ACCESS_SETUP to handle mmap walks during the collection of size statistics. Ensure clean failure handling and resource release in case of journal access errors.

* fix(dbengine): initialize `has_references` to false in v2 journal setup

Ensure `has_references` is explicitly set to false during v2 journal initialization for consistent reference tracking.

* protect(dbengine): validate v2 journal bounds during size stats walk

Add detailed bounds checks for extent and metric lists in v2 journals to prevent over-reads and ensure safe traversal. Enhance failure handling with consistent resource release on invalid mappings.

* fix(dbengine): use uint8_t* for v2 journal byte arithmetic

populate_v2_statistics() declared data_start as void* and walked the
mmap'd v2 journal with `(void *)(data_start + offset)` pointer
arithmetic. Pointer arithmetic on void* is a GCC extension, undefined
under standards-strict semantics, and CodeQL's
cpp/suspicious-pointer-scaling-void rule flags every such site (alerts
3315, 3317-3320 on this PR).

Declaring data_start as uint8_t* makes the arithmetic well-defined (1
byte per step, exactly what the GCC void* extension produced) and
silences the 5 alerts at no semantic cost. The existing (void *) casts
on the rvalues compile unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(dbengine): mark `journal_access_failed` as volatile and extend bounds checks

Declare `journal_access_failed` as volatile to ensure safety in recovery scenarios and prevent future invariance breaks. Add checks to avoid overflow in uuid entry list size computations during v2 journal setup.

* protect(dbengine): isolate UUID stack copies to prevent SIGBUS during mmap access

Safeguard `journalfile` and `rrdengine` operations by copying UUIDs to the stack before invoking mrg functions. Ensures clean recovery in protected regions on unreadable backing pages and avoids lock state corruption during SIGBUS errors.

* fix(dbengine): improve v2 header validation and clarify static analysis intent

Enhance error logging for out-of-range v2 journal header offsets, enabling clearer diagnostics during rebuild scenarios. Add a comment clarifying tautological bounds for `descr->type` to assist static analyzers.

* protect(dbengine): scope protected frame tightly in v2 journal retention updates

Reduce nesting depth and mask of unrelated faults by tightly scoping the PROTECTED_ACCESS_SETUP frame to mmap walks only. Ensure proper cleanup and avoid inflated nesting in subsequent operations.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(cherry picked from commit b5e92cb)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants


Back | FazBrowse Home | New Git URL