FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Fix TOCTOU race in dictionary_destroy() by stelfrag · Pull Request #22108 · netdata/netdata · GitHub

Fix TOCTOU race in dictionary_destroy() - #22108

Merged
stelfrag merged 11 commits into
netdata:masterfrom
stelfrag:fix_dict_destroy_race
Apr 14, 2026
Merged

Fix TOCTOU race in dictionary_destroy()#22108
stelfrag merged 11 commits into
netdata:masterfrom
stelfrag:fix_dict_destroy_race

Conversation

stelfrag commented Apr 1, 2026
edited by cubic-dev-ai Bot
Loading

Copy link
Copy Markdown
Collaborator
Summary
  • Add synchronized checks to prevent accessing destroyed dictionaries in multithreaded scenarios.
  • Introduce unit tests to reproduce and validate the fix for concurrent get/set/traverse operations during dictionary destruction.
  • Update dictionary destruction logic to mark and lock the dictionary index, mitigating race conditions.

Summary by cubic

Fixes a TOCTOU race in dictionary_destroy() that let threads access items during free. Adds synchronized destroyed-state checks in lookups/traversal, hardens the stress test to avoid crashes or hangs, and centralizes CI detection.

  • Bug Fixes

    • dictionary_destroy(): return early if null/destroyed; mark destroyed; tear down the index under the items write lock; if items are still referenced, queue deferred destruction via DICT_FLAG_QUEUED_FOR_DESTRUCTION; make queueing synchronized and idempotent.
    • Add destroyed checks under index/traversal locks in dict_item_del, dict_item_add_or_reset_value_and_acquire, dict_item_find_and_acquire, and traversal; fix reentrant traversal to unlock before exit.
    • TOCTOU stress test: run workload in a forked child; add timeout with EINTR-safe waitpid() and kill-on-timeout; handle thread-creation and acquire failures; use unique keys; remove the permanent item hold to exercise the new sync path; reduce iterations in CI; skip on Windows.
  • Refactors

    • Centralize CI detection as nd_is_running_under_ci() in libnetdata/os/ci.{c,h} (included via libnetdata.h) and replace is_ci() in status-file.c; update build files.

Written for commit 6bbda97. Summary will update on new commits.

stelfrag marked this pull request as ready for review April 1, 2026 20:41
stelfrag requested a review from thiagoftsm as a code owner April 1, 2026 20:41
Copilot AI review requested due to automatic review settings April 1, 2026 20:41
stelfrag requested a review from vkalintiris as a code owner April 1, 2026 20:41
stelfrag marked this pull request as draft April 1, 2026 20:41

cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

2 issues found across 4 files

Confidence score: 2/5

  • There is a high-confidence, high-severity regression risk in src/libnetdata/dictionary/dictionary-traversal.c: the reentrant destroyed-after-lock path can leak the traversal lock, which can lead to deadlocks in real runtime behavior.
  • A secondary issue in src/libnetdata/dictionary/dictionary-unittest.c can hang tests when nd_thread_create fails, because the ready-wait loop may wait forever instead of handling the failure path first.
  • Given the concrete deadlock risk (8/10 severity, 9/10 confidence), this is likely not safe to merge until that lock-release path is fixed.
  • Pay close attention to src/libnetdata/dictionary/dictionary-traversal.c and src/libnetdata/dictionary/dictionary-unittest.c - deadlock-prone lock handling and thread-creation failure handling need correction.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/libnetdata/dictionary/dictionary-unittest.c">

<violation number="1" location="src/libnetdata/dictionary/dictionary-unittest.c:1093">
P2: Handle `nd_thread_create` failures before waiting on the ready flags; otherwise a failed thread creation will hang the test in the ready-wait loop.</violation>
</file>

<file name="src/libnetdata/dictionary/dictionary-traversal.c">

<violation number="1" location="src/libnetdata/dictionary/dictionary-traversal.c:81">
P1: Reentrant mode can leak the traversal lock on the new destroyed-after-lock path, causing deadlocks.</violation>
</file>
Architecture diagram
sequenceDiagram
    participant D as Destroyer Thread
    participant RW as Reader/Writer Thread
    participant Idx as Dictionary Index (Hashtable)
    participant DQ as Deferred Queue

    Note over D, DQ: Fixes TOCTOU race in dictionary_destroy()

    D->>D: Set DICT_FLAG_DESTROYED
    
    rect rgb(240, 240, 240)
        Note right of D: NEW: Synchronized Index Teardown
        D->>Idx: NEW: Acquire Index Write Lock
        
        par Concurrent Access
            RW->>Idx: Attempt lock (Read or Write)
            Note over RW: RW blocks until Destroyer finishes index wipe
        and Destruction
            D->>Idx: NEW: hashtable_destroy_unsafe()
            D->>Idx: NEW: Release Index Write Lock
        end
    end

    RW->>Idx: Acquire Lock (Granted)
    RW->>RW: NEW: Re-check is_dictionary_destroyed()
    
    alt is_dictionary_destroyed is true
        RW-->>RW: NEW: Release lock and exit safely
    else is_dictionary_destroyed is false (unchanged path)
        RW->>Idx: Access items
        RW-->>RW: Release lock
    end

    Note over D, DQ: NEW: Final reference check before freeing memory

    alt Items still referenced (Refcount > 0)
        D->>DQ: NEW: Fallback: Queue for deferred destruction
        Note over D, DQ: Cleanup delayed until all acquired items are released
    else No references
        D->>D: Free all dictionary resources immediately
    end
Loading

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

This PR hardens the DICTIONARY implementation against a TOCTOU race during dictionary_destroy() in multithreaded use by synchronizing “destroyed” checks under the relevant locks and preventing concurrent index lookups from racing with teardown.

Changes:

  • Mark dictionaries as destroyed earlier and tear down the index under lock to prevent post-check concurrent lookups during destruction.
  • Add synchronized “destroyed” re-checks under traversal and index locks for get/set/del/traversal paths.
  • Add a stress/unit test that reproduces the destroy-vs-get/set/traverse race (run in a forked child to detect crashes).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/libnetdata/dictionary/dictionary.c Updates dictionary_destroy() to set the destroyed flag and tear down the index under lock, with a re-check for referenced items to choose deferred destruction safely.
src/libnetdata/dictionary/dictionary-item.h Adds destroyed re-checks under the index lock for delete/add/find operations to close TOCTOU windows.
src/libnetdata/dictionary/dictionary-traversal.c Adds destroyed re-checks after acquiring traversal locks so traversal can’t proceed on a dictionary that becomes destroyed concurrently.
src/libnetdata/dictionary/dictionary-unittest.c Introduces a forked stress test that runs concurrent get/set/traverse while destroying the dictionary to validate the fix.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/libnetdata/dictionary/dictionary-unittest.c">

<violation number="1" location="src/libnetdata/dictionary/dictionary-unittest.c:1176">
P1: Timeout detection is broken: a still-running child can be treated as exited because `status` remains 0 when `waitpid(..., WNOHANG)` never reaps it.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

github-actions Bot added area/daemon area/build Build system (autotools and cmake). labels Apr 1, 2026
stelfrag requested a review from Copilot April 1, 2026 21:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/libnetdata/dictionary/dictionary.c Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/libnetdata/dictionary/dictionary-item.h:476

  • The “destroyed” re-check in dict_item_add_or_reset_value_and_acquire() only happens while holding the index lock. After releasing the index write lock, the code proceeds to item_linked_list_add() (which takes DICTIONARY_LOCK_WRITE) without another destroyed-state check or rollback path. If dictionary_destroy() sets DICT_FLAG_DESTROYED after this function’s index-lock check but before it acquires the items lock, the writer may still link a new item into a destroyed/queued dictionary or race with deferred/free destruction. Please add a synchronized destroyed-state check on the items lock acquisition path and ensure partially-created inserts can be safely aborted/cleaned up when destruction is in progress.
    dictionary_index_lock_wrlock(dict);

    // Re-check under the index lock. This synchronizes with
    // dictionary_destroy(), which sets the destroyed flag and then takes
    // this lock before tearing down the index.
    if(unlikely(is_dictionary_destroyed(dict))) {
        dictionary_index_wrlock_unlock(dict);
        return NULL;
    }

    bool added_or_updated = false;
    size_t spins = 0;
    DICTIONARY_ITEM *item = NULL;
    do {
        void *handle = hashtable_insert_unsafe(dict, name, name_len);
        item = hashtable_insert_handle_to_item_unsafe(dict, handle);
        if (likely(item == NULL)) {
            // a new item added to the index

            // create the dictionary item
            item = dict_item_create_with_hooks(dict, name, name_len, value, value_len, constructor_data, master_item);

            pointer_add(dict, item);

            hashtable_set_item_unsafe(dict, handle, item);

            // unlock the index lock, before we add it to the linked list
            // DON'T DO IT THE OTHER WAY AROUND - DO NOT CROSS THE LOCKS!
            dictionary_index_wrlock_unlock(dict);

            item_linked_list_add(dict, item);

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/libnetdata/dictionary/dictionary.c Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

stelfrag requested a review from Copilot April 3, 2026 21:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/libnetdata/dictionary/dictionary.c Outdated
Comment thread src/libnetdata/os/ci.c Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/libnetdata/dictionary/dictionary.c:717

  • dictionary_destroy() now destroys the index early via hashtable_destroy_unsafe(), but the freed size returned from hashtable_destroy_unsafe() is ignored. Since dictionary_free_all_resources() later sees an already-destroyed index and returns 0 for index_size, dictionary_destroy() will under-report freed bytes (and any callers relying on this return value will get inaccurate results). Consider capturing the size returned from the early hashtable_destroy_unsafe() call and adding it to the final freed result (or otherwise ensuring the total includes index memory freed in this path).
    // Destroy the index while holding the items write lock.
    // This prevents a TOCTOU race: without this, a reader could pass the
    // is_dictionary_destroyed() check, then acquire an item via the index,
    // after we've decided to force-free all items.
    // By destroying the index here, any concurrent dictionary_get_and_acquire_item()
    // that acquires the index lock after this point will find an empty index.
    // This uses hashtable_destroy_unsafe(); the later cleanup path in
    // dictionary_free_all_resources() may invoke the same index teardown
    // again, so this relies on that full destruction flow being safe to repeat.
    dictionary_index_lock_wrlock(dict);
    hashtable_destroy_unsafe(dict);
    dictionary_index_wrlock_unlock(dict);

    // Re-check: a reader that held the index read lock during the destroy
    // above may have acquired an item before we got the index write lock.
    // If so, fall back to the deferred destruction path.
    if(dictionary_referenced_items(dict)) {
        dictionary_queue_for_destruction(dict);

        ll_recursive_unlock(dict, DICTIONARY_LOCK_WRITE);
        return 0;
    }

    ll_recursive_unlock(dict, DICTIONARY_LOCK_WRITE);

    size_t freed;
    dictionary_free_all_resources(dict, &freed, true);

    return freed;

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

stelfrag marked this pull request as ready for review April 7, 2026 07:04
stelfrag requested a review from a team as a code owner April 7, 2026 07:04
- Add synchronized checks to prevent accessing destroyed dictionaries in multithreaded scenarios.
- Introduce unit tests to reproduce and validate the fix for concurrent get/set/traverse operations during dictionary destruction.
- Update dictionary destruction logic to mark and lock the dictionary index, mitigating race conditions.
stelfrag added 10 commits April 14, 2026 11:16
- Add a timeout mechanism to prevent indefinite hangs during the test.
- Implement child process termination to handle potential test deadlocks.
… and unit tests

- Ensure proper unlocking in `dictionary_foreach_done` to handle reentrant mode locks.
- Add error handling in unit tests to clean up resources when thread creation fails.
…dling and readability

- Replace `timed_out` flag with `reaped` for clarity.
- Simplify logic for detecting and handling hung child processes.
- Implement `nd_is_running_under_ci()` in `libnetdata/os/ci.c` for centralized CI environment checking.
- Replace the duplicate `is_ci()` logic in `status-file.c` with the new library function.
- Update build files and headers to include the new module.
- Add `DICT_FLAG_QUEUED_FOR_DESTRUCTION` to track queued dictionaries.
- Refactor `dictionary_queue_for_destruction()` to prevent redundant operations.
- Improve `dictionary_destroy()` handling for destroyed or null dictionaries.
- Add platform-specific handling for `dictionary_destroy_race_unittest`, skipping unsupported tests on Windows.
- Hold acquired item to ensure delayed destruction path is tested.
- Add error handling for `dictionary_get_and_acquire_item` failures.
- Ensure proper item release and dictionary cleanup in all cases.
…and CI detection

- Fix EINTR handling in `dictionary_destroy()` TOCTOU tests for better reliability.
- Add synchronization to `dictionary_queue_for_destruction` to prevent redundant locking.
- Refactor CI environment detection to enhance modularity and include missing headers.
…rove synchronization

- Remove permanent item acquisition to avoid forcing the old delayed-destroy path.
- Rely on transient in-flight accesses to test synchronization during destruction.
- Reduce iterations for CI to improve test reliability and runtime.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

thiagoftsm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

No issues happening during runtime. LGTM!

stelfrag merged commit cd741f2 into netdata:master Apr 14, 2026
154 of 155 checks passed
stelfrag deleted the fix_dict_destroy_race branch April 14, 2026 14:22
nedi-app Bot pushed a commit that referenced this pull request Apr 24, 2026
* Fix TOCTOU race in dictionary_destroy()

- Add synchronized checks to prevent accessing destroyed dictionaries in multithreaded scenarios.
- Introduce unit tests to reproduce and validate the fix for concurrent get/set/traverse operations during dictionary destruction.
- Update dictionary destruction logic to mark and lock the dictionary index, mitigating race conditions.

* Improve TOCTOU test reliability in dictionary_destroy()

- Add a timeout mechanism to prevent indefinite hangs during the test.
- Implement child process termination to handle potential test deadlocks.

* Fix race condition and improve error handling in dictionary traversal and unit tests

- Ensure proper unlocking in `dictionary_foreach_done` to handle reentrant mode locks.
- Add error handling in unit tests to clean up resources when thread creation fails.

* Refactor TOCTOU test in `dictionary_destroy()` to improve process handling and readability

- Replace `timed_out` flag with `reaped` for clarity.
- Simplify logic for detecting and handling hung child processes.

* Move CI environment detection to `libnetdata` and refactor calls

- Implement `nd_is_running_under_ci()` in `libnetdata/os/ci.c` for centralized CI environment checking.
- Replace the duplicate `is_ci()` logic in `status-file.c` with the new library function.
- Update build files and headers to include the new module.

* Fix dictionary destruction logic and enhance platform compatibility

- Add `DICT_FLAG_QUEUED_FOR_DESTRUCTION` to track queued dictionaries.
- Refactor `dictionary_queue_for_destruction()` to prevent redundant operations.
- Improve `dictionary_destroy()` handling for destroyed or null dictionaries.
- Add platform-specific handling for `dictionary_destroy_race_unittest`, skipping unsupported tests on Windows.

* Clarify comments for safe repeated index teardown during dictionary destruction

* Fix dictionary destruction handling in unit tests

- Hold acquired item to ensure delayed destruction path is tested.
- Add error handling for `dictionary_get_and_acquire_item` failures.
- Ensure proper item release and dictionary cleanup in all cases.

* Improve error handling and synchronization in dictionary destruction and CI detection

- Fix EINTR handling in `dictionary_destroy()` TOCTOU tests for better reliability.
- Add synchronization to `dictionary_queue_for_destruction` to prevent redundant locking.
- Refactor CI environment detection to enhance modularity and include missing headers.

* Use unique keys in dictionary destruction unit test

* Refactor `dictionary_destroy_race_unittest` to simplify logic and improve synchronization

- Remove permanent item acquisition to avoid forcing the old delayed-destroy path.
- Rely on transient in-flight accesses to test synchronization during destruction.
- Reduce iterations for CI to improve test reliability and runtime.
stelfrag mentioned this pull request Jun 22, 2026
Ferroin pushed a commit that referenced this pull request Jul 15, 2026
* Fix TOCTOU race in dictionary_destroy()

- Add synchronized checks to prevent accessing destroyed dictionaries in multithreaded scenarios.
- Introduce unit tests to reproduce and validate the fix for concurrent get/set/traverse operations during dictionary destruction.
- Update dictionary destruction logic to mark and lock the dictionary index, mitigating race conditions.

* Improve TOCTOU test reliability in dictionary_destroy()

- Add a timeout mechanism to prevent indefinite hangs during the test.
- Implement child process termination to handle potential test deadlocks.

* Fix race condition and improve error handling in dictionary traversal and unit tests

- Ensure proper unlocking in `dictionary_foreach_done` to handle reentrant mode locks.
- Add error handling in unit tests to clean up resources when thread creation fails.

* Refactor TOCTOU test in `dictionary_destroy()` to improve process handling and readability

- Replace `timed_out` flag with `reaped` for clarity.
- Simplify logic for detecting and handling hung child processes.

* Move CI environment detection to `libnetdata` and refactor calls

- Implement `nd_is_running_under_ci()` in `libnetdata/os/ci.c` for centralized CI environment checking.
- Replace the duplicate `is_ci()` logic in `status-file.c` with the new library function.
- Update build files and headers to include the new module.

* Fix dictionary destruction logic and enhance platform compatibility

- Add `DICT_FLAG_QUEUED_FOR_DESTRUCTION` to track queued dictionaries.
- Refactor `dictionary_queue_for_destruction()` to prevent redundant operations.
- Improve `dictionary_destroy()` handling for destroyed or null dictionaries.
- Add platform-specific handling for `dictionary_destroy_race_unittest`, skipping unsupported tests on Windows.

* Clarify comments for safe repeated index teardown during dictionary destruction

* Fix dictionary destruction handling in unit tests

- Hold acquired item to ensure delayed destruction path is tested.
- Add error handling for `dictionary_get_and_acquire_item` failures.
- Ensure proper item release and dictionary cleanup in all cases.

* Improve error handling and synchronization in dictionary destruction and CI detection

- Fix EINTR handling in `dictionary_destroy()` TOCTOU tests for better reliability.
- Add synchronization to `dictionary_queue_for_destruction` to prevent redundant locking.
- Refactor CI environment detection to enhance modularity and include missing headers.

* Use unique keys in dictionary destruction unit test

* Refactor `dictionary_destroy_race_unittest` to simplify logic and improve synchronization

- Remove permanent item acquisition to avoid forcing the old delayed-destroy path.
- Rely on transient in-flight accesses to test synchronization during destruction.
- Reduce iterations for CI to improve test reliability and runtime.

(cherry picked from commit cd741f2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build Build system (autotools and cmake). area/daemon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants


Back | FazBrowse Home | New Git URL