fix systemd-journal.plugin memory retention after queries and its apps.plugin accounting - #23089
Merged
ktsaou merged 3 commits intoJul 11, 2026
Merged
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
The systemd-journald group used the substring pattern *systemd-journal*, which matches against the full process cmdline. This wrongly captured netdata's own systemd-journal.plugin (and anything else with systemd-journal in its cmdline, e.g. journalctl -u systemd-journald), polluting journald accounting with netdata's log-query workload. Match the journald family by exact process names instead: systemd-journald, systemd-journal-remote, systemd-journal-upload, systemd-journal-gatewayd. Names longer than the 15-char comm limit are reconstructed from the cmdline by apps.plugin, so exact matching works for all four. netdata's plugin now gets its own group (sd-jrnl.plugin), like every other netdata external plugin.
… idle Function plugins (systemd-journal, windows-events, macos-logs, etc.) free all per-query memory when a query completes, but glibc keeps the freed memory in its arenas and never returns it to the OS. As a result the plugin RSS stays at the high-water mark of the biggest query it ever executed (verified: 15MB -> 37MB RSS after 3 heavy journal queries with zero bytes leaked according to ASAN; a single malloc_trim(0) returns it all). After a function job completes, when the worker queue is empty (no pending and no running jobs), call mallocz_release_as_much_memory_to_the_system(). Under back-to-back load no trim happens and memory is reused; the last job of every burst always trims, so no retention window remains. On platforms without malloc_trim() (musl, macOS, Windows, FreeBSD) this is a no-op.
The hidden 'debug' command-line mode still used source:all, but the parameter was renamed to __logs_sources. The stale name is parsed as a filter on a user field named 'source', matching nothing, so debug mode evaluated 0 rows. With the fix it evaluates the journal normally.
Copy link
Copy Markdown
|
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
No issues found across 3 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
sequenceDiagram
participant AppsPlugin as apps.plugin
participant SystemdPlugin as systemd-journal.plugin
participant FuncEvloop as functions_evloop
participant Malloc as glibc malloc
Note over AppsPlugin: Process accounting
AppsPlugin->>AppsPlugin: Read process cmdline
AppsPlugin->>AppsPlugin: Match against exact names<br/>(systemd-journald, etc.)
Note right of AppsPlugin: systemd-journal.plugin cmdline does not match
AppsPlugin->>AppsPlugin: Account process under own group<br/>(sd-jrnl.plugin)
Note over SystemdPlugin,FuncEvloop: Function job handling
SystemdPlugin->>FuncEvloop: Execute function (e.g., query)
FuncEvloop->>FuncEvloop: Allocate memory for query, process, free
FuncEvloop->>FuncEvloop: Remove job from worker queue
alt Worker queue empty
FuncEvloop->>Malloc: mallocz_release_as_much_memory_to_the_system()<br/>(calls malloc_trim(0))
Malloc->>Malloc: Return freed memory to OS
else Queue not empty
FuncEvloop->>FuncEvloop: No trim (memory reused)
end
Note over SystemdPlugin: Debug mode
SystemdPlugin->>SystemdPlugin: Parse debug query parameter
SystemdPlugin->>SystemdPlugin: Uses "__logs_sources:all"<br/>— evaluates all journal sources
Sorry, something went wrong.
stelfrag
pushed a commit
to stelfrag/netdata
that referenced
this pull request
Jul 12, 2026
…s.plugin accounting (netdata#23089)
Merged
Ferroin
pushed a commit
that referenced
this pull request
Jul 15, 2026
…s.plugin accounting (#23089)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two independent issues around systemd-journal.plugin, plus one coupled debug-mode fix.
1. apps.plugin wrongly accounts systemd-journal.plugin as systemd-journald
The systemd-journald group in apps_groups.conf used the substring pattern *systemd-journal*, which matches against the full process cmdline. This wrongly captured netdata's own /usr/libexec/netdata/plugins.d/systemd-journal.plugin (and anything else with systemd-journal anywhere in its cmdline, e.g. journalctl -u systemd-journald), polluting journald accounting with netdata's log-query workload.
Fix: match the journald family by exact process names: systemd-journald, systemd-journal-remote, systemd-journal-upload, systemd-journal-gatewayd. Names longer than the 15-char comm limit are reconstructed from the cmdline by apps.plugin, so exact matching works for all four (verified live, including a running systemd-journal-upload).
Dashboard-visible change: netdata's journal plugin now appears as its own group (sd-jrnl.plugin, its runtime process name), like every other netdata external plugin, instead of inflating systemd-journald.
2. systemd-journal.plugin retains the RSS high-water mark of its biggest query forever
It is not a leak: ASAN/LSan reports identical output (1102 bytes of one-time init globals) for a trivial info request and for a 16-second full-text search over 117 GB of journals — leaked bytes do not scale with work. All per-query memory (facets, LQS, buffers, sd_journal handles) is freed at query end.
The problem is that glibc keeps the freed memory in its arenas and never returns it to the OS: the plugin never calls malloc_trim() (the only trim in the tree runs in the daemon's dbengine eviction — a different process), and M_TRIM_THRESHOLD cannot be inherited through the environment.
Measured with 3 heavy queries (7-day full-text search, ~7.7M rows evaluated each), plugin driven over the plugins.d protocol:
Fix: in the functions_evloop worker, after a function job completes and the worker queue is empty (no pending and no running jobs), call mallocz_release_as_much_memory_to_the_system(). Under back-to-back load no trim happens (memory is actively reused); the last job of every burst always trims, so no retention window remains. This covers all function plugins with the same pattern (systemd-journal, windows-events, macos-logs, systemd-units, network-viewer, debugfs, freeipmi, apps, ebpf). On platforms without malloc_trim() (musl static builds, macOS, Windows, FreeBSD) it compiles to a no-op.
With the fix, the same experiment gives:
Query results are unchanged (same evaluated/matched/returned counts), and ASAN reports no new leaks.
3. systemd-journal.plugin debug mode evaluated 0 rows
The hidden debug command-line mode still used the stale GET parameter source:all; the parameter was renamed to __logs_sources. The stale name is parsed as a filter on a user field named source, matching nothing. Fixed to use __logs_sources:all — debug mode now evaluates the journal normally (27.9M rows on the test box, was 0).
Summary by cubic
Match exact journald process names so systemd-journal.plugin is no longer counted under systemd-journald. Trim freed memory when function workers go idle to avoid retained RSS, and fix debug mode to read from all sources.
Written for commit 7c60108. Summary will update on new commits.