FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

freeipmi.plugin: fix watchdog underflow at low system uptime by ktsaou · Pull Request #22490 · netdata/netdata · GitHub

freeipmi.plugin: fix watchdog underflow at low system uptime - #22490

Merged
ktsaou merged 1 commit into
masterfrom
fix/freeipmi-watchdog-underflow
May 17, 2026
Merged

freeipmi.plugin: fix watchdog underflow at low system uptime#22490
ktsaou merged 1 commit into
masterfrom
fix/freeipmi-watchdog-underflow

Conversation

ktsaou commented May 15, 2026
edited by cubic-dev-ai Bot
Loading

Copy link
Copy Markdown
Member

Summary

The freeipmi.plugin stale-data watchdog underflows when Netdata starts within ~10 minutes of system boot, killing the plugin every cycle until plugins.d permanently disables it.

Root cause

At src/collectors/freeipmi.plugin/freeipmi_plugin.c:2031:

if(state.sensors.last_iteration_ut < now_monotonic_usec() - IPMI_RESTART_IF_SENSORS_DONT_ITERATE_EVERY_SECONDS * USEC_PER_SEC) {

Both operands are usec_t (uint64_t). IPMI_RESTART_IF_SENSORS_DONT_ITERATE_EVERY_SECONDS * USEC_PER_SEC = 600,000,000 µs. When CLOCK_MONOTONIC is below 600 s (i.e. system uptime under 10 minutes), the unsigned subtraction wraps to ~2⁶⁴, and last_iteration_ut < HUGE is trivially true. The plugin logs "sensors have not be collected for N seconds" (where N is just the few seconds since plugin start, not a real threshold breach), then exits.

Observed behavior

Reproduced on a host that boots and starts Netdata 20 s later. Journal namespace netdata:

17:02:26 freeipmi.plugin[12122]: main(): update frequency 1 seconds is too small for IPMI. Using 5.
17:02:45 freeipmi.plugin[12122]: main(): sensors have not be collected for 19 seconds. Exiting to restart.
17:02:45 netdata[3125]: PLUGINSD: freeipmi.plugin (pid 12122) disconnected after 0 successful data collections.
...
17:07:45 netdata[3125]: PLUGINSD: ... We have tried to collect something 11 times - unsuccessfully. Disabling it.

All 11 failed restarts land within the first 5 minutes of uptime, so plugins.d's SERIAL_FAILURES_THRESHOLD (10) trips and plugin_set_disabled() is called. By the time uptime exceeds 600 s and the underflow would no longer trigger, the plugin will never be retried during this Netdata lifetime. Result: no IPMI sensors/voltages/fans monitored until the next Netdata restart, which on long-uptime systems happens to be the only reason this bug is not seen more often.

Fix

Restructure the comparison to add the threshold to the last-iteration timestamp, which cannot underflow:

if(now_monotonic_usec() > state.sensors.last_iteration_ut + IPMI_RESTART_IF_SENSORS_DONT_ITERATE_EVERY_SECONDS * USEC_PER_SEC) {

last_iteration_ut + 600_000_000 cannot overflow in any realistic scenario (would require monotonic clock near UINT64_MAX).

Test plan

  • Boot a Linux host with IPMI hardware and Netdata configured to start within 10 minutes of boot (default systemd ordering does this).
  • Confirm ipmi.sensor_fan_speed, ipmi.sensor_temperature_c, ipmi.sensor_voltage contexts appear and stay live in Netdata.
  • Confirm journalctl --namespace netdata does not contain "sensors have not be collected for N seconds" within the first 10 minutes after boot.

Summary by cubic

Fixes a watchdog underflow in freeipmi.plugin that caused restarts and permanent disablement when Netdata starts within ~10 minutes of boot. Ensures IPMI sensors keep collecting after early-boot starts.

  • Bug Fixes
    • Changed the restart check to now > last + 600s to avoid unsigned underflow when uptime < 600s.
    • Stops false exits and prevents plugins.d from disabling the plugin during early uptime.

Written for commit 1527881. Summary will update on new commits.

The stale-data watchdog computes `now_monotonic_usec() - 600s` and
compares it to `last_iteration_ut`. Both operands are `usec_t`
(uint64_t), so when the monotonic clock has less than 600 seconds
since boot, the subtraction underflows to a huge value and the
comparison trivially evaluates true. The plugin self-exits, and
after 11 quick restarts plugins.d permanently disables it for the
remainder of the Netdata service lifetime.

Reproduces every time Netdata starts within ~10 minutes of system
boot. Restructure the comparison to add the threshold to the
last-iteration timestamp, which cannot underflow.
ktsaou requested a review from thiagoftsm as a code owner May 15, 2026 08:29
github-actions Bot added area/collectors Everything related to data collection collectors/freeipmi labels May 15, 2026

Copy link
Copy Markdown

cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

ktsaou merged commit 3803a32 into master May 17, 2026
151 of 157 checks passed
ilyam8 deleted the fix/freeipmi-watchdog-underflow branch May 22, 2026 10:11
stelfrag mentioned this pull request Jun 22, 2026
Ferroin pushed a commit that referenced this pull request Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/collectors Everything related to data collection collectors/freeipmi

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant


Back | FazBrowse Home | New Git URL