| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
The stale-data watchdog computes `now_monotonic_usec() - 600s` and compares it to `last_iteration_ut`. Both operands are `usec_t` (uint64_t), so when the monotonic clock has less than 600 seconds since boot, the subtraction underflows to a huge value and the comparison trivially evaluates true. The plugin self-exits, and after 11 quick restarts plugins.d permanently disables it for the remainder of the Netdata service lifetime. Reproduces every time Netdata starts within ~10 minutes of system boot. Restructure the comparison to add the threshold to the last-iteration timestamp, which cannot underflow.
|
Sorry, something went wrong.
(cherry picked from commit 3803a32)
| Back | FazBrowse Home | New Git URL |
Summary
The freeipmi.plugin stale-data watchdog underflows when Netdata starts within ~10 minutes of system boot, killing the plugin every cycle until plugins.d permanently disables it.
Root cause
At src/collectors/freeipmi.plugin/freeipmi_plugin.c:2031:
Both operands are usec_t (uint64_t). IPMI_RESTART_IF_SENSORS_DONT_ITERATE_EVERY_SECONDS * USEC_PER_SEC = 600,000,000 µs. When CLOCK_MONOTONIC is below 600 s (i.e. system uptime under 10 minutes), the unsigned subtraction wraps to ~2⁶⁴, and last_iteration_ut < HUGE is trivially true. The plugin logs "sensors have not be collected for N seconds" (where N is just the few seconds since plugin start, not a real threshold breach), then exits.
Observed behavior
Reproduced on a host that boots and starts Netdata 20 s later. Journal namespace netdata:
All 11 failed restarts land within the first 5 minutes of uptime, so plugins.d's SERIAL_FAILURES_THRESHOLD (10) trips and plugin_set_disabled() is called. By the time uptime exceeds 600 s and the underflow would no longer trigger, the plugin will never be retried during this Netdata lifetime. Result: no IPMI sensors/voltages/fans monitored until the next Netdata restart, which on long-uptime systems happens to be the only reason this bug is not seen more often.
Fix
Restructure the comparison to add the threshold to the last-iteration timestamp, which cannot underflow:
last_iteration_ut + 600_000_000 cannot overflow in any realistic scenario (would require monotonic clock near UINT64_MAX).
Test plan
Summary by cubic
Fixes a watchdog underflow in freeipmi.plugin that caused restarts and permanent disablement when Netdata starts within ~10 minutes of boot. Ensures IPMI sensors keep collecting after early-boot starts.
Written for commit 1527881. Summary will update on new commits.