Summary

When an extent write to a dbengine datafile fails with a non-transient error
(EBADF, ENOSPC, EACCES, EROFS, EINVAL), dbengine now marks the
datafile pair as failed, rotates to a freshly created datafile pair, and
retries the same extent there — instead of silently dropping the extent's
pages.

The problem

In a production deployment (Windows parent, v2.10.3) the file descriptor of
the active tier0 datafile became invalid (EBADF) while the agent kept
running — three times in 14 days. Each time, every tier0 extent flushed for
the next ~4–7 hours was silently dropped, until the datafile logically filled
(offsets are reserved even for failed writes) and the normal size-based
rotation opened a fresh fd, self-healing the instance.

The impact: multi-hour tier0 retention holes for all hosts stored on that
parent (its own metrics and all its children), with per-chart different gap
boundaries (each metric's open pages spanned different time ranges). Higher
tiers were unaffected (separate datafiles/fds), which is how it surfaced:
dashboards rendered wide views (tier1) but showed "No data" when zooming
(tier0). The only signal was one rate-limited error line every 10 seconds in
the daemon log:

DBENGINE: Tier 0, bad file descriptor

Diagnosis details: the extent write retry loop breaks immediately on the
errors listed above, extent_flush_to_open(..., have_error=true) skips the
open-cache registration and releases the pages, and the next extent picks the
same dead datafile->file. All data is still in RAM at the moment of failure
(the failure is synchronous), so dropping is unnecessary — the extent can be
retargeted.

The fix

• struct rrdengine_datafile gets a writers.failed flag; a failed datafile
  reports itself full in datafile_is_full(), so get_datafile_to_write_extent()
  rotates away from it using the existing, mutex-serialized rotation path.
• The extent buffer is position independent — only its WAL transaction embeds
  the reserved offset. On failure, the WAL is released, a new offset is
  reserved on the freshly created datafile, the WAL is rebuilt, and the same
  extent is written there. Zero data loss in the single-failure case.
• Exactly one recovery attempt per extent. If the retry also fails (or a new
  datafile pair cannot be created — e.g. the volume is truly read-only or
  full), behavior degrades to the previous one (extent dropped, rate-limited
  error), with the failed datafile still marked so subsequent extents keep
  trying to rotate away.
• Journal (WAL) write failures take the same recovery: the extent is
  re-written to a fresh pair so it is indexed and queryable; the data bytes
  already written to the failed pair become dead bytes and are sealed at
  rotation (journal indexing tolerates this — verified in the field, where the
  dying datafile's journal migrated and indexed cleanly with its surviving
  extents).
• Fixes a WAL leak on the old failure path: the WAL allocated by
  journalfile_extent_build() was only ever released inside
  journalfile_v1_extent_write(), which is never reached when the data write
  fails — one pooled WAL buffer leaked per dropped extent.

Validation

Live fault injection on a scratch instance built from this branch: closed the
tier0 datafile fd inside the running process (gdb -p <pid> -ex 'call close(fd)'),
reproducing the production failure.

Observed:

• The first natural flush after the kill failed, logged the recovery, and
  created the next datafile pair 3 ms later:

  14:27:49.758 error: DBENGINE: tier 0 datafile 1 write failed (invalid seek) -
               rotating to a new datafile and retrying the extent, to prevent data loss
  14:27:49.761 info:  DBENGINE: tier 0: created datafile-1-0000000002 (.ndf, .njf).
  

• The failed extent was written to the new datafile — no extent is lost
  line appeared at any point, including through a clean shutdown.

• After a restart (so queries are served from disk, not from the page cache),
  querying system.cpu at 1s resolution across the whole session shows
  every collected sample present — data is continuous from chart creation
  through the fd kill, the recovery, and up to the exact second of SIGTERM.
  The only empty rows are the agent's own downtime between stop and restart.

• The journals migrated and indexed normally on restart (journalfile-1-...2.njfv2: extents 53, metrics 4090, pages 5676); the failed pair (header-only) was
  handled cleanly.

• A notable variant got exercised by accident: the closed fd number was
  recycled by a pipe/socket before the flush, so the write failed with
  ESPIPE (not on the immediate-break error list). The retry loop exhausted
  its 10 attempts and the recovery still engaged — confirming it triggers on
  any final write failure, not only the immediate-break errors.

Without this change, the same injection loses every flushed extent until the
datafile logically fills and rotates — in the production incident that was
4–7 hours of tier0, three times in 14 days, for every host stored on the
parent.

Summary by cubic

On unrecoverable datafile write errors, dbengine now marks the file as failed, rotates to a new datafile pair, and retries the same extent on the new file to avoid data loss. Also handles WAL write failures and fixes a WAL buffer leak.

• Bug Fixes
  • Added a writers.failed flag; failed datafiles report full so rotation moves future writes away.
  • On a write failure, release WAL, reserve a new offset on a fresh datafile, rebuild WAL, and retry the extent once; if retry or rotation fails, fall back to dropping the extent.
  • Applied the same recovery to WAL write failures; any bytes written to the failed pair are left as dead and sealed at rotation, while indexing stays correct.
  • Improved recovery logging and ensured proper release of writer slots and WAL buffers, fixing the previous WAL leak on failed writes.

Written for commit 9eb913c. Summary will update on new commits.