Summary

Closes #3605.

Adds two histograms to make the global expiration ZSET observable in production:

Why these two signals

sandbox:storage:global:expiration is a singleton ZSET shared by every sandbox.
With 22,458 members (~3 MB) observed in production on 2026-08-14 and the evictor
calling ExpiredItems at pollInterval = 50 ms, this key is read 20 × N times
per second (N = API allocation count) — yet neither its size trend nor the
per-sweep cost was visible in any metric.

• expiration_index.size — use max/p99 to detect unbounded growth before
  it causes latency spikes; use as a before/after signal when fix(api): prune stale team index entries when ZSET orphans are swept #3567 (orphan SREM
  fix) or future sharding work lands.
• expiration_index.sweep_duration — ZRANGEBYSCORE is O(log N + K);
  rising p99 alongside rising size confirms the cost is growing and justifies
  remediation.

Implementation

heal.go — one ZCARD call appended after healExpirationIndex completes.
Cost: one Redis round-trip per 5 min, negligible next to the forEachSandboxBatch
SSCAN scan that precedes it. Sampling here avoids adding any load to the 50 ms
evictor hot path.

items.go — defer-based timer wrapping the full ExpiredItems body.
Records on every evictor tick, capturing the end-to-end cost of
ZRANGEBYSCORE + MGET pipeline + orphan cleanup.

Related

• obs(api): no metrics on global expiration ZSET size or evictor sweep latency #3605 — issue this PR closes
• obs(api): record team sandbox index SET size as a histogram #3604 — per-team index SET size histogram (same observability series)
• sandbox/storage: stale team-index SET entries accumulate when sandbox keys are lost without going through Remove() #3566 / fix(api): prune stale team index entries when ZSET orphans are swept #3567 — stale team-index entries inflating the ZSET
• perf(api): cjson.decode in startTransitionScript Lua script blocks Redis event loop under concurrent evictions #3602 / perf(api): avoid cjson.decode in startTransitionScript hot path #3603 — cjson.decode hot path (separate Redis bottleneck)
• RFC: Evaluate replacing Redis with DragonflyDB as the caching backend #3591 — DragonflyDB RFC; these metrics provide the baseline for evaluation

/cc @jakubno @dobrac @ValentaTomas