NET-371 + NET-373 + NET-375. Validates Bearer sqd_data_<id>_<rnd> on
GET /network/:dataset/:start_block/worker against an in-process LRU
(60s exists / 15s deleted, three explicit states UNDEFINED/EXISTS/DELETED
plus a brief FailedRecently sentinel) backed by POST /internal/validate.
Enforcement scope is configurable via ENFORCE_V2_AUTH_FOR_IPS — a CIDR
list whose source IPs trigger 403 on missing/invalid keys; everyone
else fails open. Empty (default) = observe-only canary mode; \`*\` =
enforce for all sources (expands to 0.0.0.0/0 + ::/0 internally);
specific CIDRs = canary scope. One knob — no separate boolean. Parsing,
cache, and Network API calls run regardless of scope, so observation-mode
metrics show what would happen under a flip.
Source-IP bypass via INTERNAL_ALLOWLIST + TRUSTED_IPS lets internal
traffic skip the Bearer path entirely. Real client IP is resolved by
walking X-Original-Forwarded-For rightmost-first, stripping TRUSTED_IPS
hops; the first non-trusted IP from the right is the source the upstream
LB observed at TCP handshake. This is robust against leftmost-XFF
spoofing (a real risk with ingress-nginx default proxy-real-ip-cidr=
0.0.0.0/0 — empirically reproduced before settling on this header).
Falls back to ConnectInfo<SocketAddr> when no XOFF is present (direct
ClusterIP path).
DISABLE_V2_AUTH global kill switch short-circuits at the very top of
the middleware: every request is allowed without parsing headers,
touching the cache, calling the Network API, or running the latency
timer. Counted as sqd_v2_auth_total{result="disabled"} so the dashboard
surfaces that the switch is engaged.
Network API client: 250ms timeout, half-open circuit breaker (50
errors / 30s window / one probe). Probe admission returns a RAII Permit
that records failure on Drop unless explicitly resolved, so a cancelled
probe re-opens the breaker instead of leaving it stuck HalfOpen.
Singleflight via DashMap of per-key tokio mutexes coalesces concurrent
misses to one upstream call; FailedRecently sentinel (1s TTL) drains
the queue without each waiter re-issuing a 250ms timeout.
Metrics: sqd_v2_auth_total{result=ok|missing|invalid|fail_open|disabled},
sqd_v2_auth_latency_seconds, sqd_v2_cache_{hit,miss}_total,
sqd_v2_validate_call_total, sqd_v2_requests_by_key (top-100 sketch
with explicit eviction so Prometheus cardinality stays bounded),
sqd_router_worker_urls_handed_total (Worker:Router ratio canary).
key_id is the only token-derived value in logs/labels; the full token
is never logged. Bypass requests use user_id="internal:<ip>" for
attribution.
90 tests covering cache states + TTL clamp, breaker half-open + probe
cancellation, fail-open paths + sentinel queue drain, singleflight
under concurrency + cleanup TOCTOU, log redaction, top-keys count
semantics, bounded label cardinality, IP-bypass + walk-rightmost XFF
anti-spoof (T1-T4 derived from empirical prod debugging), kill switch
short-circuit (no decide / no metrics drift), and canary-scope
enforcement (in-scope, out-of-scope, wildcard, narrow without
ConnectInfo, XOFF integration, allowlist precedence).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Resilience details
Wiring
Test plan
🤖 Generated with Claude Code