Summary

Fixes #2072 — When Redis (or a Redis-compatible store like Dragonfly/KeyDB) restarts, Sequin enters an unrecoverable failure state requiring a full restart. This has bitten us (and others per #2072) multiple times in production.

Root cause: MutexOwner in :has_mutex state calls acquire_mutex which returns :error when Redis is unreachable. The handler immediately returns {:shutdown, :err_keeping_mutex} — an invalid GenStateMachine return that crashes the process with {:bad_return_from_state_function, ...}. Due to MutexedSupervisor's :one_for_all strategy, this cascades and takes down the entire Runtime.Supervisor including all consumers.

Changes:

• MutexOwner: Instead of crashing on Redis errors, retry indefinitely with exponential backoff (capped at 1 hour). When Redis comes back, the error counter resets and normal operation resumes. Also fixes the GenStateMachine return value ({:stop, {:shutdown, reason}} instead of invalid {:shutdown, reason}).
• SinkConsumersLive.Index: Handle {:error, _} from metrics calls instead of crashing with MatchError.
• SinkConsumersLive.Show: Same defensive handling for all three metrics calls.
• HttpEndpointsLive.Show: Same fix for get_http_endpoint_throughput/1.

Context

We self-host Sequin on Railway with Dragonfly (Redis-compatible) as the backing store. Railway periodically auto-updates Dragonfly, which causes a brief restart. Every time this happens, Sequin fails to self-heal and requires a manual restart — we've hit this 3 times now. The :await_mutex state already retries correctly on Redis errors; this PR brings the same resilience to the :has_mutex state, but with exponential backoff so it doesn't spam during extended outages.

Test plan

• Added MutexOwnerTest with unit tests for State struct and integration tests (tagged :integration) that use iptables REJECT to simulate Redis going down and coming back
• Integration tests verify: process survives Redis outage, never crashes, recovers when Redis returns
• All existing tests pass (mix test)
• mix format --check-formatted passes
• Verify in staging by restarting Redis while Sequin is running

Note: Integration tests require NET_ADMIN capability (for iptables) and are tagged :integration so they can be excluded from normal test runs: mix test --exclude integration

🤖 Generated with Claude Code