fix(yaml_loader): refresh stale table cache on miss during config apply - #2151
ddavignon-hadrian wants to merge 1 commit into
Conversation
Sorry, something went wrong.
change_retentions resolve source/destination tables against the tables list cached on the postgres_databases record, which is only refreshed on database create and by the 6-hourly EnqueueDatabaseUpdateWorker cron. For an existing database, the config-apply update path never refreshes tables, so a table created since the last refresh is absent during apply. Because apply runs at container start, this fails the boot and crash-loops the deployment, which in turn prevents the cron from ever healing it. Refresh the referenced database's tables from the live database on a cache miss (and retry the lookup) while parsing change_retention attrs. Cached hits keep the existing in-memory fast path, and genuinely missing tables still return not_found, so this only adds a round-trip when a referenced table is absent from the cache. Fixes sequinstream#2150.
|
Quick heads up on the failing signoff check: it died at the Configure PostgreSQL step with psql: command not found (exit 127) on the warp-ubuntu-latest-arm64-32x runner. That's a setup step, so nothing after it ran and the Elixir compile and tests were skipped entirely. I don't think it's related to this PR, which only changes lib/sequin/yaml_loader.ex and its test. Looks like the runner is missing the postgres client. Could a maintainer re-run it once that's sorted? In the meantime I ran it locally on Elixir 1.19.4 / OTP 28.3 against Postgres 17 with pgvector and logical replication. The project compiles clean and the new test passes. I also checked that the test fails without the fix, with the same "No table found matching params: name=sequin_events, schema=public" error from the linked issue, so it reproduces the bug and the change resolves it. Happy to change anything on my side if it's needed. |
Sorry, something went wrong.
Problem
Fixes #2150.
When applying declarative YAML config, change_retentions resolve their source/destination tables against the tables list cached on the postgres_databases record. That cache is only refreshed when a database is created and by the 6-hourly EnqueueDatabaseUpdateWorker cron — the config-apply update path (update_db/2) never refreshes it. So if a table is created after the last refresh, apply_config fails with:
Because config apply runs at container start (before start_application), this fails the boot and crash-loops the container, which in turn prevents the 6-hourly cron from ever running — so the deployment can't self-heal. (Full analysis in #2150.)
Fix
Refresh the referenced database's tables from the live database on a cache miss while parsing change_retention attrs, then retry the lookup (option 1 from the issue). A new ensure_table_cached/3 helper:
A round-trip is only added when a referenced table is missing from the cache, so applies that don't reference new tables are unchanged.
Test
Adds a change_retentions test that creates a database, clears its cached tables to simulate a stale cache, then applies a config whose change_retention references those tables and asserts the apply succeeds and the pipeline is created.