FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Keep cache statement names within PostgreSQL's 63-byte limit by aiboupeter · Pull Request #751 · elixir-ecto/ecto_sql · GitHub

Keep cache statement names within PostgreSQL's 63-byte limit - #751

Open
aiboupeter wants to merge 1 commit into
elixir-ecto:masterfrom
aiboupeter:cache-statement-name-limit
Open

Keep cache statement names within PostgreSQL's 63-byte limit#751
aiboupeter wants to merge 1 commit into
elixir-ecto:masterfrom
aiboupeter:cache-statement-name-limit

Conversation

aiboupeter commented Aug 22, 2026
edited
Loading

Copy link
Copy Markdown

PostgreSQL silently truncates prepared statement names to NAMEDATALEN - 1 (63 bytes) — the statement hash table is keyed with HASH_STRINGS at NAMEDATALEN, so only the first 63 bytes of a name are compared (pgsql-bugs discussion; the application is expected to avoid it).

The default cache_statement names (ecto_<op>_<source>_<n>, ecto_insert_all_<source>) exceed that for long table names, so two sources that only differ after the 63rd byte map to one server-side statement while Postgrex caches them under distinct client names. On one connection:

  1. insert into business_workplace_attendance_leave_comp_rest_minutes — prepares ecto_insert_business_workplace_attendance_leave_comp_rest_minutes_0 (server stores it as ecto_insert_business_workplace_attendance_leave_comp_rest_minut)
  2. insert into business_workplace_attendance_leave_comp_rest_minutes_event_logs — a different client name, so Postgrex closes+parses ecto_insert_…_event_logs_0, which the server also truncates to the same 63 bytes → silently replaces the statement from step 1
  3. insert into the first table again — client cache hit, so it binds straight away:
    ** (Postgrex.Error) ERROR 08P01 (protocol_violation) bind message supplies 8 parameters,
    but prepared statement "ecto_insert_business_workplace_attendance_leave_comp_rest_minutes_0" requires 5
    
    08P01 isn't one of the codes that evict the client cache, so every later write to that table on that connection fails the same way until it reconnects.

This is what we hit in production; the new integration test (integration_test/pg/prepare_test.exs) reproduces it on master and passes with this change.

Change

Ecto.Adapters.SQL.cache_statement_name/3 caps the generated name at 63 bytes. Names that already fit are returned byte-for-byte unchanged, so the vast majority of applications see no difference. Longer names keep a (UTF-8 safe) prefix of the source and append :erlang.phash2/2 of the full source in base 36, e.g.

ecto_insert_business_workplace_attendance_leave_comp_1YNQTIL_0
ecto_insert_business_workplace_attendance_leave_comp_r_AFUHH7_0

The cap is applied in the shared SQL layer rather than only in the Postgres adapter: MySQL/TDS don't have the limit, but shorter names are harmless there and it keeps one code path.

Tests

  • test/ecto/adapters/sql_test.exs — unit tests for the helper (unchanged short names, ≤ 63 bytes, distinct for colliding sources, no split multibyte characters).
  • integration_test/pg/prepare_test.exs — two TEMP tables sharing a 51-byte prefix, insert A → B → A → B on one sandboxed connection. Fails on master with the 08P01 above, passes with the fix.
  • mix test: 694 tests, 0 failures. ECTO_ADAPTER=pg mix test: 504 tests, 0 failures (PostgreSQL 18.6 local).

🤖 Generated with Claude Code

PostgreSQL silently truncates prepared statement names to NAMEDATALEN - 1
(63 bytes). The default names `ecto_<op>_<source>_<n>` and
`ecto_insert_all_<source>` exceed that for long table names, so two sources
that only differ after the 63rd byte (e.g. `..._leave_comp_rest_minutes` and
`..._leave_comp_rest_minutes_event_logs`) map to a single server-side
statement while Postgrex caches them under distinct client names. Once the
second table has been prepared on a connection, the next cached execution of
the first binds against the wrong statement:

    ERROR 08P01 (protocol_violation) bind message supplies 8 parameters,
    but prepared statement "ecto_insert_..." requires 5

and, since that error does not evict the client cache, the connection keeps
failing until it reconnects.

Cap generated names at 63 bytes: names that already fit are unchanged; longer
ones keep a prefix of the source and add a hash of the full source so they
stay distinct within the limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread lib/ecto/adapters/sql.ex
if byte_size(name) <= @max_cache_statement_name_size do
name
else
hash = source |> :erlang.phash2(4_294_967_296) |> Integer.to_string(36)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Why the phash2 explicit limit?

Comment thread lib/ecto/adapters/sql.ex
Comment on lines +1191 to +1193
budget = @max_cache_statement_name_size - byte_size(prefix <> suffix <> hash) - 1
kept = source |> truncate_utf8(max(budget, 0)) |> String.trim_trailing("_")
prefix <> kept <> "_" <> hash <> suffix

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

No need to trim or add _, it is more important to be faster than have pretty names:

Suggested change
budget = @max_cache_statement_name_size - byte_size(prefix <> suffix <> hash) - 1
kept = source |> truncate_utf8(max(budget, 0)) |> String.trim_trailing("_")
prefix <> kept <> "_" <> hash <> suffix
budget = @max_cache_statement_name_size - byte_size(prefix <> suffix <> hash)
kept = source |> truncate_utf8(max(budget, 0))
prefix <> kept <> hash <> suffix

Comment thread lib/ecto/adapters/sql.ex
end
end

defp truncate_utf8(string, size) when byte_size(string) <= size, do: string

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Instead of truncating utf8, you can traverse the string and collect the first n-valid ascii bytes. It should be cheaper to implement.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants


Back | FazBrowse Home | New Git URL