| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
… path component
TableZnodeInfo::resolve substitutes {database} and {table} into a ReplicatedMergeTree
ZooKeeper path without escaping or validating them. A table or database whose name
contains '/' therefore places its own znodes inside another table's keeper subtree:
with a {database}/{table}-bearing path template (shipped commented out in
programs/server/config.xml and recommended by the docs) or an explicit path,
CREATE TABLE inj.`victim/replicas/ghost` (c0 Int) ENGINE = ReplicatedMergeTree ORDER BY c0
registers a permanently inactive replica named `ghost` under inj.victim, which then
makes ALTER ... SETTINGS alter_sync = 2 fail with KEEPER_EXCEPTION while still applying
the metadata change, and makes OPTIMIZE ... FINAL hang. The damage survives a plain DROP
of the offending table and a restart; repairing it needs SYSTEM DROP REPLICA.
Two more classes reach a znode name the same way: '.' / '..' as a whole path component,
and control bytes. Both are rejected by the ZooKeeper data model, so the same DDL is
accepted by ClickHouse Keeper and refused by Apache ZooKeeper.
Reject rather than escape, since escaping would change the resulting znode name and so
silently repoint an existing table. Three checks are needed because the classes differ
in kind:
- '/' is checked on the substituted value. It cannot be seen on the assembled path,
where a substituted '/' is indistinguishable from a template separator.
- '.', '..' and control bytes are checked on the fully expanded string, gated on
whether {database}/{table} reached it in either pass. Checking the raw value would
wrongly reject an embedded substitution such as `table_{table}` for a table named
'.', which yields the legal component `table_.`.
- '{' and '}' are checked on the substituted value, because a name carrying macro
syntax survives the other two checks and is then expanded again by the second pass.
A lone '}' is rejected too: it can close a brace opened by a configured macro.
Both macro passes are validated. The second pass expands a configured macro whose value
contains {database}/{table}, which is a real configuration - the test suite's own
macros.xml ships default_path_test = /clickhouse/tables/{database}/{shard}/.
Validation is requested explicitly by the storage factory rather than derived from
`mode`, because extractZooKeeperPathFromReplicatedTableDef re-derives the path of an
existing table with a hardcoded LoadingStrictnessLevel::CREATE and swallows BAD_ARGUMENTS
into nullopt; validating there would silently drop a pre-existing table's replicated data
path from a backup. Fresh definitions (CREATE, DDL replay in a Replicated database, and a
full-definition ATTACH) are validated; short ATTACH, server startup and RESTORE are not,
so tables that exist today keep loading and restoring.
Each expanded output now gets its own MacroExpansionInfo. Macros::expand returns early
for a string without '{' without clearing the flags, so a shared one made expanded_table
sticky and would have rejected an explicitly written replica name because the path
substituted.
An explicitly supplied overlapping or nested zookeeper_path is not addressed here; that
is the same family as issue #22970. RESTORE ... AS <name> renames to a user-chosen
destination before the storage factory runs and is likewise not addressed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Apache ZooKeeper's PathUtils.validatePath rejects U+007F-U+009F, but the
component check only tested single bytes for < 0x20 or == 0x7F, so a name
carrying a C1 control such as U+0085 (UTF-8 C2 85) still reached a znode
name. ClickHouse Keeper validates nothing, so the divergence between the
two coordination backends survived for that range.
The C1 range is detected as its 2-byte UTF-8 encoding C2 80..C2 9F, which
is exact because UTF-8 is a prefix code. Full parity with ZooKeeper is not
implemented on purpose: its rejected set also covers the UTF-16 surrogate
range, and a byte-level widening to 0x80-0x9F would reject the ordinary
continuation bytes of every non-ASCII name. Both shapes reject a name like
таблица_🚀_表, which works today; a test row and a mutant pin that.
A failure on the replica name now advises specifying the replica name
rather than the ZooKeeper path, since those are separate engine arguments.
Four branches had no automated coverage and were each independently
deletable with the suite still green: the {database} branch, a closing
brace with no opening one, the control-character loop, and the two ATTACH
arms of the opt-in guard. Rows needing a per-run unique database name or
UUID go in a .sh companion, because a fixed literal UUID makes parallel
copies collide and an Atomic database rejects a full-definition ATTACH
without one.
Five branches of the substitution validation were independently deletable with
the test suite still green, so nothing pinned them:
* the short-ATTACH exemption. A table created with a direct {database}/{table}
stores fully literal text, so on re-ATTACH there is nothing left to
substitute and the checks are skipped regardless of the flag. Removing the
exemption therefore left the suite green, even though that exemption is what
keeps existing path-unsafe tables loading at startup. Covered by a table
whose path comes from a configured macro, so the {database} does survive
into metadata, over a database later renamed to a path-unsafe name.
* the second-pass provenance record. Every existing row reached the output
through a direct macro in the first pass, so the flag never depended on the
second. Covered by a row where both macros are configured ones.
* the replica-name component check. Its only row carried a '/', so the check
on the substituted value rejected it first. Covered by a table named '..'
substituted into the replica name with an otherwise safe path.
* both ends of the C1 range. Only U+0085 in the interior was covered.
Each row was verified by observing the rejection rather than by reasoning about
the escape, and each new branch has a mutant that reddens its row and leaves the
others green.
Both tests also gain no-shared-merge-tree: under --replace-replicated-with-shared
the runner strips the path and replica-name arguments, which deletes the
substitution these tests exist to assert.
…s own metadata
A Replicated database re-derives a table by parsing the CREATE statement it stored in Keeper and
executing it. That statement can still carry an unexpanded macro: full_path_for_metadata is taken
after the first expansion pass, which unfolds only a direct {database}/{table}, so a CONFIGURED
macro whose value contains {database} survives verbatim. On replay the macro expands, the
substitution is recorded, and the new path checks run on a table that already exists.
The replay arrives BELOW LoadingStrictnessLevel::ATTACH, because parseQueryFromMetadata sets
create.attach = false, so `mode` cannot tell it apart from a fresh CREATE, and a user CREATE inside
a Replicated database must stay validated. Carry the provenance on the query context instead and set
it at the two sites that replay stored definitions: DatabaseReplicated::recoverLostReplica and the
restore path in InterpreterSystemQuery.
Measured before the fix: a lost replica of a database renamed to a path-unsafe name never recovers.
The replaying replica loops, rejecting its own stored definition every five seconds (13 attempts
observed, 0 tables recovered), where the pre-fix binary recovers on the first attempt.
Also cover four branches that had no discriminating test row: the second-pass replica-name
provenance, the replica_info term of the rename restriction, the expanded_database half of the
provenance accumulation, and the high end of the C0 control range. Test renamed 04827 -> 04832,
since 04827 was taken upstream in the meantime.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…pplied Two routes re-derive an existing table's keeper path and were still being judged as if the user had just supplied it. SYSTEM RESTART REPLICA re-attaches from stored metadata: doRestartReplica reads the definition back with getCreateTableQuery, whose attach flag DatabaseOnDisk::getCreateQueryFromMetadata clears, then sets only create.attach and calls the factory at ATTACH. That satisfies the full-definition-ATTACH arm of the guard, and full_path_for_metadata is captured after the first macro pass, so a configured macro survives into the stored text and re-substitutes. The statement then failed and, after ten retries, left the table permanently detached. Measured on a data directory written by a pre-fix server: one of six path-unsafe tables disappeared from system.tables. The re-attach now gets its own context copy carrying the recovery provenance, so the shared system context, the other branches of the same statement and the parallel tasks of SYSTEM RESTART REPLICAS are unaffected. SECONDARY_CREATE is by definition a replay of a definition another node already committed, its only producers being Replicated-database catch-up and RESTORE, so it is exempted too. Validating it prevented nothing measurable: on a lagging upgraded replica the rejected entry was retried, the replica declared itself lost, recovered from Keeper and created the offending table anyway. Measured, 31 rejections and one lost-replica recovery became zero and zero, with the same final table. A fresh unsafe CREATE is still refused, on the initiator of a Replicated database, in a plain database and through ON CLUSTER. A regression row covers the restart route, asserting that the table is still attached rather than that the statement returned successfully; a rejection leaves it detached, so the status alone can be satisfied by the wrong outcome.
doRestartReplica detaches the table, then re-creates it inside a block that retries on every exception and, if all retries fail, adjusts the in-memory metadata digest so DatabaseReplicated does not later assert a mismatch. An exception thrown after the detach but before that block reaches neither the retries nor the digest adjustment, and leaves the table permanently detached. The context copy carrying the recovery provenance was taken in exactly that window, and copying a context allocates: a Settings, a QueryAccessInfo and the table function result map. Take it before anything is detached instead, where an allocation failure is harmless. Pure relocation of two statements. The context is only read afterwards, it is still the local context argument of the factory call, and nothing between the old and the new position mutates the context being copied, so the copy's contents cannot depend on where it is taken. Widening its lifetime across the detach costs nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ing_view walk
Six readability and API-shape changes requested in review, no behaviour change:
* the "ZooKeeper path" / "replica name" subject of the messages becomes an enum
with constexpr toString and howToOverride functions, instead of a string_view
compared against a literal at the point of use
* checkSubstitutedValues finds the offending character with a single
find_first_of and reports that character, instead of two contains() calls and
a hardcoded description per case
* the component walk in checkPathComponents uses std::string_view find and
remove_prefix instead of index arithmetic over the whole string
* the control-character code point becomes std::optional<UInt16> instead of an
Int32 with -1 as the sentinel
* the resolve definition is wrapped to match the declaration
* the is_recovery_from_stored_metadata comment is shortened
Verified equivalent on the 33-row per-row prober for both tests plus a boundary
probe covering the first, middle and last component of the walk, control bytes at
the first, interior and last byte of a component, a bare C2 at the end of a
component, both ends of the C0 and C1 ranges and their neighbours, and each of
'/', '{' and '}' in a substituted value. A mutant that skips the last component
reddens 12 prober rows, 3 boundary rows and the test suite, so the probe
discriminates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tution-validation
Reject a {database}/{table} substitution that is not a safe ZooKeeper path component
| Back | FazBrowse Home | New Git URL |
Original pull-request #114006
Do not merge this PR manually
This pull-request is a first step of an automated backporting.
It contains changes similar to calling git cherry-pick locally.
If you intend to continue backporting the changes, then resolve all conflicts if any.
Otherwise, if you do not want to backport them, then just close this pull-request.
The check results does not matter at this step - you can safely ignore them.
Before you resolve anything
Conflicts are often caused by a prerequisite change that has not been backported yet, rather than by a real divergence. The bot re-tries this cherry-pick against the release branch on every run, so if that is the case here it will merge itself as soon as the prerequisite lands, and you will see a comment saying so. Manual resolution is only needed while the conflict persists.
Troubleshooting
If the conflicts were resolved in a wrong way
If this cherry-pick PR is completely screwed by a wrong conflicts resolution, and you want to recreate it:
You also need to check the Original pull-request for pr-backports-created label, and delete if it's presented there
The PR source
The PR is created in the CI job