FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Cherry pick #114006 to 26.3: Reject a {database}/{table} substitution that is not a safe ZooKeeper path component by robot-clickhouse-ci-1 · Pull Request #121289 · ClickHouse/ClickHouse · GitHub

Cherry pick #114006 to 26.3: Reject a {database}/{table} substitution that is not a safe ZooKeeper path component - #121289

Open
robot-clickhouse-ci-1 wants to merge 8 commits into
backport/26.3/114006from
cherrypick/26.3/114006
Open

robot-clickhouse-ci-1 wants to merge 8 commits into
backport/26.3/114006from
cherrypick/26.3/114006

Conversation

Copy link
Copy Markdown
Contributor

Original pull-request #114006

Do not merge this PR manually

This pull-request is a first step of an automated backporting.
It contains changes similar to calling git cherry-pick locally.
If you intend to continue backporting the changes, then resolve all conflicts if any.
Otherwise, if you do not want to backport them, then just close this pull-request.

The check results does not matter at this step - you can safely ignore them.

Before you resolve anything

Conflicts are often caused by a prerequisite change that has not been backported yet, rather than by a real divergence. The bot re-tries this cherry-pick against the release branch on every run, so if that is the case here it will merge itself as soon as the prerequisite lands, and you will see a comment saying so. Manual resolution is only needed while the conflict persists.

Troubleshooting

If the conflicts were resolved in a wrong way

If this cherry-pick PR is completely screwed by a wrong conflicts resolution, and you want to recreate it:

  • delete the pr-cherrypick label from the PR
  • delete this branch from the repository

You also need to check the Original pull-request for pr-backports-created label, and delete if it's presented there

The PR source

The PR is created in the CI job

groeneai and others added 8 commits August 8, 2026 12:43
… path component

TableZnodeInfo::resolve substitutes {database} and {table} into a ReplicatedMergeTree
ZooKeeper path without escaping or validating them. A table or database whose name
contains '/' therefore places its own znodes inside another table's keeper subtree:
with a {database}/{table}-bearing path template (shipped commented out in
programs/server/config.xml and recommended by the docs) or an explicit path,

  CREATE TABLE inj.`victim/replicas/ghost` (c0 Int) ENGINE = ReplicatedMergeTree ORDER BY c0

registers a permanently inactive replica named `ghost` under inj.victim, which then
makes ALTER ... SETTINGS alter_sync = 2 fail with KEEPER_EXCEPTION while still applying
the metadata change, and makes OPTIMIZE ... FINAL hang. The damage survives a plain DROP
of the offending table and a restart; repairing it needs SYSTEM DROP REPLICA.

Two more classes reach a znode name the same way: '.' / '..' as a whole path component,
and control bytes. Both are rejected by the ZooKeeper data model, so the same DDL is
accepted by ClickHouse Keeper and refused by Apache ZooKeeper.

Reject rather than escape, since escaping would change the resulting znode name and so
silently repoint an existing table. Three checks are needed because the classes differ
in kind:

  - '/' is checked on the substituted value. It cannot be seen on the assembled path,
    where a substituted '/' is indistinguishable from a template separator.
  - '.', '..' and control bytes are checked on the fully expanded string, gated on
    whether {database}/{table} reached it in either pass. Checking the raw value would
    wrongly reject an embedded substitution such as `table_{table}` for a table named
    '.', which yields the legal component `table_.`.
  - '{' and '}' are checked on the substituted value, because a name carrying macro
    syntax survives the other two checks and is then expanded again by the second pass.
    A lone '}' is rejected too: it can close a brace opened by a configured macro.

Both macro passes are validated. The second pass expands a configured macro whose value
contains {database}/{table}, which is a real configuration - the test suite's own
macros.xml ships default_path_test = /clickhouse/tables/{database}/{shard}/.

Validation is requested explicitly by the storage factory rather than derived from
`mode`, because extractZooKeeperPathFromReplicatedTableDef re-derives the path of an
existing table with a hardcoded LoadingStrictnessLevel::CREATE and swallows BAD_ARGUMENTS
into nullopt; validating there would silently drop a pre-existing table's replicated data
path from a backup. Fresh definitions (CREATE, DDL replay in a Replicated database, and a
full-definition ATTACH) are validated; short ATTACH, server startup and RESTORE are not,
so tables that exist today keep loading and restoring.

Each expanded output now gets its own MacroExpansionInfo. Macros::expand returns early
for a string without '{' without clearing the flags, so a shared one made expanded_table
sticky and would have rejected an explicitly written replica name because the path
substituted.

An explicitly supplied overlapping or nested zookeeper_path is not addressed here; that
is the same family as issue #22970. RESTORE ... AS <name> renames to a user-chosen
destination before the storage factory runs and is likewise not addressed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Apache ZooKeeper's PathUtils.validatePath rejects U+007F-U+009F, but the
component check only tested single bytes for < 0x20 or == 0x7F, so a name
carrying a C1 control such as U+0085 (UTF-8 C2 85) still reached a znode
name. ClickHouse Keeper validates nothing, so the divergence between the
two coordination backends survived for that range.

The C1 range is detected as its 2-byte UTF-8 encoding C2 80..C2 9F, which
is exact because UTF-8 is a prefix code. Full parity with ZooKeeper is not
implemented on purpose: its rejected set also covers the UTF-16 surrogate
range, and a byte-level widening to 0x80-0x9F would reject the ordinary
continuation bytes of every non-ASCII name. Both shapes reject a name like
таблица_🚀_表, which works today; a test row and a mutant pin that.

A failure on the replica name now advises specifying the replica name
rather than the ZooKeeper path, since those are separate engine arguments.

Four branches had no automated coverage and were each independently
deletable with the suite still green: the {database} branch, a closing
brace with no opening one, the control-character loop, and the two ATTACH
arms of the opt-in guard. Rows needing a per-run unique database name or
UUID go in a .sh companion, because a fixed literal UUID makes parallel
copies collide and an Atomic database rejects a full-definition ATTACH
without one.
Five branches of the substitution validation were independently deletable with
the test suite still green, so nothing pinned them:

  * the short-ATTACH exemption. A table created with a direct {database}/{table}
    stores fully literal text, so on re-ATTACH there is nothing left to
    substitute and the checks are skipped regardless of the flag. Removing the
    exemption therefore left the suite green, even though that exemption is what
    keeps existing path-unsafe tables loading at startup. Covered by a table
    whose path comes from a configured macro, so the {database} does survive
    into metadata, over a database later renamed to a path-unsafe name.
  * the second-pass provenance record. Every existing row reached the output
    through a direct macro in the first pass, so the flag never depended on the
    second. Covered by a row where both macros are configured ones.
  * the replica-name component check. Its only row carried a '/', so the check
    on the substituted value rejected it first. Covered by a table named '..'
    substituted into the replica name with an otherwise safe path.
  * both ends of the C1 range. Only U+0085 in the interior was covered.

Each row was verified by observing the rejection rather than by reasoning about
the escape, and each new branch has a mutant that reddens its row and leaves the
others green.

Both tests also gain no-shared-merge-tree: under --replace-replicated-with-shared
the runner strips the path and replica-name arguments, which deletes the
substitution these tests exist to assert.
…s own metadata

A Replicated database re-derives a table by parsing the CREATE statement it stored in Keeper and
executing it. That statement can still carry an unexpanded macro: full_path_for_metadata is taken
after the first expansion pass, which unfolds only a direct {database}/{table}, so a CONFIGURED
macro whose value contains {database} survives verbatim. On replay the macro expands, the
substitution is recorded, and the new path checks run on a table that already exists.

The replay arrives BELOW LoadingStrictnessLevel::ATTACH, because parseQueryFromMetadata sets
create.attach = false, so `mode` cannot tell it apart from a fresh CREATE, and a user CREATE inside
a Replicated database must stay validated. Carry the provenance on the query context instead and set
it at the two sites that replay stored definitions: DatabaseReplicated::recoverLostReplica and the
restore path in InterpreterSystemQuery.

Measured before the fix: a lost replica of a database renamed to a path-unsafe name never recovers.
The replaying replica loops, rejecting its own stored definition every five seconds (13 attempts
observed, 0 tables recovered), where the pre-fix binary recovers on the first attempt.

Also cover four branches that had no discriminating test row: the second-pass replica-name
provenance, the replica_info term of the rename restriction, the expanded_database half of the
provenance accumulation, and the high end of the C0 control range. Test renamed 04827 -> 04832,
since 04827 was taken upstream in the meantime.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…pplied

Two routes re-derive an existing table's keeper path and were still being judged
as if the user had just supplied it.

SYSTEM RESTART REPLICA re-attaches from stored metadata: doRestartReplica reads
the definition back with getCreateTableQuery, whose attach flag
DatabaseOnDisk::getCreateQueryFromMetadata clears, then sets only create.attach
and calls the factory at ATTACH. That satisfies the full-definition-ATTACH arm of
the guard, and full_path_for_metadata is captured after the first macro pass, so
a configured macro survives into the stored text and re-substitutes. The
statement then failed and, after ten retries, left the table permanently
detached. Measured on a data directory written by a pre-fix server: one of six
path-unsafe tables disappeared from system.tables. The re-attach now gets its own
context copy carrying the recovery provenance, so the shared system context, the
other branches of the same statement and the parallel tasks of SYSTEM RESTART
REPLICAS are unaffected.

SECONDARY_CREATE is by definition a replay of a definition another node already
committed, its only producers being Replicated-database catch-up and RESTORE, so
it is exempted too. Validating it prevented nothing measurable: on a lagging
upgraded replica the rejected entry was retried, the replica declared itself lost,
recovered from Keeper and created the offending table anyway. Measured, 31
rejections and one lost-replica recovery became zero and zero, with the same final
table. A fresh unsafe CREATE is still refused, on the initiator of a Replicated
database, in a plain database and through ON CLUSTER.

A regression row covers the restart route, asserting that the table is still
attached rather than that the statement returned successfully; a rejection leaves
it detached, so the status alone can be satisfied by the wrong outcome.
doRestartReplica detaches the table, then re-creates it inside a block that
retries on every exception and, if all retries fail, adjusts the in-memory
metadata digest so DatabaseReplicated does not later assert a mismatch. An
exception thrown after the detach but before that block reaches neither the
retries nor the digest adjustment, and leaves the table permanently detached.

The context copy carrying the recovery provenance was taken in exactly that
window, and copying a context allocates: a Settings, a QueryAccessInfo and the
table function result map. Take it before anything is detached instead, where an
allocation failure is harmless.

Pure relocation of two statements. The context is only read afterwards, it is
still the local context argument of the factory call, and nothing between the
old and the new position mutates the context being copied, so the copy's
contents cannot depend on where it is taken. Widening its lifetime across the
detach costs nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ing_view walk

Six readability and API-shape changes requested in review, no behaviour change:

* the "ZooKeeper path" / "replica name" subject of the messages becomes an enum
  with constexpr toString and howToOverride functions, instead of a string_view
  compared against a literal at the point of use
* checkSubstitutedValues finds the offending character with a single
  find_first_of and reports that character, instead of two contains() calls and
  a hardcoded description per case
* the component walk in checkPathComponents uses std::string_view find and
  remove_prefix instead of index arithmetic over the whole string
* the control-character code point becomes std::optional<UInt16> instead of an
  Int32 with -1 as the sentinel
* the resolve definition is wrapped to match the declaration
* the is_recovery_from_stored_metadata comment is shortened

Verified equivalent on the 33-row per-row prober for both tests plus a boundary
probe covering the first, middle and last component of the walk, control bytes at
the first, interior and last byte of a component, a bare C2 at the end of a
component, both ends of the C0 and C1 ranges and their neighbours, and each of
'/', '{' and '}' in a substituted value. A mutant that skips the last component
reddens 12 prober rows, 3 boundary rows and the test suite, so the probe
discriminates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tution-validation

Reject a {database}/{table} substitution that is not a safe ZooKeeper path component
robot-clickhouse-ci-1 added pr-cherrypick Cherry-pick of merge-commit before backporting. Do not use manually - automated use only! do not test disable testing on pull request pr-bugfix Pull request with bugfix, not backported by default labels Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not test disable testing on pull request pr-bugfix Pull request with bugfix, not backported by default pr-cherrypick Cherry-pick of merge-commit before backporting. Do not use manually - automated use only!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants


Back | FazBrowse Home | New Git URL