FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

persist: choose which shard metrics to export by mtabebe · Pull Request #39591 · MaterializeInc/materialize · GitHub

persist: choose which shard metrics to export - #39591

Open
mtabebe wants to merge 1 commit into
MaterializeInc:mainfrom
mtabebe:michaelabebe/per-102-persist-choose-shard-metrics
Open

mtabebe wants to merge 1 commit into
MaterializeInc:mainfrom
mtabebe:michaelabebe/per-102-persist-choose-shard-metrics

Conversation

mtabebe commented Oct 6, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Problem:

Persist exports 28 metric families with one series per shard, so the scrape body grows with the shard count until it goes over the scraper's size limit.

Dropping those series on the scraper side doesn't help, because the limit applies to the body the process sends, before the scraper looks at any series. The process has to stop sending them.

Solution

Add persist_shard_metrics, which picks what a process exports:

  • per_shard: only the per-shard families (what we had before)
  • summary: only their bounded aggregates (the _percentile and _topk families and mz_persist_batch_part_version_count) (what we want to do long term)
  • both: everything, which is the default (for now) and matches what's exported without this setting.
  • none: neither

Shard counts and the process-level counters are exported in every mode.

An unrecognized value exports the default. The parse itself is silent, since both readers run per scrape and would otherwise log forever. The postprocessor reports it once per change instead, and mz_persist_shard_metrics_mode_invalid counts the scrapes it affected.

Production keeps both (for now), because dashboards that read the per-shard families stop getting data without them. Tests default to summary for now

Testing

New tests switch one Metrics through all four modes and check that each exports exactly the families it should, that the process-level counters hold their value in every mode, and that an unrecognized value exports the default and counts the scrapes it hit

Closes: PER-102

Problem:

Persist exports 28 metric families with one series per shard, so the
scrape body grows with the shard count until it goes over the
scraper's size limit.

Dropping those series on the scraper side doesn't help, because the
limit applies to the body the process sends, before the scraper looks
at any series. The process has to stop sending them.

Solution

Add persist_shard_metrics, which picks what a process exports:

- per_shard: only the per-shard families (what we had before)
- summary: only their bounded aggregates (the _percentile and _topk
  families and mz_persist_batch_part_version_count) (what we want to do
  long term)
- both: everything, which is the default (for now) and matches what's exported
  without this setting.
- none: neither

Shard counts and the process-level counters are exported in every
mode.

An unrecognized value exports the default. The parse itself is silent,
since both readers run per scrape and would otherwise log forever. The
postprocessor reports it once per change instead, and
mz_persist_shard_metrics_mode_invalid counts the scrapes it affected.

Production keeps both (for now), because dashboards that read the
per-shard families stop getting data without them.
Tests default to summary for now

Testing

New tests switch one Metrics through all four modes and check that
each exports exactly the families it should, that the process-level
counters hold their value in every mode, and that an unrecognized value
exports the default and counts the scrapes it hit

Closes: PER-102

mtabebe commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

This is the second in the stack of 3: github.com//pull/39354

mtabebe requested a review from Alphadelta14 October 6, 2026 15:48
mtabebe marked this pull request as ready for review October 6, 2026 15:48
mtabebe requested review from a team as code owners October 6, 2026 15:48

This branch has not been deployed

No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants


Back | FazBrowse Home | New Git URL