| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
Problem: Persist exports 28 metric families with one series per shard, so the scrape body grows with the shard count until it goes over the scraper's size limit. Dropping those series on the scraper side doesn't help, because the limit applies to the body the process sends, before the scraper looks at any series. The process has to stop sending them. Solution Add persist_shard_metrics, which picks what a process exports: - per_shard: only the per-shard families (what we had before) - summary: only their bounded aggregates (the _percentile and _topk families and mz_persist_batch_part_version_count) (what we want to do long term) - both: everything, which is the default (for now) and matches what's exported without this setting. - none: neither Shard counts and the process-level counters are exported in every mode. An unrecognized value exports the default. The parse itself is silent, since both readers run per scrape and would otherwise log forever. The postprocessor reports it once per change instead, and mz_persist_shard_metrics_mode_invalid counts the scrapes it affected. Production keeps both (for now), because dashboards that read the per-shard families stop getting data without them. Tests default to summary for now Testing New tests switch one Metrics through all four modes and check that each exports exactly the families it should, that the process-level counters hold their value in every mode, and that an unrecognized value exports the default and counts the scrapes it hit Closes: PER-102
|
This is the second in the stack of 3: github.com//pull/39354 |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Problem:
Persist exports 28 metric families with one series per shard, so the scrape body grows with the shard count until it goes over the scraper's size limit.
Dropping those series on the scraper side doesn't help, because the limit applies to the body the process sends, before the scraper looks at any series. The process has to stop sending them.
Solution
Add persist_shard_metrics, which picks what a process exports:
Shard counts and the process-level counters are exported in every mode.
An unrecognized value exports the default. The parse itself is silent, since both readers run per scrape and would otherwise log forever. The postprocessor reports it once per change instead, and mz_persist_shard_metrics_mode_invalid counts the scrapes it affected.
Production keeps both (for now), because dashboards that read the per-shard families stop getting data without them. Tests default to summary for now
Testing
New tests switch one Metrics through all four modes and check that each exports exactly the families it should, that the process-level counters hold their value in every mode, and that an unrecognized value exports the default and counts the scrapes it hit
Closes: PER-102