FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

docs: add symptom-first troubleshooting guides by JeffLuoo · Pull Request #1296 · agent-substrate/substrate · GitHub

docs: add symptom-first troubleshooting guides - #1296

Open
Jeff Luo (JeffLuoo) wants to merge 3 commits into
agent-substrate:mainfrom
JeffLuoo:troubleshooting-guide
Open

docs: add symptom-first troubleshooting guides#1296
Jeff Luo (JeffLuoo) wants to merge 3 commits into
agent-substrate:mainfrom
JeffLuoo:troubleshooting-guide

Conversation

Copy link
Copy Markdown
Collaborator

Fixes #1294

It's converted from this public doc. Please feel free to drop comments in the doc and I can update this PR.

Copy link
Copy Markdown
Collaborator

I noticed several drifts between this guide and the metrics definition (like the error code etc) and I think it will become more programatic in the future because it will be very hard to keep the guide the metrics definition synced.

I think we should make the guide very thin. The durable structure is: guide owns the decision tree and interpretation, registry owns definitions, and the guide links rather than paraphrases. Every value taxonomy or default it copies inline is future debt.

The metric registry says what each instrument measures, and the
observability guide says how telemetry is collected, but neither takes an
operator from a symptom to a cause. A user reporting a slow request or a
503 had no path through the data.

These three guides each start at a symptom and end at a component. Every
step gives a query, says how to read the result, and names the next step
or the guide to switch to. The shared index holds what all three need:
the two spellings of the instrument names and which ingest path produces
each, the four reasons a query returns nothing, and the pointer to the
subsystems that emit no metrics at all.

Every query was run against a live cluster, and the readings drove the
text: the resume label is only meaningful next to the outcome, a fleet
gauge that every replica reports needs max rather than sum, and a phase
histogram has to exclude failures before its quantile means anything.
The guides copied label taxonomies and flag defaults inline. A copy goes out
of date without a signal, and one already had: the guide said that
ate.imagecache.requests reports _OTHER for each status that is not 401, 403
or 429, but the registry and internal/imagecache/metrics.go also pass 404,
500, 502, 503 and 504 through as literal values.

Give the guides one job. They own the decision tree: which query to run, how
to read the result, and where to go next. The registry owns the definitions.
Drop the value tables for ate.snapshot.kind, ate.failure.reason,
ate.scheduling.constraint and the parking outcome, and link to the registry
group instead. Drop the --parked-request-max default and link to
request-parking.md, which owns the flag. Drop the histogram bucket ceilings
and the two sample measurements, which the surrounding argument does not
need. The phase, reason and outcome tables that remain hold routing and not
definitions.

Fix two drifts on the way: three guides said "three reasons" for an empty
query where the README lists four, and capacity-is-full.md said "two states"
above a three-row table.

Copy link
Copy Markdown
Collaborator Author

I noticed several drifts between this guide and the metrics definition (like the error code etc) and I think it will become more programatic in the future because it will be very hard to keep the guide the metrics definition synced.

I think we should make the guide very thin. The durable structure is: guide owns the decision tree and interpretation, registry owns definitions, and the guide links rather than paraphrases. Every value taxonomy or default it copies inline is future debt.

Agreed, and updated to remove some metrics details from the guide.

Review found decision-tree arrows that send an operator to the wrong
component, and queries that contradict the paragraph beside them.

Routing. The router maps a shed request and an exhausted park budget onto
the same no_capacity outcome, so that outcome can no longer send the
reader straight to the capacity guide; step 3 now splits the two on
parking.rejected, which is the only signal that separates them. The
timeout row claimed an ended park budget, which never classifies that
way: an ended budget surfaces the condition that blocked the resume. The
capacity guide sent the common path to "make spec.replicas larger"
without passing the step that asks whether the workers are doing any
work.

Hidden capacity has two causes the guides did not name: a paused actor is
pinned to the node holding its snapshot and reports required_nodes rather
than selector, and the idle worker count includes draining workers that
the scheduler will never pick.

Queries. The CPU rate summed across the stats source that the text below
it says never to add together. The snapshot-size quantile dropped the
file name that the next line says to compare on. The step 1 restore
quantile counted failed phases, which hold their timers, while step 2
excludes them. The desired-minus-ready subtraction used a bare on() join
that fails once a second replica reports the same pool.

A paused restore was also documented as never downloading, which is
untrue for the data_on_golden scope: the golden files still come from
object storage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

troubleshooting guide

2 participants


Back | FazBrowse Home | New Git URL