…d in current code base)

…ternal to develop baseline

…SONS_LEARNED.md, update for v0.9.0 S3 paths and S3 digest extraction

…er to test infra, update docs for S3 digests

- Deploy ZkEvmVerifierPostFeynman with S3 digests and protocolVersion=10

- Move shadow-only verifier wrapper out of scroll-contracts submodule into tests/shadow-testing/contracts/

- Update GUIDE/TROUBLESHOOTING/README: prefer S3 digest files, remove proof-extraction fallback

- Prover-up script: asset detours for v0.9.0 S3 layout

- fetch-l2-blocks: store full header/transactions for coordinator chunk-task generation

- libzkp: add openvm-sdk/openvm-stark-sdk, input_commits, batch_root_verifier_vk loading

- prover-bin: adapt dumper and universal handler to v0.9.0 ProverConfig/StarkProof

- common/message: introduce OpenVMStarkProof with UserPvsProof, switch chunk/batch to StarkProof

- coordinator: update proof receiver, mock verifier and tests for StarkProof

- rust-toolchain: bump to nightly-2025-11-20 + rust-src

- regenerate Cargo.lock to resolve v0.9.0 dependency graph

- prover.rs: load child circuit handler and call enable_deferral for Batch/Bundle tasks

- UniversalHandler: expose enable_deferral wrapper around Prover::enable_deferral

- CircuitConfig: add child_circuit_vks for chunk/batch child lookup

- 04-prover-up.sh: populate child_circuit_vks from openVmVk.json

…rifier, add v0.9.0 lessons

formatProverTask/applyUniversal/InsertProverTask failures after
Update*Attempts previously refunded only active_attempts, permanently
burning total_attempts. After SessionAttempts (5) such tasks became
invisible to GetUnassigned* (total_attempts < maxAttempts) with no
prover_task row, so the cron timeout checker could never mark them
failed either — silent starvation (shadow-testing Trap 22/24).

Add RefundAttemptsByHash (chunk/batch/bundle orm) and recoverAttempts:
when the charge happened in this Assign call (hasAssignedTask == nil),
refund both counters and reset proving_status to unassigned; re-polls
of already-assigned tasks keep the old active-only recovery. Also cover
the previously unrecovered fixCompatibility failure sites in
batch/bundle assign.

- lib/: scripts and configs shared by both modes (anvil/verifier/prover/
  relayer setup, sync-queue-hashes, anvil-utils)
- follow/: follow-mode scripts, configs, Makefile, GUIDE, TROUBLESHOOTING
  (poll-sync follow of mainnet tip, finalize on Anvil fork)
- snapshot/: snapshot-replay scripts, configs, Makefile, GUIDE,
  TROUBLESHOOTING (historical fork + fixed bundle range)
- docs/COMMON-TROUBLESHOOTING.md: mode-independent traps; per-mode
  TROUBLESHOOTING files carry the mode-tagged traps verbatim
- root Makefile becomes a thin dispatcher; README rewritten as mode chooser
- 10-follow-up.sh: drop follow-run.env on --reset so a fresh run gets a
  fresh report window; assert postgres log_statement='mod' on every setup
  (also in 02-prepare-db.sh) for write attribution
- update AGENTS.md references and .gitignore for per-mode config dirs

- l2_config.disable_l2_watcher: skip the L2 missing-blocks fetch loop.
  In shadow mode the DB is populated by import/poll-sync; an empty
  l2_block table previously made the watcher crawl from genesis
  (shadow Trap 26). Default false keeps production behavior.
- sender_config.chain_nonce_only: initialize the sender nonce from the
  chain pending nonce only, ignoring pending_transaction rows, which
  are not chain/fork-scoped and poisoned nonces on fresh forks
  (shadow Traps 7/27). Default false keeps max(db+1, chain) behavior.
- estimategas: set an explicit 30M gas cap in the estimation CallMsg
  instead of 0; Anvil rejects fee-capped eth_estimateGas with Gas=0
  (shadow Trap 9). The cap only bounds the binary search, estimates
  are unchanged.
- l2_relayer: move full-calldata dumps on commit/finalize send errors
  from Error to Debug (was flooding logs every retry tick), and log a
  loud Error when a bundle is marked RollupFinalizeFailed — status 7
  is never retried by ProcessPendingBundles and needs manual reset.

- relayer.json.template: disable_l2_watcher + chain_nonce_only
- 10-follow-up.sh --reset: TRUNCATE pending_transaction (stale rows
  poison sender nonces and can replay old calldata on a fresh fork)
- mark Traps 26/7/9 as fixed upstream; document the new config knobs
  in follow/GUIDE.md

…hase

- 10-follow-up.sh: FOLLOW_FORK_HOURS_BACK (default 5) — fork at
  tip - hours*300 blocks so the run opens with a real backlog
  (committed-but-not-finalized at fork + everything produced since);
  lastFinalized/lastCommitted now read AT the fork block; 0 = old
  fork-at-tip behavior. follow-run.env records FORK_HOURS_BACK and
  MAINNET_BUNDLE_TIP_AT_START.
- generate-catchup-report.py: split report into catch-up phase (fork
  finalized index climbing to the recorded mainnet bundle tip) and
  steady-state phase; use the max_finalized_bundle watermark for
  finalized counts; backward-compatible with old env files.
- docs: follow README/GUIDE + AGENTS.md describe catch-up-then-follow.

…over

The impersonated eth_sendTransaction for updateVerifier could still be
in the txpool when addProver signed with the same owner nonce, failing
with 'replacement transaction underpriced' (hit twice at bring-up).
Poll for the receipt before continuing.

…rover in follow-up (traps 28/29)

… blob-base-fee); accept Fusaka-era fee floor

…eration

- Pin scroll-zkvm-* to zkvm-prover master bf887150 (halo2-gpu support):
  the SNARK (bundle) aggregation prover can now run on GPU via the new
  `halo2-gpu` cargo feature (24 GB-class GPUs), exposed as the
  `prover_halo2gpu` target in zkvm-prover/Makefile.
- Prover now downloads agg_vk.bin alongside app.vmexe/openvm.toml: without
  the pre-built aggregation VK the prover derives it from the SDK, which is
  slow and allocates GPU memory that is never reclaimed (the issue the
  upstream commit fixed). agg_vk.bin must exist on S3 next to each circuit.
- Cargo.lock: revert revm-family resolution drift introduced by
  `cargo update -p scroll-zkvm-*` (alloy-evm back to revm 30.1.1,
  revm-primitives edges back to the scroll-v91 fork) so the [patch]
  fork resolution keeps compiling. See follow/TROUBLESHOOTING.md Trap 32.
- Docs: new-stack checklist in follow/GUIDE.md mid-run upgrade section,
  Trap 32 (revm drift) + Trap 33 (agg_vk.bin / batch_root_verifier_vk
  mandatory assets) in follow/TROUBLESHOOTING.md, AGENTS.md build command
  and S3 asset notes.

…_vk.bin

libzkp/coordinator:
- Batch-proof verification now reads the batch circuit's agg_vk.bin from
  the assets dir instead of a separate batch_root_verifier_vk file (same
  content, one less artifact to track).

Upgrade-run fixes learned from the first live 20-upgrade.sh run:
- 20-upgrade.sh reset SQL: also reset rows already at proving_status=1 —
  their ready flags (batch_proofs_status/chunk_proofs_status) and proof
  blobs can still reference old-circuit proofs — and DELETE stale assigned
  prover_task rows while provers are stopped. Previously the new
  coordinator rebuilt poisoned tasks from old-format proofs on every poll
  and the priority dispatcher starved all task types (Trap 35).
- 03-deploy-verifier.sh: accept foundry >= 1.6 'cast receipt' status
  format ('1 (success)') so a successful updateVerifier is not misreported
  as reverted (Trap 36).

Docs:
- TROUBLESHOOTING: Trap 34 (stale ProverProofInvalid rows silently starve
  batch assignment; sweeper does not clear prover_task), Trap 35
  (post-upgrade leftover state), Trap 36 (cast status format); Trap 33
  rewritten for the agg_vk.bin-only asset story.
- GUIDE: upgrade notes — aggressive reset rationale, slow first chunk
  task generation (per-block witness fetching) is normal.
- AGENTS.md: coordinator reads batch circuit's agg_vk.bin.

…_hook_commit

Two crashes hit the first halo2-gpu bundle proving on the shadow fork
(both provers died at the SNARK stage, peak 22.9 GiB):

1. quotient.cu cudaErrorInvalidConfiguration: UniversalHandler::agg_vk()
   built the child's full GPU aggregation prover (~5.7 GiB/circuit, never
   reclaimed by the VPMM pool) just to read the VK, starving the SNARK
   phase until quotient chunking computed batch_size = 0. Now uses
   Prover::load_agg_vk() which reads the pre-built agg_vk.bin asset
   (matches zkvm-prover bf887150's VRAM-budgeting guidance).

2. 'def_hook_commit must be defined': the bundle verify circuit's hook
   commit comes from the batch child SDK's deferral prover, which used to
   be initialized as a side effect of sdk.agg_vk(). do_prove now runs
   batch.enable_deferral(chunk) before bundle.enable_deferral(batch) for
   bundle tasks, mirroring the zkvm integration tester.

Also: generate-catchup-report.py accepts +0000 timezone offsets.

Verified end-to-end on the mid-run upgrade shadow fork: bundles 18246/18247
(end_batch 518861/518862 >= upgrade boundary) were re-proven with the
halo2-gpu stack and finalized through the newly-registered wrapper; older
bundles finalized through the legacy wrapper. Documented as Trap 38;
also adds Trap 37 (stale embedded libzkp.so after coordinator rebuild).

… VRAM starvation

Fix 1 (load_agg_vk from file) removed the 5.7 GiB/circuit agg-VK
derivation, but a prover process that served chunk/batch tasks before a
bundle task still crashed at the halo2_outer SNARK stage with
quotient.cu cudaErrorInvalidConfiguration (peak 22.8 GiB on 24 GB
cards): every circuit's GPU SDK stays resident once lazily built, and
the VPMM pool never returns pages, so the quotient chunking heuristic
(raw cudaMemGetInfo) computed batch_size = 0.

do_prove now calls Prover::reset() on the child/grandchild handlers
right after enable_deferral (+ file-based agg_vk), mirroring the zkvm
integration tester: the parent captures agg_vk / cached commit /
def_hook_commit by value, so the child SDK is no longer needed and its
GPU proving keys are released before the parent's STARK/SNARK phase.

Verified live on the mainnet shadow fork (mid-run upgrade test):
bundle 18394 was proven by a mixed-type process (same process had
proved chunks minutes earlier) and finalized via the new wrapper —
halo2_outer create_proof live peak 8.6 GiB vs 22.8 GiB pre-fix.

…ssons

- 03-deploy-verifier.sh: cast 1.7 prints 'true'/'false' for receipt
  status — the genuine-register path misread a successful updateVerifier
  as reverted and aborted 20-upgrade.sh mid-cutover. Also document the
  manual recovery (finish steps i/j/k, do NOT re-run the script).
- Trap 38: Symptom 3/Cause 3 (mixed task types re-create VRAM
  starvation post-Fix-1), per-GPU supported_proof_types isolation
  workaround, Fix 3 (child SDK reset after deferral, verified on
  bundle 18394), upstream pool-free-bytes budgeting suggestion, and
  crash-recovery corollaries (reset active_attempts, wipe the local
  SDK db before restart, cleanup ordering).
- Trap 36: note the cast 1.7 output drift.

Extends (does not replace) the 20-upgrade.sh hard-switch mode: instead of
proving with the old stack locally, production bundle proofs are imported
from the remote DB into a quarantine table (remote_bundle_proof) and applied
to shadow bundles below a boundary; at the boundary the new wrapper is
registered via the genuine updateVerifier path, so old (imported proofs,
legacyVerifiers routing) and new (local proofs, latestVerifier) bundles
genuinely finalize in parallel. 31-canary-rollback.sh restores the old
wrapper at the boundary and applies quarantined proofs >= N.

- sync-mainnet-db.py: optional --import-proofs / SYNC_PROOFS=1 proof
  importer; proved-only cursor advance + 200-row lookback (late-attaching
  mainnet proofs), PROOF_IMPORT_MAX_END_BATCH boundary from canary.env
- 10-follow-up.sh: --import-proofs flag (no local coordinator/prover);
  --reset also drops canary state
- 30-canary-upgrade.sh: boundary N = max(first unproven unfinalized,
  max proven unfinalized + 1), register new wrapper, mark finalized
  history proved, start new coordinator + provers
- 31-canary-rollback.sh: stop new stack, swap wrapper back, apply proofs
- stop_component in 30/31 always falls through to the pattern sweep
  (setsid zombie-pidfile trap)
- Makefile targets follow-canary / canary-upgrade / canary-rollback;
  GUIDE.md canary chapter; TROUBLESHOOTING Trap 39/40; AGENTS.md notes
- AGENTS.md: remote RDS query discipline (partial deleted_at indexes,
  no unbounded count(*) on l2_block)
- Test report: docs/testing_reports/canary-parallel-upgrade-2026-08-25.md
  (all acceptance criteria passed, incl. interleaved finalize, rollback
  drill, and t2 new-guest proofs finalized via the new wrapper)

Mainnet indexes on l2_block/chunk/l1_message are partial
(WHERE deleted_at IS NULL), so sync queries without the predicate could
not use them: the per-minute MAX(index) watermark on chunk scanned an
entire 6.6M-row index, and the per-cycle l2_block range copy ran two
parallel seq scans of the 62 GB heap whenever new chunks arrived —
TB-scale RDS reads per day from an idle follow stack.

Add a NOT_DELETED constant and apply it to every src-side query
(get_watermarks, copy_range, baseline copy_table_pipe clauses, proof
import). EXPLAIN-verified on the remote: watermark MAX is now a 1-page
index-only backward scan, range copies are plain index scans.

Also document Trap 41 (root cause of the Trap 40 Cause 3 slowness) and
the "stop the stack when the test ends" cost discipline.