FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

[https://nvbugs/6681216][fix] Raise disagg nixl readiness timeout on Blackwell by brnguyen2 · Pull Request #18372 · NVIDIA/TensorRT-LLM · GitHub

/ TensorRT-LLM Public

[https://nvbugs/6681216][fix] Raise disagg nixl readiness timeout on Blackwell - #18372

Open
brnguyen2 wants to merge 1 commit into
NVIDIA:mainfrom
brnguyen2:user/brnguyen/fix-6681216
ConversationCommits (1)ChecksFiles changed
Open

[https://nvbugs/6681216][fix] Raise disagg nixl readiness timeout on Blackwell#18372
brnguyen2 wants to merge 1 commit into
NVIDIA:mainfrom
brnguyen2:user/brnguyen/fix-6681216

Conversation

brnguyen2 commented Aug 28, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Collaborator

Description

test_disaggregated_deepseek_v3_lite_fp8_nixl[DeepSeek-V3-Lite-fp8] times out during server startup on DGX_B200 / DGX_B300: both disaggregated workers never reach registered before the 300s readiness poll raises TimeoutError after 300s. The failure is a cold-start warmup budget overrun, not a hang. On Blackwell the CuTe DSL MLA decode JIT plus autotuner warmup (including a large B=2048 decode variant) push worker readiness to ~370s, past the 300s default. H100 registers in ~138s and is unaffected.

This raises server_start_timeout to 1200s for this test, matching the budget several other disaggregated tests already use, and removes the B200/B300 waivers so it runs again. A genuine hang would still fail at 1200s, so this does not mask real deadlocks. The default (300s) is unchanged for all other callers.

Test Coverage

disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_fp8_nixl[DeepSeek-V3-Lite-fp8] on DGX_B200 and DGX_B300 (now un-waived). Verified locally on B200: with the raised budget the workers register at ~365s and the test passes.

PR Checklist

  • PR description clearly explains what and why.
  • Follows TRT-LLM coding guidelines.
  • Test-only change (readiness budget + waiver removal); no product code paths affected.
  • No API changes.

Dev Engineer Review

  • Increased server_start_timeout for test_disaggregated_deepseek_v3_lite_fp8_nixl[DeepSeek-V3-Lite-fp8] from 300 to 1200 seconds.
  • Removed the DGX B200 and DGX B300 waiver entries for this test.
  • The timeout change is scoped to this test. Other callers retain the 300-second default.
  • The values and waiver-file changes are consistent with the reported 365–370 second worker-registration delay.
  • No product code, public API, or unrelated waiver entries changed.

QA Engineer Review

  • Modified test_disaggregated_deepseek_v3_lite_fp8_nixl.
  • Removed its DGX B200 and DGX B300 entries from tests/integration/test_lists/waives.txt.
  • The test now runs on DGX B200 and DGX B300. Local DGX B200 verification passed, with workers registering after approximately 365 seconds.
  • Verdict: sufficient.

[https://nvbugs/6681216][fix] Raise disagg nixl readiness timeout on …
45b2a2e 
…Blackwell

test_disaggregated_deepseek_v3_lite_fp8_nixl times out during server
startup on B200/B300: workers never reach registered before the 300s
readiness poll fires. Root cause is cold-start warmup, not a hang. On
Blackwell the CuTe DSL MLA decode JIT plus autotuner warmup push
worker readiness to ~370s, past the 300s default; H100 registers in
~138s and is unaffected.

Pass server_start_timeout=1200 for this test, matching the budget
other disaggregated tests already use, and remove the B200/B300
waivers so it runs again. A genuine hang still fails at 1200s.

Verified on B200: with the raised budget the test passes (workers
register ~365s; 1 passed).

Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
brnguyen2 requested a review from a team as a code owner August 28, 2026 14:50

brnguyen2 commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-,DGX_B300-4_GPUs-PyTorch-"

coderabbitai Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info ⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 102e261b-8cfa-4220-92a1-a38c04a38c4f

📥 Commits

Reviewing files that changed from the base of the PR and between 5767bed and 45b2a2e.

📒 Files selected for processing (2)
  • tests/integration/defs/disaggregated/test_disaggregated.py
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

The DeepSeek V3 Lite FP8 NIXL integration test now uses a 1200-second startup timeout. Its DGX_B200 and DGX_B300 waiver entries are removed.

Changes

DeepSeek V3 Lite FP8 NIXL

Layer / File(s) Summary
Test timeout and waiver updates
tests/integration/defs/disaggregated/test_disaggregated.py, tests/integration/test_lists/waives.txt		 The test invocation sets server_start_timeout=1200. The DGX_B200 and DGX_B300 skip records are removed.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 45b2a

This change only increases the startup wait budget for a Blackwell-specific integration test and removes its waivers; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title follows the required ticket and type format and clearly describes the main change: increasing the disaggregated NIXL readiness timeout on Blackwell.
Description check ✅ Passed The description explains the failure, root cause, solution, affected platforms, test coverage, and API impact. The included checklist is mostly complete and relevant to this test-only change.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

tensorrt-cicd commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #69985 [ run ] triggered by Bot. Commit: 45b2a2e Link to invocation

tensorrt-cicd commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #69985 [ run ] completed with state FAILURE. Commit: 45b2a2e
/LLM/main/L0_MergeRequest_PR pipeline #57265 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

brnguyen2 commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-,DGX_B300-4_GPUs-PyTorch-"

tensorrt-cicd commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #70007 [ run ] triggered by Bot. Commit: 45b2a2e Link to invocation

tensorrt-cicd commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #70007 [ run ] completed with state SUCCESS. Commit: 45b2a2e
/LLM/main/L0_MergeRequest_PR pipeline #57287 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

chienchunhung Awaiting requested review from chienchunhung chienchunhung is a code owner automatically assigned from NVIDIA/trt-llm-disagg-devs
nv-xtf Awaiting requested review from nv-xtf nv-xtf is a code owner automatically assigned from NVIDIA/trt-llm-disagg-devs

At least 1 approving review is required to merge this pull request.

Assignees

brnguyen2

Labels

ci: full pre-merge approved

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

Back | FazBrowse Home | New Git URL