FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

[fix][test] Fix flaky ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover by lhotari · Pull Request #26806 · apache/pulsar · GitHub

Repository navigation

[fix][test] Fix flaky ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover - #26806

Open
lhotari wants to merge 2 commits into
apache:masterfrom
lhotari:lh-fix-flaky-rebucket-failover
Open

lhotari wants to merge 2 commits into
apache:masterfrom
lhotari:lh-fix-flaky-rebucket-failover

Conversation

lhotari commented Oct 1, 2026

Copy link
Copy Markdown
Member

Fixes #26552

Motivation

ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover fails intermittently in CI (Broker Group 2), for example here:

java.lang.AssertionError: a hot-segment split must fire — the rollover is not a split expected [2] but found [1]
    at ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover(ScalableTopicControllerAutoScaleTest.java:507)

registerConsumer("c9") fires an asynchronous consumer-change auto-scale evaluation. When it is still in flight, evaluateAndAct coalesces the test's evaluateAutoScaleForTest() call into it. The call sets the re-evaluate flag and returns an already-completed future without evaluating. That has two consequences:

  1. The "seeded rebucket cooldown must block an immediate second rollover" assertion runs before any evaluation has actually happened, so it can pass without checking anything.
  2. The queued coalesced re-run then holds the in-flight slot during the next manual evaluation, after the hot-segment load is reported. That evaluation is coalesced as well, and the split runs later on the scheduler with trigger=coalesced, after the test has already asserted activeSegmentCount() == 2.

I reproduced this deterministically by delaying the namespace-policy read in the test's mocks by 200 ms. With the delay, the test fails every time with the CI assertion message.

Modifications

  • Add a package-private @VisibleForTesting accessor, ScalableTopicController.isAutoScaleEvaluationInFlight(). It returns true while an evaluation is running or a coalesced re-run is pending.
  • In the test, wait for the event-driven evaluation that registerConsumer("c9") fires to finish before calling evaluateAutoScaleForTest(). Both manual evaluations then really run, and both assertions check what they were written to check. The assertions themselves are unchanged.

No production behavior changes.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover. Verified locally:

  • With the 200 ms reproduction delay in place, the original test failed every time and the fixed test passed 10 of 10 runs (invocationCount = 10).
  • Without the delay, the whole test class passed (27 runs, including the 10 invocations of this test).

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

…etCooldownSurvivesLeaderFailover

registerConsumer fires an asynchronous consumer-change auto-scale evaluation.
A manual evaluateAutoScaleForTest() issued while it is in flight is coalesced
into it and returns immediately; the coalesced re-run can then swallow the
next manual evaluation too, so the hot-segment split had not happened yet when
the test asserted. Wait for the event-driven evaluation to finish before
evaluating manually.

Fixes apache#26552

Assisted-by: Claude Code (claude-opus-5-5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Test] Flaky ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover

2 participants


Back | FazBrowse Home | New Git URL