Repository navigation
fix: Do not log /metrics requests from the pre-fork metrics server thread (Gunicorn deadlock) - #6929
Merged
ntkathole merged 1 commit intoOct 7, 2026
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
Copy link
Copy Markdown
|
⚠️ Please install the Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## master #6929 +/- ##
=======================================
Coverage 49.06% 49.06%
=======================================
Files 433 433
Lines 54314 54316 +2
Branches 7912 7912
=======================================
+ Hits 26647 26649 +2
Misses 25788 25788
Partials 1879 1879
Continue to review full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Sorry, something went wrong.
jyejare
approved these changes
Oct 5, 2026
aborgatin
force-pushed
the
fix/metrics-server-quiet-handler
branch
from
76d2f7b to
8e0f1e4
Compare
October 6, 2026 07:52
ntkathole
approved these changes
Oct 7, 2026
…read
The metrics HTTP server runs as a thread in the Gunicorn master, which
forks the workers. Its default WSGIRequestHandler writes an access-log
line to stderr for every scrape. A fork while that thread holds the
stderr buffer lock leaves the new worker with the lock held forever: it
blocks on its first log line ("Booting worker") and never serves.
Use a request handler with a no-op log_message for both the IPv4 and
the dual-stack server built by _make_metrics_httpd, like
prometheus_client's own _SilentHandler.
Fixes feast-dev#6928
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alexandr Borgatin <a.borgatin@yandex.ru>
ntkathole
force-pushed
the
fix/metrics-server-quiet-handler
branch
from
8e0f1e4 to
9237fb0
Compare
October 7, 2026 04:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
The metrics HTTP server runs as a thread in the Gunicorn master, which forks the workers. Its default WSGIRequestHandler writes an access-log line to stderr for every scrape. A fork while that thread holds the stderr buffer lock leaves the new worker with the lock held forever: the worker blocks on its first log line ("Booting worker") and never serves a request. Details, stacks and a reproduction in #6928 (same class of problem as #6647).
This adds _QuietWSGIRequestHandler (no-op log_message, like prometheus_client's own _SilentHandler) and uses it for both the IPv4 and the dual-stack server built by _make_metrics_httpd. Scrapes are no longer written to stderr; nothing else changes.
Tested on Kubernetes with --max-requests 20 and a scrape storm: 5 hangs in 318 forks before, 0 hangs in 2,000 forks after. Locally (steps in #6928): a hang after 12–29 forks before, none in 336 forks after.
Which issue(s) this PR fixes:
Fixes #6928
Checks
Testing Strategy
🤖 Generated with Claude Code