FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix(data): stop leaking ristretto goroutines from MemoryConnector by andreclaro · Pull Request #1066 · erpc/erpc · GitHub

/ erpc Public

fix(data): stop leaking ristretto goroutines from MemoryConnector - #1066

Draft
andreclaro wants to merge 3 commits into
erpc:mainfrom
andreclaro:fix/memory-connector-goroutine-leak
Draft

fix(data): stop leaking ristretto goroutines from MemoryConnector#1066
andreclaro wants to merge 3 commits into
erpc:mainfrom
andreclaro:fix/memory-connector-goroutine-leak

Conversation

andreclaro commented Aug 12, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • MemoryConnector.Close() calls ristretto.Cache.Close(), but nothing in erpc ever called Close() on a memory connector — SharedStateRegistry and EvmJsonRpcCache don't expose their own Close(), so the only lifecycle signal a connector got was its ctx being cancelled, and ristretto's own processItems/defaultPolicy background goroutines don't watch that ctx themselves. Every memory connector ever created leaked 2 goroutines for the life of the process.
  • Most visible inside erpc's own erpc package test suite: each test spins up its own short-lived ERPC/network stack (and its own memory connector(s) for shared-state and/or the EVM JSON-RPC cache). Running the full package in one binary accumulated tens of thousands of live ristretto goroutines by the end of the run — enough to blow through go test's timeout on a full run of ./erpc/... (confirmed via goroutine dump: thousands of ristretto.(*Cache).processItems / defaultPolicy.processItems goroutines, all in [select], none ever collected).
  • Fix: data/memory.go — tie the ristretto cache's lifetime to the connector's ctx inside NewMemoryConnector, so cache.Close() fires on context cancellation regardless of whether any caller ever reaches for MemoryConnector.Close() explicitly. Close() itself cancels a dedicated watcher context too, so callers that build a connector with a non-cancelable ctx (context.Background(), used by several call sites in this repo) and rely solely on calling Close() don't leak the watcher goroutine either.
  • Concurrency safety: closing Ristretto's cache concurrently with an in-flight Set/Get/Delete is unsafe on its own (Cache.Close() closes internal channels — a racing Set() can data-race or panic with "send on closed channel"). Every cache-touching method now takes a read lock and checks a closed flag before touching the cache; Close() takes the write lock, so it can't proceed until every in-flight op has released, and no new op can start once closed. This was not theoretical — it's exactly what CI's -race run caught in architecture/svm (TestSvmStatePoller_ConcurrentSuggestionsDuringPoll_StaySlotMonotonic cancels a poller's ctx while a background goroutine is still writing a shared-state counter through the same connector).

Test plan

  • TestMemoryConnector_ClosesRistrettoGoroutinesOnContextCancel / ...OnExplicitClose — create/close 20 short-lived connectors each way and assert goroutine count returns near baseline; fail on the leaking version, pass with the fix.
  • TestMemoryConnector_SetRacingContextCancelDoesNotPanic — reproduces the TestSvmStatePoller_ConcurrentSuggestionsDuringPoll_StaySlotMonotonic shape (background Set() loop racing a ctx cancel) under -race; reliably data-races/panics on the intermediate sync.Once-only version, clean on the final closeMu/closed version.
  • go test ./data/... -race — passes (only Docker-testcontainer-dependent tests fail locally, no Docker daemon here, unrelated).
  • go test ./architecture/svm/... -race -count=2 and go test ./erpc/... -race -run 'TestSvm_' — pass on top of latest main (which now includes fix(svm): pass the miss reason, and stop counting cold reads as errors #1067's SVM miss-reason fix).
  • Full go test ./erpc/...: on main before this fix, a full run of the package reliably timed out via goroutine-leak-driven slowdown; with the fix the suite progresses far further per unit time before hitting remaining, separate stdlib net/http keep-alive overhead from ~100 fixture-created HTTP servers in one binary (pre-existing test-scaffolding cost, not this leak — noted for visibility, not fixed here).

Rebased onto latest main (which now includes #1067) to drop an earlier, more superficial fix I'd made to the same SVM miss-reason bug — #1067 fixes it properly (including a second bug it uncovered) and supersedes that part of this PR entirely. This PR is now scoped purely to the Ristretto goroutine leak / concurrency fix.

chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 54be017de7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread data/memory.go
Comment on lines +108 to +111
go func() {
<-ctx.Done()
cache.Close()
}()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Stop the context watcher when Close is called

When NewMemoryConnector is given a non-cancelable context such as context.Background() or a caller later cleans up with MemoryConnector.Close() without canceling the original context, this new goroutine remains parked on ctx.Done() forever. Several direct memory connector users/tests construct it with context.Background() and rely on Close(), so this change replaces the two Ristretto goroutine leak with a new one-goroutine leak per connector; make the watcher share an internal close signal or have Close() cancel the lifecycle it starts.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Good catch — fixed in 384636c: Close() now cancels the watcher's own context (separate from the caller-supplied ctx), and both paths route through a shared sync.Once since ristretto.Cache.Close() panics if it ever runs twice concurrently. Added a regression test (TestMemoryConnector_ClosesRistrettoGoroutinesOnExplicitClose) covering the context.Background() + explicit Close() case.

andreclaro added a commit to andreclaro/erpc that referenced this pull request Aug 12, 2026
Codex review on erpc#1066 correctly flagged that the ctx-watcher goroutine
added to close ristretto's own background goroutines would itself leak
forever for callers that build a MemoryConnector with a non-cancelable ctx
(context.Background(), which several call sites in this repo use) and rely
on calling Close() directly instead of cancelling a parent context.

Give Close() its own cancel func for the watcher's context, and route both
the watcher and Close() through a shared sync.Once so cache.Close() (which
panics if it ever runs twice concurrently — it closes channels) only ever
executes once regardless of which path fires first.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
andreclaro marked this pull request as draft August 12, 2026 14:39
andreclaro and others added 3 commits August 12, 2026 16:54
MemoryConnector.Close() closes the underlying ristretto cache, but nothing
in erpc ever calls it — SharedStateRegistry and EvmJsonRpcCache have no
Close() of their own, so the only way a connector's ctx is ever observed is
via the parent context being cancelled, which ristretto's own
processItems/defaultPolicy goroutines don't watch. Every connector created
over a process's lifetime leaks 2 goroutines permanently.

This is most visible in erpc's own test suite, where each test builds its
own short-lived ERPC/network stack (and therefore its own memory
connectors): running the full erpc package accumulates tens of thousands of
live ristretto goroutines over the run, degrading scheduling/GC enough to
blow through the test timeout.

Tie the cache's lifetime to the connector's ctx directly in
NewMemoryConnector so cache.Close() runs on ctx cancellation regardless of
whether any caller ever reaches for MemoryConnector.Close() explicitly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codex review on erpc#1066 correctly flagged that the ctx-watcher goroutine
added to close ristretto's own background goroutines would itself leak
forever for callers that build a MemoryConnector with a non-cancelable ctx
(context.Background(), which several call sites in this repo use) and rely
on calling Close() directly instead of cancelling a parent context.

Give Close() its own cancel func for the watcher's context, and route both
the watcher and Close() through a shared sync.Once so cache.Close() (which
panics if it ever runs twice concurrently — it closes channels) only ever
executes once regardless of which path fires first.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closing Ristretto's cache concurrently with an in-flight Set/Get/Delete is
unsafe on its own — Cache.Close() closes internal channels, so a concurrent
Set() racing it can either data-race or panic with "send on closed channel".
The previous sync.Once wrapper only made the *closing* idempotent; it did
nothing to stop a live cache operation from running into an
already-(or-concurrently-)closed cache.

This is exactly what erpc's own -race CI run found: architecture/svm's
TestSvmStatePoller_ConcurrentSuggestionsDuringPoll_StaySlotMonotonic cancels
a poller's ctx while a background goroutine is still calling Set() via
counterInt64.updateRemoteState — previously silent (Close() was dead code,
so the leak just grew instead), now a real crash risk once ctx-driven
closing is wired up.

Replace the sync.Once with an RWMutex (closeMu) + closed bool: every
Set/Get/Delete takes a read lock and checks closed before touching the
cache; Close() takes the write lock, so it can't proceed until every
in-flight op has released its read lock, and no new op can start once
closed is set. Added TestMemoryConnector_SetRacingContextCancelDoesNotPanic,
which reproduces the crash under -race on the prior version.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
andreclaro force-pushed the fix/memory-connector-goroutine-leak branch from 13b7251 to e55d44e Compare August 12, 2026 16:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant


Back | FazBrowse Home | New Git URL