FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix: abort upstream requests on client disconnect, pool connections, handle assistant prefill by mingqi · Pull Request #278 · ericc-ch/copilot-api · GitHub

Repository navigation

fix: abort upstream requests on client disconnect, pool connections, handle assistant prefill - #278

Open
mingqi wants to merge 1 commit into
ericc-ch:masterfrom
mingqi:master
Open

mingqi wants to merge 1 commit into
ericc-ch:masterfrom
mingqi:master

Conversation

mingqi commented Oct 5, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

This PR fixes three problems in how the proxy handles requests to Copilot, and adds one small performance change.

1. Upstream requests are aborted when the client disconnects

Problem: If a client cancelled a streaming request (for example, pressed Esc in Claude Code), the proxy kept reading the Copilot stream until it finished. That wasted quota and kept connections open.

Fix:

  • createChatCompletions now accepts an optional AbortSignal and passes it to both fetch and events().
  • Both /v1/chat/completions and /v1/messages create an AbortController inside streamSSE. It is aborted from stream.onAbort() and in a finally block.
  • AbortError is no longer logged as an error. Other streaming errors are still logged.
  • Streaming and non-streaming paths are now separate branches. Each one checks that it got the response type it expected and throws if not.

2. HTTP connection pooling (Node only)

Problem: The default undici Agent was used without explicit limits.

Fix: Added initDirectConnectionPool(), which sets a global undici Agent with 16 connections per origin and pipelining: 1. It runs when --proxy-env is not set. The direct agent inside initProxyFromEnv uses the same settings. This is skipped under Bun, as before.

3. Assistant prefill / conversations that end on an assistant turn

Problem: Anthropic clients can send a conversation whose last message is from the assistant (a "prefill"). Copilot's OpenAI-style endpoint does not handle that, so the request either fails or returns an empty or unexpected completion.

Fix: When the translated conversation ends with an assistant message that has no tool calls, a { role: "user", content: "Continue." } message is added at the end.

4. Debug logging is cheaper

JSON.stringify of payloads, chunks, and responses now runs only when consola.level >= 4 (debug). Before this, every streamed chunk was serialized even when debug logging was off.

Testing

  • tests/anthropic-request.test.ts: added a test for assistant prefill, and updated the existing thinking-block test to expect the trailing Continue. message.
  • tests/create-chat-completions.test.ts: added a test that checks the abort signal is passed to the upstream fetch.
  • bun test, bun run lint

Notes

  • The prefill handling is a workaround. The model continues after a "Continue." prompt, not directly from the prefilled text, so the output may not join seamlessly onto the prefix.

🤖 Generated with Claude Code

…l continuation

- Abort upstream Copilot requests when the client disconnects during streaming
- Configure undici connection pool (16 connections per origin)
- Skip debug JSON serialization unless debug logging is enabled
- Append a 'Continue.' user message when the conversation ends with an assistant turn

Co-Authored-By: Claude <noreply@anthropic.com>

coderabbitai Bot commented Oct 5, 2026 •
edited
Loading

Copy link
Copy Markdown

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Walkthrough

Direct connections use a shared agent configuration when environment proxying is disabled. The chat-completions and messages handlers select streaming or non-streaming behavior from the request setting, pass abort signals for streamed requests, and gate debug logs by log level. Message translation appends a user message containing "Continue." for qualifying assistant-ending conversations.

Priority: ➖ Normal

Merge Risk: 🟡 Moderate · up to 1d2bc

Streaming requests that fail upstream, such as those rejected for a missing token or by Copilot, now return a successful streaming response that contains an error event instead of a proper HTTP error. Clients that rely on status codes may misreport or retry incorrectly. Move the upstream call before the stream starts before merging.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 1d2bc

Streaming cancellation improves normal request cleanup, but failed upstream requests bypass the existing error-body cleanup path. This creates an availability risk for the shared connection pool. Server-configured credentials, destinations, and explicit proxy selection remain preserved.

Retained concerns

  • Medium · security · inferred: Streaming acquisition failures bypass the previous error-body consumer and the new abort cleanup. Repeated upstream rejections with unread bodies could delay connection reclamation and impair other requests sharing the bounded direct pool. Normal streaming cleanup does not cover this acquisition-error terminal state.
Security review details

Security Blast Radius

  • inferred — The identified availability risk crosses request ownership within one server process and Copilot origin. Completion calls explicitly use the shared dispatcher; model retrieval uses the same state-derived origin and is expected to share the global transport under Node’s fetch contract. Deployment-wide or cross-tenant exposure is not established.

Security Findings and Attack Paths

  • inferred — A client able to reach a completion endpoint and pass configured gates can submit streaming requests that receive upstream non-OK responses. Those failures now leave no application-level body consumer or guaranteed controller abort. If unread bodies delay connection reclamation, repeated failures can consume shared pool capacity. Exhaustion timing and exploitability were not demonstrated; small or automatically completed error bodies are important limiting counterevidence.

Trust Boundaries and Controls

  • observed — The changed branches preserve rate-limit-before-approval ordering and perform upstream creation only after approval succeeds when enabled. Approval and rate limiting remain optional server configuration. Neither control establishes tenant authentication, and no new request-controlled destination, dispatcher, or bearer credential was introduced.

Resilience and Maintainability Implications

  • observed — Cancellation ownership begins inside the streaming callback after approval. Non-streaming requests still pass no signal, as before this PR. Approval-time interruption therefore remains a dependency-contract uncertainty rather than an established introduced regression; already-disconnected callback behavior could not be verified from available dependency source.

Hardening Proposals

  • proposed — Give upstream acquisition and stream processing one terminal-cleanup owner, explicitly dispose of failed response bodies, and preserve a protocol-visible failure outcome when an HTTP error can no longer be forwarded normally.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 8 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main changes: aborting upstream requests, pooling connections, and handling assistant prefill.
Description check ✅ Passed The description explains the changes and their purpose, including abort handling, connection pooling, assistant-prefill continuation, and debug logging.
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

mingqi changed the title feat: stream abort handling, connection pooling, and assistant-prefil… fix: abort upstream requests on client disconnect, pool connections, handle assistant prefill Oct 5, 2026

coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Actionable comments posted: 1


ℹ️ Review info ⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 0dc4141b-7209-46b7-83f7-6f91c112e282
📥 Commits

Reviewing files that changed from the base of the PR and between 0ea08fe and 1d2bc7d.

📒 Files selected for processing (8)
  • src/lib/proxy.ts
  • src/routes/chat-completions/handler.ts
  • src/routes/messages/handler.ts
  • src/routes/messages/non-stream-translation.ts
  • src/services/copilot/create-chat-completions.ts
  • src/start.ts
  • tests/anthropic-request.test.ts
  • tests/create-chat-completions.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📜 Review details 🔇 Additional comments (6)
src/lib/proxy.ts (1)

5-21: LGTM!

Also applies to: 27-27

src/start.ts (1)

10-10: LGTM!

Also applies to: 33-34

src/routes/messages/non-stream-translation.ts (1)

71-78: LGTM!

tests/anthropic-request.test.ts (1)

157-161: LGTM!

Also applies to: 163-182

src/services/copilot/create-chat-completions.ts (1)

3-3: LGTM!

Also applies to: 11-11, 33-33, 37-43, 51-51

tests/create-chat-completions.test.ts (1)

58-69: LGTM!

controller.abort()
})

const response = await createChatCompletions(payload, controller.signal)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Resolve upstream failures before returning an SSE response. Both routes now await the Copilot request inside the streamSSE callback. If the token is missing or Copilot rejects the request, Hono has already returned the SSE response. Hono then writes an error event instead of allowing an HTTP error response. (github.com)

  • src/routes/chat-completions/handler.ts#L62-L62: await the upstream connection before calling streamSSE; retain the controller for disconnect cancellation.
  • src/routes/messages/handler.ts#L58-L61: make the same change before starting the translated SSE response.
📍 Affects 2 files
  • src/routes/chat-completions/handler.ts#L62-L62 (this comment)
  • src/routes/messages/handler.ts#L58-L61

This branch has not been deployed

No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants


Back | FazBrowse Home | New Git URL