feat(core): support video, audio, and PDF in openai-compatible converter - #34012
feat(core): support video, audio, and PDF in openai-compatible converter#34012Arekejoker wants to merge 1 commit into
Conversation
Sorry, something went wrong.
Extend the OpenAI-compatible message converter to handle multimodal content parts beyond images: - video/* → video_url - audio/* → input_audio (wav/mp3) - application/pdf → file Also preserve reasoning_content for non-Copilot OpenAI-compatible providers so reasoning models keep their chain-of-thought across turns. Wire the bundled @ai-sdk/openai-compatible provider through the OpenCode copilot-provider fork so these content types are supported for all OpenAI-compatible providers, not just GitHub Copilot. Add tests for video_url, input_audio, file content parts, and the reasoning_content field on assistant messages.
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
Sorry, something went wrong.
|
The following comment was made by an LLM, it may be inaccurate: Based on my search, I found one potentially related PR: Potentially Related PRPR #23501: fix: OpenAI-compatible provider improvements (system messages, image support, stream interruption) This PR addresses improvements to the OpenAI-compatible provider, including image support, which is related to the current PR's expansion of content type support. However, the current PR (34012) appears to be the primary effort for extending multimodal support to video, audio, and PDF formats. Why potentially related: Both PRs enhance the OpenAI-compatible converter's content handling capabilities, though PR #23501 focuses on image support and system messages while #34012 adds broader multimodal support. No direct duplicates found. The current PR appears to be a distinct, comprehensive effort to extend multimodal content support across multiple file types. |
Sorry, something went wrong.
|
This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window. Feel free to open a new pull request that follows our guidelines. |
Sorry, something went wrong.
Problem
The OpenAI-compatible message converter only handles image/* content parts. When a multimodal model (like MiMo, Gemini, or any provider supporting video_url) receives a video, audio, or PDF attachment, the converter throws UnsupportedFunctionalityError.
Changes
Content type support (convert-to-openai-compatible-chat-messages.ts)
Unsupported audio formats and URL-based PDFs throw UnsupportedFunctionalityError with a clear message.
Reasoning preservation
For non-Copilot OpenAI-compatible providers (no reasoning_opaque), reasoning content is now surfaced via reasoning_content on assistant messages. This preserves chain-of-thought across turns for reasoning models.
Provider wiring (provider.ts)
Wire @ai-sdk/openai-compatible through the OpenCode copilot-provider fork, which contains the multimodal converter. This ensures all OpenAI-compatible providers benefit from the new content type support.
API types (openai-compatible-api-types.ts)
Add OpenAICompatibleContentPartVideo, OpenAICompatibleContentPartAudio, and OpenAICompatibleContentPartFile types. Add reasoning_content field to OpenAICompatibleAssistantMessage.
Files changed
Test plan
All 29 turborepo typecheck packages pass.