What

All four LLM features (AI titles/summaries/chapters, transcript translation, docs Ask AI, and the support messenger) now go through one provider-agnostic layer built on the Vercel AI SDK (apps/web/lib/ai/), replacing per-feature hand-rolled Groq/OpenAI/Anthropic clients. groq-sdk is removed.

Self-hosters can now pick any provider via env:

• AI_PROVIDER — assemblyai | openai | anthropic | groq | openai-compatible
• AI_MODEL / AI_CHAT_MODEL / AI_STREAM_MODEL — per-role model overrides
• AI_BASE_URL / AI_API_KEY — any OpenAI-compatible endpoint (Ollama, OpenRouter, LM Studio…)

assemblyai routes through the AssemblyAI LLM Gateway using the existing ASSEMBLY_API_KEY, so one key can cover transcription and AI features. It is never auto-detected — transcription-only instances don't start incurring LLM charges unless they opt in.

Behavior at merge: unchanged

With AI_PROVIDER unset (the default everywhere), providers are auto-detected per role to reproduce the current behavior exactly: generation prefers Groq → OpenAI → Anthropic (same gpt-oss-120b / gpt-4o-mini), chat features prefer Anthropic → OpenAI → Groq (same claude-sonnet-5). Later providers in each chain are automatic fallbacks — previously only summaries had any fallback at all.

Design notes

• generateText + the existing JSON extraction (not generateObject): the AssemblyAI gateway rejects response_format on gpt-oss models. The two map-reduce parse sites upgraded from cleanJsonResponse to the more tolerant extractJsonObject, and the gateway path opts into server-side json-repair.
• Capability flags per selection (supportsStreaming, supportsTemperature, token budgets) encode live-verified gateway rules: no streaming for non-OpenAI-family models, no temperature for Claude models, max_tokens must always be explicit.
• Messenger keeps its exact one-support-email-per-turn guarantee via three layers (stepCountIs(2), per-provider parallel-tool-call disable, an in-execute latch), and never retries another provider after an email was sent.
• Docs Ask AI keeps its byte-identical text/plain streaming contract; an empty completion (e.g. a thinking model consuming the whole token budget) now falls through to the next provider instead of streaming an empty answer.
• Model instances are always created inside workflow steps — never across "use step" boundaries.

Verification

• Typecheck clean; full unit suite green (only the pre-existing slack-manifest failure remains); new unit suites for provider resolution and the fallback chain.
• New env-gated live e2e (__tests__/e2e/ai-provider-live-e2e.test.ts, same pattern as live-transcribe-local-e2e): real providers, real local MySQL/MinIO/media-server, no AI mocks — fresh recording → real AssemblyAI transcription → real title/summary; short + 24k-char map-reduce generation; docs-ask incremental streaming; live messenger reply. 6/6 under auto-detect and 6/6 under AI_PROVIDER=assemblyai, including a live rate-limit event absorbed by the fallback chain.

Docs

self-hosting.mdx and docker-compose.coolify.env.example document all providers, the per-role auto-detect order, and the EU-gateway caveat.

Greptile Summary

The PR centralizes AI generation, translation, docs Ask AI, and messenger behavior behind a provider-agnostic Vercel AI SDK layer with AssemblyAI gateway and OpenAI-compatible endpoint support.

• Adds role-aware provider selection, capability flags, model overrides, and fallback chains.
• Moves output validation into provider attempts so unusable completions can fall through.
• Documents the new self-hosting configuration and adds focused unit and live E2E coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Reviews (4): Last reviewed commit: "fix(web): sanitize map-reduce keyPoints ..." | Re-trigger Greptile

Context used (3)

• Context used - AGENTS.md (source)
• Knowledge Base — Web App (apps/web)
• Knowledge Base — Infra, Storage and Config