Purpose: Backend local network web API for use by VR headsets running a Unity application. Context: This server application will be used in an art installation where users all talk to an AI orb called the "centroid". An AI emotion analysis model (Tessa BiLSTM) is run on everything users say, providing the LLM with important context for discussing with the group. As the installation progresses, the AI will dynamically evolve. The web server is the source of truth — it decides when the AI responds based on cumulative emotional intensity.

See Unity Integration for detailed project architecture.

Architecture:

• Headset clients communicate from their Unity APK to the Unity game server for visualization and effects.
• Headsets directly communicate with this web server for transcription events. The headset receives its transcription response back, and handles sending it to the Unity game server.
• The Unity game server receives updates from this web server (via WebSocket) on the emotional state of the AI (cumulative from user inputs), and AI replies with TTS audio (which it plays on the central speaker and sends text to clients for display).
• This web server decides when the AI replies, not the Unity game server. Auto-response is triggered based on the cumulative emotional ratings from all user inputs exceeding an adjustable threshold.

Components:

• src/labs/services/transcription.py — Audio transcription via Google Gemini multimodal API
• src/labs/services/emotion.py — Tessa BiLSTM emotion classification (anger, fear, joy, love, sadness, surprise)
• src/labs/services/emotion_state.py — Cumulative emotion state tracker and auto-response trigger logic
• src/labs/services/gemini.py — LLM conversation history and response generation via Gemini
• src/labs/services/tts.py — ElevenLabs Text-to-Speech for the Centroid's voice
• src/labs/services/subscriptions.py — WebSocket subscription manager for push events to Unity
• src/labs/routers/centroid/ — API endpoints (REST + WebSocket)

App Flow:

1. User speech is picked up by the headset microphone
2. Headset Unity app directly calls POST /centroid/transcribe on this backend (same local network)
3. Backend transcribes the audio and runs Tessa emotion analysis
4. Transcript and emotion scores are returned directly to the headset
5. Headset forwards the transcript and emotion to the Unity game server for HUD visualization
6. Backend records the transcript (with emotion context) to the LLM conversation history
7. Backend updates the session's cumulative emotion state
8. Updated emotion state is pushed to WebSocket subscribers (Unity game server) for visual effects
9. If cumulative emotion intensity crosses the configured threshold, the backend auto-triggers: a. Gemini LLM generates a response from all accumulated speech b. ElevenLabs TTS synthesises the response into spoken audio (MP3) c. Text response + base64-encoded audio are pushed to WebSocket subscribers d. Unity game server plays the audio on the central speaker and broadcasts text to headsets