TrueFit.ai - Moving from Gemini-Locked Architecture to a Provider-Agnostic Real-Time Interview System

TrueFit.ai started as a Gemini Live–first system. The initial implementation was tightly coupled to the Gemini Live API because the entire product was originally built around a Gemini Live hackathon constraint. That made the early architecture very direct: the interview runtime, session handling, and real-time communication layer were all bound specifically to Gemini’s interface.

As we started pushing the system further, that coupling became a real limitation.

We ran into recurring issues with the Gemini Live setup, especially around session stability and interrupt handling. One of the major problems was that after triggering activityEnd, the session would often stop responding and eventually fail with a 1011 keep-alive ping timeout after ~50–60 seconds. This made real-time interviews unreliable and created a poor UX for anything production-like.

We escalated and discussed these issues on the Google AI developer forums (Gemini Live / AI Studio community), including back-and-forth with other engineers facing similar behavior. A key observation from that period was that most implementations in the ecosystem leaned heavily on WebSockets, while our system was built around a WebRTC-based real-time pipeline with explicit interrupt management handled at the application layer rather than delegated entirely to the model.

Link to related discussion on Gemini Live session instability and interrupt handling: discuss.ai.google.dev. That difference mattered. We weren’t just streaming audio - we were actively controlling interruption state, turn-taking, and session continuity ourselves.

At the same time, we evaluated the overall architecture again. The original approach for voice systems in 2024 (STT -> LLM -> TTS) was not viable for us due to latency constraints. Real-time conversational flow needs to stay under ~800ms end-to-end to feel natural, and the extra hops introduced too much delay. That pushed us fully into direct voice model usage instead of compositional pipelines.

When Gemini became unstable for our use case, we started evaluating alternatives. OpenAI Realtime stood out both in terms of latency characteristics and consistency in production voice behavior. Cost was also a factor since the system is still being built in a lean, experimental phase.

That decision led to a deeper architectural shift: instead of swapping providers ad hoc, we introduced a proper abstraction layer.

The system now uses a provider-agnostic LLM runtime built around the clean adapter pattern we already had:

• -gemini_live.py handles Gemini-specific session transport and lifecycle
• -openai_realtime.py implements OpenAI Realtime as a parallel backend
• -fallback_adapter.py handles failover and continuity when a provider degrades
• -factory.py centralizes provider selection and runtime injection

At the core of the system is now a unified LLM contract. The interview agent no longer “uses Gemini” or “uses OpenAI” directly - it interacts with a single abstraction that can resolve to any supported provider at runtime.

This change also plugged cleanly into the existing FastAPI dependency injection setup. The live interview agent no longer receives a concrete Gemini implementation; instead, it receives a generic LLM adapter, with Gemini as the default and OpenAI as a fallback. That made swapping providers almost trivial without touching core agent logic.

The result is a system that is no longer brittle to a single model provider. It can recover mid-session, switch adapters when needed, and maintain continuity while logging and monitoring behavior across providers. Over time, this also gives us a way to understand real performance trade-offs between models in live interview scenarios - latency, stability, response quality, and interruption handling.

More importantly, it sets up a foundation for something we’re actively moving toward: not just fallback-based switching, but adaptive provider selection, where the agent can eventually choose which model to rely on depending on context (speed vs reasoning quality vs stability).

This refactor wasn’t just about fixing Gemini instability - it was about turning model providers into interchangeable execution layers in a real-time system that has to stay alive under pressure.