Building TrueFit: Real-Time AI Interviews, WebRTC, and a Bug That Keeps Saying Hello

TL;DR: This is a long, technical deep dive into the architecture of TrueFit. If you are an engineer who has wrestled with aiortc, PyAV, or the Gemini Live API (native audio preview), I am specifically looking for your help with a maddening audio loop bug at the bottom of this post. If you're not into low-level audio plumbing, consider this your "too long; didn't read" warning!

I've been building TrueFit for a while now, and I think it's finally in a state where it's worth writing about — not because it's done, but because the engineering has gotten genuinely interesting. I'm writing this partly to document the architectural decisions I made, partly to think out loud about a maddening bug I'm currently fighting, and mostly because I'd love to hear from other engineers who've wrestled with similar problems.

So here's how the thing works, what I was thinking when I built it the way I did, and where I'm currently stuck.

What TrueFit Is

The short version: TrueFit is an AI-powered interview platform that lets companies screen every single applicant — not just the ones who made it past a resume filter — using a real voice conversation with an AI interviewer.

The longer version: hiring is broken in a specific, quiet way. Teams spend hours in first-round screens filtering for basic qualification signals that could be collected more efficiently. Meanwhile, candidates with real skills but weak CVs get filtered out before a human ever talks to them. TrueFit attacks this by making the interview the first step, not the last.

A candidate applies, gets a link, joins a session, and has a real spoken conversation with an AI that understands the company/team's values, knows the job, knows what skills matter, and knows how to evaluate answers. When it's over, the hiring team gets a structured evaluation — not a recording they have to sit through, but a distilled picture of how the candidate actually performs.

The infrastructure that makes this work is the interesting part.

The Stack at a Glance

-Backend: Python / FastAPI
-Real-time audio: WebRTC (aiortc server-side) + Gemini Live API
-Transport layer: WebSocket for signaling and control, WebRTC for all media
-AI: gemini-2.5-flash-native-audio-preview — native audio I/O, no TTS middleware
-Database: PostgreSQL via SQLAlchemy
-Events: Redis (queue + cache for real-time signals)
-Architecture: Domain-driven, with clear port/adapter boundaries between core logic and infrastructure

The architecture is deliberately over-engineered for a solo project. I'll explain why.

Why WebRTC Instead of Just Streaming Audio Over WebSocket

This is the first question I get, so let me answer it directly.

The naive version of this system would be: open a WebSocket, stream PCM audio from the browser mic to the server, forward it to Gemini, pipe Gemini's audio response back, play it. Done.

The problem is that WebSocket isn't designed for real-time audio delivery. It's TCP, so it has head-of-line blocking — if a packet is dropped, everything behind it waits. That's fine for JSON messages. It's terrible for audio, where a 200ms stall is audible and jarring. WebSocket also gives you zero control over jitter buffering, packet pacing, or echo cancellation.

WebRTC exists specifically to solve these problems. It uses UDP with SRTP, handles its own jitter compensation, and gives the browser's native audio stack control over echo cancellation and noise suppression. For a voice interview, these are non-negotiable — you can't have the AI hearing its own voice echoed back and treating it as candidate speech.

So the system uses two channels:

-WebSocket — a pure control channel. Signaling (SDP offer/answer, ICE candidates), session lifecycle messages, transcript events, interrupt signals. No audio.
-WebRTC — all media. Microphone audio from browser to server, AI audio from server to browser, camera and shared screen video frames from user's browser.

The signaling flow looks like this:

# From interview_ws.py — the WebSocket endpoint is intentionally thin.
# It accepts the connection, builds a session object, and hands off.

@interview_ws_router.websocket("/ws/interview/{job_id}/{candidate_id}")
async def interview_websocket(
    websocket: WebSocket,
    job_id: uuid.UUID,
    candidate_id: uuid.UUID,
    orchestration: InterviewOrchestrationService = Depends(get_orchestration),
    ...
) -> None:
    await websocket.accept()
    connection = InterviewConnection(
        websocket=websocket,
        job_id=job_id,
        candidate_id=candidate_id,
        ...
    )
    await connection.run()

Once the frontend connects over WebSocket, it sends a WebRTC offer. The server processes it, generates an answer, and after the ICE handshake completes, audio starts flowing over the peer connection — completely separate from the WebSocket that set it up.

The Three-Loop Concurrency Model

Once a session is established, three asyncio tasks run concurrently for the entire interview:

-ws_task — reads control messages from WebSocket
-interrupt_task — polls Redis every 50ms for interrupt signals
-agent_task — the LiveInterviewAgent (Gemini send + receive loops)

Inside the agent, there are two more concurrent loops:

-_send_audio_loop — pulls PCM from browser mic, pushes to Gemini
-_receive_loop — pulls events from Gemini, dispatches to handlers

So at peak, there are five concurrent coroutines driving a single session. They're coordinated through a combination of asyncio.Events (for synchronisation) and asyncio.Queues (for data flow). No shared mutable state between them except for a small set of flags and events on the connection and agent objects.

# From live_interview_agent.py
# gather() is the heartbeat — both loops run until one of them exits.

await asyncio.gather(
    self._send_audio_loop(session),
    self._receive_loop(session),
)

The AudioBridge: Where WebRTC Meets Gemini

This is probably the most important piece of infrastructure in the system, and the one I spent the most time on.

The problem is a format mismatch:

-Browser microphone (via WebRTC): 48kHz Opus, decoded to s16
-Gemini input expected: 16kHz mono s16 PCM
-Gemini output produced: 24kHz mono s16 PCM
-Browser speaker expected: 48kHz s16 via WebRTC

The AudioBridge handles all four resampling directions and sits between the aiortc peer connection and the Gemini adapter. It exposes two interfaces:

-audio_input_stream() — an async generator the agent iterates to get mic audio chunks
-push_audio(pcm_bytes) — called by the agent for each audio chunk from Gemini

# From audio_bridge.py — the inbound pipeline
async def _pump_inbound(self, track: MediaStreamTrack) -> None:
    resampler = av.AudioResampler(format="s16", layout="mono", rate=_SAMPLE_RATE)
    while not self._closed:
        frame = await asyncio.wait_for(track.recv(), timeout=1.0)

        if not self._mic_open.is_set():
            continue  # discard — agent is still speaking

        resampled_frames = resampler.resample(frame)
        for rf in resampled_frames:
            pcm_bytes = bytes(rf.planes[0])
            chunk_size = int(_SAMPLE_RATE * _CHUNK_DURATION) * _SAMPLE_WIDTH
            for i in range(0, len(pcm_bytes), chunk_size):
                chunk = pcm_bytes[i : i + chunk_size]
                if len(chunk) == chunk_size:
                    if time.monotonic() < self._speaking_cooldown_until:
                        continue  # echo suppression
                    self.inbound_queue.put_nowait(chunk)

Two things worth calling out here:

-
The mic gate. The mic starts closed. It only opens after the agent's first turn completes — after Gemini has finished its opening greeting. This is critical: if the mic were open from the start, candidate audio would reach Gemini before it's in listening mode, which produces confusing results. The gate is an asyncio.Event (_mic_open) that _pump_inbound checks on every frame.
-
Echo suppression. After the agent stops speaking, there's a 600ms cooldown window during which inbound audio is still discarded. This absorbs the acoustic echo — the agent's voice coming back through the WebRTC echo path. Without this, Gemini hears its own voice as candidate speech and starts responding to itself.

Domain-Driven Architecture (and Why It's Worth the Overhead)

The core interview logic is completely isolated from infrastructure. The agent doesn't import FastAPI, aiortc, or the Gemini SDK. It depends on abstract ports — CachePort, QueuePort, LiveSessionPort — and those are wired to concrete adapters at the FastAPI layer.

# The agent only knows about abstract ports
class LiveInterviewAgent:
    def __init__(
        self,
        *,
        live_adapter: GeminiLiveAdapter,  # injected, not imported
        orchestration: InterviewOrchestrationService,
        queue: QueuePort,
        cache: CachePort,
        audio_input_stream: AsyncIterator[bytes],
        on_audio_output: Callable[[bytes], Coroutine],
        ...
    ) -> None:

This felt over-engineered when I set it up. It's paid off twice already: once when I swapped the underlying Gemini model (from gemini-2.0-flash-live-001 to the native audio preview model) without touching the agent, and once when I needed to add Redis-backed interrupt signaling without the agent needing to know Redis exists.

How Gemini Is Structured as an Interviewer

Gemini runs with four registered tools:

| Tool | Purpose | |---|---| | record_question | Called before Gemini asks a question — creates a DB record | | persist_answer | Called after candidate responds — saves the answer | | complete_interview | Called when Gemini decides the interview is done | | flag_interrupt | Called when Gemini detects the candidate interrupting |

The tool model means Gemini drives the interview state machine. When it calls record_question, that's the signal that a new question is about to be asked. When it calls complete_interview, that triggers the InterviewCompleteSignal exception that propagates cleanly through asyncio.gather() back to run():

# From live_interview_agent.py
async def _tool_complete_interview(self, args: dict) -> dict:
    reason = args.get("reason", "questions_exhausted")
    await self._queue.publish(DomainEvent(
        event_type="interview.agent_ending",
        ...
    ))
    self._session_complete.set()
    raise InterviewCompleteSignal(reason)  # propagates through gather() to run()

Using an exception rather than a flag + break keeps the code clean. InterviewCompleteSignal is explicitly re-raised in _handle_tool_call, which means it propagates through _receive_loop → gather() → run() where it's caught and treated as a normal exit, not an error.

The Bug: "Hello Olani. Hello Olani. Hello Olani."

Alright, here's the part I actually want to get some eyes on.

The symptom is this: when a session starts, Gemini greets the candidate and asks its first question. But what comes out of the speaker is the greeting — specifically the first few words of it — on an infinite loop. "Hello Olani. Hello Olani. Hello Olani." The transcript eventually appears in the UI showing a complete, coherent sentence, but the audio just loops the opening fragment.

The interview session technically "works" — Gemini is generating a full response, the transcript is correct, the tool calls fire correctly. But the audio output is stuck in a loop of the first chunk.

Here's what I've investigated so far:

-
Multiple turn_complete events from Gemini
My first thought was that turn_complete was firing multiple times, causing _on_turn_complete to run repeatedly, which resets the resampler mid-playback and causes the outbound track to keep restarting. I added logging and confirmed that turn_complete is firing once per turn. This doesn't appear to be the source.
-
Mic gate opening too early
If open_mic() fires before the last frames of the greeting have been clocked out to the browser, the candidate's own ambient audio (or silence) reaches Gemini, which Gemini then responds to, generating another greeting fragment. I tightened the drain logic significantly:

# From interview_ws.py — _on_turn_complete
# Wait for the queue AND the track's internal buffer to fully drain
while not bridge.outbound_queue.empty() or (track and track.has_buffered_audio):
    await asyncio.sleep(0.02)

# Give the WebRTC stack time to clock out the last frames
await asyncio.sleep(0.3)

# NOW clear — all legitimate audio has played
await bridge.clear_outbound_queue()
bridge.open_mic()

The drain loop checks both the queue and the track's internal buffer (has_buffered_audio) before proceeding. Despite this, the loop persists.

-Spurious interrupt signals
If the interrupt_monitor_loop is picking up stale Redis keys and firing _on_interrupt, that clears the audio queue and triggers Gemini to re-generate output — which could cause the greeting to restart. I added key cleanup and TTL verification, and I don't see spurious interrupt events in the logs. But this feels like it's still worth exploring more carefully.

Where I think the problem might actually be:

The _AgentAudioTrack.recv() paces itself using a pts-based clock. If clear_buf() is called mid-playback — resetting _pts to 0 and _start to None — the pacing clock restarts. On the next recv() call, the track re-anchors to time.time(), which means it thinks it's at the very start of a new stream. If there are still chunks in the resampler's internal buffer at that point, they get played again — and those are the opening frames of the greeting.

The sequence I suspect:

-Gemini sends greeting audio. First few chunks hit the resampler and get buffered in _AgentAudioTrack._buf.
-Something (turn_complete? An early interrupt check?) triggers clear_outbound_queue() prematurely.
-clear_buf() resets the pts clock but doesn't fully flush the resampler's internal delay buffers.
-On the next recv() call, the resampler's internal state produces the same opening frames again.
-Repeat.

The clear_buf() method does attempt to flush the resampler:

def clear_buf(self) -> None:
    self._buf.clear()
    try:
        flush_frames = self._resampler.resample(None)  # None = flush
    except Exception:
        pass
    # Reinstantiate to guarantee clean state
    self._resampler = av.AudioResampler(
        format="s16", layout="mono", rate=_WEBRTC_SAMPLE_RATE
    )
    self._pts = 0
    self._start = None

It reinstantiates the resampler entirely, which should guarantee a clean state. But if the greeting frames are already in outbound_queue (not yet consumed by recv()) and clear_outbound_queue is called before they drain, they'd be discarded — but then the mic gate opens, silence hits Gemini, and we're back to the beginning.

I genuinely don't know if this is a timing issue in _on_turn_complete, a resampler state leak, or something about how the Gemini native audio model handles the first turn differently from subsequent ones. The native audio model is very new and documentation on its exact event ordering is thin.

What I'd Love to Hear

-If you've worked with the Gemini Live API — especially the native audio preview model — I'd love to know if you've seen turn_complete event ordering behave differently on the first turn versus subsequent turns.
-If you've built audio pipelines on top of aiortc and PyAV, I'm curious whether you've run into resampler state issues specifically at session boundaries.
-And if you've designed WebRTC + LLM voice systems and have opinions on better ways to gate the mic / synchronise the audio pipeline across async tasks, I'd genuinely appreciate the input.

The codebase is a FastAPI + aiortc backend. I'm happy to share more context if you want to dig in.

TrueFit is still early. The core loop works, and the voice quality is solid when the audio pipeline isn't misbehaving. The bug will get fixed. The interesting engineering problems never really stop.