What I was looking at
After adding tons of logging lines to _AgentAudioTrack.recv(), I discovered something. On every new turn, recv() was being called hundreds of times in rapid succession - no sleeping, no pacing - until it had drained the entire queue.
The consequence was mechanical. The audio bridge consumed the greeting at roughly 48x normal speed, burned through the buffer, and hit silence. My _on_turn_complete logic was watching has_buffered_audio - when the buffer emptied, it treated the turn as done, called clear_outbound_queue(), and reset state. From the bridge's perspective, the turn had completed cleanly. So it triggered the next turn's opening sequence. The model had said the greeting exactly once. The pipeline had played it, declared the turn over, and started again.
The model wasn't repeating itself. My audio bridge was stuck in a loop it had no idea it was in.
Two bugs, both in _AgentAudioTrack, conspired to cause this.
Bug 1: Resetting the pacing clock
The original clear_buf() reset both _pts and _start to zero and None respectively, with the intention that each turn would get a clean clock. That intention was correct in spirit but broken in practice.
The pacing logic in recv() works like this:
self._pts += self._samples_per_frame
deadline = self._start + (self._pts / self._sample_rate)
wait = deadline - time.time()
if wait > 0:
await asyncio.sleep(wait)
_start is anchored to time.time() on the very first recv() call of the session. From that point, every deadline is computed as an offset from that anchor. The system works because _pts and real time stay in sync - _pts advances by exactly 960 samples (20ms) per frame, so _pts / 48000 is always the number of seconds elapsed since the stream started.
When I reset _start = None at the end of a turn, the next call to recv() re-anchored _start to the current wall-clock time, but _pts was reset to 0. So on every subsequent frame, deadline = new_start + (0 / 48000) was in the past. wait was always negative. asyncio.sleep was never called. recv() spun as fast as the event loop could run it, which on a lightly-loaded coroutine is very fast indeed.
The fix was to not reset _pts or _start in clear_buf() at all. The pacing clock has to be continuous across turns. Only _buf, the resampler, and _input_pts are reset between turns - the output timeline stays intact.
Bug 2: The greedy drain
Even if the clock hadn't been reset, the original queue-drain logic would have caused problems. The old recv() looped greedily:
while len(self._buf) < target:
try:
pcm_24k = self._queue.get_nowait()
# resample and extend _buf
except asyncio.QueueEmpty:
break
If the queue contained, say, 40 chunks of audio (a full agent response buffered while the previous turn was finishing), this loop pulled all 40 into _buf in a single recv() call. _buf would grow to several seconds of audio. On the next 200 calls to recv(), _buf was already full, the queue was empty, and recv() just sliced from the buffer - no sleeping, because the pacing deadline had been passed while the greedy fill was happening.
The new implementation pulls at most one chunk per call:
try:
pcm_24k = await asyncio.wait_for(
self._queue.get(), timeout=0.005
)
# resample and extend _buf
except asyncio.TimeoutError:
pass # silence frame
One chunk in per 20ms call means _buf stays shallow (at most one resampled chunk ahead) and the resampler sees uniform 20ms inputs. asyncio.wait_for with a 5ms timeout means recv() blocks briefly waiting for audio, then returns silence if nothing comes, keeping the stream alive between turns without throwing getting caught in Gemini's VAD.
Bug 3: The resampler's frame size (the one I found while fixing the others)
While working on the queue drain, I noticed av.AudioResampler was producing variable-length output chunks even for fixed-size inputs. libswresample (which backs PyAV's resampler, I was seeing these two for the first time) has an internal ring buffer, and without a fixed output frame size it accumulates samples and releases them in bursts. I added frame_size=960 to the constructor:
self._resampler = av.AudioResampler(
format="s16",
layout="mono",
rate=_WEBRTC_SAMPLE_RATE,
frame_size=self._samples_per_frame, # 960 = 20ms at 48kHz
)
This forces exactly 960 output samples per resample() call, making the resampler behave as a proper stream converter, same fixed chunk in, same fixed chunk out, every time.
I also added _input_pts, an input presentation timestamp that increments by the number of input samples on each call. Without a monotonically advancing pts, some versions of libswresample re-initialise their internal compensation state on each call. With it, the resampler maintains a continuous timeline and its delay compensation is stable.
What this taught me
The most useful thing I took from this wasn't the specific fixes - it was learning why the greedy approach failed here when it's fine in most other contexts.
A greedy drain is correct when you want to process all available work as fast as possible and backpressure is handled upstream. That's most queue consumers. But recv() isn't a queue consumer, it's a clock-driven producer. It must return exactly one frame every 20ms, and the queue is just its source of input. Consuming the entire queue in one call breaks the contract between the clock and the data. The fix wasn't to drain less aggressively; it was to recognise that the call frequency is the rate limiter, not the buffer level.
The other thing: audio is just arrays of integers, but the meaning of those integers depends entirely on context - sample rate, bit depth, channel count, frame size. Every conversion between formats (24kHz Gemini output -> 48kHz WebRTC input, and the reverse path for the mic) is an opportunity to corrupt the timeline if the resampler's internal state isn't managed carefully. frame_size in av.AudioResampler is a parameter I would have ignored before this. Now I understand exactly what it's protecting against.