Setting the file. One moment.
Subchapter 5.1
references/browser-audio.mdMarkdown5 KBView on GitHub
The OpenAI Realtime SDK wires microphone and speaker automatically only for its WebRTC transport. ElevenLabs Realtime is WebSocket-only, so the app captures and plays audio itself. Both directions use PCM16, 24 kHz, mono.
This reference uses , the helper OpenAI uses in its own WebSocket examples. Any capture path that produces 24 kHz PCM16 chunks works, for example an the app already has.
wavtoolsArrayBufferAudioWorkletnpm install @openai/agents-realtime wavtoolsFramework-agnostic; in React, hold the returned handle in a useRef and call stop() on unmount.
import {
OpenAIRealtimeWebSocket,
RealtimeAgent,
RealtimeSession,
} from "@openai/agents-realtime";
import { WavRecorder, WavStreamPlayer } from "wavtools";
const SAMPLE_RATE = 24000;
export async function startVoiceSession() {
const tokenResponse = await fetch("/api/token");
if (!tokenResponse.ok)
throw new Error(`Token request failed (${tokenResponse.status})`);
const { value: clientSecret } = await tokenResponse.json();
const recorder = new WavRecorder({ sampleRate: SAMPLE_RATE });
const player = new WavStreamPlayer({ sampleRate: SAMPLE_RATE });
await recorder.begin();
await player.connect();
const session = new RealtimeSession(
new RealtimeAgent({
name: "Assistant",
instructions:
"You are a helpful, friendly voice assistant. Keep replies concise.",
}),
{
transport: new OpenAIRealtimeWebSocket({
url: "wss://api.elevenlabs.io/v1/convai/realtime",
}),
model: "eleven_realtime_v1_mini",
config: {
outputModalities: ["audio"],
audio: { output: { voice: "brian" } },
},
},
);
session.on("audio", (event) => {
player.add16BitPCM(event.data, event.responseId);
});
session.on("audio_interrupted", () => {
player.interrupt();
});
session.on("error", (event) => {
console.error(event.error);
});
try {
await session.connect({ apiKey: clientSecret });
await recorder.record((data) => {
session.sendAudio(data.mono as unknown as ArrayBuffer);
});
} catch (err) {
await stop();
throw err;
}
async function stop() {
await recorder.quit().catch(() => {});
player.interrupt();
session.close();
}
return { session, stop };
}Microphone audio streams continuously; server-side VAD decides when the user’s turn ends, so there’s no push-to-talk logic to port unless the app had one.
data.mono is mistyped. wavtools types the record() callback’s mono as Int16Array, but at runtime it’s already the PCM16 ArrayBuffer. Pass it straight to sendAudio. Reading .buffer off it, or wrapping it in a new typed array, sends garbage.recorder.begin() prompts for the microphone and player.connect() creates an AudioContext. Browsers block audio contexts created outside a click or keypress, so call startVoiceSession() from the handler that used to start the WebRTC session.audio_interrupted fires when the user barges in. player.interrupt() discards audio already queued; without it, the assistant keeps talking over the user.audio.input.format.rate all use 24000. A mismatch plays back too fast, too slow, or as noise.| WebRTC-era code | WebSocket replacement |
|---|---|
new OpenAIRealtimeWebRTC({ audioElement, mediaStream }) | new OpenAIRealtimeWebSocket({ url }) plus the recorder and player above |
<audio autoplay> element fed by the peer connection | WavStreamPlayer; remove the element |
getUserMedia stream passed as mediaStream | WavRecorder.begin() (pass a deviceId to pick an input) |
Peer-connection event handlers (changePeerConnection, ICE state) | Remove; watch session.on("error") and transport events instead |