Experimental ChatGPT voice transport

The SDK can experimentally open the signed-in user's private ChatGPT voice transport over WebRTC. Audio flows directly between the browser and the voice peer after a one-time SDP exchange through your backend. Access, refresh, and ChatGPT web-session credentials never enter the WebRTC/browser response.

OpenAI currently describes ChatGPT Voice in Desktop as a duplex system: GPT-Live manages the live conversation while a separate model coordinates app tasks. Voice and delegated tasks have separate usage accounting. The private route names below are observed implementation details, not a documented contract for that product. See OpenAI's current ChatGPT Voice in Desktop documentation for the authoritative availability and usage details.

Native tools through app-server mediation

The desktop-style app-server path preserves native WebRTC audio and interruption while routing native handoff_request items to arbitrary server-owned function schemas. It does not monitor transcripts and does not introduce a second model.

This path requires a Node/Bun backend and the codex executable on PATH. Configure tools on createChatGPTHandler(); never accept tool schemas from the browser:

import {
  createChatGPTHandler,
  type RealtimeDynamicTool,
} from "@opencoredev/loginwithchatgpt-server";

const tools: RealtimeDynamicTool[] = [{
  type: "function",
  name: "record_lookup",
  description: "Read a record owned by the authenticated application user.",
  inputSchema: {
    type: "object",
    properties: { recordId: { type: "string" } },
    required: ["recordId"],
    additionalProperties: false,
  },
}, {
  type: "function",
  name: "record_prepare_change",
  description: "Prepare a reviewable record change without applying it.",
  inputSchema: {
    type: "object",
    properties: {
      recordId: { type: "string" },
      value: { type: "string" },
    },
    required: ["recordId", "value"],
    additionalProperties: false,
  },
}];

const auth = createChatGPTHandler({
  secret: process.env.LWC_SECRET,
  realtime: {
    appServer: {
      tools,
      allowedModels: (model) => allowedExecutionModels.has(model),
      async executeTool({ request, callId, name, arguments: args }) {
        const appUser = await requireAppUser(request);
        const input = validateToolArguments(name, args);

        if (name === "record_lookup") {
          return {
            output: await readOwnedRecord(appUser.id, input.recordId),
          };
        }

        const change = await createPendingChange(appUser.id, callId, input);
        return {
          output: { status: "pending_confirmation", changeId: change.id },
          pendingConfirmation: {
            review: {
              changeId: change.id,
              recordId: change.recordId,
              previousValue: change.previousValue,
              nextValue: change.nextValue,
            },
          },
        };
      },
      async confirmTool({ request, callId, confirmation, pending }) {
        const appUser = await requireAppUser(request);
        const approved = validateConfirmation(confirmation);
        const result = approved
          ? await atomicallyApplyOwnedChange(appUser.id, callId, pending)
          : await keepPendingChange(appUser.id, callId);
        return {
          output: result,
          speech: approved ? "The change was applied." : "I left the change unapplied.",
        };
      },
    },
  },
});

The SDK adds the header-only authenticated request snapshot, loginSessionId, and opaque liveSessionId to each callback. Revalidate every argument, enforce application-user ownership, and make consequential operations idempotent by callId.

Connect from a browser user gesture:

import {
  connectChatGPTRealtimeAppServer,
} from "@opencoredev/loginwithchatgpt-core";

let pending: { callId: string; review: unknown } | undefined;

const live = await connectChatGPTRealtimeAppServer({
  session: {
    voice: "vale",
    model: selectedModelSlug,
  },
  onStateChange: renderVoiceState,
  onBridgeEvent(event) {
    if (event.type === "tool.pending_confirmation") {
      pending = { callId: event.callId, review: event.review };
      renderConfirmation(event.review);
    }
  },
});

// Call only from an explicit application UI action.
if (!pending) throw new Error("No action is waiting for confirmation.");
await live.resolveConfirmation(pending.callId, { approved: true });

// Releases browser media and closes the server process.
live.close();

connectChatGPTRealtimeAppServer() creates the peer, exchanges SDP through POST /realtime/app-server, consumes the authenticated NDJSON event stream, posts confirmations, and deletes the server session during cleanup. The handler permits one active app-server call per login session and expires abandoned calls after 30 minutes by default. It also buffers a bounded set of early lifecycle events until the browser subscribes.

Propagate the user's selected execution model in session.model. Delegated turns are entitlement-sensitive; silently falling back to a different default model can produce misleading model-specific usage errors.

For a consequential action, the bridge returns the structured pending result to the execution turn immediately. Native voice therefore remains responsive while the application waits. Speech such as “send it” may ask the tool to prepare a review, but it must never replace the authenticated UI confirmation.

Bridge events are session.started, session.closed, handoff, tool.running, tool.completed, tool.pending_confirmation, tool.failed, error, and keepalive. Never put bearer tokens, credentials, or unsafe raw tool results in the browser-facing review payload.

Start a direct voice-only call

GPT Live /realtime/wm does not accept the Codex device-login token used by the rest of this SDK. Configure the handler with a server-only auth callback backed by an encrypted ChatGPT web session. The callback should also verify that the ChatGPT email matches your application's authenticated user:

import {
  exchangeChatGPTRealtimeWebSession,
  resolveConfig,
} from "@opencoredev/loginwithchatgpt-core";
import { createChatGPTHandler } from "@opencoredev/loginwithchatgpt-server";

const config = resolveConfig();
const auth = createChatGPTHandler({
  secret: process.env.LWC_SECRET,
  realtime: {
    async getAuth({ request }) {
      const appUser = await requireAppUser(request);
      const encrypted = await loadUserRealtimeSession(appUser.id);
      const stored = await decrypt(encrypted);
      const live = await exchangeChatGPTRealtimeWebSession({
        config,
        sessionToken: stored.sessionToken,
        deviceId: stored.deviceId,
        sessionCookies: stored.sessionCookies,
        onSessionUpdate: async (next) => {
          await saveUserRealtimeSession(appUser.id, await encrypt({ ...stored, ...next }));
        },
      });
      if (live.email.toLowerCase() !== appUser.email.toLowerCase()) {
        throw new Error("ChatGPT Live identity does not match the app user");
      }
      return live;
    },
  },
});

Then connect from a user gesture so the browser can grant microphone access and play audio:

bun add @opencoredev/loginwithchatgpt-core

import {
  CHATGPT_REALTIME_VOICES,
  connectChatGPTRealtime,
} from "@opencoredev/loginwithchatgpt-core";

const call = await connectChatGPTRealtime({
  session: {
    voice: "juniper",
    transport: "wm",
    language: "en-US",
  },
  onStateChange: (state) => console.log(state),
  onTranscript: ({ kind, text }) => {
    // Render the user's recognized speech and the assistant's live captions.
    appendTranscript(kind, text);
  },
});

console.log(CHATGPT_REALTIME_VOICES);

// Explicit controls are useful for buttons and push-to-talk.
call.stopSpeaking();
call.setInputMuted(true);
call.setOutputMuted(false);

// Always release the microphone, peer, data channel, and audio graph.
call.close();

Preferred language and voice

Set session.language to the user's preferred spoken language. The SDK checks that it is a valid BCP 47 language tag, canonicalizes casing such as EN-us to en-US, and sends it in both the Live session and request headers. This improves speech recognition; it does not lock the conversation to one language. The user can still ask the assistant to speak another language during the call.

The current documented voices are exported as CHATGPT_REALTIME_VOICES: arbor, breeze, cove, ember, juniper, maple, sol, spruce, and vale. The voice option remains a string so a newly released upstream voice can work before this catalog is updated. To change voice, close the current call and reconnect. Pass the same conversationId if the upstream account and transport allow the new call to continue that chat.

language configures the direct /wm handshake. The app-server bridge does not currently expose a verified speech-recognition language field; users can still request another spoken language conversationally.

connectChatGPTRealtime() does the following:

1. captures audio with echo cancellation, noise suppression, and automatic gain;
2. creates a max-bundle WebRTC peer and negotiated data channel id 0;
3. posts the local SDP and session options to /api/chatgpt/realtime;
4. applies the SDP answer, plays the remote audio track, and decodes events;
5. monitors local voice activity while the model speaks and sends stop_speaking for natural barge-in.

Pass bargeIn: false to disable local interruption detection, or tune it:

bargeIn: { threshold: 0.045, holdMs: 120, cooldownMs: 800 }

You may supply mediaStream, mediaConstraints, peerConnection, or audioElement when your application owns those objects. A supplied stream is not stopped by close(); a stream created by the SDK is.

Connection API

The resolved ChatGPTRealtimeConnection exposes:

connectChatGPTRealtimeAppServer() returns the same connection plus:

Use onBridgeEvent for application-tool status. onEvent remains the native WebRTC/data-channel stream; these two event sources are intentionally separate. Use onTranscript for the parsed user_transcription_text and live_captioning_text events. It reports { kind, text, event }, keeping the raw event available when an application needs transport-specific fields.

For push-to-talk, disable automatic barge-in, call startListening() on press, and stopListening() on release.

This SDK does not expose video or screen-sharing media. The browser helper rejects video constraints and streams with video tracks. Its peer connection still creates an inert send-only video transceiver because the observed private handshake expects that SDP m-line; this is transport compatibility, not a video feature or a claim about every ChatGPT Voice surface.

ChatGPT Voice feature coverage

ChatGPT Voice is a product with account-, plan-, device-, and region-specific features. This package is an experimental transport SDK, so some items from OpenAI's ChatGPT Voice guide are provided by the SDK, some come from ChatGPT upstream, and some must be built by the host application.

Availability can change upstream. Treat a successful signaling response and runtime events as the source of truth for a particular signed-in account.

Session schema

The browser sends ergonomic camel-case ChatGPTRealtimeSessionOptions. The core package converts them into the upstream wire keys shown below.

The fixed /wm wire payload includes backend_reasoning_effort: "high", chat_mode: "chat", and an empty client_tools array, matching the current ChatGPT clients. The SDK deliberately does not expose arbitrary private metadata, prompt overrides, delegation controls, or model-tool fields as supported features.

voice_session_id and voice_status_request_id are generated per call. Use buildChatGPTRealtimeSession() when you need the exact wire object without opening a connection.

Direct HTTP endpoint

POST /api/chatgpt/realtime is cookie-authenticated and protected by the same origin checks as /responses.

{
  "sdp": "v=0\r\n…",
  "session": {
    "voice": "juniper",
    "transport": "wm",
    "language": "en-US"
  }
}

A successful response is 201 application/sdp containing the WebRTC answer. Errors are JSON:

Configure the route with:

createChatGPTHandler({
  enableRealtime: true,
  realtime: {
    maxRequestBytes: 256 * 1024,
    allowedTransports: ["wm"],
    allowedModes: ["wingman"],
    sessionDefaults: { transport: "wm", historyAndTrainingDisabled: false },
    getAuth: async ({ request }) => getUserBoundWebAuth(request),
  },
});

The handler converts the request to multipart sdp + session fields and posts to the selected ChatGPT endpoint:

wm is the default because it is the currently observed GPT Live transport. The server permits only wm unless realtime.allowedTransports explicitly opts into a compatibility path. vp and vps are undocumented, may disappear, and do not make Advanced or Standard product features available through this SDK. Codex OAuth tokens do not authorize wm; getAuth must return a short-lived token minted for the ChatGPT web client. exchangeChatGPTRealtimeWebSession() performs that server-only exchange and validates the token client id.

The app-server route family is separate:

These routes use the normal authenticated Codex session, not realtime.getAuth. Cross-session ids return 404, non-GET requests keep the same origin protections, logout closes owned calls, and no access or refresh token enters the response.

Data-channel schema

Messages arrive in a nested { "type": "data_message", "data": "<json>" } envelope. The parser accepts strings, ArrayBuffer, typed arrays, byte arrays, and numeric-key byte objects. Unknown event types pass through unchanged. Event-specific fields may be on the event or its payload; getChatGPTRealtimePayload(event) normalizes both forms.

Client events must use that same outer envelope. /wm silently ignores a raw inner action or relay event. encodeChatGPTRealtimeEvent() and every connection method add the required framing.

Observed event names exported in CHATGPT_REALTIME_EVENT_TYPES:

This catalog is intentionally narrow. The parser passes unknown events through for forward compatibility, but their presence is not an SDK capability claim.

The client-tool parsing and result helpers remain available for compatibility with ChatGPT's reserved first-party tools, but the SDK does not advertise arbitrary schemas in /wm.

Control messages use this shape:

{
  "type": "action_request",
  "payload": { "action": "stop_speaking" }
}

Known actions are start_listening, stop_listening, stop_speaking, resume_listening, and relay_message. A conversational text relay is not an action_request; createChatGPTRealtimeRelayMessage() builds the currently observed private message shape. It is not a function call or tool result. Use createChatGPTRealtimeAction(), encodeChatGPTRealtimeEvent(), and parseChatGPTRealtimeEvent() for a custom transport.

Low-level server use

Custom backends can call createChatGPTRealtimeCall() directly with a resolved config, an auth callback, SDP, and session options. This is a server-only primitive because getAuth() returns bearer material. Some runtime HTTP stacks may be challenged by ChatGPT's web edge; inject a compatible server-side fetch through resolveConfig({ fetch }) when necessary. Never solve that by moving the ChatGPT access token into the browser.

The web-session credential is longer-lived and more sensitive than the minted access token. Encrypt it at rest, bind it to the application's user identity, never log it, and never put it in local storage or a client-readable cookie. ChatGPT's web edge also requires a stable oai-did device identity and can rotate its NextAuth session into numbered cookie chunks. Persist the deviceId and sessionCookies delivered to onSessionUpdate; serialize concurrent exchanges for one user so two requests cannot race that rotation. The helper distinguishes a true 401 expiry from a retryable 403 edge rejection.

Custom Node/Bun backends can also instantiate ChatGPTRealtimeAppServerSession from @opencoredev/loginwithchatgpt-server/realtime-app-server directly. They must provide server-held tokens, token refresh, dynamic schemas, execution and confirmation handlers, user-scoped session storage, an authenticated event stream, a confirmation route, TTL cleanup, and model propagation. Most applications should use the built-in handler routes, which provide that lifecycle.

The app-server bridge requires a long-lived Node/Bun instance because its child process and Live-session registry are process-local. Use sticky routing for all routes belonging to one Live session; edge and request-isolated serverless runtimes cannot host this bridge. The normal login and response routes remain Web-standard when realtime.appServer is unset. The child starts in a private temporary directory, receives only required system environment variables, and runs with a read-only Codex sandbox instead of inheriting the hosting server's application secrets.

Treat every dynamic-tool invocation as untrusted input. Revalidate it against the configured server-side schema and allowlist, bind execution to the authenticated application user, make calls idempotent by call id, and return only JSON-serializable results. Keep preparation separate from consequential mutation, and require an explicit application UI confirmation immediately before the mutation.