gemini-live-api

SKILL.md

LiveAPI Service Skill

This skill provides instructions for generating a LiveAPI client service
class that connects to the Gemini Enterprise Live API over WebSockets. The
generated client handles bidirectional streaming, bearer-token authentication
via Application Default Credentials (ADC), transparent session resumption, and
ClientMessage / ServerMessage proto exchange.

The skill also produces a demo frontend + backend service so the user can
interactively validate the generated client (text, audio, video, transcription,
and interrupt handling).

Prerequisites

Before running the generation flow, ensure the following are available on the
host:

A Google Cloud project with the Vertex AI / Gemini Enterprise Agent Platform
APIs enabled.

Application Default Credentials configured on the host running the generated
client:

gcloud auth application-default login

A destination output folder supplied by the user (e.g. /tmp/liveapi_out)
where the generated code, environment, and demo will be written. Never
mutate the host's system Python environment.

The user's chosen implementation language (Python is the default and
reference language for this skill).

Reference Files

Provided files in references/ (do not treat these as standalone skills —
they are loaded on demand):

client_server_messages.md: Public reference for the ClientMessage /
ServerMessage schemas used by the Live API.

client_server_messages.proto: The proto definition generated from
client_server_messages.md.

session_manager.md: Describes how to correctly handle sessions, buffering,
and resumption on disconnection.

Steps

Step 1: Copy the reference files

Copy client_server_messages.md, client_server_messages.proto, and
session_manager.md from this skill's references/ folder into the user's
destination output folder. These files become the source of truth for the
generated client.

Step 2: Reconcile with the public documentation

Examine the public documents linked from client_server_messages.md. If there
are any discrepancies between the public documents and the copied
client_server_messages.md / client_server_messages.proto, update the copies
in the destination folder so the generated client compiles and runs against the
current server contract.

Step 3: Implement the client class

Implement a class in the user's chosen language that:

Imports the local client_server_messages.proto types (ClientMessage,
ServerMessage).

Opens a WebSocket connection to the Live API endpoint.

Exposes async methods so the user can send and receive data to/from the
model.

For languages that require an isolated runtime (e.g. Python), create an isolated
environment (e.g. venv) inside the destination folder and generate a bash
script (e.g. setup.sh) that recreates the environment and installs
dependencies. Never install into the system interpreter or the user's global
site-packages, and never instruct the user to run sudo pip install.

Initialization parameters

The user provides the following at construction time:

project_id

location

model_id

config: a ClientMessage with the setup field populated.

Authentication

Obtain a bearer token via Application Default Credentials, attach it to the
WebSocket connect request as Authorization: Bearer <token>, refresh the token
before or upon expiry, and reuse the refreshed token on every reconnection
(including go_away and unexpected disconnects). Do not hard-code a
long-lived API key as the only auth mechanism.

Public async API

The class MUST expose the following async methods, gated on receipt of a
setup_complete ServerMessage before sending:

send_realtime_data(data): send realtime input. data is a ClientMessage
carrying a realtime_input field.

send_client_content(data): send non-realtime, turn-based content that
contributes to history. data is a ClientMessage carrying a
client_content field.

receive(): yield ServerMessage instances parsed from the WebSocket
stream.

Do not expose synchronous blocking variants as the primary API surface.

Step 4: Write a test file

Once the client is implemented, generate a test file that initializes the
connection and exercises sending text, audio, and video data and receiving
the responses. Ask the user for any information required to run the test
(project, model, media samples).

Step 5: Generate how_to_run.md

Provide a how_to_run.md in the destination folder that documents the generated
class. Include full examples showing how to build ClientMessage payloads for
every supported modality, how to send them, and how to receive data from the
model.

Step 6: Generate a demo frontend + backend service

Create scripts that deploy the implementation as a service with both a frontend
UI and a backend service (any language). The service MUST reuse the
ClientMessage / ServerMessage protos from Step 1 for wire traffic. Through
the UI the user should be able to:

Start a new connection / close the current connection.

Select the model to use.

Select input sources (audio and/or video from camera or screenshot) and
stream them to the model.

Send a text message to the model.

Hear model audio and see the interleaved model and user transcription /
conversation history.

While implementing audio and transcription playback, follow the guidance in
Live API best practices.

Handling the interrupt signal

When a ServerMessage's server_content arrives with interrupted: true, the
UI MUST:

Ensure played audio and its corresponding transcription remain time-aligned.

Immediately stop the currently playing model audio and stop appending to the
in-progress transcription bubble.

Clear the unplayed audio buffer and any pending unrendered transcription so
stale content does not bleed into the next turn.

Start new chat bubbles for the next user and model turns.

Handling the transcription finished signal

For streamed input_transcription / output_transcription chunks, append to
the currently active bubble while finished is unset, and close that bubble and
start a fresh one when finished is observed. Route input_transcription text
to user-role bubbles and output_transcription text to model-role bubbles.

Step 7: Generate how_to_test_with_ui.md

Write how_to_test_with_ui.md describing how to launch and use the demo
service. It MUST include:

The exact shell command(s) or script invocation(s) to start the backend
service.

The exact shell command(s) or script invocation(s) to start the frontend UI.

The host and port (e.g. http://localhost:PORT) the user should open in
their browser.

How to start a session, select a model, choose input sources (mic, camera,
screen), send a text message, and observe model audio and transcription in
the UI.

Validation Checklist

Before considering the generation complete, verify each item:

 client_server_messages.md, client_server_messages.proto, and
session_manager.md were copied into the destination folder.

 The generated client imports the local proto-generated ClientMessage
and ServerMessage types.

 The client connects to the Live API WebSocket at
wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent
(or the wss://aiplatform.googleapis.com/... global variant), and formats
the setup model field as
projects/{project_id}/locations/{location}/publishers/google/models/{model_id}.

 Authentication uses ADC-provided bearer tokens sent as Authorization: Bearer <token>, is refreshed before expiry, and reattached on every
reconnect.

 Public async methods send_realtime_data, send_client_content, and
receive are present, correctly typed, and gated on setup_complete.

 Transparent session resumption is enabled
(session_resumption.transparent = true), the latest new_handle is
tracked, sent-message indexing starts at 1, the buffer is pruned via
last_consumed_client_message_index, and buffered messages are replayed on
reconnect (including on go_away and WebSocket close codes 1000 / 1006).

 If using python, an isolated environment (e.g. venv) plus a setup.sh
and requirements.txt (or equivalent) exist inside the destination folder;
no changes were made to system or user-global Python.

 how_to_run.md and how_to_test_with_ui.md are present, and the demo
UI reuses the same ClientMessage / ServerMessage protos.

 Interrupt handling and transcription finished handling behave as
described above.

 The client does not target generativelanguage.googleapis.com and
does not authenticate via API key in a query string.

34