Skip to main content
The gateway accepts audio and video input for models that declare the modality — check input_modalities in GET /v1/models. Sending audio or video to any other model fails fast with 400 invalid_request_error before anything is billed. Output is still text only. Today that means the Gemini family (google/gemini-*): every catalog Gemini model accepts image, audio, and video input.

Audio

Chat Completions

Use OpenAI-style input_audio parts: base64 data plus a format token (wav or mp3; aiff, aac, ogg and flac also pass through):

Responses API

The same input_audio shape, paired with input_text:

Video

Chat Completions

Use video_url parts (the de-facto extension Qwen and Zhipu clients already send). The url is a data: URI or an http(s) URL. YouTube URLs pass straight through to Gemini; any other video URL is fetched by the gateway and sent inline, so it must fit Gemini’s ~20MB inline request limit:
A data: URI carries the video inline: data:video/mp4;base64,....

Gemini-native endpoint

/v1beta/models/{model}:generateContent accepts Google’s own shapes: inlineData for base64 audio/video (routed by mimeType) and fileData for video URLs:

Billing

Audio and video are tokenized by the provider (Gemini: ~32 tokens per second of audio, ~300 per second of video) and billed as input tokens. Some models price audio input above the text rate — pricing.audio_input_per_mtok_usd in GET /v1/models shows the rate where one applies (e.g. google/gemini-2.5-flash bills audio at 1.00/1Mvs1.00/1M vs 0.30 text). Video frames bill at the model’s input rate, but a video’s audio track is tokenized separately and bills at the audio rate where the model declares one.