input_modalities in GET /v1/models.
Sending audio or video to any other model fails fast with
400 invalid_request_error before anything is billed. Output is still text
only.
Today that means the Gemini family (google/gemini-*): every catalog
Gemini model accepts image, audio, and video input.
Audio
Chat Completions
Use OpenAI-styleinput_audio parts: base64 data plus a format token
(wav or mp3; aiff, aac, ogg and flac also pass through):
Responses API
The sameinput_audio shape, paired with input_text:
Video
Chat Completions
Usevideo_url parts (the de-facto extension Qwen and Zhipu clients already
send). The url is a data: URI or an http(s) URL. YouTube URLs pass
straight through to Gemini; any other video URL is fetched by the gateway and
sent inline, so it must fit Gemini’s ~20MB inline request limit:
data: URI carries the video inline: data:video/mp4;base64,....
Gemini-native endpoint
/v1beta/models/{model}:generateContent accepts Google’s own shapes:
inlineData for base64 audio/video (routed by mimeType) and fileData for
video URLs:
Billing
Audio and video are tokenized by the provider (Gemini: ~32 tokens per second of audio, ~300 per second of video) and billed as input tokens. Some models price audio input above the text rate —pricing.audio_input_per_mtok_usd in
GET /v1/models shows the rate where one applies (e.g.
google/gemini-2.5-flash bills audio at 0.30 text). Video frames
bill at the model’s input rate, but a video’s audio track is tokenized
separately and bills at the audio rate where the model declares one.
