Skip to main content

Translation management API

Control the Palabra real-time translation pipeline over WebSockets or a WebRTC data channel.

Prerequisites​

  • WebSockets — you only need an API Key. Optionally, you can create a streaming session yourself and use its ws_url and publisher token instead.
  • WebRTC — create a streaming session first. The response contains the webrtc_url (the URL of the Palabra WebRTC Speech-to-Speech Translation API) and the publisher JWT access token used to authorize your connection.

Choose a transport​

Option 1. WebRTC​

  1. Create a streaming session.
  2. Connect to webrtc_url with any LiveKit client using your publisher token.
  3. Once connected, send commands through the default (empty-topic) WebRTC data channel.

Option 2. WebSockets​

Connect to the WebSocket endpoint, passing your API Key as the token query parameter (or in the Authorization header). The server validates the key and creates a streaming session for the lifetime of the connection automatically:

const randomHash = crypto.randomUUID(); // any random URL-safe string
const endpoint = `wss://streaming.palabra.ai/streaming-api/${randomHash}/v1/speech-to-speech/stream?token=${apiKey}`;
const socket = new WebSocket(endpoint);

Replace the {random_hash} path segment with any random URL-safe string generated by your client for the connection — it is used to distribute connections across translation servers.

Alternatively, create a streaming session yourself and connect to the returned ws_url with the session's publisher token:

const endpoint = `${ws_url}?token=${publisher}`;
const socket = new WebSocket(endpoint);

Once the connection is open, you can start sending commands.

Message format​

Every API message — request and response — has the same envelope:

{
  "message_type": "<string>",
  "data": { /* payload */ }
}

If message_type is "error", the data field contains diagnostic information.

Typical workflow​

  1. Create a task — send set_task to start the translation.
  2. Update the task — send another set_task to change settings during an ongoing translation.
  3. Pause processing — send pause_task to pause the translation (billing stops). Resume with another set_task.
  4. Interrupt processing — send interrupt_task to instantly stop transcription, translation, and speech of the current phrase (e.g. the speaker got interrupted) without pausing the translation of subsequent phrases.
  5. Finish the task — send end_task. No new audio is accepted after it; the server finishes processing everything it already received, sends an end_of_stream message and closes your connection automatically; the session is invalidated within 1 minute.

Settings reference​

Rate limits​

ScopeLimit
WebSocket connections20 per minute per token
set_task, get_task1 per 2 seconds
pause_task, end_task1 per 3 seconds
tts_task60 per minute

Exceeding a command limit returns an error message; exceeding the connection limit closes the connection with code 1008.

Streaming audio configuration​

Option 1. WebRTC audio I/O​

Use the following input/output configuration in your set_task:

{
  "data": {
    "input_stream": {
      "content_type": "audio",
      "source": {
        "type": "webrtc"
      }
    },
    "output_stream": {
      "content_type": "audio",
      "target": {
        "type": "webrtc"
      }
    }
    // ...
  }
}
  1. Publish your microphone track to the LiveKit room.
  2. Subscribe to the translated tracks that Palabra publishes to the same room after you send set_task.

Option 2. WebSocket audio I/O​

Use this configuration instead:

{
  "data": {
    "input_stream": {
      "content_type": "audio",
      "source": {
        "type": "ws",
        "format": "pcm_s16le", // or opus, wav
        "sample_rate": 24000,  // 16000 – 48000
        "channels": 1          // 1 or 2
      }
    },
    "output_stream": {
      "content_type": "audio",
      "target": {
        "type": "ws",
        "format": "pcm_s16le" // or zlib_pcm_s16le
      }
    }
  }
}

The output is always 24000 Hz, mono (sample_rate and channels cannot be changed).

  1. Send base64-encoded audio chunks that exactly match the declared format.
  2. Receive base64-encoded TTS chunks in output_audio_data responses (see below).

API messages​

Request message schema​

Loading ....

Response message schema​

Loading ....

Requests (client → server)​

Message typeDescription
set_taskCreate or update the translation task
get_taskReturn the current task
pause_taskPause the current task; resume with set_task
interrupt_taskInstantly stop the speech of the current (ongoing) phrase
end_taskFinish the translation task
tts_taskGenerate TTS from text
input_audio_dataInput audio data chunk (WebSocket audio transport only)

set_task​

Create a new task or modify the current one:

  • Sent for the first time after creating a session — starts the translation.
  • Sent after pause_task — resumes the translation.
  • Sent during an ongoing translation — updates the settings in real time, no need to stop the translation.
{
  "message_type": "set_task",
  "data": {
    "input_stream":  { /* Depends on transport — see "Streaming audio configuration" above */ },
    "output_stream": { /* Depends on transport — see "Streaming audio configuration" above */ },
    "pipeline": {
      "transcription": {
        "source_language": "string",
        "detectable_languages": ["string"],
        "segment_confirmation_silence_threshold": "float",
        "only_confirm_by_silence": "bool",
        "sentence_splitter": {
          "enabled": "bool"
         },
        "verification": {
          "auto_transcription_correction": "bool",
          "transcription_correction_style": "string"
        }
      },
      // Translation and speech generation settings for one or more target languages
      "translations": [
        {
          "target_language": "string",
          "translate_partial_transcriptions": "bool",
          "speech_generation": {
            "voice_cloning": "bool",
            "voice_id": "string",
            "voice_timbre_detection": {
              "enabled": "bool",
              "high_timbre_voices": [],
              "low_timbre_voices": []
            }
          }
        }
        // You can add more targets
      ],
      "translation_queue_configs": {
        "global": {
          "desired_queue_level_ms": "int",
          "max_queue_level_ms": "int",
          "auto_tempo": "bool"
        },
        "es": {
          "desired_queue_level_ms": "int",
          "max_queue_level_ms": "int"
        }
      },
      // Select response types to receive
      "allowed_message_types": [
        "translated_transcription",
        "partial_transcription",
        "partial_translated_transcription",
        "validated_transcription"
      ]
    }
  }
}

See the Translation settings breakdown for details on each field and the Recommended settings for best-practice values.

ASR-only mode​

To use this API in ASR only mode, do either of the following:

  • Set output_stream to null — you will still get text translations, but no TTS audio.
  • Use an empty translations list — you will get neither text translations nor TTS audio.

get_task​

Return the current task (as a current_task response). Set exclude_hidden to false to include advanced (hidden) fields in the response.

{ "message_type": "get_task", "data": { "exclude_hidden": true } }

pause_task​

Pause the current task. No audio is processed and no billing applies while the task is paused. Use set_task to resume.

{ "message_type": "pause_task", "data": {} }

interrupt_task​

Instantly stop the speech (TTS) of the phrase currently being processed, without pausing upcoming ones. Useful when the current phrase no longer needs finishing — for example, if your conversation partner interrupts you. Unlike end_task, nothing is finalized: the current phrase is dropped.

{
  "message_type": "interrupt_task",
  "data": {
    "languages": ["global"], // 'global' (all languages, default) or target languages from `translations`
    "pause_task": false      // pause the task after interrupting
  }
}

end_task​

Finish the current task. After end_task no new audio is accepted; the server finishes processing everything it already received (the tail of the translation is fully delivered), sends an end_of_stream message (see below) and then closes the connection automatically.

{
  "message_type": "end_task",
  "data": {
    "force": false // drop the pipeline and disconnect immediately, without processing the tail
  }
}

tts_task​

Generates TTS from text within the active task. The behavior depends on translate_text (default false):

  • "translate_text": false — TTS only. language is the language to speak in and must be one of the target_language values configured in the task's translations section (otherwise a VALIDATION_ERROR is returned).
  • "translate_text": true — the text is first translated into every target_language of the task, then synthesized. language is the language of the input text and must be a valid source language.

text is limited to 2048 characters. See the Text-to-Translated-Speech guide for details.

{
  "message_type": "tts_task",
  "data": {
    "text": "Hello, how are you?",
    "language": "en",       // text language
    "translate_text": false // default
  }
}

input_audio_data​

Send a base64-encoded input audio chunk when WebSockets is selected as the audio transport. The audio chunks you push must match the format / sample_rate / channels declared in your set_task command. The optimal chunk length is 320 ms. The base64 payload must be between 1 KB and 512 KB (≈384 KB of raw audio per message).

{
  "message_type": "input_audio_data",
  "data": {
    "data": "base64 encoded data"
  }
}

Responses (server → client)​

Message typeDescription
partial_transcriptionUnconfirmed ASR segment
partial_translated_transcriptionUnconfirmed translation segment
validated_transcriptionFinal ASR segment
translated_transcriptionFinal translation
output_audio_dataChunk of generated TTS audio (WebSocket audio transport only)
current_taskResponse to the get_task command
end_of_streamEnd-of-stream confirmation (after end_task, right before close)
warningAudio stream health warning
errorValidation or runtime error
  • To receive partial_transcription, validated_transcription, and translated_transcription messages, include these message types in the allowed_message_types field of your set_task command.

partial_transcription​

Transcription of an uncompleted segment:

{
  "message_type": "partial_transcription",
  "data": {
    "transcription": {
      "transcription_id": "19f84f5e10a20085",
      "language": "en",
      "text": "The weather is nice",
      "segments": [
        {
          "text": "The weather is nice",
          "start": 7.36,
          "end": 8.32,
          "start_timestamp": 1784642133.39,
          "end_timestamp": 1784642134.35,
          "words": [
            {
              "word": "The",
              "start": 7.36,
              "end": 7.42
            },
            {
              "word": "weather",
              "start": 7.54,
              "end": 7.90
            },
            {
              "word": "is",
              "start": 7.96,
              "end": 8.04
            },
            {
              "word": "nice",
              "start": 8.12,
              "end": 8.32
            }
          ]
        }
      ]
    }
  }
}

partial_translated_transcription​

Translation of an uncompleted segment (optional):

{
  "message_type": "partial_translated_transcription",
  "data": {
    "transcription": {
      "transcription_id": "19f84f5e10a20085",
      "translation_part_id": 0,
      "language": "es",
      "text": "Hoy hace buen",
      "segments": [
        {
          "text": "Hoy hace buen",
          "start": 7.36,
          "end": 8.32,
          "start_timestamp": 1784642133.39,
          "end_timestamp": 1784642134.35,
          "words": [...] // the same as in `partial_transcription`
        }
      ]
    }
  }
}

validated_transcription​

Transcription of a completed segment:

{
  "message_type": "validated_transcription",
  "data": {
    "transcription": {
      "transcription_id": "19f84f5e10a20085",
      "language": "en",
      "text": "The weather is nice today.",
      "segments": [
        {
          "text": "The weather is nice today.",
          "start": 7.36,
          "end": 8.74,
          "start_timestamp": 1784642133.39,
          "end_timestamp": 1784642134.77,
          "words": [
            {
              "word": "The",
              "start": 7.36,
              "end": 7.42
            },
            {
              "word": "weather",
              "start": 7.54,
              "end": 7.90
            },
            {
              "word": "is",
              "start": 7.96,
              "end": 8.04
            },
            {
              "word": "nice",
              "start": 8.12,
              "end": 8.32
            },
            {
              "word": "today.",
              "start": 8.40,
              "end": 8.74
            }
          ]
        }
      ]
    }
  }
}

translated_transcription​

Translation of a completed segment:

{
  "message_type": "translated_transcription",
  "data": {
    "transcription": {
      "transcription_id": "19f84f5e10a20085",
      "translation_part_id": 0,
      "language": "es",
      "text": "Hoy hace buen tiempo.",
      "segments": [
        {
          "text": "Hoy hace buen tiempo.",
          "start": 7.36,
          "end": 8.74,
          "start_timestamp": 1784642133.39,
          "end_timestamp": 1784642134.77,
          "words": [...] // the same as in `validated_transcription`
        }
      ]
    }
  }
}

output_audio_data​

TTS audio chunk (when WebSockets is the audio transport):

{
  "message_type": "output_audio_data",
  "data": {
    "transcription_id": "190983855fe3404e",
    "translation_part_id": "0",  // part index when a phrase is split into several TTS pieces
    "language": "es",            // TTS language
    "last_chunk": false,         // true on the last generated chunk for this `transcription_id`
    "data": "base64 string",     // ≤512 KB base64 (≈384 KB of raw audio)
    "chunk_generation_delta": 120 // optional, ms between generation start and this chunk
  }
}

The audio is always 24 kHz mono; the encoding (pcm_s16le or zlib_pcm_s16le) matches the output_stream.target.format of your task.

current_task​

The response to the get_task command. Contains the current task configuration plus its status:

{
  "message_type": "current_task",
  "data": {
    "input_stream":  { /* ... */ },
    "output_stream": { /* ... */ },
    "pipeline":      { /* ... */ },
    "task_status": "running" // running | paused | unknown
  }
}

current_task is sent in response to get_task, and also automatically when you reconnect to an already running pipeline. The server does not send it as a confirmation of set_task — to verify that a new task has started, poll get_task (it returns a NOT_FOUND error until the pipeline is up).

end_of_stream​

End-of-stream confirmation. Sent after an end_task, once the remaining audio has been processed and delivered; the connection is closed right after.

{ "message_type": "end_of_stream", "data": {} }

warning​

Audio stream health warnings. The server monitors the pace of your input_audio_data stream:

{
  "message_type": "warning",
  "data": {
    "code": "AUDIO_STREAM_TOO_FAST", // AUDIO_STREAM_TOO_FAST | AUDIO_STREAM_TOO_SLOW | AUDIO_STREAM_STALLED
    "message": "Human-readable description"
  }
}

Send audio at real-time rate (e.g., one 320 ms chunk every 320 ms): pushing faster, slower, or stopping mid-stream triggers these warnings and degrades translation quality.

error​

Validation, authorization, or other kinds of errors:

{
  "message_type": "error",
  "data": {
    "code": "VALIDATION_ERROR",
    "desc": "ValidationError(model='SetTaskMessage', errors=[{'loc': ('input_stream', 'content_type')",
    "msg": "value is not a valid enumeration member; permitted: 'audio'\", 'type': 'type_error.enum'",
    "param": null
  }
}