Audio#

Learn how to turn audio into text, synthesize speech, generate music, or extract speaker embeddings with Xinference.

Introduction#

The Audio API provides four methods for interacting with audio:

  • The transcriptions endpoint transcribes audio into the input language.

  • The translations endpoint translates audio into English.

  • The speech endpoint generates audio from the input text.

  • The embeddings endpoint extracts a speaker embedding from an audio file.

API ENDPOINT

OpenAI-compatible ENDPOINT

Transcription API

/v1/audio/transcriptions

Translation API

/v1/audio/translations

Speech API

/v1/audio/speech

Speaker Embedding API

/v1/audio/embeddings

Invalid and silent audio#

  • FunASR transcription models return HTTP 200 with an empty transcription when no speech is detected. An empty audio upload is invalid and returns HTTP 400.

  • MegaTTS3 voice cloning requires decodable reference audio with samples and detectable sound. Empty, undecodable, or effectively silent prompt_speech input, and empty prompt_latent input, return HTTP 400 before inference.

Supported models#

The audio API is supported with the following models in Xinference:

Audio to text#

Audio engines#

Audio models with multiple implementations use one model name and select the runtime with --model-engine:

  • The Whisper models listed above use transformers by default and also support MLX on Mac computers with Apple silicon.

  • F5-TTS and Kokoro-82M use PyTorch by default and also support MLX on Mac computers with Apple silicon.

  • SenseVoiceSmall and Fun-ASR-Nano-2512 use PyTorch by default and also support MLX on Mac computers with Apple silicon.

  • Qwen3-ASR-0.6B and Qwen3-ASR-1.7B use transformers by default. On Linux with NVIDIA GPUs, they can use vLLM for faster transcriptions; on Mac computers with Apple silicon, they can use MLX.

  • The Qwen3-TTS models, MeloTTS-English, MeloTTS-English-v3, and VoxCPM2 use PyTorch by default and also support MLX on Mac computers with Apple silicon.

For example:

xinference launch --model-name whisper-large-v3 --model-type audio --model-engine MLX
xinference launch --model-name F5-TTS --model-type audio --model-engine MLX
xinference launch --model-name Qwen3-ASR-1.7B --model-type audio --model-engine vLLM
xinference launch --model-name Qwen3-TTS-12Hz-0.6B-Base --model-type audio --model-engine MLX

The former *-mlx model names remain accepted as launch compatibility aliases. Registration, cache, version, and virtual-environment lookups use the canonical model name. New integrations should use that name with --model-engine MLX. The Web UI presents the available engines in the launch dialog.

Text to audio (TTS)#

Models supporting zero-shot (direct synthesis without reference audio):

Models supporting voice design (natural-language voice description):

Music generation#

Speaker embeddings#

Models supporting voice cloning (requires reference audio):

Models supporting emotion control:

Quickstart#

Speaker Embeddings#

The Speaker Embedding API accepts one audio file and returns one speaker embedding. The built-in CAMPPlus models return a 192-dimensional vector. The endpoint is intentionally stateless: applications can store the returned vectors and use cosine similarity for speaker verification or 1:N speaker identification. The request uses multipart/form-data: model is the UID of a running speaker-embedding model and file is the audio sample. Unlike the general /v1/embeddings endpoint, this endpoint returns one embedding object rather than a list of text embeddings.

Open Running Models, select a running CAMPPlus model, and upload a clear speech sample in the Speaker Embedding panel. Select Extract embedding to inspect the vector and copy it from the results panel.

ModelScope decodes the input, converts multi-channel audio to one channel, and resamples it to the model’s 16 kHz sample rate. The returned vector preserves the model output. Use cosine similarity when comparing two vectors; choose a verification or identification threshold using representative audio from your own application.

Transcription#

The Transcription API mimics OpenAI’s create transcriptions API. We can try Transcription API out either via cURL, OpenAI Client, or Xinference’s python client:

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/transcriptions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "file": "<audio bytes>",
  }'

Translation#

The Translation API mimics OpenAI’s create translations API. We can try Translation API out either via cURL, OpenAI Client, or Xinference’s python client:

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/translations' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "file": "<audio bytes>",
  }'

Speech#

The Speech API mimics OpenAI’s create speech API. We can try Speech API out either via cURL, OpenAI Client, or Xinference’s python client:

Speech API use non-stream by default as

  1. The stream output of ChatTTS is not as good as the non-stream output, please refer to: 2noise/ChatTTS#564

  2. The stream requires ffmpeg<7: https://pytorch.org/audio/stable/installation.html#optional-dependencies

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "<The text to generate audio for>",
    "voice": "echo",
    "stream": True,
  }'

Breeze-TTS-2 Usage#

Breeze-TTS-2 supports English and Chinese voice design, voice cloning, voice direction, and native streaming. It requires Linux and an NVIDIA CUDA GPU. The model weights and self-hosted outputs are licensed for research and non-commercial use only; review the upstream BreezeBlue model license before launching the model.

For voice design, omit prompt_speech and pass a natural-language voice description in instruct. For voice cloning, pass reference audio bytes in prompt_speech and their exact transcript in prompt_text. Voice direction combines all three fields: prompt_speech, prompt_text, and instruct. cfg_scale defaults to 1; the upstream project recommends 4 when stronger instruction following is needed. seed defaults to 42. The model does not support the OpenAI speed or preset voice controls.

instruct, prompt_text, cfg_scale, and seed use the existing Speech API kwargs channel; raw REST requests encode kwargs as a JSON string. prompt_speech is sent as a multipart file. The Xinference sync and async clients handle both forms through their speech method.

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("<MODEL_UID>")

# Voice design
designed_voice = model.speech(
    input="Welcome aboard. Your journey begins now.",
    voice="",
    response_format="wav",
    instruct="A warm, thoughtful young woman with a calm delivery.",
    cfg_scale=4,
    seed=42,
)

# Voice cloning or voice direction
with open("reference.wav", "rb") as reference_file:
    reference_audio = reference_file.read()
directed_voice = model.speech(
    input="We need to discuss what happened last night.",
    voice="",
    response_format="wav",
    prompt_speech=reference_audio,
    prompt_text="This is the exact transcript of the reference audio.",
    instruct="Speak slowly with a restrained, serious tone.",
    cfg_scale=4,
    seed=42,
)

Set stream=True to receive encoded audio chunks from the model’s native streaming runtime. CUDA Graph acceleration can be enabled at launch with --fast_all true or with the individual fast_text_encoder, fast_backbone_prefill, fast_backbone_decode, fast_depth_decoder, and fast_codec model options. These modes increase startup work and GPU memory use.

ACE-Step1.5 Usage#

The built-in registration uses the PyTorch engine and requires Python 3.11 or 3.12. It downloads the complete ACE-Step 1.5 checkpoint bundle from Hugging Face or ModelScope. The bundle contains the default acestep-v15-turbo DiT, vae, Qwen3-Embedding-0.6B, and acestep-5Hz-lm-1.7B. Xinference uses the official ACE-Step 1.5 Python API in a per-model virtual environment. ACE-Step is available under the MIT license.

This integration supports the bundled DiT, LM, and VAE checkpoints. Standalone checkpoint combinations cannot currently be selected through config_path or lm_model_path. ACE-Step supports CUDA, ROCm, Apple Silicon, Intel XPU, and CPU; accelerator availability and performance depend on the system PyTorch build.

ACE-Step1.5 reuses the Speech endpoint for text-to-music generation. Put lyrics in input and the required music description in instruct inside the existing kwargs field. Use [Instrumental] as the lyrics when generating an instrumental track.

duration defaults to 60 seconds. It accepts -1 for model-selected duration or a value from 10 through 600 seconds. seed=-1 selects a random seed; non-negative integers provide reproducible generation. Supported output formats are aac, flac, mp3, ogg, opus, wav, and wav32. Generation is non-streaming, speed must be 1.0, and voice must be default, an empty string, or null.

Launch the default DiT-only configuration, where thinking is false:

xinference launch --model-name ACE-Step1.5 --model-type audio --model-engine PyTorch

To enable LM planning, metadata completion, and audio-code reasoning, load the bundled 1.7B LM:

xinference launch --model-name ACE-Step1.5 --model-type audio \
  --model-engine PyTorch --lm_model_path acestep-5Hz-lm-1.7B

This enables thinking=true and the related use_cot_* options. The LM uses its PyTorch backend by default. offload_to_cpu, offload_dit_to_cpu, quantization, and compile_model can be supplied as launch options for supported hardware. lm_backend accepts pt, vllm, or mlx; vllm requires CUDA for native execution, while mlx targets Apple Silicon. Other ACE-Step controls, including bpm, keyscale, timesignature, inference_steps, guidance_scale, shift, infer_method, and timesteps, can be supplied through the same kwargs channel.

When the LM is loaded, thinking=false disables audio-code reasoning by default. An explicitly enabled use_cot_caption, use_cot_language, or use_cot_metas still uses the LM for that planning step. MP3, AAC, and Opus output require a working FFmpeg installation. OGG output is generated as WAV and then encoded as OGG/Vorbis with libsndfile.

Raw REST requests encode kwargs as a JSON string. The Xinference sync and async clients accept these names through their existing **kwargs argument.

curl 'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "[Verse]\nMorning light across the city\n[Chorus]\nSing it back to me",
    "voice": "default",
    "response_format": "wav",
    "speed": 1.0,
    "stream": false,
    "kwargs": "{\"instruct\": \"Warm acoustic pop with intimate vocals\", \"seed\": 7, \"duration\": 60}"
  }' \
  --output ace-step.wav

MiniMax-Music3 Usage#

MiniMax-Music3 reuses the Speech endpoint for text-to-music generation. Put the lyrics in input and the required music description in instruct inside the existing kwargs field. Preserve line breaks and put tags such as [Verse] and [Chorus] on their own lines.

duration is the maximum generated length in seconds. Its range is 0.04 through 360 and its default is 60. Xinference passes it directly to the Diffusers pipeline as audio_duration. The model may emit an end-of-audio token and finish before the limit. Supported output formats are flac, mp3, ogg, and wav; WAV is the default. Generation requires stream=false, speed=1.0, and voice set to default, an empty string, or null. Inference requires NVIDIA CUDA. Sampling steps and classifier-free guidance remain at the Diffusers defaults and are not request parameters.

Xinference preserves the Diffusers pipeline’s native 44.1 kHz stereo samples. It wraps them in an IEEE-float WAV container without resampling or integer PCM quantization. FLAC, MP3, and OGG responses are encoded from those samples with libsndfile.

instruct, seed, and duration are model options passed through the existing kwargs channel rather than additional Speech API parameters. Raw REST requests encode kwargs as a JSON string. The Xinference sync and async clients accept these names through their existing **kwargs argument.

curl 'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "[Verse]\nMorning light filtering through the pine\n[Chorus]\nSoftly the world begins to breathe",
    "voice": "default",
    "response_format": "wav",
    "speed": 1.0,
    "stream": false,
    "kwargs": "{\"instruct\": \"Warm acoustic pop with intimate female vocals, fingerpicked guitar, soft piano, and a wide final chorus.\", \"seed\": 7, \"duration\": 60}"
  }' \
  --output music3.wav

ChatTTS Usage#

Basic usage, refer to audio speech usage.

Fixed tone color. We can use fixed tone color provided by 6drf21e/ChatTTS_Speaker, Download the evaluation_result.csv , take seed_2155 as example, we get the emb_data of it.

import pandas as pd

df = pd.read_csv("evaluation_results.csv")
emb_data_2155 = df[df['seed_id'] == 'seed_2155'].iloc[0]["emb_data"]

Use the fixed tone color of seed_2155 to generate speech.

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
resp_bytes = model.speech(
    voice=emb_data_2155,
    input=<The text to generate audio for>
)

CosyVoice Usage#

CosyVoice has two versions: CosyVoice 1.0 and CosyVoice 2.0. CosyVoice 1.0 has three different models:

  • CosyVoice-300M-SFT: Choose this model if you just want to convert text to audio. There are pretrained voices available: [‘中文女’, ‘中文男’, ‘日语男’, ‘粤语女’, ‘英文女’, ‘英文男’, ‘韩语女’]

  • CosyVoice-300M: Choose this model if you want to clone voice or convert text to audio in different languages. The prompt_speech is always required and should be a WAV file. For optimal performance, use a sample rate of 16,000 Hz.

  • CosyVoice-300M-Instruct: Choose this model If you need precise control over the tone and pitch.

Basic usage, launch model CosyVoice-300M-SFT.

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "<The text to generate audio for>",
    # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
    "voice": "中文女"
  }'

Clone voice, launch model CosyVoice-300M.

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

zero_shot_prompt_text = ("<the words in the text exactly match "
                         "the audio file of the zero-shot prompt>")
# The words said in the audio file should be identical
# to zero_shot_prompt_text.
#
# The audio input file must be in WAV format.
# For optimal performance, use a 16,000 Hz sample rate.
#
# Files with different sample rates will be resampled to 16,000 Hz,
# which may increase processing time.
with open(zero_shot_prompt_file, "rb") as f:
    zero_shot_prompt = f.read()

speech_bytes = model.speech(
    "<The text to generate audio for>",
    prompt_text=zero_shot_prompt_text,
    prompt_speech=zero_shot_prompt,
)

Cross lingual usage, launch model CosyVoice-300M.

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

# The audio input file must be in WAV format.
# For optimal performance, use a 16,000 Hz sample rate.
#
# Files with different sample rates will be resampled to 16,000 Hz,
# which may increase processing time.
with open(cross_lingual_prompt_file, "rb") as f:
    cross_lingual_prompt = f.read()

speech_bytes = model.speech(
    "<The text to generate audio for>",  # text could be another language
    prompt_speech=cross_lingual_prompt,
)

Instruction based, launch model CosyVoice-300M-Instruct.

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

response = model.speech(
    "在面对挑战时，他展现了非凡的<strong>勇气</strong>与<strong>智慧</strong>。",
    voice="中文男",
    instruct_text="Theo 'Crimson', is a fiery, passionate rebel leader. "
    "Fights with fervor for justice, but struggles with impulsiveness.",
)

CosyVoice 2.0 only has one model, it provides all the capabilities of the three CosyVoice models. The usage is the same as CosyVoice.

CosyVoice 2.0 stream usage, launch model CosyVoice2-0.5B.

# Launch model
from xinference.client import Client

model_uid = client.launch_model(
    model_name=model_name,
    model_type="audio",
    download_hub="modelscope",
)

endpoint = "http://127.0.0.1:9997"
input_string = "你好，我是通义生成式语音大模型，请问有什么可以帮您的吗？"

# Stream request by openai client
import openai
import tempfile

openai_client = openai.Client(api_key="not empty", base_url=f"{endpoint}/v1")
# ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
response = openai_client.audio.speech.with_streaming_response.create(
    model=model_uid, input=input_string, voice="英文女"
)
with tempfile.NamedTemporaryFile(suffix=".mp3", delete=True) as f:
    response.stream_to_file(f.name)
    assert os.stat(f.name).st_size > 0

# Stream request by xinference client
response = model.speech(input_string, stream=True)
assert inspect.isgenerator(response)
with tempfile.NamedTemporaryFile(suffix=".mp3", delete=True) as f:
    for chunk in response:
        f.write(chunk)

More instructions and examples, could be found at https://fun-audio-llm.github.io/ .

FishSpeech Usage#

Basic usage, refer to audio speech usage.

Clone voice, launch model FishSpeech-1.5. Please use prompt_speech instead of reference_audio and prompt_text instead of reference_text to clone voice from the reference audio for the FishSpeech model. This arguments is aligned to voice cloning of CosyVoice.

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

# The reference audio file is the voice file
# the words said in the file should be identical to reference_text
with open(reference_audio_file, "rb") as f:
    reference_audio = f.read()
reference_text = ""  # text in the audio

speech_bytes = model.speech(
    "<The text to generate audio for>",
    prompt_speech=reference_audio,
    prompt_text=reference_text
)

Paraformer Usage#

model

vad

punc

timestamp

speaker

hotword

paraformer-zh

yes

yes

no

no

no

paraformer-zh-hotword

yes

yes

no

no

yes

paraformer-zh-spk

yes

yes

yes

yes

no

paraformer-zh-long

yes

yes

yes

yes

no

seaco-paraformer-zh (recommend)

yes

yes

yes

yes

yes

  1. VAD & Punctuation Usage

    All Paraformer models support VAD and punctuation.

  2. Timestamp & Speaker Usage

    Only the following models support timestamp and speaker:

    • paraformer-zh-spk

    • paraformer-zh-long

    • seaco-paraformer-zh

    Among them, only paraformer-zh-spk enables speaker info by default.

    If you need speaker info when using paraformer-zh-long or seaco-paraformer-zh:

    • In Web UI: add an extra parameter with key spk_model and value cam++

    • In command line: add the option --spk_model cam++

    Example:

    from xinference.client import Client
client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("seaco-paraformer-zh")
with open("asr_example.wav", "rb") as audio_file:
    audio = audio_file.read()
    model.transcriptions(audio, response_format="verbose_json")

  3. Hotword Usage

    Only the following models support hotword:

    • paraformer-zh-hotword

    • seaco-paraformer-zh

    Example:

    from xinference.client import Client
client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("seaco-paraformer-zh")
with open("asr_example.wav", "rb") as audio_file:
    audio = audio_file.read()
    model.transcriptions(audio, hotword="小艾 魔搭")

SenseVoiceSmall Offline Usage#

Now SenseVoiceSmall use a small vad model fsmn-vad, it will be downloaded thus network required.

For offline environment, you can download the vad model in advance.

Download from huggingface or modelscope. Assume downloaded to /path/to/fsmn-vad.

Then when launching SenseVoiceSmall with Web UI, you can add an additional parameter with key vad_model and value /path/to/fsmn-vad which is the downloaded path. When launching with command line, you can add an option --vad_model /path/to/fsmn-vad.

Kokoro Usage#

The Kokoro model supports multiple languages, but the default language is English. If you want to use other languages, such as Chinese, you need to install additional dependency packages and add an additional parameter when starting the model.

  1. pip install misaki[zh]

  2. Initialize the model with the parameter lang_code=’z’, For all available lang_code options, please refer to kokoro source code. If the model is started through the web UI, an additional parameter needs to be added, with the key as lang_code and the value as z. If the model is started through the xinference client, the parameters are passed via the launch_model interface:

    model_uid = client.launch_model(
    model_name="Kokoro-82M",
    model_type="audio",
    compile=False,
    download_hub="huggingface",
    lang_code="z",
)

  3. When inferring, the voice must start with ‘z’, for example: zf_xiaoyi. The currently supported voices are: https://huggingface.co/hexgrad/Kokoro-82M/tree/main/voices. For example:

    input_string = "重新启动即可更新"
response = model.speech(input_string, voice="zf_xiaoyi")

IndexTTS2 Usage#

The IndexTTS2 model supports emotion control, you can use this feature by specifying some additional parameters. Here are several examples of how to use IndexTTS2:

  1. Synthesize new speech with a single reference audio file (voice cloning):

    from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

response = model.speech(
    input="Translate for me, what is a surprise!",
    prompt_speech=test_prompt_speech,
)

  2. Using a separate, emotional reference audio file to condition the speech synthesis:

    from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

with open("example/emo_sad.wav", "rb") as f:
    emo_prompt_speech = f.read()

response = model.speech(
    input="It's such a shame the singer didn't make it to the finals.",
    prompt_speech=test_prompt_speech,
    emo_audio_prompt=emo_prompt_speech
)

  3. When an emotional reference audio file is specified, you can optionally set the emo_alpha to adjust how much it affects the output. Valid range is 0.0 - 1.0 , and the default value is 1.0 (100%):

    from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

with open("example/emo_sad.wav", "rb") as f:
    emo_prompt_speech = f.read()

response = model.speech(
    input="It's such a shame the singer didn't make it to the finals.",
    prompt_speech=test_prompt_speech,
    emo_audio_prompt=emo_prompt_speech,
    emo_alpha=0.9
)

  4. It’s also possible to omit the emotional reference audio and instead provide an 8-float list specifying the intensity of each emotion, in the following order: [happy, angry, sad, afraid, disgusted, melancholic, surprised, calm] . You can additionally use the use_random parameter to introduce stochasticity during inference; the default is False , and setting it to True enables randomness:

    from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

response = model.speech(
    input="Wow, I'm so lucky!",
    prompt_speech=test_prompt_speech,
    emo_vector=[0, 0, 0, 0, 0, 0, 0.45, 0],
    use_random=False
)

  5. Alternatively, you can enable use_emo_text to guide the emotions based on your provided text script. Your text script will then automatically be converted into emotion vectors. It’s recommended to use emo_alpha around 0.6 (or lower) when using the text emotion modes, for more natural sounding speech. You can introduce randomness with use_random (default: False; True enables randomness):

    from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

response = model.speech(
    input="Quick, hide! He's coming! He's coming to get us!",
    prompt_speech=test_prompt_speech,
    emo_alpha=0.6,
    use_emo_text=True,
    use_random=False
)

  6. It’s also possible to directly provide a specific text emotion description via the emo_text parameter. Your emotion text will then automatically be converted into emotion vectors. This gives you separate control of the text script and the text emotion description:

    from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

response = model.speech(
    input="Quick, hide! He's coming! He's coming to get us!",
    prompt_speech=test_prompt_speech,
    emo_alpha=0.6,
    use_emo_text=True,
    emo_text="You scared the hell out of me! Are you a ghost?",
    use_random=False
)

IndexTTS-2.5 Usage#

IndexTTS-2.5 adds Japanese, Spanish, and Arabic, as well as pronunciation and speaking-speed control. A reference audio file and a language are used for each request. Supported language values are ZH, EN, JA, ES, and AR:

from xinference.client import Client

client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS-2.5")

with open("prompt.wav", "rb") as f:
    prompt_speech = f.read()

response = model.speech(
    input="Hello, this is a voice cloning demo.",
    prompt_speech=prompt_speech,
    language="EN",
    speed=1.2,
)

The standard speed parameter uses OpenAI semantics: values above 1.0 speed up speech. The supported range is 0.5 to 2.0. You can instead pass the upstream duration_factor option with the same supported range, where values above 1.0 slow speech down. Pronunciation can be controlled with <word|reading> notation, for example 他在银<行|XING2>里<行|HANG2>走了半天。.

Upstream officially supports Python 3.10 and 3.11. A GPU is strongly recommended; enable use_bf16 at launch to reduce GPU memory use. Text-based emotion guidance additionally requires use_qwen_emo=True.

IndexTTS2 Offline Usage#

IndexTTS2 requires several small models that are downloaded automatically during initialization. For offline environments, you can download these models to a single directory and specify the directory path.

Easy Setup Method

The simplest way to set up offline usage is to Use the hf download command to download the small model in advance:

# Create your local models directory
mkdir -p /path/to/small_models

# Download models from Hugging Face
hf download facebook/w2v-bert-2.0 --local-dir /path/to/small_models/w2v-bert-2.0
hf download funasr/campplus --local-dir /path/to/small_models/campplus
hf download nvidia/bigvgan_v2_22khz_80band_256x --local-dir /path/to/small_models/bigvgan
hf download amphion/MaskGCT --local-dir /path/to/small_models/MaskGCT

The final directory structure should look like this:

/path/to/small_models/
├── w2v-bert-2.0/                 # Feature extraction model
├── campplus/                     # Speaker recognition model
├── bigvgan/                      # Vocoder model
└── MaskGCT/                      # Semantic codec model

Required Models

The small models are automatically mapped as follows:

  1. w2v-bert-2.0 (models--facebook--w2v-bert-2.0) - Feature extraction model

  2. campplus (models--funasr--campplus) - Speaker recognition model

  3. bigvgan (models--nvidia--bigvgan_v2_22khz_80band_256x) - Vocoder model

  4. semantic_codec (models--amphion--MaskGCT) - Semantic encoding/decoding model

Launching IndexTTS2 with Offline Models

When launching IndexTTS2 with Web UI, you can add an additional parameter: - small_models_dir - Path to directory containing all small models

When launching with command line, you can add the option:

xinference launch --model-name IndexTTS2 --model-type audio \
    --small_models_dir /path/to/small_models

When launching with Python client:

model_uid = client.launch_model(
    model_name="IndexTTS2",
    model_type="audio",
    small_models_dir="/path/to/small_models"
)