音频#

Learn how to turn audio into text, synthesize speech, generate music, or extract speaker embeddings with Xinference.

介绍#

The Audio API provides four methods for interacting with audio:

转录终端将音频转录为输入语言。
翻译端点将音频转换为英文。
转录终端将音频转录为输入语言。
The embeddings endpoint extracts a speaker embedding from an audio file.

API 端点	OpenAI 兼容端点
转录 API	/v1/audio/transcriptions
翻译 API	/v1/audio/translations
语音 API	/v1/audio/speech
Speaker Embedding API	/v1/audio/embeddings

支持的模型列表#

在Xinference中，以下模型支持音频API：

语音转文本#

Audio engines#

Audio models with multiple implementations use one model name and select the runtime with --model-engine:

The Whisper models listed above use transformers by default and also support MLX on Mac computers with Apple silicon.
F5-TTS and Kokoro-82M use PyTorch by default and also support MLX on Mac computers with Apple silicon.
SenseVoiceSmall and Fun-ASR-Nano-2512 use PyTorch by default and also support MLX on Mac computers with Apple silicon.
Qwen3-ASR-0.6B and Qwen3-ASR-1.7B use transformers by default. On Linux with NVIDIA GPUs, they can use vLLM for faster transcriptions; on Mac computers with Apple silicon, they can use MLX.
The Qwen3-TTS models, MeloTTS-English, MeloTTS-English-v3, and VoxCPM2 use PyTorch by default and also support MLX on Mac computers with Apple silicon.

For example:

xinference launch --model-name whisper-large-v3 --model-type audio --model-engine MLX
xinference launch --model-name F5-TTS --model-type audio --model-engine MLX
xinference launch --model-name Qwen3-ASR-1.7B --model-type audio --model-engine vLLM
xinference launch --model-name Qwen3-TTS-12Hz-0.6B-Base --model-type audio --model-engine MLX

The former *-mlx model names remain accepted as launch compatibility aliases. Registration, cache, version, and virtual-environment lookups use the canonical model name. New integrations should use that name with --model-engine MLX. The Web UI presents the available engines in the launch dialog.

文本转语音（TTS）#

支持zero-shot的模型 （无需参考音频）

Models supporting voice design (natural-language voice description):

Music generation#

ACE-Step1.5
MiniMax-Music3 (NVIDIA CUDA only)

Speaker embeddings#

支持语音克隆的模型 （需要参考音频）

支持情感控制的模型

快速入门#

Speaker Embeddings#

The Speaker Embedding API accepts one audio file and returns one speaker embedding. The built-in CAMPPlus models return a 192-dimensional vector. The endpoint is intentionally stateless: applications can store the returned vectors and use cosine similarity for speaker verification or 1:N speaker identification. The request uses multipart/form-data: model is the UID of a running speaker-embedding model and file is the audio sample. Unlike the general /v1/embeddings endpoint, this endpoint returns one embedding object rather than a list of text embeddings.

Open Running Models, select a running CAMPPlus model, and upload a clear speech sample in the Speaker Embedding panel. Select Extract embedding to inspect the vector and copy it from the results panel.

curl -X POST \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/embeddings' \
  -H 'accept: application/json' \
  -F 'model=<MODEL_UID>' \
  -F 'file=@speaker.wav'

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("<MODEL_UID>")

with open("speaker.wav", "rb") as audio_file:
    result = model.create_embedding(audio_file.read())

embedding = result["embedding"]

{
  "object": "embedding",
  "model": "<MODEL_UID>",
  "dimensions": 192,
  "embedding": [0.0123, -0.0456, 0.0789]
}

ModelScope decodes the input, converts multi-channel audio to one channel, and resamples it to the model's 16 kHz sample rate. The returned vector preserves the model output. Use cosine similarity when comparing two vectors; choose a verification or identification threshold using representative audio from your own application.

转录#

Transcription API 模仿了 OpenAI 的 create transcriptions API。你可以通过 cURL、OpenAI Client 或者 Xinference 的 Python 客户端来尝试 Transcription API：

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/transcriptions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "file": "<audio bytes>",
  }'

import openai

client = openai.Client(
    api_key="cannot be empty",
    base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
)
with open("speech.mp3", "rb") as audio_file:
    client.audio.transcriptions.create(
        model=<MODEL_UID>,
        file=audio_file,
    )

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
with open("speech.mp3", "rb") as audio_file:
    model.transcriptions(audio=audio_file.read())

{
  "text": "Imagine the wildest idea that you've ever had, and you're curious about how it might scale to something that's a 100, a 1,000 times bigger. This is a place where you can get to do that."
}

翻译#

Translation API 模仿了 OpenAI 的 create translations API。你可以通过 cURL、OpenAI Client 或 Xinference 的 Python 客户端来尝试使用 Translation API：

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/translations' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "file": "<audio bytes>",
  }'

import openai

client = openai.Client(
    api_key="cannot be empty",
    base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
)
with open("speech.mp3", "rb") as audio_file:
    client.audio.translations.create(
        model=<MODEL_UID>,
        file=audio_file,
    )

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
with open("speech.mp3", "rb") as audio_file:
    model.translations(audio=audio_file.read())

{
  "text": "Hello, my name is Wolfgang and I come from Germany. Where are you heading today?"
}

语音#

Transcription API 模仿了 OpenAI 的 create speech API。你可以通过 cURL、OpenAI Client 或者 Xinference 的 Python 客户端来尝试 Speech API：

Speech API 默认使用非流式

ChatTTS 的流式输出不如非流式的效果好，参考：2noise/ChatTTS#564
流式要求 ffmpeg<7：https://pytorch.org/audio/stable/installation.html#optional-dependencies

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "<The text to generate audio for>",
    "voice": "echo",
    "stream": True,
  }'

import openai

client = openai.Client(
    api_key="cannot be empty",
    base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
)
client.audio.speech.create(
    model=<MODEL_UID>,
    input=<The text to generate audio for>,
    voice="echo",
)

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
model.speech(
    input=<The text to generate audio for>,
    voice="echo",
    stream: True,
)

The output will be an audio binary.

Breeze-TTS-2 Usage#

Breeze-TTS-2 supports English and Chinese voice design, voice cloning, voice direction, and native streaming. It requires Linux and an NVIDIA CUDA GPU. The model weights and self-hosted outputs are licensed for research and non-commercial use only; review the upstream BreezeBlue model license before launching the model.

For voice design, omit prompt_speech and pass a natural-language voice description in instruct. For voice cloning, pass reference audio bytes in prompt_speech and their exact transcript in prompt_text. Voice direction combines all three fields: prompt_speech, prompt_text, and instruct. cfg_scale defaults to 1; the upstream project recommends 4 when stronger instruction following is needed. seed defaults to 42. The model does not support the OpenAI speed or preset voice controls.

instruct, prompt_text, cfg_scale, and seed use the existing Speech API kwargs channel; raw REST requests encode kwargs as a JSON string. prompt_speech is sent as a multipart file. The Xinference sync and async clients handle both forms through their speech method.

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("<MODEL_UID>")

# Voice design
designed_voice = model.speech(
    input="Welcome aboard. Your journey begins now.",
    voice="",
    response_format="wav",
    instruct="A warm, thoughtful young woman with a calm delivery.",
    cfg_scale=4,
    seed=42,
)

# Voice cloning or voice direction
with open("reference.wav", "rb") as reference_file:
    reference_audio = reference_file.read()
directed_voice = model.speech(
    input="We need to discuss what happened last night.",
    voice="",
    response_format="wav",
    prompt_speech=reference_audio,
    prompt_text="This is the exact transcript of the reference audio.",
    instruct="Speak slowly with a restrained, serious tone.",
    cfg_scale=4,
    seed=42,
)

Set stream=True to receive encoded audio chunks from the model's native streaming runtime. CUDA Graph acceleration can be enabled at launch with --fast_all true or with the individual fast_text_encoder, fast_backbone_prefill, fast_backbone_decode, fast_depth_decoder, and fast_codec model options. These modes increase startup work and GPU memory use.

ACE-Step1.5 使用说明#

内置注册使用 PyTorch 引擎，并要求 Python 3.11 或 3.12。它会从 Hugging Face 或 ModelScope 下载完整的 ACE-Step 1.5 检查点套件。该套件包含默认的 acestep-v15-turbo DiT、vae、Qwen3-Embedding-0.6B 和 acestep-5Hz-lm-1.7B。Xinference 在每个模型独立的虚拟环境中使用官方 ACE-Step 1.5 Python API。ACE-Step 采用 MIT 许可证。

此集成支持套件中包含的 DiT、LM 和 VAE 检查点。目前不能通过 config_path 或 lm_model_path 选择独立检查点的任意组合。ACE-Step 支持 CUDA、ROCm、Apple Silicon、Intel XPU 和 CPU；加速器的可用性和性能取决于系统安装的 PyTorch 版本。

ACE-Step1.5 复用 Speech 端点进行文本到音乐生成。将歌词放入 input，并将必需的音乐描述放入现有 kwargs 字段中的 instruct。生成纯音乐时，使用 [Instrumental] 作为歌词。

duration defaults to 60 seconds. It accepts -1 for model-selected duration or a value from 10 through 600 seconds. seed=-1 selects a random seed; non-negative integers provide reproducible generation. Supported output formats are aac, flac, mp3, ogg, opus, wav, and wav32. Generation is non-streaming, speed must be 1.0, and voice must be default, an empty string, or null.

启动默认的纯 DiT 配置，其中 thinking 为 false:

xinference launch --model-name ACE-Step1.5 --model-type audio --model-engine PyTorch

要启用 LM 规划、元数据补全和音频编码推理，请加载随附的 1.7B LM:

xinference launch --model-name ACE-Step1.5 --model-type audio \
  --model-engine PyTorch --lm_model_path acestep-5Hz-lm-1.7B

这会启用 thinking=true 及相关的 use_cot_* 选项。LM 默认使用 PyTorch 后端。对于支持的硬件，可以在启动时提供 offload_to_cpu、offload_dit_to_cpu、quantization 和 compile_model。lm_backend 接受 pt、vllm 或 mlx；vllm 的原生执行需要 CUDA，而 mlx 面向 Apple Silicon。其他 ACE-Step 控制项，包括 bpm、keyscale、timesignature、inference_steps、guidance_scale、shift、infer_method 和 timesteps，可通过同一个 kwargs 通道提供。

When the LM is loaded, thinking=false disables audio-code reasoning by default. An explicitly enabled use_cot_caption, use_cot_language, or use_cot_metas still uses the LM for that planning step. MP3, AAC, and Opus output require a working FFmpeg installation. OGG output is generated as WAV and then encoded as OGG/Vorbis with libsndfile.

原始 REST 请求将 kwargs 编码为 JSON 字符串。Xinference 同步和异步客户端通过现有的 **kwargs 参数接受这些名称。

curl 'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "[Verse]\nMorning light across the city\n[Chorus]\nSing it back to me",
    "voice": "default",
    "response_format": "wav",
    "speed": 1.0,
    "stream": false,
    "kwargs": "{\"instruct\": \"Warm acoustic pop with intimate vocals\", \"seed\": 7, \"duration\": 60}"
  }' \
  --output ace-step.wav

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("<MODEL_UID>")
wav = model.speech(
    input="[Verse]\nMorning light\n[Chorus]\nSing again",
    instruct="Warm acoustic pop with intimate vocals",
    voice="default",
    response_format="wav",
    seed=7,
    duration=60,
)
with open("ace-step.wav", "wb") as output:
    output.write(wav)

MiniMax-Music3 Usage#

MiniMax-Music3 reuses the Speech endpoint for text-to-music generation. Put the lyrics in input and the required music description in instruct inside the existing kwargs field. Preserve line breaks and put tags such as [Verse] and [Chorus] on their own lines.

duration is the maximum generated length in seconds. Its range is 0.04 through 360 and its default is 60. Xinference passes it directly to the Diffusers pipeline as audio_duration. The model may emit an end-of-audio token and finish before the limit. Supported output formats are flac, mp3, ogg, and wav; WAV is the default. Generation requires stream=false, speed=1.0, and voice set to default, an empty string, or null. Inference requires NVIDIA CUDA. Sampling steps and classifier-free guidance remain at the Diffusers defaults and are not request parameters.

Xinference preserves the Diffusers pipeline's native 44.1 kHz stereo samples. It wraps them in an IEEE-float WAV container without resampling or integer PCM quantization. FLAC, MP3, and OGG responses are encoded from those samples with libsndfile.

instruct, seed, and duration are model options passed through the existing kwargs channel rather than additional Speech API parameters. Raw REST requests encode kwargs as a JSON string. The Xinference sync and async clients accept these names through their existing **kwargs argument.

curl 'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "[Verse]\nMorning light filtering through the pine\n[Chorus]\nSoftly the world begins to breathe",
    "voice": "default",
    "response_format": "wav",
    "speed": 1.0,
    "stream": false,
    "kwargs": "{\"instruct\": \"Warm acoustic pop with intimate female vocals, fingerpicked guitar, soft piano, and a wide final chorus.\", \"seed\": 7, \"duration\": 60}"
  }' \
  --output music3.wav

import json

from openai import OpenAI

client = OpenAI(
    api_key="cannot be empty",
    base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1",
)
response = client.audio.speech.create(
    model="<MODEL_UID>",
    input="[Verse]\nMorning light\n[Chorus]\nSing again",
    voice="default",
    response_format="wav",
    extra_body={
        "kwargs": json.dumps({
            "instruct": "Warm acoustic pop with intimate vocals and soft piano.",
            "seed": 7,
            "duration": 60,
        })
    },
)
response.write_to_file("music3.wav")

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("<MODEL_UID>")
wav = model.speech(
    input="[Verse]\nMorning light\n[Chorus]\nSing again",
    instruct="Warm acoustic pop with intimate vocals and soft piano.",
    voice="default",
    response_format="wav",
    seed=7,
    duration=60,
)
with open("music3.wav", "wb") as output:
    output.write(wav)

ChatTTS 使用#

基本使用，参考语音使用章节。

固定音色。我们可以使用由 6drf21e/ChatTTS_Speaker 提供的固定音色，下载 evaluation_result.csv ，以 seed_2155 音色作为例子，我们使用 emb_data 列的数据。

import pandas as pd

df = pd.read_csv("evaluation_results.csv")
emb_data_2155 = df[df['seed_id'] == 'seed_2155'].iloc[0]["emb_data"]

使用 seed_2155 固定音色来创建语音。

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
resp_bytes = model.speech(
    voice=emb_data_2155,
    input=<The text to generate audio for>
)

CosyVoice 模型使用#

CosyVoice 有两个版本：CosyVoice 1.0 和 CosyVoice 2.0。CosyVoice 1.0 有 3 个不同模型：

CosyVoice-300M-SFT: 如果你只想把文本转换为语音，选择这个模型。它提供了一些预训练的音色: ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
CosyVoice-300M: 如果你想克隆声音或者把文本转换成另一种语言的语音，选择这个模型。使用这个模型，你必须提供 prompt_speech WAV格式音频文件，请使用 16,000 Hz 采样率以获得更好的性能。
CosyVoice-300M-Instruct: 如果你想精确控制音调和音色，选择这个模型。

基本使用，加载模型 CosyVoice-300M-SFT。

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "<The text to generate audio for>",
    # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
    "voice": "中文女"
  }'

import openai

client = openai.Client(
    api_key="cannot be empty",
    base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
)
response = client.audio.speech.create(
    model=<MODEL_UID>,
    input=<The text to generate audio for>,
    # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
    voice="中文女",
)
response.stream_to_file('1.mp3')

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
speech_bytes = model.speech(
    input=<The text to generate audio for>,
    # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
    voice="中文女"
)
with open('1.mp3', 'wb') as f:
    f.write(speech_bytes)

克隆声音，加载模型 CosyVoice-300M。

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

zero_shot_prompt_text = ("<the words in the text exactly match "
                         "the audio file of the zero-shot prompt>")
# The words said in the audio file should be identical
# to zero_shot_prompt_text.
#
# The audio input file must be in WAV format.
# For optimal performance, use a 16,000 Hz sample rate.
#
# Files with different sample rates will be resampled to 16,000 Hz,
# which may increase processing time.
with open(zero_shot_prompt_file, "rb") as f:
    zero_shot_prompt = f.read()

speech_bytes = model.speech(
    "<The text to generate audio for>",
    prompt_text=zero_shot_prompt_text,
    prompt_speech=zero_shot_prompt,
)

跨语言使用，加载模型 CosyVoice-300M。

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

# The audio input file must be in WAV format.
# For optimal performance, use a 16,000 Hz sample rate.
#
# Files with different sample rates will be resampled to 16,000 Hz,
# which may increase processing time.
with open(cross_lingual_prompt_file, "rb") as f:
    cross_lingual_prompt = f.read()

speech_bytes = model.speech(
    "<The text to generate audio for>",  # text could be another language
    prompt_speech=cross_lingual_prompt,
)

基于指令的声音合成，加载模型 CosyVoice-300M-Instruct。

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

response = model.speech(
    "在面对挑战时，他展现了非凡的<strong>勇气</strong>与<strong>智慧</strong>。",
    voice="中文男",
    instruct_text="Theo 'Crimson', is a fiery, passionate rebel leader. "
    "Fights with fervor for justice, but struggles with impulsiveness.",
)

CosyVoice 2.0 只有一个模型，但它包含了 CosyVoice 三个模型的所有能力。使用方法与 CosyVoice 一样。

CosyVoice 2.0 流式使用，加载模型 CosyVoice2-0.5B。

# Launch model
from xinference.client import Client

model_uid = client.launch_model(
    model_name=model_name,
    model_type="audio",
    download_hub="modelscope",
)

endpoint = "http://127.0.0.1:9997"
input_string = "你好，我是通义生成式语音大模型，请问有什么可以帮您的吗？"

# Stream request by openai client
import openai
import tempfile

openai_client = openai.Client(api_key="not empty", base_url=f"{endpoint}/v1")
# ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
response = openai_client.audio.speech.with_streaming_response.create(
    model=model_uid, input=input_string, voice="英文女"
)
with tempfile.NamedTemporaryFile(suffix=".mp3", delete=True) as f:
    response.stream_to_file(f.name)
    assert os.stat(f.name).st_size > 0

# Stream request by xinference client
response = model.speech(input_string, stream=True)
assert inspect.isgenerator(response)
with tempfile.NamedTemporaryFile(suffix=".mp3", delete=True) as f:
    for chunk in response:
        f.write(chunk)

更多指令和例子，可以参考 https://fun-audio-llm.github.io/ 。

FishSpeech 模型使用#

基本使用，参考语音使用章节。

克隆语音，启动模型 FishSpeech-1.5。请使用 prompt_speech`而不是 `reference_audio 以及 prompt_text 而不是 reference_text 来为 FishSpeech 模型提供参考音频。这个参数和 CosyVoice 的语音克隆保持一致。

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

# The reference audio file is the voice file
# the words said in the file should be identical to reference_text
with open(reference_audio_file, "rb") as f:
    reference_audio = f.read()
reference_text = ""  # text in the audio

speech_bytes = model.speech(
    "<The text to generate audio for>",
    prompt_speech=reference_audio,
    prompt_text=reference_text
)

Paraformer 使用说明#

model	语音活动检测（vad）	标点恢复（punc）	时间戳	说话人	热词
paraformer-zh	是	是	否	否	否
paraformer-zh-hotword	是	是	否	否	是
paraformer-zh-spk	是	是	是	是	否
paraformer-zh-long	是	是	是	是	否
seaco-paraformer-zh （推荐）	是	是	是	是	是

VAD 与标点符号的使用

所有 Paraformer 模型均支持 VAD 和标点功能。
时间戳和说话人识别使用说明

仅以下模型支持时间戳和说话人识别：
- paraformer-zh-spk
- paraformer-zh-long
- seaco-paraformer-zh
其中，仅 paraformer-zh-spk 默认启用说话人识别功能。

如果你使用的是 paraformer-zh-long 或 seaco-paraformer-zh，且需要启用说话人识别功能：
- 在 Web UI 中：添加名为 spk_model、值为 cam++ 的参数
- 在命令行中：添加参数 --spk_model cam++
示例：
```
from xinference.client import Client
client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("seaco-paraformer-zh")
with open("asr_example.wav", "rb") as audio_file:
    audio = audio_file.read()
    model.transcriptions(audio, response_format="verbose_json")
```

热词功能使用说明

仅以下模型支持 hotword （热词功能）：

paraformer-zh-hotword
seaco-paraformer-zh

示例：

from xinference.client import Client
client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("seaco-paraformer-zh")
with open("asr_example.wav", "rb") as audio_file:
    audio = audio_file.read()
    model.transcriptions(audio, hotword="小艾 魔搭")

SenseVoiceSmall 离线使用#

现在 SenseVoiceSmall 使用一个小的 VAD 模型 fsmn-vad，因此它需要网络来下载。

对于离线环境，你可以提前下载这个 VAD 模型。

从 huggingface 或者 modelscope 下载。假设下载到 /path/to/fsmn-vad。

然后当用 Web UI 加载 SenseVoiceSmall 时，添加额外选项，key 是 vad_model，值是之前的下载路径 /path/to/fsmn-vad。用命令行加载时，增加选项 --vad_model /path/to/fsmn-vad。

Kokoro 模型使用#

Kokoro模型支持多语言，默认是英文。如果你想使用非默认语言，例如中文，则需要安装额外依赖包并且在模型启动时增加对应参数。

pip install misaki[zh]
使用 lang_code='z' 参数初始化模型，可以参考 kokoro source code 查看所有支持的 lang_code。如果你是通过 Web UI启动的模型，则需要添加额外参数，key是 lang_code，value是 z。如果你是通过 xinference client启动的模型，则可以参考如下代码传递参数：
```
model_uid = client.launch_model(
    model_name="Kokoro-82M",
    model_type="audio",
    compile=False,
    download_hub="huggingface",
    lang_code="z",
)
```
当推理时，需要使用 'z' 开头的 voice，例如：zf_xiaoyi。目前支持的 voices 可以参考 https://huggingface.co/hexgrad/Kokoro-82M/tree/main/voices。使用方法如下：
```
input_string = "重新启动即可更新"
response = model.speech(input_string, voice="zf_xiaoyi")
```

IndexTTS2 使用#

IndexTTS2模型支持情感控制，你可以通过使用一些额外的参数来时用这个功能。以下为IndexTTS2的使用方式：

单一参考音频（音色克隆）：

from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

response = model.speech(
    input="Translate for me, what is a surprise!",
    prompt_speech=test_prompt_speech,
)

指定情感参考音频：

from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

with open("example/emo_sad.wav", "rb") as f:
    emo_prompt_speech = f.read()

response = model.speech(
    input="It's such a shame the singer didn't make it to the finals.",
    prompt_speech=test_prompt_speech,
    emo_audio_prompt=emo_prompt_speech
)

当指定情感参考音频时，可以选择设置 emo_alpha 参数以调整其对输出的影响程度。有效范围为 0.0 - 1.0 ，默认值为 1.0 (100%)。

from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

with open("example/emo_sad.wav", "rb") as f:
    emo_prompt_speech = f.read()

response = model.speech(
    input="It's such a shame the singer didn't make it to the finals.",
    prompt_speech=test_prompt_speech,
    emo_audio_prompt=emo_prompt_speech,
    emo_alpha=0.9
)

可以省略情绪参考音频，转而提供一个包含8个浮点数的列表，按以下顺序指定每种情绪的强度: [快乐, 愤怒, 悲伤, 恐惧, 厌恶, 忧郁, 惊讶, 平静] 。您还可以使用 use_random 参数在推理过程中引入随机性情绪；默认值为 False ，设置为 True 即可启用随机性情绪。

from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

response = model.speech(
    input="Wow, I'm so lucky!",
    prompt_speech=test_prompt_speech,
    emo_vector=[0, 0, 0, 0, 0, 0, 0.45, 0],
    use_random=False
)

或者，您可以启用 use_emo_text 功能，根据您提供的 text 脚本引导情感表达。您的文本脚本将自动转换为情感向量。使用文本情感模式时，建议将 emo_alpha 设置为 0.6 左右（或更低），以获得更自然的语音效果。您可通过 use_random 引入随机性（默认值：False ；True 启用随机性）：

from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

response = model.speech(
    input="Quick, hide! He's coming! He's coming to get us!",
    prompt_speech=test_prompt_speech,
    emo_alpha=0.6,
    use_emo_text=True,
    use_random=False
)

您也可以通过 emo_text 参数直接提供特定的文本情绪描述。您的情绪文本将自动转换为情绪向量。这使您能够分别控制文本脚本和文本情绪描述：

from xinference.client import Client
client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS2")

with open("../mp3_test_voice.mp3", "rb") as f:
    test_prompt_speech = f.read()

response = model.speech(
    input="Quick, hide! He's coming! He's coming to get us!",
    prompt_speech=test_prompt_speech,
    emo_alpha=0.6,
    use_emo_text=True,
    emo_text="You scared the hell out of me! Are you a ghost?",
    use_random=False
)

IndexTTS-2.5 Usage#

IndexTTS-2.5 adds Japanese, Spanish, and Arabic, as well as pronunciation and speaking-speed control. A reference audio file and a language are used for each request. Supported language values are ZH, EN, JA, ES, and AR:

from xinference.client import Client

client = Client("http://0.0.0.0:6735")
model = client.get_model("IndexTTS-2.5")

with open("prompt.wav", "rb") as f:
    prompt_speech = f.read()

response = model.speech(
    input="Hello, this is a voice cloning demo.",
    prompt_speech=prompt_speech,
    language="EN",
    speed=1.2,
)

The standard speed parameter uses OpenAI semantics: values above 1.0 speed up speech. The supported range is 0.5 to 2.0. You can instead pass the upstream duration_factor option with the same supported range, where values above 1.0 slow speech down. Pronunciation can be controlled with <word|reading> notation, for example 他在银<行|XING2>里<行|HANG2>走了半天。.

Upstream officially supports Python 3.10 and 3.11. A GPU is strongly recommended; enable use_bf16 at launch to reduce GPU memory use. Text-based emotion guidance additionally requires use_qwen_emo=True.

IndexTTS2 离线使用#

IndexTTS2需要多个小型模型，这些模型会在初始化过程中自动下载。在离线环境中，您可以将这些模型下载到单一目录，并指定该目录路径。

简易设置方法

设置离线使用的最简单方法是使用： hf download 命令去提前下载所有小模型

# Create your local models directory
mkdir -p /path/to/small_models

# Download models from Hugging Face
hf download facebook/w2v-bert-2.0 --local-dir /path/to/small_models/w2v-bert-2.0
hf download funasr/campplus --local-dir /path/to/small_models/campplus
hf download nvidia/bigvgan_v2_22khz_80band_256x --local-dir /path/to/small_models/bigvgan
hf download amphion/MaskGCT --local-dir /path/to/small_models/MaskGCT

最终的目录结构应如下所示：

/path/to/small_models/
├── w2v-bert-2.0/                 # Feature extraction model
├── campplus/                     # Speaker recognition model
├── bigvgan/                      # Vocoder model
└── MaskGCT/                      # Semantic codec model

支持的模型列表

小型模型将按以下方式自动映射：

w2v-bert-2.0 (models--facebook--w2v-bert-2.0) - 特征提取模型
campplus (models--funasr--campplus) - 说话人识别模型
bigvgan (models--nvidia--bigvgan_v2_22khz_80band_256x) - 语音编码器模型
语义编解码器 (models--amphion--MaskGCT) - 语义编码/解码模型

使用离线模式启动IndexTTS2

在通过Web UI启动IndexTTS2时，可添加额外参数：- small_models_dir - 包含所有小型模型的目录路径

在通过命令行启动时，您可以添加以下选项：

xinference launch --model-name IndexTTS2 --model-type audio \
    --small_models_dir /path/to/small_models

使用 Python 客户端启动时：

model_uid = client.launch_model(
    model_name="IndexTTS2",
    model_type="audio",
    small_models_dir="/path/to/small_models"
)