音频（实验性质）#

学习如何使用 Xinference 将音频转换为文本或将文本转换为音频。

介绍#

Audio API提供了三种与音频交互的方法：

转录终端将音频转录为输入语言。
翻译端点将音频转换为英文。
转录终端将音频转录为输入语言。

API 端点	OpenAI 兼容端点
Transcription API	/v1/audio/transcriptions
Translation API	/v1/audio/translations
Speech API	/v1/audio/speech

支持的模型列表#

在Xinference中，以下模型支持音频API：

语音转文本#

whisper-tiny
whisper-tiny.en
whisper-base
whisper-base.en
whisper-medium
whisper-medium.en
whisper-large-v3
whisper-large-v3-turbo
Belle-distilwhisper-large-v2-zh
Belle-whisper-large-v2-zh
Belle-whisper-large-v3-zh
SenseVoiceSmall

文本转语音#

ChatTTS
CosyVoice

快速入门#

转录#

Transcription API 模仿了 OpenAI 的 create transcriptions API。你可以通过 cURL、OpenAI Client 或者 Xinference 的 Python 客户端来尝试 Transcription API：

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/transcriptions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "file": "<audio bytes>",
  }'

import openai

client = openai.Client(
    api_key="cannot be empty",
    base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
)
with open("speech.mp3", "rb") as audio_file:
    client.audio.transcriptions.create(
        model=<MODEL_UID>,
        file=audio_file,
    )

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
with open("speech.mp3", "rb") as audio_file:
    model.transcriptions(audio=audio_file.read())

{
  "text": "Imagine the wildest idea that you've ever had, and you're curious about how it might scale to something that's a 100, a 1,000 times bigger. This is a place where you can get to do that."
}

翻译#

Translation API 模仿了 OpenAI 的 create translations API。你可以通过 cURL、OpenAI Client 或 Xinference 的 Python 客户端来尝试使用 Translation API：

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/translations' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "file": "<audio bytes>",
  }'

import openai

client = openai.Client(
    api_key="cannot be empty",
    base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
)
with open("speech.mp3", "rb") as audio_file:
    client.audio.translations.create(
        model=<MODEL_UID>,
        file=audio_file,
    )

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
with open("speech.mp3", "rb") as audio_file:
    model.translations(audio=audio_file.read())

{
  "text": "Hello, my name is Wolfgang and I come from Germany. Where are you heading today?"
}

语音#

Transcription API 模仿了 OpenAI 的 create speech API。你可以通过 cURL、OpenAI Client 或者 Xinference 的 Python 客户端来尝试 Speech API：

Speech API 默认使用非流式

ChatTTS 的流式输出不如非流式的效果好，参考：2noise/ChatTTS#564
流式要求 ffmpeg<7：https://pytorch.org/audio/stable/installation.html#optional-dependencies

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "<The text to generate audio for>",
    "voice": "echo",
    "stream": True,
  }'

import openai

client = openai.Client(
    api_key="cannot be empty",
    base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
)
client.audio.speech.create(
    model=<MODEL_UID>,
    input=<The text to generate audio for>,
    voice="echo",
)

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
model.speech(
    input=<The text to generate audio for>,
    voice="echo",
    stream: True,
)

The output will be an audio binary.

ChatTTS 使用#

基本使用，参考语音使用章节。

固定音色。我们可以使用由 6drf21e/ChatTTS_Speaker 提供的固定音色，下载 evaluation_result.csv ，以 seed_2155 音色作为例子，我们使用 emb_data 列的数据。

import pandas as pd

df = pd.read_csv("evaluation_results.csv")
emb_data_2155 = df[df['seed_id'] == 'seed_2155'].iloc[0]["emb_data"]

使用 seed_2155 固定音色来创建语音。

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
resp_bytes = model.speech(
    voice=emb_data_2155,
    input=<The text to generate audio for>
)

CosyVoice 模型使用#

基本使用，加载模型 CosyVoice-300M-SFT。

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "input": "<The text to generate audio for>",
    # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
    "voice": "中文女"
  }'

import openai

client = openai.Client(
    api_key="cannot be empty",
    base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
)
response = client.audio.speech.create(
    model=<MODEL_UID>,
    input=<The text to generate audio for>,
    # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
    voice="中文女",
)
response.stream_to_file('1.mp3')

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")
speech_bytes = model.speech(
    input=<The text to generate audio for>,
    # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
    voice="中文女"
)
with open('1.mp3', 'wb') as f:
    f.write(speech_bytes)

克隆声音，加载模型 CosyVoice-300M。

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

zero_shot_prompt_text = ""
# The zero shot prompt file is the voice file
# the words said in the file shoule be identical to zero_shot_prompt_text
with open(zero_shot_prompt_file, "rb") as f:
    zero_shot_prompt = f.read()

speech_bytes = model.speech(
    "<The text to generate audio for>",
    prompt_text=zero_shot_prompt_text,
    prompt_speech=zero_shot_prompt,
)

跨语言使用，加载模型 CosyVoice-300M。

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

# the file that reads in some language
with open(cross_lingual_prompt_file, "rb") as f:
    cross_lingual_prompt = f.read()

speech_bytes = model.speech(
    "<The text to generate audio for>",  # text could be another language
    prompt_speech=cross_lingual_prompt,
)

基于指令的声音合成，加载模型 CosyVoice-300M-Instruct。

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

model = client.get_model("<MODEL_UID>")

response = model.speech(
    "在面对挑战时，他展现了非凡的<strong>勇气</strong>与<strong>智慧</strong>。",
    voice="中文男",
    instruct_text="Theo 'Crimson', is a fiery, passionate rebel leader. "
    "Fights with fervor for justice, but struggles with impulsiveness.",
)

更多指令和例子，可以参考 https://fun-audio-llm.github.io/ 。