视频#

学习如何使用 Xinference 生成视频

介绍#

Video API 提供了和视频交互的方式：

  • Text-to-video 端点将一段文本提示词从头开始创建视频

  • Image-to-video 端点将一张图片从头开始创建视频

  • firstlastframe-to-video 接口根据首帧和尾帧之间的过渡生成视频。

API

Endpoint

Text-to-Video API

/v1/video/generations

Image-to-Video API

/v1/video/generations/image

FirstLastFrame-to-Video API

/v1/video/generations/flf

支持的模型列表#

Text-to-video API 在 Xinference 中支持以下模型：

Image-to-video API 在 Xinference 中支持以下模型：

Xinference 中支持以下模型使用 firstlastframe-to-video 接口：

Video engines#

Video runtimes are selected with --model-engine. The diffusers engine remains the default for models that expose both runtimes. For example, launch Wan2.2-A14B with either engine explicitly:

xinference launch --model-name Wan2.2-A14B --model-type video --model-engine diffusers
xinference launch --model-name Wan2.2-A14B --model-type video --model-engine MLX

The MLX engine uses Blaizzy/mlx-video and is available on Apple Silicon with Python 3.11 or newer. It supports Wan2.1 T2V, Wan2.2 T2V/I2V/TI2V, and the LTX-2/LTX-2.3 distilled and dev models listed above. The LTX models are MLX-only in Xinference.

Wan2.1's official checkpoints are converted to the native MLX layout on first load and the converted copy is reused by later launches. This first launch therefore needs additional time and disk space. Wan2.2 and LTX use pre-converted checkpoints.

The Web UI obtains the engines supported by each model from the engine query API and presents them in the launch dialog. Additional video runtimes can be registered independently without changing the Video API.

快速入门#

文生视频#

可以通过 cURL 或 Xinference 的方式尝试使用 text-to-video API

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/video/generations' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<MODEL_UID>",
    "prompt": "<your prompt>"
  }'

图生视频#

可以通过 cURL 或 Xinference 的方式尝试使用 image-to-video API

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/video/generations/image' \
  -F model=<MODEL_UID> \
  -F image=@xxx.jpg \
  -F prompt=<prompt>

首尾帧生视频#

你可以通过 cURL 或 Xinference 的 Python 客户端来体验 firstlastframe-to-video 接口：

curl -X 'POST' \
  'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/video/generations/flf' \
  -F model=<MODEL_UID> \
  -F first_frame=@xxx.jpg \
  -F last_frame=@xxx2.jpg \
  -F prompt=<prompt>

Lightning LoRA acceleration#

Lightning LoRA checkpoints distill a video model into fewer denoising steps. Select a supported version with --lightning_version when launching the model; Xinference downloads the LoRA, applies its training alpha and scheduler shifts, and uses the version's recommended inference-step count when the request does not override num_inference_steps. Lightning reduces denoising time, but does not reduce model size or peak memory; MiniMax-H3's default INT4 quantization and group offload remain enabled.

Model

Lightning version

Evaluations

Video shift

Recommended canvas

MiniMax-H3

4step_v0.1

4

12

544p mixed aspect ratios

MiniMax-H3

8step_v1.0_bf16

8

12

544p mixed aspect ratios

MiniMax-H3

4step_v1.0_768p_bf16

4

6

1344x768

In the Web UI, open the MiniMax-H3 launch dialog, expand Advanced Configuration, and select a value under Lightning Versions. Leave Lightning Model Path empty to download the selected checkpoint automatically. After the model starts, set Inference Steps on the video generation page to the evaluation count in the table. The generation page currently starts with 25 steps, which overrides the Lightning default if left unchanged.

For example, launch the 768p four-step version from the command line:

xinference launch --model-name MiniMax-H3 --model-type video \
    --lightning_version 4step_v1.0_768p_bf16

Then generate with four inference steps. MiniMax-H3 outputs at a fixed 24 FPS; 124 frames produce a video of about five seconds:

from xinference.client import Client

client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
model = client.get_model("<MODEL_UID>")
model.text_to_video(
    prompt="A running cat",
    width=1344,
    height=768,
    num_frames=124,
    fps=24,
    num_inference_steps=4,
)

Xinference downloads the Lightning checkpoint from the same hub selected for the base model. Both Hugging Face and ModelScope are supported. To use an already downloaded checkpoint, pass both its path and version:

xinference launch --model-name MiniMax-H3 --model-type video \
    --lightning_version 4step_v0.1 \
    --lightning_model_path /path/to/minimax_h3_fl2v_turbo_4step_v0.1.safetensors

num_inference_steps represents actual transformer evaluations and remains a per-request override. Match it to the selected Lightning version: use 4 for a 4step checkpoint and 8 for an 8step checkpoint. MiniMax-H3's scheduler internally adds the terminal sigma grid point required to run that number of evaluations.

备注

The evaluation-count semantics apply to MiniMax-H3 with or without Lightning. A request for N evaluations now passes N + 1 scheduler grid points so the terminal sigma does not consume one of the requested evaluations. Therefore, non-Lightning output may differ from earlier Xinference versions for the same num_inference_steps value.

内存优化#

视频生成会占用大量显存，举例来说，运行 CogVideoX 可能会使用到约 35 GB 的显存。

Xinference 支持若干选项，来优化视频模型显存（VRAM）使用。

  • CPU 卸载或块级分组卸载。

  • 逐层类型转换（Layerwise casting）。

  • Weight quantization.

备注

CPU 卸载和块级分组卸载不能同时开启，但逐层类型转换可以与其中之一配合使用。

CPU 卸载#

CPU 卸载会将模型权重保留在 CPU 上，仅在执行前向传播时才加载到 GPU。适用于显存极其有限的场景，但对性能影响较大。

当使用显存小于 24 GB 的 GPU 时，建议在启动模型时添加 --cpu_offload True。对于 Web UI，可添加额外选项 cpu_offload，值设为 True。

xinference launch --model-name Wan2.1-i2v-14B-480p --model-type video --cpu_offload True

块级分组卸载#

块级分组卸载将模型的多个内部层（如 torch.nn.ModuleList 或 torch.nn.Sequential）分组，并根据需要在推理过程中将这些分组从 CPU 加载到 GPU。与 CPU 卸载相比，它使用更多的内存，但对性能的影响更小。

对于命令行，添加 --group_offload True 选项；对于 Web UI，添加一个额外选项 group_offload，值设为 True。

通过启用 CUDA 流，我们可以加速分组卸载推理。然而，使用 CUDA 流需要将模型参数移动到固定内存中。这项分配由 Pytorch 在后台处理，并可能导致 CPU RAM 使用量显著增加。如果您的 CPU RAM 至少是模型大小的两倍，请考虑使用此选项。通过在命令行中添加 --use_stream True 启用 CUDA 流；对于 Web UI，添加一个额外选项 use_stream，值设为 True。

xinference launch --model-name Wan2.1-i2v-14B-480p --model-type video --group_offload True --use_stream True

Weight quantization#

Some video models support weight-only quantization through the --quantization option. Quantization lowers both GPU and host memory usage, with a possible quality and performance trade-off.

MiniMax-H3 supports the following values for quantization:

  • int4: the default. Most large linear weights use TorchAO INT4, while a few BF16 blocks remain on the CPU during loading. Together with block-level group offloading, this allows the model to load on a 24GB consumer GPU without additional launch options. CUDA streams are disabled on this path to avoid an extra pinned host-memory copy.

  • int8: use TorchAO INT8 weight-only quantization for higher weight precision. This requires at least 75GB of available host RAM.

  • none or bf16: disable weight quantization. With the default torch_dtype, weights are loaded in BF16 and require substantially more GPU and host memory.

  • torchao: a compatibility alias for int8. Use int8 in new launch configurations.

For example, select INT8 with:

xinference launch --model-name MiniMax-H3 --model-type video \
    --quantization int8

将逐层类型转换应用于 Transformer#

逐层类型转换将把每个层的权重降级为 torch.float8_e4m3fn，在层的前向传播过程中暂时升级为 torch.bfloat16，然后在之后恢复为 torch.float8_e4m3fn。这种方法将内存需求减少约 50%，同时由于精度折衷，生成的视频质量会略有下降。通过在命令行中添加 --layerwise_cast True 来启用逐层类型转换；对于 Web UI，添加一个额外选项 layerwise_cast，值设为 True。

此示例将需要 20GB 的显存。

xinference launch --model-name Wan2.1-i2v-14B-480p --model-type video --layerwise_cast True --cpu_offload True