.. _video: ==================== Video (Experimental) ==================== Learn how to generate videos with Xinference. Introduction ================== The Video API provides the ability to interact with videos: * The text-to-video endpoint create videos from scratch based on a text prompt. * The image-to-video endpoint create videos from scratch based on an input image. * The firstlastframe-to-video endpoint creates videos based on the transition between a first and a last frame. .. list-table:: :widths: 25 50 :header-rows: 1 * - API - Endpoint * - Text-to-Video API - /v1/video/generations * - Image-to-Video API - /v1/video/generations/image * - FirstLastFrame-to-Video API - /v1/video/generations/flf Supported models ------------------- The text-to-video API is supported with the following models in Xinference: * :ref:`MiniMax-H3 ` * :ref:`CogVideoX-2b ` * :ref:`CogVideoX-5b ` * :ref:`HunyuanVideo ` * :ref:`Wan2.1-1.3B ` * :ref:`Wan2.1-14B ` The image-to-video API is supported with the following models in Xinference: * :ref:`MiniMax-H3 ` * :ref:`Wan2.1-i2v-14B-480p ` * :ref:`Wan2.1-i2v-14B-720p ` The firstlastframe-to-video API is supported with the following models in Xinference: * :ref:`MiniMax-H3 ` * :ref:`Wan2.1-flf2v-14B-720p ` Quickstart =================== Text-to-video -------------------- You can try text-to-video API out either via cURL, or Xinference's python client: .. tabs:: .. code-tab:: bash cURL curl -X 'POST' \ 'http://:/v1/video/generations' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "model": "", "prompt": "" }' .. code-tab:: python Xinference Python Client from xinference.client import Client client = Client("http://:") model = client.get_model("") input_text = "an apple" model.text_to_video(input_text) Image-to-video -------------------- You can try image-to-video API out either via cURL, or Xinference's python client: .. tabs:: .. code-tab:: bash cURL curl -X 'POST' \ 'http://:/v1/video/generations/image' \ -F model= \ -F image=@xxx.jpg \ -F prompt= .. code-tab:: python Xinference Python Client from xinference.client import Client client = Client("http://:") model = client.get_model("") with open("xxx.jpg", "rb") as f: prompt = "" model.image_to_video(image=f.read(), prompt=prompt) FirstLastFrame-to-video -------------------------- You can try firstlastframe-to-video API out either via cURL, or Xinference's python client: .. tabs:: .. code-tab:: bash cURL curl -X 'POST' \ 'http://:/v1/video/generations/flf' \ -F model= \ -F first_frame=@xxx.jpg \ -F last_frame=@xxx2.jpg \ -F prompt= .. code-tab:: python Xinference Python Client from xinference.client import Client client = Client("http://:") model = client.get_model("") with open("xxx.jpg", "rb") as f1, open("xxx2.jpg", "rb") as f2: prompt = "" model.flf_to_video(first_frame=f1.read(), last_frame=f2.read(), prompt=prompt) Memory optimization =================== Video generation will occupy huge GPU memory, for instance, running CogVideoX may require up to around 35 GB GPU memory. Xinference supports several options to optimize video model memory (VRAM) usage. * CPU offloading or block level group offloading. * Layerwise casting. * Weight quantization. .. note:: CPU offloading and Block Level Group Offloading cannot be enabled at the same time, but layerwise casting can be used in combination with either of them. CPU offloading -------------------- CPU offloading keeps the model weights on the CPU and only loads them to the GPU when a forward pass needs to be executed. It is suitable for scenarios with extremely limited GPU memory, but it has a significant impact on performance. When running on GPU whose memory is less than 24 GB, we recommend to add ``--cpu_offload True`` when launching model. For Web UI, add an extra option, ``cpu_offload`` with value set to ``True``. .. code-block:: bash xinference launch --model-name Wan2.1-i2v-14B-480p --model-type video --cpu_offload True Block Level Group Offloading ------------------------------- Block Level Group Offloading groups multiple internal layers of the model (such as ``torch.nn.ModuleList`` or ``torch.nn.Sequential``) and loads these groups from the CPU to the GPU as needed during inference. Compared to CPU offloading, it uses more memory but has less impact on performance. For the command line, add the ``--group_offload True`` option; for the Web UI, add an additional option ``group_offload`` with the value set to ``True``. We can speed up group offloading inference, by enabling the use of CUDA streams. However, using CUDA streams requires moving the model parameters into pinned memory. This allocation is handled by Pytorch under the hood, and can result in a significant spike in CPU RAM usage. Please consider this option if your CPU RAM is atleast 2X the size of the model you are group offloading. Enable CUDA streams via adding ``--use_stream True`` for command line; for the Web UI, add an additional option ``use_stream`` with the value set to ``True``. .. code-block:: bash xinference launch --model-name Wan2.1-i2v-14B-480p --model-type video --group_offload True --use_stream True Weight quantization ------------------- Some video models support weight-only quantization through the ``--quantization`` option. Quantization lowers both GPU and host memory usage, with a possible quality and performance trade-off. MiniMax-H3 supports the following values for ``quantization``: * ``int4``: the default. Most large linear weights use TorchAO INT4, while a few BF16 blocks remain on the CPU during loading. Together with block-level group offloading, this allows the model to load on a 24GB consumer GPU without additional launch options. CUDA streams are disabled on this path to avoid an extra pinned host-memory copy. * ``int8``: use TorchAO INT8 weight-only quantization for higher weight precision. This requires at least 75GB of available host RAM. * ``none`` or ``bf16``: disable weight quantization. With the default ``torch_dtype``, weights are loaded in BF16 and require substantially more GPU and host memory. * ``torchao``: a compatibility alias for ``int8``. Use ``int8`` in new launch configurations. For example, select INT8 with:: xinference launch --model-name MiniMax-H3 --model-type video \ --quantization int8 Applying Layerwise Casting to the Transformer ------------------------------------------------ Layerwise casting will downcast each layer’s weights to ``torch.float8_e4m3fn``, temporarily upcast to ``torch.bfloat16`` during the forward pass of the layer, then revert to ``torch.float8_e4m3fn`` afterward. This approach reduces memory requirements by approximately 50% while introducing a minor quality reduction in the generated video due to the precision trade-off. Enable layerwise casting via adding ``--layerwise_cast True`` for command line; for the Web UI, add an additional option ``layerwise_cast`` with the value set to ``True``. This example will require 20GB of VRAM. .. code-block:: bash xinference launch --model-name Wan2.1-i2v-14B-480p --model-type video --layerwise_cast True --cpu_offload True