.. _models_llm_index: ===================== Large language Models ===================== The following is a list of built-in LLM in Xinference: .. list-table:: :widths: 25 25 25 50 :header-rows: 1 * - MODEL NAME - ABILITIES - COTNEXT_LENGTH - DESCRIPTION * - :ref:`aquila2 ` - generate - 2048 - Aquila2 series models are the base language models * - :ref:`aquila2-chat ` - chat - 2048 - Aquila2-chat series models are the chat models * - :ref:`aquila2-chat-16k ` - chat - 16384 - AquilaChat2-16k series models are the long-text chat models * - :ref:`baichuan-2 ` - generate - 4096 - Baichuan2 is an open-source Transformer based LLM that is trained on both Chinese and English data. * - :ref:`baichuan-2-chat ` - chat - 4096 - Baichuan2-chat is a fine-tuned version of the Baichuan LLM, specializing in chatting. * - :ref:`c4ai-command-r-v01 ` - chat - 131072 - C4AI Command-R(+) is a research release of a 35 and 104 billion parameter highly performant generative model. * - :ref:`code-llama ` - generate - 100000 - Code-Llama is an open-source LLM trained by fine-tuning LLaMA2 for generating and discussing code. * - :ref:`code-llama-instruct ` - chat - 100000 - Code-Llama-Instruct is an instruct-tuned version of the Code-Llama LLM. * - :ref:`code-llama-python ` - generate - 100000 - Code-Llama-Python is a fine-tuned version of the Code-Llama LLM, specializing in Python. * - :ref:`codegeex4 ` - chat - 131072 - the open-source version of the latest CodeGeeX4 model series * - :ref:`codeqwen1.5 ` - generate - 65536 - CodeQwen1.5 is the Code-Specific version of Qwen1.5. It is a transformer-based decoder-only language model pretrained on a large amount of data of codes. * - :ref:`codeqwen1.5-chat ` - chat - 65536 - CodeQwen1.5 is the Code-Specific version of Qwen1.5. It is a transformer-based decoder-only language model pretrained on a large amount of data of codes. * - :ref:`codeshell ` - generate - 8194 - CodeShell is a multi-language code LLM developed by the Knowledge Computing Lab of Peking University. * - :ref:`codeshell-chat ` - chat - 8194 - CodeShell is a multi-language code LLM developed by the Knowledge Computing Lab of Peking University. * - :ref:`codestral-v0.1 ` - generate - 32768 - Codestrall-22B-v0.1 is trained on a diverse dataset of 80+ programming languages, including the most popular ones, such as Python, Java, C, C++, JavaScript, and Bash * - :ref:`cogvlm2 ` - chat, vision - 8192 - CogVLM2 have achieved good results in many lists compared to the previous generation of CogVLM open source models. Its excellent performance can compete with some non-open source models. * - :ref:`cogvlm2-video-llama3-chat ` - chat, vision - 8192 - CogVLM2-Video achieves state-of-the-art performance on multiple video question answering tasks. * - :ref:`csg-wukong-chat-v0.1 ` - chat - 32768 - csg-wukong-1B is a 1 billion-parameter small language model(SLM) pretrained on 1T tokens. * - :ref:`deepseek ` - generate - 4096 - DeepSeek LLM, trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese. * - :ref:`deepseek-chat ` - chat - 4096 - DeepSeek LLM is an advanced language model comprising 67 billion parameters. It has been trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese. * - :ref:`deepseek-coder ` - generate - 16384 - Deepseek Coder is composed of a series of code language models, each trained from scratch on 2T tokens, with a composition of 87% code and 13% natural language in both English and Chinese. * - :ref:`deepseek-coder-instruct ` - chat - 16384 - deepseek-coder-instruct is a model initialized from deepseek-coder-base and fine-tuned on 2B tokens of instruction data. * - :ref:`deepseek-v2 ` - generate - 128000 - DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. * - :ref:`deepseek-v2-chat ` - chat - 128000 - DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. * - :ref:`deepseek-v2-chat-0628 ` - chat - 128000 - DeepSeek-V2-Chat-0628 is an improved version of DeepSeek-V2-Chat. * - :ref:`deepseek-v2.5 ` - chat - 128000 - DeepSeek-V2.5 is an upgraded version that combines DeepSeek-V2-Chat and DeepSeek-Coder-V2-Instruct. The new model integrates the general and coding abilities of the two previous versions. * - :ref:`deepseek-vl-chat ` - chat, vision - 4096 - DeepSeek-VL possesses general multimodal understanding capabilities, capable of processing logical diagrams, web pages, formula recognition, scientific literature, natural images, and embodied intelligence in complex scenarios. * - :ref:`gemma-2-it ` - chat - 8192 - Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. * - :ref:`gemma-it ` - chat - 8192 - Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. * - :ref:`glm-4v ` - chat, vision - 8192 - GLM4 is the open source version of the latest generation of pre-trained models in the GLM-4 series launched by Zhipu AI. * - :ref:`glm-edge-chat ` - chat - 8192 - The GLM-Edge series is our attempt to face the end-side real-life scenarios, which consists of two sizes of large-language dialogue models and multimodal comprehension models (GLM-Edge-1.5B-Chat, GLM-Edge-4B-Chat, GLM-Edge-V-2B, GLM-Edge-V-5B). Among them, the 1.5B / 2B model is mainly for platforms such as mobile phones and cars, and the 4B / 5B model is mainly for platforms such as PCs. * - :ref:`glm-edge-v ` - chat, vision - 8192 - The GLM-Edge series is our attempt to face the end-side real-life scenarios, which consists of two sizes of large-language dialogue models and multimodal comprehension models (GLM-Edge-1.5B-Chat, GLM-Edge-4B-Chat, GLM-Edge-V-2B, GLM-Edge-V-5B). Among them, the 1.5B / 2B model is mainly for platforms such as mobile phones and cars, and the 4B / 5B model is mainly for platforms such as PCs. * - :ref:`glm4-chat ` - chat, tools - 131072 - GLM4 is the open source version of the latest generation of pre-trained models in the GLM-4 series launched by Zhipu AI. * - :ref:`glm4-chat-1m ` - chat, tools - 1048576 - GLM4 is the open source version of the latest generation of pre-trained models in the GLM-4 series launched by Zhipu AI. * - :ref:`gorilla-openfunctions-v2 ` - chat - 4096 - OpenFunctions is designed to extend Large Language Model (LLM) Chat Completion feature to formulate executable APIs call given natural language instructions and API context. * - :ref:`gpt-2 ` - generate - 1024 - GPT-2 is a Transformer-based LLM that is trained on WebTest, a 40 GB dataset of Reddit posts with 3+ upvotes. * - :ref:`internlm2-chat ` - chat - 32768 - The second generation of the InternLM model, InternLM2. * - :ref:`internlm2.5-chat ` - chat - 32768 - InternLM2.5 series of the InternLM model. * - :ref:`internlm2.5-chat-1m ` - chat - 262144 - InternLM2.5 series of the InternLM model supports 1M long-context * - :ref:`internvl-chat ` - chat, vision - 32768 - InternVL 1.5 is an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. * - :ref:`internvl2 ` - chat, vision - 32768 - InternVL 2 is an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. * - :ref:`llama-2 ` - generate - 4096 - Llama-2 is the second generation of Llama, open-source and trained on a larger amount of data. * - :ref:`llama-2-chat ` - chat - 4096 - Llama-2-Chat is a fine-tuned version of the Llama-2 LLM, specializing in chatting. * - :ref:`llama-3 ` - generate - 8192 - Llama 3 is an auto-regressive language model that uses an optimized transformer architecture * - :ref:`llama-3-instruct ` - chat - 8192 - The Llama 3 instruction tuned models are optimized for dialogue use cases and outperform many of the available open source chat models on common industry benchmarks.. * - :ref:`llama-3.1 ` - generate - 131072 - Llama 3.1 is an auto-regressive language model that uses an optimized transformer architecture * - :ref:`llama-3.1-instruct ` - chat, tools - 131072 - The Llama 3.1 instruction tuned models are optimized for dialogue use cases and outperform many of the available open source chat models on common industry benchmarks.. * - :ref:`llama-3.2-vision ` - generate, vision - 131072 - The Llama 3.2-Vision instruction-tuned models are optimized for visual recognition, image reasoning, captioning, and answering general questions about an image... * - :ref:`llama-3.2-vision-instruct ` - chat, vision - 131072 - Llama 3.2-Vision instruction-tuned models are optimized for visual recognition, image reasoning, captioning, and answering general questions about an image... * - :ref:`llama-3.3-instruct ` - chat, tools - 131072 - The Llama 3.3 instruction tuned models are optimized for dialogue use cases and outperform many of the available open source chat models on common industry benchmarks.. * - :ref:`minicpm-2b-dpo-bf16 ` - chat - 4096 - MiniCPM is an End-Size LLM developed by ModelBest Inc. and TsinghuaNLP, with only 2.4B parameters excluding embeddings. * - :ref:`minicpm-2b-dpo-fp16 ` - chat - 4096 - MiniCPM is an End-Size LLM developed by ModelBest Inc. and TsinghuaNLP, with only 2.4B parameters excluding embeddings. * - :ref:`minicpm-2b-dpo-fp32 ` - chat - 4096 - MiniCPM is an End-Size LLM developed by ModelBest Inc. and TsinghuaNLP, with only 2.4B parameters excluding embeddings. * - :ref:`minicpm-2b-sft-bf16 ` - chat - 4096 - MiniCPM is an End-Size LLM developed by ModelBest Inc. and TsinghuaNLP, with only 2.4B parameters excluding embeddings. * - :ref:`minicpm-2b-sft-fp32 ` - chat - 4096 - MiniCPM is an End-Size LLM developed by ModelBest Inc. and TsinghuaNLP, with only 2.4B parameters excluding embeddings. * - :ref:`minicpm-llama3-v-2_5 ` - chat, vision - 8192 - MiniCPM-Llama3-V 2.5 is the latest model in the MiniCPM-V series. The model is built on SigLip-400M and Llama3-8B-Instruct with a total of 8B parameters. * - :ref:`minicpm-v-2.6 ` - chat, vision - 32768 - MiniCPM-V 2.6 is the latest model in the MiniCPM-V series. The model is built on SigLip-400M and Qwen2-7B with a total of 8B parameters. * - :ref:`minicpm3-4b ` - chat - 32768 - MiniCPM3-4B is the 3rd generation of MiniCPM series. The overall performance of MiniCPM3-4B surpasses Phi-3.5-mini-Instruct and GPT-3.5-Turbo-0125, being comparable with many recent 7B~9B models. * - :ref:`mistral-instruct-v0.1 ` - chat - 8192 - Mistral-7B-Instruct is a fine-tuned version of the Mistral-7B LLM on public datasets, specializing in chatting. * - :ref:`mistral-instruct-v0.2 ` - chat - 8192 - The Mistral-7B-Instruct-v0.2 Large Language Model (LLM) is an improved instruct fine-tuned version of Mistral-7B-Instruct-v0.1. * - :ref:`mistral-instruct-v0.3 ` - chat - 32768 - The Mistral-7B-Instruct-v0.2 Large Language Model (LLM) is an improved instruct fine-tuned version of Mistral-7B-Instruct-v0.1. * - :ref:`mistral-large-instruct ` - chat - 131072 - Mistral-Large-Instruct-2407 is an advanced dense Large Language Model (LLM) of 123B parameters with state-of-the-art reasoning, knowledge and coding capabilities. * - :ref:`mistral-nemo-instruct ` - chat - 1024000 - The Mistral-Nemo-Instruct-2407 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-Nemo-Base-2407 * - :ref:`mistral-v0.1 ` - generate - 8192 - Mistral-7B is a unmoderated Transformer based LLM claiming to outperform Llama2 on all benchmarks. * - :ref:`mixtral-8x22b-instruct-v0.1 ` - chat - 65536 - The Mixtral-8x22B-Instruct-v0.1 Large Language Model (LLM) is an instruct fine-tuned version of the Mixtral-8x22B-v0.1, specializing in chatting. * - :ref:`mixtral-instruct-v0.1 ` - chat - 32768 - Mistral-8x7B-Instruct is a fine-tuned version of the Mistral-8x7B LLM, specializing in chatting. * - :ref:`mixtral-v0.1 ` - generate - 32768 - The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts. * - :ref:`omnilmm ` - chat, vision - 2048 - OmniLMM is a family of open-source large multimodal models (LMMs) adept at vision & language modeling. * - :ref:`openhermes-2.5 ` - chat - 8192 - Openhermes 2.5 is a fine-tuned version of Mistral-7B-v0.1 on primarily GPT-4 generated data. * - :ref:`opt ` - generate - 2048 - Opt is an open-source, decoder-only, Transformer based LLM that was designed to replicate GPT-3. * - :ref:`orion-chat ` - chat - 4096 - Orion-14B series models are open-source multilingual large language models trained from scratch by OrionStarAI. * - :ref:`orion-chat-rag ` - chat - 4096 - Orion-14B series models are open-source multilingual large language models trained from scratch by OrionStarAI. * - :ref:`phi-2 ` - generate - 2048 - Phi-2 is a 2.7B Transformer based LLM used for research on model safety, trained with data similar to Phi-1.5 but augmented with synthetic texts and curated websites. * - :ref:`phi-3-mini-128k-instruct ` - chat - 128000 - The Phi-3-Mini-128K-Instruct is a 3.8 billion-parameter, lightweight, state-of-the-art open model trained using the Phi-3 datasets. * - :ref:`phi-3-mini-4k-instruct ` - chat - 4096 - The Phi-3-Mini-4k-Instruct is a 3.8 billion-parameter, lightweight, state-of-the-art open model trained using the Phi-3 datasets. * - :ref:`platypus2-70b-instruct ` - generate - 4096 - Platypus-70B-instruct is a merge of garage-bAInd/Platypus2-70B and upstage/Llama-2-70b-instruct-v2. * - :ref:`qvq-72b-preview ` - chat, vision - 32768 - QVQ-72B-Preview is an experimental research model developed by the Qwen team, focusing on enhancing visual reasoning capabilities. * - :ref:`qwen-chat ` - chat - 32768 - Qwen-chat is a fine-tuned version of the Qwen LLM trained with alignment techniques, specializing in chatting. * - :ref:`qwen-vl-chat ` - chat, vision - 4096 - Qwen-VL-Chat supports more flexible interaction, such as multiple image inputs, multi-round question answering, and creative capabilities. * - :ref:`qwen1.5-chat ` - chat, tools - 32768 - Qwen1.5 is the beta version of Qwen2, a transformer-based decoder-only language model pretrained on a large amount of data. * - :ref:`qwen1.5-moe-chat ` - chat, tools - 32768 - Qwen1.5-MoE is a transformer-based MoE decoder-only language model pretrained on a large amount of data. * - :ref:`qwen2-audio ` - chat, audio - 32768 - Qwen2-Audio: A large-scale audio-language model which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. * - :ref:`qwen2-audio-instruct ` - chat, audio - 32768 - Qwen2-Audio: A large-scale audio-language model which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. * - :ref:`qwen2-instruct ` - chat, tools - 32768 - Qwen2 is the new series of Qwen large language models * - :ref:`qwen2-moe-instruct ` - chat, tools - 32768 - Qwen2 is the new series of Qwen large language models. * - :ref:`qwen2-vl-instruct ` - chat, vision - 32768 - Qwen2-VL: To See the World More Clearly.Qwen2-VL is the latest version of the vision language models in the Qwen model familities. * - :ref:`qwen2.5 ` - generate - 32768 - Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. * - :ref:`qwen2.5-coder ` - generate - 32768 - Qwen2.5-Coder is the latest series of Code-Specific Qwen large language models (formerly known as CodeQwen). * - :ref:`qwen2.5-coder-instruct ` - chat, tools - 32768 - Qwen2.5-Coder is the latest series of Code-Specific Qwen large language models (formerly known as CodeQwen). * - :ref:`qwen2.5-instruct ` - chat, tools - 32768 - Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. * - :ref:`qwq-32b-preview ` - chat - 32768 - QwQ-32B-Preview is an experimental research model developed by the Qwen Team, focused on advancing AI reasoning capabilities. * - :ref:`seallm_v2 ` - generate - 8192 - We introduce SeaLLM-7B-v2, the state-of-the-art multilingual LLM for Southeast Asian (SEA) languages * - :ref:`seallm_v2.5 ` - generate - 8192 - We introduce SeaLLM-7B-v2.5, the state-of-the-art multilingual LLM for Southeast Asian (SEA) languages * - :ref:`skywork ` - generate - 4096 - Skywork is a series of large models developed by the Kunlun Group · Skywork team. * - :ref:`skywork-math ` - generate - 4096 - Skywork is a series of large models developed by the Kunlun Group · Skywork team. * - :ref:`starling-lm ` - chat - 4096 - We introduce Starling-7B, an open large language model (LLM) trained by Reinforcement Learning from AI Feedback (RLAIF). The model harnesses the power of our new GPT-4 labeled ranking dataset * - :ref:`telechat ` - chat - 8192 - The TeleChat is a large language model developed and trained by China Telecom Artificial Intelligence Technology Co., LTD. The 7B model base is trained with 1.5 trillion Tokens and 3 trillion Tokens and Chinese high-quality corpus. * - :ref:`tiny-llama ` - generate - 2048 - The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. * - :ref:`wizardcoder-python-v1.0 ` - chat - 100000 - * - :ref:`wizardmath-v1.0 ` - chat - 2048 - WizardMath is an open-source LLM trained by fine-tuning Llama2 with Evol-Instruct, specializing in math. * - :ref:`xverse ` - generate - 2048 - XVERSE is a multilingual large language model, independently developed by Shenzhen Yuanxiang Technology. * - :ref:`xverse-chat ` - chat - 2048 - XVERSEB-Chat is the aligned version of model XVERSE. * - :ref:`yi ` - generate - 4096 - The Yi series models are large language models trained from scratch by developers at 01.AI. * - :ref:`yi-1.5 ` - generate - 4096 - Yi-1.5 is an upgraded version of Yi. It is continuously pre-trained on Yi with a high-quality corpus of 500B tokens and fine-tuned on 3M diverse fine-tuning samples. * - :ref:`yi-1.5-chat ` - chat - 4096 - Yi-1.5 is an upgraded version of Yi. It is continuously pre-trained on Yi with a high-quality corpus of 500B tokens and fine-tuned on 3M diverse fine-tuning samples. * - :ref:`yi-1.5-chat-16k ` - chat - 16384 - Yi-1.5 is an upgraded version of Yi. It is continuously pre-trained on Yi with a high-quality corpus of 500B tokens and fine-tuned on 3M diverse fine-tuning samples. * - :ref:`yi-200k ` - generate - 262144 - The Yi series models are large language models trained from scratch by developers at 01.AI. * - :ref:`yi-chat ` - chat - 4096 - The Yi series models are large language models trained from scratch by developers at 01.AI. * - :ref:`yi-coder ` - generate - 131072 - Yi-Coder is a series of open-source code language models that delivers state-of-the-art coding performance with fewer than 10 billion parameters.Excelling in long-context understanding with a maximum context length of 128K tokens.Supporting 52 major programming languages, including popular ones such as Java, Python, JavaScript, and C++. * - :ref:`yi-coder-chat ` - chat - 131072 - Yi-Coder is a series of open-source code language models that delivers state-of-the-art coding performance with fewer than 10 billion parameters.Excelling in long-context understanding with a maximum context length of 128K tokens.Supporting 52 major programming languages, including popular ones such as Java, Python, JavaScript, and C++. * - :ref:`yi-vl-chat ` - chat, vision - 4096 - Yi Vision Language (Yi-VL) model is the open-source, multimodal version of the Yi Large Language Model (LLM) series, enabling content comprehension, recognition, and multi-round conversations about images. .. toctree:: :maxdepth: 3 aquila2 aquila2-chat aquila2-chat-16k baichuan-2 baichuan-2-chat c4ai-command-r-v01 code-llama code-llama-instruct code-llama-python codegeex4 codeqwen1.5 codeqwen1.5-chat codeshell codeshell-chat codestral-v0.1 cogvlm2 cogvlm2-video-llama3-chat csg-wukong-chat-v0.1 deepseek deepseek-chat deepseek-coder deepseek-coder-instruct deepseek-v2 deepseek-v2-chat deepseek-v2-chat-0628 deepseek-v2.5 deepseek-vl-chat gemma-2-it gemma-it glm-4v glm-edge-chat glm-edge-v glm4-chat glm4-chat-1m gorilla-openfunctions-v2 gpt-2 internlm2-chat internlm2.5-chat internlm2.5-chat-1m internvl-chat internvl2 llama-2 llama-2-chat llama-3 llama-3-instruct llama-3.1 llama-3.1-instruct llama-3.2-vision llama-3.2-vision-instruct llama-3.3-instruct minicpm-2b-dpo-bf16 minicpm-2b-dpo-fp16 minicpm-2b-dpo-fp32 minicpm-2b-sft-bf16 minicpm-2b-sft-fp32 minicpm-llama3-v-2_5 minicpm-v-2.6 minicpm3-4b mistral-instruct-v0.1 mistral-instruct-v0.2 mistral-instruct-v0.3 mistral-large-instruct mistral-nemo-instruct mistral-v0.1 mixtral-8x22b-instruct-v0.1 mixtral-instruct-v0.1 mixtral-v0.1 omnilmm openhermes-2.5 opt orion-chat orion-chat-rag phi-2 phi-3-mini-128k-instruct phi-3-mini-4k-instruct platypus2-70b-instruct qvq-72b-preview qwen-chat qwen-vl-chat qwen1.5-chat qwen1.5-moe-chat qwen2-audio qwen2-audio-instruct qwen2-instruct qwen2-moe-instruct qwen2-vl-instruct qwen2.5 qwen2.5-coder qwen2.5-coder-instruct qwen2.5-instruct qwq-32b-preview seallm_v2 seallm_v2.5 skywork skywork-math starling-lm telechat tiny-llama wizardcoder-python-v1.0 wizardmath-v1.0 xverse xverse-chat yi yi-1.5 yi-1.5-chat yi-1.5-chat-16k yi-200k yi-chat yi-coder yi-coder-chat yi-vl-chat