.. _index: ====================== Welcome to Xinference! ====================== .. toctree:: :maxdepth: 2 :hidden: getting_started/index models/index user_guide/index examples/index reference/index development/index Xorbits Inference (Xinference) is an open-source platform to streamline the operation and integration of a wide array of AI models. With Xinference, you're empowered to run inference using any open-source LLMs, embedding models, and multimodal models either in the cloud or on your own premises, and create robust AI-driven applications. Developing Real-world AI Applications with Xinference ----------------------------------------------------- .. tabs:: .. code-tab:: python LLM from xinference.client import Client client = Client("http://localhost:9997") model = client.get_model("MODEL_UID") # Chat to LLM model.chat( messages=[{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "What is the largest animal?"}], generate_config={"max_tokens": 1024} ) # Chat to VL model model.chat( messages=[ { "role": "user", "content": [ {"type": "text", "text": "What’s in this image?"}, { "type": "image_url", "image_url": { "url": "http://i.epochtimes.com/assets/uploads/2020/07/shutterstock_675595789-600x400.jpg", }, }, ], } ], generate_config={"max_tokens": 1024} ) .. code-tab:: python Embedding from xinference.client import Client client = Client("http://localhost:9997") model = client.get_model("MODEL_UID") model.create_embedding("What is the capital of China?") .. code-tab:: python Image from xinference.client import Client client = Client("http://localhost:9997") model = client.get_model("MODEL_UID") model.text_to_image("An astronaut walking on the mars") .. code-tab:: python Audio from xinference.client import Client client = Client("http://localhost:9997") model = client.get_model("MODEL_UID") with open("speech.mp3", "rb") as audio_file: model.transcriptions(audio_file.read()) .. code-tab:: python Rerank from xinference.client import Client client = Client("http://localhost:9997") model = client.get_model("MODEL_UID") query = "A man is eating pasta." corpus = [ "A man is eating food.", "A man is eating a piece of bread.", "The girl is carrying a baby.", "A man is riding a horse.", "A woman is playing violin." ] print(model.rerank(corpus, query)) .. code-tab:: python Video from xinference.client import Client client = Client("http://localhost:9997") model = client.get_model("MODEL_UID") model.text_to_video("") Getting Started --------------- .. grid:: 2 .. grid-item-card:: Install Xinference :link: installation :link-type: ref Install Xinference on Linux, Windows, and macOS. .. grid-item-card:: Try it out! :link: using_xinference :link-type: ref Start by running Xinference on a local machine. .. grid:: 2 .. grid-item-card:: Explore models :link: models_builtin_index :link-type: ref Explore a wide range of models supported by Xinference. .. grid-item-card:: Register your own model :link: models_custom :link-type: ref Register model weights and turn it into an API. Explore the API --------------- .. grid:: 2 .. grid-item-card:: Chat & Generate :link: chat :link-type: ref Learn how to chat with LLMs in Xinference. .. grid-item-card:: Tools :link: tools :link-type: ref Learn how to connect LLM with external tools. .. grid:: 2 .. grid-item-card:: Embeddings :link: embed :link-type: ref Learn how to create text embeddings in Xinference. .. grid-item-card:: Rerank :link: rerank :link-type: ref Learn how to use rerank models in Xinference. .. grid:: 2 .. grid-item-card:: Images :link: image :link-type: ref Learn how to generate images with Xinference. .. grid-item-card:: Vision :link: vision :link-type: ref Learn how to process image with LLMs. .. grid:: 2 .. grid-item-card:: Audio :link: audio :link-type: ref Learn how to turn audio into text or text into audio with Xinference. .. grid-item-card:: Video :link: video :link-type: ref Learn how to generate video with Xinference. Getting Involved ---------------- .. grid:: :gutter: 1 .. grid-item:: .. div:: sd-font-weight-normal sd-fs-5 Get Latest News .. grid:: 1 :gutter: 3 .. grid-item-card:: :link: https://twitter.com/Xorbitsio :fab:`twitter` Follow us on Twitter .. grid-item-card:: :link: https://zhihu.com/org/xorbits :fab:`zhihu` Read our blogs .. grid-item:: .. div:: sd-font-weight-normal sd-fs-5 Get Support .. grid:: 1 :gutter: 3 .. grid-item-card:: :link: https://xorbits.cn/assets/images/wechat_work_qr.png :fab:`weixin` Find community on WeChat .. grid-item-card:: :link: https://join.slack.com/t/xorbitsio/shared_invite/zt-1o3z9ucdh-RbfhbPVpx7prOVdM1CAuxg :fab:`slack` Find community on Slack .. grid-item-card:: :link: https://github.com/xorbitsai/inference/issues/new/choose :fab:`github` Open an issue .. grid-item:: .. div:: sd-fs-5 Contribute to Xinference .. grid:: 1 :gutter: 3 .. grid-item-card:: :link: https://github.com/xorbitsai/inference/pulls :fab:`github` Create a pull request