Example: CLI chatbot 🤖️#

Description:

Demonstrate how to interact with Xinference to play with LLM chat functionality with an AI agent in command line💻

Used Technology:

@ ggerganov ‘s ggml

@ Xinference as a launcher

@ All LLaMA and Chatglm models supported by Xorbitsio inference

Detailed Explanation on the Demo Functionality :

  1. Take the user command line input in the terminal and grab the required parameters for model launching.

  2. Launch the Xinference frameworks and automatically deploy the model user demanded into the cluster.

  3. Initialize an empty chat history to store all the context in the chatroom.

  4. Recursively ask for user’s input as prompt and let the model to generate response based on the prompt and the chat history. Show the Output of the response in the terminal.

  5. Store the user’s input and agent’s response into the chat history as context for the upcoming rounds.

Source Code :