xinference.client.handlers.ChatglmCppChatModelHandle.chat

ChatglmCppChatModelHandle.chat(prompt: str, system_prompt: str | None = None, chat_history: List[ChatCompletionMessage] | None = None, tools: List[Dict] | None = None, generate_config: ChatglmCppGenerateConfig | None = None) → ChatCompletion | Iterator[ChatCompletionChunk]

Given a list of messages comprising a conversation, the ChatGLM model will return a response via RESTful APIs.

Parameters:

• prompt (str) – The user’s input.

• system_prompt (Optional[str]) – The system context provide to Model prior to any chats.

• chat_history (Optional[List["ChatCompletionMessage"]]) – A list of messages comprising the conversation so far.

• tools (Optional[List[Dict]]) – A tool list.

• generate_config (Optional["ChatglmCppGenerateConfig"]) – Additional configuration for ChatGLM chat generation.

Returns:

Stream is a parameter in generate_config. When stream is set to True, the function will return Iterator[“ChatCompletionChunk”]. When stream is set to False, the function will return “ChatCompletion”.

Return type:

Union[“ChatCompletion”, Iterator[“ChatCompletionChunk”]]

Raises:

RuntimeError – Report the failure to generate the chat from the server. Detailed information provided in error message.