orichain.llm

class orichain.llm.LLM(**kwds: Any)

Bases: object

Synchronous Language Model class for interacting with various LLM providers.

This class provides a unified interface to interact with different language models from providers such as OpenAI, AWS Bedrock, Google Gemini and Vertex AI, Anthropic, and Azure OpenAI.

default_model = 'gpt-5-mini'

default_model_provider = 'OpenAI'

__init__(**kwds: Any) → None

Initialize the Language Model class with the required parameters.

Args:

• model_name (str, optional): Name of the model to be used. Default: “gpt-4.1-mini”

• provider (str, optional): Name of the model provider. Default: “OpenAI”. Allowed values:

  • OpenAI

  • AzureOpenAI

  • AWSBedrock

  • GoogleGemini

  • GoogleVertexAI

  • AnthropicBedrock

  • Anthropic

  • TogetherAI

Authentication Arguments by provider:

OpenAI models:

• api_key (str): OpenAI API key.

• timeout (Timeout, optional): Request timeout parameter like connect, read, write. Default: 60.0, 5.0, 10.0, 2.0

• max_retries (int, optional): Number of retries for the request. Default: 2

AWS Bedrock models:

• aws_access_key (str): AWS access key.

• aws_secret_key (str): AWS secret key.

• aws_region (str): AWS region name.

• prompt_caching (bool, optional): Whether to use prompt caching. Default: True

• config (botocore.config.Config, optional):

  • connect_timeout (float or int, optional): The time in seconds till a timeout exception is thrown when attempting to make a connection. Default: 60

  • read_timeout: (float or int, optional): The time in seconds till a timeout exception is thrown when attempting to read from a connection. Default: 60

  • region_name (str, optional): region name Note: If specifing config you need to still pass region_name even if you have already passed in aws_region

  • max_pool_connections: The maximum number of connections to keep in a connection pool. Defualt: 10

  • retries (Dict, optional):

    • total_max_attempts: Number of retries for the request. Default: 2

Google Gemini models:

• api_key (str): Gemini API key

• http_options (types.HttpOptions, optional): HTTP options to be used in each of the requests. Default is None

• debug_config (DebugConfig, optional): Configuration options that change client network behavior when testing. Default is None

Google Vertex AI models:

• api_key (str): Vertex AI API key

• credentials (google.auth.credentials.Credentials): The credentials to use for authentication when calling the Vertex AI APIs.

• project (str): The Google Cloud project ID to use for quota.

• location (str): The location to send API requests to (for example, us-central1).

• http_options (types.HttpOptions, optional): HTTP options to be used in each of the requests. Default is None

• debug_config (DebugConfig, optional): Configuration options that change client network behavior when testing. Default is None

Anthropic models:

• api_key (str): Anthropic API key.

• timeout (Timeout, optional): Request timeout parameter like connect, read, write. Default: 60.0, 5.0, 10.0, 2.0

• max_retries (int, optional): Number of retries for the request. Default: 2

• prompt_caching (bool, optional): Whether to use prompt caching. Default: True

Azure OpenAI models:

• api_key (str): Azure OpenAI API key.

• azure_endpoint (str): Azure OpenAI endpoint.

• api_version (str): Azure OpenAI API version.

• timeout (Timeout, optional): Request timeout parameter like connect, read, write. Default: 60.0, 5.0, 10.0, 2.0

• max_retries (int, optional): Number of retries for the request. Default: 2

TogetherAI models:

• api_key (str): TogetherAI API key.

• timeout (float or int, optional): Request timeout in seconds. Default: 60

• max_retries (int, optional): Number of retries for the request. Default: 2

Raises:

• ValueError: If an unsupported model is specified.

• KeyError: If required parameters are not provided.

• TypeError: If an invalid type is provided for a parameter.

Warns:

• UserWarning: If the model name is not provided, it defaults to the default model.

__call__(user_message: str, matched_sentence: List[str] | None = None, system_prompt: str | None = None, chat_hist: List[Dict[str, str]] | None = None, sampling_paras: Dict | None = None, tools: List[Dict] | None = None, tool_choice: str | None = None, extra_metadata: Dict | None = None, do_json: bool = False, **kwds: Any) → Dict

Generate a synchronous response from the language model.

Args:

• user_message (str): The user’s input message.

• system_prompt (str, optional): System prompt to guide the model’s behavior.

• chat_hist (List[Dict[str, str]], optional): Chat history for context.

• sampling_paras (Dict, optional): Parameters for sampling (temperature, top_p, etc.).

• model_name (str, optional): Specifies the model to use. If not provided, the default is the model set during class instantiation.

• do_json (bool, optional): Whether to return a JSON response. Default: False.

• tools (List[Dict], optional): List of tools to be used by the model. Example format

  [{“name”: “tool name”, “description”: “tool description”, “parameters”: {“type”: “object”, “properties”: {“arg_1”: {“type”: “string”, “description”: “An example argument for the tool.”}}, “required”: [“arg_1”]}}, …..]

• tool_choice (str, optional): Defines tool usage:

  • “auto” (default) lets the model decide

  • “none” disables tools (not supported on AWSBedrock/AnthropicBedrock and TogetherAI)

  • “required” forces tool use (unsupported on AzureOpenAI < 2024-06-01)

  • provide a tool name to call it directly.

• matched_sentence (List[str], optional): A list of matched text chunks for context. Not used internally, but included in the response under the matched_sentence key.

• extra_metadata (Dict, optional): Additional metadata to be included in the response.

Generation Arguments by provider:

AWS Bedrock models:

• additional_model_fields (Dict, optional): additionalModelRequestFields passed to the client in the request body.

Google Gemini & Vertex AI models:

• config (google.genai.types.GenerateContentConfig, optional): Optional model configuration parameters provided to the client.chats.create API.

• response_mime_type (str, optional): Output response mimetype of the generated candidate text. Supported mimetype: “text/plain” (Default), “application/json” (if do_json=True)

Anthropic & AnthropicBedrock models:

• timeout (httpx.Timeout, optional): - timeout (httpx.Timeout, optional): Request timeout parameter like connect, read, write. Default is 60.0, 5.0, 10.0, 2.0

Returns:

Dict: The model’s response with tool calls and metadata.

stream(user_message: str, matched_sentence: List[str] | None = None, system_prompt: str | None = None, chat_hist: List | None = None, sampling_paras: Dict | None = None, tools: List[Dict] | None = None, tool_choice: str | None = None, extra_metadata: Dict | None = None, do_json: bool = False, do_sse: bool = True, **kwds: Any) → Generator

Stream responses from the language model.

Args:

• user_message (str): The user’s input message.

• system_prompt (str, optional): System prompt to guide the model’s behavior.

• chat_hist (List[Dict[str, str]], optional): Chat history for context.

• sampling_paras (Dict, optional): Parameters for sampling (temperature, top_p, etc.).

• model_name (str, optional): Specifies the model to use. If not provided, the default is the model set during class instantiation.

• do_json (bool, optional): Whether to return JSON responses. Default: False.

• do_sse (bool, optional): Whether to format responses as Server-Sent Events. Default: True.

• tools (List[Dict], optional): List of tools to be used by the model. Example format

  [{“name”: “tool name”, “description”: “tool description”, “parameters”: {“type”: “object”, “properties”: {“arg_1”: {“type”: “string”, “description”: “An example argument for the tool.”}}, “required”: [“arg_1”]}}, …..]

• tool_choice (str, optional): Defines tool usage:

  • “auto” (default) lets the model decide

  • “none” disables tools (not supported on AWSBedrock/AnthropicBedrock)

  • “required” forces tool use (unsupported on AzureOpenAI < 2024-06-01)

  • provide a tool name to call it directly.

• matched_sentence (List[str], optional): A list of matched text chunks for context. Not used internally, but included in the response under the matched_sentence key.

• extra_metadata (Dict, optional): Additional metadata to be included in the response.

Generation Arguments by provider:

AWS Bedrock models:

• additional_model_fields (Dict, optional): additionalModelRequestFields passed to the client in the request body.

Google Gemini & Vertex AI models:

• config (google.genai.types.GenerateContentConfig, optional): Optional model configuration parameters provided to the client.chats.create API.

• response_mime_type (str, optional): Output response mimetype of the generated candidate text. Supported mimetype: “text/plain” (Default), “application/json” (if do_json=True)

Yields:

Generator: Stream of responses from the language model, followed by a final dictionary containing the complete response, including tool calls and metadata.

class orichain.llm.AsyncLLM(**kwds: Any)

Bases: object

Asynchronous Language Model class for interacting with various LLM providers.

This class provides a unified interface to interact with different language models from providers such as OpenAI, AWS Bedrock, Google Gemini and Vertex AI, Anthropic, and Azure OpenAI.

default_model = 'gpt-5-mini'

default_model_provider = 'OpenAI'

__init__(**kwds: Any) → None

Initialize the Language Model class with the required parameters.

Args:

• model_name (str, optional): Name of the model to be used. Default: “gpt-4.1-mini”

• provider (str, optional): Name of the model provider. Default: “OpenAI”. Allowed values:

  • OpenAI

  • AzureOpenAI

  • AWSBedrock

  • GoogleGemini

  • GoogleVertexAI

  • AnthropicBedrock

  • Anthropic

  • TogetherAI

Authentication Arguments by provider:

OpenAI models:

• api_key (str): OpenAI API key.

• timeout (Timeout, optional): Request timeout parameter like connect, read, write. Default: 60.0, 5.0, 10.0, 2.0

• max_retries (int, optional): Number of retries for the request. Default: 2

AWS Bedrock models:

• aws_access_key (str): AWS access key.

• aws_secret_key (str): AWS secret key.

• aws_region (str): AWS region name.

• prompt_caching (bool, optional): Whether to use prompt caching. Default: True

• config (botocore.config.Config, optional):

  • connect_timeout (float or int, optional): The time in seconds till a timeout exception is thrown when attempting to make a connection. Default: 60

  • read_timeout: (float or int, optional): The time in seconds till a timeout exception is thrown when attempting to read from a connection. Default: 60

  • region_name (str, optional): region name Note: If specifing config you need to still pass region_name even if you have already passed in aws_region

  • max_pool_connections: The maximum number of connections to keep in a connection pool. Defualt: 10

  • retries (Dict, optional):

    • total_max_attempts: Number of retries for the request. Default: 2

Google Gemini models:

• api_key (str): Gemini API key

• http_options (types.HttpOptions, optional): HTTP options to be used in each of the requests. Default is None

• debug_config (DebugConfig, optional): Configuration options that change client network behavior when testing. Default is None

Google Vertex AI models:

• api_key (str): Vertex AI API key

• credentials (google.auth.credentials.Credentials): The credentials to use for authentication when calling the Vertex AI APIs.

• project (str): The Google Cloud project ID to use for quota.

• location (str): The location to send API requests to (for example, us-central1).

• http_options (types.HttpOptions, optional): HTTP options to be used in each of the requests. Default is None

• debug_config (DebugConfig, optional): Configuration options that change client network behavior when testing. Default is None

Anthropic models:

• api_key (str): Anthropic API key.

• timeout (Timeout, optional): Request timeout parameter like connect, read, write. Default: 60.0, 5.0, 10.0, 2.0

• max_retries (int, optional): Number of retries for the request. Default: 2

• prompt_caching (bool, optional): Whether to use prompt caching. Default: True

Azure OpenAI models:

• api_key (str): Azure OpenAI API key.

• azure_endpoint (str): Azure OpenAI endpoint.

• api_version (str): Azure OpenAI API version.

• timeout (Timeout, optional): Request timeout parameter like connect, read, write. Default: 60.0, 5.0, 10.0, 2.0

• max_retries (int, optional): Number of retries for the request. Default: 2

TogetherAI models:

• api_key (str): TogetherAI API key.

• timeout (float or int, optional): Request timeout in seconds. Default: 60

• max_retries (int, optional): Number of retries for the request. Default: 2

Raises:

• ValueError: If an unsupported model is specified.

• KeyError: If required parameters are not provided.

• TypeError: If an invalid type is provided for a parameter.

Warns:

• UserWarning: If the model name is not provided, it defaults to the default model.

async __call__(user_message: str, request: Request | None = None, matched_sentence: List[str] | None = None, system_prompt: str | None = None, chat_hist: List[Dict[str, str]] | None = None, sampling_paras: Dict | None = None, tools: List[Dict] | None = None, tool_choice: str | None = None, extra_metadata: Dict | None = None, do_json: bool = False, **kwds: Any) → Dict

Generate a synchronous response from the language model.

Args:

• user_message (str): The user’s input message.

• system_prompt (str, optional): System prompt to guide the model’s behavior.

• chat_hist (List[Dict[str, str]], optional): Chat history for context.

• sampling_paras (Dict, optional): Parameters for sampling (temperature, top_p, etc.).

• model_name (str, optional): Specifies the model to use. If not provided, the default is the model set during class instantiation.

• do_json (bool, optional): Whether to return a JSON response. Default: False.

• tools (List[Dict], optional): List of tools to be used by the model. Example format

  [{“name”: “tool name”, “description”: “tool description”, “parameters”: {“type”: “object”, “properties”: {“arg_1”: {“type”: “string”, “description”: “An example argument for the tool.”}}, “required”: [“arg_1”]}}, …..]

• tool_choice (str, optional): Defines tool usage:

  • “auto” (default) lets the model decide

  • “none” disables tools (not supported on AWSBedrock/AnthropicBedrock and TogetherAI)

  • “required” forces tool use (unsupported on AzureOpenAI < 2024-06-01)

  • provide a tool name to call it directly.

• request (Request, optional): FastAPI Request object for cancellation detection.

• matched_sentence (List[str], optional): A list of matched text chunks for context. Not used internally, but included in the response under the matched_sentence key.

• extra_metadata (Dict, optional): Additional metadata to be included in the response.

Generation Arguments by provider:

AWS Bedrock models:

• additional_model_fields (Dict, optional): additionalModelRequestFields passed to the client in the request body.

Google Gemini & Vertex AI models:

• config (google.genai.types.GenerateContentConfig, optional): Optional model configuration parameters provided to the client.chats.create API.

• response_mime_type (str, optional): Output response mimetype of the generated candidate text. Supported mimetype: “text/plain” (Default), “application/json” (if do_json=True)

Anthropic & AnthropicBedrock models:

• timeout (httpx.Timeout, optional): - timeout (httpx.Timeout, optional): Request timeout parameter like connect, read, write. Default is 60.0, 5.0, 10.0, 2.0

Returns:

Dict: The model’s response with tool calls and metadata.

async stream(user_message: str, request: Request | None = None, matched_sentence: List[str] | None = None, system_prompt: str | None = None, chat_hist: List | None = None, sampling_paras: Dict | None = None, tools: List[Dict] | None = None, tool_choice: str | None = None, extra_metadata: Dict | None = None, do_json: bool = False, do_sse: bool = True, **kwds: Any) → AsyncGenerator

Stream responses from the language model.

Args:

• user_message (str): The user’s input message.

• system_prompt (str, optional): System prompt to guide the model’s behavior.

• chat_hist (List[Dict[str, str]], optional): Chat history for context.

• sampling_paras (Dict, optional): Parameters for sampling (temperature, top_p, etc.).

• model_name (str, optional): Specifies the model to use. If not provided, the default is the model set during class instantiation.

• do_json (bool, optional): Whether to return JSON responses. Default: False.

• do_sse (bool, optional): Whether to format responses as Server-Sent Events. Default: True.

• tools (List[Dict], optional): List of tools to be used by the model. Example format

  [{“name”: “tool name”, “description”: “tool description”, “parameters”: {“type”: “object”, “properties”: {“arg_1”: {“type”: “string”, “description”: “An example argument for the tool.”}}, “required”: [“arg_1”]}}, …..]

• tool_choice (str, optional): Defines tool usage:

  • “auto” (default) lets the model decide

  • “none” disables tools (not supported on AWSBedrock/AnthropicBedrock)

  • “required” forces tool use (unsupported on AzureOpenAI < 2024-06-01)

  • provide a tool name to call it directly.

• request (Request, optional): FastAPI Request object for cancellation detection.

• matched_sentence (List[str], optional): A list of matched text chunks for context. Not used internally, but included in the response under the matched_sentence key.

• extra_metadata (Dict, optional): Additional metadata to be included in the response.

Generation Arguments by provider:

AWS Bedrock models:

• additional_model_fields (Dict, optional): additionalModelRequestFields passed to the client in the request body.

Google Gemini & Vertex AI models:

• config (google.genai.types.GenerateContentConfig, optional): Optional model configuration parameters provided to the client.chats.create API.

• response_mime_type (str, optional): Output response mimetype of the generated candidate text. Supported mimetype: “text/plain” (Default), “application/json” (if do_json=True)

Yields:

AsyncGenerator: Stream of responses from the language model, followed by a final dictionary containing the complete response, including tool calls and metadata.