Generic LLM Selection with Fallback and Error Handling

[tim-heinsohn]

Is your feature request related to a problem? Please describe.

In a Langchain.rb based application that is setup and hosted by different users
it's not known which LLMs can be used via API keys.

So the idea would be to allow users to run such an application with agents that

• either do not rely on a specific LLM, or
• use specific LLM methods and parameters on an optional base, or
• raise a specific error when the LLM does not match.

Users could just provide the specific LLM API keys they have access to, and
the application would select and use the corresponding LLM class.

While the application could handle this on its own, it might be a common use
case for Langchain.rb users.

Providing a mechanism for generic LLM configuration in Langchain.rb directly
would thus allow application developers to use Langchain.rb out of the box
without implementing their own LLM setup.

This would also be important for multi-tenant applications, cost
optimization, failover scenarios, and experimentation with different LLMs.

Describe the solution you'd like

A generic Langchain::LLM::Generic model could handle several parts:

1. Dynamic LLM Selection:

   • Use an LANGCHAINRB_LLM environment variable containing the provider and
     model name (e.g., "anthropic.claude-3-sonnet").
   • Support fallback values or a prioritized list of LLMs (e.g.,
     "openai.gpt-4,anthropic.claude-3-sonnet").

2. LLM Initialization:

   • Instantiate the corresponding LLM class based on the environment variable.
   • Validate configuration and handle initialization errors gracefully.

3. Method Availability Check:

   • Provide a check on the llm instance (or a maintained list of methods)
     before forwarding a call to let agents know whether a method is available
     in that LLM.

4. Parameter Mapping:

   • Map generic parameters to provider-specific ones (e.g., temperature to
     top_p for certain providers).

5. Method Forwarding:

   • Forward all method calls to the underlying LLM instance.

Describe alternatives you've considered

1. Factory Pattern:

   • A factory could deliver the specific LLM instance directly based on the environment.
   • Pros: Simpler implementation, clearer separation of concerns.
   • Cons: Less flexibility for dynamic method handling.

2. Plugin System:

   • A plugin system could allow users to add new LLM providers dynamically.
   • Pros: Highly extensible.
   • Cons: More complex to implement and maintain.

Additional context

This feature would be particularly useful for:

• Multi-tenant or locally hosted applications where different users may have
  access to different LLMs.
• Cost optimization by dynamically selecting the most cost-effective LLM.
  Agents could provide a list of required llm methods.
• Failover scenarios where the application can switch to a backup LLM if the
  primary one fails.
• Experimentation with different LLMs to compare performance and results.

Examples

1. Environment Variable Configuration:

export LANGCHAINRB_LLM="openai.gpt-4,anthropic.claude-3-sonnet"

2. Usage in Code:

llm = Langchain::LLM::Generic.new

# Attempt to use a method that is not supported by the selected LLM
if llm.supports_message?(:summarize)
  summary = llm.summarize(text: "A long article about AI advancements...")
  puts "Summary: #{summary}"
else
  puts "Summarization is not supported by the selected LLM."
end

If for example the selected LLM is Anthropic (which doesn't support
summarize):

The `summarize` method is not supported by the selected LLM.
Supported LLMs for `summarize`: AI21, Cohere, OpenAI

Implementation notes:

module Langchain::LLM
  class Generic < Base
    SUPPORTED_METHODS = {
      summarize: %w[AI21 Cohere OpenAI],
      embed: %w[OpenAI Anthropic GoogleGemini Cohere HuggingFace MistralAI Ollama]
    }.freeze

    def supports_method?(method_name)
      if SUPPORTED_METHODS.key?(method_name)
        super
      else
        raise MethodNotSupportedError.new(method_name, SUPPORTED_METHODS[method_name])
      end
    end
  end
end

module Langchain::LLM
  class MethodNotSupportedError < StandardError
    attr_reader :supported_llms

    def initialize(method_name, supported_llms)
      @supported_llms = supported_llms
      super("#{method_name} is not supported by the selected LLM. Supported LLMs: #{supported_llms.sort.join(', ')}")
    end
  end
end

Usage with rescue:

begin
  response = llm.chat(messages: [{ role: "user", content: "Hello!" }])
rescue Langchain::LLM::MethodNotSupportedError => e
  puts "Error: #{e.message}. Supported LLMs: #{e.supported_llms.join(', ')}"
end

[tim-heinsohn]

Just seeing this for a good part is covered by LLM Adapters already.