Issues getting agent based on non-openai models to use local function/tool call

[rdstettler]

I'm fairly new to autogen but can't really make local function calls work. I tried with openai 3.5 turbo, it appears to work, when running mistral-7b-v0.1.Q6_K.gguf or mistral-7b-instruct-v0.2.Q4_K_M.gguf the model appears to be completely ignorant about local functions, maybe there is something I'm missing. So first I added a python function that can call our erp system and get the description of a machine id

llm_config = {
    "timeout": 600,
    "seed": 42,
    "config_list": config_list,
    "temperature": 0,
    "functions": [
        {
            "name": "get_description_of_erp_machine",
            "description": "Get the ERP description of a given machine id",
            "parameters": {
                "type": "object",
                "properties": {
                    "machineid": {
                        "type": "string",
                        "description": "The ERP machine id"
                    }
                },
                "required": ["machineid"]
            }
        },
....

I import the skills

import skills
print(skills.get_description_of_erp_machine('M123456'))

> My Machine Description

I then created 2 models like this

assistant = autogen.AssistantAgent(
    name='assistant',
    llm_config=llm_config,
    is_termination_msg=lambda x: x.get("content", "") and "terminate" in x.get("content", "").lower(),
    system_message="""PROVIDE USEFUL PYTHON CODE INCORPORATING THE LOCALLY AVAILABLE FUNCTIONS FOR ERP REQUESTS.""",
)

user_proxy = autogen.UserProxyAgent(
    name='user_proxy',
    human_input_mode='NEVER',
    max_consecutive_auto_reply=10,
    is_termination_msg=lambda x: x.get("content", "") and "terminate" in x.get("content", "").lower(),
    code_execution_config={
        "work_dir": "coding",
        "use_docker": False
    },
    llm_config=llm_config,
    system_message="""You are a local ERP API assistant
    If the task was answered to full satisfaction, summarize with repeating the task and add the result and end the answer with the word TERMINATE.""",
)

user_proxy.register_function(
    function_map={
        "get_description_of_erp_machine": skills.get_description_of_erp_machine,
    }
)
assistant.register_function(
    function_map={
        "get_description_of_erp_machine": skills.get_description_of_erp_machine,
    }
)

user_proxy.initiate_chat(
    assistant,
    message="What name does machine 511V0001 have? "
)

So far so good, when calling open ai 3.5 for whatever reason it works. But with mistral the llm just returns

To answer your question, I'd need to assume you are using the SAP Python Client pySAP for interacting with an SAP system. Here's a code snippet that demonstrates how to retrieve the machine name bla bla

It appears that mistral isn't capable to retrieve the function list I passed. Is that correct? Is it "just" the model having less features than openai?

Update

So I just told the system message that it should know the functions (I added another function to the original post)

assistant_system_message = """PROVIDE USEFUL PYTHON CODE INCORPORATING THE LOCALLY AVAILABLE FUNCTIONS FOR SAP REQUESTS. You are being given the following python functions with description. Assume the functions available to the user, without having to import anything
    function name: get_description_of_erp_machine
    argument (string, use quotes): machineId
    description: Get the ERP description of a given machineid
   
    ...
    Use those functions when answering. 
    """

Then it works in a sense, that the provided code uses those functions, BUT : I get the following errors

...
machine_name = get_description_of_erp_machine("511V0001")
^^^^^^
NameError: name 'get_description_of_erp_machine' is not defined

Why isn't the function not available?

[ekzhu]

First of all, regarding the function not being available, it seems that the user proxy agent is in code execution mode not in function calling mode. I have seen this happen for Non-OpenAI models because they don't follow the OpenAI's function/tool call message format. So there is no function_call or tool_call field in the message from the model. Without those fields, user proxy agent cannot pick up the functions. Since you also mentioned the functions in the assistant's system message, the assistant produced code blocks in the message's content field. A code block triggers the user proxy agent into code execution mode, but the function definition is not in the code block, and that's why you see the error.

To make user proxy use the registered functions, the message from assistant must include a structured function_call or tool_call field following OpenAI schema. One way to do that is to use structured generation through something like Guidance.

Lastly, I you can use register_for_llm and register_for_execution decorators. They will make the code much simpler and you don't have to write any JSON schema. See: https://microsoft.github.io/autogen/docs/reference/agentchat/conversable_agent#register_for_llm

[rdstettler]

@ekzhu Thank you very much for the elaboration. As far as the decorators go, I didn't use them because I wanted to export the functions to a separate file, however I feel like registering those in a proper module that can be imported appears to be significantly better. Apart from that, I added the function call into the whole script, I added the decorator and retried. Still the same issue, what does the user proxy do wrong? This is was autogen outputs, what additional code does the assistant need to give?

user_proxy (to assistant):

What name does machine 511V0001 have? 

assistant (to user_proxy):

 To get the description of machine 511V0001, you can use the `get_description_of_erp_machine` function as follows:

```python
machine_description = get_description_of_erp_machine("511V0001")
print(f"The description of machine 511V0001 is: {machine_description}")
"""

 EXECUTING CODE BLOCK 0 (inferred language is python)...
SIGALRM is not supported on Windows. No timeout will be enforced.
user_proxy (to assistant):

exitcode: 1 (execution failed)
Code output: 
Traceback (most recent call last):
  File "", line 1, in <module>
    machine_description = get_description_of_erp_machine("511V0001")
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'get_description_of_erp_machine' is not defined

[ekzhu]

It looks like it is the same problem as before. The underlying model cannot generate the function_call and tool_call fields so the user proxy cannot pick it up, and your functions are not available in the code execution mode -- they are only available in function execution mode. Have you tried using Guidance?

Thanks for raising this issue. I think it is an important one to address. I will create an issue for this. We will create some example notebooks and perhaps a way to address it using autogen itself.

Meanwhile, a quick and dirty solution would be to customize the system message of the AssistantAgent. You can append the complete function definitions there and ask it to always include the full function definition when generating code blocks.

[rdstettler]

The funny thing is, with autogenstudio for whatever reason it works (mistral instruct model). I have never head of Guidance and will look into it.

I generated a github repo with all the necessary data (except the config files)
https://github.com/rdstettler/autogen_localLLM_erpfunctions/tree/main

[ekzhu]

Thank you!

[sonichi]

@qingyun-wu @victordibia

[niuhuluzhihao]

Are skills in autogenstudio the same as functioning calling/tool calling? @ekzhu @sonichi It seems that it does not have the function of functional calling. Why can it be run in autogenstudio?

The funny thing is, with autogenstudio for whatever reason it works (mistral instruct model). I have never head of Guidance and will look into it.

I generated a github repo with all the necessary data (except the config files) https://github.com/rdstettler/autogen_localLLM_erpfunctions/tree/main

[niuhuluzhihao]

@ekzhu Skills are preloaded module in the user proxy agent?I execute autogenstudio ui --port 8081，the default result is as shown below.

In fact, he is configured in AssistantAgent.
Does this mean that skills are configured in Assistant, but skills are executed in UserProxyagent?

[ekzhu]

Correct.

[niuhuluzhihao]

Thank you

[rizickus]

Skills are preloaded module in the code executor of the user proxy agent. It is different from functions, which are executed by the function executors in the user proxy agent. Currently the are defined through separate interfaces. Skills are currently only available in autogenstudio. Functions are available in the library.

@ekzhu, are there any plans to unify skills and tools? In some sense, I prefer tools because you pass only very limited API information. With skills looks like in general you have to use more tokens to enable your agent to achieve the same thing.

To sum up, the studio uses skills, but tools look "leaner" approach, so I am having paralysis on which way to move forward at this moment.

[ekzhu]

We are working on integrating skill-flavored tools to the core library. #1421 is addressing it. Would you like to give some feedback on that? You can comment on that roadmap issue so we can ping you when the PR is ready for review.