Can not get consistent hand off to user and have agent respond with message prior to tool call

[MHVosovski]

I have been at this for only two months, so still learning and lots more to learn. What I ran into might be just lack of experience, so want to ask for feedback and maybe to be steered into right direction.

I have a fairly simple multi-agent swarm setup using autogen 0.4.0.dev13

• DataAnalyst - receives task or question from user. Has tool to execute SQL query which will save result to file in work_dir and respond with name
• SchemaExpert - receives name of table or description of needed data and returns schema. In short term has tool that takes exact table name and return SQL schema of the table. Instructions include MD table of all tables and description of contents. Long term solution is to fine-tune or train (still don't know correct term) on database schema
• CodeExecutor - basic llm-less agent to execute python code

Handoff to schema expert is great, even with follow up questions or adjustments to task. Handig off to user is hit or miss, and I suspect maybe it is the prompt I have (snippet below):

Do not use tools right away. Send a message explaining steps and plan first.
[...cut...]
You must first send a message explaining your plan and next step before executing a tool or handing off.
If you need information from the user, you must first send your message, then you can handoff to the user.
Before executing SQL query, first provide it in code block and hand off to the user for approval to run the query.
Use TERMINATE when question is fully answered or task is completed.

There are instances when LLM is saying that it wants user to confirm, but handoff is not happening and query executes. When more information is needed, user handoff is mostly consisent and hands off to user, but the approval/confirmation for running query is not.

Second thing I am facing is not being able to get LLM to send message before calling tool. Data analyst agent will always start with tool call to transfer to schema expert if question/task involves database schema, and once it receives schema it layout the plan and steps it will take next. Out of frustration I added first line but it made no difference.

What I would like it to be is state that it needs schema and it will hand off this request to schema expert.
OpenAI Assistant, despite not being multi-agent, does this kind of nicely by sending a message that it will use a tool to complete "such and such task".

Other than that, it has been a great experience learning autogen 0.2 and now 0.4. Putting things like this together and watch it work is quite something. I suspect in next few months Semantic Kernel will be on my radar.

Edit: Oups, meant to include relevant portions of code

agents = [
    AssistantAgent(
        name="DataAnalyst",
        model_client=self.client,
        handoffs=["SchemaExpert","CodeExecutor", "user"],
        tools=[execute_query2],
        system_message="""Do not use tools right away. Send a message explaining steps and plan first.
        ...
        You do not have names or schema of existing databases and tables, but you can ask SchemaExpert to provide this information.
        You can handoff python code execution task to CodeExecutor.
        You must first send a message explaining your plan and next step before executing a tool or handing off.
        If you need information from the user, you must first send your message, then you can handoff to the user.
        Before executing SQL query, first provide it in code block and hand off to the user for approval to run the query.
        Use TERMINATE when question is fully answered or task is completed.
        """
    ),
    AssistantAgent(
        name="SchemaExpert",
        model_client=self.client,
        handoffs=["DataAnalyst"],
        tools=[get_schema],
        system_message="""You are an agent specialized in retreiving detailed schema about tables across multiple SQL Server databases.
        ...
        Use your only tool get_schema to retreive detailed information about requested table's schema.
        If you can not answer with schema details do not make up or imagine a schema. Ask for clarification
        When complete, handoff to the data analyst to continue"""
    ),
    CodeExecutorAgent(
        name="CodeExecutor",
        code_executor=LocalCommandLineCodeExecutor(
            timeout=60,
            work_dir="workdir_agentteam3",
        )
    )
]

termination = HandoffTermination(target="user") | TextMentionTermination("TERMINATE") | MaxMessageTermination(25)

self.team = Swarm(
    participants=agents,
    termination_condition=termination,
)```

[ekzhu]

Handig off to user is hit or miss

You can also tune the handoff by customizing the hand off tool (its description, name, returned message) by using autogen_agentchat.base.Handoff rather than a just a string for the agent name.

the approval/confirmation for running query is not.

I believe for approval of running the tool, it should not depend on the agent, rather, it should be built into the tool itself, and have the tool returns a well-formed error message if the user rejects it. It is a general pattern that we may incorporate into the AssistantAgent itself. A PR contribution is welcome!

Second thing I am facing is not being able to get LLM to send message before calling tool.

This is a good idea. Have you tried providing some in-context learning examples in the system message, or through the model_context parameter of AssistantAgent, by adding a few pairs of UserMessage and AssistantMessage in the initial_message field?

I have created #4895 and #4894 to track these two features. We can continue relevant discussion for these issues there.