System_message vs Description

[abhishekrai43]

Describe the issue

https://microsoft.github.io/autogen/blog/2023/12/29/AgentDescriptions/ So as per this, description replaces system_message for better conversational performance. The article says 'the description is for the benefit of the GroupChatManager, not for the Agent's own use or instruction.'

So, How is one to give different personas to the Assistant agents in the groupchat?

I am using Autogen for conversational discussions without any programming tasks and the conversation quality is not at all good.
I see a lot of 'As an AI assistant' messages, Agents talking randomly, I aska ll agents to contribute, only 1 will answer and then end tjhe conversation, calling themselves different names etc which makes me think I am not setting this up right maybe even when I just copy standard Autogen Examples and only change the system_message prompt.

Is there some additional step that needs to be taken to improve the quality of the conversation?

Version 0.2.6 GPT Model is gpt-4-turbo

Steps to reproduce

Copying any standard Autogen examples and setting up agents with conversational roles(Marketing office, CTO, CMO etc. No programming tasks.

Screenshots and logs

No response

Additional Information

No response

[ekzhu]

@afourney would have better answer for this. Could you share your code for setting up a group chat? Did you set the agent description field? I think the performance depends on the task and how clearly the task is articulated and what is the objective.

[abhishekrai43]

import autogen
import os
from autogen.agentchat.contrib.gpt_assistant_agent import GPTAssistantAgent
config_list = [
    {
        "model": "gpt-4-turbo",
        "api_key": os.getenv('OPENAI_API_KEY')
    },
    {
        "model": "gpt-3.5-turbo",
        "api_key":os.getenv('OPENAI_API_KEY')
        
    }
]
llm_config={"config_list": config_list}
user_proxy = autogen.UserProxyAgent(
    name="User_proxy",
    description="A human Admin that facilitates the conversation between the agents ensuring that each agent maintains their distinctive roles and cross checks the responses to ensure they are within their respective roles. The context of the conversation is that all roles, all working within the same company and collaborate with each other in solving strategic problems.The company is global technology company providing IT product and services. There is a staff of 10000 people in the company. They operate in the US, EU, and Asia.",
    code_execution_config={
        "last_n_messages": 2,
        "work_dir": "groupchat",
        "use_docker": False,
    },  # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.
    human_input_mode="TERMINATE",
)
ceo = autogen.AssistantAgent(
    system_message="CEO. needs accurate answers from my CTO and CMO . The answers need to address the queries in entirety and then some. They should show the brilliance of the team in coming up with practical, unique if needed and robust solutions to the queries. Each Team member must stick to their respective roles only and not comment as another team member",
    name="Peter",
    llm_config=llm_config,
)
cto = autogen.AssistantAgent(
    name="Sarah",
    system_message="CTO. Expert with 30 years of experience and a PhD in Computer Science from MIT. I have a wealth of knowledge in all technical aspects of the underlying projects and product. I take all scientific , product development, and delivery related judgements, ensuring they are sound.  My answers are rooted in authenticity and accuracy. I work in tandem with the CMO to output the best possible solution to present to our CEO",
    llm_config=llm_config,
)
cmo = autogen.AssistantAgent(
    name="Andy",
    system_message="CMO. Expert with 30 years of experience and MBA from Harvard Business School. I look at all marketing, advertising, sales, promotion, press-releases, company communications, related aspects and  issues deeply, think step by step and then answer after carefully considering all edge cases, unforeseen events and also cover the basics with ironclad logical thinking.",
    llm_config=llm_config,
)
groupchat = autogen.GroupChat(agents=[user_proxy, ceo, cmo,cto], messages=[], max_round=5,speaker_selection_method='manual')
manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)
user_proxy.initiate_chat(
    manager, message="we are considering a massive LinkedIn ad campaign for out chatbot product. This product is a game changer but, we have not done any real outreach to date. Please provide the pluses and minuses to coming out of the gat with a large LinkedIn campaign. When you respond, please announce your name in the first part of your answer."
)

@afourney ...I think I am using a very standard sort of an example, kindly let me know if I have made some mistake. So, I see that these 'names' sometimes are recognized when responding, by the agents, most of the times they call themselves assistant. One thing that happens most frequently is CMO calling itself CEO or CEO saying As the CMO, etc and a lot of qualifiers "As an AI assistant" ....Then also it would say "As an AI assistant I cannot connnect with CTO or CMO" ...sometimes they will start using their correct names. So it's a bit here and there leaving me wondering what wrong am I doing? Also description and system_message , are they the same , can we use one to provide context of the conversation or do they just replcae each other as instructions for the Agent? I don't mean to ask these many questions but trying to put the complete behaviour to provide as much context as possible.

[abhishekrai43]

@ekzhu Can you comment on the other points?

[afourney]

Can you confirm that gpt-4 is actually being used? I believe the OpenAI model is actually called "gpt-4-turbo-preview". Is it possible that it's falling back to gpt-3.5-turbo maybe, as per the config list? GroupChat and related activities don't work as well with weaker models. Agents assuming the role of other agents is a very common failure case in this case.

[abhishekrai43]

@afourney No, I am using gpt-4-turbo. I did try it with gpt-4 as well. My takeaway is this. That Autogen groupchat is meant to give the group of agents 1 task..that they can achieve using Python, and return back a result.

It does not seem as if it is meant for having a conversation back and forth which does not include any 'tasks'. Tasks referrring to running some python code to find/calculate/deduce/etc ....so would I be correct in assuming that all personas by means of either system_message or description (assuming they are interchangeable) are sent basically to the same LLM with personas as a context and that is why sometimes it will end up responding with As an AI model or I apologize or something. Also I think that the speaker method seems best when it is set to auto .Would specifically directing it to an agent by manually choosing that agent result in better responses aligned to that Agent's persona? Then that would sort of mean , that I can have 1 agent and pass different personas to it if I have model with large input and I can have the same thing? Since this is a dicussion , I am taking the liberty to use this language, Kindly let me know if you'd like me to be more concise.

[abhishekrai43]

@afourney Sorry to repeat myself but given a choice between auto and manual select speaker...what would you recommend? Assuming there is 1 task to be solved within the output generation token limit of the model?

[abhishekrai43]

"gpt-4-turbo", this is indeed strange..Not sure which model is being used then?...but I have run it multiple times...how does it find it? regex?

[afourney]

Typically, we mode down the OAI config list, so that if a call fails, we try the next model on the list. I don't personally know if this happens for every error, or only some (like rate limit). @sonichi would know.

It is conceivable that it's failing the gpt-4-turbo call, and then succeeding on the gpt-3.5-turbo call (since this is a model). You could test this by removing gpt-3.5.-tubo and trying again. Do you get an error this time?

[abhishekrai43]

@afourney The response time does not indicate there is any mode failing though or that is my impression because I tried GPTAssistant(Not conversable Agent) and there was clearly a much longer response time when I selected gpt-3-turbo for assistants and gpt-4 for manager. Also I am a very heavy user of GPT-4 the browser, the content of the answers indicate that it is gpt-4-turbo, it's only the context that this is happening in a groupchat with other agents who are working towards a common goal, is where it seems lacking, They retain their 'I am an AI assistant' persona.

Any links to this(github or any other)? People do have non-coding multi-agent group chats though. ..Maybe I can look at those and fill holes in my understanding on these?

[abhishekrai43]

@afourney No errors in the chat. Do you get an error this time?