Guidance Requset: Complying with prompt formats (e.g StableBeluga)

[kfsone]

I've now encountered public models that sound like they would fit agent styles/roles I've found interesting, but these models require a particular prompt format. I'll use StableBeluga as an example; at a glance it seems quite apropos to autogen:

https://huggingface.co/stabilityai/StableBeluga-13B

It uses a multi-stop(?) format:

System:

This is a system prompt, please behave and help the user.

User:

Your prompt here

Assistant

The output of Stable Beluga 13B

I guess this is similar to Google Vertex's "Context" component.

Assuming I have the model available via a local, openai endpoint (e.g thru LM Studio) how might I integrate it, say, for the critic role of the research https://github.com/microsoft/autogen/blob/main/notebook/agentchat_groupchat_research.ipynb example?

Note I'm not specifically looking to use beluga, and I realize I could just prefix the messages with '### System:\nThis is your role...' but I'm also considering more nuanced than that, if that isn't a little more than autogen is ready for yet :)

[sonichi]

Would this help? https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#templating
and https://microsoft.github.io/autogen/docs/reference/agentchat/conversable_agent#send

[kfsone]

I'd initially assumed that was how to achieve it, but could not get it to work, and ended up feeling stupid. So I tasked some of the example web-access agents with the problem and they either concluded it wasn't supported, required writing a custom conversable in whole, or in the case of one local LM, reciting poetry about flying on an airplane: My first encounter with hallucination, I believe? Honestly, I was grateful for that, heh (ref "feeling stupid").

The Use-Cases template example is fairly dense and maybe its just lacking a concrete example. A lot of the huggingface models I tried running require specific prompt sequences; I threw up a sandbox and used LM Studio to host them, and found that it's "chat" interface goes an extra mile to massage inputs to the required form but leaves the server api raw.

[j-loquat]

Agree! I customized LM Studio chat interface for the right prompt format for Mistral 7B - but wondered how to handle the same thing for Autogen through the api. It works ok by default but probably would be better performance if I could modify Autogen code to use the right prompt template.