Proposal: Topical Guidelines

[kichanyurd]

Motivation

Imagine we have a Parlant insurance agent "working at" an insurance company with strict regulatory requirements.

Because of the strict regulations of this use case, the agent must not work by a "best effort" strategy in trying to help the customer. Instead, it must only offer support in cases where human input and guidance was explicitly provided to it by the system—specifically, in this scope, through its provided set of guidelines.

Let's construct a minimal example to illustrate the challenge that may arise here.

The Use Case

1. There's only one guideline installed:
   1. Condition: The customer requests an appointment with a human rep
   2. Action: Help them find a slot and book it
2. A customer comes along and tells the agent: "How long is a normal consultation appointment?"
3. Clearly, in this case, the installed guideline—while essentially on-topic—is not activated, since the customer is not really requesting an appointment at this point (or at least not yet).

Current Behavior

Today, the MessageEventGenerator should evaluate the context and see that it doesn't have the necessary information to answer the customer's question, and ideally (WIP) respond with some equivalent of "I don't have that information."

In short, in today's version of the engine, the agent should reply, "Sorry but I don't have that information."

Desired Behavior

While, strictly speaking, the agent doesn't have the information, there may still be a more productive answer than "I don't know." This is because, while the agent doesn't have that information, it could still proactively offer to help with things within the same topic that it does know about.

For example, since it can help with booking an appointment, it could arguably handle the interaction as follows:

• Customer: How long is a normal consultation appointment?
• Agent: Sorry, I don't have that information at this point, but if you want I can nonetheless help you schedule one.

Solution Proposal

1. Add a new first-class feature to the engine: Topics. Each Topic as a name, a description, and a set of zero or more (embeddable) short documents called TopicalGroundingDocument that are automatically retrieved in the appropriate conversational contexts.
2. Add a property topics to each guideline. Each guideline can be associated with zero or more topics.
3. In addition, there is a special built-in topic, called no-topic.
4. If a guideline is not associated with any topic (including no-topic), it relates to the engine in exactly the same way it does today.
5. Now, before matching applicable guidelines, we evaluate the conversation and classify it into zero or more supported topics which have been manually added to the agent. Here—and this is important—suppose that no guidelines are matched during the engine's execution. This what we do next in this case:
   1. If the conversation matches no topic, then we look for guidelines associated with no-topic.
      1. If there are no such guidelines, we proceed with the execution as usual (this maintains the behavior we have today).
      2. Otherwise, we explicitly load all guidelines marked with no-topic, allowing them to affect the rest of the generation.
      3. Effectively this means that as soon as we add even one no-topic guideline, this feature kicks into effect, constraining the agent to never reply in an unguided way.
   2. If the conversation matches one or more topics (remember that still no guidelines were automatically matched here, so we're "guideline-less" at this point):
      1. Fetch all guidelines within the matched topics. Run an LLM request that, given the current interaction context, provides up to 3 top ways to continue the conversation in a guided way. In other words, extract actionable content from those topical guidelines, and use that to lead the conversation forward.
      2. As an edge case, we need to account for cases where none of the fetched guidelines are actually actionable from a functional standpoint. For example, they may try to guide mannerisms or things of that nature. Here, we fall back to a no-topic state and follow the same logic as outline above.

Discussion

Let's try to consider various use cases and see if the current proposal properly addresses them. Also, let's try to consider where this approach could go wrong.

[MCBarKar]

Sounds good to me overall, it should solve current problems we have where the agent doesn't know what it can and cannot do.

One point of potential improvement - if no guidelines were proposed in the classical way (by the guideline proposer), instead of having one (new) module suggest up to 3 continuations, and then sending them to the message event producer, maybe it should directly choose a reply to the user? Thereby saving an LLM call.

Meaning, instead of building this new module, we could build an alternative version of the message event generator which is used exclusively for when we're in the 'no-match' mode, where no guidelines were activated.

[sramichetty20019]

Great proposal for improving agent responses! I'd like to suggest an enhancement using intent classification while adhering to the guidelines:

Example Enhancement:

1)Current:
Customer: "How long is a normal consultation appointment?"
System: [Topic Match → No Guidelines → Fallback]
Response: "Sorry, I don't have that information, but I can help schedule one."

2)Proposed Intent-Based Enhancement:
Intent Classification Layer:

• Identifies primary intent (appointment information)
• Maps to approved guideline intents (appointment scheduling)
• Uses transformer-based classification to handle variations in queries(while topics may not be able to cover all variations because users have free flow text , which is indeterministic in nature )

I am thinking more on this ......this is a basic idea i have in mind

[MCBarKar]

I didn't think about intent classification here but I agree that it could help.

Just to clarify, your proposed improvement is:

1. Identify the intent of the user in natural language.
2. Classify that intent into one of many predetermined intents (we could think of these as topics, or purely as 'intent-classes')
3. Continue with the suggested flow, inserting guidelines with the same intent-class into the prompt used to generate a response to the user.

Did I get it right?

[sramichetty20019]

yeah thats the way even if there are some synomyms that user writes in his free text they would be identified to right inent or the closest intent

[vahuja4]

Looks pretty solid to me! @MCBarKar's optimization is very important! Considering my latest discord message about having "global", "meta", and "task-specific" tags for every guideline, one simple optimization could be to add topics to task-specific guidelines for which no topics have been added. But this would be a one-time exercise.

[vahuja4]

Let us consider the following scenario
G1: "when a customer wants to set up a recurring transaction, then..." topic ["transaction"]
G2: "when a customer wants to do wire transfer, then.." topic:["transaction"]

Let us say that the user's query is "dispute transaction". This will match at the topic level, but not at the guideline level, obviously. I am assuming that in this case, the msg gen will just output something like "Sorry, I can't help with disputing a transaction". Because the guidelines are not at all related

[mc-dorzo]

@vahuja4 In the example you gave, would you want it to say only "Sorry I can't help" or since there are guidelines on that topic, would you want it to offer practical ways to continue based on them?

[vahuja4]

Yes, only "Sorry, I can't help with disputing a transaction" because including things like "Sorry, I can't help with disputing a transaction. But I can help you do a wire transfer" might seem very off. Basically, what I am saying is that the msg generator should do proper filtering even if there are guidelines with the same topic

[vahuja4]

Along the same lines, if there is faq integration, then we could possibly answer like so:
"Sorry, I can't help you dispute a transaction right here, but here is some info:......"

[mc-dorzo]

Got it! Makes sense. So one step is to detect the situation and then give you custom ways to hook into it and handle it. One way is to disambiguate but another way would be to first run another evaluation and potentially do something else based on its result. This does sound more flexible.

[kichanyurd]

@vahuja4 @MCBarKar Interesting advice came from an experience Conversation Designer I was talking to today.

He was saying, from a practical standpoint, it may make sense to allow a default guideline for any topic. When the conversation matches that topic, and no other guidelines are matched in that topic, we include this default guideline.

This would allow a designer to provide a clear fallback strategy on any topic, as well as covering the case where there is a no-guideline and no-topic match, in which case we just say, "We can't help with that" or something of that nature.

WDYT?

[MCBarKar]

Makes sense to me