[Microsoft.Extensions.AI] Questions on undocumented features

[raffaeler]

What's the intended way to manage the following features:

1. Refusals coming from the LLMs
2. Manually handing tools in streaming mode for parallel invocations

TIA

[stephentoub]

Refusals coming from the LLMs

You can access state from the underlying provider via RawRepresentation and/or AdditionalProperties. So if you know you're working with a provider that surfaces refusal information in a certain way, you can access it there. Refusals haven't become widespread enough for us to want to promote it up to first-class API, though we recently added ErrorContent, and we could possibly choose to map refusal information to that.

Manually handing tools in streaming mode for parallel invocations

Can you elaborate on the question?

[raffaeler]

Thanks for the answer.

Manually handing tools in streaming mode for parallel invocations
Can you elaborate on the question?

I work in streaming mode and need to manually invoke the tools.
Currently I am doing this:

• Outer loop repeating the GetStreamingResponseAsync until the FinishReason is stop
• Notifying the user interface exclusively with the AIContent items of type: TextContent, TextReasoningContent, DataContent, UriContent, ErrorContent and UsageContent.
• Inner loop examining the updates and collecting the FunctionCallContent items in a separate collection.
• Outside the inner loop (at the end of the updates) I invoke the tools (either serially or in parallel).

Is this the correct/intended workflow?

[stephentoub]

If you don't want to use the automatic invocation and instead want to use manual invocation, you would just do effectively the same thing that FunctionInvocationChatClient is doing here:

extensions/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/FunctionInvokingChatClient.cs

Lines 326 to 416 in 7b2620c

	for (int iteration = 0; ; iteration++)
	{
	updates.Clear();
	functionCallContents?.Clear();
	await foreach (var update in base.GetStreamingResponseAsync(messages, options, cancellationToken))
	{
	if (update is null)
	{
	Throw.InvalidOperationException($"The inner {nameof(IChatClient)} streamed a null {nameof(ChatResponseUpdate)}.");
	}
	updates.Add(update);
	_ = CopyFunctionCalls(update.Contents, ref functionCallContents);
	if (totalUsage is not null)
	{
	IList<AIContent> contents = update.Contents;
	int contentsCount = contents.Count;
	for (int i = 0; i < contentsCount; i++)
	{
	if (contents[i] is UsageContent uc)
	{
	totalUsage.Add(uc.Details);
	}
	}
	}
	yield return update;
	Activity.Current = activity; // workaround for https://github.com/dotnet/runtime/issues/47802
	}
	// If there are no tools to call, or for any other reason we should stop, return the response.
	if (functionCallContents is not { Count: > 0 } ||
	options?.Tools is not { Count: > 0 } ||
	iteration >= _maximumIterationsPerRequest)
	{
	break;
	}
	// Reconstitute a response from the response updates.
	var response = updates.ToChatResponse();
	(responseMessages ??= []).AddRange(response.Messages);
	// Prepare the history for the next iteration.
	FixupHistories(originalMessages, ref messages, ref augmentedHistory, response, responseMessages, ref lastIterationHadConversationId);
	// Process all of the functions, adding their results into the history.
	var modeAndMessages = await ProcessFunctionCallsAsync(augmentedHistory, options, functionCallContents, iteration, consecutiveErrorCount, isStreaming: true, cancellationToken);
	responseMessages.AddRange(modeAndMessages.MessagesAdded);
	consecutiveErrorCount = modeAndMessages.NewConsecutiveErrorCount;
	// This is a synthetic ID since we're generating the tool messages instead of getting them from
	// the underlying provider. When emitting the streamed chunks, it's perfectly valid for us to
	// use the same message ID for all of them within a given iteration, as this is a single logical
	// message with multiple content items. We could also use different message IDs per tool content,
	// but there's no benefit to doing so.
	string toolResponseId = Guid.NewGuid().ToString("N");
	// Stream any generated function results. This mirrors what's done for GetResponseAsync, where the returned messages
	// includes all activities, including generated function results.
	foreach (var message in modeAndMessages.MessagesAdded)
	{
	var toolResultUpdate = new ChatResponseUpdate
	{
	AdditionalProperties = message.AdditionalProperties,
	AuthorName = message.AuthorName,
	ConversationId = response.ConversationId,
	CreatedAt = DateTimeOffset.UtcNow,
	Contents = message.Contents,
	RawRepresentation = message.RawRepresentation,
	ResponseId = toolResponseId,
	MessageId = toolResponseId, // See above for why this can be the same as ResponseId
	Role = message.Role,
	};
	yield return toolResultUpdate;
	Activity.Current = activity; // workaround for https://github.com/dotnet/runtime/issues/47802
	}
	if (modeAndMessages.ShouldTerminate)
	{
	break;
	}
	UpdateOptionsForNextIteration(ref options, response.ConversationId);
	}
	AddUsageTags(activity, totalUsage);
	}

It's basically what you said, except you don't need really need to pay attention to FinishReason; just iterate until the end of the IAsyncEnumerable.

[raffaeler]

Thanks.
I check the FinishReason because the code managing the tools, adds the tool responses to the history, which needs to be sent backk to the LLM service.
That said, I can rely on a private variable telling there are new messages in the history or just check the FinishReason.
Is FinishReason reliable to achieve this?

[stephentoub]

I don't understand. What are you going to do differently based on the finish reason? If there are no FunctionCallContents, then there's nothing more to do. If there are, you turn the crank to iterate again.

[raffaeler]

It's very simple: i repeat GetStreamingResponseAsync until the FinishReason is stop because my code invokes the tool and add the FunctionResultContent to the history.
As I wrote, I can rely on a boolean for that, but I was wondering if I can assume that stop is always sent at the end or there may be other conditions where this is not true.

[stephentoub]

It's very simple: i repeat GetStreamingResponseAsync until the FinishReason is stop because my code invokes the tool and add the FunctionResultContent to the history.

Sure, but adding the FunctionResultContent into the history doesn't cause the response stream to run any longer. You need to issue a new request with the new content, which will produce a separate stream to be consumed. Each turn of the crank, where you add in the function call contents and results, is a separate request with a separate response stream.

[raffaeler]

ok, thank you.