Async init of chat client

[verdie-g]

To build my IChatClient I need to get credentials with a HTTP call, something like this

services.AddChatClient(sp =>
    {
        var creds = GetSecretsAsync().GetAwaiter().GetResult();
        return new AzureSomethingClient(creds);
    })
    .UseLogging()
    .UseOpenTelemetry();

I would like to avoid that sync-over-async, especially that I have a use-case where the secret depends on the request so the IChatClient is registered as scoped.

One work-around that I tried is to have a custom IChatClient that fetches the credentials in GetResponseAsync so the initialization is just

services.AddChatClient(sp => new CustomChatClient())
    .UseLogging()
    .UseOpenTelemetry();

class CustomChatClient : IChatClient
{
    public virtual async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        var innerClient = await GetOrCreateClientAsync();
        return await innerClient.GetResponseAsync(messages, options, cancellationToken);
    }
}

The problem is that the first method that will be called is IChatClient.GetService which is synchronous so the sync-over-async is not avoided.

What is the recommended way to achieve that?

[stephentoub]

The problem is that the first method that will be called is IChatClient.GetService which is synchronous so the sync-over-async is not avoided.

What's calling GetService and to resolve what?

[stephentoub]

You could add your own IChatClient to the pipeline, either with an anonymous one using the Use method that takes a delegate, or writing a custom one. Have it do the resolution in GetResponse{Streaming}Async before delegating to later in the pipeline, and then store whatever info it gathers into the ChatOptions.AdditionalProperties for your later IChatClient to read.

[verdie-g]

This is getting quite complex :/

I'm thinking I should probably forget about the service collection extensions, and write my own factory

class ChatClientFactory
{
    public async Task<IChatClient> CreateChatClientAsync(/* ... */)
    {
            var credential = GetCredentialAsync();
            return new OpenTelemetryChatClient(new AzureChatClient(credential)));
    }
}

Do you think my use-case is uncommon?

[stephentoub]

Do you think my use-case is uncommon?

I've spoken with a lot of folks using these APIs and you're the first I've encountered that's making per-user, per-call requests to determine which chat backend should be used and where the metadata about which backend is used is material for telemetry purposes. From that perspective it's uncommon. Doesn't mean others aren't doing it, of course, just none I've seen.

and write my own factory

If that's reasonable for you, sounds like a good approach. At the end of the day, seems like what you really want is asynchronous DI support, which as a broader category is a much more common request:
dotnet/runtime#65656

[verdie-g]

I found an easy workaround.

Instead of calling .UseOpenTelemetry() on the ChatClientBuilder, I do a new OpenTelemetryChatClient(client) on the client created after the async operation. That avoids the call to GetService on my custom IChatClient.

[verdie-g]

Ah sad, that workaround does not work with SemanticKernel. Calling _kernel.InvokePromptAsync will first call GetService before GetResponseAsync.

   at Microsoft.Extensions.AI.ChatClientExtensions.GetService[TService](IChatClient client, Object serviceKey)
   at Microsoft.SemanticKernel.ChatCompletion.ChatClientAIService..ctor(IChatClient chatClient)
   at Microsoft.SemanticKernel.KernelFunctionFromPrompt.RenderPromptAsync(Kernel kernel, KernelArguments arguments, Boolean isStreaming, CancellationToken cancellationToken)