Reuse provider HTTP sessions and add connection-level STT diagnostics

[MyButtermilk]

Summary

Scriber currently creates short-lived aiohttp.ClientSession instances around individual Live Mic and direct-file transcription workflows. That prevents the HTTP connection pool, DNS cache, TCP/TLS state, and dual-stack connection strategy from being reused between sequential dictations.

Introduce an application-owned provider HTTP transport with one reusable session per asyncio event loop, then inject that session into STT adapters and direct-upload workflows. Add privacy-safe connection diagnostics so benchmarks can distinguish provider time from DNS, connection setup, upload, and local finalization time.

This is the Python/aiohttp equivalent of the connection-reuse and Happy-Eyeballs work that materially improved the warm dictation path in DictateKeyboard [1].

Motivation

The current provider adapters already accept an optional aiohttp.ClientSession, but src/pipeline.py commonly creates a new session for each recording or file job. As a result, a second warm transcription can still pay for:

• DNS lookup;
• TCP connection establishment;
• TLS negotiation;
• a new per-session connector and connection pool.

aiohttp.ClientSession is designed to encapsulate a connection pool and be reused for the lifetime of an application or logical client [2]. aiohttp >= 3.10 also exposes RFC 8305 Happy Eyeballs controls through TCPConnector, so Scriber can retain normal system DNS ordering and work correctly on IPv4-only, IPv6-only, and dual-stack networks without an IPv4-first resolver [3].

Proposed implementation

1. Add a provider HTTP transport owner

Create a small module such as:

src/runtime/provider_http.py

Suggested responsibilities:

• create and own one ClientSession per asyncio event loop;
• never share a session or connector across threads/event loops;
• expose an explicit async close() used during backend shutdown;
• provide the shared session to ScriberPipeline, direct file/YouTube jobs, meeting finalization, and provider adapters;
• make tests able to inject a fake or local-session implementation without touching production globals.

A registry keyed by the current event loop is acceptable, but lifecycle ownership should remain explicit. Avoid a module-level session created at import time.

2. Configure the connector for warm reuse and dual-stack correctness

Initial configuration to benchmark:

connector = aiohttp.TCPConnector(
    family=socket.AF_UNSPEC,
    happy_eyeballs_delay=0.25,
    interleave=1,
    ttl_dns_cache=300,
    keepalive_timeout=60,
    limit=32,
    limit_per_host=8,
)

The exact pool sizes and TTL should remain easy to tune, but the important contracts are:

• preserve system DNS results rather than forcing IPv4;
• enable connection reuse;
• keep a bounded global and per-host pool;
• use separate request-level connect, upload/read, and total timeouts;
• retain existing proxy, certificate, and cancellation behavior.

Consider aiohttp.DummyCookieJar() because provider APIs do not need browser-style cookies.

3. Pin the required aiohttp capability explicitly

Add an explicit runtime dependency, for example:

aiohttp>=3.10,<4

Scriber should not rely on a transitive Pipecat dependency for a networking feature that is part of the product's latency and reliability contract.

4. Inject the session through existing boundaries

Migrate incrementally:

1. ScriberPipeline.start() and transcribe_file();
2. transcribe_file_direct() and direct provider helpers;
3. meeting finalization and remaining provider-specific jobs;
4. summarization only if doing so does not couple unrelated timeout/retry policies.

Provider helpers should continue accepting an explicit session argument. Avoid hiding session acquisition deep inside each adapter.

5. Add privacy-safe TraceConfig diagnostics

Use aiohttp.TraceConfig to record bounded phase markers such as:

• request started;
• DNS cache hit/miss and DNS duration;
• connection queued/created/reused;
• first and last request chunk sent;
• response headers received;
• response body complete;
• request exception/cancellation.

Attach a generated request/flow ID and provider name, but never log:

• API keys or authorization headers;
• request or response bodies;
• audio bytes or filenames containing personal data;
• transcript text;
• full query strings.

The tracing API exposes connection reuse, DNS, connection creation, and chunk events directly [4].

6. Keep retry policy outside this issue

This issue must not add broad application-level retries for billable transcription POSTs. A failed request whose upload may have reached the provider must be surfaced to the caller rather than silently replayed.

Non-goals

• Replacing Python networking with OkHttp.
• Migrating every provider to HTTP/2.
• Adding an IPv4-first resolver.
• Automatically retrying or cross-provider replaying transcription requests.
• Changing provider request/response schemas.

Acceptance criteria

• Consecutive STT requests to the same origin can reuse a pooled connection.
• A warm second request does not perform DNS resolution or create a new socket when the existing connection remains healthy.
• Live Mic sessions and direct file jobs use the shared transport rather than creating a session per job.
• Sessions are never shared across event loops or threads.
• Backend shutdown closes all provider sessions without Unclosed client session or connector warnings.
• Dual-stack behavior uses system DNS plus Happy Eyeballs; no address-family preference is installed.
• Connect timeout is independently bounded from upload/read/total request timeouts.
• Trace output proves connection creation versus reuse without exposing credentials, audio, filenames, or transcript content.
• Existing unit, provider replay, installed workflow, and packaging checks continue to pass.

Suggested tests

• Local aiohttp.web or equivalent test server: two sequential requests use one connection.
• Trace assertions: first call reports connection creation; second reports on_connection_reuseconn.
• Session lifecycle test: explicit close is idempotent and leaves no pending transports.
• Event-loop isolation test: a session created on loop A is not returned on loop B.
• IPv4-only, IPv6-only, and dual-stack integration matrix where available.
• Cancellation during upload does not leave a pooled response or task hanging.
• Packaged-sidecar smoke test to ensure the pinned aiohttp version and optional resolver dependencies are present.

Benchmark evidence to collect

For one cold call and at least ten warm calls, record:

requestStarted
DNS start/end/cache hit
connectionCreated or connectionReused
firstRequestChunkSent
lastRequestChunkSent
responseHeadersReceived
responseBodyComplete
transcriptCommitted

Compare median and p95 stop-to-committed-text latency before and after the change using identical audio and provider configuration.

References

1. Speed up OpenRouter dictation by 22% without forcing IPv4 DevEmperor/DictateKeyboard#210
2. https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession
3. https://docs.aiohttp.org/en/v3.10.10/client_reference.html#aiohttp.TCPConnector
4. https://docs.aiohttp.org/en/stable/tracing_reference.html