Skip to content

AGENTS.md

Latest commit

 

History

History
159 lines (126 loc) · 16.5 KB

AGENTS.md

File metadata and controls

159 lines (126 loc) · 16.5 KB
Raw

LLM TCK Contributor Guide

Scope

This repository builds ManagedCode.LlmTck, a deterministic Technology Compatibility Kit for LLM APIs. Treat the runtime, protocol adapters, clients, Aspire integration, docs, and samples as separate package surfaces.

Follows MCAF.

Conversations (Self-Learning)

Learn the user's stable habits, preferences, and corrections. Record durable rules here instead of relying on chat history.

Before doing any non-trivial task, evaluate the latest user message. If it contains a durable rule, correction, preference, or workflow change, update AGENTS.md first. If it is only task-local scope, do not turn it into a lasting rule.

Update this file when the user gives:

  • a repeated correction
  • a permanent requirement
  • a lasting preference
  • a workflow change
  • a high-signal frustration that indicates a rule was missed

Extract rules aggressively when the user says things equivalent to:

  • "never", "don't", "stop", "avoid"
  • "always", "must", "make sure", "should"
  • "remember", "keep in mind", "note that"
  • "from now on", "going forward"
  • "the workflow is", "we do it like this"

Preferences belong in ## Preferences:

  • positive preferences go under Likes
  • negative preferences go under Dislikes
  • comparisons should become explicit rules or preferences

Corrections should update an existing rule when possible instead of creating duplicates.

Treat these as strong signals and record them immediately:

  • anger, swearing, sarcasm, or explicit frustration
  • ALL CAPS, repeated punctuation, or "don't do this again"
  • the same mistake happening twice
  • the user manually undoing or rejecting a recurring pattern

Do not record:

  • one-off instructions for the current task
  • temporary exceptions
  • requirements that are already captured elsewhere without change

Rule format:

  • one instruction per bullet
  • place it in the right section
  • capture the why, not only the literal wording
  • remove obsolete rules when a better one replaces them

Commands

  • Restore: dotnet restore ManagedCode.LlmTck.slnx
  • Build: dotnet build ManagedCode.LlmTck.slnx --configuration Release --no-restore
  • Test: dotnet test tests/ManagedCode.LlmTck.Tests/ManagedCode.LlmTck.Tests.csproj --configuration Release --no-build --verbosity normal
  • Pack: for project in src/*/*.csproj; do dotnet pack "$project" --configuration Release --no-build --output artifacts/packages; done
  • Release: pushing to main must start the release workflow; it reads the package version from Directory.Build.props, creates the matching vX.Y.Z tag and GitHub Release, attaches package artifacts, and publishes NuGet packages only when that version is not already released.
  • Format check: dotnet format ManagedCode.LlmTck.slnx --verify-no-changes
  • Tool restore: dotnet tool restore
  • Coverage gate: production code line coverage must stay at or above 90%; measure with coverlet/reportgenerator when coverage expectations change.
  • Coverage: rm -rf tests/ManagedCode.LlmTck.Tests/TestResults/coverage-analysis && mkdir -p tests/ManagedCode.LlmTck.Tests/TestResults/coverage-analysis/raw && dotnet tool run coverlet tests/ManagedCode.LlmTck.Tests/bin/Release/net10.0/ManagedCode.LlmTck.Tests.dll --target "dotnet" --targetargs "test tests/ManagedCode.LlmTck.Tests/ManagedCode.LlmTck.Tests.csproj --configuration Release --no-build --verbosity normal" --format cobertura --format json --output tests/ManagedCode.LlmTck.Tests/TestResults/coverage-analysis/raw/coverage --include "[ManagedCode.LlmTck]*" --include "[ManagedCode.LlmTck.*]*" --exclude "[ManagedCode.LlmTck.Tests]*" --exclude "[ManagedCode.LlmTck.Service]*" --exclude "[ManagedCode.LlmTck.AppHost]*" --threshold 90 --threshold-type line --threshold-stat Total
  • Quality gate: dotnet format ManagedCode.LlmTck.slnx --verify-no-changes --no-restore && dotnet build ManagedCode.LlmTck.slnx --configuration Release --no-restore && dotnet test tests/ManagedCode.LlmTck.Tests/ManagedCode.LlmTck.Tests.csproj --configuration Release --no-build --verbosity normal && git diff --check

global.json opts dotnet test into Microsoft.Testing.Platform; do not add VSTest-specific logger arguments to normal test commands.

Preferences

Likes

  • Keep checked-in UI screenshots synchronized with the current rendered product; after visible dashboard changes, refresh and verify the affected images in a real browser before commit and push.
  • When token usage is requested, count response tokens with tiktoken and include the measured usage in replies; if tiktoken is unavailable, report that blocker instead of guessing token counts.
  • Token usage workflow changes need executable code and tests, not only contributor-guide documentation, so callers can verify usage accounting behavior directly.

Dislikes

  • Do not introduce real-LLM toggles, external project environment variable names, or WA.Storied-specific constants into this repository; LlmTck tests and samples must remain deterministic, repo-owned, and free of cross-project naming leaks.

Code Quality Skills

  • Repository-local skills live in .codex/skills; keep them checked in with the repo.
  • MCAF governance skills remain the mcaf-* set installed from the canonical MCAF tutorial.
  • Code-quality skills are installed from the current Managed Code dotnet-skills catalog for .NET Quality, Frontend Quality, Diagnostics & Metrics, Testing, and Testing Research.
  • Use analyzer-config, code-analysis, format, quality-ci, roslynator, meziantou-analyzer, stylecop-analyzers, complexity, and resharper-clt when changing analyzer, naming, formatting, or warning policy.
  • Use run-tests, tunit, coverage-analysis, coverlet, reportgenerator, stryker, assertion-quality, test-anti-patterns, test-gap-analysis, and test-smell-detection when changing test behavior or coverage expectations.
  • Use biome, eslint, htmlhint, stylelint, webhint, and sonarjs for browser-facing files under site or future frontend assets.
  • Use codeql, quickdup, cloc, analyzing-dotnet-performance, microbenchmarking, and profiling for deeper static, duplication, size, or performance analysis.
  • Root .editorconfig is the source of truth for formatting, naming, and C# style. Prefer changing it once over suppressing analyzer noise in individual files.

Design Rules

  • Prefer official .NET packages and abstractions before writing custom protocol code.
  • Tests that fake IChatClient should use the dotnet/extensions pattern: a tiny fake class with delegate callbacks, not Moq or NSubstitute.
  • Keep provider wire formats in provider packages such as ManagedCode.LlmTck.OpenAI; keep deterministic scenario behavior in ManagedCode.LlmTck.
  • Keep the provider package matrix explicit. Do not ship the TCK as only OpenAI-compatible; maintain package surfaces for OpenAI, Azure OpenAI, Microsoft Foundry, Anthropic, Gemini, Groq, Mistral, Ollama, Cohere, Amazon Bedrock, OpenRouter, DeepSeek, and Perplexity.
  • Azure OpenAI and Microsoft Foundry compatibility must be proven with official Azure SDK clients, not only raw HTTP requests.
  • Built-in defaults, samples, and scenario examples must use official provider model IDs such as gpt-4.1-mini, text-embedding-3-small, gpt-image-1, and gpt-4o-mini-tts; do not use synthetic llm-tck-* model IDs for normal fixtures.
  • Keep custom AddModel(...) support, but expose typed convenience APIs for common official fixture models so callers do not have to hand-type model IDs for standard chat, embedding, image, and audio setup.
  • Reasoning-capable models must carry deterministic reasoning-token usage through runtime summaries and provider response usage details; do not flatten reasoning usage into ordinary visible output tokens only.
  • Prompt-cache accounting is provider-specific: expose cache read/write usage for every provider surface that has a documented cache contract, and report no cache usage for providers or operations without one.
  • Compatibility tag values in C# code must come from named constants everywhere they are assigned or asserted, so CompatibilityTags cannot drift through inline string literals.
  • Provider ApiContract operation id values and behavior-evidence operation ids in C# must come from LlmTckProviderOperationIds; do not type raw operation-id strings such as videos.content.retrieve.
  • The client package must expose a universal pre-test configuration API so tests can spawn a client, reset or configure the hosted TCK, load models, auth, datasets, scenarios, scripted errors, embeddings, images, and audio fixtures without hand-authoring raw DTOs.
  • The browser admin panel must be served from /, and runtime control APIs must use /admin-api/... paths; do not add hidden or aliased admin/control routes for reset, configure, models, assertions, or operator UI.
  • The browser admin panel must be implemented as Blazor server-side components in ManagedCode.LlmTck.Hosting; do not serve it from a static HTML string endpoint, and use component state plus StateHasChanged() for refresh behavior instead of client-side polling fetch code.
  • The browser admin panel is a full-viewport debugging surface, not a narrow landing page; show models/fixtures, actions, calls/requests, responses, provider/control API endpoints, assertions, and runtime events as separate grouped, inspectable areas so users can inspect each part without digging through one collapsed feed.
  • The browser admin dashboard must be request-centric: use a scannable table with one entry per provider request, show provider and model at a glance, and let operators inspect the full request, response, multi-turn conversation, cache accounting, token usage, timing, status, and related metadata without reconstructing a call from separate feeds; keep retention explicitly bounded and visibly mark the exceptional payload that exceeds that diagnostic budget.
  • Keep the request inbox visually primary and easy to scan; do not compress it beside a narrow, endlessly tall inspector. Constrain long conversations and payloads to independent scroll regions, collapse oversized system/tool content by default while keeping it expandable, and validate the resulting hierarchy with realistic long multi-turn data at desktop, tablet, and mobile sizes.
  • Keep the dashboard's useful horizontal master-detail structure, but use an Apple-inspired content-first visual language rather than Material or Fluent styling: quiet system typography, generous clean surfaces, restrained separators and status color, sidebar depth only where it clarifies hierarchy, and no grid of badges or boxed metrics. Keep the filter/navigation rail visible at ordinary laptop widths and the selected response visible in the first reading-pane viewport; on narrow mobile screens use an explicit list-to-detail transition with a Back action.
  • Render JSON payloads as properly indented code with an SF Mono-style font and preserve readable Unicode characters instead of exposing serializer escape sequences such as \u0027.
  • Use Microsoft Fluent UI Blazor primitives for dashboard overlays and dismissible controls, but theme them to the dashboard's Apple-inspired visual language instead of adopting stock Fluent styling. Popovers must close on outside click, Escape, and completed actions. Blend Prostir.build's restrained glass, lime selection rail, and technical polish with managed-code.com's warm canvas, ink typography, and tiny coral-to-magenta accents without copying either site's branding or marketing-page layout.
  • The browser admin panel must auto-refresh by default every 15 seconds, but do not add visible countdown UI unless the user explicitly asks for it.
  • Provider APIs must be explicitly namespaced by provider, such as /openai, /anthropic, /gemini, or /azure-openai; do not expose generic root /v1/* provider routes that make the TCK look like only the OpenAI API.
  • Aspire examples must show endpoint retrieval from the Aspire resource (GetEndpoint("http") or CreateHttpClient(...)) and API key wiring so users see the complete integration path.
  • Aspire sample services must use ManagedCode.LlmTck.Hosting service-defaults methods such as AddServiceDefaults() and MapDefaultEndpoints() for health/liveness wiring; do not create a separate ServiceDefaults sample project, and do not hand-roll app.MapGet("/health", ...) or placeholder root status endpoints in sample Program.cs files.
  • Aspire integration must expose a package-owned builder.AddLlmTck() entry point so consumers can install the Aspire NuGet package and add the TCK without caller-supplied project paths or Projects.* metadata types.
  • builder.AddLlmTck() must be .NET/Aspire-first and must not require Docker or a container runtime by default; container-backed hosting may exist only as an explicit opt-in API.
  • Aspire integration must start an Aspire AppHost in tests before a change is considered covered, with the AppHost model built directly in test code or a test fixture.
  • Aspire test coverage should build the AppHost directly inside the test code or fixture; do not add a standalone Obhost/test AppHost file just to host Aspire for tests.
  • Aspire tests and samples must resolve service/provider URLs from the Aspire resource endpoint allocated by the AppHost; do not hard-code 127.0.0.1, localhost, ports, or invented provider endpoints in configuration.
  • Do not add an Aspire provider-endpoint setter to ManagedCode.LlmTck.Aspire; consumers must use the endpoint exposed by the LlmTckResource.
  • Do not add Aspire compatibility environment flags, sample-service config echoes, or fluent compatibility methods unless they drive real route/runtime behavior; prove compatibility through actual provider clients instead.
  • Provider API compatibility must be doc-backed: every claimed provider method, route, streaming mode, modality, and API-version constraint must be represented in a checked-in contract with official documentation links and covered by tests before the provider package claims support.
  • Provider API verification must cover the full claimed provider matrix: every implemented ApiContract operation for every provider needs behavior-test evidence, and fixes that touch provider compatibility must rerun the provider contract/evidence tests plus the full quality gate before closeout.
  • Provider fault simulation must be provider-neutral and always available across the full claimed provider matrix: configurable rate-limit (too_many_requests/429) and content-filter (content_filter) behavior must flow through runtime configuration and be proven on all provider families, not only OpenAI-compatible routes.
  • Do not hide nondeterminism behind retries. Model responses, stream chunks, errors, auth requirements, embeddings, images, and audio fixtures should be explicit.

Ownership Map

  • src/ManagedCode.LlmTck: provider-neutral runtime, scenarios, assertions, and model catalog.
  • src/ManagedCode.LlmTck.OpenAI: OpenAI-compatible request/response shapes and mapping.
  • src/ManagedCode.LlmTck.AzureOpenAI: Azure OpenAI compatibility profile and future Azure OpenAI wire contracts.
  • src/ManagedCode.LlmTck.Foundry: Microsoft Foundry compatibility profile and future Foundry wire contracts.
  • src/ManagedCode.LlmTck.Anthropic: Anthropic Messages compatibility profile and future Anthropic wire contracts.
  • src/ManagedCode.LlmTck.Gemini: Gemini compatibility profile and future Gemini wire contracts.
  • src/ManagedCode.LlmTck.Groq: Groq compatibility profile and future Groq wire contracts.
  • src/ManagedCode.LlmTck.Mistral: Mistral compatibility profile and future Mistral wire contracts.
  • src/ManagedCode.LlmTck.Ollama: Ollama compatibility profile and future Ollama wire contracts.
  • src/ManagedCode.LlmTck.Cohere: Cohere compatibility profile and future Cohere wire contracts.
  • src/ManagedCode.LlmTck.Bedrock: Amazon Bedrock compatibility profile and future Bedrock wire contracts.
  • src/ManagedCode.LlmTck.OpenRouter: OpenRouter compatibility profile and future OpenRouter wire contracts.
  • src/ManagedCode.LlmTck.DeepSeek: DeepSeek compatibility profile and future DeepSeek wire contracts.
  • src/ManagedCode.LlmTck.Perplexity: Perplexity compatibility profile and future Perplexity wire contracts.
  • src/ManagedCode.LlmTck.Hosting: ASP.NET Core provider and control endpoints.
  • src/ManagedCode.LlmTck.Client: control client and Microsoft.Extensions.AI clients.
  • src/ManagedCode.LlmTck.Aspire: Aspire AppHost extension methods.
  • samples: runnable AppHost and service sample.
  • tests: TUnit unit, endpoint, client, and Aspire integration coverage.
  • docs: architecture, feature specs, ADRs, and test inventory.
  • site: static public landing page.