diff --git a/gateway/1.12/modules/ROOT/pages/flex-gateway-llm-proxy-policies.adoc b/gateway/1.12/modules/ROOT/pages/flex-gateway-llm-proxy-policies.adoc index c358e1713..275b7238e 100644 --- a/gateway/1.12/modules/ROOT/pages/flex-gateway-llm-proxy-policies.adoc +++ b/gateway/1.12/modules/ROOT/pages/flex-gateway-llm-proxy-policies.adoc @@ -36,7 +36,7 @@ For policy configuration parameters, see xref:policies-included-directory.adoc[] == LLM Proxy Authentication Policy -By default, the LLM Proxy has the Client ID Enforcement policy applied. +By default, the LLM Proxy has the Client ID Enforcement policy applied. This is required because the Client ID Enforcement populates the `Authentication.clientName` variable in the `Authentication` object that is used as a unique identifier for LLM Metrics. To remove the Client ID Enforcement policy, ensure that you either: diff --git a/gateway/1.13/modules/ROOT/nav.adoc b/gateway/1.13/modules/ROOT/nav.adoc index 2eb5a25bc..392e8d277 100644 --- a/gateway/1.13/modules/ROOT/nav.adoc +++ b/gateway/1.13/modules/ROOT/nav.adoc @@ -105,6 +105,7 @@ *** xref:policies-included-bedrock-guardrails.adoc[Amazon Bedrock Guardrails] *** xref:policies-included-azure-content-safety.adoc[Azure Content Safety] *** xref:policies-included-llm-pii-detection.adoc[LLM PII Detection] +*** xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy] *** xref:policies-included-llm-token-rate-limit.adoc[LLM Token Based Rate Limit] *** xref:policies-included-basic-auth-ldap.adoc[Basic Authentication: LDAP] *** xref:policies-included-basic-auth-simple.adoc[Basic Authentication: Simple] diff --git a/gateway/1.13/modules/ROOT/pages/flex-gateway-llm-proxy-policies.adoc b/gateway/1.13/modules/ROOT/pages/flex-gateway-llm-proxy-policies.adoc index a7c1122b1..dfb850c1c 100644 --- a/gateway/1.13/modules/ROOT/pages/flex-gateway-llm-proxy-policies.adoc +++ b/gateway/1.13/modules/ROOT/pages/flex-gateway-llm-proxy-policies.adoc @@ -8,7 +8,7 @@ endif::[] By default, LLM Proxy applies these policies: * Client ID Enforcement -* LLM Proxy Core Policy +* xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy] * Model Based Routing Policy or Semantic Routing Policy (policy name dependent on embedded service provider) You don't need to modify these policies. @@ -19,6 +19,7 @@ NOTE: LLM Proxy doesn't support xref:policies-included-rate-limiting-sla.adoc[]. These policies are specific and useful for LLM Proxies: +* xref:policies-included-llm-proxy-core.adoc[] * xref:policies-included-bedrock-guardrails.adoc[] * xref:policies-included-azure-content-safety.adoc[] * xref:policies-included-llm-pii-detection.adoc[] @@ -40,8 +41,8 @@ For policy configuration parameters, see xref:policies-included-directory.adoc[] == LLM Proxy Authentication Policy -By default, the LLM Proxy has the Client ID Enforcement policy applied. -This is required because the Client ID Enforcement populates the `Authentication.clientName` variable in the `Authentication` object that is used as a unique identifier for LLM Metrics. +By default, the LLM Proxy has the Client ID Enforcement policy applied. +This is required because the Client ID Enforcement populates the `Authentication.clientName` variable in the `Authentication` object that is used by the xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy] as a unique identifier for LLM Metrics. To remove the Client ID Enforcement policy, ensure that you either: @@ -54,7 +55,7 @@ To remove the Client ID Enforcement policy, ensure that you either: ** xref:policies-included-jwt-validation.adoc[] (If Client ID enforcement is configured, `skipClientIdValidation=false`) ** A custom policy that populates `Authentication.clientName` -* Edit the dataweave variable in LLM Proxy Core to extract a different unique identifier, such as `clientid`, `userid`, or `departmentid`. +* Edit the `Client Identifier` parameter in the xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy] to extract a different unique identifier, such as `#[attributes.headers['client-id']]`, `#[attributes.headers['user-id']]`, or `#[attributes.headers['department-id']]`. + NOTE: You can't filter by this unique identifier in Usage Reports. diff --git a/gateway/1.13/modules/ROOT/pages/flex-gateway-llm-proxy-token-reports.adoc b/gateway/1.13/modules/ROOT/pages/flex-gateway-llm-proxy-token-reports.adoc index fe3b11c33..10bf2eaef 100644 --- a/gateway/1.13/modules/ROOT/pages/flex-gateway-llm-proxy-token-reports.adoc +++ b/gateway/1.13/modules/ROOT/pages/flex-gateway-llm-proxy-token-reports.adoc @@ -9,7 +9,7 @@ endif::[] [[view-token-usage-reports]] == View Token Usage Reports -With Token Usage reports, you can view the amount of API tokens each LLM Proxy uses for individual models. +With Token Usage reports, you can view the amount of API tokens each LLM Proxy uses for individual models. Token usage is collected by the xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy]. NOTE: To limit token usage, apply the xref:policies-included-llm-token-rate-limit.adoc[LLM Token Based Rate Limit policy] to your LLM Proxy. diff --git a/gateway/1.13/modules/ROOT/pages/policies-included-directory.adoc b/gateway/1.13/modules/ROOT/pages/policies-included-directory.adoc index c20d2d1b6..a79739684 100644 --- a/gateway/1.13/modules/ROOT/pages/policies-included-directory.adoc +++ b/gateway/1.13/modules/ROOT/pages/policies-included-directory.adoc @@ -17,6 +17,7 @@ endif::[] | xref:policies-included-bedrock-guardrails.adoc[Amazon Bedrock Guardrails] | LLM | Evaluates LLM prompts and responses against Amazon Bedrock guardrails for content safety, PII detection, and contextual grounding | xref:policies-included-azure-content-safety.adoc[Azure Content Safety] | LLM | Evaluates LLM prompts and responses against Azure AI Content Safety for harmful content, jailbreak attempts, hallucinations, and copyrighted material | xref:policies-included-llm-pii-detection.adoc[LLM PII Detection] | LLM | Detects personally identifiable information (PII) in OpenAI-format LLM Proxy requests and responses +| xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy] | LLM | Establishes core functionality for LLM Proxy, including input format detection and metrics collection | xref:policies-included-llm-token-rate-limit.adoc[LLM Token Based Rate Limit] | LLM | Limits LLM Proxy usage based on the number of tokens consumed (OpenAI-format APIs) | xref:policies-included-agent-connection-telemetry.adoc[Agent Connection Telemetry] | A2A and MCP | Tracks the number of requests made to agent or MCP server instances | xref:policies-included-basic-auth-ldap.adoc[Basic Authentication: LDAP] | Security | Allows access based on the basic authorization mechanism, with user-password defined on LDAP diff --git a/gateway/1.13/modules/ROOT/pages/policies-included-llm-proxy-core.adoc b/gateway/1.13/modules/ROOT/pages/policies-included-llm-proxy-core.adoc new file mode 100644 index 000000000..bbf4e67e9 --- /dev/null +++ b/gateway/1.13/modules/ROOT/pages/policies-included-llm-proxy-core.adoc @@ -0,0 +1,61 @@ += LLM Proxy Core Policy +ifndef::env-site,env-github[] +include::_attributes.adoc[] +endif::[] +:imagesdir: ../assets/images +:keywords: api gateway, flex gateway, gateway, policy, llm, metrics + +[width="100%", cols="5,15"] +|=== +>s| Policy Name | LLM Proxy Core Policy +>s|Summary | Establishes core functionality and metrics collection for LLM Proxies +>s|First Flex Gateway version available | v1.11.0 +>s| Returned Status Codes | No return codes exist for this policy. +|=== + +== Summary + +The LLM Proxy Core Policy is the foundational policy in the LLM Proxy policy chain and enables LLM Proxy metric collection. The policy is applied automatically when creating an LLM Proxy. + +NOTE: This policy doesn't collect metrics for Gemini format LLM Proxies. + + +== Configuring Policy Parameters + +include::partial$policy-title-headers.adoc[tag=configFileTitleOnly] + +The LLM Proxy Core policy isn't supported in Local Mode. + +include::partial$policy-title-headers.adoc[tag=ui] + +[%header%autowidth.spread,cols="a,a,a"] +|=== +| Element | Required | Description +| Client Identifier | No | DataWeave expression to resolve the client name used in usage metrics. + +Default: `#[authentication.clientName]` + +This identifier is used to group token usage metrics per client in the LLM Token Usage Report. +|=== + +== LLM Proxy Authentication Requirements + +The LLM Proxy Core Policy requires a unique client identifier to properly attribute token usage in metrics. By default, it uses `authentication.clientName`, which is populated by the Client ID Enforcement policy. + +If you remove the Client ID Enforcement policy, ensure that you either: + +* Apply a policy that populates `authentication.clientName`: +** xref:policies-included-client-id-enforcement.adoc[] +** xref:policies-included-oauth-token-introspection.adoc[] (if Client ID enforcement is configured, `skipClientIdValidation=false`) +** xref:policies-included-openid-token-enforcement.adoc[] (if Client ID enforcement is configured, `skipClientIdValidation=false`) +** xref:policies-included-jwt-validation.adoc[] (if Client ID enforcement is configured, `skipClientIdValidation=false`) +** A custom policy that populates `authentication.clientName` + +* Edit the `Client Identifier` parameter in the LLM Proxy Core Policy to extract a different unique identifier, such as `#[attributes.headers['client-id']]` or `#[attributes.headers['user-id']]`. + +NOTE: If you use a custom identifier, you can't filter by `authentication.clientName` in the LLM Token Usage Report. + +== See Also + +* xref:flex-gateway-llm-proxy.adoc[LLM Proxy] - Overview of LLM Proxy and routing +* xref:flex-gateway-llm-proxy-policies.adoc[LLM Proxy Policies] - LLM Proxy policy chain +* xref:flex-gateway-llm-proxy-token-reports.adoc[Viewing Token Usage and LLM Metrics] - Token usage reports and limiting usage +* xref:policies-included-llm-token-rate-limit.adoc[LLM Token Based Rate Limit] - Token-based rate limiting for LLM traffic