Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ For policy configuration parameters, see xref:policies-included-directory.adoc[]

== LLM Proxy Authentication Policy

By default, the LLM Proxy has the Client ID Enforcement policy applied.
By default, the LLM Proxy has the Client ID Enforcement policy applied.
This is required because the Client ID Enforcement populates the `Authentication.clientName` variable in the `Authentication` object that is used as a unique identifier for LLM Metrics.

To remove the Client ID Enforcement policy, ensure that you either:
Expand Down
1 change: 1 addition & 0 deletions gateway/1.13/modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@
*** xref:policies-included-bedrock-guardrails.adoc[Amazon Bedrock Guardrails]
*** xref:policies-included-azure-content-safety.adoc[Azure Content Safety]
*** xref:policies-included-llm-pii-detection.adoc[LLM PII Detection]
*** xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy]
*** xref:policies-included-llm-token-rate-limit.adoc[LLM Token Based Rate Limit]
*** xref:policies-included-basic-auth-ldap.adoc[Basic Authentication: LDAP]
*** xref:policies-included-basic-auth-simple.adoc[Basic Authentication: Simple]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ endif::[]
By default, LLM Proxy applies these policies:

* Client ID Enforcement
* LLM Proxy Core Policy
* xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy]
* Model Based Routing Policy or Semantic Routing Policy (policy name dependent on embedded service provider)

You don't need to modify these policies.
Expand All @@ -19,6 +19,7 @@ NOTE: LLM Proxy doesn't support xref:policies-included-rate-limiting-sla.adoc[].

These policies are specific and useful for LLM Proxies:

* xref:policies-included-llm-proxy-core.adoc[]
* xref:policies-included-bedrock-guardrails.adoc[]
* xref:policies-included-azure-content-safety.adoc[]
* xref:policies-included-llm-pii-detection.adoc[]
Expand All @@ -40,8 +41,8 @@ For policy configuration parameters, see xref:policies-included-directory.adoc[]

== LLM Proxy Authentication Policy

By default, the LLM Proxy has the Client ID Enforcement policy applied.
This is required because the Client ID Enforcement populates the `Authentication.clientName` variable in the `Authentication` object that is used as a unique identifier for LLM Metrics.
By default, the LLM Proxy has the Client ID Enforcement policy applied.
This is required because the Client ID Enforcement populates the `Authentication.clientName` variable in the `Authentication` object that is used by the xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy] as a unique identifier for LLM Metrics.

To remove the Client ID Enforcement policy, ensure that you either:

Expand All @@ -54,7 +55,7 @@ To remove the Client ID Enforcement policy, ensure that you either:
** xref:policies-included-jwt-validation.adoc[] (If Client ID enforcement is configured, `skipClientIdValidation=false`)
** A custom policy that populates `Authentication.clientName`

* Edit the dataweave variable in LLM Proxy Core to extract a different unique identifier, such as `clientid`, `userid`, or `departmentid`.
* Edit the `Client Identifier` parameter in the xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy] to extract a different unique identifier, such as `#[attributes.headers['client-id']]`, `#[attributes.headers['user-id']]`, or `#[attributes.headers['department-id']]`.
+
NOTE: You can't filter by this unique identifier in Usage Reports.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ endif::[]
[[view-token-usage-reports]]
== View Token Usage Reports

With Token Usage reports, you can view the amount of API tokens each LLM Proxy uses for individual models.
With Token Usage reports, you can view the amount of API tokens each LLM Proxy uses for individual models. Token usage is collected by the xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy].

NOTE: To limit token usage, apply the xref:policies-included-llm-token-rate-limit.adoc[LLM Token Based Rate Limit policy] to your LLM Proxy.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ endif::[]
| xref:policies-included-bedrock-guardrails.adoc[Amazon Bedrock Guardrails] | LLM | Evaluates LLM prompts and responses against Amazon Bedrock guardrails for content safety, PII detection, and contextual grounding
| xref:policies-included-azure-content-safety.adoc[Azure Content Safety] | LLM | Evaluates LLM prompts and responses against Azure AI Content Safety for harmful content, jailbreak attempts, hallucinations, and copyrighted material
| xref:policies-included-llm-pii-detection.adoc[LLM PII Detection] | LLM | Detects personally identifiable information (PII) in OpenAI-format LLM Proxy requests and responses
| xref:policies-included-llm-proxy-core.adoc[LLM Proxy Core Policy] | LLM | Establishes core functionality for LLM Proxy, including input format detection and metrics collection
| xref:policies-included-llm-token-rate-limit.adoc[LLM Token Based Rate Limit] | LLM | Limits LLM Proxy usage based on the number of tokens consumed (OpenAI-format APIs)
| xref:policies-included-agent-connection-telemetry.adoc[Agent Connection Telemetry] | A2A and MCP | Tracks the number of requests made to agent or MCP server instances
| xref:policies-included-basic-auth-ldap.adoc[Basic Authentication: LDAP] | Security | Allows access based on the basic authorization mechanism, with user-password defined on LDAP
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
= LLM Proxy Core Policy
ifndef::env-site,env-github[]
include::_attributes.adoc[]
endif::[]
:imagesdir: ../assets/images
:keywords: api gateway, flex gateway, gateway, policy, llm, metrics

[width="100%", cols="5,15"]
|===
>s| Policy Name | LLM Proxy Core Policy
>s|Summary | Establishes core functionality and metrics collection for LLM Proxies
>s|First Flex Gateway version available | v1.11.0
>s| Returned Status Codes | No return codes exist for this policy.
|===

== Summary

The LLM Proxy Core Policy is the foundational policy in the LLM Proxy policy chain and enables LLM Proxy metric collection. The policy is applied automatically when creating an LLM Proxy.

NOTE: This policy doesn't collect metrics for Gemini format LLM Proxies.


== Configuring Policy Parameters

include::partial$policy-title-headers.adoc[tag=configFileTitleOnly]

The LLM Proxy Core policy isn't supported in Local Mode.

include::partial$policy-title-headers.adoc[tag=ui]

[%header%autowidth.spread,cols="a,a,a"]
|===
| Element | Required | Description
| Client Identifier | No | DataWeave expression to resolve the client name used in usage metrics. +
Default: `#[authentication.clientName]` +
This identifier is used to group token usage metrics per client in the LLM Token Usage Report.
|===

== LLM Proxy Authentication Requirements

The LLM Proxy Core Policy requires a unique client identifier to properly attribute token usage in metrics. By default, it uses `authentication.clientName`, which is populated by the Client ID Enforcement policy.

If you remove the Client ID Enforcement policy, ensure that you either:

* Apply a policy that populates `authentication.clientName`:
** xref:policies-included-client-id-enforcement.adoc[]
** xref:policies-included-oauth-token-introspection.adoc[] (if Client ID enforcement is configured, `skipClientIdValidation=false`)
** xref:policies-included-openid-token-enforcement.adoc[] (if Client ID enforcement is configured, `skipClientIdValidation=false`)
** xref:policies-included-jwt-validation.adoc[] (if Client ID enforcement is configured, `skipClientIdValidation=false`)
** A custom policy that populates `authentication.clientName`

* Edit the `Client Identifier` parameter in the LLM Proxy Core Policy to extract a different unique identifier, such as `#[attributes.headers['client-id']]` or `#[attributes.headers['user-id']]`.

NOTE: If you use a custom identifier, you can't filter by `authentication.clientName` in the LLM Token Usage Report.

== See Also

* xref:flex-gateway-llm-proxy.adoc[LLM Proxy] - Overview of LLM Proxy and routing
* xref:flex-gateway-llm-proxy-policies.adoc[LLM Proxy Policies] - LLM Proxy policy chain
* xref:flex-gateway-llm-proxy-token-reports.adoc[Viewing Token Usage and LLM Metrics] - Token usage reports and limiting usage
* xref:policies-included-llm-token-rate-limit.adoc[LLM Token Based Rate Limit] - Token-based rate limiting for LLM traffic