Add support for customizing HTTP headers by kyokuping · Pull Request #1196 · apple/pkl

kyokuping · 2025-08-31T16:28:45Z

This PR adds support for custom HTTP headers, introducing a --http-header CLI flag to accept key=value pairs. These headers can also be specified within the setting.pkl file.

Closes #633

SPICE: apple/pkl-evolution#24

HT154

Thanks for the contribution! 🎉

I have a few primary concerns here:

Although rare, it is valid to send a specific header more than once in a request. When there are headers with the same name their order may also matter.
How does your implementation handle "reserved" headers that users should not override like Origin or User-Agent?
It seems less than ideal to send the same headers to all hosts, eg. if host A requires auth but host B does not, Pkl should not send a configured Authorization header to both hosts. I think a prefix matching scheme like the one used for rewrites is necessary here to ensure the proper security controls are in place.

To this end, I think the Pkl API for this needs to be something along these lines:

headers: Mapping<HttpRewrite, Listing<Pair<String, String>>>

This complicates the CLI a bit, of course, but maybe something like this is reasonable?

--http-header <prefix>='<header name>: <header value>'

HT154 · 2025-09-17T15:58:21Z

Just a heads up, we want reviews from the entire core team because this feature affects the core language and has security implications. One member of the team is currently out on leave so it's going to be a little bit before we can fully review this, but we haven't forgotten about it!

HT154 · 2025-09-07T07:46:49Z

stdlib/EvaluatorSettings.pkl

+  /// HTTP headers to add to outbound requests targeting specified URLs.
+  headers: Mapping<HttpRewrite, Listing<Pair<String, String>>>?
 }


We need the @Since annotation here for documentation purposes. It would also be good to add specificity and examples to the doc comment, similar to rewrites above.

Suggested change

/// HTTP headers to add to outbound requests targeting specified URLs.

headers: Mapping<HttpRewrite, Listing<Pair<String, String>>>?

}

/// HTTP headers to add to outbound requests targeting specified URLs.

@Since { version = "0.30.0" }

headers: Mapping<HttpRewrite, Listing<Pair<String, String>>>?

}

I'm also curious if it makes sense to rename HttpRewrite to HttpPrefix (with typealias marked @Deprecated for backwards compatibility) and refactoring the corresponding Java APIs. Interested in input from @bioball and @stackoverflow on this.

HT154 · 2025-09-17T19:59:58Z

pkl-commons-cli/src/main/kotlin/org/pkl/commons/cli/commands/BaseOptions.kt

+              val headerParts = header.split(":", limit = 2)
+              require(headerParts.size == 2) { "Header '$header' is not in 'name:value' format. " }
+              PPair(headerParts[0], headerParts[1])


curl accepts this same syntax, but also accepts <name>:[ \t]+<value> and appears to strip the leading (but not trailing) whitespace from header values before sending the request. To avoid surprise for users of Pkl, it might be good to do the same. In theory, HTTP servers should be ignoring this leading (and trailing) whitespace when parsing headers so it shouldn't matter, but it might be best to avoid taking chances here.

I'm also wondering if it makes sense to do some validation of characters in headers here (and in the EvaluatorSettings API) to eagerly report invalid characters instead of lazily doing so when a matching request is made for the first time.

stackoverflow

The design sounds sensible at first glance, but due to the complexity and security issues it may cause it may be a good idea to create a SPICE for it.
What do you think @bioball, @HT154?

bioball

Hey, thanks for the PR! I agree with @stackoverflow; it'd be good to have this discussed in a SPICE.

If you'd like to drive this feature, can you write one up? It should be submitted as PR to pkl-evolution.

Your SPICE should start by copy/pasting the template. Your SPICE number would be the next number after the latest proposed SPICE see (https://github.com/apple/pkl-evolution/pulls)

bioball · 2025-09-22T18:48:10Z

pkl-core/src/main/java/org/pkl/core/http/JdkHttpClient.java

+          }
+        }
+      }
+      return underlying.send(wrappedRequestBuilder.build(), responseBodyHandler);


The logic of adding headers should be handled in RequestRewritingClient

kyokuping · 2025-09-25T04:29:10Z

I will resume working on this PR after getting the SPICE merged!

stackoverflow · 2025-09-25T08:01:06Z

As a heads up: the SPICE can only be merged once this pr is merged and released on a new version. So you only need the SPICE to be approved.

bioball · 2025-11-13T16:37:28Z

@kyokuping was the close intentional?

kyokuping · 2025-11-13T17:35:48Z

Sorry, accidentally closed the PR. 😢

bioball · 2025-11-13T18:22:33Z

Gotcha, feel free to re-open a new PR!

discount-elf · 2026-01-13T21:48:10Z

Happy to see this, just went looking for http(s) auth support :D

danievanzyl · 2026-03-12T07:50:06Z

When will this be merged? Really looking forward to access private repos 🙏🏻

HT154

Hey @kyokuping would you mind rebasing this to resolve the conflict(s)?

I have some pretty minor feedback, but from a technical standpoint this looks pretty good. I'd like to try and ship this in 0.32 in a few months, so it'd be great if this was ready for another set of eyes by the time the core team's all back together in a few weeks!

pkl-commons-cli/src/main/kotlin/org/pkl/commons/cli/commands/BaseOptions.kt

pkl-core/src/main/java/org/pkl/core/http/JdkHttpClient.java

stdlib/EvaluatorSettings.pkl

pkl-commons-cli/src/main/kotlin/org/pkl/commons/cli/commands/BaseOptions.kt

Co-authored-by: Jen Basch <jbasch94@gmail.com>

- Introduce pattern-based URL matches - Both accept a single and multiple values for header values - Move header append logic to RequestRewritingClient

pkl-commons-cli/src/main/kotlin/org/pkl/commons/cli/commands/BaseOptions.kt

HT154 · 2026-03-16T21:01:21Z

pkl-core/src/main/java/org/pkl/core/util/IoUtils.java

+    if (!headerName.equals(headerName.toLowerCase())) {
+      throw new IllegalArgumentException(
+          "HTTP header '%s' should be all lowercase".formatted(headerName));
+    }


Is this validation actually necessary? Why would this be preferable to normalizing header names during CLI parsing?

Based on the previous review on the SPICE, I updated the settings to enforce lowercase values and therefore expected the CLI to match this behavior. If we want to change this in the CLI, we should probably align the EvaluatorSettings implementation too. Which way do you prefer to go with?

In the evaluator settings, "pre"-normalization is required to ensure the Mapping has unique keys. On the CLI, each instance of the option is parsed linearly, so the CLI could perform its own normalization without issue.

In general, I think it's best that we accept the broadest input possible for each input method (even if those are not consistent). It's very possible the other core folks could disagree, so I'm happy to put a pin in this until they're back over the next few weeks.

Co-authored-by: Jen Basch <jbasch94@gmail.com>

HT154 reviewed Aug 31, 2025

View reviewed changes

kyokuping requested a review from HT154 September 6, 2025 14:45

HT154 reviewed Sep 17, 2025

View reviewed changes

kyokuping force-pushed the feat/headers branch from 70161b3 to c2632cc Compare September 19, 2025 06:38

stackoverflow reviewed Sep 22, 2025

View reviewed changes

bioball reviewed Sep 22, 2025

View reviewed changes

kyokuping mentioned this pull request Sep 23, 2025

SPICE-0022: Custom HTTP Headers apple/pkl-evolution#24

Open

kyokuping marked this pull request as draft September 25, 2025 04:30

kyokuping closed this Nov 13, 2025

kyokuping force-pushed the feat/headers branch from 94962c3 to ecf2d8b Compare November 13, 2025 15:57

kyokuping reopened this Nov 14, 2025

kyokuping force-pushed the feat/headers branch from e8ea9f6 to ffbf0cf Compare January 17, 2026 14:25

kyokuping force-pushed the feat/headers branch from ffbf0cf to be62d56 Compare March 3, 2026 09:02

kyokuping marked this pull request as ready for review March 3, 2026 13:36

kyokuping requested a review from HT154 March 3, 2026 13:36

HT154 reviewed Mar 12, 2026

View reviewed changes

kyokuping and others added 6 commits March 14, 2026 00:19

Add support for customizing HTTP headers

661a5c6

Allow customizing headers for different URLs

7edd7fc

add @Since annotation in pkl option

857805c

Co-authored-by: Jen Basch <jbasch94@gmail.com>

validate header syntax

a5b6e63

Rename HttpRewrite to HttpPrefix for clarity

a68daf1

Add strict HTTP header validation to EvaluatorSettings

4a60635

Update implementation based on the SPICE changes

f44e39f

- Introduce pattern-based URL matches - Both accept a single and multiple values for header values - Move header append logic to RequestRewritingClient

kyokuping force-pushed the feat/headers branch from 6818324 to f44e39f Compare March 13, 2026 15:20

kyokuping added 5 commits March 16, 2026 21:56

Revert Copyright date change

fb2f89d

Update @Since annotations

dc62816

Rewrite lambdas as expressions

0486d28

Disallow using commas as header separator

e44d577

Implement strict header validation in CLI

6772430

kyokuping requested a review from HT154 March 16, 2026 15:10

HT154 reviewed Mar 16, 2026

View reviewed changes

Use computeIfAbsent

1e51d66

Co-authored-by: Jen Basch <jbasch94@gmail.com>

Conversation

kyokuping commented Aug 31, 2025 • edited by HT154 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HT154 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HT154 commented Sep 17, 2025

Uh oh!

HT154 Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

HT154 Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

HT154 Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

stackoverflow left a comment

Choose a reason for hiding this comment

Uh oh!

bioball left a comment

Choose a reason for hiding this comment

Uh oh!

bioball Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

kyokuping commented Sep 25, 2025

Uh oh!

stackoverflow commented Sep 25, 2025

Uh oh!

bioball commented Nov 13, 2025

Uh oh!

kyokuping commented Nov 13, 2025

Uh oh!

bioball commented Nov 13, 2025

Uh oh!

discount-elf commented Jan 13, 2026

Uh oh!

danievanzyl commented Mar 12, 2026

Uh oh!

HT154 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HT154 Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

kyokuping Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

HT154 Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kyokuping commented Aug 31, 2025 •

edited by HT154

Loading

HT154 left a comment •

edited

Loading