queue-proxy can crash with concurrent map iteration and map write during HTTP/2 response finalization

[junyicc]

What version of Knative?

knative version：1.17.1

Observed in a downstream build based on Knative Serving. The queue-proxy logs report:

"commit":"20c92a1-dirty"

The downstream commit resolves to 20c92a10d2ff6aa0162e0c74f9080e1fac7ec441. The relevant timeout handler code matches current upstream main in the suspected critical path: timeoutWriter.Header() directly returns the underlying http.ResponseWriter header map without synchronization.

I also checked upstream main at dbce551; pkg/http/handler/timeout.go still contains:

func (tw *timeoutWriter) Header() http.Header { return tw.w.Header() }

Expected Behavior

queue-proxy should not crash due to concurrent HTTP/2 response header/trailer access. Request cancellation or timeout should fail only the affected request, not the queue-proxy process.

Actual Behavior

queue-proxy crashed and restarted with exitCode=2. The fatal runtime error was:

fatal error: concurrent map iteration and map write

The panic stack points to HTTP/2 response finalization iterating/promoting trailers while the response header map was concurrently modified:

fatal error: concurrent map iteration and map write

goroutine 1123810148 [running]:
internal/runtime/maps.fatal({0x128031e?, 0xc000524530?})
	runtime/panic.go:1058 +0x18
internal/runtime/maps.(*Iter).Next(0xc000948c20?)
	internal/runtime/maps/table.go:683 +0x86
golang.org/x/net/http2.(*responseWriterState).promoteUndeclaredTrailers(0xc001d67780)
	golang.org/x/net@v0.34.0/http2/server.go:2851 +0x65
golang.org/x/net/http2.(*responseWriterState).writeChunk(0xc001d67780, {0xc0008aa000, 0xf, 0x1000})
	golang.org/x/net@v0.34.0/http2/server.go:2717 +0x77
golang.org/x/net/http2.chunkWriter.Write({0x10610e0?}, {0xc0008aa000?, 0xc0012831e0?, 0x1ce0680?})
	golang.org/x/net@v0.34.0/http2/server.go:2670 +0x1d
bufio.(*Writer).Flush(0xc000b093c0)
	bufio/bufio.go:643 +0x55
golang.org/x/net/http2.(*responseWriter).FlushError(0x142e0a0?)
	golang.org/x/net@v0.34.0/http2/server.go:2935 +0x31
golang.org/x/net/http2.(*responseWriter).Flush(...)
	golang.org/x/net@v0.34.0/http2/server.go:2925
golang.org/x/net/http2.(*responseWriter).handlerDone(0xc001425a50)
	golang.org/x/net@v0.34.0/http2/server.go:3100 +0x25
golang.org/x/net/http2.(*serverConn).runHandler.func1()
	golang.org/x/net@v0.34.0/http2/server.go:2473 +0x205
golang.org/x/net/http2.(*serverConn).runHandler(0x44cf12?, 0x0?, 0x0?, 0xc000146fb8?)
	golang.org/x/net@v0.34.0/http2/server.go:2477 +0x109
created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 624989236
	golang.org/x/net@v0.34.0/http2/server.go:2409 +0x21d

At the same timestamp, other goroutines were in the queue-proxy timeout / reverse-proxy handler path:

knative.dev/pkg/network.ErrorHandler.func1
	knative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/network/error_handler.go:33
net/http/httputil.(*ReverseProxy).ServeHTTP
	net/http/httputil/reverseproxy.go:486
knative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP
	knative.dev/serving/pkg/queue/request_metric.go:205
knative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3
	knative.dev/serving/pkg/queue/handler.go:76
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2294
knative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4
	knative.dev/serving/pkg/queue/forwarded_shim.go:54
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2294
knative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4
	knative.dev/serving/pkg/http/handler/timeout.go:118

Steps to Reproduce the Problem

I do not yet have a minimal standalone reproducer. The observed production trigger appears to involve:

1. queue-proxy serving HTTP/2 traffic.
2. Concurrent proxied requests.
3. Request cancellation or timeout while response handling is being finalized.
4. A response header/trailer map is modified while the HTTP/2 server iterates/promotes trailers during handlerDone.

Suspected Cause

The crash looks like concurrent access to the underlying http.Header map owned by the HTTP/2 response writer.

timeoutWriter synchronizes Flush, Write, WriteHeader, and timeout error writes with tw.mu, but Header() returns the raw underlying header map without synchronization:

func (tw *timeoutWriter) Header() http.Header { return tw.w.Header() }

Because callers can mutate the returned map outside timeoutWriter's mutex, the HTTP/2 server may concurrently iterate the same map in promoteUndeclaredTrailers, causing:

fatal error: concurrent map iteration and map write

This may be related to or another manifestation of #15850.