Skip to content

exp/api/remote: dont panic on malformed remote-write label refs - #2087

Open
samarth70 wants to merge 1 commit into
prometheus:mainfrom
samarth70:fix/desymbolize-labels-panic
Open

exp/api/remote: dont panic on malformed remote-write label refs#2087
samarth70 wants to merge 1 commit into
prometheus:mainfrom
samarth70:fix/desymbolize-labels-panic

Conversation

@samarth70

Copy link
Copy Markdown

Problem

DesymbolizeLabels walks labelRefs two entries at a time and indexes into symbols directly, with no validation of either:

func DesymbolizeLabels(labelRefs []uint32, symbols, buf []string) []string {
	result := buf[:0]
	for i := 0; i < len(labelRefs); i += 2 {
		result = append(result, symbols[labelRefs[i]], symbols[labelRefs[i+1]])
	}
	return result
}

Both inputs come off the wire. NewWriteHandler accepts remote-write v2 requests and hands them to a writeStorage implementation, and this is the helper that implementation uses to turn TimeSeries.LabelsRefs into labels. A sender that emits an odd number of references, or a reference past the end of the symbols table, panics the decoding goroutine:

DesymbolizeLabels([]uint32{1}, []string{"", "a"}, nil)
// panic: runtime error: index out of range [1] with length 1

DesymbolizeLabels([]uint32{5, 6}, []string{"", "a"}, nil)
// panic: runtime error: index out of range [5] with length 2

Neither condition is checked anywhere before this point, so a malformed or malicious request reaches it unfiltered.

Fix

Return an error for both cases. This matches the reference implementation of the same spec in prometheus/prometheus (prompb/io/prometheus/write/v2/symbols.go), which already guards exactly these two conditions:

func desymbolizeLabels(b *labels.ScratchBuilder, labelRefs []uint32, symbols []string) (labels.Labels, error) {
	if len(labelRefs)%2 != 0 {
		return labels.EmptyLabels(), fmt.Errorf("invalid labelRefs length %d", len(labelRefs))
	}
	...
	if int(nameRef) >= len(symbols) || int(valueRef) >= len(symbols) {
		return labels.EmptyLabels(), fmt.Errorf("labelRefs %d (name) = %d (value) outside of symbols table (size %d)", ...)
	}

Prometheus's own appendV2 treats this as a fallible operation and turns a failure into a bad-request error rather than a crash, which seems like the behaviour a receiver built on this package would want too.

This does change the signature to return an error. That is a breaking change, which I took to be acceptable here because the module documents itself as experimental ("This package is experimental and may contain breaking changes or be removed in the future"), and because silently dropping the malformed pairs would discard metric data without telling the caller. Happy to take a non-breaking variant instead if you would rather not change the signature in this module yet.

I left SymbolizeLabels alone. It has the same i += 2 shape, but its input is the caller's own label slice rather than wire data, so an odd length there is a programming error in the caller and out of scope for this change. Glad to include it if you would prefer them consistent.

Testing

Added TestDesymbolizeLabelsInvalidInput, covering an odd reference count (two lengths), an out-of-range name reference, an out-of-range value reference, and references against an empty symbols table. It fails on current main with the panics above and passes with the fix.

The existing TestSymbolsTable round trip is updated for the new signature and still passes. Full exp module suite is green (go test ./...), and gofmt and go vet are clean.

DesymbolizeLabels stepped through labelRefs two at a time and indexed
symbols directly, without checking that the slice has an even length or
that each reference is within the symbols table. Both are attacker or
peer controlled: they come from a remote-write v2 request decoded by a
receiver, so a malformed request panicked the decoding goroutine with an
index out of range error.

Return an error for both cases instead, matching the reference
implementation in prometheus/prometheus, which rejects an odd labelRefs
length and references outside the symbols table.

Signed-off-by: Sam Agarwal <samarthagrawal526@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant