AI Gateway - Benchmarks, and thank you for featuring GoModel

[SantiagoDePolonia]

Jakub - GoModel author here. I came across this repo by chance and was really happy to find GoModel featured in the list - thank you for that. 🙏

Today I published a small benchmark that might be useful here: a reproducible comparison of GoModel, LiteLLM, Portkey and Bifrost. All pointed at the same mock backend, one at a time, in Docker on an AWS c7i.large. It focuses on the "boring" runtime costs that decide where a gateway can run rather than on feature counts - proxy-overhead latency (p50/p99), sustained throughput, peak RAM under load, cold-start time and Docker image size.

The harness is fully open and self-contained: one command provisions the box, runs each gateway against the same backend, prints the tables, and tears the infra back down - so it's easy to rerun, change the machine, or add your own workloads.

I'm obviously not a neutral party (I wrote one of the four), which is exactly why I kept the whole thing reproducible — corrections, a different methodology, or other people's numbers are all genuinely welcome.

Just sharing in case it's useful: https://enterpilot.io/blog/benchmarking-ai-gateways-gomodel-litellm-portkey-bifrost-june-2026/

Thanks again for featuring GoMode on this list!

[cuihuan]

Jakub — maintainer here. Sorry for the slow reply, and thank you both for the kind words and for sharing the harness.

This is exactly the kind of artifact this list exists to reward: a one-command, tear-down-included, reproducible run beats a feature-count table every time, and naming your own conflict of interest up front is the right way to publish a vendor-run benchmark. Genuinely well done.

Two thoughts:

1. We run an independent overhead benchmark ourselves (llm-gateway-bench) — on our rig the per-request proxy overhead came out Bifrost 0.56 ms / Portkey OSS 2.69 ms / LiteLLM 5.41 ms. It would be interesting to line your p50/p99 numbers up against ours where the tools overlap; if they diverge, that difference is itself useful methodology signal, and I would be happy to dig into why.

2. If you would like the benchmark referenced in the list, the honest path is the same one every number here takes: it gets labeled vendor-run (author of one of the four gateways measured) with the reproduction command next to it — sourced-and-labeled beats unlisted. A PR adding it under the evaluations section works, or I can pick it up on the next evaluation pass.

The GoModel listing itself stays on the usual independent-evidence bar regardless of this thread — but the reproducible harness is a real contribution to the space. Thanks again for building in the open.

[SantiagoDePolonia]

The response prepared with AI assistance below.

Thank you — and the labeling terms are the right ones, so I took both of them up.

Your point 2, the listing: awesome-ai-gateway#96 adds the harness under 📊 Latest evaluations, labeled vendor-run with the reproduction command in the entry, mirrored into README.zh-CN.md. Fold it in on your next pass instead if you prefer — the entry is a starting point, not a demand.

Your point 1, the numbers, which was the more interesting half. Your probe/overhead.mjs is sequential; my headline table is c=10, and that single difference explains almost all of the gap. Read at the c=1 point of my concurrency sweep, against data/overhead.json (2026-09-03):

Gateway	llm-gateway-bench	this harness @ c=1	this harness @ c=10
Bifrost	0.66 ms	0.63 ms	4.03 ms
LiteLLM	5.34 ms	5.49 ms	43.78 ms
Portkey OSS	2.60 ms	1.14 ms	9.60 ms

Two of three agree to within 5% on unrelated hardware, which neither of us could have established alone. Portkey is the one divergence — same 1.15.2 on both sides, so my untested guess is distribution rather than the gateway: you run npx @portkey-ai/gateway on the CI host, I run the official Docker image behind a bridge network. I can re-run against the npm build if that is worth settling.

The ranking is identical in both regimes; only the magnitudes move, by 5–8×. That seems like the transferable finding: a sequential figure prices the translation, a concurrent one prices the translation plus the runtime's queueing, and quoting one where the other belongs is how "50× faster" claims get made.

And the part that matters more than either: llm-gateway-bench#1 adds GoModel to your own runner — overhead, fidelity and cross-format, at the image defaults with the audit log and usage tracking left on, the same way you measure the others. My harness turns per-request logging off everywhere, so your number may land above my 0.37 ms. If it does, yours is the one to print.

Writing that up also made me find an undisclosed asymmetry in my own setup: Prometheus metrics were off for GoModel (its default) and on for Bifrost and TensorZero, which my parity note never mentioned. Now disclosed in the harness README. Small effect, but it is precisely the kind of thing a reader of a vendor-run benchmark should not have to find for themselves.

Fully agree that the GoModel listing stays on the independent-evidence bar — that is what PR #1 above is for, and I would rather earn it than argue for it.