Skip to content

Latest commit

 

History

History
423 lines (351 loc) · 43.9 KB

COMMUNITY_BENCHMARKS.md

File metadata and controls

423 lines (351 loc) · 43.9 KB

Community benchmark results

Share measurements from your own hardware so other users can judge how Strata might run on a similar machine. This guide suggests a report format and a pull request workflow; a single model, GPU, or context length is enough to contribute. The quickest way is the built-in benchmark: BENCH.bat / ./bench.sh writes a report in this format.

Existing benchmark reports provide examples. The engine 0.1.26 speed report includes hardware, settings, and per-run JSON. The technical details explain the published measurements and their limits. Report what you actually measured and label estimates separately.

The models PolyStrata adds

Qwen3.6-35B-A3B, Ornith 1.5, Gemma 4 26B-A4B and GLM 5.3 Flash run on PolyStrata's own engine, strata-poly, so their reports are listed here, apart from Strata's. The three below were posted on Reddit by people who ran the engine of v0.1.41-poly.1 or an earlier one. The numbers are the posters' own, and "not stated" marks what a post leaves out.

Reported by GPU CPU, RAM, system Model and file Context MTP Writes, tokens/s Reads, tokens/s Before, or compared with
u/Present_Resource4022 RTX 4070 SUPER 12 GB CPU not stated; 32 GB DDR5-6000; Windows 11 Qwen3.6-35B-A3B, HauhauCS uncensored Aggressive TTQ Q4_K_M about 8K / about 32K on 140 / 100 2,800 to 3,300 llama.cpp at 8K: about 77 writing and 2,600 reading, settings not stated
u/hope_it_helps RTX 4070 Ti 12 GB not stated Ornith 1.5 APEX-I-Compact not stated not stated about 90, 60 at the slowest not stated usually 40 before, engine not stated
u/Sensitive_Song4219 RTX 5060 Laptop 8 GB Core Ultra 9 275HX; 32 GB DDR5; system not stated Tiel Coder 35B-A3B, file not stated not stated not stated maybe 20% faster than before, no figure 1,000 to 1,500; 1,016 at the slowest on a large read, over 1,400 in an auto-compact 200 to 350 reading before, engine not stated
iMilnb RX 7600 8 GB (gfx1102), the AMD beta Ryzen 5 5600X; 30 GB; Linux, ROCm from AMD's wheels Qwen3.6-35B-A3B UD-Q2_K_XL, and Ornith 1.5 APEX-I-Mini 32K; the 24,000-token prompt is 31,834 tokens; --kv-whole on Qwen3.6: 74 in a short chat, 53 after the long prompt; Ornith: 40 and 43 Qwen3.6 481, Ornith 444 the first run of this engine on an AMD card, through the author's own port (#11)
biurquezjoselina-design RTX 5090 D v2 24 GB Core i7-14700KF, 13 threads; 128 GB DDR5; Windows 10 GLM 5.3 Flash 3.0BIT (neuralll), 113.6 GB, with the draft block; 1,483 experts in VRAM 32K, and 256K to 1M set in the run config on 26 to 30 after a short prompt, 23 to 30 after 2.5K to 10K 213 to 395 at 2.5K to 10K; 488 at 112K down to 286 at 557K in one request each, every planted fact found; with 0.1.44.1-poly.5, 1,002,452 tokens in one request in 45.7 min (366 tok/s; 66.3 min with poly.2), the three facts found, and 17.7 tok/s writing after it the first run of GLM on Windows; this project's 32 GB Linux pod reads 29K at 740 to 820
biurquezjoselina-design RTX 5090 D v2 24 GB Core i7-14700KF, 13 threads; 128 GB DDR5; Windows 10 DeepSeek V4 Flash UD-IQ2_XXS (Unsloth), 90.9 GB, on the release's ready-made engine 32K, and 256K to 1M set in the run config no MTP; guesses from repeated text 35 to 36 after a short prompt, 29 to 34 after 2.5K to 10K 303 to 469 at 2.5K, 688 to 805 at 10K; 393 at 252K, 247 at 504K and 138 at 1,008K in one request each, every planted fact found the first run of DeepSeek V4 Flash on Windows; this project's 32 GB Linux pod read 487,529 tokens in 1,923 s
biurquezjoselina-design RTX 5090 D v2 24 GB Core i7-14700KF, 13 threads; 128 GB DDR5; Windows 10 DeepSeek V4 Flash UD-IQ3_XXS (Unsloth), 104.2 GB, with 0.1.44.1-poly.5; 1,367 experts in VRAM 32K, and 512K set in the run config no MTP; guesses from repeated text 27 to 31 after up to 10K tokens 502 at 100K; 519,757 tokens in one request at 231 (705 at the start, 95 to 157 at the end), every planted fact found; RAM at most 119.8 of 127.8 GB UD-IQ2_XXS on the same PC: 31 to 33 writing, 600 at 100K and 243 at 512K
biurquezjoselina-design RTX 5090 D v2 24 GB Core i7-14700KF, 13 threads; 128 GB DDR5; Windows 10 DeepSeek V4 Flash UD-IQ4_XS (Unsloth), 136.7 GB, more than the RAM; 992 experts in VRAM 32K, 256K set in the run config no MTP; guesses from repeated text with 0.1.44.1-poly.6: 3.4 after a short prompt, 2.2 to 2.8 after 2.5K to 10K; with poly.5: 1.0 to 2.1 poly.6: 50 at 2.5K, 100 at 10K; poly.5: 23 and 53 RAM at its 127.6 GiB ceiling and the rest paged from the disk; poly.6 loads in 28 s where poly.5 took 110 s and kept 58 GiB of converted MXFP4 experts in RAM; llama.cpp b11706, experts on the CPU, the same file and prompt: 8.6; with UD-Q4_K_XL (155 GB), llama.cpp 9.5 to 10.6 and poly.6 1.4 to 2.2
Haroldyolo RTX 4070 Ti SUPER 16 GB Ryzen 7 7800X3D; 30.9 GiB; Windows 11 Qwen3.6-35B-A3B UD-Q3_K_M, 0.1.45-poly.1, BENCH.bat 64K on 165 in a short chat (121 to 196), 143 in an agent session 3,287 at 4K, 3,615 at 16K the same PC runs Qwen3.8-Flash-Next Coder IQ1_M on the Strata project's engine at 59 writing, 2,033 and 2,646 reading
Haroldyolo RTX 4070 Ti SUPER 16 GB Ryzen 7 7800X3D; 30.9 GiB; Windows 11 Qwen3.6-35B-A3B UD-Q4_K_M, 0.1.45-poly.1, BENCH.bat 128K on 164 in a short chat (119 to 212), 157 in an agent session 2,043 at 4K, 3,078 at 16K
Haroldyolo RTX 4070 Ti SUPER 16 GB Ryzen 7 7800X3D; 30.9 GiB; Windows 11 Tiel Coder 35B-A3B UD-Q4_K_XL, main at a418c11a, tools/strata_bench.py 64K on 141 in a short chat (113 to 166), 118 in an agent session 1,959 at 4K, 2,989 at 16K the Qwen3.6 UD-Q4_K_M rows above, on the same PC
Haroldyolo RTX 4070 Ti SUPER 16 GB Ryzen 7 7800X3D; 30.9 GiB; Windows 11 MiniCPM-V 4.7 35B-A3B Q4_K_M, main at a418c11a, tools/strata_bench.py 64K no 134 in a short chat (95 to 142), 123 in an agent session 1,998 at 4K, 3,180 at 16K the Tiel Coder and Qwen3.6 rows above, on the same PC
Danielkgr RX 7900 XTX 24 GB (gfx1100) and Radeon AI PRO R9700 32 GB (gfx1201, on a chipset x4 link), the layers split over both Core i5-14600KF, 9 threads; 96 GB DDR4-3200; Linux, ROCm 7.14.1 from AMD's wheels Ornith 1.5 122B Q5_K_M (ornith-ai), 82.6 GiB, main at 90347087; 6,213 of 12,544 experts in VRAM 128K, --kv-whole on, 65% to 85% of the guesses kept on short replies 20.7 to 22.0 at 98K to 100K tokens; 28 to 31 in a short reply 536 for a 98K-token prompt with --prompt-batch 2048, 472 with 1,024; later turns of the session 1.8 to 2.0 s llama.cpp Vulkan, the same cards, file and session: 527 s for the first prompt where PolyStrata took 184, 6.9 s for later turns, 13.5 writing at 100K; against llama.cpp on the CPU, the same most likely token at 93% of 281 steps
Dampfinchen RTX 2060 Core i7-9750H; 32 GB; system not stated Gemma 4 26B-A4B QAT, file not stated; 555 experts in VRAM, --vram-reserve-mib 120 64K, measured at 18K not stated 30 at 18K; slower as the context grows, about llama.cpp's speed near 85K 900 at 18K llama.cpp at 18K: 16 writing and 470 reading, steady as the context grows
HaldunEymen RTX 3060 12 GB, PCIe 3.0 Ryzen 5 5600G; 63 GB DDR4; Linux Mint 22.3, the models on a SATA hard disk Tiel Coder 35B-A3B UD-Q3_K_XL and Ornith 1.5 APEX-I-Compact, main at 5e16c627, polystrata.py benchmark --prompt-tokens 24000 32K; the long prompt is 32,201 tokens for Tiel, 32,022 for Ornith on; Tiel 71% to 81% of the guesses kept, Ornith 74% to 95% Tiel: 73 to 85 in a short chat, 52 to 60 after the long prompt; Ornith: 80 to 82 and 58 to 59 Tiel 1,078, Ornith 1,141 to 1,166 for the long prompt; its first token after 79 s and 28 s Qwen3.8-Flash-Next IQ3_XXS on the Strata project's engine, the same PC: 41 to 44 writing
DRojasLey RX 7800 XT 16 GB (gfx1101) Ryzen 7 5700X; 31.3 GB; Ubuntu 24.04.5, ROCm from AMD's wheels Tiel Coder 35B-A3B UD-Q3_K_XL, main at 7b700758, tools/strata_bench.py 128K, --kv-whole, --kv q8_0, --vram-reserve-mib 4096 on, 67% to 79% of the guesses kept 86 in a short chat (80 to 102), 67 after a 32K prompt 885 at 4K, 873 at 16K, 661 at 32K the first report of Tiel Coder on an RX 7800 XT
Csavoldi 2x Tesla V100-PCIE-32GB (sm_70), the layers split over both Xeon E5-2682 v4; 47 GB; Linux Qwen3.6-35B-A3B UD-Q3_K_M, main at 7b700758 built with CUDA 12.9.1 and -DSTRATA_EXPERIMENTAL_SM60=ON, tools/strata_bench.py 32K not stated 157.5 in a short chat (135 to 196), 134.2 in an agent session 2,615 at 4K, 2,778 at 16K the first report of strata-poly on a Volta card
GodrezJr2 RTX 4060 Laptop 8 GB Core i9-14900HX, 8 threads for the experts; 31.7 GB; Windows 11 Qwen3.6-35B-A3B UD-Q3_K_M, 0.1.45-poly.4's ready-made engine, a 117,977-token prompt and 64-token answers 128K, --kv q8_0, --tail-skip 6 not stated 32.1 after the long prompt; 51.0 with STRATA_FDEC=1 814; 1,318 with STRATA_FATTN=1 the two switches at 118K, which DETAILS.md measured only on an RTX 3070: 1.62x reading, 1.60x writing; both together 1,308 / 51.4
brocasm RX 7900 XTX 24 GB (gfx1100) Ryzen 7 5700X; 31.2 GiB; CachyOS Linux, the HIP backend Clover-35B-A3B-Preview Q4_K_M, 0.1.45-poly.6 (20d839db), tools/strata_bench.py 128K, --kv-whole not stated 123 in a short chat (116 to 126), 112 in an agent session 3,990 at 4K, 3,021 at 16K
LionelColaso GTX 1650 4 GB Core i5-9300H; 23.9 GiB; Windows 11 Clover-35B-A3B-Preview APEX-I-NanoPlus, 0.1.45-poly.6 (20d839d), tools/strata_bench.py 32K 13 of 33 guesses kept 11.6 in a short chat (10.6 to 13.7), 9.4 in an agent session 136 at 4K, 110 at 16K
LionelColaso GTX 1650 4 GB Core i5-9300H; 23.9 GiB; Windows 11 MiniCPM-V 4.7 35B-A3B mix_q3gud_im, 0.1.45-poly.6 (20d839d), tools/strata_bench.py 32K 26 of 60 guesses kept 17.8 in a short chat (15.6 to 20.6), 15.0 in an agent session 147 at 4K, 116 at 16K the Clover row above, on the same PC
HexhamAllstar RTX 3080 10 GB Ryzen 7 5700X3D; 63.9 GiB; Windows 11 Tiel Coder 35B-A3B UD-Q4_K_XL, 0.1.45, tools/strata_bench.py 64K on, 73% of the guesses kept 77 in a short chat (66 to 85), 80 in an agent session 1,420 at 4K, 2,527 at 16K
benchman-ai RTX 5070 Ti 16 GB Ryzen 7 5700X3D; 63.9 GiB; Windows 10 Qwen3.6-35B-A3B UD-Q4_K_M, 0.1.45, tools/strata_bench.py 32K on, 78% of the guesses kept 164 in a short chat (134 to 201), 153 in an agent session 2,158 at 4K, 3,105 at 16K
benchman-ai RTX 5070 Ti 16 GB Ryzen 7 5700X3D; 63.9 GiB; Windows 10 DeepSeek V4 Flash UD-IQ2_XXS, 0.1.45, tools/strata_bench.py 32K 52 of 120 guesses kept 3.4 in a short chat (2.4 to 3.9), 1.7 in an agent session 104 at 4K, 129 at 16K the 90.9 GB file does not fit in 64 GB of RAM: the disk was busy while it wrote
Hazelhof 4x RTX 5080 16 GB on a PEX89048 PCIe 5.0 switch, Gen5 x8 each, the layers split over all four Core Ultra 7 270K Plus; 125.2 GiB DDR5-6400; Linux, CUDA 13.3 GLM 5.3 Flash GSQ-RCO 3.0bit, main at 2d6a8f9b built from source; 4,652 of 12,384 experts in VRAM 32K on, 75% of the guesses kept 64 in a short chat (51 to 72), 62 in an agent session 479 at 4K, 573 at 16K a one-hour soak: 1,621 requests, no failures, decode 69.4 tok/s median in every quarter of the hour
shindocha RTX 5060 Ti 16 GB Xeon E5-2697 v4, 16 threads; 61.8 GiB; Linux 7.0, driver 610.43 Qwen3.6-35B-A3B UD-Q4_K_XL, 0.1.45 (a82999fd), tools/strata_bench.py 180K, the install's own configuration not stated 138 in a short chat (117 to 161), 131 in an agent session 1,470 at 4K, 2,565 at 16K
dontaskmeformyname RX 7900 XTX 24 GB (gfx1100) Ryzen 7 7800X3D; 30.6 GiB; CachyOS Linux, the HIP backend Flint 40B-A2.5B UD-Q3_K_XL, 0.1.45-poly.7 (fabae0d6), tools/strata_bench.py 64K, --kv-whole not stated 121 in a short chat (114 to 145), 111 in an agent session 3,496 at 4K, 2,315 at 16K the first report of Flint on an AMD card; an 8-task agent harness passed every task
narical RTX 2080 8 GB Core i3-10100, 4 cores; 31.2 GiB; Linux 7.3.4 Gemma 4 26B-A4B UD-Q4_K_XL with the Q8_0 draft file, 0.1.45, tools/strata_bench.py 17K not stated 46.6 in a short chat (43.7 to 63.2), 48.7 in an agent session 1,245 at 4K, 1,878 at 16K
Keoian 2x RX 5700 XT 8 GB (gfx1010), the layers split over both Core i7-3770 (AVX, no AVX2), 6 threads; 32 GB DDR3-1600; Ubuntu 26.04.1, ROCm from AMD's wheels Cyber-Tiel Coder 35B-A3B UD-Q4_K_XL, main at 67dde645, tools/strata_bench.py; 4,053 of 10,496 experts in VRAM 256K, --kv-whole, --kv q8_0, --cache-bias 0.5, --vram-reserve-mib 512 not stated 77.2 in a short chat (60.5 to 95.2), 75.1 in an agent session 430 at 4K, 454 at 16K, 419 at 32K the first report of strata-poly on gfx1010 under Linux; without a named reserve the engine faulted there when the expert cache grew
dontaskmeformyname RX 7900 XTX 24 GB (gfx1100) Ryzen 7 7800X3D; 30.6 GiB; CachyOS Linux, the HIP backend Ornith 1.5 APEX-I-Compact, main at 67dde645, tools/strata_bench.py 64K, --kv-whole on 153.7 in a short chat (146.0 to 186.8), 138.3 in an agent session 3,435 at 4K, 2,799 at 16K, 2,189 at 32K llama.cpp 0a66ac7 on the same file at a 36,864 context: after a 4K prompt PolyStrata wrote at 136 and llama.cpp at 113 (117 with draft-mtp); after a 32K prompt 105 against 90 (104), and llama.cpp read that prompt at 2,664 where PolyStrata read it at 2,105
dontaskmeformyname RX 7900 XTX 24 GB (gfx1100) Ryzen 7 7800X3D; 30.6 GiB; CachyOS Linux, the HIP backend Nex-N2.5-mini IQ4_XS (bartowski), installed with --custom-gguf, main at 67dde645, tools/strata_bench.py 64K, --kv-whole none: the file has no MTP block 113.2 in a short chat (93.0 to 114.1), 105.8 in an agent session 4,107 at 4K, 3,065 at 16K, 2,261 at 32K the Ornith row above, on the same PC
vsech RX 7600 XT 16 GB Ryzen 7 5700X; 63.9 GiB; Windows 10, the HIP backend Gemma 4 26B-A4B UD-Q4_K_M with the Q8_0 draft file, main at acd46131, tools/strata_bench.py 32K, --kv-whole not stated 66.7 in a short chat (55.2 to 81.0), 58.3 in an agent session 1,026 at 4K, 939 at 16K
vsech RX 7600 XT 16 GB Ryzen 7 5700X; 63.9 GiB; Windows 10, the HIP backend Qwen3.6-35B-A3B UD-Q4_K_M, main at acd46131, tools/strata_bench.py 131K, --kv-whole not stated 74.2 in a short chat (55.6 to 82.7), 67.2 in an agent session 702 at 4K, 777 at 16K the Gemma 4 row above, on the same PC
vsech RX 7600 XT 16 GB Ryzen 7 5700X; 63.9 GiB; Windows 10, the HIP backend Tiel Coder 35B-A3B UD-Q4_K_XL, main at acd46131, tools/strata_bench.py 32K, --kv-whole not stated 60.2 in a short chat (42.5 to 70.4), 52.0 in an agent session 762 at 4K, 802 at 16K the two rows above, on the same PC
JuggyBob RX 7600 XT 16 GB (gfx1102) Ryzen 5 7600X; 31.1 GB; Windows 11, the HIP backend LFM2.5-24B-A2B UD-Q3_K_XL, main at 7a6c0983, the prompts of an agent CLI, about 12K tokens each; 2,432 of 2,432 experts in VRAM 64K, --kv-whole none: the file has no MTP block 90.0 over 81 requests (65.4 to 130.3) up to 1,534 on a fresh 12K prompt

Setup offers Tiel Coder now (--family tiel), and not the HauhauCS file. How to give setup such a file is in the FAQ.

To add yours, open an issue at VecSzn/PolyStrata with what What to record lists. python polystrata.py benchmark --model <name> --prompt-tokens 24000 gives the two speeds the README uses (what it sends and prints).

An upstream-suite report from an RX 6700 XT 12 GB, Ryzen 5 3600 and 32 GB RAM includes Qwen3.6, Ornith, Tiel Coder, Gemma 4, MiniCPM-V and LFM2.5 at 32K and 64K, with per-request data, exact settings and a separate Qwen UD-Q4_K_M speed control. A second one from an RX 6700 XT 12 GB, Ryzen 5 5600X and 32 GB RAM has Qwen3.6, LFM2.5, Tiel Coder and Gemma 4 QAT at 32K and 64K, a rerun of Qwen on a newer commit and a Gemma draft A/B; its Qwen3.6 and LFM2.5 rows land within a few percent of the first.

An official-suite report from an RTX 3060 12 GB, Xeon W-2145 and 32 GB RAM compares v0.1.45-poly.2 and v0.1.45-poly.3 on Windows: five model configurations at 32K, a Qwen MMA pair, and Qwen 64K/128K allocations with about 32K actual input. It includes thirteen suites with per-request data, exact release commits and binary hashes, two retained LFM request failures, and an older-driver appendix. These October 22/24 results do not test v0.1.45-poly.4 or later releases.

A follow-up on the RX 6700 XT / Ryzen 5 3600, now with 64 GB RAM compares v0.1.45-poly.2 and v0.1.45-poly.4 at 28 and 52 GiB container memory limits, all at 64K configured context. It includes 28 official suites, frozen expert-profile seeds, exact native HIP build hashes and four retained LFM agent-request failures. The earlier report stays intact; the new limits compare available capacity on the upgraded machine rather than recreating its old physical RAM.

The same RX 6700 XT on v0.1.45-poly.9, the HIP engine built for gfx1031: Qwen3.6 UD-Q3_K_M, UD-Q4_K_M and UD-Q4_K_XL, Gemma 4 UD-Q4_K_M and Tiel Coder UD-Q3_K_XL at 64K configured context, the official suite with 36 requests each and no failures. Chats wrote 72 to 90 tokens/s, and prompts near 32K read at 819 to 859.

The official suite on a GTX 1650 4 GB with Clover APEX-I-NanoPlus and with MiniCPM-V mix_q3gud_im, and on an RTX 3080 10 GB with Tiel Coder UD-Q4_K_XL, all three with v0.1.45-poly.6 and per-request data; their rows are in the table above.

The official suite on an RX 7900 XTX with Flint UD-Q3_K_XL adds an 8-task agent harness run through Hermes Agent: every task made the right file, and one of them took two minutes because the model kept re-reading its input.

Comparing with llama.cpp

bench/bench_vs_llama.py is the script Strata used for its own comparison. It starts PolyStrata and llama.cpp's llama-server in turns on the same GGUF, one engine on the card at a time, sends both the same prompts and prints the medians. With a size that setup has installed:

  1. Stop PolyStrata and anything else that uses the card.

  2. Download llama-server from llama.cpp's releases, the CUDA build for an NVIDIA card. Qwen3.6's draft block needs --spec-type draft-mtp, which b11540 has.

  3. Make a folder for the run and copy prompt-4k.txt and prompt-32k.txt into it from bench/results/2026-10-06-llamacpp-0.6.0/: the first 4,000 and 32,000 tokens of Strata's docs.

  4. Save a file like this one as machine.json in the PolyStrata folder, with your paths: the size's GGUF, llama-server, the run config setup wrote for the size (strata-<size>.json in the PolyStrata folder), the PolyStrata folder's serve folder and the PolyStrata folder.

    {
     "gguf": "D:/PolyStrata-data/models/qwen36-UD-Q3_K_M/Qwen3.6-35B-A3B-UD-Q3_K_M.gguf",
     "llama_server": "D:/llama.cpp/llama-server.exe",
     "strata_cfg": "D:/PolyStrata/strata-qwen36-ud-q3_k_m.json",
     "serve": "D:/PolyStrata/serve",
     "cwd": "D:/PolyStrata",
     "configs": [
      {"name": "polystrata", "kind": "strata", "args": [], "override": {"--max-context": "36864"}},
      {"name": "llama-mtp", "kind": "llama", "args": ["--fit", "on", "-c", "36864", "-fa", "on",
       "-ctk", "q8_0", "-ctv", "q8_0", "--spec-type", "draft-mtp", "--spec-draft-n-max", "2"]},
      {"name": "llama", "kind": "llama", "args": ["--fit", "on", "-c", "36864", "-fa", "on",
       "-ctk", "q8_0", "-ctv", "q8_0"]}
     ]
    }
  5. Run it from the PolyStrata folder with setup's Python (.venv/bin/python on Linux), then print the table:

    .venv\Scripts\python.exe bench\bench_vs_llama.py run --machine machine.json --out <run folder> --rounds 3
    .venv\Scripts\python.exe bench\bench_vs_llama.py summary --out <run folder>
    

A round starts one engine, warms it up with two requests, sends the 4K prompt twice and the 32K prompt once at temperature 0 and again at 0.7, each for up to 256 tokens of answer, and stops it; then the next engine starts. Rounds alternate the order of the engines. The summary gives medians: prompt and answer speed, the server's prompt time, the share of draft tokens accepted and the peak VRAM. Every request, and the command that started each engine, is in <run folder>/raw/, and the engines' logs are in the run folder.

--fit on lets llama.cpp decide how many layers' experts go to the processor. A number picked by hand with -ngl 99 --n-cpu-moe N can be faster; on the README's 8 GB laptop it was N = 30 for Qwen3.6 UD-Q3_K_M. The context of 36,864 on both sides leaves room for the 32K prompt and its answer, and -fa on with the q8_0 cache is how Strata's comparison ran llama.cpp. The example is for Qwen3.6 UD-Q3_K_M.

The README's laptop ran it on 2026-10-11 with the engine of 2ba29a383baf and llama.cpp b11540, for Qwen3.6 UD-Q3_K_M and Tiel Coder UD-Q3_K_XL. Against llama.cpp's fastest setting in each cell, PolyStrata read the prompts 2.7 to 3.5 times as fast and wrote 1.5 to 1.8 times as fast.

Community reports

Run the benchmark

The simplest report is one command after setup. Windows: double-click BENCH.bat. Linux: ./bench.sh. Or python tools/strata_bench.py from the Strata folder. Setup also offers it at the end of an interactive install (default no).

It starts the model with the settings you already have (the newest strata-*.json; --config FILE picks another one), or uses a Strata server that is already running on that port (--url http://127.0.0.1:8080 for another address). It then sends a fixed suite through the HTTP API, three runs each, and reports medians with ranges. The default takes about 3 to 10 minutes once the model has loaded:

Test What it sends What it reports
chat three short questions, up to 100 new tokens decode tok/s, time to first token
4K and 16K prompt a fresh word-list prompt of about 4,096 and 16,384 tokens (skipped when your context is smaller) prompt tok/s, time to first token
agent session a system prompt and four tools, then five turns that each add about 1,500 words of a project file and continue from the model's own earlier answers tokens the conversation cache reused, tokens read again, time to first token per turn
--clients N N requests at the same moment (the server needs --batch to run them together) total tok/s, p50 and p95 first-token time and latency
--long the prompt test at about 32,768 tokens prompt tok/s, time to first token

Speeds are the engine's own timings (prompt_per_second, predicted_per_second in the answer); first-token time and latency are measured by the client. Thinking is off for the suite so the answers are short and the runs comparable (--think leaves it on). Every prompt carries a unique tag, so the fresh-prompt tests never get help from the cache. Close other programs first: a browser or a game changes the numbers.

The result goes to bench/results/<date>-community-<gpu>-<model>/ (or --out DIR):

  • README.md in the format below, with the GPU, VRAM, CPU, RAM, OS, driver, Strata and engine versions, model and quantization, context and your settings filled in from the machine and the server
  • results.json with every request (tokens, timings, errors) and the machine facts
  • engine-log-excerpt.txt with the engine's start-up lines (when the benchmark started the server)

Home folders, user and host names, local network addresses and API keys are replaced by placeholders before anything is written. Read the report once before you share it: the benchmark cannot know what else you consider private.

To share it, open a pull request that adds the folder (see Submit a report), or paste README.md into a GitHub issue. Add what the tool does not know: background load, power limits, storage.

To compare with an earlier run on the same machine (before and after an update or a setting change):

python tools/strata_bench.py --compare bench/results/2026-10-01-community-rtx-3060-qwen3-8-flash-next
python tools/strata_bench.py --compare OLD_FOLDER --compare-only NEW_FOLDER   # two saved runs, nothing is started

The tool is a speed check, not a quality test. For recall and other checks, see Optional correctness checks; for A/B tests of two builds use tools/ab_engine.py.

What to record

Include enough information for someone else to repeat your run:

  • Hardware: GPU model, VRAM, selected GPUs for a multi-GPU run, CPU, installed RAM, and storage type. Include PCIe link speed and width, GPU power limits, and other workloads when known or relevant.
  • Software: OS, Strata commit and engine version, driver and CUDA version (or ROCm for HIP), and release binary or source build. Include changed build options.
  • Model: exact repository and revision when available, quantization, GGUF filenames, and vision encoder if enabled. Identify custom packs, expert profiles, or draft vocabularies, with hashes or reproducible preparation steps.
  • Settings: launch command and relevant run configuration, context limit, KV type and streaming window, expert cache, low-RAM mode, prefill size, CPU workers, MTP, reasoning effort, sampling, and vision. State whether calibration or experimental speed projection was enabled; attach the resulting settings.
  • Workload: benchmark command or script, shareable prompts or instructions to build them, actual prompt and generated token counts, output cap, repetitions, and cache state for each run.

Use GB or GiB consistently and name the unit. Mark unavailable values explicitly instead of guessing. Remove credentials and private content from configurations, prompts, and logs before publishing them.

How to measure

Start with a configuration that already works on your machine. You do not need to download every model or test the largest context. A useful initial speed report tests one short prompt and one longer prompt that fits, with a fixed output cap (for example, 256 tokens). Record the actual output length if the model ends early.

For each configuration, try at least three measured runs and publish all of them with a median and range. If you only have one run, say so. Explain any warm-up and whether model loading is included in the timing.

Strata reuses conversation prefixes and adapts its expert cache. Separate prompts that are fully processed from follow-ups that reuse tokens, and report the number of reused and freshly read tokens from the engine log. Describe how you reset or retained state between runs. A warmed expert cache and a reused prompt prefix are different conditions; record both when possible.

Keep these measurements separate:

  • Prompt throughput: freshly processed prompt tokens per second, from the engine's timing output. Keep the corresponding token count and duration.
  • Decode throughput: generated tokens per second, from the engine's timing output. Count reasoning tokens too, or explain any different convention.
  • Time to first token: seconds from sending the request to its first generated token. For a streaming API measurement, ignore keep-alives and empty deltas and state whether the first token is reasoning or answer text.
  • Total latency: elapsed seconds for the whole request. State whether measured at the engine or client, and whether loading, vision encoding, or queueing is included.
  • Memory: observed RAM and VRAM usage, including whether the value is a startup snapshot or a peak during inference. Report paging or out-of-memory failures.

Attach the engine timing lines and your measurement script if you used one. Avoid calculating decode throughput by dividing generated tokens by total request time, which also includes prompt processing. Cancelled or failed requests belong in the report as failures, outside the successful throughput summary.

When comparing versions or settings, run both on the same machine and workload and list every changed setting. Preserve the per-run results: output text and draft acceptance can change speed even when the input is identical.

Optional correctness checks

Speed measurements alone do not establish answer quality. Include a small check relevant to your workload, such as a completed coding task with test results, a tool call followed by its result, or an image question with an expected answer.

For long-context recall, the repository includes tools/needle_bench.py. With a running local server, from the repository root:

python tools/needle_bench.py --help
python tools/needle_bench.py --url http://127.0.0.1:8080 --lengths 32k,128k --depths 10,50,90 --out needles.json

Choose lengths your configured context can hold. The script supports --api-key for an authenticated server; keep that key out of published commands. Save the reported actual prompt lengths, misses, errors, and skipped cases as well as successes. A needle test measures recall on those inputs, not overall model quality.

Submit a report

  1. Fork Strata and create a branch for your report.
  2. Add a folder such as bench/results/YYYY-MM-DD-community-rtx-5090/. Use the measurement date and a short hardware label; add a suffix if the name exists.
  3. Put a README.md based on the template below in that folder, together with per-run JSON or CSV and small scripts or prompts needed to repeat it. Use your own data format and document its fields and units. Keep large model files and generated packs out of the PR.
  4. Open a pull request against Niko1221/Strata:main. Summarize the hardware, model, configurations tested, and limitations. Keep a results-only submission separate from engine changes so reviewers can assess the measurements directly.

Maintainers can review the report and decide where to include it. Adding a report does not require changing the README's headline performance claims. For a failed run or a suspected bug, a GitHub issue with the configuration and relevant logs may be more useful than a benchmark PR.

Report template

Copy this into your report's README.md, replace the placeholders, and remove sections that do not apply. Use not measured for missing measurements; the table contains no example performance numbers.

# Community benchmark on GPU name

Measured on DATE by HANDLE. State what was tested and the main limitation.

## Hardware and software

- GPU and VRAM; CPU; installed RAM; storage; PCIe link if known:
- OS; driver; CUDA or ROCm:
- Strata commit; engine version; release binary or source build:
- Background workloads and any power limits:

## Model and configuration

- Model repository and revision; quantization; GGUF filenames:
- Vision encoder; custom packs or profiles:
- Context; KV type and streaming; cache; prefill; low-RAM mode:
- MTP; reasoning; sampling; calibration; experimental speed projection:

```text
Exact launch command and relevant configuration, with credentials removed.
```

## Method

Link prompts and scripts. Describe output cap, repetitions, warm-up, loading,
prompt reuse, expert-cache state, timing boundaries, and memory measurement.

## Results

| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range |
| --- | ---: | ---: | ---: | ---: | --- | --- | --- |
| Fill in measured values | | | | | | | |

Link per-run data and logs. Include total latency and memory observations when
measured, with units and timing boundaries. List failures and skipped cases.

## Correctness and limitations

Describe checks, expected answers, observed results, and untested behavior.