m5-max-model-testing

Experimental. The pages and scores under docs/ are an unfinished local bench. Older runs used different servers, context sizes, and slot counts. A shared-profile rerun (one slot, 16384 context, temperature 0, seed 42) is still in progress, and the hub and compare pages have not been rebuilt from it. Matched runs still differ by GPU backend, compiler, shell, and Python. Treat any ranking as provisional.

Local LLM benchmarking for agentic coding. Scores are machine-verified (compile + hidden tests, no LLM judge).

• macOS: MLX on Apple Silicon (this repo’s original target: M5 Max, 128 GB).
• Linux: llama.cpp CUDA / vLLM / SGLang / Ollama on NVIDIA. Baseline: docs/benchmarks-linux.md (RTX 6000 Pro Workstation 96 GB). Not ROCm.

Mac deployed: gpt-oss-20b — 83.3 tok/s, 43/48 C, 11.7 GB, 19/19 one-shot self-repair. Mac most accurate: qwen3.8-flash-next 125B — 46/48 C, 24/24 Python, 19/19 self-repair, 33.5 tok/s.

Quick start

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt

Mac

.venv/bin/pip install mlx-lm mlx-vlm
scripts/serve-mlx.sh &          # gpt-oss-20b on :8080
.venv/bin/python scripts/bench.py --target gptoss --case both --trials 2
.venv/bin/python scripts/eval_code.py --target gptoss --trials 1 --json
# full sweep: scripts/run-all-benchmarks.sh

Linux (NVIDIA CUDA) — scripts/linux-setup.sh is Rocky/EL10, writes under /home, builds for Blackwell (sm_120). On Ubuntu set MODELS_DIR/HF_HOME, install gcc/cmake/ninja/CUDA yourself, and change CMAKE_CUDA_ARCHITECTURES to your GPU.

export MODELS_DIR=$HOME/models HF_HOME=$HOME/hf
scripts/linux-setup.sh                 # add --with-vllm for vLLM pairs; scripts/linux-sglang-setup.sh for SGLang
scripts/download-models.sh scripts/models-linux.txt   # ~200 GB; or edit the list down to gpt-oss
export LLAMA_SERVER_BIN=$HOME/llama.cpp/build/bin/llama-server   # wherever it built
scripts/serve-llamacpp.sh &
.venv/bin/python scripts/bench.py --target gptoss --case both --trials 2
.venv/bin/python scripts/eval_code.py --target gptoss --trials 1 --json
# full sweep: scripts/run-all-benchmarks.sh

Set HF_TOKEN or Hugging Face downloads crawl. Linux tok/s is not comparable to the Mac; quality rankings are.

Docs

• docs/index.html — hub: machines first
• docs/compare.html — same model + same tests across machines
• docs/results/report.html — M5 Max report · docs/results-linux/report.html — RTX 6000 Pro Workstation
• docs/benchmarks.md — Mac writeup · docs/benchmarks-linux.md — Linux writeup
• START-HERE.md — Mac live stack and gotchas
• scripts/run-all-benchmarks.sh — speed + quality + C + perplexity (platform-detected)
• scripts/run-linux-missing.sh — remaining Linux suites (Python/Bash/hard/repair/…)