Skip to content

Commit c7afe59

Browse files
nicko170nicko170
committed
Initial commit: fit — vLLM deployment planner
Stateless Go HTTP service + React UI that takes a HuggingFace model ID, GPU inventory, and workload constraints, then returns a ranked list of valid vLLM deployment configurations (quantization, KV dtype, parallelism, placement) that fit and meet the SLO. The React SPA is embedded into the Go binary via //go:embed, so the service runs as a single static binary serving the UI at / and the JSON API at /api/v1/*. HuggingFace tokens are read from .env via godotenv, with per-request overrides supported by the API.
0 parents  commit c7afe59

80 files changed

Lines changed: 10985 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.example‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copy to .env (loaded automatically by the binary via godotenv).
2+
# .env is gitignored; never commit a real token.
3+
4+
# HuggingFace Hub access token (required for gated/private models).
5+
# Acts as the server-side default; per-request tokens in the PlanRequest
6+
# take priority and override this.
7+
HF_TOKEN=
8+
9+
# Optional: override the HF Hub base URL (e.g. for a mirror).
10+
# HF_API_BASE=https://huggingface.co
11+
12+
# Optional: HTTP port. Default 8080.
13+
# PORT=8080
14+
15+
# Optional: slog level. One of: debug, info, warn, error. Default info.
16+
# LOG_LEVEL=info
17+
18+
# Optional: per-request timeout in milliseconds. Default 5000.
19+
# REQUEST_TIMEOUT_MS=5000
20+
21+
# Optional: in-memory HF metadata cache size (entries). Default 10000.
22+
# CACHE_MAX_ENTRIES=10000
23+
24+
# Optional: path to a custom estimator constants JSON file. Empty = built-in defaults.
25+
# CONSTANTS_PATH=

‎.github/workflows/build.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
name: build
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
build:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
15+
- uses: actions/setup-node@v4
16+
with:
17+
node-version: '20'
18+
cache: 'npm'
19+
cache-dependency-path: web/package-lock.json
20+
21+
- uses: actions/setup-go@v5
22+
with:
23+
go-version-file: go.mod
24+
cache: true
25+
26+
- name: Build frontend
27+
run: |
28+
npm --prefix web ci
29+
npm --prefix web run build
30+
31+
- name: Verify frontend was embedded
32+
run: |
33+
test -f internal/webui/dist/index.html
34+
ls -la internal/webui/dist/
35+
36+
- name: Go vet
37+
run: go vet ./cmd/... ./internal/... ./pkg/...
38+
39+
- name: Go test
40+
run: go test ./cmd/... ./internal/... ./pkg/...
41+
42+
- name: Go build (embeds frontend)
43+
run: |
44+
mkdir -p bin
45+
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o bin/fit ./cmd/fit
46+
47+
- name: Smoke test embedded binary
48+
run: |
49+
./bin/fit &
50+
PID=$!
51+
sleep 2
52+
curl -fsS http://localhost:8080/api/v1/healthz
53+
curl -fsS http://localhost:8080/ | head -3
54+
kill $PID
55+
56+
- uses: actions/upload-artifact@v4
57+
with:
58+
name: fit-linux-amd64
59+
path: bin/fit
60+
if-no-files-found: error

‎.gitignore‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Secrets
2+
.env
3+
.env.*
4+
!.env.example
5+
6+
# Go build artifacts
7+
/fit
8+
/bin/
9+
/dist/
10+
*.test
11+
*.out
12+
coverage.txt
13+
14+
# Stale dev-server output (Vite now builds into internal/webui/dist for embedding)
15+
/web/dist/
16+
17+
# Node / Vite
18+
/web/node_modules/
19+
/web/.vite/
20+
21+
# Editor / OS
22+
.DS_Store
23+
.idea/
24+
.vscode/
25+
*.swp

‎Dockerfile‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# syntax=docker/dockerfile:1
2+
3+
# Stage 1: build the React SPA into internal/webui/dist
4+
FROM node:20-alpine AS web
5+
WORKDIR /src
6+
COPY web/package.json web/package-lock.json ./web/
7+
RUN npm --prefix web ci
8+
COPY web ./web
9+
RUN npm --prefix web run build
10+
11+
# Stage 2: build the Go binary (embeds the built SPA)
12+
FROM golang:1.25-alpine AS build
13+
WORKDIR /src
14+
COPY go.mod go.sum ./
15+
RUN go mod download
16+
COPY . .
17+
COPY --from=web /src/internal/webui/dist ./internal/webui/dist
18+
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/fit ./cmd/fit
19+
20+
# Stage 3: runtime
21+
FROM gcr.io/distroless/static
22+
COPY --from=build /out/fit /fit
23+
EXPOSE 8080
24+
ENTRYPOINT ["/fit"]

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 greenthread-ai
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# fit
2+
3+
**Does this model fit on your GPUs — and at what shape?**
4+
5+
`fit` is a stateless planner for vLLM deployments. Give it a HuggingFace model
6+
ID, your GPU inventory, and your workload (concurrency, context length, SLOs),
7+
and it returns a ranked list of valid `(quantization, KV dtype, parallelism,
8+
placement)` configurations that fit in memory and meet the SLO — along with
9+
the rejected ones and **why** they were rejected.
10+
11+
It is a pure planning engine. It does not schedule, deploy, or talk to a
12+
cluster. Hand the top-ranked config to your orchestrator and let it act.
13+
14+
The full design spec lives in [SPEC.md](./SPEC.md).
15+
16+
---
17+
18+
## What's in the box
19+
20+
- **Go HTTP server** (`cmd/fit`) — single static binary. Embeds the React UI,
21+
exposes the planning API at `/api/v1/*`.
22+
- **React UI** (`web/`) — model + workload + inventory editor with live
23+
re-planning on every edit. Built into the binary; no separate web server.
24+
- **Planner core** (`internal/{estimator,solver,model,gpu,hf}`) — memory and
25+
throughput math (roofline model), combinatorial config search, HuggingFace
26+
introspection, GPU SKU catalog, and per-architecture family handlers (dense,
27+
MoE, MLA-MoE, hybrid SSM, multimodal vision/audio).
28+
- **Calibration tooling** (`cmd/calibrate`) — skeleton for empirical
29+
calibration against real vLLM runs.
30+
- **vLLM architecture generator** (`cmd/gen-vllm-archs`) — scrapes upstream
31+
vLLM's model registry and emits Go bindings.
32+
33+
---
34+
35+
## Quick start
36+
37+
```bash
38+
# 1. Configure your HuggingFace token (only required for gated/private models)
39+
cp .env.example .env
40+
echo 'HF_TOKEN=hf_xxx' >> .env
41+
42+
# 2. Build the React UI (output is embedded into the Go binary)
43+
npm --prefix web ci
44+
npm --prefix web run build
45+
46+
# 3. Build the binary
47+
go build -o fit ./cmd/fit
48+
49+
# 4. Run
50+
./fit
51+
# → http://localhost:8080 (UI + API on the same port)
52+
```
53+
54+
The binary serves the SPA at `/` and the JSON API at `/api/v1/*`.
55+
56+
### Frontend dev loop
57+
58+
For live frontend reloads, run Vite separately and let it proxy to the Go
59+
backend:
60+
61+
```bash
62+
go run ./cmd/fit # terminal 1 — backend on :8080
63+
npm --prefix web run dev # terminal 2 — Vite on :5173 proxying /api → :8080
64+
```
65+
66+
---
67+
68+
## Configuration
69+
70+
All configuration is via environment variables. `fit` loads `.env` on startup
71+
(via [godotenv](https://github.com/joho/godotenv)); real env vars always win.
72+
See [`.env.example`](./.env.example) for the complete list.
73+
74+
| Var | Default | Purpose |
75+
|----------------------|-----------------------------|----------------------------------------------------|
76+
| `HF_TOKEN` | _(none)_ | Server-side default HF token. Per-request tokens override. |
77+
| `HF_API_BASE` | `https://huggingface.co` | Override for HF Hub mirror. |
78+
| `PORT` | `8080` | HTTP port. |
79+
| `LOG_LEVEL` | `info` | One of `debug`, `info`, `warn`, `error`. |
80+
| `REQUEST_TIMEOUT_MS` | `5000` | Per-request timeout. |
81+
| `CACHE_MAX_ENTRIES` | `10000` | In-memory HF metadata cache size. |
82+
| `CONSTANTS_PATH` | _(built-in)_ | Path to a custom estimator constants JSON file. |
83+
84+
---
85+
86+
## API
87+
88+
| Method | Path | Purpose |
89+
|--------|-------------------------------------------|--------------------------------------|
90+
| GET | `/api/v1/healthz` | Liveness probe. |
91+
| GET | `/api/v1/readyz` | Readiness (constants + HF reachable). |
92+
| GET | `/api/v1/gpus` | Built-in GPU SKU catalog. |
93+
| GET | `/api/v1/models/{owner}/{repo}/profile` | Model introspection (no planning). |
94+
| POST | `/api/v1/plan` | Main endpoint. Returns ranked configs. |
95+
96+
Request/response schemas are documented in [SPEC.md §2](./SPEC.md).
97+
98+
---
99+
100+
## Docker
101+
102+
```bash
103+
docker build -t fit .
104+
docker run --rm -p 8080:8080 --env-file .env fit
105+
```
106+
107+
The Dockerfile is a multi-stage build: Node stage builds the React UI into
108+
`internal/webui/dist/`, Go stage compiles the binary (which embeds the dist),
109+
final stage runs distroless static.
110+
111+
---
112+
113+
## Repository layout
114+
115+
```
116+
cmd/
117+
fit/ # HTTP server entrypoint
118+
calibrate/ # empirical calibration CLI (skeleton)
119+
gen-vllm-archs/ # generator for internal/model/vllm_archs.go
120+
internal/
121+
api/ # request/response wire types + validation
122+
estimator/ # memory + throughput math, calibration constants
123+
gpu/ # built-in GPU SKU catalog + inventory resolution
124+
hf/ # HuggingFace Hub client + in-memory cache
125+
model/ # per-family handlers (dense, MoE, MLA, hybrid SSM, multimodal)
126+
server/ # chi routes, middleware, handlers
127+
solver/ # combinatorial config search + ranking
128+
webui/ # //go:embed of the built React SPA + static handler
129+
pkg/
130+
planner/ # public planner type re-exports
131+
web/ # React 19 + Vite TypeScript SPA
132+
testdata/ # golden files + sample configs
133+
```
134+
135+
---
136+
137+
## License
138+
139+
[MIT](./LICENSE)

0 commit comments

Comments
 (0)