Repository navigation
Feat/m12 devshardd integration - #11
Merged
Merged
Conversation
…es for m11 Move from internal/phases/07_registration.go to internal/chain/: - RunComposeExec (verbatim) - FetchConsensusKey + helpers (verbatim) - LoadKeyringPassword (refactored from mutating method to pure function; M11 Slice 1 Review Notes F1 fix) Move from internal/cmd/repair.go to internal/cosmovisor/: - ReleaseAsset, FindBinaryAsset, DownloadAsset, VerifySHA256, ExtractZipBinary, FixSymlink, RunHostCmd, ShellQuote (verbatim) - copyFile (kept package-private, used by ExtractZipBinary) - Type alias `ReleaseAsset = cosmovisor.ReleaseAsset` in cmd/repair.go to minimize call-site churn in unmoved helpers (parseGitHubReleaseAssets, parseUpgradeInfoBinaries, fetchReleaseAssets, buildDirectDownloadAssets) Centralize sudo/docker shell command identifiers: - New: internal/docker/exec.go exports docker.CmdSudo and docker.CmdDocker - Removed local cmdSudo/cmdDocker consts from internal/phases/06_deploy.go - Updated callers in 07_registration.go (via the chain extraction) and 09_mlnode_firewall.go to reference the docker package Tests migrated to the new package locations: - internal/chain/consensus_test.go (4 TestFetchConsensusKey_* cases) - internal/chain/exec_test.go (2 TestRunComposeExec_* + 4 new TestLoadKeyringPassword_* cases verifying pure-function semantics) - internal/cosmovisor/binary_test.go (TestVerifySHA256, TestFindBinaryAsset, TestExtractZipBinary) - internal/cosmovisor/symlink_test.go (TestFixSymlink, adjusted signature) - internal/cosmovisor/host_test.go (TestShellQuote, exported name) Zero behavior change for non-test code; all existing tests remain green. go vet ./... clean; go test -race ./... all pass. LOC: ~310 prod moved (architect-approved exception above the 200-line guideline; see docs/plan-m11.md Review Notes F14). Pre-review: architect. Refs: M11 Slice 1, FR-M11-7..16 enablement, architect conditions #1 and #2.
…recipe
Replaces the existing `SupportedModels []string` slice with a
`map[string]ModelRecipe` keyed by upstream model ID. Each recipe declares:
- GPUArchAllowed []string (e.g. {"sm_100","sm_103","sm_120"} for Kimi)
- VLLMArgs []string (recipe-authoritative when non-nil)
- MinVRAMPerGPUGB / MinTotalVRAMGB (architecture-aware floors)
- ImageVariant ("standard" | "blackwell")
Four entries shipped:
- Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 — legacy, VLLMArgs nil
- Qwen/Qwen3-32B-FP8 — legacy, VLLMArgs nil
- Qwen/QwQ-32B — legacy, VLLMArgs nil
- moonshotai/Kimi-K2.6 — recipe-driven, 20-token VLLMArgs verbatim
from FR-M11-3 (12 flag groups: --enable-auto-tool-choice,
--tensor-parallel-size 4, --enable-expert-parallel, --trust-remote-code,
--mm-encoder-tp-mode data, --tool-call-parser kimi_k2,
--reasoning-parser kimi_k2, --attention-backend FLASHINFER_MLA,
--disable-custom-all-reduce, --gpu-memory-utilization 0.95,
--max-num-seqs 128, --max-model-len 240000)
`recommendConfig` signature now `(gpuCount, vramMB, arch, hasNVLink,
requestedModel string) (GPURecommendation, error)`:
- requestedModel == "": legacy tier switch returns (rec, nil) — byte-
identical behavior to pre-M11 (NFR-M11-5 backward compatibility)
- requestedModel != "": looked up in SupportedModels, validated against
GPUArchAllowed + MinTotalVRAMGB, then base recommendation overlaid
with recipe.Model + (if VLLMArgs populated) deriveDisplayFromArgs to
expose TP/MemoryUtil/MaxModelLen for UI display
- Failure returns clear error citing observed vs required arch / VRAM
`buildVLLMArgs` consults SupportedModels[state.SelectedModel]: when a
recipe has non-nil VLLMArgs, emits them verbatim (defensive copy);
otherwise falls back to the historical state-driven path.
`selectMLNodeImage` now takes an additional `requestedModel` param.
When the model resolves to a recipe with ImageVariant == "blackwell",
the blackwell image is selected regardless of arch (defensive — the
recommender would already have rejected non-Blackwell hosts upstream).
Tests added (13 new subtests + 1 golden file):
- TestKimiRecipe_VLLMArgsGoldenOrder (pinned testdata/kimi_vllm_args.golden.json)
- TestSupportedModels_AllFourPresent
- TestSupportedModelNames_DeterministicOrder (ASCII-sort contract)
- TestKimiRecipe_BlackwellOnly
- TestLegacyRecipes_NoVLLMArgs
- TestLookupRecipe
- TestRecommendConfig_KimiArchValidation (4 subtests: reject sm_80/sm_90,
accept sm_100/sm_120)
- TestRecommendConfig_KimiVRAMRejection
- TestRecommendConfig_UnknownModel
- TestRecommendConfig_EmptyModelFallsBackToTier
- TestSelectMLNodeImage_RecipeImageVariant (3 subtests for ImageVariant override)
All existing TestRecommendConfig and TestSelectMLNodeImage cases updated
to the new signatures (zero behavioral change for the empty-model path).
internal/cmd/download_model.go and download_model_test.go switched from
slice iteration to SupportedModelNames() (sorted) and map membership.
LOC: ~150 production (model_recipes.go 100 + 02_gpu_detection.go +50) +
~250 tests. Pre-review: architect.
Refs: M11 Slice 2, FR-M11-1..5, AC-M11-2, NFR-M11-5; architect Review
Notes F3 (string-typed GPUArchAllowed), F4 (ImageVariant wiring), F13
(20-token Kimi arg list pinned in golden file).
… SelectedModel gate
CLI surface:
- New `setup --model <id>` flag. When set, state.SelectedModel is
populated before phase 02 runs; recommendConfig validates arch+VRAM
and fast-fails (exit 1) with a clear error before any files are written.
Verified by TestPhase02_KimiOnNonBlackwell_FastFails (TC-1.6 / AC-M11-2).
Phase 02 gate (Review Notes F5):
- The recommender's `rec.Model` assignment to state.SelectedModel is now
gated: `if state.SelectedModel == "" { state.SelectedModel = rec.Model }`.
Pre-set values (from --model flag or saved state) survive the phase.
Verified by TestPhase02_HonorsPreSetSelectedModel.
Menu filter for interactive selectors:
- New `phases.BuildModelMenuOptions(arch string) []string` filters out
recipes whose GPUArchAllowed excludes the host arch. Kimi-K2.6 is
hidden on sm_80/sm_86/sm_89/sm_90; shown on sm_100/sm_103/sm_120.
Recipes without GPUArchAllowed restrictions (Qwen variants) are
always present regardless of arch.
- 5 unit-test subtests cover Blackwell + non-Blackwell + unknown-arch +
unrestricted-recipe paths.
Tests added:
- internal/phases/02_model_gate_test.go: 2 phase-level tests
- internal/phases/model_selector.go + _test.go: 4 menu tests
LOC: ~60 production + ~150 tests. Pre-review: none.
Refs: M11 Slice 3, FR-M11-6, AC-M11-2; architect Review Notes F5
(phase 02 honors pre-set SelectedModel). UC-1 main + UC-1-A1 + UC-1-E2
covered.
…gating
Adds the shared safety predicate used by Slices 5-8 (update --api-only,
ml-node update, govmodel) to refuse disruptive operations during PoC,
PoC validation, and Confirmation PoC windows.
Implementation queries `inferenced query inference epoch-info -o json`
via docker exec on the api container. Schema verified against mainnet-1
(gonka-mainnet, epoch 265, block 4102021) on 2026-05-16. The response
exposes block_height, params.epoch_params.{epoch_length, poc_stage_duration,
poc_validation_delay, poc_validation_duration, confirmation_poc_safety_window},
and latest_epoch.poc_start_block_height. There is no direct current_phase
field — phase is derived from offset = block_height - poc_start_block_height:
- [0, pocStageDuration) → PoCGenerate (unsafe)
- [pocStage, pocStage + delay) → PoCGenerate (still unsafe; inter-stage gap)
- [pocStage+delay, +pocValDur) → PoCValidate (unsafe)
- [epochLength - cpocWindow, end) → ConfirmationPoC (unsafe)
- otherwise → Inference (safe)
Fail-closed contract (NFR-M11-4): chain query failure, parse failure,
non-numeric fields, missing fields, or out-of-window block all produce
SafetyCheck{Safe: false, Phase: Unknown, Reason: <err>}. Returns nil error
from Check() — the error channel is reserved for programming bugs only.
API:
- type PoCPhase string + 5 constants
- type SafetyCheck struct {Safe, Phase, CurrentBlock, NextPhaseBlock,
SecondsUntilSafe, Reason}
- type QueryRunner interface {Query(ctx, args...) ([]byte, error)}
- type PoCSafetyChecker {State, Runner, Timeout}
- NewPoCSafetyChecker(state) defaults to dockerExecQueryRunner + 2s timeout
- (c) Check(ctx) (*SafetyCheck, error)
Tests (16 subtests / ~250 lines):
- TestDerivePhase_TableDriven: 9 cases across all phases + 2 out-of-window
- TestDerivePhase_RealisticMainnetSnapshot: against captured mainnet JSON
- TestDerivePhase_{MalformedJSON,NonNumericFields}
- TestPoCSafetyChecker_HappyPath / _RunnerError / _Timeout / _MalformedQuery /
_DefaultTimeoutAppliedWhenZero: all fail-closed paths via injected fakeRunner
Pre-review: security-auditor (gating predicate for destructive ops).
Refs: M11 Slice 4, FR-M11-13, NFR-M11-4; architect Review Notes F10
(epoch-info schema verified against running mainnet before code written).
…allowlist, and lock
The M11 hotfix installer — security-critical surface that downloads,
verifies, and atomically swaps the Cosmovisor api binary outside PoC
phase windows.
New files (8):
- allowlist.go — ValidateReleaseURL with path-prefix host check
(rejects substring/suffix attacks like
github.com.evil.com or gonka-ai/gonka.evil.com)
- manifest.go — Manifest struct + ParseManifest with versioned-
contract gate (schema_version: 1 required; unknown
values rejected before any download per R-M11-3)
- lock_unix.go — AcquireLock using unix.LOCK_EX|LOCK_NB; ErrLockHeld
sentinel; build-tagged !windows
- installer.go — Installer struct + Install pipeline:
1. ValidateVersion (regex: v\d+\.\d+\.\d+(-suffix)*)
2. ValidateDeployDir (abs, clean, no .., not /proc /sys /dev)
3. Safety gate (Force short-circuits per F20)
4. flock acquire (refuses concurrent installs)
5. Manifest fetch + parse + version lookup
6. Allowlist BOTH manifest URL and binary URL (F16)
7. Capture os.Readlink for rollback (in-memory)
8. Download with 3 retries + 5min/attempt timeout
(4xx non-retryable, 5xx retryable with backoff)
9. VerifySHA256 (constant-time, F8)
10. chmod 0755, place via RenameFunc (EXDEV seam)
11. Atomic same-dir sibling symlink swap (NFR-M11-1)
12. RestartFunc; HealthCheckFunc; rollback on either
failure via second same-dir atomic rename
Function-typed SafetyCheckFunc dependency keeps
cosmovisor independent of chain (clean module boundary)
Updates to existing files (2):
- binary.go — VerifySHA256 uses crypto/subtle.ConstantTimeCompare
over hex-decoded byte slices (F8 fix); validates
expected-length BEFORE reading the file so attackers
can't probe filesystem state with malformed hashes
- go.mod / go.sum — promote golang.org/x/sys to direct (for unix.Flock)
Tests (4 new test files, 24 distinct cases across TC-6.* coverage):
- allowlist_test.go — 12 cases (happy paths + substring attacks + scheme attacks)
- manifest_test.go — 7 cases (happy, schema versioning, missing fields, forward-compat)
- lock_unix_test.go — 4 cases (acquire, contention, re-acquire after release, bad path)
- installer_test.go — 14 cases (ValidateVersion + ValidateDeployDir tables,
happy path, idempotent re-install, off-allowlist
manifest, safety unsafe, Force bypass, sha256
mismatch, restart rollback, health rollback,
EXDEV via injected RenameFunc, lock contention,
unknown version, schema_version 2 rejected,
constant-time sha guard, transient 5xx retried,
404 non-retryable)
Architect-approved exception: ~600 LOC in installer.go (target was 280) due
to comprehensive error coverage + helper functions; cyclomatic complexity
of Install is 33 (above golangci default 15) — annotated with
//nolint:gocyclo. Splitting would fragment the deliberately linear pipeline
that ~14 test cases exercise end-to-end. Two #nosec annotations on inherent
security/installer patterns (G302 chmod 0755 — binary must be executable;
G304 lock file path — constructed from validated DeployDir).
Pre-review: security-auditor + architect.
Refs: M11 Slice 5, FR-M11-7/8/9/10, NFR-M11-1/2/3, AC-M11-4/5/6/7/9;
architect Review Notes F6 (atomic rollback via sibling rename, not sudo
ln -snf), F7 (Download wrapper with injected http.Client), F8 (constant-
time sha256), F15 (Version+DeployDir validation), F16 (manifest URL also
allowlist-checked), F20 (Force short-circuits safety predicate), F23
(lock path defaults to <DeployDir>/.dapi/cosmovisor/.update.lock).
End-to-end CLI: `gonka-nop update --api-only --version v0.2.12-api-post3` fetches the race-releases manifest, gates on PoC safety (or skips with --force), invokes cosmovisor.Installer for download/verify/atomic-swap/ rollback, restarts the api container, polls /admin/v1/setup/report for health, and persists state.Versions.APIBinaryVersion on success. Flag surface (registered on existing updateCmd): - --api-only (bool) - --version <v> (required with --api-only) - --force (skips PoC safety check; loud warning emitted) - --api-only and --check are MarkFlagsMutuallyExclusive (architect cond #7) internal/config additions: - ImageVersions.APIBinaryVersion field (json:"api_binary_version,omitempty") cleared by state.Reset()'s existing `s.Versions = ImageVersions{}` line - RaceReleasesManifestURL constant — production manifest URL is hardcoded; no --manifest-url flag exists so there is NOTHING for an attacker to override (resolves Review Notes F16 by elimination rather than build tag) - ParsePostNumber + SortVersionsByPostN + LatestVersion — integer-aware sort so v0.2.12-api-post10 ranks above v0.2.12-api-post2 (Review Notes F17) - TestMainImageVersion_IgnoresAPIBinaryVersion invariant (Review Notes F27) internal/cmd/update_api.go bridges: - buildSafetyCheckFunc — adapts chain.PoCSafetyChecker → cosmovisor.SafetyCheckFunc (cosmovisor stays independent of chain) - buildAPIRestartFunc — `docker [compose] restart api` (with state.UseSudo) - buildAPIHealthCheckFunc — polls /admin/v1/setup/report every 2s Tests (10 cases across 3 files): - versions_apirelease_test.go: 4 cases (ParsePostNumber, Sort, Latest, MainImageVersion invariant) - update_api_test.go: 6 cases (required version, required deployment, invalid version rejection, build helpers non-nil, flag registration, field settability) The full happy-path install is covered by internal/cosmovisor/installer_test.go; this slice's tests exercise the cmd-layer wiring and flag handling. Refs: M11 Slice 6, FR-M11-7/8/11/17, AC-M11-4/5, NFR-M11-3; architect Review Notes F16 (no manifest-URL flag bypass), F17 (integer postN sort), F27 (MainImageVersion invariant test), F20 (Force short-circuit honored via cosmovisor.InstallOpts.Force flag).
Per-governance-model PoC opt-in management. Three tx subcommands sign with
the cold key (KEY_NAME from config.env) and refuse to run during PoC /
Confirmation PoC windows unless --force is set. Plus a read-only `list`
subcommand that shows the chain's per-(participant, model) state.
CLI surface:
- gonka-nop govmodel list
- gonka-nop govmodel delegate <model-id> <delegatee-addr> [--force]
- gonka-nop govmodel refuse <model-id> [--force]
- gonka-nop govmodel declare-intent <model-id> [--force]
internal/chain additions:
- ValidateGonkaAddress — bech32 SHAPE check (prefix gonka1 + 38-58
charset chars). Full bech32 checksum needs
cosmos-sdk (~10MB dep) just for client UX;
chain rejects invalid addresses on tx submit
anyway (Review Notes F22 decision)
- TxBroadcaster interface — production impl shells out via `docker exec
-i api inferenced tx inference <action>...`
with password PIPED via cmd.Stdin so it
NEVER appears in argv (/proc/PID/cmdline)
(NFR-M11-6 secret-hygiene contract)
- SetPoCDelegation / RefusePoCDelegation / DeclarePoCIntent typed
wrappers around TxBroadcaster
- PoCDelegationState + ListPoCDelegations + parsePoCDelegationResponse
— reads `inferenced query inference poc-delegation <participant> -o json`
schema (delegations / refusals / intents arrays)
- QueryRunnerFromState constructor (reuses dockerExecQueryRunner)
Critical security ordering (architect cond #9): runGovmodelTx calls
chain.NewPoCSafetyChecker.Check() BEFORE chain.LoadKeyringPassword so
fail-closed safety doesn't unnecessarily decrypt secrets in memory.
When --force is set, the safety call is skipped ENTIRELY (not invoked
with result ignored) per Review Notes F20.
Tests (12 cases):
- TestValidateGonkaAddress (8 subcases incl. typo, wrong prefix, charset)
- TestSetPoCDelegation/Refuse/DeclarePoCIntent_BuildsCorrectArgs via
recordingBroadcaster fake (asserts password recorded in stdin-equivalent
position, never in argv)
- TestParsePoCDelegationResponse (happy + empty + malformed)
- TestListPoCDelegations (happy + query error fail-closed)
govmodelCmd registered on rootCmd; gonka-nop govmodel --help now works
in the live binary.
Refs: M11 Slice 7, FR-M11-12/13/14, NFR-M11-6, AC-M11-8; architect Review
Notes F22 (bech32 shape-check, no cosmos-sdk), cond #9 (safety before
keyring), F20 (Force short-circuits safety predicate). Pre-review:
security-auditor.
Adds the M6.5 remainder: `gonka-nop ml-node update --id <node-id>` issues
a PUT /admin/v1/nodes/:id with CLI-flag overrides merged onto the existing
record fetched via GET. Refuses to run during PoC/CPoC unless --force.
CLI:
gonka-nop ml-node update --id node1 --model moonshotai/Kimi-K2.6
gonka-nop ml-node update --id node1 --tp 4 --max-concurrent 128
gonka-nop ml-node update --id node1 --model Qwen/QwQ-32B --vllm-args "--max-num-seqs 64"
gonka-nop ml-node update --id node1 --force --model X
Workflow:
1. chain.PoCSafetyChecker.Check() — refuses if unsafe (skip with --force)
2. GET /admin/v1/nodes — find target by ID; error if missing
3. mergeNodeUpdate — flags override, unspecified fields preserved
- --model with recipe.VLLMArgs populated → recipe args verbatim
- --vllm-args overrides recipe args (with explicit warning)
- --tp patches --tensor-parallel-size in args (inserts if missing)
- --max-concurrent → MaxConcurrent field
4. Idempotency check: byte-equal merged vs existing → "no changes required"
5. PUT /admin/v1/nodes/:id with merged body
Tests (12 cases):
- TestMergeNodeUpdate: identity, MaxConcurrent override, TP override,
Model swap with recipe, --vllm-args override+warning, unknown model
- TestOverrideTP: replaces existing + inserts when missing
- TestPutAdminNode: happy path against httptest server (asserts method+path+body),
5xx error path
- TestListKnownNodeIDs
Refs: M11 Slice 8, FR-M11-13/16, AC-M11-3; finishes M6.5 remainder.
NodeStatus gets a new GovernanceModels []GovModelState field, populated by cmd/root.go's runStatus via chain.ListPoCDelegations (best-effort; chain query failure → nil → section omitted entirely). The display appends a "Governance Models" section after the existing sections. Architect condition #10: when chain returns zero governance state for the participant, the section header is NOT printed — operators who haven't touched governance see no noise. Verified by TestPrintGovernanceModels_OmittedWhenEmpty. internal/status: - NodeStatus.GovernanceModels []GovModelState (new field) - GovModelState local type mirrors chain.PoCDelegationState (status pkg stays independent of internal/chain; cmd layer bridges) - printGovernanceModels in display.go — renders delegated / refused / intent / unknown states with consistent color treatment - Display() invokes printGovernanceModels last so it doesn't disturb pre-M11 byte-equivalent output when empty internal/cmd/root.go: - runStatus enriches NodeStatus.GovernanceModels via fetchGovernanceModelsForStatus (5s timeout, fail-soft) - New helper imports chain.QueryRunnerFromState + ListPoCDelegations Tests (2 new cases): - TestPrintGovernanceModels_OmittedWhenEmpty (architect cond #10) - TestPrintGovernanceModels_RendersDelegationLine Note: F18 (`validator_in_set` false-FAIL override) is ALREADY IMPLEMENTED at status.go:reconcileValidatorStatus with TestReconcileValidatorStatus. Slice 9 scope reduced accordingly. Refs: M11 Slice 9 (final), FR-M11-14/15/18, AC-M11-3; architect Review Notes F18 (already in code) + cond #10 (omit-when-empty).
Discovered during the M11 binary's first real-world deploy on mainnet-2 (2026-05-16):
phase 02 calls ui.Input("Custom MLNode image") and ui.Select("Attention backend")
unconditionally, but neither prompt has a default value, so non-interactive mode
errors with 'no default value for required input' even though the operator passed
--yes.
Fix:
- MLNode image prompt: when ui.IsNonInteractive(), skip the prompt entirely and
log that the auto-selected image (state.MLNodeImageTag from selectMLNodeImage)
is being used. Explicit --mlnode-image flag still wins via the existing
state.CustomMLNodeImage check.
- Attention backend prompt: when ui.IsNonInteractive() and no explicit
--attention-backend was passed, default to FLASHINFER (the recommended option
per the existing prompt label) without asking.
Operator UX preserved for interactive use; --yes flows now complete phase 02
without errors. Caught by M11 deploy on mainnet-2 (8x A100 80GB sm_80).
Refs: M11 deploy hotfix.
…-byte ceiling) Discovered on 2026-05-16 mainnet redeploy: a 24-char keyring password caused 'bcrypt: password length exceeds 72 bytes' during inferenced keys add. Cosmos-sdk's file keyring backend uses bcrypt to derive the encryption key, and concatenates the user password with internal metadata (key name, salt, derivation context) before hashing. With long passwords the concatenated input exceeds bcrypt's hard 72-byte limit and the key creation fails silently from NOP's perspective. Practical safe ceiling is ~50-60 chars. Hard-cap at 50 with an explicit error so operators don't hit the cryptic bcrypt error mid-deploy. Both runQuickReal and runSecureReal had the missing-too-long guard. Refs: M11 mainnet deploy hotfix; pairs with the earlier 71369ed phase-02 non-interactive fix.
CometBFT's /status endpoint reports height=0 + catching_up=true for the
ENTIRE state-sync window (can be 5+ minutes), giving operators no
indication that anything is happening. Discovered the hard way on the
2026-05-16 mainnet redeploy: nine peers connected, no visible block
progress, no clue whether the node was healthy.
This adds a 'Sync' section to gonka-nop status that reads node container
logs and parses CometBFT's statesync chunk-progress lines:
Sync
State-syncing snapshot @ height 4,102,000 — chunk 309/1168 (26%)
→ ETA: ~2m30s at observed chunk rate
→ Peers: 9 connected
→ Last log: Applied snapshot chunk to ABCI app chunk=309 ...
Phases:
- Initializing — RPC unreachable or no peers
- SnapshotDiscover — peers connected, awaiting snapshot offers
- SnapshotApplying — chunks streaming (height=0 in /status is normal here)
- BlockSync — snapshot done, fast-syncing block-by-block
- Synced — caught up
Implementation:
- New internal/status/sync.go with SyncPhase enum, SyncStatus struct,
DeriveSyncStatus pure function (RPC + optional DockerLogsTail injection),
parseSyncFromLogs regex-based parser, EstimateETA wall-clock projection
- NodeStatus.Sync field populated by fetchBlockchainStatus
- New printSync() display section, always shown when Sync is populated
(confirms 'Synced' reassuringly; explains pre-block phases meaningfully)
- defaultDockerLogsTail shells out to 'docker logs --tail N node' (5s timeout)
- DockerLogsTail func-type seam lets tests inject log content
Tests (10 cases): Synced / BlockSync / HeightZero-NoPeers / NoLogs /
LogsApplying / LogsDiscovery / LogReadError + EstimateETA + regex +
ANSI stripping. All injected fakes, no real docker dependency.
Refs: M11 mainnet-1 deploy operational gap.
CometBFT writes colorized logs by default (e.g. 'chunk=\x1b[0m309'). My
regex 'chunk=(\d+)' was looking for digits immediately after '=' but the
log line has '=\x1b[0m' between. Result: the parser fell through to
'Connected to N peers, awaiting state-sync snapshot offers' instead of
showing chunk progress, even though the logs were full of valid chunk
lines.
Discovered immediately on the first live test against mainnet-1 mid-sync.
Stripping ANSI once at the top of parseSyncFromLogs fixes all three regex
matchers (applied chunk, discovered snapshot, block-sync target).
Live result on mainnet-1:
Sync
State-syncing snapshot @ height 4,102,000 — chunk 845/1168 (72%)
ETA: ~1m0s at observed chunk rate
Peers: 10 connected
…ate)
Slice 7's parser used struct tag json:"delegate" but the chain returns
delegate_to. Discovered immediately after broadcasting the first real
on-chain delegation tx on mainnet-1 (height 4,104,217) — gonka-nop
govmodel list rendered an empty section even though the chain confirmed
the delegation.
Mainnet schema verified live: poc-delegation query returns
delegations: [{model_id, delegator, delegate_to}]
Fixed field tag + unit-test fixture body.
Refs: M11 mainnet hotfix #4.
… module Adds gonka-nop collateral subcommands for the chain's separate 'collateral' cosmos module (not part of inference module). Discovered when the user hit 'Collateral 0.0% (needed 3,095 ngonka)' on the dashboard and realized our fresh deploy hadn't deposited anything. Subcommands: gonka-nop collateral show [addr] # defaults to our cold key gonka-nop collateral deposit <amount> # e.g. 3095ngonka gonka-nop collateral withdraw <amount> # unbonds (1 epoch on mainnet) Chain primitives wrapped: Tx: inferenced tx collateral deposit-collateral [amount] Tx: inferenced tx collateral withdraw-collateral [amount] Query: inferenced query collateral show-collateral [participant] NotFound query response (fresh participants) handled gracefully — returns Found=false instead of error, prompts operator with 'deposit' hint. Plumbing additions: - internal/chain/collateral.go — GetCollateral, DepositCollateralViaModule, WithdrawCollateralViaModule, CollateralBalance struct, ParseCollateralResponse, ValidateCoinAmount (regex enforces <digits><alpha-denom>, rejects shell injection like '100;rm') - internal/chain/govmodel.go — TxModuleBroadcaster interface + BroadcastModule method on dockerExecTxBroadcaster (lets us target non-'inference' modules while keeping the existing TxBroadcaster shape for govmodel) - internal/cmd/collateral.go — cobra tree mirroring govmodel pattern - internal/cmd/root.go — registers collateralCmd Security: - Password piped via stdin (cmd.Stdin) — never argv (NFR-M11-6 pattern reused) - Amount regex defangs shell injection - All txs signed with cold key (state.ColdKeyName), keyring loaded same as govmodel Tests pass; live verification next. Refs: post-M11 add-on, prompted by mainnet deploy collateral gap.
…+ format human-friendly
Two improvements from the first live test of the new collateral command:
1. show: NotFound was missed because inferenced writes the error to stderr
and cmd.Output() only captures stdout. cmd.ExitError.Stderr has it —
plumb it through isCollateralNotFound. Result: 'No collateral deposited
(chain returned NotFound)' now renders correctly for fresh participants.
2. deposit/withdraw: previous code printed 'code: 0' which was the BROADCAST
acceptance, not the on-chain execution result. Misleading when the tx
later fails (e.g. insufficient funds, code 5). Now parses txhash from the
YAML output, prints a concise success line + tells the operator how to
check the actual tx_result (which lives in a future block).
3. format: 'amount: 1000000 ngonka' now also shows '~0.001 GONKA' for sums
≥ 1M ngonka.
Verified on mainnet-1:
- collateral show → 'No collateral deposited (chain returned NotFound)' ✓
- collateral deposit 3095ngonka → tx DDB22BFD... broadcast,
landed at height 4,105,851 with code 5 'insufficient funds 0<3095' ✓
- feegrant cold→warm covered the gas (49,704 used, no balance burned)
Refs: post-M11 collateral add-on, mainnet smoke-test fixes.
Surfaces participant collateral balance alongside the existing Governance
Models section. Format:
Collateral
✓ 10000 ngonka (0.000010 GONKA)
When no collateral deposited (fresh participant):
Collateral
⚠ None deposited — `gonka-nop collateral deposit <amount>` to enable PoC eligibility
When fetch fails: section omitted entirely (best-effort, like Governance Models).
Implementation mirrors GovernanceModels pattern: NodeStatus.Collateral
field (status pkg stays independent of internal/chain), cmd/root.go's
runStatus bridges via chain.GetCollateral with 5s timeout.
Live-verified on mainnet-1.
…R-M12-15) Extends FetchImageVersions to surface the versiond image tag from upstream docker-compose.yml. The Versiond field joins Node and API as a critical-field check requirement: an upstream branch missing the versiond line would silently render an empty image tag in NOP-generated compose, breaking FR-M12-1. Fallback mainnet/testnet versions populate Versiond to keep offline setups working. Existing extractImageTag regex correctly disambiguates proxy vs proxy-ssl via exact-match comparison; added test coverage at the ParseComposeImageVersions API level for that property. Net production diff: +12 / -3 lines on internal/config/versions.go. Tests: 4 new (TestParseComposeImageVersions_Versiond, TestFallback_VersiondPopulated, TestFetchImageVersions_VersiondMissingErrors, TestParseComposeImageVersions_ProxySSLDisambiguation) + 2 fixture updates (testComposeContent, testnetCompose) so existing tests pass the new critical-field check. Refs: M12 Slice 1, FR-M12-15.
…M12-4) Implements the atomic mutation helper for M12 Slice 2. Sequence is read → transform → yaml.Unmarshal validate → write backup (0600, with crypto/rand 8-hex suffix) → write sibling tmp via os.CreateTemp → os.Rename over original. Backup created AFTER transform succeeds so failing transforms leave no .bak; YAML validation happens BEFORE rename so invalid output leaves the original untouched. Acquires per-directory advisory flock at <dir>/.nop.lock via the M11 cosmovisor primitive (non-blocking; returns wrapped cosmovisor.ErrLockHeld on contention so callers can errors.Is(...) it). Preserves source file mode on the new file; backup always 0600 since it may pass-through sensitive env values. Path validation rejects relative paths, paths containing "..", and NUL bytes BEFORE any I/O (security rule). Test contract substitution: TC-MUTATION-5 (SIGINT subprocess) replaced with TestAtomicReplace_RecoveryFromHalfState_BakAndTmpPresent — invariant-based test that proves no-half-state via stale-bak/stale-tmp recovery, deterministic and CI-stable. 12 tests + 4 subtests, all green with -race. Backup rotation deferred (tracked under UC-3-EDGE2). Refs: NFR-M12-4, PRD § Compose Mutation Strategy, plan-m12.md Slice 2, architect pre-review 2026-05-17.
…/2/3/10) Security review (security-auditor) APPROVED-WITH-CONDITIONS 2026-05-17: - strings.ReplaceAll (not os.Expand) — state values may contain literal $ - empty KeyName/AccountPubKey/KeyringPassword fail fast (not silent empty) - error messages never echo state values - no log/slog/fmt.Print/ui imports in this package (canary-tested) - regex anchored at string boundaries (Go default, not multiline) — rejects newline/NUL/prefix/suffix injection (TC-SEC-4 + 3 new tests) - bare semver expansion narrow by design — non-semver tags require --allow-non-upstream + full image ref - placeholder tokens factored into unnamed constants (goconst) - rendered compose contains plaintext KEYRING_PASSWORD; caller (Slice 5) MUST write 0600. See VersiondServiceBlock godoc warning. This slice adds verbatim upstream service blocks copied from gonka-ai/gonka:main/deploy/join/docker-compose.yml at PRD time (per PRD R-M12-1, hardcoded locally to avoid drift between FetchImageVersions snapshots and migrate-devshardd invocations), the substitution function that renders them with state-derived values, and the image allowlist regex enforcing INV-M12-3/4. Tests: 26 (13 substitution + 13 allowlist incl. injection/ReDoS guards). All green; lint 0 issues. Refs: FR-M12-1/2/3/10, INV-M12-3/4, NFR-M12-1, R-M12-1, security pre-review.
state.DevshardConfig holds the optional Postgres connection params for
the devshardd payload-storage backend. Empty PostgresHost = file backend
(default per UC-1). Pre-M12 state.json files load with empty
DevshardConfig per FR-M12-14 backward-compat contract (omitempty per
field; mtime preserved on load).
5 new setup flags: --postgres-host, --postgres-port, --postgres-db,
--postgres-user, --postgres-password. Password may also come from
GONKA_POSTGRES_PASSWORD env var which is scrubbed after read so
subprocesses don't inherit it (NFR-M12-1). When --yes mode is active
and --postgres-host is set without a password from any source, fatal
before state mutation (UC-2-A2).
state.json mode remains 0600 — guarded by new TestState_SaveFileMode_Is0600
(TC-SEC-7, R-M12-5). Reset() clears DevshardConfig.
Empty DevshardConfig serializes as `"devshard_config":{}` because Go's
omitempty does not zero-check inner struct types; this is acceptable
backward-compat (pre-M12 readers ignore unknown keys) and verified by
TestState_JSON_EmptyDevshardConfig_NoPostgresFieldsLeak which guards
against accidental postgres_* field serialization at zero state.
Tests: 6 state tests + 7 flag tests, all green; lint 0 issues.
Refs: FR-M12-5/6/14, NFR-M12-1, R-M12-5, AC-M12-8.
…-1/2/3)
Phase 05 compose generator now emits the versiond service block via
devshard.RenderVersiondBlock (Slice 3), the 7 api env additions for
devshardd integration (DAPI_API__NODE_MANAGER_GRPC_PORT/_LOCK_TTL_SECONDS,
PGHOST/PORT/DB/USER/PASSWORD), and the proxy GONKA_API_EXEMPT_ROUTES
default now includes 'subnet devshard' tokens with new EXEMPT_CONN_LIMIT=200.
Phase 05 config.env generator gains an optional POSTGRES_* block emitted
only when state.DevshardConfig.PostgresHost is non-empty. Values flow
into the api container at `docker compose up` via the existing MergeEnv
path (internal/docker/compose.go), resolving upstream's
${POSTGRES_*:-default} substitutions in api env. File mode remains
0600 (no new credential-exposure surface).
resolveVersions adds a defensive Versiond fallback (FR-M12-15) — defends
against state migrated from pre-Slice-1 binaries that lacked the field.
Architect pre-review APPROVED-WITH-CONDITIONS 2026-05-17 (4 stale-design
items dropped, 1 added: Postgres values flow via config.env not shell env).
Existing TestGenerateDockerCompose + TestGenerateConfigEnv + 2 NAT tests
updated for new state fixture (KeyName/AccountPubKey/KeyringPassword
required by RenderVersiondBlock) and new EXEMPT_ROUTES/CONN_LIMIT
assertions. 7 new M12 tests cover versiond block, api env, proxy env,
Postgres emit-on-host, no-emit-on-empty, fallback, two-run idempotency,
password-exactly-once. All green; lint 0 new issues.
Slice 5b (devshards dirs + port 9400 + versiond pull WARN) follows.
Refs: FR-M12-1/2/3/15, NFR-M12-1, AC-M12-1/4, INV-M12-1/2, R-M12-5.
…-4/12)
Slice 5b operational glue per architect-approved split:
- ensureDevshardsDirs(outputDir) creates devshards/{bin,data} at 0750
with full idempotency: pre-existing dirs left untouched (mode
preserved per UC-1-A3 — operator may want 0700 or a symlink to a
big-disk mount per UC-1-EDGE3). Pre-existing 'bin' as a non-directory
returns a clear error.
- checkPorts adds port 9400 (api NodeManager gRPC for devshardd) per
UC-1-E4 / TC-1.8. Added to both mocked and real check lists.
- checkVersiondPullable runs 'docker pull --quiet
ghcr.io/product-science/versiond:<tag>' as a non-fatal WARN check.
Per architect Q4 / option (b): migrate-devshardd (Slice 6) will run
its own FATAL check; phase 01 stays single-purpose. 60s timeout via
context.WithTimeout. Honors state.UseSudo.
Tests: 4 dir helper tests (create, idempotent, leaves-existing,
conflicting-file), all green; lint clean (full-package run); existing
phases test suite unaffected.
Refs: FR-M12-4, FR-M12-12, UC-1-A3, UC-1-E4, UC-1-EDGE3, TC-1.8,
TC-IDEM-3, architect Q3/Q4/Q5 (option b: WARN in setup, FATAL in migrate).
Slice 7 — surgical api image swap for INV-M12-4 (the only NOP-supported
path to deploy a non-upstream api image). Validates against the
devshard.IsUpstreamAPIImage allowlist (Slice 3); bare-semver input is
expanded to canonical form via ExpandBareSemver. Non-upstream images
require explicit --allow-non-upstream + WARN.
Compose mutation uses docker.AtomicReplace (Slice 2): flock-serialized,
YAML-validated, .bak.<ts>.<rand> rollback path. Container recreation
uses new docker.RecreateService("api") wrapper around
`docker compose up -d --no-deps --force-recreate api` — only the api
container is touched; node/proxy/bridge/explorer/versiond stay running.
This is the path we'll use to revert mainnet-1 from
ghcr.io/segovchik/api:otel-test-4 to upstream ghcr.io/product-science/api
during M12 Phase 4 deploy.
Tests: 12 (truth-table for imageIsAPI, swap rewrites only api,
preserves indentation, only-first-match, skips comments, no-api
errors). All green; lint clean (one nolint:gocyclo on runUpdate
dispatcher with 3 short-circuit branches).
Refs: FR-M12-10, INV-M12-3, INV-M12-4, plan-m12.md Slice 7.
…ntMissing (FR-M12-11) M12 Slice 8a — re-grants missing ml-ops permissions to the warm key via a SINGLE grant-ml-ops-permissions tx. Idempotent: 27/27 already-granted exits success without broadcasting (NFR-M12-5, AC-M12-7). Security: stdin-pipe password ONLY (reuses M11 dockerExecTxBroadcaster which already has the right exec contract). Password is NEVER echoed into argv, log lines, or error wraps — redactPassword scrubs every error-wrap path; canary-tested via TestGrantMissing_ErrorNeverContains Password. NEVER calls register-new-participant (chain code 1110 on existing accounts): pinned to the single grantAction constant and verified by TestGrantMissing_NeverConstructsForbiddenAction. Balance pre-check via BalanceQuerier interface (mockable for tests, production wraps the existing dockerExecQueryRunner). Insufficient balance → no tx, error names the shortfall + signer address. Security-auditor pre-review APPROVED-WITH-CONDITIONS 2026-05-17 — all conditions applied: interface seam (no exec.Command mock needed), balance check in chain package, redactPassword helper, no full argv echo, register-new-participant guarded by test assertion. Tests: 18 (14 permissions + 4 balance parse). All green; lint clean. Slice 8b (reset base + status Devshardd section + api-image WARN) ships separately per security-auditor split recommendation — operationally orthogonal, AC-M12-2 mainnet recovery depends on 8a only. Refs: FR-M12-11, NFR-M12-1, NFR-M12-5, AC-M12-2, AC-M12-7, security-auditor pre-review 2026-05-17.
…alidation) Live mainnet-1 deploy 2026-05-17 surfaced two production-only issues that unit tests couldn't have caught: 1) grant-ml-ops-permissions CLI takes EXACTLY 2 positional args (granter-key-name, grantee-address) — the chain grants the full ml-ops set itself, the caller does NOT pass a filtered permission list. Initial Slice 8a design passed (grantee + 8 missing actions) = 9 args → CLI rejected with "accepts 2 arg(s), received 9". Fix: GrantMissing now passes only (signerKey, grantee). The DiffPermissions list is informational (gates the call + operator log). Updated TestGrantMissing_HappyPath assertion. 2) --gas 200000 default in dockerExecTxBroadcaster.Broadcast and .BroadcastModule was too low for grant txs — actual usage 200657 > limit 200000 → "out of gas" mid-block. Bumped both to 2000000 (matches the legacy 07_registration.go hardcoded value). Chain charges only what's used; 10x headroom is safe. Live validation: mainnet-1 swapped api segovchik/api:otel-test-4 → ghcr.io/product-science/api:0.2.12 via M12 update --service api --image (Slice 7), then grant tx confirmed at block 4119697. Authz query shows 20 grants on warm key including MsgSubmitPocValidationsV2 — the v0.2.12 PoC validation permission. Setup/report still shows "8 missing" because the api check looks for the pre-v2 name MsgSubmitPocValidation; this is an api-side display drift, NOT a real permission gap. Refs: M12 deploy, Slice 7 + 8a live integration verified.
- Bump workflow Go versions 1.25.8 → 1.25.10 (ci, release, security)
- Extract testAPIBinaryVersion + defaultAttentionBackend constants (goconst)
- Add #nosec G204 justifications on hard-coded iptables/sh argv (gosec)
- Drop dead `if phase != PhaseInference {}` no-op in poc_safety.go
- Fix American-English spelling: signalled → signaled (per project rule)
No behavior change.
…govmodel Fresh deploys had no signal that per-model PoC opt-in was even a thing — status omitted the Governance Models section entirely when empty, and the only path to find a delegate was knowing an address out of band. status: - NodeStatus.AvailableGovernanceModels []string (chain-known models, populated best-effort in root.go via fetchAvailableGovernanceModelsForStatus) - printGovernanceModels renders an informative section when delegations are empty but chain has governance models — lists them + the `govmodel delegate` command. Residual "omit when both empty" preserves architect cond #10 for the chains-with-zero-models edge case. govmodel: - New `gonka-nop govmodel candidates <model-id>` subcommand. Queries hardware-nodes-all, filters by model, aggregates per participant, ranks best-first (INFERENCE count desc, FAILED count asc, total nodes desc). - Flags: --healthy-only (drop any-FAILED candidates), --limit N (default 15). chain: - ListGovernanceModels (wraps `inferenced query inference models-all`) - ListHardwareNodesAll (wraps `inferenced query inference hardware-nodes-all`) - DelegationCandidate{Participant, InferenceNodes, FailedNodes, OtherNodes, Hardware} with IsHealthy()/TotalNodes() helpers - RankCandidatesForModel(entries, modelID) — deterministic ranking + compact hardware summary ("8x B300 ×17 + 8x B200 ×3 + +1 more") Tests: - 13 new chain tests using a trimmed realistic mainnet fixture - 2 status display tests covering the new empty-with-available path - All pass with -race; vet+golangci-lint clean Live-verified on mainnet-1 (epoch 272): collateral deposit 5M ngonka + Kimi-K2.6 delegation to gonka1qa90... both committed via this code path.
- Drop "What's New (v0.2.1-rc5)" section: tied to a specific RC and mentions personal namespace images that should not live in public docs. - Drop "GPU-Specific Deployment Guides" section entirely: contained hardcoded image tags (mlnode:3.0.12-post6, :3.0.12-post6-blackwell, :3.0.13-alpha4) that drift out of date as upstream churns, and pointed operators at a personal segovchik namespace for B300 builds. Per-host tuning notes belong in operator-specific docs or release notes, not the entry-point README. README now focuses on what NOP does and how to use it generically. Trimmed 357 → 216 lines.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| return "", fmt.Errorf("random suffix: %w", err) | ||
| } | ||
| bakPath := fmt.Sprintf("%s.bak.%s.%s", path, ts, suffix) | ||
| if err := os.WriteFile(bakPath, original, 0600); err != nil { |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #11 +/- ##
==========================================
+ Coverage 37.34% 41.67% +4.33%
==========================================
Files 32 59 +27
Lines 6009 7868 +1859
==========================================
+ Hits 2244 3279 +1035
- Misses 3664 4410 +746
- Partials 101 179 +78 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…guide xref
README:
- Add govmodel {list,candidates,delegate,refuse,declare-intent},
collateral {show,deposit,withdraw}, and repair-permissions to the
Commands table — these landed across M11/M12 but were never linked
from the README, leaving operators with no signal they exist.
- Cross-reference the new guide at the bottom of the Commands section.
docs/nop-guide.md (new, ~270 lines):
- Three-topology walkthrough (full / multi-server / network-only) with
copy-pasteable non-interactive setup commands per topology.
- Post-deploy section covers the post-grace (epoch ≥ 181) collateral
deposit flow with the live formula and the multi-model PoC opt-in
flow (delegate / refuse / declare-intent + the candidates discovery
command). Penalty hierarchy from gonka.ai/docs documented verbatim.
- Day-2 ops + recovery sections cross-reference existing M9/M11
commands (update --check/--service/--api-only, repair,
repair-permissions, reset, cleanup).
- Flags the known multi-server gotcha: mlnode-registration.json
defaults id=node1 which can clobber an existing validator-local node
if not renamed before ml-node add.
- Flags the api-induced restart loop on freshly-added remote mlnodes
and documents the safe disable → wait → enable sequence.
Sources: live-validated on mainnet-1 (epoch 272) 2026-05-22 — every
command in the guide was run end-to-end before commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.