Skip to content

development.md

Latest commit

 

History

History
229 lines (177 loc) · 9.42 KB

development.md

File metadata and controls

229 lines (177 loc) · 9.42 KB
Raw

Cerebro Development Guide

This document describes the current bootstrap service on main. Historical warehouse, Cedar runtime, and legacy port-variable workflows are retired; use the Makefile and README.md as the source of truth.

Prerequisites

  • Go 1.26+; the repository pins go1.26.5 in go.mod.
  • Rust 1.93.1 with Cargo; rust-toolchain.toml installs the pinned toolchain, rustfmt, Clippy, and the Wasm build target.
  • cargo-deny 0.20.2 for local Rust dependency-policy checks.
  • The nightly-2026-06-09 toolchain and cargo-fuzz 0.13.1 only when running the static validator fuzz target locally.
  • cargo-llvm-cov 0.8.7 plus the llvm-tools-preview Rust component for the Rust coverage gate.
  • Docker and Docker Compose for the durable local stack.
  • Make.

Local Setup

git clone https://github.com/writer/cerebro.git
cd cerebro
make doctor
make build

Run the lightweight server without durable stores:

make serve-dev

Run the durable local stack with NATS JetStream, Postgres, Neo4j, Cerebro, and the local bearer key local-dev-key:

docker compose pull
docker compose up -d

Plain Compose initializes the local Postgres volume with the compose-file password. The onboarding Make targets use tmp/local-postgres-password. Before switching from plain Compose to make agent-onboard-e2e or make github-business-demo, run docker compose down -v to recreate local volumes, or run the Make target with CEREBRO_LOCAL_POSTGRES_PASSWORD=cerebro to reuse that volume. docker compose down -v deletes local stack data.

Use docker compose -f docker-compose.yml -f docker-compose.build.yml up --build -d when you need the durable stack to run the current checkout instead of the published image.

The compose stack uses service-local CEREBRO_* variables. For a standalone local template, start from .env.example.

Common Commands

make build          # compile ./bin/cerebro
make serve-dev      # run local server with acknowledged dev-mode opt-out
make test           # go test ./...
make lint           # golangci-lint over app packages
make changed-check  # changed-path validation for local iteration and hooks
make proto-lint     # buf lint
make check          # build, tests, lint, proto lint, structural checks, arch tests
make verify         # CI-parity validation
make sourcegen-check # connector definition sourcegen readiness
make readme-check   # README drift checks
make docs-drift-check  # generated docs drift checks
make oss-audit      # public repository hygiene scan
make clean          # remove bin/ and Cargo target output

Rust tooling runs on Unix hosts. Linux is the CI and release host contract; macOS is supported for local development. The graph action generator depends on Unix file-permission semantics, so Windows is not a supported host. The static validator guest targets wasm32-unknown-unknown and is built through make graphagent-static-validator-check.

Run the complete Rust checks with:

make rust-deny graph-action-check rust-wasm-check

The checks include formatting, Clippy, tests, warning-free rustdoc, dependency advisories and policy, the generated graph action registry, and all embedded Wasm artifacts.

Rust dependency versions and shared features are owned by [workspace.dependencies] in the root Cargo.toml. Workspace members inherit those entries with workspace = true; a member may add features required by that crate but may not repeat or override a version, path, or Git source.

Every workspace member also inherits [workspace.lints]. Unsafe code is denied by default, including unsafe operations inside unsafe functions. The only allowed exceptions are the narrow audited Wasm ABI modules that expose guest functions and the shared guest-memory module that contains their documented pointer operations. Run make rust-workspace-policy after adding a workspace member, dependency, or lint.

The static Cypher validator has deterministic security properties for parser stability, refusal preservation, row-limit monotonicity, and ABI values. Run them directly with:

make rust-validator-properties

Coverage-guided fuzzing is scheduled weekly and can be started manually from the Rust validator fuzz GitHub Actions workflow. It is not a pull-request wall-clock gate. To reproduce the bounded 10,000-input smoke locally:

rustup toolchain install nightly-2026-06-09 --profile minimal
cargo +nightly-2026-06-09 install cargo-fuzz --version 0.13.1 --locked
make rust-validator-fuzz-smoke

New fuzz crashes are written under internal/graphagent/staticvalidator/fuzz/artifacts/. Move a minimized input into internal/graphagent/staticvalidator/fuzz/corpus/validate/ and add a deterministic regression assertion before fixing the parser.

The source record kernel has two focused checks:

make sourceruntime-record-kernel-check
make rust-source-kernel-evidence

The first compiles the native tests and the embedded wasm32-unknown-unknown guest. The second reports the guest's imports, exports, and byte size next to the runtime floor produced by the repository's pinned standard Go toolchain. The comparison is a capability-surface check, not a throughput benchmark.

make rust-wasm-check also enforces a 90% Rust workspace line-coverage floor and runs 20 fixed iterations of each embedded Wasm benchmark. The coverage floor was set below the measured 90.09% workspace baseline so the gate catches meaningful regressions without treating rounding noise as a failure. wasm_abi.rs files are excluded because they are target-specific allocation and descriptor glue exercised by the Go/Wasm integration tests, not by native Rust coverage. No validator or evaluator logic is excluded.

Run these lanes directly with:

make rust-coverage
make rust-benchmark-smoke
make rust-event-admission-benchmark

The benchmark smoke lane covers the static validator, source coverage, MITRE, Panopticon extraction, and the shared Go host invocation path. It writes the complete Go benchmark output to tmp/rust-wasm-benchmark.txt. CI checks that the harnesses compile and finish; latency numbers remain evidence for comparison and are not pass/fail thresholds.

make build places cerebro and cerebro-event-admission-worker together in bin/. Source sync resolves the worker beside the running cerebro binary by default. make rust-event-admission-benchmark builds the release worker, checks native CBOR and JSON outcomes against the embedded Wasm corpus, and records boundary-specific Go and Rust results under tmp/.

Focused validation:

make workflow-e2e-test
make workflow-replay-test
make finding-rule-test
make graph-rebuild-dryrun
make sourcecoverage-evaluator-check

Embedded Rust Wasm Modules

scripts/embedded_wasm.py is the module registry for Rust packages embedded in the Go runtime. Each entry owns the Cargo package, checked-in artifact, stable Make targets, changed-path routing, and canonical artifact platform.

Use the module's existing Make targets to check or regenerate an artifact. Add new modules to the registry instead of adding build commands directly to the Makefile or scripts/changed_checks.py. Artifact generation that requires Linux-x86_64 fails on other platforms; checks still compile the module and report when byte comparison is deferred to CI.

tools/archtests/embedded_wasm_artifacts.json records the canonical artifact path, SHA-256 digest, exact byte size, ABI version, pinned Rust builder inputs, and an explicit maximum size for every registered module. After regenerating the Wasm files on Linux-x86_64, run make rust-wasm-manifest-generate and commit the manifest update. make rust-wasm-manifest-check reports digest, ABI, size, or budget drift; increasing a budget requires an intentional change to internal/wasmartifacts/manifest.go.

Embedded host failures expose a typed diagnostic through wasmhost.DiagnosticKindOf and errors.Is sentinels for invalid input, ABI or memory violations, guest status, invalid output, timeout, and cancellation. The diagnostic wrapper preserves the existing error text and wrapped causes so operator output and caller fallback behavior do not change.

Architecture Notes

  • External data enters through source packages under sources/.
  • Source runtime state is stored in Postgres when configured.
  • Append-log-backed sync and replay use NATS JetStream.
  • Graph projection and query operations use Neo4j/Aura.
  • Runtime finding behavior lives in Go rule packages under internal/findings/.
  • PolicyFindingRule YAML files under policies/ are the authoring DSL for generated policy rules. Run make finding-dsl-check, make policy-rule-generate, and make detection-catalog-generate after policy edits.

Before Opening A PR

Run the narrowest relevant package tests while developing, then run changed-path validation:

make changed-check

make changed-check maps touched paths to focused package tests and contract checks such as sourcegen-check, catalog-check, policy-rule checks, OpenAPI checks, protobuf checks, script tests, and structural checks. The pre-commit hook uses it by default to keep local iteration fast. Set CEREBRO_PRE_COMMIT_FULL_VERIFY=1 when you want the hook to run full CI-equivalent validation locally.

Before broad PRs or before handing off a change with high blast radius, run:

make check

For public config, docs, or packaging changes, also run:

make oss-audit

For generated docs changes, also run:

make docs-drift-check