diff --git a/docs/superpowers/plans/2026-11-05-usecase-metadata-standard.md b/docs/superpowers/plans/2026-11-05-usecase-metadata-standard.md new file mode 100644 index 0000000..6e9c9eb --- /dev/null +++ b/docs/superpowers/plans/2026-11-05-usecase-metadata-standard.md @@ -0,0 +1,331 @@ +# Open Security Usecase & Blue Team Benchmark Metadata Standard Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:subagent-driven-development` (recommended) or `superpowers:executing-plans` to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Implement the vendor-neutral Open Security Usecase & Blue Team Evaluation Metadata Standard (`schema_version: "2.0"`) in `logstory` (models, JSON Schema Draft 2020-12 exporter, 5-gate validator with anti-spoiler verification, v1/v2 doc generator shim, CLI subcommands `validate`, `inspect`, `export-schemas`, and `replay usecase --with-paired-baselines`) and migrate the two built-in `logstory` usecases plus all 47 usecases in `logstory_usecases` to v2 `.md` + `eval.yaml`. + +**Architecture:** Two-file physical split per usecase package (`/.md` for telemetry/replay/platform bindings with `` anti-spoiler boundary, and `/eval.yaml` for held-out multi-task Blue Team ground truth and Signal Detection Theory counterfactual pairs). Implemented in pure Python 3.10+ stdlib (`dataclasses`, `enum`, `typing`, `json`, `re`, `pathlib`) + `yaml` / `typer` so `logstory` requires zero additional third-party runtime dependencies. + +**Tech Stack:** Python 3.10+, `dataclasses`, `oyaml`/`PyYAML`, `typer`, `pytest`, `ruff`, `pyink` (2-space indentation, 88-character line length, Google-style docstrings). + +**Spec:** `docs/superpowers/specs/2026-11-05-usecase-metadata-standard-design.md` + +--- + +## Global Constraints + +1. **No emojis anywhere:** Never use emojis in code, comments, CLI output, documentation, tests, or commit messages. +2. **Formatting & Linting:** All Python files in `logstory` MUST use **2-space indentation**, `line-length = 88`, and Google-style docstrings (`pyink` + `ruff check`). +3. **Git Staging Hygiene:** Never run `git add .`, `git add -A`, or `git add --all`. Always check `git status` and stage explicit file paths (`git add `). +4. **Zero New Runtime Dependencies:** `.venv` in `logstory` has `typer`, `oyaml`/`yaml`, and `pytest` installed (`pydantic`, `jsonschema`, and `jinja2` are not installed in the base virtualenv). Implement `src/logstory/schema/models.py` and `src/logstory/schema/validator.py` using stdlib `dataclasses`, `enum`, `typing`, `json`, and `re` plus `yaml`, and provide a built-in Jinja2-free fallback renderer in `usecases/generate_usecase_docs.py` when `jinja2` is not installed. +5. **Product Naming:** Use **Google Security Operations** (or **SecOps**) in documentation and descriptions; reserve `"Chronicle"` only for API/IAM technical identifiers. + +--- + +## File Structure + +### `logstory` Worktree (`/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2`) +- **Create:** `src/logstory/schema/__init__.py` - Public package exports for schema models, parser, JSON Schema exporter, and 5-gate validator. +- **Create:** `src/logstory/schema/models.py` - Strongly typed Python dataclasses, enums, dict validation/serialization, `parse_usecase_package`, and Draft 2020-12 JSON Schema generation (`export_json_schemas`). +- **Create:** `src/logstory/schema/validator.py` - 5-gate validator (`validate_usecase_package`, `validate_usecase_corpus`) including anti-spoiler leakage detection (``) and counterfactual pair verification. +- **Create:** `schemas/usecase.schema.json` - Exported JSON Schema (Draft 2020-12) for `.md` YAML frontmatter. +- **Create:** `schemas/eval.schema.json` - Exported JSON Schema (Draft 2020-12) for `eval.yaml`. +- **Modify:** `usecases/generate_usecase_docs.py` - Add `normalize_frontmatter_for_template(frontmatter: dict) -> dict` v1/v2 compatibility shim and a stdlib fallback renderer when `jinja2` is not installed. +- **Modify:** `usecases/usecase_template.j2` - Render OCSF stream mappings, paired baselines, and multi-platform (`google_secops`, `databricks`, `sigma`, `splunk`) sections alongside legacy tables. +- **Modify:** `usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS.md` and create `usecases/NETWORK_ANALYSIS/eval.yaml` (plus symlink in `src/logstory/usecases/NETWORK_ANALYSIS/eval.yaml`). +- **Modify:** `usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP.md` and create `usecases/RULES_SEARCH_WORKSHOP/eval.yaml`. +- **Modify:** `src/logstory/logstory.py` - Add `usecases validate`, `usecases inspect`, `usecases export-schemas`, and `--with-paired-baselines` on `replay usecase`. +- **Modify:** `pyproject.toml` - Include `"usecases/**/*.yaml"`, `"usecases/**/*.j2"`, and `"schemas/*.json"` in `[tool.setuptools.package-data]`. +- **Create:** `tests/test_usecase_schema_models.py` - Unit tests for schema models, v1/v2 parsing, `` splitting, and JSON Schema export. +- **Create:** `tests/test_usecase_validator.py` - Unit tests for all 5 validation gates and anti-spoiler checks. +- **Create:** `tests/test_generate_usecase_docs.py` - Unit tests for v1/v2 doc generation and normalization shim. +- **Create:** `tests/test_usecase_cli_metadata.py` - CLI tests for `usecases validate`, `usecases inspect --blind`, `usecases export-schemas`, and `replay usecase --with-paired-baselines`. + +### `logstory_usecases` Worktree (`/usr/local/google/home/dandye/Projects/logstory_usecases__worktrees/schema__usecase-metadata-v2`) +- **Create:** `scripts/migrate_usecases_v2.py` - Deterministic migration script transforming all 43 attack scenarios and 4 `CYMBAL_BASELINE_*` composable baselines to v2 `.md` + `eval.yaml`. +- **Create:** `schemas/usecase.schema.json` and `schemas/eval.schema.json` - Synchronized JSON Schemas. +- **Modify/Create:** `/.md` and `/eval.yaml` across all 47 usecase directories. + +--- + +### Task 1: Core Schema Models, Parser, and JSON Schema Exporter + +**Files:** +- Create: `src/logstory/schema/__init__.py` +- Create: `src/logstory/schema/models.py` +- Create: `schemas/usecase.schema.json` +- Create: `schemas/eval.schema.json` +- Test: `tests/test_usecase_schema_models.py` + +- [ ] **Step 1: Write the failing unit tests in `tests/test_usecase_schema_models.py`** + +Create `tests/test_usecase_schema_models.py` testing: +1. Validating and serializing a complete v2 `SecurityUsecaseManifest` (including OKF v0.2 provenance fields `schema_version`, `id`, `title`, `version`, `created`, `updated`, `author`, `license`, `description`, `tags`, `environment`, `telemetry`, and `platforms` for `google_secops`, `databricks`, `sigma`, `splunk`). +2. Validating and serializing a complete v2 `SecurityUsecaseEval` (including `threat_context`, `tasks`, `ground_truth`, `compliance_gates`, `counterfactual_pair`, and `reference_queries`). +3. Rejecting unknown/extra keys and invalid enum values with descriptive `SchemaValidationError` exceptions. +4. Splitting `.md` on `` via `parse_usecase_package(usecase_dir, blind=True)` vs `blind=False`. +5. Generating deterministic Draft 2020-12 JSON Schemas (`usecase.schema.json` and `eval.schema.json`) via `export_json_schemas(output_dir)`. + +- [ ] **Step 2: Run pytest to verify the new test file fails** + +Run: +```bash +/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2/.venv/bin/pytest tests/test_usecase_schema_models.py -v +``` +Expected: `ModuleNotFoundError: No module named 'logstory.schema'` + +- [ ] **Step 3: Implement `src/logstory/schema/models.py` and `src/logstory/schema/__init__.py`, and export `schemas/*.schema.json`** + +Implement in `src/logstory/schema/models.py`: +- Custom `SchemaValidationError(ValueError)` with field path reporting. +- Enum classes (`ScenarioType`, `GroundingTier`, `ExpectedDisposition`, `ExpectedVerdict`, `CriticalityTier`, `PairRelationship`, `DifficultyTier`). +- Dataclasses with `.from_dict(data: dict)` (enforcing required fields, types, and rejecting unknown keys) and `.to_dict() -> dict`: + - Manifest models: `EnvironmentContext`, `TimestampPolicy`, `TelemetryStream`, `TelemetrySection`, `GoogleSecOpsRunFrequency`, `GoogleSecOpsDetection`, `GoogleSecOpsSavedSearch`, `GoogleSecOpsReferenceList`, `GoogleSecOpsPlatform`, `DatabricksPlatform`, `SigmaPlatform`, `SplunkPlatform`, `PlatformBindings`, `SecurityUsecaseManifest`. + - Eval models: `ThreatContext`, `AlertTriageTask`, `ThreatHuntTask`, `DetectionEngineeringTask`, `EvaluationTasks`, `KillChainStage`, `AffectedAsset`, `MaliciousIoc`, `ChangeWindow`, `BenignConfusers`, `EntitiesAndIocs`, `GroundTruth`, `ExpectedActions`, `ComplianceGates`, `CounterfactualPair`, `ReferenceQueries`, `SecurityUsecaseEval`. + - Package container: `ParsedUsecasePackage` (`usecase_dir`, `schema_version`, `manifest`, `raw_frontmatter`, `public_markdown_body`, `solution_markdown_body`, `eval_data`). +- Functions: + - `split_frontmatter_and_body(content: str) -> tuple[dict[str, Any], str]` + - `split_solution_section(markdown_body: str) -> tuple[str, str | None]` (splits on ``) + - `parse_usecase_package(usecase_dir: Path, blind: bool = False) -> ParsedUsecasePackage` + - `build_usecase_json_schema() -> dict[str, Any]` and `build_eval_json_schema() -> dict[str, Any]` (Draft 2020-12 JSON Schemas with `$schema: "https://json-schema.org/draft/2020-12/schema"`, `additionalProperties: false`, enum constraints, and property descriptions) + - `export_json_schemas(output_dir: Path) -> tuple[Path, Path]` + +- [ ] **Step 4: Run pytest, ruff, and pyink on Task 1 files** + +Run: +```bash +WORKTREE_DIR="/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2" +"$WORKTREE_DIR/.venv/bin/pytest" "$WORKTREE_DIR/tests/test_usecase_schema_models.py" -v +"$WORKTREE_DIR/.venv/bin/ruff" check "$WORKTREE_DIR/src/logstory/schema" "$WORKTREE_DIR/tests/test_usecase_schema_models.py" +"$WORKTREE_DIR/.venv/bin/pyink" --check "$WORKTREE_DIR/src/logstory/schema" "$WORKTREE_DIR/tests/test_usecase_schema_models.py" +``` +Expected: All tests pass, 0 ruff warnings, pyink formatted cleanly. + +- [ ] **Step 5: Stage and commit Task 1 files explicitly** + +Run: +```bash +git status +git add src/logstory/schema/__init__.py src/logstory/schema/models.py schemas/usecase.schema.json schemas/eval.schema.json tests/test_usecase_schema_models.py +git commit -m "Add v2 security usecase and Blue Team eval schema models and JSON schemas" +``` + +--- + +### Task 2: 5-Gate Validator and Anti-Spoiler Checker + +**Files:** +- Create: `src/logstory/schema/validator.py` +- Modify: `src/logstory/schema/__init__.py` +- Test: `tests/test_usecase_validator.py` + +- [ ] **Step 1: Write failing unit tests in `tests/test_usecase_validator.py`** + +Create `tests/test_usecase_validator.py` covering: +1. **Gate 1 (Schema & Provenance):** Valid v2 package passes; missing required OKF v0.2 fields (`license`, `version`, `author`) or invalid `eval.yaml` enum fails with `gate="G1_SCHEMA"`. +2. **Gate 2 (Stream & File Integrity):** Missing on-disk log file referenced by `telemetry.events[].path`, missing detection rule referenced by `platforms.google_secops.detections[].path`, or `kill_chain[].stream_id` referencing an unknown `stream_id` fails with `gate="G2_INTEGRITY"`. +3. **Gate 3 (Anti-Spoiler Leakage):** + - Ground-truth key (`ground_truth`, `kill_chain`, `malicious_iocs`, `compliance_gates`) placed in `.md` frontmatter fails with `gate="G3_ANTI_SPOILER"`. + - Any non-trivial IOC value from `eval.yaml -> ground_truth.entities_and_iocs.malicious_iocs[].value`, compromised account from `compromised_accounts`, or CISA advisory ID from `threat_context.cisa_advisories` appearing in `.md` frontmatter (`title`, `description`, `tags`, `notes`) or in the Markdown body above `` fails with `gate="G3_ANTI_SPOILER"`. + - The same IOC/advisory appearing *below* `` passes Gate 3. +4. **Gate 4 (Counterfactual Twin Consistency):** + - When `eval.yaml` specifies `counterfactual_pair.paired_usecase_id` with `pair_relationship == "malicious_variant_of_benign_baseline"`, `.md` must include that baseline in `environment.paired_baselines` (unless the paired usecase is an attack variant and this usecase is a `benign_baseline`). + - When `corpus_root` is provided, verify that the paired directory exists on disk. +5. **Gate 5 (v1 Legacy Compatibility Mode):** + - A legacy v1 `.md` (without `schema_version: "2.0"`) passes with a warning when `strict_v2=False`, and fails with `gate="G5_LEGACY_V1"` when `strict_v2=True`. + +- [ ] **Step 2: Run pytest to verify `tests/test_usecase_validator.py` fails** + +Run: +```bash +/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2/.venv/bin/pytest tests/test_usecase_validator.py -v +``` +Expected: `ImportError` / `ModuleNotFoundError`. + +- [ ] **Step 3: Implement `src/logstory/schema/validator.py` and export from `src/logstory/schema/__init__.py`** + +Implement in `src/logstory/schema/validator.py`: +- `ValidationFinding` (`gate: str`, `severity: Literal["ERROR", "WARNING"]`, `message: str`, `path: str | None = None`) +- `UsecaseValidationResult` (`usecase_id: str`, `usecase_dir: Path`, `schema_version: str`, `findings: list[ValidationFinding]`, property `is_valid -> bool`, property `errors -> list[ValidationFinding]`, property `warnings -> list[ValidationFinding]`, `.to_dict() -> dict`) +- `validate_usecase_package(usecase_dir: Path, strict_v2: bool = False, corpus_root: Path | None = None) -> UsecaseValidationResult` +- `validate_usecase_corpus(corpus_dir: Path, strict_v2: bool = False) -> list[UsecaseValidationResult]` + +- [ ] **Step 4: Run pytest, ruff, and pyink on Task 2 files** + +Run: +```bash +WORKTREE_DIR="/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2" +"$WORKTREE_DIR/.venv/bin/pytest" "$WORKTREE_DIR/tests/test_usecase_validator.py" -v +"$WORKTREE_DIR/.venv/bin/ruff" check "$WORKTREE_DIR/src/logstory/schema" "$WORKTREE_DIR/tests/test_usecase_validator.py" +"$WORKTREE_DIR/.venv/bin/pyink" --check "$WORKTREE_DIR/src/logstory/schema" "$WORKTREE_DIR/tests/test_usecase_validator.py" +``` +Expected: All tests pass, 0 lint or formatting errors. + +- [ ] **Step 5: Stage and commit Task 2 files explicitly** + +Run: +```bash +git status +git add src/logstory/schema/__init__.py src/logstory/schema/validator.py tests/test_usecase_validator.py +git commit -m "Add 5-gate usecase package validator and anti-spoiler checker" +``` + +--- + +### Task 3: Documentation Generator v1/v2 Shim and Built-in `logstory` Usecase Migration + +**Files:** +- Modify: `usecases/generate_usecase_docs.py` +- Modify: `usecases/usecase_template.j2` +- Modify: `usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS.md` +- Create: `usecases/NETWORK_ANALYSIS/eval.yaml` +- Create: `src/logstory/usecases/NETWORK_ANALYSIS/eval.yaml` (symlink to `../../../../usecases/NETWORK_ANALYSIS/eval.yaml`) +- Modify: `usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP.md` +- Create: `usecases/RULES_SEARCH_WORKSHOP/eval.yaml` +- Test: `tests/test_generate_usecase_docs.py` + +- [ ] **Step 1: Write failing unit tests in `tests/test_generate_usecase_docs.py`** + +Create `tests/test_generate_usecase_docs.py` testing: +1. `normalize_frontmatter_for_template(frontmatter)` converts a v2 `SecurityUsecaseManifest` dict into backwards-compatible template variables (`events`, `entities`, `rules`, `saved_searches`, `reference_lists`, `run_frequency`, `data_rbac`) while preserving `telemetry`, `platforms`, `environment`, `schema_version`, `version`, `author`, and `license`. +2. `generate_usecase_doc` works for both v1 and v2 frontmatter (using Jinja2 if installed, or the built-in stdlib template renderer fallback when Jinja2 is absent) and renders OCSF classes, paired baselines, and platform bindings (`Google SecOps`, `Databricks`, `Sigma`, `Splunk`) without spoilers. +3. Both built-in usecases (`usecases/NETWORK_ANALYSIS` and `usecases/RULES_SEARCH_WORKSHOP`) validate cleanly under `validate_usecase_package(..., strict_v2=True)`. + +- [ ] **Step 2: Run pytest to verify `tests/test_generate_usecase_docs.py` fails** + +Run: +```bash +/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2/.venv/bin/pytest tests/test_generate_usecase_docs.py -v +``` +Expected: Fails because `normalize_frontmatter_for_template` does not exist yet and built-in usecases are still v1. + +- [ ] **Step 3: Update `usecases/generate_usecase_docs.py`, `usecases/usecase_template.j2`, and migrate `NETWORK_ANALYSIS` and `RULES_SEARCH_WORKSHOP` to v2** + +In `usecases/generate_usecase_docs.py`: +- Add `normalize_frontmatter_for_template(frontmatter: dict) -> dict` mapping v2 `telemetry.events`, `telemetry.entities`, and `platforms.google_secops` into legacy keys when `schema_version == "2.0"`. +- Make `from jinja2 import Environment, FileSystemLoader` conditional (`try ... except ImportError`) and implement `_render_without_jinja2(normalized: dict) -> str` producing identical Markdown tables and sections when `jinja2` is not installed in `.venv`. +- Migrate `usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS.md` and `usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP.md` to `schema_version: "2.0"`, placing any detailed exercise notes below ``, and create `usecases/NETWORK_ANALYSIS/eval.yaml` (plus symlink in `src/logstory/usecases/NETWORK_ANALYSIS/eval.yaml`) and `usecases/RULES_SEARCH_WORKSHOP/eval.yaml`. + +- [ ] **Step 4: Run pytest, ruff, and pyink on Task 3 files** + +Run: +```bash +WORKTREE_DIR="/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2" +"$WORKTREE_DIR/.venv/bin/pytest" "$WORKTREE_DIR/tests/test_generate_usecase_docs.py" "$WORKTREE_DIR/tests/test_cli.py" -v +"$WORKTREE_DIR/.venv/bin/ruff" check "$WORKTREE_DIR/usecases/generate_usecase_docs.py" "$WORKTREE_DIR/tests/test_generate_usecase_docs.py" +"$WORKTREE_DIR/.venv/bin/pyink" --check "$WORKTREE_DIR/usecases/generate_usecase_docs.py" "$WORKTREE_DIR/tests/test_generate_usecase_docs.py" +``` +Expected: All tests pass, 0 lint/format errors. + +- [ ] **Step 5: Stage and commit Task 3 files explicitly** + +Run: +```bash +git status +git add usecases/generate_usecase_docs.py usecases/usecase_template.j2 usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS.md usecases/NETWORK_ANALYSIS/eval.yaml src/logstory/usecases/NETWORK_ANALYSIS/eval.yaml usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP.md usecases/RULES_SEARCH_WORKSHOP/eval.yaml tests/test_generate_usecase_docs.py +git commit -m "Upgrade generate_usecase_docs for v2 manifests and migrate built-in usecases" +``` + +--- + +### Task 4: CLI Subcommands (`validate`, `inspect`, `export-schemas`) and `--with-paired-baselines` + +**Files:** +- Modify: `src/logstory/logstory.py` +- Modify: `pyproject.toml` +- Test: `tests/test_usecase_cli_metadata.py` + +- [ ] **Step 1: Write failing CLI tests in `tests/test_usecase_cli_metadata.py`** + +Create `tests/test_usecase_cli_metadata.py` testing via `CliRunner`: +1. `logstory usecases validate ` on a single usecase directory and on a corpus directory, with `--strict-v2` and `--format json` / `--format text`. Verify exit code `0` on valid packages and exit code `1` when any gate fails. +2. `logstory usecases inspect --blind --format json` returns `manifest` and `public_markdown_body` (with zero content from below `` and `eval_data: null`), whereas `--include-eval` includes `eval_data` and `solution_markdown_body`. +3. `logstory usecases export-schemas --output-dir ` writes `usecase.schema.json` and `eval.schema.json`. +4. `logstory replay usecase --with-paired-baselines --local-file-output --no-get` reads `environment.paired_baselines` from `.md` and replays each paired baseline usecase prior to (or alongside) the target usecase, deduplicating baselines if multiple are listed. + +- [ ] **Step 2: Run pytest to verify `tests/test_usecase_cli_metadata.py` fails** + +Run: +```bash +/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2/.venv/bin/pytest tests/test_usecase_cli_metadata.py -v +``` +Expected: Fails because the new CLI subcommands and `--with-paired-baselines` option do not exist yet. + +- [ ] **Step 3: Implement CLI subcommands in `src/logstory/logstory.py` and update `pyproject.toml`** + +In `src/logstory/logstory.py`: +- Add `@usecases_app.command("validate")` accepting `path: str | None = typer.Argument(None)`, `--strict-v2: bool = typer.Option(False, "--strict-v2")`, `--format: str = typer.Option("text", "--format")` (`text` or `json`). Resolves either a filesystem path or an installed usecase name (or all installed usecases if omitted). +- Add `@usecases_app.command("inspect")` accepting `usecase: str = typer.Argument(...)`, `--blind/--include-eval: bool = typer.Option(True, "--blind/--include-eval")`, `--format: str = typer.Option("json", "--format")` (`json` or `text`). +- Add `@usecases_app.command("export-schemas")` accepting `--output-dir: str = typer.Option("schemas", "--output-dir")`. +- Add `with_paired_baselines: bool = typer.Option(False, "--with-paired-baselines", help="Also replay composable paired baselines declared in environment.paired_baselines")` to `@replay_app.command("usecase")`. Add helper `_get_paired_baselines(usecase: str) -> list[str]` that reads `/.md` frontmatter and returns `environment.paired_baselines`. +- In `pyproject.toml`, update `[tool.setuptools.package-data]` to include `"usecases/**/*.yaml"`, `"usecases/**/*.j2"`, and `"schemas/*.json"`. + +- [ ] **Step 4: Run full pytest suite, ruff, and pyink in `logstory`** + +Run: +```bash +WORKTREE_DIR="/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2" +"$WORKTREE_DIR/.venv/bin/pytest" "$WORKTREE_DIR/tests" -v +"$WORKTREE_DIR/.venv/bin/ruff" check "$WORKTREE_DIR/src" "$WORKTREE_DIR/tests" "$WORKTREE_DIR/usecases/generate_usecase_docs.py" +"$WORKTREE_DIR/.venv/bin/pyink" --check "$WORKTREE_DIR/src" "$WORKTREE_DIR/tests" "$WORKTREE_DIR/usecases/generate_usecase_docs.py" +``` +Expected: All unit and CLI tests pass with coverage >= 80%, 0 ruff or pyink errors. + +- [ ] **Step 5: Stage and commit Task 4 files explicitly** + +Run: +```bash +git status +git add pyproject.toml src/logstory/logstory.py tests/test_usecase_cli_metadata.py +git commit -m "Add usecases validate, inspect, export-schemas CLI and --with-paired-baselines" +``` + +--- + +### Task 5: Migrate All 47 Scenarios in `logstory_usecases` to v2 `.md` + `eval.yaml` + +**Files (in `/usr/local/google/home/dandye/Projects/logstory_usecases__worktrees/schema__usecase-metadata-v2`):** +- Create: `scripts/migrate_usecases_v2.py` +- Create: `schemas/usecase.schema.json` +- Create: `schemas/eval.schema.json` +- Modify/Create: `/.md` and `/eval.yaml` across all 47 usecase directories (`43` attack scenarios + `4` `CYMBAL_BASELINE_*` composable baselines) +- Test (in `logstory`): `tests/test_logstory_usecases_corpus_v2.py` + +- [ ] **Step 1: Create isolated Git worktree `/usr/local/google/home/dandye/Projects/logstory_usecases__worktrees/schema__usecase-metadata-v2`** + +Run: +```bash +USECASES_MAIN="/usr/local/google/home/dandye/Projects/logstory_usecases__worktrees/main" +USECASES_WT="/usr/local/google/home/dandye/Projects/logstory_usecases__worktrees/schema__usecase-metadata-v2" +git -C "$USECASES_MAIN" worktree add -b schema__usecase-metadata-v2 "$USECASES_WT" main +``` + +- [ ] **Step 2: Write `scripts/migrate_usecases_v2.py` in `logstory_usecases__worktrees/schema__usecase-metadata-v2`** + +Write `scripts/migrate_usecases_v2.py` that: +1. Parses existing `/.md` frontmatter and Markdown body, `RULES/*.yl2` metadata (`meta:` section including MITRE `ta`, `t1`, `author`, `severity`, `priority`, and events/outcomes), `THREAT_ACTORS.md`, `CYMBAL_BASELINE.md`, `RELEASE_PLAN.md`, and the 47-scenario metadata table (`expected_skills`, `forbid_lolbin_hashes`, `requires_168h_horizon`, `change_window`, `compromised_accounts`, `affected_assets`, `malicious_iocs`, `kill_chain`). +2. Maps every Google SecOps log type (`POWERSHELL`, `WINDOWS_SYSMON`, `WINDOWS_DEFENDER_AV`, `WINEVTLOG`, `WINDOWS_AD`, `GCP_CLOUDAUDIT`, `WORKSPACE_ACTIVITY`, `WORKSPACE_USERS`, `SURICATA_EVE`, `BRO_JSON`, `LIN_AUDITD`, `OSQUERY`, `PAN_FIREWALL`, `OKTA`, `OFFICE_365`, `AWS_CLOUDTRAIL`, `AZURE_AD`, `AZURE_AD_CONTEXT`, `ASSET_CONTEXT`, `USER_CONTEXT`, `CROWDSTRIKE_FALCON`, `FIREEYE_HX`, etc.) to its canonical OCSF v1.3+ category/class (`ocsf_category_uid`, `ocsf_class_uid`, `ocsf_class_name`) and Databricks Unity Catalog OCSF table (`secops_ocsf.`). +3. Rewrites `/.md` with `schema_version: "2.0"` frontmatter, neutral `description` (scrubbing spoiler IOCs/actor names/CISA advisory IDs from the public description and placing the full analyst walkthrough and IOC tables below ``), and writes `/eval.yaml` for each of the 47 directories. +4. Copies `schemas/usecase.schema.json` and `schemas/eval.schema.json` into `logstory_usecases/schemas/`. + +- [ ] **Step 3: Run `scripts/migrate_usecases_v2.py` and validate all 47 usecases with `logstory usecases validate --strict-v2`** + +Run: +```bash +WORKTREE_DIR="/usr/local/google/home/dandye/Projects/logstory__worktrees/schema__usecase-metadata-v2" +USECASES_WT="/usr/local/google/home/dandye/Projects/logstory_usecases__worktrees/schema__usecase-metadata-v2" +"$WORKTREE_DIR/.venv/bin/python" "$USECASES_WT/scripts/migrate_usecases_v2.py" "$USECASES_WT" +"$WORKTREE_DIR/.venv/bin/logstory" usecases validate "$USECASES_WT" --strict-v2 +``` +Expected: `47/47 usecases passed all 5 validation gates (0 errors, 0 warnings)`. + +- [ ] **Step 4: Add corpus integration test `tests/test_logstory_usecases_corpus_v2.py` in `logstory` and run full test suites** + +Create `tests/test_logstory_usecases_corpus_v2.py` in `logstory` (skipping cleanly if `logstory_usecases` directory is not mounted, and validating all 47 scenarios when present) and run `pytest`, `ruff`, and `pyink`. + +- [ ] **Step 5: Stage and commit changes in both worktrees explicitly** + +In `logstory_usecases__worktrees/schema__usecase-metadata-v2`: +- Stage `schemas/usecase.schema.json`, `schemas/eval.schema.json`, `scripts/migrate_usecases_v2.py`, and all 47 `/.md` and `/eval.yaml` files by explicit path and commit. +In `logstory__worktrees/schema__usecase-metadata-v2`: +- Stage `tests/test_logstory_usecases_corpus_v2.py` and commit. diff --git a/docs/superpowers/specs/2026-11-05-usecase-metadata-standard-design.md b/docs/superpowers/specs/2026-11-05-usecase-metadata-standard-design.md new file mode 100644 index 0000000..761f510 --- /dev/null +++ b/docs/superpowers/specs/2026-11-05-usecase-metadata-standard-design.md @@ -0,0 +1,246 @@ +--- +type: DesignSpec +title: "Open Security Scenario & Logstory Usecase Metadata Standard (v2.0)" +description: > + Vendor-neutral usecase metadata and held-out Blue Team evaluation specification + for Logstory and cross-vendor InfoSec community benchmarks (Google SecOps, + Databricks Lakehouse / OCSF, Sigma, and Splunk). +tags: + - logstory + - ocsf + - blue-team-evals + - secops + - databricks + - benchmark-standard +created: 2026-11-05 +updated: 2026-11-05 +status: draft +generated: + by: jetski/gemini-3.1-pro + at: 2026-11-05T00:53:00Z +verified: + - by: human:dandye + at: 2026-11-05T00:53:00Z +sources: + - id: logstory-repo + resource: https://github.com/chronicle/logstory + - id: logstory-usecases + resource: https://github.com/dandye/logstory_usecases +--- + +# Open Security Scenario & Logstory Usecase Metadata Standard (v2.0) + +## 1. Purpose and Motivation + +Security operations teams and AI agent researchers need a vendor-neutral standard to package synthetic or anonymized security scenarios ("log stories") so that: + +1. **Telemetry Replay is Portable Across Platforms**: A single scenario package can be replayed into Google Security Operations (SecOps UDM), loaded into Databricks Lakehouse (Delta tables / OCSF), or queried in Splunk / Sigma pipelines. +2. **Blue Team Agent Evaluations are Blind and Reproducible**: Evaluating autonomous SOC agents across Alert Triage, Unalerted Threat Hunting ("Hunt Cymbal" mode), and Detection Engineering requires a strict anti-spoiler boundary between the **Scenario Stimulus** (`.md` + `EVENTS/` + `ENTITIES/`) and the **Held-Out Answer Key** (`eval.yaml` + `RULES/`). +3. **Signal Detection Theory (SDT) and Operational Safety are First-Class**: Benchmarks must test whether agents distinguish real attacks from confusable benign baselines (`CYMBAL_BASELINE_*` counterfactual twins), widen search horizons from 72h to 168h when required, respect maintenance windows, avoid blocking signed OS LOLBins, and avoid leaking RFC-1918/RFC-5737 internal IPs to external CTI tools. + +--- + +## 2. Package Layout and Anti-Spoiler Boundary + +Each usecase resides in a directory `/` with two distinct metadata files separating stimulus from evaluation ground truth: + +```text +/ + .md # 1. Telemetry & Replay Package Manifest (safe for blind agent setup) + eval.yaml # 2. Held-out Blue Team Benchmark & Answer Key (NEVER mounted in agent sandbox) + EVENTS/.log # 3. Native-format event telemetry files + ENTITIES/.log # 4. Entity context roster files (optional) + PARSER_EXTENSIONS/*.proto # 5. Platform parser extensions (optional) + RULES/ # 6. Reference detection rules (.yl2, .sigma.yml, .sql) + _generate_events.py # 7. Deterministic synthetic timeline generator (optional) +``` + +### 2.1 Boundary Invariants + +1. **Zero Spoilers in `.md` Frontmatter**: The YAML frontmatter of `.md` declares only OKF v0.2 provenance, time windows, paired benign baselines, OCSF telemetry streams, and platform ingestion bindings. It must never contain attacker IOCs, compromised account names, CISA advisory IDs, MITRE ATT&CK kill-chain sequences, or expected verdicts. +2. **Markdown Body Delimiter (``)**: Any human-facing attack narrative, indicator table, or rule walkthrough in `.md` must appear below an explicit `` HTML comment delimiter. +3. **Blind Inspection Mode**: Running `logstory usecases inspect --blind` returns only the `.md` frontmatter and Markdown content above ``, omitting `eval.yaml` and `RULES/`. +4. **Hermetic Sandbox Mounting**: Evaluation runners mount only `EVENTS/`, `ENTITIES/`, and blind metadata inside the agent runtime container; `eval.yaml` and `RULES/` remain exclusively in the host verifier/judge process. + +--- + +## 3. `.md` Manifest Specification (`usecase.schema.json`) + +`.md` begins with YAML frontmatter conforming to both Open Knowledge Format (OKF v0.2) and the `SecurityUsecaseManifest` v2.0 schema. + +### 3.1 Schema Structure + +| Field | Type | Required | Description | +|---|---|---|---| +| `schema_version` | `"2.0"` | Yes | Version of the usecase metadata specification. | +| `type` | `"SecurityUsecase"` | Yes | OKF v0.2 concept type discriminator. | +| `id` | `string` (`^[A-Z0-9_]+$`) | Yes | Canonical uppercase identifier matching directory name. | +| `title` | `string` | Yes | Human-readable scenario or telemetry package title. | +| `kind` | `"attack" \| "baseline" \| "hybrid"` | Yes | Whether the package contains attack events, benign baseline noise, or both. | +| `category` | `string` | Yes | High-level domain (`identity_cloud`, `endpoint_ransomware`, `network_c2`, `insider_risk`, `baseline`, `workshop`). | +| `description` | `string` | Yes | Blind-safe summary of the telemetry streams and operational window (no spoilers). | +| `tags` | `list[str]` | Yes | Non-spoiler discovery tags (e.g., `endpoint`, `windows`, `ocsf`, `google-secops`, `databricks`). | +| `created` | `date` (`YYYY-MM-DD`) | Yes | Initial authoring date. | +| `updated` | `date` (`YYYY-MM-DD`) | Yes | Last modification date. | +| `status` | `"draft" \| "stable" \| "deprecated"` | Yes | OKF v0.2 lifecycle status. | +| `sources` | `list[{id, resource}]` | No | OKF v0.2 source references (non-spoiler specs or generator pointers). | +| `generated` | `{by: str, at: datetime}` | No | OKF v0.2 generation provenance. | +| `verified` | `list[{by: str, at: datetime}]` | No | OKF v0.2 verification provenance. | +| `environment` | `EnvironmentSpec` | Yes | Organization canon, base date, UTC window, and `paired_baselines`. | +| `telemetry` | `TelemetrySpec` | Yes | Vendor-neutral `events` and `entities` stream definitions with OCSF mappings. | +| `platforms` | `PlatformBindings` | Yes | Platform-specific bindings (`google_secops`, `databricks`, `sigma`, `splunk`). | + +### 3.2 `environment` Block + +```yaml +environment: + organization: cymbal-investments.org + base_date: "2025-08-04" + time_window_utc: + start: "2025-08-04T13:00:00Z" + end: "2025-08-04T17:00:00Z" + paired_baselines: + - CYMBAL_BASELINE_ENDPOINT +``` + +* `paired_baselines`: Lists one or more `CYMBAL_BASELINE_*` usecase IDs that share the same entity roster and log types and should be co-replayed into the same time window for realistic background noise and false-positive testing. + +### 3.3 `telemetry` Block (OCSF v1.3+ Aligned) + +Each file in `EVENTS/` and `ENTITIES/` is declared once in `telemetry.events` or `telemetry.entities` with a stable `stream_id`: + +```yaml +telemetry: + events: + - stream_id: windows_security + path: EVENTS/WINEVTLOG.log + vendor_name: Microsoft + product_name: Microsoft-Windows-Security-Auditing + ocsf_category_uid: 3 + ocsf_class_uid: 3002 + ocsf_class_name: Authentication + notes: "Windows Security Event Log stream" + entities: + - stream_id: windows_ad + path: ENTITIES/WINDOWS_AD.log + vendor_name: Microsoft + product_name: Microsoft AD + ocsf_category_uid: 5 + ocsf_class_uid: 5003 + ocsf_class_name: User Inventory +``` + +### 3.4 `platforms` Block + +Platform-specific ingestion, table names, RBAC notes, and rule files live under `platforms.`: + +* **`platforms.google_secops`**: + * `run_frequency`: `{ events: str, entities: Optional[str] }` + * `data_rbac`: `str` + * `events`: `list[{ stream_id: str, log_type: str, notes: Optional[str] }]` + * `entities`: `list[{ stream_id: str, log_type: str, notes: Optional[str] }]` + * `rules`: `list[{ name: str, path: str, live: bool, alerting: bool, notes: Optional[str] }]` + * `saved_searches`: `list[{ name: str, creator: Optional[str], notes: Optional[str] }]` + * `reference_lists`: `list[{ name: str, type: Optional[str], notes: Optional[str] }]` +* **`platforms.databricks`**: + * `catalog`: `str` (default `"security_lakehouse"`) + * `schema_name`: `str` (default `"ocsf_silver"`) + * `tables`: `list[{ stream_id: str, bronze_table: str, ocsf_table: str, format: str }]` + * `queries`: `list[{ name: str, path: str, notes: Optional[str] }]` +* **`platforms.sigma`**: + * `rules`: `list[{ name: str, path: str, level: Optional[str] }]` +* **`platforms.splunk`**: + * `sourcetypes`: `list[{ stream_id: str, sourcetype: str, index: Optional[str] }]` + +--- + +## 4. `eval.yaml` Benchmark Specification (`eval.schema.json`) + +`eval.yaml` resides alongside `.md` and holds the platform-agnostic ground truth, compliance gates, counterfactual pairing, and task definitions for Blue Team agent evaluations. + +### 4.1 Schema Structure + +| Field | Type | Required | Description | +|---|---|---|---| +| `schema_version` | `"2.0"` | Yes | Evaluation schema version. | +| `type` | `"SecurityUsecaseEval"` | Yes | OKF v0.2 / schema type discriminator. | +| `usecase_id` | `string` | Yes | Must match `id` in `.md`. | +| `difficulty_tier` | `int` (`1..5`) | Yes | Progression difficulty (`1` Fundamentals .. `5` LOTL/Deception/Multi-Domain). | +| `release_day` | `Optional[int]` (`1..43`) | No | Canonical day index in the 43-day Hunt Cymbal curriculum. | +| `grounding` | `GroundingSpec` | Yes | `tier` (`stix_grounded`, `tradecraft_modeled`, `behavioral_synthetic`, `benign_baseline`) and `threat_actor` profile (`name`, `aliases`, `cisa_advisories`, `cves`, `mitre_group_id`). | +| `verdict` | `VerdictSpec` | Yes | `expected_disposition` (`TRUE_POSITIVE`, `BENIGN_POSITIVE`, `FALSE_POSITIVE`, `SUSPICIOUS_TRUE_POSITIVE`, `INCONCLUSIVE`), `expected_verdict` (`MALICIOUS`, `BENIGN`, `SUSPICIOUS`, `INCONCLUSIVE`), and `severity` (`LOW`, `MEDIUM`, `HIGH`, `CRITICAL`). | +| `kill_chain` | `list[KillChainStage]` | Yes | Chronological attack stages (`stage`, `timestamp_utc`, `tactic`, `technique_id`, `stream_id`, `summary`). Empty list `[]` for pure benign baselines. | +| `entities_and_iocs` | `EntitiesAndIocsSpec` | Yes | Ground-truth `compromised_accounts`, `affected_assets` (with `criticality_tier`, `patient_zero`, `blast_radius_fp_usd`, `blast_radius_fn_usd`), `malicious_iocs`, and `benign_confusers` (`signed_lolbins`, `internal_ips`, `change_window`). | +| `compliance_gates` | `ComplianceGatesSpec` | Yes | `requires_pre_containment_verification`, `requires_168h_horizon_widening`, `forbid_external_cti_on_internal_ips`, `forbid_lolbin_hash_blocking`, and `expected_actions` (`primary`, `secondary`, `prohibited`). | +| `counterfactual_pair` | `Optional[CounterfactualPairSpec]` | No | Links an attack scenario to its confusable benign twin (`pair_id`, `twin_usecase_id`, `shared_entities`, `discriminating_invariant`). | +| `tasks` | `EvalTasksSpec` | Yes | Multi-task benchmark definitions (`alert_triage`, `threat_hunt`, `detection_engineering`). | + +### 4.2 Multi-Task Evaluation Contracts (`tasks`) + +1. **`tasks.alert_triage`**: + * `case_id`: Canonical SOAR/SIEM case identifier (e.g., `"CASE-AKIRA-01"`). + * `initial_alerts`: Array of seed alerts (`alert_id`, `rule_name`, `severity`, `risk_score`). + * `prompt`: Blind triage instruction presented to the agent under test. + * `routing`: `{ expected_capability: str, skill_bindings: dict[str, str] }` mapping abstract defensive capability to harness-specific skill slugs (`google_secops`, `goose`, `databricks`). + * `assertions`: `{ must_cite_entities: list[str], must_cite_mitre: list[str], must_not_contain: list[str] }`. +2. **`tasks.threat_hunt`**: + * `prompt_unalerted`: Blind threat-hunt prompt ("Hunt Cymbal" mode) naming only the available telemetry streams and UTC time window. + * `prompt_intel_driven`: Optional advisory/collection-driven hunt prompt. + * `expected_patient_zero`: Hostname or account of initial compromise (`Optional[str]`, `null` for benign baselines). +3. **`tasks.detection_engineering`**: + * `prompt`: Rule authoring prompt. + * `target_techniques`: List of MITRE ATT&CK technique IDs to cover. + * `expected_rule_count_min`: Minimum number of detection rules. + * `max_false_positives_on_paired_baseline`: Maximum allowed false-positive detections when evaluated against `environment.paired_baselines` (default `0`). + +--- + +## 5. Components, Data Flow, and Error Handling + +```mermaid +flowchart LR + MD[".md\n(v1 or v2 Frontmatter)"] --> Parser["logstory.schema.parser\nparse_usecase_package()"] + EVAL["eval.yaml\n(Held-Out Answer Key)"] --> Parser + Parser --> Validator["logstory.schema.validator\nvalidate_usecase_package()"] + Validator -->|"5-Gate Check"| CLI["logstory usecases\nvalidate / inspect / export-schemas"] + Validator -->|"v1/v2 Shim"| DocGen["generate_usecase_docs.py\n+ usecase_template.j2"] + Parser -->|"environment.paired_baselines"| Replay["logstory replay usecase\n--with-paired-baselines"] +``` + +### 5.1 Module Responsibilities + +1. **`src/logstory/schema/models.py`**: + * Defines strict Pydantic v2 models (`SecurityUsecaseManifest`, `SecurityUsecaseEval`, and nested types) with `extra="allow"` on `platforms` so third-party vendors can add experimental platform blocks while keeping core fields strictly typed. + * Provides `export_json_schemas(output_dir: Path)` to write `schemas/usecase.schema.json` and `schemas/eval.schema.json` (JSON Schema Draft 2020-12). +2. **`src/logstory/schema/validator.py`**: + * Executes 5 validation gates returning a structured `UsecaseValidationResult(usecase_id, is_valid, schema_version, errors, warnings)`: + * **Gate 1 (Schema & OKF v0.2)**: Validates Pydantic models and OKF v0.2 top-level provenance keys. + * **Gate 2 (Stream & Path Integrity)**: Verifies all `stream_id` references across `platforms.*` and `eval.yaml` `kill_chain` exist in `telemetry.{events,entities}`, and all referenced files (`EVENTS/*.log`, `ENTITIES/*.log`, `RULES/*`) exist on disk. + * **Gate 3 (Anti-Spoiler Leakage)**: Verifies that none of `eval.yaml`'s `malicious_iocs` values, `compromised_accounts` user IDs, or `threat_actor.cisa_advisories` appear in `.md` frontmatter or above `` (for attack usecases). + * **Gate 4 (Counterfactual Twin Consistency)**: Verifies that `counterfactual_pair.twin_usecase_id` in `eval.yaml` is declared in `.md`'s `environment.paired_baselines`. + * **Gate 5 (v1 Legacy Compatibility)**: Accepts legacy v1 frontmatter when `strict_v2=False` (emitting a migration warning) and fails closed when `strict_v2=True`. +3. **`usecases/generate_usecase_docs.py` & `usecases/usecase_template.j2`**: + * Updated with `normalize_frontmatter_for_template(frontmatter: dict) -> dict` so v2 manifests populate the legacy `events`, `entities`, `rules`, `run_frequency`, and `data_rbac` template keys from `telemetry` + `platforms.google_secops`, while also rendering OCSF and multi-platform sections. +4. **CLI Subcommands in `src/logstory/logstory.py`**: + * `logstory usecases validate [USECASE_OR_PATH] [--strict-v2] [--json]` + * `logstory usecases inspect [--blind] [--format json|yaml|md]` + * `logstory usecases export-schemas [--output-dir schemas/]` + * `logstory replay usecase [--with-paired-baselines]`: When `--with-paired-baselines` is enabled, resolves `environment.paired_baselines` from `.md` and replays each baseline usecase into the same `--timestamp-delta` window before replaying ``. +5. **Corpus Migration Script (`scripts/migrate_usecases_v2.py`)**: + * Idempotent CLI tool that migrates all 47 usecases in `logstory_usecases` (43 attack scenarios + 4 `CYMBAL_BASELINE_*` scenarios) to v2 `.md` + `eval.yaml`, inserting `` before the spoiler narrative in `.md`. + +--- + +## 6. Testing and Verification Strategy + +1. **Unit & Schema Tests (`tests/test_usecase_schema.py`)**: + * Validates v2 `SecurityUsecaseManifest` and `SecurityUsecaseEval` models on valid fixtures. + * Verifies Gate 2 catches dangling `stream_id` and missing `EVENTS/*.log` files. + * Verifies Gate 3 catches an IOC, compromised account, or CISA advisory leaked into `.md` frontmatter or above ``. + * Verifies Gate 4 catches a mismatch between `counterfactual_pair.twin_usecase_id` and `environment.paired_baselines`. + * Verifies v1 backwards compatibility in both `validate_usecase_package(..., strict_v2=False)` and `generate_usecase_docs.py`. + * Verifies `export_json_schemas()` produces deterministic, valid Draft 2020-12 JSON Schemas. +2. **CLI Integration Tests (`tests/test_usecase_cli_metadata.py`)**: + * Tests `logstory usecases validate`, `logstory usecases inspect --blind` (confirming zero spoilers in output), `logstory usecases export-schemas`, and `logstory replay usecase --with-paired-baselines --local-file-output`. +3. **Full Corpus Conformance Gate**: + * Runs `logstory usecases validate --strict-v2` across all migrated usecases in `logstory` (`RULES_SEARCH_WORKSHOP`, `NETWORK_ANALYSIS`) and all 47 scenarios in `logstory_usecases` with 0 errors. diff --git a/pyproject.toml b/pyproject.toml index 276dbdf..6f9f924 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -73,10 +73,13 @@ logstory = "logstory.logstory:entry_point" "logstory" = [ "usecases/**/*.log", "*.yaml", + "usecases/**/*.yaml", "usecases/**/*.yl2", "usecases/**/*.json", "usecases/**/*.md", - "usecases/**/*.proto" + "usecases/**/*.j2", + "usecases/**/*.proto", + "schemas/*.json", ] [tool.ruff] diff --git a/schemas/eval.schema.json b/schemas/eval.schema.json new file mode 100644 index 0000000..32198b4 --- /dev/null +++ b/schemas/eval.schema.json @@ -0,0 +1,671 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://chronicle.github.io/logstory/schemas/eval.schema.json", + "title": "SecurityUsecaseEval", + "description": "Held-out Blue Team Evaluation & Answer Key Schema (v2.0) for autonomous SOC agent benchmarking across Alert Triage, Threat Hunting, and Detection Engineering.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "usecase_id", + "threat_context", + "tasks", + "ground_truth", + "compliance_gates" + ], + "properties": { + "schema_version": { + "type": "string", + "const": "2.0", + "description": "Evaluation schema version ('2.0')." + }, + "usecase_id": { + "type": "string", + "pattern": "^[A-Z0-9_]+$", + "description": "Usecase identifier matching .md id." + }, + "threat_context": { + "$ref": "#/$defs/ThreatContext" + }, + "tasks": { + "$ref": "#/$defs/EvaluationTasks" + }, + "ground_truth": { + "$ref": "#/$defs/GroundTruth" + }, + "compliance_gates": { + "$ref": "#/$defs/ComplianceGates" + }, + "counterfactual_pair": { + "oneOf": [ + { + "$ref": "#/$defs/CounterfactualPair" + }, + { + "type": "null" + } + ], + "description": "Optional Signal Detection Theory twin pairing." + }, + "reference_queries": { + "$ref": "#/$defs/ReferenceQueries" + } + }, + "$defs": { + "ThreatContext": { + "type": "object", + "additionalProperties": false, + "required": [ + "grounding_tier" + ], + "description": "Adversary, advisory, and MITRE ATT&CK context.", + "properties": { + "grounding_tier": { + "type": "string", + "enum": [ + "stix_grounded", + "tradecraft_modeled", + "behavioral_synthetic", + "benign_baseline" + ] + }, + "difficulty_tier": { + "type": "string", + "enum": [ + "beginner", + "intermediate", + "advanced", + "expert", + "tier_1", + "tier_2", + "tier_3", + "tier_4", + "tier_5" + ] + }, + "threat_actor": { + "type": [ + "string", + "null" + ] + }, + "actor_aliases": { + "type": "array", + "items": { + "type": "string" + } + }, + "campaign_name": { + "type": [ + "string", + "null" + ] + }, + "cisa_advisories": { + "type": "array", + "items": { + "type": "string" + } + }, + "cve_ids": { + "type": "array", + "items": { + "type": "string" + } + }, + "mitre_tactics": { + "type": "array", + "items": { + "type": "string" + } + }, + "mitre_techniques": { + "type": "array", + "items": { + "type": "string" + } + }, + "release_day": { + "type": [ + "integer", + "null" + ] + }, + "cohort": { + "type": [ + "string", + "null" + ] + }, + "notes": { + "type": "string" + } + } + }, + "AlertTriageTask": { + "type": "object", + "additionalProperties": false, + "description": "Alert triage benchmark task configuration.", + "properties": { + "enabled": { + "type": "boolean" + }, + "case_id": { + "type": [ + "string", + "null" + ] + }, + "alert_ids": { + "type": "array", + "items": { + "type": "string" + } + }, + "prompt": { + "type": "string" + }, + "expected_skill": { + "type": [ + "string", + "null" + ] + }, + "skill_bindings": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "time_horizon_hours": { + "type": "integer" + } + } + }, + "ThreatHuntTask": { + "type": "object", + "additionalProperties": false, + "description": "Proactive threat hunt task configuration.", + "properties": { + "enabled": { + "type": "boolean" + }, + "prompt_unalerted": { + "type": "string" + }, + "prompt_intel_driven": { + "type": [ + "string", + "null" + ] + }, + "expected_patient_zero": { + "type": [ + "string", + "null" + ] + }, + "expected_skill": { + "type": [ + "string", + "null" + ] + }, + "skill_bindings": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "time_horizon_hours": { + "type": "integer" + } + } + }, + "DetectionEngineeringTask": { + "type": "object", + "additionalProperties": false, + "description": "Detection engineering task configuration.", + "properties": { + "enabled": { + "type": "boolean" + }, + "prompt": { + "type": "string" + }, + "target_techniques": { + "type": "array", + "items": { + "type": "string" + } + }, + "reference_rules": { + "type": "array", + "items": { + "type": "string" + } + }, + "expected_rule_count_min": { + "type": "integer" + }, + "max_false_positives_on_paired_baseline": { + "type": "integer" + } + } + }, + "EvaluationTasks": { + "type": "object", + "additionalProperties": false, + "description": "Multi-task Blue Team benchmark suite.", + "properties": { + "alert_triage": { + "$ref": "#/$defs/AlertTriageTask" + }, + "threat_hunt": { + "$ref": "#/$defs/ThreatHuntTask" + }, + "detection_engineering": { + "$ref": "#/$defs/DetectionEngineeringTask" + } + } + }, + "KillChainStage": { + "type": "object", + "additionalProperties": false, + "required": [ + "stage", + "stream_id" + ], + "description": "Single chronological attack stage.", + "properties": { + "stage": { + "type": "string", + "minLength": 1 + }, + "stream_id": { + "type": "string", + "minLength": 1 + }, + "timestamp_utc": { + "type": [ + "string", + "null" + ] + }, + "tactic": { + "type": [ + "string", + "null" + ] + }, + "technique_id": { + "type": [ + "string", + "null" + ] + }, + "summary": { + "type": "string" + }, + "key_evidence": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "AffectedAsset": { + "type": "object", + "additionalProperties": false, + "required": [ + "hostname" + ], + "description": "Affected enterprise asset and blast-radius cost.", + "properties": { + "hostname": { + "type": "string", + "minLength": 1 + }, + "criticality_tier": { + "type": "string", + "enum": [ + "TIER_0", + "TIER_1", + "TIER_2", + "TIER_3", + "tier_0_crown_jewel", + "tier_1_core_infra", + "tier_2_workstation", + "critical", + "high", + "medium", + "low" + ] + }, + "ip_addresses": { + "type": "array", + "items": { + "type": "string" + } + }, + "role": { + "type": "string" + }, + "patient_zero": { + "type": "boolean" + }, + "blast_radius_fp_usd": { + "type": "number" + }, + "blast_radius_fn_usd": { + "type": "number" + } + } + }, + "MaliciousIoc": { + "type": "object", + "additionalProperties": false, + "required": [ + "type", + "value" + ], + "description": "Ground-truth malicious indicator of compromise.", + "properties": { + "type": { + "type": "string", + "minLength": 1 + }, + "value": { + "type": "string", + "minLength": 1 + }, + "role": { + "type": "string" + }, + "notes": { + "type": "string" + } + } + }, + "ChangeWindow": { + "type": "object", + "additionalProperties": false, + "required": [ + "ticket_id" + ], + "description": "Maintenance or change window record.", + "properties": { + "ticket_id": { + "type": "string", + "minLength": 1 + }, + "approved": { + "type": "boolean" + }, + "start_utc": { + "type": [ + "string", + "null" + ] + }, + "end_utc": { + "type": [ + "string", + "null" + ] + }, + "summary": { + "type": "string" + } + } + }, + "BenignConfusers": { + "type": "object", + "additionalProperties": false, + "description": "Benign confusers and signed OS LOLBins.", + "properties": { + "signed_lolbins": { + "type": "array", + "items": { + "type": "string" + } + }, + "internal_ips": { + "type": "array", + "items": { + "type": "string" + } + }, + "change_window": { + "oneOf": [ + { + "$ref": "#/$defs/ChangeWindow" + }, + { + "type": "null" + } + ] + }, + "admin_tools": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "EntitiesAndIocs": { + "type": "object", + "additionalProperties": false, + "description": "Ground-truth entities, IOCs, and confusers.", + "properties": { + "compromised_accounts": { + "type": "array", + "items": { + "type": "string" + } + }, + "affected_assets": { + "type": "array", + "items": { + "$ref": "#/$defs/AffectedAsset" + } + }, + "malicious_iocs": { + "type": "array", + "items": { + "$ref": "#/$defs/MaliciousIoc" + } + }, + "benign_confusers": { + "$ref": "#/$defs/BenignConfusers" + } + } + }, + "GroundTruth": { + "type": "object", + "additionalProperties": false, + "required": [ + "expected_disposition", + "expected_verdict" + ], + "description": "Ground-truth disposition, verdict, and timeline.", + "properties": { + "expected_disposition": { + "type": "string", + "enum": [ + "TRUE_POSITIVE", + "BENIGN_POSITIVE", + "FALSE_POSITIVE", + "SUSPICIOUS_TRUE_POSITIVE", + "INCONCLUSIVE" + ] + }, + "expected_verdict": { + "type": "string", + "enum": [ + "MALICIOUS", + "BENIGN", + "SUSPICIOUS", + "INCONCLUSIVE" + ] + }, + "severity": { + "type": "string" + }, + "patient_zero_host": { + "type": [ + "string", + "null" + ] + }, + "patient_zero_account": { + "type": [ + "string", + "null" + ] + }, + "root_cause_summary": { + "type": "string" + }, + "kill_chain": { + "type": "array", + "items": { + "$ref": "#/$defs/KillChainStage" + } + }, + "entities_and_iocs": { + "$ref": "#/$defs/EntitiesAndIocs" + } + } + }, + "ExpectedActions": { + "type": "object", + "additionalProperties": false, + "description": "Primary, secondary, and prohibited response actions.", + "properties": { + "primary": { + "type": "array", + "items": { + "type": "string" + } + }, + "secondary": { + "type": "array", + "items": { + "type": "string" + } + }, + "prohibited": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "ComplianceGates": { + "type": "object", + "additionalProperties": false, + "description": "Safety and SOP compliance requirements.", + "properties": { + "requires_pre_containment_verification": { + "type": "boolean" + }, + "requires_168h_horizon_widening": { + "type": "boolean" + }, + "forbid_external_cti_on_internal_ips": { + "type": "boolean" + }, + "forbid_lolbin_hash_blocking": { + "type": "boolean" + }, + "respect_change_window": { + "type": "boolean" + }, + "expected_actions": { + "$ref": "#/$defs/ExpectedActions" + } + } + }, + "CounterfactualPair": { + "type": "object", + "additionalProperties": false, + "required": [ + "paired_usecase_id", + "pair_relationship" + ], + "description": "Counterfactual benign/malicious twin link.", + "properties": { + "paired_usecase_id": { + "type": "string", + "minLength": 1 + }, + "pair_relationship": { + "type": "string", + "enum": [ + "malicious_variant_of_benign_baseline", + "benign_baseline_for_malicious_variant", + "confusable_twin", + "benign_twin", + "attack_twin" + ] + }, + "shared_entities": { + "type": "array", + "items": { + "type": "string" + } + }, + "shared_telemetry_streams": { + "type": "array", + "items": { + "type": "string" + } + }, + "discriminating_invariant": { + "type": "string" + } + } + }, + "ReferenceQueries": { + "type": "object", + "additionalProperties": false, + "description": "Reference investigative queries by platform.", + "properties": { + "google_secops_udm": { + "type": "array", + "items": { + "type": "string" + } + }, + "google_secops_yaral": { + "type": "array", + "items": { + "type": "string" + } + }, + "databricks_sql": { + "type": "array", + "items": { + "type": "string" + } + }, + "sigma": { + "type": "array", + "items": { + "type": "string" + } + }, + "splunk_spl": { + "type": "array", + "items": { + "type": "string" + } + } + } + } + } +} diff --git a/schemas/usecase.schema.json b/schemas/usecase.schema.json new file mode 100644 index 0000000..648b1fe --- /dev/null +++ b/schemas/usecase.schema.json @@ -0,0 +1,475 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://chronicle.github.io/logstory/schemas/usecase.schema.json", + "title": "SecurityUsecaseManifest", + "description": "Open Security Scenario & Logstory Usecase Manifest Schema (v2.0) combining OKF v0.2 provenance, OCSF telemetry streams, and multi-platform bindings.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "id", + "title", + "version", + "created", + "updated", + "author", + "license", + "description", + "tags", + "environment", + "telemetry", + "platforms" + ], + "properties": { + "schema_version": { + "type": "string", + "const": "2.0", + "description": "Usecase manifest schema version ('2.0')." + }, + "id": { + "type": "string", + "pattern": "^[A-Z0-9_]+$", + "description": "Canonical uppercase usecase identifier." + }, + "title": { + "type": "string", + "minLength": 1, + "description": "Human-readable blind-safe package title." + }, + "version": { + "type": "string", + "minLength": 1, + "description": "Semantic version of the usecase package." + }, + "created": { + "type": "string", + "pattern": "^\\d{4}-\\d{2}-\\d{2}$", + "description": "Initial authoring date (YYYY-MM-DD)." + }, + "updated": { + "type": "string", + "pattern": "^\\d{4}-\\d{2}-\\d{2}$", + "description": "Last modification date (YYYY-MM-DD)." + }, + "author": { + "type": "string", + "minLength": 1, + "description": "Package author or organization." + }, + "license": { + "type": "string", + "minLength": 1, + "description": "SPDX license identifier (e.g., Apache-2.0)." + }, + "description": { + "type": "string", + "minLength": 1, + "description": "Blind-safe summary of the telemetry streams and scenario context (no spoilers)." + }, + "tags": { + "type": "array", + "minItems": 1, + "items": { + "type": "string" + }, + "description": "Discovery tags for the usecase package." + }, + "environment": { + "$ref": "#/$defs/EnvironmentContext" + }, + "telemetry": { + "$ref": "#/$defs/TelemetrySection" + }, + "platforms": { + "$ref": "#/$defs/PlatformBindings" + } + }, + "$defs": { + "TimestampPolicy": { + "type": "object", + "additionalProperties": false, + "description": "Replay timestamp adjustment policy.", + "properties": { + "base_date": { + "type": "string", + "description": "Base date of synthetic logs (YYYY-MM-DD)." + }, + "preserve_relative_offsets": { + "type": "boolean", + "description": "Whether relative inter-event offsets are preserved." + }, + "default_delta": { + "type": "string", + "description": "Default timestamp shift expression." + }, + "entity_offset_days": { + "type": "integer", + "description": "Days prior to events to replay entity context logs." + }, + "window_start": { + "type": [ + "string", + "null" + ], + "description": "Optional UTC window start timestamp." + }, + "window_end": { + "type": [ + "string", + "null" + ], + "description": "Optional UTC window end timestamp." + } + } + }, + "EnvironmentContext": { + "type": "object", + "additionalProperties": false, + "description": "Simulated organization, scenario classification, and composable paired baselines.", + "properties": { + "organization": { + "type": "string", + "description": "Simulated enterprise domain name." + }, + "scenario_type": { + "type": "string", + "enum": [ + "attack", + "benign_baseline", + "baseline", + "hybrid", + "workshop" + ], + "description": "High-level scenario classification." + }, + "paired_baselines": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Composable benign baseline usecase IDs to co-replay." + }, + "timestamp_policy": { + "$ref": "#/$defs/TimestampPolicy" + }, + "notes": { + "type": "string", + "description": "Optional environment notes." + } + } + }, + "TelemetryStream": { + "type": "object", + "additionalProperties": false, + "required": [ + "stream_id", + "path", + "log_type" + ], + "description": "Vendor-neutral event or entity telemetry stream.", + "properties": { + "stream_id": { + "type": "string", + "minLength": 1, + "description": "Stable identifier for the stream." + }, + "path": { + "type": "string", + "minLength": 1, + "description": "Relative path to the log file inside the usecase." + }, + "log_type": { + "type": "string", + "minLength": 1, + "description": "Canonical ingestion log type identifier." + }, + "vendor_name": { + "type": "string", + "description": "Telemetry vendor name." + }, + "product_name": { + "type": "string", + "description": "Telemetry product name." + }, + "ocsf_category_uid": { + "type": [ + "integer", + "null" + ], + "description": "OCSF v1.3+ category UID." + }, + "ocsf_class_uid": { + "type": [ + "integer", + "null" + ], + "description": "OCSF v1.3+ class UID." + }, + "ocsf_class_name": { + "type": [ + "string", + "null" + ], + "description": "OCSF v1.3+ class name." + }, + "notes": { + "type": "string", + "description": "Blind-safe stream notes." + } + } + }, + "TelemetrySection": { + "type": "object", + "additionalProperties": false, + "description": "Event and entity telemetry streams.", + "properties": { + "events": { + "type": "array", + "items": { + "$ref": "#/$defs/TelemetryStream" + }, + "description": "Event log streams under EVENTS/." + }, + "entities": { + "type": "array", + "items": { + "$ref": "#/$defs/TelemetryStream" + }, + "description": "Entity context streams under ENTITIES/." + } + } + }, + "GoogleSecOpsRunFrequency": { + "type": "object", + "additionalProperties": false, + "required": [ + "events" + ], + "description": "Recommended Google SecOps replay cadence.", + "properties": { + "events": { + "type": "string", + "description": "Replay frequency for event streams." + }, + "entities": { + "type": [ + "string", + "null" + ], + "description": "Replay frequency for entity streams." + } + } + }, + "GoogleSecOpsDetection": { + "type": "object", + "additionalProperties": false, + "required": [ + "name" + ], + "description": "Google SecOps YARA-L 2.0 detection rule entry.", + "properties": { + "name": { + "type": "string", + "minLength": 1, + "description": "Rule name." + }, + "path": { + "type": [ + "string", + "null" + ], + "description": "Optional relative path to .yl2 file." + }, + "live": { + "type": "boolean", + "description": "Whether live rule evaluation is enabled." + }, + "alerting": { + "type": "boolean", + "description": "Whether alerting is enabled." + }, + "notes": { + "type": "string", + "description": "Rule notes." + } + } + }, + "GoogleSecOpsSavedSearch": { + "type": "object", + "additionalProperties": false, + "required": [ + "name" + ], + "description": "Saved UDM search definition.", + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "creator": { + "type": "string" + }, + "notes": { + "type": "string" + }, + "query": { + "type": [ + "string", + "null" + ] + } + } + }, + "GoogleSecOpsReferenceList": { + "type": "object", + "additionalProperties": false, + "required": [ + "name" + ], + "description": "Google SecOps reference list metadata.", + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "type": { + "type": "string" + }, + "notes": { + "type": "string" + } + } + }, + "GoogleSecOpsPlatform": { + "type": "object", + "additionalProperties": false, + "description": "Google Security Operations platform bindings.", + "properties": { + "run_frequency": { + "$ref": "#/$defs/GoogleSecOpsRunFrequency" + }, + "data_rbac": { + "type": [ + "string", + "null" + ] + }, + "log_type_mappings": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "detections": { + "type": "array", + "items": { + "$ref": "#/$defs/GoogleSecOpsDetection" + } + }, + "saved_searches": { + "type": "array", + "items": { + "$ref": "#/$defs/GoogleSecOpsSavedSearch" + } + }, + "reference_lists": { + "type": "array", + "items": { + "$ref": "#/$defs/GoogleSecOpsReferenceList" + } + }, + "parser_extensions": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "DatabricksPlatform": { + "type": "object", + "additionalProperties": false, + "description": "Databricks Unity Catalog OCSF Lakehouse bindings.", + "properties": { + "catalog": { + "type": "string" + }, + "schema_name": { + "type": "string" + }, + "table_mappings": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "detections": { + "type": "array", + "items": { + "type": "string" + } + }, + "notes": { + "type": "string" + } + } + }, + "SigmaPlatform": { + "type": "object", + "additionalProperties": false, + "description": "Sigma detection rule bindings.", + "properties": { + "rules": { + "type": "array", + "items": { + "type": "string" + } + }, + "notes": { + "type": "string" + } + } + }, + "SplunkPlatform": { + "type": "object", + "additionalProperties": false, + "description": "Splunk sourcetype and search bindings.", + "properties": { + "sourcetypes": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "saved_searches": { + "type": "array", + "items": { + "type": "string" + } + }, + "notes": { + "type": "string" + } + } + }, + "PlatformBindings": { + "type": "object", + "additionalProperties": false, + "description": "Multi-platform ingestion and detection bindings.", + "properties": { + "google_secops": { + "$ref": "#/$defs/GoogleSecOpsPlatform" + }, + "databricks": { + "$ref": "#/$defs/DatabricksPlatform" + }, + "sigma": { + "$ref": "#/$defs/SigmaPlatform" + }, + "splunk": { + "$ref": "#/$defs/SplunkPlatform" + } + } + } + } +} diff --git a/src/logstory/logstory.py b/src/logstory/logstory.py index f7c3b29..d60810d 100644 --- a/src/logstory/logstory.py +++ b/src/logstory/logstory.py @@ -23,6 +23,7 @@ import tempfile import uuid from importlib.metadata import version +from pathlib import Path try: from . import main as imported_main @@ -36,6 +37,14 @@ from google.oauth2 import service_account from logstory.auth import has_application_default_credentials +from logstory.schema import ( + SchemaValidationError, + export_json_schemas, + parse_usecase_package, + split_frontmatter_and_body, + validate_usecase_corpus, + validate_usecase_package, +) UTC = datetime.UTC @@ -734,6 +743,183 @@ def usecase_get( raise typer.Exit(1) +@usecases_app.command("validate") +def usecases_validate( + target: str | None = typer.Argument( + None, + help=( + "Path to a usecase directory, a corpus directory, or an installed" + " usecase name (validates all installed usecases if omitted)" + ), + ), + strict_v2: bool = typer.Option( + False, + "--strict-v2", + help="Require schema_version 2.0 and fail on legacy v1 manifests", + ), + output_format: str = typer.Option( + "text", "--format", help="Output format: text or json" + ), +) -> None: + """Validate one or more usecase packages across the 5 verification gates.""" + package_dir = Path(os.path.dirname(os.path.abspath(__file__))) + installed_root = package_dir / "usecases" + + if target is None: + results = [ + validate_usecase_package(uc_dir, strict_v2=strict_v2) + for uc_dir in sorted(installed_root.iterdir(), key=lambda p: p.name) + if uc_dir.is_dir() and (uc_dir / f"{uc_dir.name}.md").exists() + ] + elif Path(target).exists(): + target_path = Path(target) + if (target_path / f"{target_path.name}.md").exists(): + results = [validate_usecase_package(target_path, strict_v2=strict_v2)] + else: + results = validate_usecase_corpus(target_path, strict_v2=strict_v2) + else: + installed_dir = installed_root / target + if not installed_dir.is_dir(): + typer.echo(f"Error: Usecase or path '{target}' not found.") + raise typer.Exit(code=1) + results = [validate_usecase_package(installed_dir, strict_v2=strict_v2)] + + passed = sum(1 for r in results if r.is_valid) + failed = len(results) - passed + all_valid = bool(results) and failed == 0 + + if output_format.lower() == "json": + payload = { + "valid": all_valid, + "total": len(results), + "passed": passed, + "failed": failed, + "results": [r.to_dict() for r in results], + } + typer.echo(json.dumps(payload, indent=2)) + else: + for r in results: + status_label = "PASS" if r.is_valid else "FAIL" + typer.echo(f"[{status_label}] {r.usecase_id} (schema_version={r.schema_version})") + for finding in r.findings: + loc = f" ({finding.path})" if finding.path else "" + typer.echo(f" [{finding.severity}] {finding.gate}{loc}: {finding.message}") + typer.echo(f"Summary: total={len(results)} passed={passed} failed={failed}") + + if not all_valid: + raise typer.Exit(code=1) + + +@usecases_app.command("inspect") +def usecases_inspect( + target: str = typer.Argument( + ..., help="Path to a usecase directory or installed usecase name" + ), + blind: bool = typer.Option( + True, + "--blind/--include-eval", + help=( + "Blind mode strips solution section below " + " and omits eval.yaml" + ), + ), + output_format: str = typer.Option( + "json", "--format", help="Output format: json or text" + ), +) -> None: + """Inspect a usecase package in blind mode (default) or full evaluation mode.""" + package_dir = Path(os.path.dirname(os.path.abspath(__file__))) + target_path = Path(target) + if target_path.exists(): + usecase_dir = target_path + else: + usecase_dir = package_dir / "usecases" / target + if not usecase_dir.is_dir(): + typer.echo(f"Error: Usecase or path '{target}' not found.") + raise typer.Exit(code=1) + + try: + pkg = parse_usecase_package(usecase_dir, blind=blind) + except SchemaValidationError as exc: + typer.echo(f"Error parsing usecase '{target}': {exc}") + raise typer.Exit(code=1) from exc + + if output_format.lower() == "json": + typer.echo(json.dumps(pkg.to_dict(blind=blind), indent=2)) + else: + manifest_dict = ( + pkg.manifest.to_dict() if pkg.manifest is not None else pkg.raw_frontmatter + ) + uc_id = manifest_dict.get("id", usecase_dir.name) + title = manifest_dict.get("title", uc_id) + description = manifest_dict.get("description", "") + typer.echo(f"id: {uc_id}") + typer.echo(f"schema_version: {pkg.schema_version}") + typer.echo(f"title: {title}") + typer.echo(f"description: {description}") + if pkg.manifest is not None: + event_ids = [s.stream_id for s in pkg.manifest.telemetry.events] + entity_ids = [s.stream_id for s in pkg.manifest.telemetry.entities] + typer.echo(f"event_streams: {', '.join(event_ids)}") + typer.echo(f"entity_streams: {', '.join(entity_ids)}") + typer.echo("\n--- Public Markdown Body ---") + typer.echo(pkg.public_markdown_body) + if not blind: + if pkg.solution_markdown_body is not None: + typer.echo("\n--- Solution Markdown Body ---") + typer.echo(pkg.solution_markdown_body) + if pkg.eval_data is not None: + typer.echo("\n--- Eval Data ---") + typer.echo(json.dumps(pkg.eval_data.to_dict(), indent=2)) + + +@usecases_app.command("export-schemas") +def usecases_export_schemas( + output_dir: str = typer.Option( + "schemas", + "--output-dir", + help="Directory to write usecase.schema.json and eval.schema.json", + ), +) -> None: + """Export Draft 2020-12 JSON Schemas for usecase.md and eval.yaml.""" + usecase_schema_path, eval_schema_path = export_json_schemas(Path(output_dir)) + typer.echo(f"Wrote {usecase_schema_path}") + typer.echo(f"Wrote {eval_schema_path}") + + +def _get_paired_baselines( + usecase: str, search_roots: list[Path] | None = None +) -> list[str]: + """Locate `/.md` and return `environment.paired_baselines`.""" + package_dir = Path(os.path.dirname(os.path.abspath(__file__))) + roots: list[Path] = [] + if search_roots: + roots.extend(Path(r) for r in search_roots) + roots.extend([ + package_dir / "usecases", + Path("usecases"), + ]) + for root in roots: + md_path = root / usecase / f"{usecase}.md" + if md_path.is_file(): + try: + frontmatter, _ = split_frontmatter_and_body(md_path.read_text(encoding="utf-8")) + except Exception: + return [] + env_block = frontmatter.get("environment") + if not isinstance(env_block, dict): + return [] + raw_baselines = env_block.get("paired_baselines", []) + if not isinstance(raw_baselines, list): + return [] + deduped: list[str] = [] + for item in raw_baselines: + if isinstance(item, str) and item.strip() and item not in deduped: + deduped.append(item) + return deduped + return [] + + def _get_logtypes(usecase: str, entities: bool = False) -> list[str]: """Get logtype names for a usecase without printing.""" entity_or_event = "ENTITIES" if entities else "EVENTS" @@ -984,6 +1170,14 @@ def replay_usecase( project_id: str | None = ProjectIdOption, forwarder_name: str | None = ForwarderNameOption, impersonate_service_account: str | None = ImpersonateServiceAccountOption, + with_paired_baselines: bool = typer.Option( + False, + "--with-paired-baselines", + help=( + "Also replay composable paired baselines declared in" + " environment.paired_baselines" + ), + ), ): """Replay a specific usecase.""" # Load environment file first (needed for download logic) @@ -1043,6 +1237,29 @@ def replay_usecase( impersonate_service_account, ) + if with_paired_baselines: + for baseline_uc in _get_paired_baselines(usecase): + if get_if_missing and baseline_uc not in get_usecases(): + print(f"Paired baseline '{baseline_uc}' not found locally, downloading...") + if not _download_usecase(baseline_uc): + raise typer.Exit(1) + current_available = get_usecases() + if baseline_uc not in current_available: + print( + f"Paired baseline '{baseline_uc}' not found. Available usecases:" + f" {', '.join(sorted(current_available))}" + ) + raise typer.Exit(1) + baseline_logtypes = _get_logtypes(baseline_uc, entities=entities) + if baseline_logtypes: + _replay_usecases( + [baseline_uc], + baseline_logtypes, + entities, + timestamp_delta, + local_file_output, + ) + usecases = [usecase] logtypes = _get_logtypes(usecase, entities=entities) if not logtypes: diff --git a/src/logstory/schema/__init__.py b/src/logstory/schema/__init__.py new file mode 100644 index 0000000..d091708 --- /dev/null +++ b/src/logstory/schema/__init__.py @@ -0,0 +1,145 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Public exports for Logstory v2 usecase and evaluation schema models.""" + +from __future__ import annotations + +from logstory.schema.models import ( + JSON_SCHEMA_DRAFT_2020_12, + SCHEMA_VERSION_V1, + SCHEMA_VERSION_V2, + SOLUTION_DELIMITER, + AffectedAsset, + AlertTriageTask, + BenignConfusers, + ChangeWindow, + ComplianceGates, + CounterfactualPair, + CriticalityTier, + DatabricksPlatform, + DetectionEngineeringTask, + DifficultyTier, + EntitiesAndIocs, + EnvironmentContext, + EvaluationTasks, + ExpectedActions, + ExpectedDisposition, + ExpectedVerdict, + GoogleSecOpsDetection, + GoogleSecOpsPlatform, + GoogleSecOpsReferenceList, + GoogleSecOpsRunFrequency, + GoogleSecOpsSavedSearch, + GroundingTier, + GroundTruth, + KillChainStage, + MaliciousIoc, + PairRelationship, + ParsedUsecasePackage, + PlatformBindings, + ReferenceQueries, + ScenarioType, + SchemaValidationError, + SecurityUsecaseEval, + SecurityUsecaseManifest, + SigmaPlatform, + SplunkPlatform, + TelemetrySection, + TelemetryStream, + ThreatContext, + ThreatHuntTask, + TimestampPolicy, + build_eval_json_schema, + build_usecase_json_schema, + export_json_schemas, + parse_usecase_package, + split_frontmatter_and_body, + split_solution_section, +) +from logstory.schema.validator import ( + FORBIDDEN_FRONTMATTER_KEYS, + GATE_G1_SCHEMA, + GATE_G2_INTEGRITY, + GATE_G3_ANTI_SPOILER, + GATE_G4_COUNTERFACTUAL, + GATE_G5_LEGACY_V1, + UsecaseValidationResult, + ValidationFinding, + validate_usecase_corpus, + validate_usecase_package, +) + +__all__ = [ + "FORBIDDEN_FRONTMATTER_KEYS", + "GATE_G1_SCHEMA", + "GATE_G2_INTEGRITY", + "GATE_G3_ANTI_SPOILER", + "GATE_G4_COUNTERFACTUAL", + "GATE_G5_LEGACY_V1", + "JSON_SCHEMA_DRAFT_2020_12", + "SCHEMA_VERSION_V1", + "SCHEMA_VERSION_V2", + "SOLUTION_DELIMITER", + "AffectedAsset", + "AlertTriageTask", + "BenignConfusers", + "ChangeWindow", + "ComplianceGates", + "CounterfactualPair", + "CriticalityTier", + "DatabricksPlatform", + "DetectionEngineeringTask", + "DifficultyTier", + "EntitiesAndIocs", + "EnvironmentContext", + "EvaluationTasks", + "ExpectedActions", + "ExpectedDisposition", + "ExpectedVerdict", + "GoogleSecOpsDetection", + "GoogleSecOpsPlatform", + "GoogleSecOpsReferenceList", + "GoogleSecOpsRunFrequency", + "GoogleSecOpsSavedSearch", + "GroundingTier", + "GroundTruth", + "KillChainStage", + "MaliciousIoc", + "PairRelationship", + "ParsedUsecasePackage", + "PlatformBindings", + "ReferenceQueries", + "ScenarioType", + "SchemaValidationError", + "SecurityUsecaseEval", + "SecurityUsecaseManifest", + "SigmaPlatform", + "SplunkPlatform", + "TelemetrySection", + "TelemetryStream", + "ThreatContext", + "ThreatHuntTask", + "TimestampPolicy", + "UsecaseValidationResult", + "ValidationFinding", + "build_eval_json_schema", + "build_usecase_json_schema", + "export_json_schemas", + "parse_usecase_package", + "split_frontmatter_and_body", + "split_solution_section", + "validate_usecase_corpus", + "validate_usecase_package", +] diff --git a/src/logstory/schema/models.py b/src/logstory/schema/models.py new file mode 100644 index 0000000..94801ae --- /dev/null +++ b/src/logstory/schema/models.py @@ -0,0 +1,3080 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Core v2 schema models, package parser, and Draft 2020-12 JSON Schema exporter.""" + +from __future__ import annotations + +import datetime +import json +import re +from dataclasses import dataclass, field +from enum import StrEnum +from pathlib import Path +from typing import Any, TypeVar + +import yaml + +SCHEMA_VERSION_V1 = "1.0" +SCHEMA_VERSION_V2 = "2.0" +SOLUTION_DELIMITER = "" +JSON_SCHEMA_DRAFT_2020_12 = "https://json-schema.org/draft/2020-12/schema" +USECASE_ID_PATTERN = r"^[A-Z0-9_]+$" +ISO_DATE_PATTERN = r"^\d{4}-\d{2}-\d{2}$" +MIN_FRONTMATTER_PARTS = 3 +DEFAULT_TRIAGE_HORIZON_HOURS = 72 +DEFAULT_HUNT_HORIZON_HOURS = 168 + +_USECASE_ID_RE = re.compile(USECASE_ID_PATTERN) +_ISO_DATE_RE = re.compile(ISO_DATE_PATTERN) + +E = TypeVar("E", bound=StrEnum) + + +class SchemaValidationError(ValueError): + """Raised when usecase manifest or eval metadata violates the v2 schema.""" + + def __init__(self, message: str, path: str | None = None) -> None: + """Initialize SchemaValidationError with an optional field path. + + Args: + message: Human-readable validation failure description. + path: Dotted field path where validation failed. + """ + self.path = path + self.reason = message + full_message = f"{path}: {message}" if path else message + super().__init__(full_message) + + +class ScenarioType(StrEnum): + """High-level scenario classification for a usecase package.""" + + ATTACK = "attack" + BENIGN_BASELINE = "benign_baseline" + BASELINE = "baseline" + HYBRID = "hybrid" + WORKSHOP = "workshop" + + +class GroundingTier(StrEnum): + """Real-world threat intelligence grounding tier for a scenario.""" + + STIX_GROUNDED = "stix_grounded" + TRADECRAFT_MODELED = "tradecraft_modeled" + BEHAVIORAL_SYNTHETIC = "behavioral_synthetic" + BENIGN_BASELINE = "benign_baseline" + + +class ExpectedDisposition(StrEnum): + """Canonical SOC 4-way (+ inconclusive) case triage disposition.""" + + TRUE_POSITIVE = "TRUE_POSITIVE" + BENIGN_POSITIVE = "BENIGN_POSITIVE" + FALSE_POSITIVE = "FALSE_POSITIVE" + SUSPICIOUS_TRUE_POSITIVE = "SUSPICIOUS_TRUE_POSITIVE" + INCONCLUSIVE = "INCONCLUSIVE" + + +class ExpectedVerdict(StrEnum): + """High-level threat verdict for evaluation scoring.""" + + MALICIOUS = "MALICIOUS" + BENIGN = "BENIGN" + SUSPICIOUS = "SUSPICIOUS" + INCONCLUSIVE = "INCONCLUSIVE" + + +class CriticalityTier(StrEnum): + """Enterprise asset criticality tier for blast-radius evaluation.""" + + TIER_0 = "TIER_0" + TIER_1 = "TIER_1" + TIER_2 = "TIER_2" + TIER_3 = "TIER_3" + TIER_0_CROWN_JEWEL = "tier_0_crown_jewel" + TIER_1_CORE_INFRA = "tier_1_core_infra" + TIER_2_WORKSTATION = "tier_2_workstation" + CRITICAL = "critical" + HIGH = "high" + MEDIUM = "medium" + LOW = "low" + + +class PairRelationship(StrEnum): + """Signal Detection Theory relationship between counterfactual twin usecases.""" + + MALICIOUS_VARIANT_OF_BENIGN_BASELINE = "malicious_variant_of_benign_baseline" + BENIGN_BASELINE_FOR_MALICIOUS_VARIANT = "benign_baseline_for_malicious_variant" + CONFUSABLE_TWIN = "confusable_twin" + BENIGN_TWIN = "benign_twin" + ATTACK_TWIN = "attack_twin" + + +class DifficultyTier(StrEnum): + """Progression difficulty tier for Blue Team evaluation tasks.""" + + BEGINNER = "beginner" + INTERMEDIATE = "intermediate" + ADVANCED = "advanced" + EXPERT = "expert" + TIER_1 = "tier_1" + TIER_2 = "tier_2" + TIER_3 = "tier_3" + TIER_4 = "tier_4" + TIER_5 = "tier_5" + + +def _normalize_yaml_scalars(value: Any) -> Any: + """Recursively normalize YAML date/datetime scalars into ISO strings.""" + if isinstance(value, datetime.datetime): + return value.isoformat() + if isinstance(value, datetime.date): + return value.isoformat() + if isinstance(value, dict): + return {str(k): _normalize_yaml_scalars(v) for k, v in value.items()} + if isinstance(value, list): + return [_normalize_yaml_scalars(item) for item in value] + return value + + +def _ensure_mapping(data: Any, path: str) -> dict[str, Any]: + """Verify that data is a dict and return a normalized shallow copy.""" + if not isinstance(data, dict): + raise SchemaValidationError( + f"Expected mapping/object, got {type(data).__name__}", path=path + ) + return dict(data) + + +def _reject_unknown_keys(data: dict[str, Any], allowed: set[str], path: str) -> None: + """Raise SchemaValidationError if any keys in data are outside allowed.""" + unknown = sorted(set(data.keys()) - allowed) + if unknown: + joined = ", ".join(unknown) + raise SchemaValidationError(f"Unknown field(s): {joined}", path=path) + + +def _require_non_empty_str(data: dict[str, Any], key: str, path: str) -> str: + """Extract a required non-empty string field from data.""" + field_path = f"{path}.{key}" if path else key + if key not in data or data[key] is None: + raise SchemaValidationError(f"Missing required field '{key}'", path=field_path) + val = data[key] + if isinstance(val, (datetime.date, datetime.datetime)): + val = val.isoformat() + if not isinstance(val, str) or not val.strip(): + raise SchemaValidationError( + f"Field '{key}' must be a non-empty string", path=field_path + ) + return val + + +def _optional_str( + data: dict[str, Any], + key: str, + path: str, + default: str | None = None, +) -> str | None: + """Extract an optional string field from data.""" + field_path = f"{path}.{key}" if path else key + if key not in data or data[key] is None: + return default + val = data[key] + if isinstance(val, (datetime.date, datetime.datetime)): + val = val.isoformat() + if not isinstance(val, str): + raise SchemaValidationError(f"Field '{key}' must be a string", path=field_path) + return val + + +def _optional_bool( + data: dict[str, Any], key: str, path: str, default: bool = False +) -> bool: + """Extract an optional boolean field from data.""" + field_path = f"{path}.{key}" if path else key + if key not in data or data[key] is None: + return default + val = data[key] + if not isinstance(val, bool): + raise SchemaValidationError(f"Field '{key}' must be a boolean", path=field_path) + return val + + +def _optional_int( + data: dict[str, Any], + key: str, + path: str, + default: int | None = None, +) -> int | None: + """Extract an optional integer field from data.""" + field_path = f"{path}.{key}" if path else key + if key not in data or data[key] is None: + return default + val = data[key] + if isinstance(val, bool) or not isinstance(val, int): + raise SchemaValidationError(f"Field '{key}' must be an integer", path=field_path) + return val + + +def _optional_float( + data: dict[str, Any], key: str, path: str, default: float = 0.0 +) -> float: + """Extract an optional numeric field as float.""" + field_path = f"{path}.{key}" if path else key + if key not in data or data[key] is None: + return default + val = data[key] + if isinstance(val, bool) or not isinstance(val, (int, float)): + raise SchemaValidationError(f"Field '{key}' must be a number", path=field_path) + return float(val) + + +def _str_list( + data: dict[str, Any], + key: str, + path: str, + required: bool = False, +) -> list[str]: + """Extract a list of strings from data.""" + field_path = f"{path}.{key}" if path else key + if key not in data or data[key] is None: + if required: + raise SchemaValidationError(f"Missing required field '{key}'", path=field_path) + return [] + val = data[key] + if not isinstance(val, list) or not all(isinstance(item, str) for item in val): + raise SchemaValidationError( + f"Field '{key}' must be a list of strings", path=field_path + ) + return list(val) + + +def _str_dict(data: dict[str, Any], key: str, path: str) -> dict[str, str]: + """Extract a dict[str, str] mapping from data.""" + field_path = f"{path}.{key}" if path else key + if key not in data or data[key] is None: + return {} + val = data[key] + if not isinstance(val, dict) or not all( + isinstance(k, str) and isinstance(v, str) for k, v in val.items() + ): + raise SchemaValidationError( + f"Field '{key}' must be a mapping of string keys to string values", + path=field_path, + ) + return dict(val) + + +def _parse_enum(value: Any, enum_cls: type[E], path: str) -> E: + """Convert a raw value into an Enum member or raise SchemaValidationError.""" + if isinstance(value, enum_cls): + return value + if ( + isinstance(value, int) + and not isinstance(value, bool) + and enum_cls is DifficultyTier + ): + tier_key = f"tier_{value}" + for member in enum_cls: + if member.value == tier_key: + return member + if isinstance(value, str): + for member in enum_cls: + if member.value == value: + return member + allowed = ", ".join(m.value for m in enum_cls) + raise SchemaValidationError( + f"Invalid value {value!r}; expected one of: {allowed}", path=path + ) + + +@dataclass +class TimestampPolicy: + """Replay timestamp adjustment policy for a usecase.""" + + base_date: str = "2025-08-04" + preserve_relative_offsets: bool = True + default_delta: str = "1d" + entity_offset_days: int = 1 + window_start: str | None = None + window_end: str | None = None + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "environment.timestamp_policy" + ) -> TimestampPolicy: + """Validate and construct a TimestampPolicy from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "base_date", + "preserve_relative_offsets", + "default_delta", + "entity_offset_days", + "window_start", + "window_end", + }, + path, + ) + base_date = ( + _optional_str(mapping, "base_date", path, default="2025-08-04") or "2025-08-04" + ) + preserve_offsets = _optional_bool( + mapping, "preserve_relative_offsets", path, default=True + ) + default_delta = _optional_str(mapping, "default_delta", path, default="1d") or "1d" + entity_offset = _optional_int(mapping, "entity_offset_days", path, default=1) + return cls( + base_date=base_date, + preserve_relative_offsets=preserve_offsets, + default_delta=default_delta, + entity_offset_days=1 if entity_offset is None else entity_offset, + window_start=_optional_str(mapping, "window_start", path), + window_end=_optional_str(mapping, "window_end", path), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize TimestampPolicy to a plain dictionary.""" + result: dict[str, Any] = { + "base_date": self.base_date, + "preserve_relative_offsets": self.preserve_relative_offsets, + "default_delta": self.default_delta, + "entity_offset_days": self.entity_offset_days, + } + if self.window_start is not None: + result["window_start"] = self.window_start + if self.window_end is not None: + result["window_end"] = self.window_end + return result + + +@dataclass +class EnvironmentContext: + """Enterprise environment context, scenario type, and paired baselines.""" + + organization: str = "cymbal-investments.org" + scenario_type: ScenarioType = ScenarioType.ATTACK + paired_baselines: list[str] = field(default_factory=list) + timestamp_policy: TimestampPolicy = field(default_factory=TimestampPolicy) + notes: str = "" + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "environment" + ) -> EnvironmentContext: + """Validate and construct an EnvironmentContext from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "organization", + "scenario_type", + "paired_baselines", + "timestamp_policy", + "notes", + }, + path, + ) + org = ( + _optional_str(mapping, "organization", path, default="cymbal-investments.org") + or "cymbal-investments.org" + ) + raw_scenario_type = mapping.get("scenario_type", ScenarioType.ATTACK.value) + scenario_type = _parse_enum( + raw_scenario_type, ScenarioType, f"{path}.scenario_type" + ) + paired_baselines = _str_list(mapping, "paired_baselines", path) + raw_ts_policy = mapping.get("timestamp_policy") + ts_policy = ( + TimestampPolicy.from_dict(raw_ts_policy, f"{path}.timestamp_policy") + if raw_ts_policy is not None + else TimestampPolicy() + ) + notes = _optional_str(mapping, "notes", path, default="") or "" + return cls( + organization=org, + scenario_type=scenario_type, + paired_baselines=paired_baselines, + timestamp_policy=ts_policy, + notes=notes, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize EnvironmentContext to a plain dictionary.""" + result: dict[str, Any] = { + "organization": self.organization, + "scenario_type": self.scenario_type.value, + "paired_baselines": list(self.paired_baselines), + "timestamp_policy": self.timestamp_policy.to_dict(), + } + if self.notes: + result["notes"] = self.notes + return result + + +@dataclass +class TelemetryStream: + """Vendor-neutral event or entity telemetry stream with OCSF alignment.""" + + stream_id: str + path: str + log_type: str + vendor_name: str = "" + product_name: str = "" + ocsf_category_uid: int | None = None + ocsf_class_uid: int | None = None + ocsf_class_name: str | None = None + notes: str = "" + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "telemetry.stream" + ) -> TelemetryStream: + """Validate and construct a TelemetryStream from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "stream_id", + "path", + "log_type", + "vendor_name", + "product_name", + "ocsf_category_uid", + "ocsf_class_uid", + "ocsf_class_name", + "notes", + }, + path, + ) + stream_id = _require_non_empty_str(mapping, "stream_id", path) + rel_path = _require_non_empty_str(mapping, "path", path) + log_type = ( + _optional_str(mapping, "log_type", path, default=Path(rel_path).stem) + or Path(rel_path).stem + ) + vendor_name = _optional_str(mapping, "vendor_name", path, default="") or "" + product_name = _optional_str(mapping, "product_name", path, default="") or "" + ocsf_category_uid = _optional_int(mapping, "ocsf_category_uid", path) + ocsf_class_uid = _optional_int(mapping, "ocsf_class_uid", path) + ocsf_class_name = _optional_str(mapping, "ocsf_class_name", path) + notes = _optional_str(mapping, "notes", path, default="") or "" + return cls( + stream_id=stream_id, + path=rel_path, + log_type=log_type, + vendor_name=vendor_name, + product_name=product_name, + ocsf_category_uid=ocsf_category_uid, + ocsf_class_uid=ocsf_class_uid, + ocsf_class_name=ocsf_class_name, + notes=notes, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize TelemetryStream to a plain dictionary.""" + result: dict[str, Any] = { + "stream_id": self.stream_id, + "path": self.path, + "log_type": self.log_type, + "vendor_name": self.vendor_name, + "product_name": self.product_name, + } + if self.ocsf_category_uid is not None: + result["ocsf_category_uid"] = self.ocsf_category_uid + if self.ocsf_class_uid is not None: + result["ocsf_class_uid"] = self.ocsf_class_uid + if self.ocsf_class_name is not None: + result["ocsf_class_name"] = self.ocsf_class_name + result["notes"] = self.notes + return result + + +@dataclass +class TelemetrySection: + """Container for event and entity telemetry streams in a usecase.""" + + events: list[TelemetryStream] = field(default_factory=list) + entities: list[TelemetryStream] = field(default_factory=list) + + @classmethod + def from_dict(cls, data: dict[str, Any], path: str = "telemetry") -> TelemetrySection: + """Validate and construct a TelemetrySection from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys(mapping, {"events", "entities"}, path) + + raw_events = mapping.get("events", []) + if not isinstance(raw_events, list): + raise SchemaValidationError( + "Field 'events' must be a list", path=f"{path}.events" + ) + events = [ + TelemetryStream.from_dict(item, f"{path}.events[{idx}]") + for idx, item in enumerate(raw_events) + ] + + raw_entities = mapping.get("entities", []) + if raw_entities is None: + raw_entities = [] + if not isinstance(raw_entities, list): + raise SchemaValidationError( + "Field 'entities' must be a list", path=f"{path}.entities" + ) + entities = [ + TelemetryStream.from_dict(item, f"{path}.entities[{idx}]") + for idx, item in enumerate(raw_entities) + ] + return cls(events=events, entities=entities) + + def to_dict(self) -> dict[str, Any]: + """Serialize TelemetrySection to a plain dictionary.""" + return { + "events": [stream.to_dict() for stream in self.events], + "entities": [stream.to_dict() for stream in self.entities], + } + + +@dataclass +class GoogleSecOpsRunFrequency: + """Recommended replay schedule for Google Security Operations.""" + + events: str = "Every 1 day" + entities: str | None = None + + @classmethod + def from_dict( + cls, + data: dict[str, Any], + path: str = "platforms.google_secops.run_frequency", + ) -> GoogleSecOpsRunFrequency: + """Validate and construct GoogleSecOpsRunFrequency from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys(mapping, {"events", "entities"}, path) + events = _require_non_empty_str(mapping, "events", path) + entities = _optional_str(mapping, "entities", path) + return cls(events=events, entities=entities) + + def to_dict(self) -> dict[str, Any]: + """Serialize GoogleSecOpsRunFrequency to a plain dictionary.""" + result: dict[str, Any] = {"events": self.events} + if self.entities is not None: + result["entities"] = self.entities + return result + + +@dataclass +class GoogleSecOpsDetection: + """YARA-L 2.0 detection rule metadata for Google Security Operations.""" + + name: str + path: str | None = None + live: bool = False + alerting: bool = False + notes: str = "" + + @classmethod + def from_dict( + cls, + data: dict[str, Any], + path: str = "platforms.google_secops.detections", + ) -> GoogleSecOpsDetection: + """Validate and construct GoogleSecOpsDetection from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys(mapping, {"name", "path", "live", "alerting", "notes"}, path) + name = _require_non_empty_str(mapping, "name", path) + rule_path = _optional_str(mapping, "path", path) + live = _optional_bool(mapping, "live", path, default=False) + alerting = _optional_bool(mapping, "alerting", path, default=False) + notes = _optional_str(mapping, "notes", path, default="") or "" + return cls( + name=name, + path=rule_path, + live=live, + alerting=alerting, + notes=notes, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize GoogleSecOpsDetection to a plain dictionary.""" + return { + "name": self.name, + "path": self.path, + "live": self.live, + "alerting": self.alerting, + "notes": self.notes, + } + + +@dataclass +class GoogleSecOpsSavedSearch: + """Saved UDM search metadata for Google Security Operations.""" + + name: str + creator: str = "" + notes: str = "" + query: str | None = None + + @classmethod + def from_dict( + cls, + data: dict[str, Any], + path: str = "platforms.google_secops.saved_searches", + ) -> GoogleSecOpsSavedSearch: + """Validate and construct GoogleSecOpsSavedSearch from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys(mapping, {"name", "creator", "notes", "query"}, path) + name = _require_non_empty_str(mapping, "name", path) + creator = _optional_str(mapping, "creator", path, default="") or "" + notes = _optional_str(mapping, "notes", path, default="") or "" + query = _optional_str(mapping, "query", path) + return cls(name=name, creator=creator, notes=notes, query=query) + + def to_dict(self) -> dict[str, Any]: + """Serialize GoogleSecOpsSavedSearch to a plain dictionary.""" + result: dict[str, Any] = { + "name": self.name, + "creator": self.creator, + "notes": self.notes, + } + if self.query is not None: + result["query"] = self.query + return result + + +@dataclass +class GoogleSecOpsReferenceList: + """Reference list metadata for Google Security Operations.""" + + name: str + type: str = "String" + notes: str = "" + + @classmethod + def from_dict( + cls, + data: dict[str, Any], + path: str = "platforms.google_secops.reference_lists", + ) -> GoogleSecOpsReferenceList: + """Validate and construct GoogleSecOpsReferenceList from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys(mapping, {"name", "type", "notes"}, path) + name = _require_non_empty_str(mapping, "name", path) + list_type = _optional_str(mapping, "type", path, default="String") or "String" + notes = _optional_str(mapping, "notes", path, default="") or "" + return cls(name=name, type=list_type, notes=notes) + + def to_dict(self) -> dict[str, Any]: + """Serialize GoogleSecOpsReferenceList to a plain dictionary.""" + return { + "name": self.name, + "type": self.type, + "notes": self.notes, + } + + +@dataclass +class GoogleSecOpsPlatform: + """Platform bindings for Google Security Operations (UDM and YARA-L 2.0).""" + + run_frequency: GoogleSecOpsRunFrequency | None = None + data_rbac: str | None = None + log_type_mappings: dict[str, str] = field(default_factory=dict) + detections: list[GoogleSecOpsDetection] = field(default_factory=list) + saved_searches: list[GoogleSecOpsSavedSearch] = field(default_factory=list) + reference_lists: list[GoogleSecOpsReferenceList] = field(default_factory=list) + parser_extensions: list[str] = field(default_factory=list) + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "platforms.google_secops" + ) -> GoogleSecOpsPlatform: + """Validate and construct GoogleSecOpsPlatform from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "run_frequency", + "data_rbac", + "log_type_mappings", + "detections", + "saved_searches", + "reference_lists", + "parser_extensions", + }, + path, + ) + raw_freq = mapping.get("run_frequency") + run_frequency = ( + GoogleSecOpsRunFrequency.from_dict(raw_freq, f"{path}.run_frequency") + if raw_freq is not None + else None + ) + data_rbac = _optional_str(mapping, "data_rbac", path) + log_type_mappings = _str_dict(mapping, "log_type_mappings", path) + + raw_detections = mapping.get("detections", []) or [] + if not isinstance(raw_detections, list): + raise SchemaValidationError( + "Field 'detections' must be a list", path=f"{path}.detections" + ) + detections = [ + GoogleSecOpsDetection.from_dict(item, f"{path}.detections[{idx}]") + for idx, item in enumerate(raw_detections) + ] + + raw_searches = mapping.get("saved_searches", []) or [] + if not isinstance(raw_searches, list): + raise SchemaValidationError( + "Field 'saved_searches' must be a list", + path=f"{path}.saved_searches", + ) + saved_searches = [ + GoogleSecOpsSavedSearch.from_dict(item, f"{path}.saved_searches[{idx}]") + for idx, item in enumerate(raw_searches) + ] + + raw_ref_lists = mapping.get("reference_lists", []) or [] + if not isinstance(raw_ref_lists, list): + raise SchemaValidationError( + "Field 'reference_lists' must be a list", + path=f"{path}.reference_lists", + ) + reference_lists = [ + GoogleSecOpsReferenceList.from_dict(item, f"{path}.reference_lists[{idx}]") + for idx, item in enumerate(raw_ref_lists) + ] + + parser_extensions = _str_list(mapping, "parser_extensions", path) + return cls( + run_frequency=run_frequency, + data_rbac=data_rbac, + log_type_mappings=log_type_mappings, + detections=detections, + saved_searches=saved_searches, + reference_lists=reference_lists, + parser_extensions=parser_extensions, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize GoogleSecOpsPlatform to a plain dictionary.""" + result: dict[str, Any] = {} + if self.run_frequency is not None: + result["run_frequency"] = self.run_frequency.to_dict() + if self.data_rbac is not None: + result["data_rbac"] = self.data_rbac + result["log_type_mappings"] = dict(self.log_type_mappings) + result["detections"] = [det.to_dict() for det in self.detections] + result["saved_searches"] = [s.to_dict() for s in self.saved_searches] + result["reference_lists"] = [r.to_dict() for r in self.reference_lists] + if self.parser_extensions: + result["parser_extensions"] = list(self.parser_extensions) + return result + + +@dataclass +class DatabricksPlatform: + """Platform bindings for Databricks Unity Catalog OCSF Lakehouse.""" + + catalog: str = "security_lakehouse" + schema_name: str = "ocsf_silver" + table_mappings: dict[str, str] = field(default_factory=dict) + detections: list[str] = field(default_factory=list) + notes: str = "" + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "platforms.databricks" + ) -> DatabricksPlatform: + """Validate and construct DatabricksPlatform from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + {"catalog", "schema_name", "table_mappings", "detections", "notes"}, + path, + ) + catalog = ( + _optional_str(mapping, "catalog", path, default="security_lakehouse") + or "security_lakehouse" + ) + schema_name = ( + _optional_str(mapping, "schema_name", path, default="ocsf_silver") + or "ocsf_silver" + ) + table_mappings = _str_dict(mapping, "table_mappings", path) + detections = _str_list(mapping, "detections", path) + notes = _optional_str(mapping, "notes", path, default="") or "" + return cls( + catalog=catalog, + schema_name=schema_name, + table_mappings=table_mappings, + detections=detections, + notes=notes, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize DatabricksPlatform to a plain dictionary.""" + result: dict[str, Any] = { + "catalog": self.catalog, + "schema_name": self.schema_name, + "table_mappings": dict(self.table_mappings), + "detections": list(self.detections), + } + if self.notes: + result["notes"] = self.notes + return result + + +@dataclass +class SigmaPlatform: + """Platform bindings for Sigma vendor-neutral detection rules.""" + + rules: list[str] = field(default_factory=list) + notes: str = "" + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "platforms.sigma" + ) -> SigmaPlatform: + """Validate and construct SigmaPlatform from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys(mapping, {"rules", "notes"}, path) + rules = _str_list(mapping, "rules", path) + notes = _optional_str(mapping, "notes", path, default="") or "" + return cls(rules=rules, notes=notes) + + def to_dict(self) -> dict[str, Any]: + """Serialize SigmaPlatform to a plain dictionary.""" + result: dict[str, Any] = {"rules": list(self.rules)} + if self.notes: + result["notes"] = self.notes + return result + + +@dataclass +class SplunkPlatform: + """Platform bindings for Splunk sourcetypes and saved searches.""" + + sourcetypes: dict[str, str] = field(default_factory=dict) + saved_searches: list[str] = field(default_factory=list) + notes: str = "" + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "platforms.splunk" + ) -> SplunkPlatform: + """Validate and construct SplunkPlatform from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys(mapping, {"sourcetypes", "saved_searches", "notes"}, path) + sourcetypes = _str_dict(mapping, "sourcetypes", path) + saved_searches = _str_list(mapping, "saved_searches", path) + notes = _optional_str(mapping, "notes", path, default="") or "" + return cls( + sourcetypes=sourcetypes, + saved_searches=saved_searches, + notes=notes, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize SplunkPlatform to a plain dictionary.""" + result: dict[str, Any] = {"sourcetypes": dict(self.sourcetypes)} + if self.saved_searches: + result["saved_searches"] = list(self.saved_searches) + if self.notes: + result["notes"] = self.notes + return result + + +@dataclass +class PlatformBindings: + """Cross-vendor platform bindings for a security usecase package.""" + + google_secops: GoogleSecOpsPlatform | None = None + databricks: DatabricksPlatform | None = None + sigma: SigmaPlatform | None = None + splunk: SplunkPlatform | None = None + + @classmethod + def from_dict(cls, data: dict[str, Any], path: str = "platforms") -> PlatformBindings: + """Validate and construct PlatformBindings from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, {"google_secops", "databricks", "sigma", "splunk"}, path + ) + secops = ( + GoogleSecOpsPlatform.from_dict( + mapping["google_secops"], f"{path}.google_secops" + ) + if mapping.get("google_secops") is not None + else None + ) + databricks = ( + DatabricksPlatform.from_dict(mapping["databricks"], f"{path}.databricks") + if mapping.get("databricks") is not None + else None + ) + sigma = ( + SigmaPlatform.from_dict(mapping["sigma"], f"{path}.sigma") + if mapping.get("sigma") is not None + else None + ) + splunk = ( + SplunkPlatform.from_dict(mapping["splunk"], f"{path}.splunk") + if mapping.get("splunk") is not None + else None + ) + return cls( + google_secops=secops, + databricks=databricks, + sigma=sigma, + splunk=splunk, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize PlatformBindings to a plain dictionary.""" + result: dict[str, Any] = {} + if self.google_secops is not None: + result["google_secops"] = self.google_secops.to_dict() + if self.databricks is not None: + result["databricks"] = self.databricks.to_dict() + if self.sigma is not None: + result["sigma"] = self.sigma.to_dict() + if self.splunk is not None: + result["splunk"] = self.splunk.to_dict() + return result + + +@dataclass +class SecurityUsecaseManifest: + """Top-level v2 .md frontmatter manifest (OKF v0.2 + OCSF).""" + + schema_version: str + id: str + title: str + version: str + created: str + updated: str + author: str + license: str + description: str + tags: list[str] + environment: EnvironmentContext + telemetry: TelemetrySection + platforms: PlatformBindings + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "manifest" + ) -> SecurityUsecaseManifest: + """Validate and construct SecurityUsecaseManifest from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "schema_version", + "id", + "title", + "version", + "created", + "updated", + "author", + "license", + "description", + "tags", + "environment", + "telemetry", + "platforms", + }, + path, + ) + schema_version = _require_non_empty_str(mapping, "schema_version", path) + if schema_version != SCHEMA_VERSION_V2: + raise SchemaValidationError( + f"Unsupported schema_version {schema_version!r}; expected '2.0'", + path=f"{path}.schema_version", + ) + usecase_id = _require_non_empty_str(mapping, "id", path) + if not _USECASE_ID_RE.match(usecase_id): + raise SchemaValidationError( + f"Usecase id {usecase_id!r} must match {USECASE_ID_PATTERN}", + path=f"{path}.id", + ) + title = _require_non_empty_str(mapping, "title", path) + version = _require_non_empty_str(mapping, "version", path) + created = _require_non_empty_str(mapping, "created", path) + if not _ISO_DATE_RE.match(created): + raise SchemaValidationError( + f"Field 'created' {created!r} must match {ISO_DATE_PATTERN}", + path=f"{path}.created", + ) + updated = _require_non_empty_str(mapping, "updated", path) + if not _ISO_DATE_RE.match(updated): + raise SchemaValidationError( + f"Field 'updated' {updated!r} must match {ISO_DATE_PATTERN}", + path=f"{path}.updated", + ) + author = _require_non_empty_str(mapping, "author", path) + license_id = _require_non_empty_str(mapping, "license", path) + description = _require_non_empty_str(mapping, "description", path) + tags = _str_list(mapping, "tags", path, required=True) + if not tags: + raise SchemaValidationError( + "Field 'tags' must contain at least one tag", path=f"{path}.tags" + ) + + if "environment" not in mapping or mapping["environment"] is None: + raise SchemaValidationError( + "Missing required field 'environment'", path=f"{path}.environment" + ) + environment = EnvironmentContext.from_dict( + mapping["environment"], f"{path}.environment" + ) + + if "telemetry" not in mapping or mapping["telemetry"] is None: + raise SchemaValidationError( + "Missing required field 'telemetry'", path=f"{path}.telemetry" + ) + telemetry = TelemetrySection.from_dict(mapping["telemetry"], f"{path}.telemetry") + + if "platforms" not in mapping or mapping["platforms"] is None: + raise SchemaValidationError( + "Missing required field 'platforms'", path=f"{path}.platforms" + ) + platforms = PlatformBindings.from_dict(mapping["platforms"], f"{path}.platforms") + + return cls( + schema_version=schema_version, + id=usecase_id, + title=title, + version=version, + created=created, + updated=updated, + author=author, + license=license_id, + description=description, + tags=tags, + environment=environment, + telemetry=telemetry, + platforms=platforms, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize SecurityUsecaseManifest to a plain dictionary.""" + return { + "schema_version": self.schema_version, + "id": self.id, + "title": self.title, + "version": self.version, + "created": self.created, + "updated": self.updated, + "author": self.author, + "license": self.license, + "description": self.description, + "tags": list(self.tags), + "environment": self.environment.to_dict(), + "telemetry": self.telemetry.to_dict(), + "platforms": self.platforms.to_dict(), + } + + +@dataclass +class ThreatContext: + """Held-out adversary, advisory, CVE, and MITRE context for eval.yaml.""" + + grounding_tier: GroundingTier + difficulty_tier: DifficultyTier = DifficultyTier.INTERMEDIATE + threat_actor: str | None = None + actor_aliases: list[str] = field(default_factory=list) + campaign_name: str | None = None + cisa_advisories: list[str] = field(default_factory=list) + cve_ids: list[str] = field(default_factory=list) + mitre_tactics: list[str] = field(default_factory=list) + mitre_techniques: list[str] = field(default_factory=list) + release_day: int | None = None + cohort: str | None = None + notes: str = "" + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "threat_context" + ) -> ThreatContext: + """Validate and construct ThreatContext from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "grounding_tier", + "difficulty_tier", + "threat_actor", + "actor_aliases", + "campaign_name", + "cisa_advisories", + "cve_ids", + "mitre_tactics", + "mitre_techniques", + "release_day", + "cohort", + "notes", + }, + path, + ) + if "grounding_tier" not in mapping or mapping["grounding_tier"] is None: + raise SchemaValidationError( + "Missing required field 'grounding_tier'", + path=f"{path}.grounding_tier", + ) + grounding_tier = _parse_enum( + mapping["grounding_tier"], GroundingTier, f"{path}.grounding_tier" + ) + raw_diff = mapping.get("difficulty_tier", DifficultyTier.INTERMEDIATE.value) + difficulty_tier = _parse_enum(raw_diff, DifficultyTier, f"{path}.difficulty_tier") + return cls( + grounding_tier=grounding_tier, + difficulty_tier=difficulty_tier, + threat_actor=_optional_str(mapping, "threat_actor", path), + actor_aliases=_str_list(mapping, "actor_aliases", path), + campaign_name=_optional_str(mapping, "campaign_name", path), + cisa_advisories=_str_list(mapping, "cisa_advisories", path), + cve_ids=_str_list(mapping, "cve_ids", path), + mitre_tactics=_str_list(mapping, "mitre_tactics", path), + mitre_techniques=_str_list(mapping, "mitre_techniques", path), + release_day=_optional_int(mapping, "release_day", path), + cohort=_optional_str(mapping, "cohort", path), + notes=_optional_str(mapping, "notes", path, default="") or "", + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize ThreatContext to a plain dictionary.""" + result: dict[str, Any] = { + "grounding_tier": self.grounding_tier.value, + "difficulty_tier": self.difficulty_tier.value, + "threat_actor": self.threat_actor, + "actor_aliases": list(self.actor_aliases), + "campaign_name": self.campaign_name, + "cisa_advisories": list(self.cisa_advisories), + "cve_ids": list(self.cve_ids), + "mitre_tactics": list(self.mitre_tactics), + "mitre_techniques": list(self.mitre_techniques), + } + if self.release_day is not None: + result["release_day"] = self.release_day + if self.cohort is not None: + result["cohort"] = self.cohort + if self.notes: + result["notes"] = self.notes + return result + + +@dataclass +class AlertTriageTask: + """Alert triage evaluation task definition.""" + + enabled: bool = True + case_id: str | None = None + alert_ids: list[str] = field(default_factory=list) + prompt: str = "" + expected_skill: str | None = None + skill_bindings: dict[str, str] = field(default_factory=dict) + time_horizon_hours: int = DEFAULT_TRIAGE_HORIZON_HOURS + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "tasks.alert_triage" + ) -> AlertTriageTask: + """Validate and construct AlertTriageTask from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "enabled", + "case_id", + "alert_ids", + "prompt", + "expected_skill", + "skill_bindings", + "time_horizon_hours", + }, + path, + ) + horizon = _optional_int( + mapping, + "time_horizon_hours", + path, + default=DEFAULT_TRIAGE_HORIZON_HOURS, + ) + return cls( + enabled=_optional_bool(mapping, "enabled", path, default=True), + case_id=_optional_str(mapping, "case_id", path), + alert_ids=_str_list(mapping, "alert_ids", path), + prompt=_optional_str(mapping, "prompt", path, default="") or "", + expected_skill=_optional_str(mapping, "expected_skill", path), + skill_bindings=_str_dict(mapping, "skill_bindings", path), + time_horizon_hours=( + DEFAULT_TRIAGE_HORIZON_HOURS if horizon is None else horizon + ), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize AlertTriageTask to a plain dictionary.""" + return { + "enabled": self.enabled, + "case_id": self.case_id, + "alert_ids": list(self.alert_ids), + "prompt": self.prompt, + "expected_skill": self.expected_skill, + "skill_bindings": dict(self.skill_bindings), + "time_horizon_hours": self.time_horizon_hours, + } + + +@dataclass +class ThreatHuntTask: + """Unalerted and intelligence-driven threat hunt evaluation task.""" + + enabled: bool = True + prompt_unalerted: str = "" + prompt_intel_driven: str | None = None + expected_patient_zero: str | None = None + expected_skill: str | None = None + skill_bindings: dict[str, str] = field(default_factory=dict) + time_horizon_hours: int = DEFAULT_HUNT_HORIZON_HOURS + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "tasks.threat_hunt" + ) -> ThreatHuntTask: + """Validate and construct ThreatHuntTask from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "enabled", + "prompt_unalerted", + "prompt_intel_driven", + "expected_patient_zero", + "expected_skill", + "skill_bindings", + "time_horizon_hours", + }, + path, + ) + horizon = _optional_int( + mapping, + "time_horizon_hours", + path, + default=DEFAULT_HUNT_HORIZON_HOURS, + ) + return cls( + enabled=_optional_bool(mapping, "enabled", path, default=True), + prompt_unalerted=( + _optional_str(mapping, "prompt_unalerted", path, default="") or "" + ), + prompt_intel_driven=_optional_str(mapping, "prompt_intel_driven", path), + expected_patient_zero=_optional_str(mapping, "expected_patient_zero", path), + expected_skill=_optional_str(mapping, "expected_skill", path), + skill_bindings=_str_dict(mapping, "skill_bindings", path), + time_horizon_hours=(DEFAULT_HUNT_HORIZON_HOURS if horizon is None else horizon), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize ThreatHuntTask to a plain dictionary.""" + result: dict[str, Any] = { + "enabled": self.enabled, + "prompt_unalerted": self.prompt_unalerted, + "prompt_intel_driven": self.prompt_intel_driven, + "expected_patient_zero": self.expected_patient_zero, + "time_horizon_hours": self.time_horizon_hours, + } + if self.expected_skill is not None: + result["expected_skill"] = self.expected_skill + if self.skill_bindings: + result["skill_bindings"] = dict(self.skill_bindings) + return result + + +@dataclass +class DetectionEngineeringTask: + """Detection rule authoring and false-positive evaluation task.""" + + enabled: bool = True + prompt: str = "" + target_techniques: list[str] = field(default_factory=list) + reference_rules: list[str] = field(default_factory=list) + expected_rule_count_min: int = 1 + max_false_positives_on_paired_baseline: int = 0 + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "tasks.detection_engineering" + ) -> DetectionEngineeringTask: + """Validate and construct DetectionEngineeringTask from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "enabled", + "prompt", + "target_techniques", + "reference_rules", + "expected_rule_count_min", + "max_false_positives_on_paired_baseline", + }, + path, + ) + min_rules = _optional_int(mapping, "expected_rule_count_min", path, default=1) + max_fp = _optional_int( + mapping, "max_false_positives_on_paired_baseline", path, default=0 + ) + return cls( + enabled=_optional_bool(mapping, "enabled", path, default=True), + prompt=_optional_str(mapping, "prompt", path, default="") or "", + target_techniques=_str_list(mapping, "target_techniques", path), + reference_rules=_str_list(mapping, "reference_rules", path), + expected_rule_count_min=1 if min_rules is None else min_rules, + max_false_positives_on_paired_baseline=(0 if max_fp is None else max_fp), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize DetectionEngineeringTask to a plain dictionary.""" + return { + "enabled": self.enabled, + "prompt": self.prompt, + "target_techniques": list(self.target_techniques), + "reference_rules": list(self.reference_rules), + "expected_rule_count_min": self.expected_rule_count_min, + "max_false_positives_on_paired_baseline": ( + self.max_false_positives_on_paired_baseline + ), + } + + +@dataclass +class EvaluationTasks: + """Multi-task evaluation definitions for Blue Team benchmarking.""" + + alert_triage: AlertTriageTask | None = None + threat_hunt: ThreatHuntTask | None = None + detection_engineering: DetectionEngineeringTask | None = None + + @classmethod + def from_dict(cls, data: dict[str, Any], path: str = "tasks") -> EvaluationTasks: + """Validate and construct EvaluationTasks from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + {"alert_triage", "threat_hunt", "detection_engineering"}, + path, + ) + triage = ( + AlertTriageTask.from_dict(mapping["alert_triage"], f"{path}.alert_triage") + if mapping.get("alert_triage") is not None + else None + ) + hunt = ( + ThreatHuntTask.from_dict(mapping["threat_hunt"], f"{path}.threat_hunt") + if mapping.get("threat_hunt") is not None + else None + ) + det_eng = ( + DetectionEngineeringTask.from_dict( + mapping["detection_engineering"], f"{path}.detection_engineering" + ) + if mapping.get("detection_engineering") is not None + else None + ) + return cls( + alert_triage=triage, + threat_hunt=hunt, + detection_engineering=det_eng, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize EvaluationTasks to a plain dictionary.""" + result: dict[str, Any] = {} + if self.alert_triage is not None: + result["alert_triage"] = self.alert_triage.to_dict() + if self.threat_hunt is not None: + result["threat_hunt"] = self.threat_hunt.to_dict() + if self.detection_engineering is not None: + result["detection_engineering"] = self.detection_engineering.to_dict() + return result + + +@dataclass +class KillChainStage: + """Single chronological stage in an attack scenario's kill chain.""" + + stage: str + stream_id: str + timestamp_utc: str | None = None + tactic: str | None = None + technique_id: str | None = None + summary: str = "" + key_evidence: list[str] = field(default_factory=list) + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "ground_truth.kill_chain" + ) -> KillChainStage: + """Validate and construct KillChainStage from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "stage", + "stream_id", + "timestamp_utc", + "tactic", + "technique_id", + "summary", + "key_evidence", + }, + path, + ) + stage = _require_non_empty_str(mapping, "stage", path) + stream_id = _require_non_empty_str(mapping, "stream_id", path) + return cls( + stage=stage, + stream_id=stream_id, + timestamp_utc=_optional_str(mapping, "timestamp_utc", path), + tactic=_optional_str(mapping, "tactic", path), + technique_id=_optional_str(mapping, "technique_id", path), + summary=_optional_str(mapping, "summary", path, default="") or "", + key_evidence=_str_list(mapping, "key_evidence", path), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize KillChainStage to a plain dictionary.""" + return { + "stage": self.stage, + "stream_id": self.stream_id, + "timestamp_utc": self.timestamp_utc, + "tactic": self.tactic, + "technique_id": self.technique_id, + "summary": self.summary, + "key_evidence": list(self.key_evidence), + } + + +@dataclass +class AffectedAsset: + """Ground-truth affected asset with criticality tier and blast-radius costs.""" + + hostname: str + criticality_tier: CriticalityTier = CriticalityTier.TIER_2_WORKSTATION + ip_addresses: list[str] = field(default_factory=list) + role: str = "" + patient_zero: bool = False + blast_radius_fp_usd: float = 0.0 + blast_radius_fn_usd: float = 0.0 + + @classmethod + def from_dict( + cls, + data: dict[str, Any], + path: str = "ground_truth.entities_and_iocs.affected_assets", + ) -> AffectedAsset: + """Validate and construct AffectedAsset from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "hostname", + "criticality_tier", + "ip_addresses", + "role", + "patient_zero", + "blast_radius_fp_usd", + "blast_radius_fn_usd", + }, + path, + ) + hostname = _require_non_empty_str(mapping, "hostname", path) + raw_tier = mapping.get("criticality_tier", CriticalityTier.TIER_2_WORKSTATION.value) + criticality_tier = _parse_enum( + raw_tier, CriticalityTier, f"{path}.criticality_tier" + ) + return cls( + hostname=hostname, + criticality_tier=criticality_tier, + ip_addresses=_str_list(mapping, "ip_addresses", path), + role=_optional_str(mapping, "role", path, default="") or "", + patient_zero=_optional_bool(mapping, "patient_zero", path, default=False), + blast_radius_fp_usd=_optional_float( + mapping, "blast_radius_fp_usd", path, default=0.0 + ), + blast_radius_fn_usd=_optional_float( + mapping, "blast_radius_fn_usd", path, default=0.0 + ), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize AffectedAsset to a plain dictionary.""" + return { + "hostname": self.hostname, + "criticality_tier": self.criticality_tier.value, + "ip_addresses": list(self.ip_addresses), + "role": self.role, + "patient_zero": self.patient_zero, + "blast_radius_fp_usd": self.blast_radius_fp_usd, + "blast_radius_fn_usd": self.blast_radius_fn_usd, + } + + +@dataclass +class MaliciousIoc: + """Ground-truth malicious indicator of compromise.""" + + type: str + value: str + role: str = "" + notes: str = "" + + @classmethod + def from_dict( + cls, + data: dict[str, Any], + path: str = "ground_truth.entities_and_iocs.malicious_iocs", + ) -> MaliciousIoc: + """Validate and construct MaliciousIoc from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys(mapping, {"type", "value", "role", "notes"}, path) + ioc_type = _require_non_empty_str(mapping, "type", path) + value = _require_non_empty_str(mapping, "value", path) + role = _optional_str(mapping, "role", path, default="") or "" + notes = _optional_str(mapping, "notes", path, default="") or "" + return cls(type=ioc_type, value=value, role=role, notes=notes) + + def to_dict(self) -> dict[str, Any]: + """Serialize MaliciousIoc to a plain dictionary.""" + return { + "type": self.type, + "value": self.value, + "role": self.role, + "notes": self.notes, + } + + +@dataclass +class ChangeWindow: + """Maintenance or change management ticket context.""" + + ticket_id: str + approved: bool = True + start_utc: str | None = None + end_utc: str | None = None + summary: str = "" + + @classmethod + def from_dict( + cls, + data: dict[str, Any], + path: str = "ground_truth.entities_and_iocs.benign_confusers.change_window", + ) -> ChangeWindow: + """Validate and construct ChangeWindow from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + {"ticket_id", "approved", "start_utc", "end_utc", "summary"}, + path, + ) + ticket_id = _require_non_empty_str(mapping, "ticket_id", path) + return cls( + ticket_id=ticket_id, + approved=_optional_bool(mapping, "approved", path, default=True), + start_utc=_optional_str(mapping, "start_utc", path), + end_utc=_optional_str(mapping, "end_utc", path), + summary=_optional_str(mapping, "summary", path, default="") or "", + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize ChangeWindow to a plain dictionary.""" + return { + "ticket_id": self.ticket_id, + "approved": self.approved, + "start_utc": self.start_utc, + "end_utc": self.end_utc, + "summary": self.summary, + } + + +@dataclass +class BenignConfusers: + """Benign confusers (signed LOLBins, internal IPs, change windows).""" + + signed_lolbins: list[str] = field(default_factory=list) + internal_ips: list[str] = field(default_factory=list) + change_window: ChangeWindow | None = None + admin_tools: list[str] = field(default_factory=list) + + @classmethod + def from_dict( + cls, + data: dict[str, Any], + path: str = "ground_truth.entities_and_iocs.benign_confusers", + ) -> BenignConfusers: + """Validate and construct BenignConfusers from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + {"signed_lolbins", "internal_ips", "change_window", "admin_tools"}, + path, + ) + raw_cw = mapping.get("change_window") + change_window = ( + ChangeWindow.from_dict(raw_cw, f"{path}.change_window") + if raw_cw is not None + else None + ) + return cls( + signed_lolbins=_str_list(mapping, "signed_lolbins", path), + internal_ips=_str_list(mapping, "internal_ips", path), + change_window=change_window, + admin_tools=_str_list(mapping, "admin_tools", path), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize BenignConfusers to a plain dictionary.""" + result: dict[str, Any] = { + "signed_lolbins": list(self.signed_lolbins), + "internal_ips": list(self.internal_ips), + "change_window": ( + self.change_window.to_dict() if self.change_window is not None else None + ), + } + if self.admin_tools: + result["admin_tools"] = list(self.admin_tools) + return result + + +@dataclass +class EntitiesAndIocs: + """Ground-truth entities, IOCs, and benign confusers for an evaluation.""" + + compromised_accounts: list[str] = field(default_factory=list) + affected_assets: list[AffectedAsset] = field(default_factory=list) + malicious_iocs: list[MaliciousIoc] = field(default_factory=list) + benign_confusers: BenignConfusers = field(default_factory=BenignConfusers) + + @classmethod + def from_dict( + cls, + data: dict[str, Any], + path: str = "ground_truth.entities_and_iocs", + ) -> EntitiesAndIocs: + """Validate and construct EntitiesAndIocs from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "compromised_accounts", + "affected_assets", + "malicious_iocs", + "benign_confusers", + }, + path, + ) + compromised_accounts = _str_list(mapping, "compromised_accounts", path) + + raw_assets = mapping.get("affected_assets", []) or [] + if not isinstance(raw_assets, list): + raise SchemaValidationError( + "Field 'affected_assets' must be a list", + path=f"{path}.affected_assets", + ) + affected_assets = [ + AffectedAsset.from_dict(item, f"{path}.affected_assets[{idx}]") + for idx, item in enumerate(raw_assets) + ] + + raw_iocs = mapping.get("malicious_iocs", []) or [] + if not isinstance(raw_iocs, list): + raise SchemaValidationError( + "Field 'malicious_iocs' must be a list", + path=f"{path}.malicious_iocs", + ) + malicious_iocs = [ + MaliciousIoc.from_dict(item, f"{path}.malicious_iocs[{idx}]") + for idx, item in enumerate(raw_iocs) + ] + + raw_confusers = mapping.get("benign_confusers") + benign_confusers = ( + BenignConfusers.from_dict(raw_confusers, f"{path}.benign_confusers") + if raw_confusers is not None + else BenignConfusers() + ) + + return cls( + compromised_accounts=compromised_accounts, + affected_assets=affected_assets, + malicious_iocs=malicious_iocs, + benign_confusers=benign_confusers, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize EntitiesAndIocs to a plain dictionary.""" + return { + "compromised_accounts": list(self.compromised_accounts), + "affected_assets": [a.to_dict() for a in self.affected_assets], + "malicious_iocs": [ioc.to_dict() for ioc in self.malicious_iocs], + "benign_confusers": self.benign_confusers.to_dict(), + } + + +@dataclass +class GroundTruth: + """Held-out ground-truth verdict, disposition, kill chain, and entities.""" + + expected_disposition: ExpectedDisposition + expected_verdict: ExpectedVerdict + severity: str = "HIGH" + patient_zero_host: str | None = None + patient_zero_account: str | None = None + root_cause_summary: str = "" + kill_chain: list[KillChainStage] = field(default_factory=list) + entities_and_iocs: EntitiesAndIocs = field(default_factory=EntitiesAndIocs) + + @classmethod + def from_dict(cls, data: dict[str, Any], path: str = "ground_truth") -> GroundTruth: + """Validate and construct GroundTruth from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "expected_disposition", + "expected_verdict", + "severity", + "patient_zero_host", + "patient_zero_account", + "root_cause_summary", + "kill_chain", + "entities_and_iocs", + }, + path, + ) + if "expected_disposition" not in mapping or mapping["expected_disposition"] is None: + raise SchemaValidationError( + "Missing required field 'expected_disposition'", + path=f"{path}.expected_disposition", + ) + expected_disposition = _parse_enum( + mapping["expected_disposition"], + ExpectedDisposition, + f"{path}.expected_disposition", + ) + + if "expected_verdict" not in mapping or mapping["expected_verdict"] is None: + raise SchemaValidationError( + "Missing required field 'expected_verdict'", + path=f"{path}.expected_verdict", + ) + expected_verdict = _parse_enum( + mapping["expected_verdict"], + ExpectedVerdict, + f"{path}.expected_verdict", + ) + + severity = _optional_str(mapping, "severity", path, default="HIGH") or "HIGH" + patient_zero_host = _optional_str(mapping, "patient_zero_host", path) + patient_zero_account = _optional_str(mapping, "patient_zero_account", path) + root_cause_summary = ( + _optional_str(mapping, "root_cause_summary", path, default="") or "" + ) + + raw_kc = mapping.get("kill_chain", []) or [] + if not isinstance(raw_kc, list): + raise SchemaValidationError( + "Field 'kill_chain' must be a list", path=f"{path}.kill_chain" + ) + kill_chain = [ + KillChainStage.from_dict(item, f"{path}.kill_chain[{idx}]") + for idx, item in enumerate(raw_kc) + ] + + raw_entities = mapping.get("entities_and_iocs") + entities_and_iocs = ( + EntitiesAndIocs.from_dict(raw_entities, f"{path}.entities_and_iocs") + if raw_entities is not None + else EntitiesAndIocs() + ) + + return cls( + expected_disposition=expected_disposition, + expected_verdict=expected_verdict, + severity=severity, + patient_zero_host=patient_zero_host, + patient_zero_account=patient_zero_account, + root_cause_summary=root_cause_summary, + kill_chain=kill_chain, + entities_and_iocs=entities_and_iocs, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize GroundTruth to a plain dictionary.""" + return { + "expected_disposition": self.expected_disposition.value, + "expected_verdict": self.expected_verdict.value, + "severity": self.severity, + "patient_zero_host": self.patient_zero_host, + "patient_zero_account": self.patient_zero_account, + "root_cause_summary": self.root_cause_summary, + "kill_chain": [stage.to_dict() for stage in self.kill_chain], + "entities_and_iocs": self.entities_and_iocs.to_dict(), + } + + +@dataclass +class ExpectedActions: + """Primary, secondary, and prohibited containment/response actions.""" + + primary: list[str] = field(default_factory=list) + secondary: list[str] = field(default_factory=list) + prohibited: list[str] = field(default_factory=list) + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "compliance_gates.expected_actions" + ) -> ExpectedActions: + """Validate and construct ExpectedActions from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys(mapping, {"primary", "secondary", "prohibited"}, path) + return cls( + primary=_str_list(mapping, "primary", path), + secondary=_str_list(mapping, "secondary", path), + prohibited=_str_list(mapping, "prohibited", path), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize ExpectedActions to a plain dictionary.""" + return { + "primary": list(self.primary), + "secondary": list(self.secondary), + "prohibited": list(self.prohibited), + } + + +@dataclass +class ComplianceGates: + """Operational safety and SOP compliance gates for agent evaluation.""" + + requires_pre_containment_verification: bool = True + requires_168h_horizon_widening: bool = False + forbid_external_cti_on_internal_ips: bool = True + forbid_lolbin_hash_blocking: bool = True + respect_change_window: bool = False + expected_actions: ExpectedActions = field(default_factory=ExpectedActions) + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "compliance_gates" + ) -> ComplianceGates: + """Validate and construct ComplianceGates from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "requires_pre_containment_verification", + "requires_168h_horizon_widening", + "forbid_external_cti_on_internal_ips", + "forbid_lolbin_hash_blocking", + "respect_change_window", + "expected_actions", + }, + path, + ) + raw_actions = mapping.get("expected_actions") + expected_actions = ( + ExpectedActions.from_dict(raw_actions, f"{path}.expected_actions") + if raw_actions is not None + else ExpectedActions() + ) + return cls( + requires_pre_containment_verification=_optional_bool( + mapping, + "requires_pre_containment_verification", + path, + default=True, + ), + requires_168h_horizon_widening=_optional_bool( + mapping, "requires_168h_horizon_widening", path, default=False + ), + forbid_external_cti_on_internal_ips=_optional_bool( + mapping, "forbid_external_cti_on_internal_ips", path, default=True + ), + forbid_lolbin_hash_blocking=_optional_bool( + mapping, "forbid_lolbin_hash_blocking", path, default=True + ), + respect_change_window=_optional_bool( + mapping, "respect_change_window", path, default=False + ), + expected_actions=expected_actions, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize ComplianceGates to a plain dictionary.""" + return { + "requires_pre_containment_verification": ( + self.requires_pre_containment_verification + ), + "requires_168h_horizon_widening": self.requires_168h_horizon_widening, + "forbid_external_cti_on_internal_ips": self.forbid_external_cti_on_internal_ips, + "forbid_lolbin_hash_blocking": self.forbid_lolbin_hash_blocking, + "respect_change_window": self.respect_change_window, + "expected_actions": self.expected_actions.to_dict(), + } + + +@dataclass +class CounterfactualPair: + """Signal Detection Theory counterfactual pairing between twin usecases.""" + + paired_usecase_id: str + pair_relationship: PairRelationship + shared_entities: list[str] = field(default_factory=list) + shared_telemetry_streams: list[str] = field(default_factory=list) + discriminating_invariant: str = "" + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "counterfactual_pair" + ) -> CounterfactualPair: + """Validate and construct CounterfactualPair from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "paired_usecase_id", + "pair_relationship", + "shared_entities", + "shared_telemetry_streams", + "discriminating_invariant", + }, + path, + ) + paired_id = _require_non_empty_str(mapping, "paired_usecase_id", path) + if "pair_relationship" not in mapping or mapping["pair_relationship"] is None: + raise SchemaValidationError( + "Missing required field 'pair_relationship'", + path=f"{path}.pair_relationship", + ) + relationship = _parse_enum( + mapping["pair_relationship"], + PairRelationship, + f"{path}.pair_relationship", + ) + return cls( + paired_usecase_id=paired_id, + pair_relationship=relationship, + shared_entities=_str_list(mapping, "shared_entities", path), + shared_telemetry_streams=_str_list(mapping, "shared_telemetry_streams", path), + discriminating_invariant=( + _optional_str(mapping, "discriminating_invariant", path, default="") or "" + ), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize CounterfactualPair to a plain dictionary.""" + return { + "paired_usecase_id": self.paired_usecase_id, + "pair_relationship": self.pair_relationship.value, + "shared_entities": list(self.shared_entities), + "shared_telemetry_streams": list(self.shared_telemetry_streams), + "discriminating_invariant": self.discriminating_invariant, + } + + +@dataclass +class ReferenceQueries: + """Held-out reference queries and rules across supported platforms.""" + + google_secops_udm: list[str] = field(default_factory=list) + google_secops_yaral: list[str] = field(default_factory=list) + databricks_sql: list[str] = field(default_factory=list) + sigma: list[str] = field(default_factory=list) + splunk_spl: list[str] = field(default_factory=list) + + @classmethod + def from_dict( + cls, data: dict[str, Any], path: str = "reference_queries" + ) -> ReferenceQueries: + """Validate and construct ReferenceQueries from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "google_secops_udm", + "google_secops_yaral", + "databricks_sql", + "sigma", + "splunk_spl", + }, + path, + ) + return cls( + google_secops_udm=_str_list(mapping, "google_secops_udm", path), + google_secops_yaral=_str_list(mapping, "google_secops_yaral", path), + databricks_sql=_str_list(mapping, "databricks_sql", path), + sigma=_str_list(mapping, "sigma", path), + splunk_spl=_str_list(mapping, "splunk_spl", path), + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize ReferenceQueries to a plain dictionary.""" + return { + "google_secops_udm": list(self.google_secops_udm), + "google_secops_yaral": list(self.google_secops_yaral), + "databricks_sql": list(self.databricks_sql), + "sigma": list(self.sigma), + "splunk_spl": list(self.splunk_spl), + } + + +@dataclass +class SecurityUsecaseEval: + """Top-level v2 eval.yaml held-out Blue Team benchmark specification.""" + + schema_version: str + usecase_id: str + threat_context: ThreatContext + tasks: EvaluationTasks + ground_truth: GroundTruth + compliance_gates: ComplianceGates = field(default_factory=ComplianceGates) + counterfactual_pair: CounterfactualPair | None = None + reference_queries: ReferenceQueries = field(default_factory=ReferenceQueries) + + @classmethod + def from_dict(cls, data: dict[str, Any], path: str = "eval") -> SecurityUsecaseEval: + """Validate and construct SecurityUsecaseEval from a dictionary.""" + mapping = _ensure_mapping(data, path) + _reject_unknown_keys( + mapping, + { + "schema_version", + "usecase_id", + "threat_context", + "tasks", + "ground_truth", + "compliance_gates", + "counterfactual_pair", + "reference_queries", + }, + path, + ) + schema_version = _require_non_empty_str(mapping, "schema_version", path) + if schema_version != SCHEMA_VERSION_V2: + raise SchemaValidationError( + f"Unsupported eval schema_version {schema_version!r}; expected '2.0'", + path=f"{path}.schema_version", + ) + usecase_id = _require_non_empty_str(mapping, "usecase_id", path) + if not _USECASE_ID_RE.match(usecase_id): + raise SchemaValidationError( + f"usecase_id {usecase_id!r} must match {USECASE_ID_PATTERN}", + path=f"{path}.usecase_id", + ) + + if "threat_context" not in mapping or mapping["threat_context"] is None: + raise SchemaValidationError( + "Missing required field 'threat_context'", + path=f"{path}.threat_context", + ) + threat_context = ThreatContext.from_dict( + mapping["threat_context"], f"{path}.threat_context" + ) + + if "tasks" not in mapping or mapping["tasks"] is None: + raise SchemaValidationError( + "Missing required field 'tasks'", path=f"{path}.tasks" + ) + tasks = EvaluationTasks.from_dict(mapping["tasks"], f"{path}.tasks") + + if "ground_truth" not in mapping or mapping["ground_truth"] is None: + raise SchemaValidationError( + "Missing required field 'ground_truth'", path=f"{path}.ground_truth" + ) + ground_truth = GroundTruth.from_dict( + mapping["ground_truth"], f"{path}.ground_truth" + ) + + raw_gates = mapping.get("compliance_gates") + compliance_gates = ( + ComplianceGates.from_dict(raw_gates, f"{path}.compliance_gates") + if raw_gates is not None + else ComplianceGates() + ) + + raw_pair = mapping.get("counterfactual_pair") + counterfactual_pair = ( + CounterfactualPair.from_dict(raw_pair, f"{path}.counterfactual_pair") + if raw_pair is not None + else None + ) + + raw_queries = mapping.get("reference_queries") + reference_queries = ( + ReferenceQueries.from_dict(raw_queries, f"{path}.reference_queries") + if raw_queries is not None + else ReferenceQueries() + ) + + return cls( + schema_version=schema_version, + usecase_id=usecase_id, + threat_context=threat_context, + tasks=tasks, + ground_truth=ground_truth, + compliance_gates=compliance_gates, + counterfactual_pair=counterfactual_pair, + reference_queries=reference_queries, + ) + + def to_dict(self) -> dict[str, Any]: + """Serialize SecurityUsecaseEval to a plain dictionary.""" + return { + "schema_version": self.schema_version, + "usecase_id": self.usecase_id, + "threat_context": self.threat_context.to_dict(), + "tasks": self.tasks.to_dict(), + "ground_truth": self.ground_truth.to_dict(), + "compliance_gates": self.compliance_gates.to_dict(), + "counterfactual_pair": ( + self.counterfactual_pair.to_dict() + if self.counterfactual_pair is not None + else None + ), + "reference_queries": self.reference_queries.to_dict(), + } + + +@dataclass +class ParsedUsecasePackage: + """Parsed representation of a usecase directory on disk.""" + + usecase_dir: Path + schema_version: str + manifest: SecurityUsecaseManifest | None + raw_frontmatter: dict[str, Any] + public_markdown_body: str + solution_markdown_body: str | None + eval_data: SecurityUsecaseEval | None + + def to_dict(self, blind: bool = False) -> dict[str, Any]: + """Serialize ParsedUsecasePackage to a JSON-compatible dictionary.""" + return { + "usecase_dir": str(self.usecase_dir), + "schema_version": self.schema_version, + "manifest": ( + self.manifest.to_dict() + if self.manifest is not None + else dict(self.raw_frontmatter) + ), + "public_markdown_body": self.public_markdown_body, + "solution_markdown_body": None if blind else self.solution_markdown_body, + "eval_data": ( + None if blind or self.eval_data is None else self.eval_data.to_dict() + ), + } + + +def split_frontmatter_and_body(content: str) -> tuple[dict[str, Any], str]: + """Split a Markdown file into YAML frontmatter dict and Markdown body. + + Args: + content: Raw text content of `.md`. + + Returns: + Tuple of `(frontmatter_dict, markdown_body)`. + + Raises: + SchemaValidationError: If frontmatter delimiters or YAML syntax are invalid. + """ + normalized = content.replace("\r\n", "\n") + if normalized.startswith("\n---\n"): + normalized = normalized[1:] + if not normalized.startswith("---\n"): + raise SchemaValidationError( + "Markdown file is missing leading YAML frontmatter delimiter '---'", + path="frontmatter", + ) + + parts = normalized.split("---\n", 2) + if len(parts) < MIN_FRONTMATTER_PARTS: + raise SchemaValidationError( + "Markdown file is missing closing YAML frontmatter delimiter '---'", + path="frontmatter", + ) + + frontmatter_yaml = parts[1] + body = parts[2] + try: + loaded = yaml.safe_load(frontmatter_yaml) + except yaml.YAMLError as exc: + raise SchemaValidationError( + f"Invalid YAML syntax in frontmatter: {exc}", path="frontmatter" + ) from exc + + if not isinstance(loaded, dict): + raise SchemaValidationError( + "YAML frontmatter must be a key-value mapping", path="frontmatter" + ) + return _normalize_yaml_scalars(loaded), body + + +def split_solution_section(markdown_body: str) -> tuple[str, str | None]: + """Split Markdown body on ``. + + Args: + markdown_body: Markdown text following the YAML frontmatter. + + Returns: + Tuple of `(public_markdown_body, solution_markdown_body)`. When the + delimiter is absent, `solution_markdown_body` is `None`. + """ + if SOLUTION_DELIMITER not in markdown_body: + return markdown_body.strip(), None + public_part, solution_part = markdown_body.split(SOLUTION_DELIMITER, 1) + return public_part.strip(), solution_part.strip() + + +def parse_usecase_package( + usecase_dir: Path, blind: bool = False +) -> ParsedUsecasePackage: + """Parse `/.md` and optional `eval.yaml` from a directory. + + Args: + usecase_dir: Path to the `` directory. + blind: When True, omits content below `` and does + not load `eval.yaml`. + + Returns: + Populated `ParsedUsecasePackage`. + """ + resolved_dir = Path(usecase_dir) + usecase_name = resolved_dir.name + md_path = resolved_dir / f"{usecase_name}.md" + if not md_path.is_file(): + raise SchemaValidationError( + f"Missing required usecase manifest file: {md_path}", + path=str(md_path), + ) + + raw_text = md_path.read_text(encoding="utf-8") + raw_frontmatter, markdown_body = split_frontmatter_and_body(raw_text) + public_body, solution_body = split_solution_section(markdown_body) + + schema_version = str(raw_frontmatter.get("schema_version", SCHEMA_VERSION_V1)) + manifest: SecurityUsecaseManifest | None = None + if schema_version == SCHEMA_VERSION_V2: + manifest = SecurityUsecaseManifest.from_dict(raw_frontmatter) + + eval_data: SecurityUsecaseEval | None = None + if not blind: + eval_path = resolved_dir / "eval.yaml" + if eval_path.is_file(): + try: + raw_eval = yaml.safe_load(eval_path.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise SchemaValidationError( + f"Invalid YAML syntax in eval.yaml: {exc}", path="eval.yaml" + ) from exc + eval_data = SecurityUsecaseEval.from_dict( + _normalize_yaml_scalars(raw_eval), path="eval" + ) + + return ParsedUsecasePackage( + usecase_dir=resolved_dir, + schema_version=schema_version, + manifest=manifest, + raw_frontmatter=raw_frontmatter, + public_markdown_body=public_body, + solution_markdown_body=None if blind else solution_body, + eval_data=None if blind else eval_data, + ) + + +def build_usecase_json_schema() -> dict[str, Any]: + """Build Draft 2020-12 JSON Schema for `.md` frontmatter.""" + return { + "$schema": JSON_SCHEMA_DRAFT_2020_12, + "$id": "https://chronicle.github.io/logstory/schemas/usecase.schema.json", + "title": "SecurityUsecaseManifest", + "description": ( + "Open Security Scenario & Logstory Usecase Manifest Schema (v2.0) " + "combining OKF v0.2 provenance, OCSF telemetry streams, and " + "multi-platform bindings." + ), + "type": "object", + "additionalProperties": False, + "required": [ + "schema_version", + "id", + "title", + "version", + "created", + "updated", + "author", + "license", + "description", + "tags", + "environment", + "telemetry", + "platforms", + ], + "properties": { + "schema_version": { + "type": "string", + "const": SCHEMA_VERSION_V2, + "description": "Usecase manifest schema version ('2.0').", + }, + "id": { + "type": "string", + "pattern": USECASE_ID_PATTERN, + "description": "Canonical uppercase usecase identifier.", + }, + "title": { + "type": "string", + "minLength": 1, + "description": "Human-readable blind-safe package title.", + }, + "version": { + "type": "string", + "minLength": 1, + "description": "Semantic version of the usecase package.", + }, + "created": { + "type": "string", + "pattern": ISO_DATE_PATTERN, + "description": "Initial authoring date (YYYY-MM-DD).", + }, + "updated": { + "type": "string", + "pattern": ISO_DATE_PATTERN, + "description": "Last modification date (YYYY-MM-DD).", + }, + "author": { + "type": "string", + "minLength": 1, + "description": "Package author or organization.", + }, + "license": { + "type": "string", + "minLength": 1, + "description": "SPDX license identifier (e.g., Apache-2.0).", + }, + "description": { + "type": "string", + "minLength": 1, + "description": ( + "Blind-safe summary of the telemetry streams and scenario " + "context (no spoilers)." + ), + }, + "tags": { + "type": "array", + "minItems": 1, + "items": {"type": "string"}, + "description": "Discovery tags for the usecase package.", + }, + "environment": {"$ref": "#/$defs/EnvironmentContext"}, + "telemetry": {"$ref": "#/$defs/TelemetrySection"}, + "platforms": {"$ref": "#/$defs/PlatformBindings"}, + }, + "$defs": { + "TimestampPolicy": { + "type": "object", + "additionalProperties": False, + "description": "Replay timestamp adjustment policy.", + "properties": { + "base_date": { + "type": "string", + "description": "Base date of synthetic logs (YYYY-MM-DD).", + }, + "preserve_relative_offsets": { + "type": "boolean", + "description": ( + "Whether relative inter-event offsets are preserved." + ), + }, + "default_delta": { + "type": "string", + "description": "Default timestamp shift expression.", + }, + "entity_offset_days": { + "type": "integer", + "description": ( + "Days prior to events to replay entity context logs." + ), + }, + "window_start": { + "type": ["string", "null"], + "description": "Optional UTC window start timestamp.", + }, + "window_end": { + "type": ["string", "null"], + "description": "Optional UTC window end timestamp.", + }, + }, + }, + "EnvironmentContext": { + "type": "object", + "additionalProperties": False, + "description": ( + "Simulated organization, scenario classification, and " + "composable paired baselines." + ), + "properties": { + "organization": { + "type": "string", + "description": "Simulated enterprise domain name.", + }, + "scenario_type": { + "type": "string", + "enum": [e.value for e in ScenarioType], + "description": "High-level scenario classification.", + }, + "paired_baselines": { + "type": "array", + "items": {"type": "string"}, + "description": ( + "Composable benign baseline usecase IDs to co-replay." + ), + }, + "timestamp_policy": {"$ref": "#/$defs/TimestampPolicy"}, + "notes": { + "type": "string", + "description": "Optional environment notes.", + }, + }, + }, + "TelemetryStream": { + "type": "object", + "additionalProperties": False, + "required": ["stream_id", "path", "log_type"], + "description": "Vendor-neutral event or entity telemetry stream.", + "properties": { + "stream_id": { + "type": "string", + "minLength": 1, + "description": "Stable identifier for the stream.", + }, + "path": { + "type": "string", + "minLength": 1, + "description": ( + "Relative path to the log file inside the usecase." + ), + }, + "log_type": { + "type": "string", + "minLength": 1, + "description": "Canonical ingestion log type identifier.", + }, + "vendor_name": { + "type": "string", + "description": "Telemetry vendor name.", + }, + "product_name": { + "type": "string", + "description": "Telemetry product name.", + }, + "ocsf_category_uid": { + "type": ["integer", "null"], + "description": "OCSF v1.3+ category UID.", + }, + "ocsf_class_uid": { + "type": ["integer", "null"], + "description": "OCSF v1.3+ class UID.", + }, + "ocsf_class_name": { + "type": ["string", "null"], + "description": "OCSF v1.3+ class name.", + }, + "notes": { + "type": "string", + "description": "Blind-safe stream notes.", + }, + }, + }, + "TelemetrySection": { + "type": "object", + "additionalProperties": False, + "description": "Event and entity telemetry streams.", + "properties": { + "events": { + "type": "array", + "items": {"$ref": "#/$defs/TelemetryStream"}, + "description": "Event log streams under EVENTS/.", + }, + "entities": { + "type": "array", + "items": {"$ref": "#/$defs/TelemetryStream"}, + "description": "Entity context streams under ENTITIES/.", + }, + }, + }, + "GoogleSecOpsRunFrequency": { + "type": "object", + "additionalProperties": False, + "required": ["events"], + "description": "Recommended Google SecOps replay cadence.", + "properties": { + "events": { + "type": "string", + "description": "Replay frequency for event streams.", + }, + "entities": { + "type": ["string", "null"], + "description": "Replay frequency for entity streams.", + }, + }, + }, + "GoogleSecOpsDetection": { + "type": "object", + "additionalProperties": False, + "required": ["name"], + "description": "Google SecOps YARA-L 2.0 detection rule entry.", + "properties": { + "name": { + "type": "string", + "minLength": 1, + "description": "Rule name.", + }, + "path": { + "type": ["string", "null"], + "description": "Optional relative path to .yl2 file.", + }, + "live": { + "type": "boolean", + "description": "Whether live rule evaluation is enabled.", + }, + "alerting": { + "type": "boolean", + "description": "Whether alerting is enabled.", + }, + "notes": { + "type": "string", + "description": "Rule notes.", + }, + }, + }, + "GoogleSecOpsSavedSearch": { + "type": "object", + "additionalProperties": False, + "required": ["name"], + "description": "Saved UDM search definition.", + "properties": { + "name": {"type": "string", "minLength": 1}, + "creator": {"type": "string"}, + "notes": {"type": "string"}, + "query": {"type": ["string", "null"]}, + }, + }, + "GoogleSecOpsReferenceList": { + "type": "object", + "additionalProperties": False, + "required": ["name"], + "description": "Google SecOps reference list metadata.", + "properties": { + "name": {"type": "string", "minLength": 1}, + "type": {"type": "string"}, + "notes": {"type": "string"}, + }, + }, + "GoogleSecOpsPlatform": { + "type": "object", + "additionalProperties": False, + "description": "Google Security Operations platform bindings.", + "properties": { + "run_frequency": {"$ref": "#/$defs/GoogleSecOpsRunFrequency"}, + "data_rbac": {"type": ["string", "null"]}, + "log_type_mappings": { + "type": "object", + "additionalProperties": {"type": "string"}, + }, + "detections": { + "type": "array", + "items": {"$ref": "#/$defs/GoogleSecOpsDetection"}, + }, + "saved_searches": { + "type": "array", + "items": {"$ref": "#/$defs/GoogleSecOpsSavedSearch"}, + }, + "reference_lists": { + "type": "array", + "items": {"$ref": "#/$defs/GoogleSecOpsReferenceList"}, + }, + "parser_extensions": { + "type": "array", + "items": {"type": "string"}, + }, + }, + }, + "DatabricksPlatform": { + "type": "object", + "additionalProperties": False, + "description": "Databricks Unity Catalog OCSF Lakehouse bindings.", + "properties": { + "catalog": {"type": "string"}, + "schema_name": {"type": "string"}, + "table_mappings": { + "type": "object", + "additionalProperties": {"type": "string"}, + }, + "detections": { + "type": "array", + "items": {"type": "string"}, + }, + "notes": {"type": "string"}, + }, + }, + "SigmaPlatform": { + "type": "object", + "additionalProperties": False, + "description": "Sigma detection rule bindings.", + "properties": { + "rules": { + "type": "array", + "items": {"type": "string"}, + }, + "notes": {"type": "string"}, + }, + }, + "SplunkPlatform": { + "type": "object", + "additionalProperties": False, + "description": "Splunk sourcetype and search bindings.", + "properties": { + "sourcetypes": { + "type": "object", + "additionalProperties": {"type": "string"}, + }, + "saved_searches": { + "type": "array", + "items": {"type": "string"}, + }, + "notes": {"type": "string"}, + }, + }, + "PlatformBindings": { + "type": "object", + "additionalProperties": False, + "description": "Multi-platform ingestion and detection bindings.", + "properties": { + "google_secops": {"$ref": "#/$defs/GoogleSecOpsPlatform"}, + "databricks": {"$ref": "#/$defs/DatabricksPlatform"}, + "sigma": {"$ref": "#/$defs/SigmaPlatform"}, + "splunk": {"$ref": "#/$defs/SplunkPlatform"}, + }, + }, + }, + } + + +def build_eval_json_schema() -> dict[str, Any]: + """Build Draft 2020-12 JSON Schema for `eval.yaml` benchmark specifications.""" + return { + "$schema": JSON_SCHEMA_DRAFT_2020_12, + "$id": "https://chronicle.github.io/logstory/schemas/eval.schema.json", + "title": "SecurityUsecaseEval", + "description": ( + "Held-out Blue Team Evaluation & Answer Key Schema (v2.0) for " + "autonomous SOC agent benchmarking across Alert Triage, Threat " + "Hunting, and Detection Engineering." + ), + "type": "object", + "additionalProperties": False, + "required": [ + "schema_version", + "usecase_id", + "threat_context", + "tasks", + "ground_truth", + "compliance_gates", + ], + "properties": { + "schema_version": { + "type": "string", + "const": SCHEMA_VERSION_V2, + "description": "Evaluation schema version ('2.0').", + }, + "usecase_id": { + "type": "string", + "pattern": USECASE_ID_PATTERN, + "description": "Usecase identifier matching .md id.", + }, + "threat_context": {"$ref": "#/$defs/ThreatContext"}, + "tasks": {"$ref": "#/$defs/EvaluationTasks"}, + "ground_truth": {"$ref": "#/$defs/GroundTruth"}, + "compliance_gates": {"$ref": "#/$defs/ComplianceGates"}, + "counterfactual_pair": { + "oneOf": [ + {"$ref": "#/$defs/CounterfactualPair"}, + {"type": "null"}, + ], + "description": "Optional Signal Detection Theory twin pairing.", + }, + "reference_queries": {"$ref": "#/$defs/ReferenceQueries"}, + }, + "$defs": { + "ThreatContext": { + "type": "object", + "additionalProperties": False, + "required": ["grounding_tier"], + "description": "Adversary, advisory, and MITRE ATT&CK context.", + "properties": { + "grounding_tier": { + "type": "string", + "enum": [e.value for e in GroundingTier], + }, + "difficulty_tier": { + "type": "string", + "enum": [e.value for e in DifficultyTier], + }, + "threat_actor": {"type": ["string", "null"]}, + "actor_aliases": { + "type": "array", + "items": {"type": "string"}, + }, + "campaign_name": {"type": ["string", "null"]}, + "cisa_advisories": { + "type": "array", + "items": {"type": "string"}, + }, + "cve_ids": { + "type": "array", + "items": {"type": "string"}, + }, + "mitre_tactics": { + "type": "array", + "items": {"type": "string"}, + }, + "mitre_techniques": { + "type": "array", + "items": {"type": "string"}, + }, + "release_day": {"type": ["integer", "null"]}, + "cohort": {"type": ["string", "null"]}, + "notes": {"type": "string"}, + }, + }, + "AlertTriageTask": { + "type": "object", + "additionalProperties": False, + "description": "Alert triage benchmark task configuration.", + "properties": { + "enabled": {"type": "boolean"}, + "case_id": {"type": ["string", "null"]}, + "alert_ids": { + "type": "array", + "items": {"type": "string"}, + }, + "prompt": {"type": "string"}, + "expected_skill": {"type": ["string", "null"]}, + "skill_bindings": { + "type": "object", + "additionalProperties": {"type": "string"}, + }, + "time_horizon_hours": {"type": "integer"}, + }, + }, + "ThreatHuntTask": { + "type": "object", + "additionalProperties": False, + "description": "Proactive threat hunt task configuration.", + "properties": { + "enabled": {"type": "boolean"}, + "prompt_unalerted": {"type": "string"}, + "prompt_intel_driven": {"type": ["string", "null"]}, + "expected_patient_zero": {"type": ["string", "null"]}, + "expected_skill": {"type": ["string", "null"]}, + "skill_bindings": { + "type": "object", + "additionalProperties": {"type": "string"}, + }, + "time_horizon_hours": {"type": "integer"}, + }, + }, + "DetectionEngineeringTask": { + "type": "object", + "additionalProperties": False, + "description": "Detection engineering task configuration.", + "properties": { + "enabled": {"type": "boolean"}, + "prompt": {"type": "string"}, + "target_techniques": { + "type": "array", + "items": {"type": "string"}, + }, + "reference_rules": { + "type": "array", + "items": {"type": "string"}, + }, + "expected_rule_count_min": {"type": "integer"}, + "max_false_positives_on_paired_baseline": {"type": "integer"}, + }, + }, + "EvaluationTasks": { + "type": "object", + "additionalProperties": False, + "description": "Multi-task Blue Team benchmark suite.", + "properties": { + "alert_triage": {"$ref": "#/$defs/AlertTriageTask"}, + "threat_hunt": {"$ref": "#/$defs/ThreatHuntTask"}, + "detection_engineering": {"$ref": "#/$defs/DetectionEngineeringTask"}, + }, + }, + "KillChainStage": { + "type": "object", + "additionalProperties": False, + "required": ["stage", "stream_id"], + "description": "Single chronological attack stage.", + "properties": { + "stage": {"type": "string", "minLength": 1}, + "stream_id": {"type": "string", "minLength": 1}, + "timestamp_utc": {"type": ["string", "null"]}, + "tactic": {"type": ["string", "null"]}, + "technique_id": {"type": ["string", "null"]}, + "summary": {"type": "string"}, + "key_evidence": { + "type": "array", + "items": {"type": "string"}, + }, + }, + }, + "AffectedAsset": { + "type": "object", + "additionalProperties": False, + "required": ["hostname"], + "description": "Affected enterprise asset and blast-radius cost.", + "properties": { + "hostname": {"type": "string", "minLength": 1}, + "criticality_tier": { + "type": "string", + "enum": [e.value for e in CriticalityTier], + }, + "ip_addresses": { + "type": "array", + "items": {"type": "string"}, + }, + "role": {"type": "string"}, + "patient_zero": {"type": "boolean"}, + "blast_radius_fp_usd": {"type": "number"}, + "blast_radius_fn_usd": {"type": "number"}, + }, + }, + "MaliciousIoc": { + "type": "object", + "additionalProperties": False, + "required": ["type", "value"], + "description": "Ground-truth malicious indicator of compromise.", + "properties": { + "type": {"type": "string", "minLength": 1}, + "value": {"type": "string", "minLength": 1}, + "role": {"type": "string"}, + "notes": {"type": "string"}, + }, + }, + "ChangeWindow": { + "type": "object", + "additionalProperties": False, + "required": ["ticket_id"], + "description": "Maintenance or change window record.", + "properties": { + "ticket_id": {"type": "string", "minLength": 1}, + "approved": {"type": "boolean"}, + "start_utc": {"type": ["string", "null"]}, + "end_utc": {"type": ["string", "null"]}, + "summary": {"type": "string"}, + }, + }, + "BenignConfusers": { + "type": "object", + "additionalProperties": False, + "description": "Benign confusers and signed OS LOLBins.", + "properties": { + "signed_lolbins": { + "type": "array", + "items": {"type": "string"}, + }, + "internal_ips": { + "type": "array", + "items": {"type": "string"}, + }, + "change_window": { + "oneOf": [ + {"$ref": "#/$defs/ChangeWindow"}, + {"type": "null"}, + ] + }, + "admin_tools": { + "type": "array", + "items": {"type": "string"}, + }, + }, + }, + "EntitiesAndIocs": { + "type": "object", + "additionalProperties": False, + "description": "Ground-truth entities, IOCs, and confusers.", + "properties": { + "compromised_accounts": { + "type": "array", + "items": {"type": "string"}, + }, + "affected_assets": { + "type": "array", + "items": {"$ref": "#/$defs/AffectedAsset"}, + }, + "malicious_iocs": { + "type": "array", + "items": {"$ref": "#/$defs/MaliciousIoc"}, + }, + "benign_confusers": {"$ref": "#/$defs/BenignConfusers"}, + }, + }, + "GroundTruth": { + "type": "object", + "additionalProperties": False, + "required": ["expected_disposition", "expected_verdict"], + "description": "Ground-truth disposition, verdict, and timeline.", + "properties": { + "expected_disposition": { + "type": "string", + "enum": [e.value for e in ExpectedDisposition], + }, + "expected_verdict": { + "type": "string", + "enum": [e.value for e in ExpectedVerdict], + }, + "severity": {"type": "string"}, + "patient_zero_host": {"type": ["string", "null"]}, + "patient_zero_account": {"type": ["string", "null"]}, + "root_cause_summary": {"type": "string"}, + "kill_chain": { + "type": "array", + "items": {"$ref": "#/$defs/KillChainStage"}, + }, + "entities_and_iocs": {"$ref": "#/$defs/EntitiesAndIocs"}, + }, + }, + "ExpectedActions": { + "type": "object", + "additionalProperties": False, + "description": "Primary, secondary, and prohibited response actions.", + "properties": { + "primary": { + "type": "array", + "items": {"type": "string"}, + }, + "secondary": { + "type": "array", + "items": {"type": "string"}, + }, + "prohibited": { + "type": "array", + "items": {"type": "string"}, + }, + }, + }, + "ComplianceGates": { + "type": "object", + "additionalProperties": False, + "description": "Safety and SOP compliance requirements.", + "properties": { + "requires_pre_containment_verification": {"type": "boolean"}, + "requires_168h_horizon_widening": {"type": "boolean"}, + "forbid_external_cti_on_internal_ips": {"type": "boolean"}, + "forbid_lolbin_hash_blocking": {"type": "boolean"}, + "respect_change_window": {"type": "boolean"}, + "expected_actions": {"$ref": "#/$defs/ExpectedActions"}, + }, + }, + "CounterfactualPair": { + "type": "object", + "additionalProperties": False, + "required": ["paired_usecase_id", "pair_relationship"], + "description": "Counterfactual benign/malicious twin link.", + "properties": { + "paired_usecase_id": {"type": "string", "minLength": 1}, + "pair_relationship": { + "type": "string", + "enum": [e.value for e in PairRelationship], + }, + "shared_entities": { + "type": "array", + "items": {"type": "string"}, + }, + "shared_telemetry_streams": { + "type": "array", + "items": {"type": "string"}, + }, + "discriminating_invariant": {"type": "string"}, + }, + }, + "ReferenceQueries": { + "type": "object", + "additionalProperties": False, + "description": "Reference investigative queries by platform.", + "properties": { + "google_secops_udm": { + "type": "array", + "items": {"type": "string"}, + }, + "google_secops_yaral": { + "type": "array", + "items": {"type": "string"}, + }, + "databricks_sql": { + "type": "array", + "items": {"type": "string"}, + }, + "sigma": { + "type": "array", + "items": {"type": "string"}, + }, + "splunk_spl": { + "type": "array", + "items": {"type": "string"}, + }, + }, + }, + }, + } + + +def export_json_schemas(output_dir: Path) -> tuple[Path, Path]: + """Write `usecase.schema.json` and `eval.schema.json` to `output_dir`. + + Args: + output_dir: Directory where JSON Schema files will be written. + + Returns: + Tuple of `(usecase_schema_path, eval_schema_path)`. + """ + target_dir = Path(output_dir) + target_dir.mkdir(parents=True, exist_ok=True) + usecase_path = target_dir / "usecase.schema.json" + eval_path = target_dir / "eval.schema.json" + + usecase_path.write_text( + json.dumps(build_usecase_json_schema(), indent=2) + "\n", + encoding="utf-8", + ) + eval_path.write_text( + json.dumps(build_eval_json_schema(), indent=2) + "\n", + encoding="utf-8", + ) + return usecase_path, eval_path diff --git a/src/logstory/schema/validator.py b/src/logstory/schema/validator.py new file mode 100644 index 0000000..3f4e680 --- /dev/null +++ b/src/logstory/schema/validator.py @@ -0,0 +1,766 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""5-gate validator and anti-spoiler boundary checker for Logstory usecases.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Literal + +import yaml + +from logstory.schema.models import ( + SCHEMA_VERSION_V1, + PairRelationship, + ScenarioType, + SchemaValidationError, + SecurityUsecaseEval, + SecurityUsecaseManifest, + _normalize_yaml_scalars, + split_frontmatter_and_body, + split_solution_section, +) + +GATE_G1_SCHEMA = "G1_SCHEMA" +GATE_G2_INTEGRITY = "G2_INTEGRITY" +GATE_G3_ANTI_SPOILER = "G3_ANTI_SPOILER" +GATE_G4_COUNTERFACTUAL = "G4_COUNTERFACTUAL" +GATE_G5_LEGACY_V1 = "G5_LEGACY_V1" + +MIN_SPOILER_TOKEN_LEN = 3 + +FORBIDDEN_FRONTMATTER_KEYS: frozenset[str] = frozenset({ + "ground_truth", + "kill_chain", + "malicious_iocs", + "compliance_gates", + "threat_context", + "compromised_accounts", + "affected_assets", + "expected_disposition", + "expected_verdict", + "cisa_advisories", + "patient_zero_host", + "patient_zero_account", + "counterfactual_pair", +}) + +_TRIVIAL_PLACEHOLDERS: frozenset[str] = frozenset({ + "n/a", + "none", + "null", + "tbd", + "unknown", +}) + + +@dataclass +class ValidationFinding: + """Single validation finding emitted by one of the 5 verification gates.""" + + gate: str + severity: Literal["ERROR", "WARNING"] + message: str + path: str | None = None + + def to_dict(self) -> dict[str, Any]: + """Serialize ValidationFinding to a plain dictionary.""" + return { + "gate": self.gate, + "severity": self.severity, + "message": self.message, + "path": self.path, + } + + +@dataclass +class UsecaseValidationResult: + """Structured 5-gate validation outcome for a single usecase package.""" + + usecase_id: str + usecase_dir: Path + schema_version: str + findings: list[ValidationFinding] = field(default_factory=list) + + @property + def is_valid(self) -> bool: + """Return True when zero ERROR-severity findings were recorded.""" + return not self.errors + + @property + def errors(self) -> list[ValidationFinding]: + """Return all ERROR-severity findings.""" + return [f for f in self.findings if f.severity == "ERROR"] + + @property + def warnings(self) -> list[ValidationFinding]: + """Return all WARNING-severity findings.""" + return [f for f in self.findings if f.severity == "WARNING"] + + def to_dict(self) -> dict[str, Any]: + """Serialize UsecaseValidationResult to a JSON-compatible dictionary.""" + return { + "usecase_id": self.usecase_id, + "usecase_dir": str(self.usecase_dir), + "schema_version": self.schema_version, + "is_valid": self.is_valid, + "findings": [f.to_dict() for f in self.findings], + "errors": [f.to_dict() for f in self.errors], + "warnings": [f.to_dict() for f in self.warnings], + } + + +def _check_forbidden_frontmatter_keys( + data: Any, path: str, findings: list[ValidationFinding] +) -> None: + """Recursively detect held-out ground-truth keys placed in .md.""" + if isinstance(data, dict): + for key, value in data.items(): + field_path = f"{path}.{key}" if path else str(key) + if key in FORBIDDEN_FRONTMATTER_KEYS: + findings.append( + ValidationFinding( + gate=GATE_G3_ANTI_SPOILER, + severity="ERROR", + message=( + f"Forbidden ground-truth key '{key}' is not permitted in" + " .md frontmatter" + ), + path=field_path, + ) + ) + else: + _check_forbidden_frontmatter_keys(value, field_path, findings) + elif isinstance(data, list): + for idx, item in enumerate(data): + _check_forbidden_frontmatter_keys(item, f"{path}[{idx}]", findings) + + +def _strip_forbidden_frontmatter_keys(data: dict[str, Any]) -> dict[str, Any]: + """Return a copy of frontmatter with forbidden ground-truth keys removed.""" + cleaned: dict[str, Any] = {} + for key, value in data.items(): + if key in FORBIDDEN_FRONTMATTER_KEYS: + continue + if isinstance(value, dict): + cleaned[key] = _strip_forbidden_frontmatter_keys(value) + else: + cleaned[key] = value + return cleaned + + +def _validate_v1_frontmatter( + raw_frontmatter: dict[str, Any], + strict_v2: bool, + findings: list[ValidationFinding], +) -> None: + """Run Gate 5 legacy v1 compatibility and basic structure checks.""" + if strict_v2: + findings.append( + ValidationFinding( + gate=GATE_G5_LEGACY_V1, + severity="ERROR", + message=( + "Usecase manifest lacks schema_version '2.0' (legacy v1" + " frontmatter rejected under strict_v2=True)" + ), + path="manifest.schema_version", + ) + ) + return + + findings.append( + ValidationFinding( + gate=GATE_G5_LEGACY_V1, + severity="WARNING", + message=( + "Usecase uses legacy v1 frontmatter; migrate to schema_version" + " '2.0' and held-out eval.yaml" + ), + path="manifest.schema_version", + ) + ) + for required_v1_key in ("title", "description"): + val = raw_frontmatter.get(required_v1_key) + if not isinstance(val, str) or not val.strip(): + findings.append( + ValidationFinding( + gate=GATE_G1_SCHEMA, + severity="ERROR", + message=f"Legacy v1 frontmatter is missing '{required_v1_key}'", + path=f"manifest.{required_v1_key}", + ) + ) + + +def _validate_gate2_integrity( + resolved_dir: Path, + manifest: SecurityUsecaseManifest, + eval_data: SecurityUsecaseEval | None, + findings: list[ValidationFinding], +) -> None: + """Run Gate 2 (G2_INTEGRITY) stream ID cross-reference and file path checks.""" + declared_stream_ids: set[str] = set() + + for idx, event_stream in enumerate(manifest.telemetry.events): + stream_path = f"manifest.telemetry.events[{idx}]" + if event_stream.stream_id in declared_stream_ids: + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=f"Duplicate telemetry stream_id {event_stream.stream_id!r}", + path=f"{stream_path}.stream_id", + ) + ) + declared_stream_ids.add(event_stream.stream_id) + if not (resolved_dir / event_stream.path).is_file(): + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=f"Missing event telemetry file on disk: {event_stream.path}", + path=f"{stream_path}.path", + ) + ) + + for idx, entity_stream in enumerate(manifest.telemetry.entities): + stream_path = f"manifest.telemetry.entities[{idx}]" + if entity_stream.stream_id in declared_stream_ids: + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=f"Duplicate telemetry stream_id {entity_stream.stream_id!r}", + path=f"{stream_path}.stream_id", + ) + ) + declared_stream_ids.add(entity_stream.stream_id) + if not (resolved_dir / entity_stream.path).is_file(): + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=f"Missing entity telemetry file on disk: {entity_stream.path}", + path=f"{stream_path}.path", + ) + ) + + secops = manifest.platforms.google_secops + if secops is not None: + for stream_id in secops.log_type_mappings: + if stream_id not in declared_stream_ids: + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=( + "google_secops.log_type_mappings references unknown" + f" stream_id {stream_id!r}" + ), + path=f"manifest.platforms.google_secops.log_type_mappings.{stream_id}", + ) + ) + for idx, detection in enumerate(secops.detections): + if detection.path and not (resolved_dir / detection.path).is_file(): + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=f"Missing detection rule file on disk: {detection.path}", + path=f"manifest.platforms.google_secops.detections[{idx}].path", + ) + ) + for idx, ext_path in enumerate(secops.parser_extensions): + if ext_path and not (resolved_dir / ext_path).is_file(): + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=f"Missing parser extension file on disk: {ext_path}", + path=f"manifest.platforms.google_secops.parser_extensions[{idx}]", + ) + ) + + databricks = manifest.platforms.databricks + if databricks is not None: + for stream_id in databricks.table_mappings: + if stream_id not in declared_stream_ids: + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=( + "databricks.table_mappings references unknown stream_id" + f" {stream_id!r}" + ), + path=f"manifest.platforms.databricks.table_mappings.{stream_id}", + ) + ) + + splunk = manifest.platforms.splunk + if splunk is not None: + for stream_id in splunk.sourcetypes: + if stream_id not in declared_stream_ids: + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=( + f"splunk.sourcetypes references unknown stream_id {stream_id!r}" + ), + path=f"manifest.platforms.splunk.sourcetypes.{stream_id}", + ) + ) + + sigma = manifest.platforms.sigma + if sigma is not None: + for idx, rule_path in enumerate(sigma.rules): + if rule_path and not (resolved_dir / rule_path).is_file(): + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=f"Missing Sigma rule file on disk: {rule_path}", + path=f"manifest.platforms.sigma.rules[{idx}]", + ) + ) + + if eval_data is not None: + for idx, stage in enumerate(eval_data.ground_truth.kill_chain): + if stage.stream_id not in declared_stream_ids: + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=( + f"kill_chain[{idx}] references unknown stream_id" + f" {stage.stream_id!r}" + ), + path=f"eval.ground_truth.kill_chain[{idx}].stream_id", + ) + ) + if eval_data.counterfactual_pair is not None: + for idx, stream_id in enumerate( + eval_data.counterfactual_pair.shared_telemetry_streams + ): + if stream_id not in declared_stream_ids: + findings.append( + ValidationFinding( + gate=GATE_G2_INTEGRITY, + severity="ERROR", + message=( + "counterfactual_pair.shared_telemetry_streams references" + f" unknown stream_id {stream_id!r}" + ), + path=f"eval.counterfactual_pair.shared_telemetry_streams[{idx}]", + ) + ) + + +def _collect_frontmatter_text_surfaces( + raw_frontmatter: dict[str, Any], +) -> list[tuple[str, str]]: + """Collect blind-visible frontmatter text surfaces for anti-spoiler scanning.""" + surfaces: list[tuple[str, str]] = [] + for top_key in ("title", "description", "notes"): + val = raw_frontmatter.get(top_key) + if isinstance(val, str) and val.strip(): + surfaces.append((f"manifest.{top_key}", val)) + + tags = raw_frontmatter.get("tags") + if isinstance(tags, list): + for idx, tag in enumerate(tags): + if isinstance(tag, str) and tag.strip(): + surfaces.append((f"manifest.tags[{idx}]", tag)) + + def _walk_notes(node: Any, path: str) -> None: + if isinstance(node, dict): + for k, v in node.items(): + subpath = f"{path}.{k}" if path else str(k) + if k == "notes" and isinstance(v, str) and v.strip(): + surfaces.append((subpath, v)) + else: + _walk_notes(v, subpath) + elif isinstance(node, list): + for idx, item in enumerate(node): + _walk_notes(item, f"{path}[{idx}]") + + for section_key in ("environment", "telemetry", "platforms"): + if section_key in raw_frontmatter: + _walk_notes(raw_frontmatter[section_key], f"manifest.{section_key}") + + return surfaces + + +def _collect_spoiler_indicators( + eval_data: SecurityUsecaseEval, +) -> list[tuple[str, str]]: + """Extract non-trivial IOCs, compromised accounts, and CISA advisories.""" + indicators: list[tuple[str, str]] = [] + entities = eval_data.ground_truth.entities_and_iocs + + for ioc in entities.malicious_iocs: + candidate = ioc.value.strip() + if ( + len(candidate) >= MIN_SPOILER_TOKEN_LEN + and candidate.lower() not in _TRIVIAL_PLACEHOLDERS + ): + indicators.append((f"malicious_ioc ({ioc.type})", candidate)) + + for account in entities.compromised_accounts: + candidate = account.strip() + if ( + len(candidate) >= MIN_SPOILER_TOKEN_LEN + and candidate.lower() not in _TRIVIAL_PLACEHOLDERS + ): + indicators.append(("compromised_account", candidate)) + + for advisory in eval_data.threat_context.cisa_advisories: + candidate = advisory.strip() + if ( + len(candidate) >= MIN_SPOILER_TOKEN_LEN + and candidate.lower() not in _TRIVIAL_PLACEHOLDERS + ): + indicators.append(("cisa_advisory", candidate)) + + return indicators + + +def _validate_gate3_anti_spoiler( + raw_frontmatter: dict[str, Any], + public_markdown_body: str, + eval_data: SecurityUsecaseEval | None, + findings: list[ValidationFinding], +) -> None: + """Run Gate 3 (G3_ANTI_SPOILER) leakage checks against eval.yaml indicators.""" + if eval_data is None: + return + + indicators = _collect_spoiler_indicators(eval_data) + if not indicators: + return + + surfaces = _collect_frontmatter_text_surfaces(raw_frontmatter) + for surface_path, surface_text in surfaces: + lower_surface = surface_text.lower() + for indicator_kind, indicator_value in indicators: + if indicator_value.lower() in lower_surface: + findings.append( + ValidationFinding( + gate=GATE_G3_ANTI_SPOILER, + severity="ERROR", + message=( + f"Spoiler {indicator_kind} {indicator_value!r} leaked in" + f" frontmatter field '{surface_path}'" + ), + path=surface_path, + ) + ) + + if public_markdown_body: + lower_public_body = public_markdown_body.lower() + for indicator_kind, indicator_value in indicators: + if indicator_value.lower() in lower_public_body: + findings.append( + ValidationFinding( + gate=GATE_G3_ANTI_SPOILER, + severity="ERROR", + message=( + f"Spoiler {indicator_kind} {indicator_value!r} leaked in" + " Markdown body above " + ), + path="public_markdown_body", + ) + ) + + +def _validate_gate4_counterfactual( + manifest: SecurityUsecaseManifest, + eval_data: SecurityUsecaseEval | None, + corpus_root: Path | None, + findings: list[ValidationFinding], +) -> None: + """Run Gate 4 (G4_COUNTERFACTUAL) counterfactual twin consistency checks.""" + checked_missing_dirs: set[str] = set() + + if eval_data is not None and eval_data.counterfactual_pair is not None: + pair = eval_data.counterfactual_pair + paired_id = pair.paired_usecase_id + is_baseline_scenario = manifest.environment.scenario_type in ( + ScenarioType.BENIGN_BASELINE, + ScenarioType.BASELINE, + ) + if ( + pair.pair_relationship == PairRelationship.MALICIOUS_VARIANT_OF_BENIGN_BASELINE + and not is_baseline_scenario + and paired_id not in manifest.environment.paired_baselines + ): + findings.append( + ValidationFinding( + gate=GATE_G4_COUNTERFACTUAL, + severity="ERROR", + message=( + f"Counterfactual paired baseline {paired_id!r} in eval.yaml" + " is not listed in manifest environment.paired_baselines" + ), + path="manifest.environment.paired_baselines", + ) + ) + + if corpus_root is not None and not (Path(corpus_root) / paired_id).is_dir(): + checked_missing_dirs.add(paired_id) + findings.append( + ValidationFinding( + gate=GATE_G4_COUNTERFACTUAL, + severity="ERROR", + message=( + "Counterfactual paired usecase directory does not exist in" + f" corpus_root: {paired_id}" + ), + path="eval.counterfactual_pair.paired_usecase_id", + ) + ) + + if corpus_root is not None: + for idx, baseline_id in enumerate(manifest.environment.paired_baselines): + if baseline_id in checked_missing_dirs: + continue + if not (Path(corpus_root) / baseline_id).is_dir(): + checked_missing_dirs.add(baseline_id) + findings.append( + ValidationFinding( + gate=GATE_G4_COUNTERFACTUAL, + severity="ERROR", + message=( + "Paired baseline directory does not exist in corpus_root:" + f" {baseline_id}" + ), + path=f"manifest.environment.paired_baselines[{idx}]", + ) + ) + + +def validate_usecase_package( + usecase_dir: Path, + strict_v2: bool = False, + corpus_root: Path | None = None, +) -> UsecaseValidationResult: + """Execute the 5-gate validation pipeline on a single usecase directory. + + Args: + usecase_dir: Path to `` directory. + strict_v2: When True, rejects legacy v1 packages and requires `eval.yaml`. + corpus_root: Optional path to the parent corpus root directory for verifying + that paired counterfactual baseline directories exist on disk. + + Returns: + Populated `UsecaseValidationResult` containing all errors and warnings. + """ + resolved_dir = Path(usecase_dir) + usecase_id = resolved_dir.name + findings: list[ValidationFinding] = [] + + md_path = resolved_dir / f"{usecase_id}.md" + if not md_path.is_file(): + findings.append( + ValidationFinding( + gate=GATE_G1_SCHEMA, + severity="ERROR", + message=f"Missing required usecase manifest file: {md_path}", + path=str(md_path), + ) + ) + return UsecaseValidationResult( + usecase_id=usecase_id, + usecase_dir=resolved_dir, + schema_version="unknown", + findings=findings, + ) + + try: + raw_frontmatter, markdown_body = split_frontmatter_and_body( + md_path.read_text(encoding="utf-8") + ) + except SchemaValidationError as exc: + findings.append( + ValidationFinding( + gate=GATE_G1_SCHEMA, + severity="ERROR", + message=exc.reason, + path=exc.path or "frontmatter", + ) + ) + return UsecaseValidationResult( + usecase_id=usecase_id, + usecase_dir=resolved_dir, + schema_version="unknown", + findings=findings, + ) + + public_markdown_body, _ = split_solution_section(markdown_body) + _check_forbidden_frontmatter_keys(raw_frontmatter, "manifest", findings) + + has_schema_version = ( + "schema_version" in raw_frontmatter + and raw_frontmatter["schema_version"] is not None + ) + raw_version = ( + str(raw_frontmatter["schema_version"]) + if has_schema_version + else SCHEMA_VERSION_V1 + ) + + if not has_schema_version or raw_version == SCHEMA_VERSION_V1: + _validate_v1_frontmatter(raw_frontmatter, strict_v2, findings) + return UsecaseValidationResult( + usecase_id=usecase_id, + usecase_dir=resolved_dir, + schema_version=SCHEMA_VERSION_V1, + findings=findings, + ) + + cleaned_frontmatter = _strip_forbidden_frontmatter_keys(raw_frontmatter) + manifest: SecurityUsecaseManifest | None = None + try: + manifest = SecurityUsecaseManifest.from_dict(cleaned_frontmatter, path="manifest") + if manifest.id != usecase_id: + findings.append( + ValidationFinding( + gate=GATE_G1_SCHEMA, + severity="ERROR", + message=( + f"Manifest id {manifest.id!r} does not match directory name" + f" {usecase_id!r}" + ), + path="manifest.id", + ) + ) + except SchemaValidationError as exc: + findings.append( + ValidationFinding( + gate=GATE_G1_SCHEMA, + severity="ERROR", + message=exc.reason, + path=exc.path or "manifest", + ) + ) + + eval_data: SecurityUsecaseEval | None = None + eval_path = resolved_dir / "eval.yaml" + if eval_path.is_file(): + try: + loaded_eval = yaml.safe_load(eval_path.read_text(encoding="utf-8")) + eval_data = SecurityUsecaseEval.from_dict( + _normalize_yaml_scalars(loaded_eval), path="eval" + ) + if eval_data.usecase_id != usecase_id: + findings.append( + ValidationFinding( + gate=GATE_G1_SCHEMA, + severity="ERROR", + message=( + f"eval.yaml usecase_id {eval_data.usecase_id!r} does not" + f" match directory name {usecase_id!r}" + ), + path="eval.usecase_id", + ) + ) + except yaml.YAMLError as exc: + findings.append( + ValidationFinding( + gate=GATE_G1_SCHEMA, + severity="ERROR", + message=f"Invalid YAML syntax in eval.yaml: {exc}", + path="eval.yaml", + ) + ) + except SchemaValidationError as exc: + findings.append( + ValidationFinding( + gate=GATE_G1_SCHEMA, + severity="ERROR", + message=exc.reason, + path=exc.path or "eval", + ) + ) + else: + findings.append( + ValidationFinding( + gate=GATE_G1_SCHEMA, + severity="ERROR" if strict_v2 else "WARNING", + message="Missing eval.yaml for v2 usecase package", + path="eval.yaml", + ) + ) + + if manifest is not None: + _validate_gate2_integrity(resolved_dir, manifest, eval_data, findings) + + _validate_gate3_anti_spoiler( + raw_frontmatter, public_markdown_body, eval_data, findings + ) + + if manifest is not None: + _validate_gate4_counterfactual(manifest, eval_data, corpus_root, findings) + + return UsecaseValidationResult( + usecase_id=usecase_id, + usecase_dir=resolved_dir, + schema_version=raw_version, + findings=findings, + ) + + +def validate_usecase_corpus( + corpus_dir: Path, strict_v2: bool = False +) -> list[UsecaseValidationResult]: + """Discover and validate all usecase directories under `corpus_dir`. + + Args: + corpus_dir: Root directory containing multiple `/` subdirectories, + or a single `/` directory. + strict_v2: When True, requires `schema_version: "2.0"` and `eval.yaml` for + every discovered usecase package. + + Returns: + List of `UsecaseValidationResult` objects sorted by `usecase_id`. + """ + resolved_corpus = Path(corpus_dir) + if (resolved_corpus / f"{resolved_corpus.name}.md").is_file(): + return [ + validate_usecase_package( + resolved_corpus, + strict_v2=strict_v2, + corpus_root=resolved_corpus.parent, + ) + ] + + usecase_dirs: list[Path] = [] + for child in sorted(resolved_corpus.iterdir(), key=lambda p: p.name): + if not child.is_dir() or child.name.startswith("."): + continue + if ( + (child / f"{child.name}.md").is_file() + or (child / "eval.yaml").is_file() + or (child / "EVENTS").is_dir() + ): + usecase_dirs.append(child) + + return [ + validate_usecase_package(uc_dir, strict_v2=strict_v2, corpus_root=resolved_corpus) + for uc_dir in usecase_dirs + ] diff --git a/src/logstory/usecases/NETWORK_ANALYSIS/eval.yaml b/src/logstory/usecases/NETWORK_ANALYSIS/eval.yaml new file mode 120000 index 0000000..764f297 --- /dev/null +++ b/src/logstory/usecases/NETWORK_ANALYSIS/eval.yaml @@ -0,0 +1 @@ +../../../../usecases/NETWORK_ANALYSIS/eval.yaml \ No newline at end of file diff --git a/tests/test_generate_usecase_docs.py b/tests/test_generate_usecase_docs.py new file mode 100644 index 0000000..61115c0 --- /dev/null +++ b/tests/test_generate_usecase_docs.py @@ -0,0 +1,404 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for v1/v2 usecase documentation generator and built-in usecases.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest + +from logstory.schema import parse_usecase_package, validate_usecase_package +from usecases.generate_usecase_docs import ( + _render_without_jinja2, + generate_usecase_doc, + normalize_frontmatter_for_template, + parse_frontmatter, + process_usecase_directory, +) + +REPO_ROOT = Path(__file__).resolve().parent.parent +TEMPLATE_PATH = REPO_ROOT / "usecases" / "usecase_template.j2" + + +def _sample_v2_frontmatter() -> dict[str, Any]: + """Return a complete v2 SecurityUsecaseManifest frontmatter dictionary.""" + return { + "schema_version": "2.0", + "id": "DNS_TUNNELING_EXFIL_GTI", + "title": "DNS Tunneling Data Exfiltration Telemetry Package", + "version": "2.0.0", + "created": "2026-10-18", + "updated": "2026-11-05", + "author": "Google Cloud Security", + "license": "Apache-2.0", + "description": ( + "Enterprise DNS and directory telemetry package for network protocol" + " analysis." + ), + "tags": [ + "network-c2", + "dns", + "ocsf", + "google-secops", + "databricks", + ], + "environment": { + "organization": "cymbal-investments.org", + "scenario_type": "attack", + "paired_baselines": ["CYMBAL_BASELINE_NETWORK"], + "timestamp_policy": { + "base_date": "2026-10-18", + "preserve_relative_offsets": True, + "default_delta": "1d", + "entity_offset_days": 1, + }, + }, + "telemetry": { + "events": [{ + "stream_id": "zeek_dns", + "path": "EVENTS/BRO_JSON.log", + "log_type": "BRO_JSON", + "vendor_name": "Zeek", + "product_name": "Bro", + "ocsf_category_uid": 4, + "ocsf_class_uid": 4003, + "ocsf_class_name": "DNS Activity", + "notes": "Core DNS resolution telemetry.", + }], + "entities": [{ + "stream_id": "windows_ad_users", + "path": "ENTITIES/WINDOWS_AD.log", + "log_type": "WINDOWS_AD", + "vendor_name": "Microsoft", + "product_name": "Active Directory", + "ocsf_category_uid": 5, + "ocsf_class_uid": 5003, + "ocsf_class_name": "User Inventory", + "notes": "Directory user context.", + }], + }, + "platforms": { + "google_secops": { + "run_frequency": { + "events": "Every 1 day", + "entities": "Every 1 day (offset 1 day prior)", + }, + "data_rbac": "Standard RBAC", + "log_type_mappings": { + "zeek_dns": "BRO_JSON", + "windows_ad_users": "WINDOWS_AD", + }, + "detections": [{ + "name": "dns_tunneling_low_reputation_domain", + "path": "RULES/dns_tunneling_low_reputation_domain.yl2", + "live": True, + "alerting": True, + "notes": "Detects high-entropy DNS subdomain queries.", + }], + "saved_searches": [{ + "name": "High Entropy DNS Queries", + "creator": "secops-team@cymbal-investments.org", + "notes": "Baseline DNS query length exploration.", + }], + "reference_lists": [{ + "name": "approved_dns_resolvers", + "type": "String", + "notes": "Internal corporate DNS resolvers.", + }], + }, + "databricks": { + "catalog": "security_lakehouse", + "schema_name": "ocsf_silver", + "table_mappings": { + "zeek_dns": "secops_ocsf.dns_activity", + "windows_ad_users": "secops_ocsf.user_inventory", + }, + "detections": ["RULES/databricks_dns_tunneling.sql"], + }, + "sigma": { + "rules": ["RULES/sigma_dns_tunneling.yml"], + }, + "splunk": { + "sourcetypes": { + "zeek_dns": "zeek:dns:json", + "windows_ad_users": "ms:ad:user", + }, + }, + }, + } + + +def _sample_v1_frontmatter() -> dict[str, Any]: + """Return a legacy v1 frontmatter dictionary.""" + return { + "title": "LEGACY_WORKSHOP", + "description": "Legacy v1 workshop usecase.", + "tags": ["workshop", "legacy"], + "created": "2025-07-26", + "updated": "2025-07-26", + "run_frequency": { + "events": "Every 3 days", + "entities": "Every 3 days (offset 1 day)", + }, + "data_rbac": "Must be disabled", + "events": [{ + "log_type": "POWERSHELL.log", + "product_name": "PowerShell", + "vendor_name": "Microsoft", + "notes": "PowerShell script block logs", + }], + "entities": [{ + "log_type": "N/A", + "product_name": "", + "vendor_name": "", + "notes": "", + }], + "rules": [{ + "name": "whoami_execution.yaral", + "live": True, + "alerting": False, + "notes": "Slide 17", + }], + "saved_searches": [{ + "name": "Failed User Logins", + "creator": "Curated", + "notes": "Slide 15", + }], + "reference_lists": [{ + "name": "key_servers", + "type": "String", + "notes": "Key server hostnames", + }], + } + + +class TestFrontmatterNormalization: + """Test v1/v2 frontmatter normalization shim for template rendering.""" + + def test_normalize_v2_frontmatter_populates_legacy_and_preserves_v2_keys( + self, + ): + v2 = _sample_v2_frontmatter() + normalized = normalize_frontmatter_for_template(v2) + + # Preserved v2 keys + assert normalized["schema_version"] == "2.0" + assert normalized["id"] == "DNS_TUNNELING_EXFIL_GTI" + assert normalized["version"] == "2.0.0" + assert normalized["author"] == "Google Cloud Security" + assert normalized["license"] == "Apache-2.0" + assert normalized["environment"]["organization"] == "cymbal-investments.org" + assert normalized["environment"]["paired_baselines"] == ["CYMBAL_BASELINE_NETWORK"] + assert "telemetry" in normalized + assert "platforms" in normalized + + # Populated legacy compatibility keys + assert len(normalized["events"]) == 1 + assert normalized["events"][0]["log_type"] == "BRO_JSON" + assert normalized["events"][0]["vendor_name"] == "Zeek" + assert normalized["events"][0]["product_name"] == "Bro" + assert normalized["events"][0]["ocsf_class_uid"] == 4003 + assert normalized["events"][0]["ocsf_class_name"] == "DNS Activity" + + assert len(normalized["entities"]) == 1 + assert normalized["entities"][0]["log_type"] == "WINDOWS_AD" + assert normalized["entities"][0]["ocsf_class_uid"] == 5003 + + assert normalized["run_frequency"] == { + "events": "Every 1 day", + "entities": "Every 1 day (offset 1 day prior)", + } + assert normalized["data_rbac"] == "Standard RBAC" + + assert len(normalized["rules"]) == 1 + assert normalized["rules"][0]["name"] == "dns_tunneling_low_reputation_domain" + assert normalized["rules"][0]["live"] is True + assert normalized["rules"][0]["alerting"] is True + + assert len(normalized["saved_searches"]) == 1 + assert normalized["saved_searches"][0]["name"] == "High Entropy DNS Queries" + + assert len(normalized["reference_lists"]) == 1 + assert normalized["reference_lists"][0]["name"] == "approved_dns_resolvers" + + def test_normalize_v1_frontmatter_preserves_existing_keys(self): + v1 = _sample_v1_frontmatter() + normalized = normalize_frontmatter_for_template(v1) + assert normalized["title"] == "LEGACY_WORKSHOP" + assert normalized["events"][0]["log_type"] == "POWERSHELL.log" + assert normalized["rules"][0]["name"] == "whoami_execution.yaral" + assert normalized["data_rbac"] == "Must be disabled" + + +class TestGenerateUsecaseDocRendering: + """Test documentation rendering for v1 and v2 frontmatter.""" + + def test_generate_usecase_doc_v1(self, tmp_path: Path): + v1 = _sample_v1_frontmatter() + out_file = tmp_path / "LEGACY_WORKSHOP_generated.md" + rendered = generate_usecase_doc(v1, TEMPLATE_PATH, out_file) + + assert out_file.is_file() + assert "# LEGACY_WORKSHOP" in rendered + assert "Legacy v1 workshop usecase." in rendered + assert "POWERSHELL.log" in rendered + assert "No entities configured for this usecase." in rendered + assert "whoami_execution.yaral" in rendered + assert "Failed User Logins" in rendered + assert "key_servers" in rendered + + def test_generate_usecase_doc_v2_renders_ocsf_baselines_and_platforms( + self, tmp_path: Path + ): + v2 = _sample_v2_frontmatter() + out_file = tmp_path / "DNS_TUNNELING_EXFIL_GTI_generated.md" + rendered = generate_usecase_doc(v2, TEMPLATE_PATH, out_file) + + assert out_file.is_file() + assert "# DNS Tunneling Data Exfiltration Telemetry Package" in rendered + assert "Apache-2.0" in rendered + assert "Google Cloud Security" in rendered + assert "2.0.0" in rendered + + # OCSF classes and paired baselines + assert "DNS Activity" in rendered + assert "4003" in rendered + assert "User Inventory" in rendered + assert "5003" in rendered + assert "CYMBAL_BASELINE_NETWORK" in rendered + + # Platform bindings + assert "Google SecOps" in rendered + assert "Databricks" in rendered + assert "secops_ocsf.dns_activity" in rendered + assert "Sigma" in rendered + assert "RULES/sigma_dns_tunneling.yml" in rendered + assert "Splunk" in rendered + assert "zeek:dns:json" in rendered + + # Fallback renderer parity + normalized = normalize_frontmatter_for_template(v2) + fallback_rendered = _render_without_jinja2(normalized) + assert "DNS Activity" in fallback_rendered + assert "CYMBAL_BASELINE_NETWORK" in fallback_rendered + assert "Databricks" in fallback_rendered + assert "Sigma" in fallback_rendered + assert "Splunk" in fallback_rendered + + def test_process_usecase_directory_and_parse_frontmatter(self, tmp_path: Path): + uc_dir = tmp_path / "SAMPLE_UC" + uc_dir.mkdir() + md_path = uc_dir / "SAMPLE_UC.md" + md_path.write_text( + "---\ntitle: SAMPLE_UC\ndescription: Sample description\n---\n\n# Body\n", + encoding="utf-8", + ) + + fm, body = parse_frontmatter(md_path) + assert fm["title"] == "SAMPLE_UC" + assert "# Body" in body + + generated = process_usecase_directory(uc_dir, TEMPLATE_PATH) + assert generated is not None + assert "# SAMPLE_UC" in generated + assert (uc_dir / "SAMPLE_UC_generated.md").is_file() + + +class TestBuiltinUsecasesStrictV2: + """Verify built-in logstory usecases are migrated to v2 and pass all 5 gates.""" + + @pytest.mark.parametrize( + "usecase_rel_path", + [ + "usecases/NETWORK_ANALYSIS", + "usecases/RULES_SEARCH_WORKSHOP", + "src/logstory/usecases/NETWORK_ANALYSIS", + "src/logstory/usecases/RULES_SEARCH_WORKSHOP", + ], + ) + def test_builtin_usecase_validates_strict_v2(self, usecase_rel_path: str): + usecase_dir = REPO_ROOT / usecase_rel_path + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is True, f"Validation failed: {result.findings}" + assert result.schema_version == "2.0" + assert result.errors == [] + assert result.warnings == [] + + def test_network_analysis_package_contents_and_symlink(self): + symlink_eval = REPO_ROOT / "src/logstory/usecases/NETWORK_ANALYSIS/eval.yaml" + assert symlink_eval.is_symlink() + assert symlink_eval.is_file() + + pkg = parse_usecase_package(REPO_ROOT / "usecases/NETWORK_ANALYSIS", blind=True) + assert pkg.schema_version == "2.0" + assert pkg.manifest is not None + assert pkg.manifest.id == "NETWORK_ANALYSIS" + assert pkg.solution_markdown_body is None + assert pkg.eval_data is None + + full_pkg = parse_usecase_package( + REPO_ROOT / "usecases/NETWORK_ANALYSIS", blind=False + ) + assert full_pkg.eval_data is not None + assert full_pkg.eval_data.usecase_id == "NETWORK_ANALYSIS" + + def test_rules_search_workshop_telemetry_and_detections(self): + pkg = parse_usecase_package( + REPO_ROOT / "usecases/RULES_SEARCH_WORKSHOP", blind=False + ) + assert pkg.schema_version == "2.0" + assert pkg.manifest is not None + assert pkg.manifest.id == "RULES_SEARCH_WORKSHOP" + + event_paths = [s.path for s in pkg.manifest.telemetry.events] + assert event_paths == [ + "EVENTS/POWERSHELL.log", + "EVENTS/WINDOWS_DEFENDER_AV.log", + "EVENTS/WINDOWS_SYSMON.log", + "EVENTS/WINEVTLOG.log", + ] + + entity_paths = [s.path for s in pkg.manifest.telemetry.entities] + assert entity_paths == ["ENTITIES/WINDOWS_AD.log"] + + secops = pkg.manifest.platforms.google_secops + assert secops is not None + rule_paths = [d.path for d in secops.detections if d.path is not None] + assert sorted(rule_paths) == [ + "RULES/mitre_attack_T1021_002_windows_admin_share_basic.yl2", + "RULES/rw_mimikatz_T1003.yl2", + "RULES/win_password_spray.yl2", + "RULES/win_repeatedAuthFailure_thenSuccess.yl2", + ] + assert pkg.eval_data is not None + assert pkg.eval_data.usecase_id == "RULES_SEARCH_WORKSHOP" + + @pytest.mark.parametrize( + "usecase_name", ["NETWORK_ANALYSIS", "RULES_SEARCH_WORKSHOP"] + ) + def test_builtin_usecase_process_directory_generates_docs( + self, tmp_path: Path, usecase_name: str + ): + usecase_dir = REPO_ROOT / "usecases" / usecase_name + rendered = process_usecase_directory( + usecase_dir, TEMPLATE_PATH, output_dir=tmp_path + ) + assert rendered is not None + assert f"# {usecase_name}" in rendered + assert "OCSF Class" in rendered + assert "Platform Bindings" in rendered + assert (tmp_path / f"{usecase_name}_generated.md").is_file() diff --git a/tests/test_logstory_usecases_corpus_v2.py b/tests/test_logstory_usecases_corpus_v2.py new file mode 100644 index 0000000..5d7424d --- /dev/null +++ b/tests/test_logstory_usecases_corpus_v2.py @@ -0,0 +1,172 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Full 47-scenario corpus validation tests for logstory_usecases v2 migration.""" + +from __future__ import annotations + +import json +import os +import re +from pathlib import Path + +import pytest + +from logstory.schema import ( + build_eval_json_schema, + build_usecase_json_schema, + parse_usecase_package, + validate_usecase_corpus, +) + +EXPECTED_USECASE_COUNT = 47 +EXPECTED_ATTACK_COUNT = 43 +EXPECTED_BASELINE_IDS = frozenset({ + "CYMBAL_BASELINE_CLOUD", + "CYMBAL_BASELINE_ENDPOINT", + "CYMBAL_BASELINE_IDENTITY", + "CYMBAL_BASELINE_NETWORK", +}) + +_EMOJI_RE = re.compile( + "[\U0001f300-\U0001f9ff\U0001fa00-\U0001faff\u2600-\u26ff\u2700-\u27bf]" +) + + +def _resolve_usecases_corpus_dir() -> Path | None: + """Locate the logstory_usecases worktree directory if mounted locally.""" + env_override = os.environ.get("LOGSTORY_USECASES_DIR", "").strip() + if env_override: + env_path = Path(env_override) + if env_path.is_dir(): + return env_path + + repo_root = Path(__file__).resolve().parent.parent + projects_dir = repo_root.parent.parent + candidates = [ + projects_dir / "logstory_usecases__worktrees" / "schema__usecase-metadata-v2", + projects_dir / "logstory_usecases__worktrees" / "main", + projects_dir / "logstory_usecases", + ] + for candidate in candidates: + if candidate.is_dir() and (candidate / "AKIRA_RANSOMWARE").is_dir(): + return candidate + return None + + +@pytest.fixture(scope="module", name="corpus_dir") +def fixture_corpus_dir() -> Path: + """Return the mounted logstory_usecases corpus directory or skip cleanly.""" + resolved = _resolve_usecases_corpus_dir() + if resolved is None: + pytest.skip("logstory_usecases repository is not mounted locally") + return resolved + + +class TestLogstoryUsecasesCorpusV2: + """Validate all 47 scenarios in logstory_usecases against v2 5-gate checks.""" + + def test_all_47_usecases_pass_strict_v2_validation(self, corpus_dir: Path): + results = validate_usecase_corpus(corpus_dir, strict_v2=True) + assert len(results) == EXPECTED_USECASE_COUNT + + failures = { + r.usecase_id: [f.to_dict() for f in r.findings] + for r in results + if not r.is_valid or r.warnings + } + assert failures == {}, f"Corpus validation findings: {failures}" + assert all(r.schema_version == "2.0" for r in results) + + def test_synchronized_json_schemas_exist_in_corpus(self, corpus_dir: Path): + usecase_schema_path = corpus_dir / "schemas" / "usecase.schema.json" + eval_schema_path = corpus_dir / "schemas" / "eval.schema.json" + assert usecase_schema_path.is_file() + assert eval_schema_path.is_file() + + assert ( + json.loads(usecase_schema_path.read_text(encoding="utf-8")) + == build_usecase_json_schema() + ) + assert ( + json.loads(eval_schema_path.read_text(encoding="utf-8")) + == build_eval_json_schema() + ) + + def test_blind_and_full_parsing_and_ocsf_mappings_across_corpus( + self, corpus_dir: Path + ): + results = validate_usecase_corpus(corpus_dir, strict_v2=True) + assert len(results) == EXPECTED_USECASE_COUNT + + attack_count = 0 + baseline_ids: set[str] = set() + + for res in results: + uc_dir = res.usecase_dir + blind_pkg = parse_usecase_package(uc_dir, blind=True) + assert blind_pkg.schema_version == "2.0" + assert blind_pkg.manifest is not None + assert blind_pkg.eval_data is None + assert blind_pkg.solution_markdown_body is None + assert f"# {res.usecase_id}" in blind_pkg.public_markdown_body + assert "## Overview" in blind_pkg.public_markdown_body + assert "## Telemetry Streams" in blind_pkg.public_markdown_body + assert "## Replay Instructions" in blind_pkg.public_markdown_body + + full_pkg = parse_usecase_package(uc_dir, blind=False) + assert full_pkg.eval_data is not None + assert full_pkg.solution_markdown_body is not None + + manifest = full_pkg.manifest + assert manifest is not None + eval_data = full_pkg.eval_data + + all_streams = list(manifest.telemetry.events) + list(manifest.telemetry.entities) + assert len(all_streams) >= 1 + secops = manifest.platforms.google_secops + databricks = manifest.platforms.databricks + assert secops is not None + assert databricks is not None + + for stream in all_streams: + assert stream.ocsf_category_uid is not None + assert stream.ocsf_class_uid is not None + assert stream.ocsf_class_name + assert stream.stream_id in secops.log_type_mappings + assert stream.stream_id in databricks.table_mappings + assert databricks.table_mappings[stream.stream_id].startswith("secops_ocsf.") + + if res.usecase_id in EXPECTED_BASELINE_IDS: + baseline_ids.add(res.usecase_id) + assert manifest.environment.scenario_type.value == "benign_baseline" + assert eval_data.ground_truth.expected_verdict.value == "BENIGN" + assert eval_data.ground_truth.expected_disposition.value == "BENIGN_POSITIVE" + assert eval_data.counterfactual_pair is not None + else: + attack_count += 1 + assert manifest.environment.scenario_type.value == "attack" + assert eval_data.ground_truth.expected_verdict.value == "MALICIOUS" + assert eval_data.ground_truth.expected_disposition.value == "TRUE_POSITIVE" + assert len(eval_data.ground_truth.kill_chain) >= 1 + + md_text = (uc_dir / f"{res.usecase_id}.md").read_text(encoding="utf-8") + eval_text = (uc_dir / "eval.yaml").read_text(encoding="utf-8") + assert not _EMOJI_RE.search(md_text), f"Emoji found in {res.usecase_id}.md" + assert not _EMOJI_RE.search( + eval_text + ), f"Emoji found in {res.usecase_id}/eval.yaml" + + assert attack_count == EXPECTED_ATTACK_COUNT + assert baseline_ids == EXPECTED_BASELINE_IDS diff --git a/tests/test_usecase_cli_metadata.py b/tests/test_usecase_cli_metadata.py new file mode 100644 index 0000000..40ea0c7 --- /dev/null +++ b/tests/test_usecase_cli_metadata.py @@ -0,0 +1,564 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""CLI tests for usecases validate, inspect, export-schemas, and paired baselines.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any +from unittest.mock import patch + +import yaml +from typer.testing import CliRunner + +from logstory.logstory import _get_paired_baselines, app + +runner = CliRunner() + + +def _valid_manifest_dict( + usecase_id: str = "DNS_TUNNELING_EXFIL_GTI", + paired_baselines: list[str] | None = None, +) -> dict[str, Any]: + """Return a valid v2 SecurityUsecaseManifest dictionary.""" + return { + "schema_version": "2.0", + "id": usecase_id, + "title": f"{usecase_id} Telemetry Package", + "version": "2.0.0", + "created": "2026-10-18", + "updated": "2026-11-05", + "author": "Google Cloud Security", + "license": "Apache-2.0", + "description": ( + "Enterprise DNS and directory telemetry package for network protocol" + " analysis." + ), + "tags": ["network-c2", "dns", "ocsf", "google-secops"], + "environment": { + "organization": "cymbal-investments.org", + "scenario_type": "attack", + "paired_baselines": ( + paired_baselines + if paired_baselines is not None + else ["CYMBAL_BASELINE_NETWORK"] + ), + "timestamp_policy": { + "base_date": "2026-10-18", + "preserve_relative_offsets": True, + "default_delta": "1d", + "entity_offset_days": 1, + }, + }, + "telemetry": { + "events": [{ + "stream_id": "zeek_dns", + "path": "EVENTS/BRO_JSON.log", + "log_type": "BRO_JSON", + "vendor_name": "Zeek", + "product_name": "Bro", + "ocsf_category_uid": 4, + "ocsf_class_uid": 4003, + "ocsf_class_name": "DNS Activity", + }], + "entities": [{ + "stream_id": "windows_ad_users", + "path": "ENTITIES/WINDOWS_AD.log", + "log_type": "WINDOWS_AD", + "vendor_name": "Microsoft", + "product_name": "Active Directory", + "ocsf_category_uid": 5, + "ocsf_class_uid": 5003, + "ocsf_class_name": "User Inventory", + }], + }, + "platforms": { + "google_secops": { + "run_frequency": { + "events": "Every 1 day", + "entities": "Every 1 day (offset 1 day prior)", + }, + "data_rbac": "Standard RBAC", + "log_type_mappings": { + "zeek_dns": "BRO_JSON", + "windows_ad_users": "WINDOWS_AD", + }, + "detections": [{ + "name": "dns_tunneling_low_reputation_domain", + "path": "RULES/dns_tunneling_low_reputation_domain.yl2", + "live": True, + "alerting": True, + }], + }, + }, + } + + +def _valid_eval_dict( + usecase_id: str = "DNS_TUNNELING_EXFIL_GTI", + paired_baseline_id: str | None = "CYMBAL_BASELINE_NETWORK", +) -> dict[str, Any]: + """Return a valid v2 SecurityUsecaseEval dictionary.""" + eval_dict: dict[str, Any] = { + "schema_version": "2.0", + "usecase_id": usecase_id, + "threat_context": { + "grounding_tier": "stix_grounded", + "difficulty_tier": "advanced", + "threat_actor": "UNC-STRATIO", + "actor_aliases": ["StratioGroup"], + "campaign_name": "Operation HexSubdomain", + "cisa_advisories": ["AA24-242A"], + "cve_ids": ["CVE-2024-12345"], + "mitre_tactics": ["TA0011", "TA0010"], + "mitre_techniques": ["T1071.004", "T1048.003"], + }, + "tasks": { + "alert_triage": { + "enabled": True, + "case_id": "CASE-4012", + "alert_ids": ["de_4cfa127e-8a09-3f1b-05b8-588dbfaf257b"], + "prompt": "Triage CASE-4012 and determine disposition.", + "expected_skill": "dns-tunneling-network-beacon-hunt", + "time_horizon_hours": 72, + }, + "threat_hunt": { + "enabled": True, + "prompt_unalerted": "Hunt across zeek_dns and windows_ad_users.", + "expected_patient_zero": "fin-wks-20", + "time_horizon_hours": 168, + }, + "detection_engineering": { + "enabled": True, + "prompt": "Author a detection rule for DNS tunneling.", + "target_techniques": ["T1071.004"], + "reference_rules": ["RULES/dns_tunneling_low_reputation_domain.yl2"], + "expected_rule_count_min": 1, + "max_false_positives_on_paired_baseline": 0, + }, + }, + "ground_truth": { + "expected_disposition": "TRUE_POSITIVE", + "expected_verdict": "MALICIOUS", + "severity": "HIGH", + "patient_zero_host": "fin-wks-20", + "patient_zero_account": "alice.smith", + "root_cause_summary": "DNS tunneling exfiltration to stratioai.org.", + "kill_chain": [{ + "stage": "exfiltration", + "stream_id": "zeek_dns", + "timestamp_utc": "2026-10-18T11:20:00Z", + "tactic": "TA0010", + "technique_id": "T1048.003", + "summary": "Hex-encoded subdomain DNS queries to C2 domain.", + }], + "entities_and_iocs": { + "compromised_accounts": ["alice.smith"], + "affected_assets": [{ + "hostname": "fin-wks-20", + "criticality_tier": "tier_2_workstation", + "ip_addresses": ["10.128.0.20"], + "patient_zero": True, + "blast_radius_fp_usd": 500.0, + "blast_radius_fn_usd": 50000.0, + }], + "malicious_iocs": [{ + "type": "domain", + "value": "stratioai.org", + "role": "c2_domain", + }], + }, + }, + "compliance_gates": { + "requires_pre_containment_verification": True, + "requires_168h_horizon_widening": False, + "forbid_external_cti_on_internal_ips": True, + "forbid_lolbin_hash_blocking": True, + }, + } + if paired_baseline_id is not None: + eval_dict["counterfactual_pair"] = { + "paired_usecase_id": paired_baseline_id, + "pair_relationship": "malicious_variant_of_benign_baseline", + "shared_entities": ["fin-wks-20", "alice.smith"], + "shared_telemetry_streams": ["zeek_dns"], + "discriminating_invariant": "Subdomain labels exceed 40 hex chars.", + } + return eval_dict + + +def _write_usecase_fixture( + root_dir: Path, + usecase_name: str = "DNS_TUNNELING_EXFIL_GTI", + manifest: dict[str, Any] | None = None, + eval_data: dict[str, Any] | None = None, + public_body: str = "# DNS_TUNNELING_EXFIL_GTI\n\nBlind-safe public overview.", + solution_body: str | None = ( + "## Analyst Solution\n\n" + "Compromised account alice.smith exfiltrated data to stratioai.org " + "matching advisory AA24-242A." + ), + create_files: bool = True, +) -> Path: + """Materialize a usecase directory fixture on disk for CLI tests.""" + usecase_dir = root_dir / usecase_name + usecase_dir.mkdir(parents=True, exist_ok=True) + + manifest_dict = ( + manifest if manifest is not None else _valid_manifest_dict(usecase_name) + ) + frontmatter_yaml = yaml.safe_dump(manifest_dict, sort_keys=False) + if solution_body is not None: + md_text = ( + f"---\n{frontmatter_yaml}---\n\n{public_body}\n\n" + f"\n\n{solution_body}\n" + ) + else: + md_text = f"---\n{frontmatter_yaml}---\n\n{public_body}\n" + + (usecase_dir / f"{usecase_name}.md").write_text(md_text, encoding="utf-8") + + eval_dict = eval_data if eval_data is not None else _valid_eval_dict(usecase_name) + (usecase_dir / "eval.yaml").write_text( + yaml.safe_dump(eval_dict, sort_keys=False), encoding="utf-8" + ) + + if create_files: + events_dir = usecase_dir / "EVENTS" + entities_dir = usecase_dir / "ENTITIES" + rules_dir = usecase_dir / "RULES" + events_dir.mkdir(parents=True, exist_ok=True) + entities_dir.mkdir(parents=True, exist_ok=True) + rules_dir.mkdir(parents=True, exist_ok=True) + (events_dir / "BRO_JSON.log").write_text('{"ts": "2026-10-18"}\n', encoding="utf-8") + (entities_dir / "WINDOWS_AD.log").write_text( + '{"user": "alice"}\n', encoding="utf-8" + ) + (rules_dir / "dns_tunneling_low_reputation_domain.yl2").write_text( + "rule dns_tunneling_low_reputation_domain {}\n", encoding="utf-8" + ) + + return usecase_dir + + +class TestUsecasesValidateCli: + """Tests for `logstory usecases validate` subcommand.""" + + def test_validate_single_usecase_dir_json_and_text(self, tmp_path: Path): + usecase_dir = _write_usecase_fixture(tmp_path) + + res_json = runner.invoke( + app, + ["usecases", "validate", str(usecase_dir), "--strict-v2", "--format", "json"], + ) + assert res_json.exit_code == 0, res_json.stdout + payload = json.loads(res_json.stdout) + assert payload["valid"] is True + assert payload["total"] == 1 + assert payload["passed"] == 1 + assert payload["failed"] == 0 + assert payload["results"][0]["usecase_id"] == "DNS_TUNNELING_EXFIL_GTI" + assert payload["results"][0]["is_valid"] is True + + res_text = runner.invoke( + app, + ["usecases", "validate", str(usecase_dir), "--strict-v2", "--format", "text"], + ) + assert res_text.exit_code == 0, res_text.stdout + assert "DNS_TUNNELING_EXFIL_GTI" in res_text.stdout + assert "PASS" in res_text.stdout + + def test_validate_corpus_directory_success_and_failure(self, tmp_path: Path): + _write_usecase_fixture( + tmp_path, + usecase_name="CYMBAL_BASELINE_NETWORK", + manifest=_valid_manifest_dict("CYMBAL_BASELINE_NETWORK", paired_baselines=[]), + eval_data=_valid_eval_dict("CYMBAL_BASELINE_NETWORK", paired_baseline_id=None), + ) + _write_usecase_fixture( + tmp_path, + usecase_name="DNS_TUNNELING_EXFIL_GTI", + ) + + res = runner.invoke( + app, + ["usecases", "validate", str(tmp_path), "--strict-v2", "--format", "json"], + ) + assert res.exit_code == 0, res.stdout + payload = json.loads(res.stdout) + assert payload["valid"] is True + assert payload["total"] == 2 + assert payload["passed"] == 2 + + # Introduce spoiler leakage into public markdown body to fail Gate 3 + bad_dir = _write_usecase_fixture( + tmp_path, + usecase_name="SPOILER_USECASE", + manifest=_valid_manifest_dict("SPOILER_USECASE", paired_baselines=[]), + eval_data=_valid_eval_dict("SPOILER_USECASE", paired_baseline_id=None), + public_body="Public text leaking stratioai.org before delimiter.", + ) + assert bad_dir.is_dir() + + res_fail = runner.invoke( + app, + ["usecases", "validate", str(tmp_path), "--strict-v2", "--format", "json"], + ) + assert res_fail.exit_code == 1 + fail_payload = json.loads(res_fail.stdout) + assert fail_payload["valid"] is False + assert fail_payload["failed"] == 1 + + res_fail_text = runner.invoke( + app, + ["usecases", "validate", str(tmp_path), "--strict-v2", "--format", "text"], + ) + assert res_fail_text.exit_code == 1 + assert "FAIL" in res_fail_text.stdout + assert "G3_ANTI_SPOILER" in res_fail_text.stdout + + def test_validate_installed_usecases_and_missing_target(self): + res_all = runner.invoke( + app, ["usecases", "validate", "--strict-v2", "--format", "json"] + ) + assert res_all.exit_code == 0, res_all.stdout + payload = json.loads(res_all.stdout) + assert payload["valid"] is True + assert payload["total"] >= 2 + + res_named = runner.invoke( + app, + ["usecases", "validate", "NETWORK_ANALYSIS", "--strict-v2", "--format", "json"], + ) + assert res_named.exit_code == 0, res_named.stdout + named_payload = json.loads(res_named.stdout) + assert named_payload["results"][0]["usecase_id"] == "NETWORK_ANALYSIS" + + res_missing = runner.invoke( + app, ["usecases", "validate", "NONEXISTENT_USECASE_XYZ"] + ) + assert res_missing.exit_code == 1 + + +class TestUsecasesInspectCli: + """Tests for `logstory usecases inspect` subcommand.""" + + def test_inspect_blind_strips_solution_and_eval(self, tmp_path: Path): + usecase_dir = _write_usecase_fixture(tmp_path) + + res_blind = runner.invoke( + app, + ["usecases", "inspect", str(usecase_dir), "--blind", "--format", "json"], + ) + assert res_blind.exit_code == 0, res_blind.stdout + blind_data = json.loads(res_blind.stdout) + assert blind_data["manifest"]["id"] == "DNS_TUNNELING_EXFIL_GTI" + assert "Blind-safe public overview." in blind_data["public_markdown_body"] + assert "stratioai.org" not in res_blind.stdout + assert "alice.smith" not in res_blind.stdout + assert "AA24-242A" not in res_blind.stdout + assert blind_data["solution_markdown_body"] is None + assert blind_data["eval_data"] is None + + res_blind_text = runner.invoke( + app, + ["usecases", "inspect", str(usecase_dir), "--blind", "--format", "text"], + ) + assert res_blind_text.exit_code == 0, res_blind_text.stdout + assert "DNS_TUNNELING_EXFIL_GTI" in res_blind_text.stdout + assert "stratioai.org" not in res_blind_text.stdout + + def test_inspect_include_eval_returns_eval_and_solution(self, tmp_path: Path): + usecase_dir = _write_usecase_fixture(tmp_path) + + res_full = runner.invoke( + app, + ["usecases", "inspect", str(usecase_dir), "--include-eval", "--format", "json"], + ) + assert res_full.exit_code == 0, res_full.stdout + full_data = json.loads(res_full.stdout) + assert full_data["eval_data"] is not None + assert full_data["eval_data"]["usecase_id"] == "DNS_TUNNELING_EXFIL_GTI" + assert full_data["solution_markdown_body"] is not None + assert "stratioai.org" in full_data["solution_markdown_body"] + + res_full_text = runner.invoke( + app, + ["usecases", "inspect", str(usecase_dir), "--include-eval", "--format", "text"], + ) + assert res_full_text.exit_code == 0, res_full_text.stdout + assert "TRUE_POSITIVE" in res_full_text.stdout + + def test_inspect_installed_usecase_and_missing(self): + res_installed = runner.invoke( + app, ["usecases", "inspect", "NETWORK_ANALYSIS", "--blind", "--format", "json"] + ) + assert res_installed.exit_code == 0, res_installed.stdout + data = json.loads(res_installed.stdout) + assert data["manifest"]["id"] == "NETWORK_ANALYSIS" + assert data["eval_data"] is None + + res_missing = runner.invoke(app, ["usecases", "inspect", "NONEXISTENT_USECASE_XYZ"]) + assert res_missing.exit_code == 1 + + +class TestUsecasesExportSchemasCli: + """Tests for `logstory usecases export-schemas` subcommand.""" + + def test_export_schemas_writes_json_schema_files(self, tmp_path: Path): + out_dir = tmp_path / "exported_schemas" + res = runner.invoke( + app, ["usecases", "export-schemas", "--output-dir", str(out_dir)] + ) + assert res.exit_code == 0, res.stdout + usecase_schema = out_dir / "usecase.schema.json" + eval_schema = out_dir / "eval.schema.json" + assert usecase_schema.is_file() + assert eval_schema.is_file() + + uc_json = json.loads(usecase_schema.read_text(encoding="utf-8")) + ev_json = json.loads(eval_schema.read_text(encoding="utf-8")) + assert uc_json["title"] == "SecurityUsecaseManifest" + assert ev_json["title"] == "SecurityUsecaseEval" + + +class TestReplayWithPairedBaselines: + """Tests for `logstory replay usecase --with-paired-baselines`.""" + + def test_replay_usecase_with_paired_baselines_deduplicates_and_replays_first( + self, tmp_path: Path + ): + manifest = _valid_manifest_dict( + "ATTACK_UC", + paired_baselines=[ + "CYMBAL_BASELINE_NETWORK", + "CYMBAL_BASELINE_ENDPOINT", + "CYMBAL_BASELINE_NETWORK", + ], + ) + _write_usecase_fixture( + tmp_path, + usecase_name="ATTACK_UC", + manifest=manifest, + eval_data=_valid_eval_dict("ATTACK_UC"), + ) + + replayed_calls: list[tuple[str, str, bool]] = [] + + def fake_replay_logtype( + use_case, + log_type, + _logstory_exe_time, + old_base_time, + timestamp_delta=None, + entities=False, + local_file_output=False, + ): + del timestamp_delta, local_file_output + replayed_calls.append((use_case, log_type, entities)) + return old_base_time + + def fake_get_logtypes(_usecase: str, entities: bool = False) -> list[str]: + if entities: + return ["WINDOWS_AD"] + return ["BRO_JSON"] + + with ( + patch( + "logstory.logstory.get_usecases", + return_value=[ + "ATTACK_UC", + "CYMBAL_BASELINE_NETWORK", + "CYMBAL_BASELINE_ENDPOINT", + ], + ), + patch("logstory.logstory._get_logtypes", side_effect=fake_get_logtypes), + patch( + "logstory.logstory.imported_main.usecase_replay_logtype", + side_effect=fake_replay_logtype, + ), + patch( + "logstory.logstory._get_paired_baselines", + side_effect=lambda uc: _get_paired_baselines(uc, search_roots=[tmp_path]), + ), + ): + res = runner.invoke( + app, + [ + "replay", + "usecase", + "ATTACK_UC", + "--with-paired-baselines", + "--local-file-output", + "--no-get", + ], + ) + assert res.exit_code == 0, res.stdout + replayed_usecases = [call[0] for call in replayed_calls] + assert replayed_usecases == [ + "CYMBAL_BASELINE_NETWORK", + "CYMBAL_BASELINE_ENDPOINT", + "ATTACK_UC", + ] + + def test_replay_usecase_with_paired_baselines_downloads_when_get_enabled( + self, tmp_path: Path + ): + manifest = _valid_manifest_dict( + "ATTACK_UC", + paired_baselines=["CYMBAL_BASELINE_NETWORK"], + ) + _write_usecase_fixture( + tmp_path, + usecase_name="ATTACK_UC", + manifest=manifest, + eval_data=_valid_eval_dict("ATTACK_UC"), + ) + + installed = ["ATTACK_UC"] + downloaded: list[str] = [] + + def fake_download(uc: str, _bucket: str | None = None) -> bool: + downloaded.append(uc) + installed.append(uc) + return True + + with ( + patch("logstory.logstory.get_usecases", side_effect=lambda: list(installed)), + patch("logstory.logstory._download_usecase", side_effect=fake_download), + patch("logstory.logstory._get_logtypes", return_value=["BRO_JSON"]), + patch( + "logstory.logstory.imported_main.usecase_replay_logtype", + return_value=None, + ), + patch( + "logstory.logstory._get_paired_baselines", + side_effect=lambda uc: _get_paired_baselines(uc, search_roots=[tmp_path]), + ), + ): + res = runner.invoke( + app, + [ + "replay", + "usecase", + "ATTACK_UC", + "--with-paired-baselines", + "--local-file-output", + "--get", + ], + ) + assert res.exit_code == 0, res.stdout + assert downloaded == ["CYMBAL_BASELINE_NETWORK"] diff --git a/tests/test_usecase_schema_models.py b/tests/test_usecase_schema_models.py new file mode 100644 index 0000000..685f5ec --- /dev/null +++ b/tests/test_usecase_schema_models.py @@ -0,0 +1,589 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for v2 security usecase and Blue Team eval schema models.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import pytest +import yaml + +from logstory.schema import ( + AffectedAsset, + AlertTriageTask, + BenignConfusers, + ChangeWindow, + ComplianceGates, + CounterfactualPair, + CriticalityTier, + DatabricksPlatform, + DetectionEngineeringTask, + DifficultyTier, + EntitiesAndIocs, + EnvironmentContext, + EvaluationTasks, + ExpectedActions, + ExpectedDisposition, + ExpectedVerdict, + GoogleSecOpsDetection, + GoogleSecOpsPlatform, + GoogleSecOpsReferenceList, + GoogleSecOpsRunFrequency, + GoogleSecOpsSavedSearch, + GroundingTier, + GroundTruth, + KillChainStage, + MaliciousIoc, + PairRelationship, + ParsedUsecasePackage, + PlatformBindings, + ReferenceQueries, + ScenarioType, + SchemaValidationError, + SecurityUsecaseEval, + SecurityUsecaseManifest, + SigmaPlatform, + SplunkPlatform, + TelemetrySection, + TelemetryStream, + ThreatContext, + ThreatHuntTask, + TimestampPolicy, + build_eval_json_schema, + build_usecase_json_schema, + export_json_schemas, + parse_usecase_package, + split_frontmatter_and_body, + split_solution_section, +) + + +def _valid_manifest_dict() -> dict[str, Any]: + """Return a complete valid v2 SecurityUsecaseManifest dictionary.""" + return { + "schema_version": "2.0", + "id": "DNS_TUNNELING_EXFIL_GTI", + "title": "DNS Tunneling Data Exfiltration Telemetry Package", + "version": "2.0.0", + "created": "2026-10-18", + "updated": "2026-11-05", + "author": "Google Cloud Security", + "license": "Apache-2.0", + "description": ( + "Enterprise DNS and endpoint process telemetry package for network" + " protocol analysis." + ), + "tags": [ + "network-c2", + "dns", + "ocsf", + "google-secops", + "databricks", + ], + "environment": { + "organization": "cymbal-investments.org", + "scenario_type": "attack", + "paired_baselines": ["CYMBAL_BASELINE_NETWORK"], + "timestamp_policy": { + "base_date": "2026-10-18", + "preserve_relative_offsets": True, + "default_delta": "1d", + "entity_offset_days": 1, + }, + }, + "telemetry": { + "events": [{ + "stream_id": "zeek_dns", + "path": "EVENTS/BRO_JSON.log", + "log_type": "BRO_JSON", + "vendor_name": "Zeek", + "product_name": "Bro", + "ocsf_category_uid": 4, + "ocsf_class_uid": 4003, + "ocsf_class_name": "DNS Activity", + "notes": "Core DNS resolution telemetry.", + }], + "entities": [{ + "stream_id": "windows_ad_users", + "path": "ENTITIES/WINDOWS_AD.log", + "log_type": "WINDOWS_AD", + "vendor_name": "Microsoft", + "product_name": "Active Directory", + "ocsf_category_uid": 5, + "ocsf_class_uid": 5003, + "ocsf_class_name": "User Inventory", + "notes": "Directory user context.", + }], + }, + "platforms": { + "google_secops": { + "run_frequency": { + "events": "Every 1 day", + "entities": "Every 1 day (offset 1 day prior)", + }, + "data_rbac": "Standard RBAC", + "log_type_mappings": { + "zeek_dns": "BRO_JSON", + "windows_ad_users": "WINDOWS_AD", + }, + "detections": [{ + "name": "dns_tunneling_low_reputation_domain", + "path": "RULES/dns_tunneling_low_reputation_domain.yl2", + "live": True, + "alerting": True, + "notes": "Detects high-entropy DNS subdomain queries.", + }], + "saved_searches": [{ + "name": "High Entropy DNS Queries", + "creator": "secops-team@cymbal-investments.org", + "notes": "Baseline DNS query length exploration.", + }], + "reference_lists": [{ + "name": "approved_dns_resolvers", + "type": "String", + "notes": "Internal corporate DNS resolvers.", + }], + }, + "databricks": { + "catalog": "security_lakehouse", + "schema_name": "ocsf_silver", + "table_mappings": { + "zeek_dns": "secops_ocsf.dns_activity", + "windows_ad_users": "secops_ocsf.user_inventory", + }, + "detections": ["RULES/databricks_dns_tunneling.sql"], + }, + "sigma": { + "rules": ["RULES/sigma_dns_tunneling.yml"], + }, + "splunk": { + "sourcetypes": { + "zeek_dns": "zeek:dns:json", + "windows_ad_users": "ms:ad:user", + }, + }, + }, + } + + +def _valid_eval_dict() -> dict[str, Any]: + """Return a complete valid v2 SecurityUsecaseEval dictionary.""" + udm_query = ( + 'metadata.event_type = "NETWORK_DNS" AND' + " network.dns.questions.name = /stratioai\\.org$/" + ) + databricks_sql_query = ( + "SELECT * FROM security_lakehouse.ocsf_silver.dns_activity" + " WHERE query_hostname LIKE '%.stratioai.org'" + ) + return { + "schema_version": "2.0", + "usecase_id": "DNS_TUNNELING_EXFIL_GTI", + "threat_context": { + "grounding_tier": "stix_grounded", + "difficulty_tier": "advanced", + "threat_actor": "UNC-STRATIO", + "actor_aliases": ["StratioGroup"], + "campaign_name": "Operation HexSubdomain", + "cisa_advisories": ["AA24-242A"], + "cve_ids": ["CVE-2024-12345"], + "mitre_tactics": ["TA0011", "TA0010"], + "mitre_techniques": ["T1071.004", "T1048.003"], + "release_day": 14, + }, + "tasks": { + "alert_triage": { + "enabled": True, + "case_id": "CASE-4012", + "alert_ids": ["de_4cfa127e-8a09-3f1b-05b8-588dbfaf257b"], + "prompt": ( + "Triage CASE-4012 and determine disposition, affected assets," + " and containment actions." + ), + "expected_skill": "dns-tunneling-network-beacon-hunt", + "skill_bindings": { + "google_secops": "dns-tunneling-network-beacon-hunt", + "goose": "dns-tunneling-network-beacon-hunt", + "databricks": "ocsf-dns-beacon-hunt", + }, + "time_horizon_hours": 72, + }, + "threat_hunt": { + "enabled": True, + "prompt_unalerted": ( + "Hunt across zeek_dns and windows_ad_users streams for" + " covert C2 or DNS exfiltration activity." + ), + "prompt_intel_driven": ( + "Sweep telemetry for indicators associated with advisory AA24-242A." + ), + "expected_patient_zero": "fin-wks-20", + "time_horizon_hours": 168, + }, + "detection_engineering": { + "enabled": True, + "prompt": ( + "Author a detection rule identifying hex-encoded subdomain" + " exfiltration over DNS." + ), + "target_techniques": ["T1071.004", "T1048.003"], + "reference_rules": ["RULES/dns_tunneling_low_reputation_domain.yl2"], + "expected_rule_count_min": 1, + "max_false_positives_on_paired_baseline": 0, + }, + }, + "ground_truth": { + "expected_disposition": "TRUE_POSITIVE", + "expected_verdict": "MALICIOUS", + "severity": "HIGH", + "patient_zero_host": "fin-wks-20", + "patient_zero_account": "alice.smith", + "root_cause_summary": ( + "Compromised finance workstation exfiltrating sensitive payload" + " chunks via hex-encoded DNS TXT/A queries to stratioai.org." + ), + "kill_chain": [{ + "stage": "exfiltration", + "stream_id": "zeek_dns", + "timestamp_utc": "2026-10-18T11:20:00Z", + "tactic": "TA0010", + "technique_id": "T1048.003", + "summary": "Hex-encoded subdomain DNS queries to C2 domain.", + "key_evidence": [ + "4a6f686e446f655365637265745061796c6f6164.t.stratioai.org" + ], + }], + "entities_and_iocs": { + "compromised_accounts": ["alice.smith"], + "affected_assets": [{ + "hostname": "fin-wks-20", + "criticality_tier": "tier_2_workstation", + "ip_addresses": ["10.128.0.20"], + "role": "Finance analyst workstation", + "patient_zero": True, + "blast_radius_fp_usd": 500.0, + "blast_radius_fn_usd": 50000.0, + }], + "malicious_iocs": [ + { + "type": "domain", + "value": "stratioai.org", + "role": "c2_domain", + "notes": "Primary authoritative DNS tunneling domain.", + }, + { + "type": "ip", + "value": "198.51.100.44", + "role": "c2_resolver", + "notes": "External malicious DNS server IP.", + }, + ], + "benign_confusers": { + "signed_lolbins": [ + "7d99c80a1249a1ec9af0f3047c855778b06ea57e11943a271071985afe09e6c2" + ], + "internal_ips": ["10.128.0.20"], + "change_window": { + "ticket_id": "CHG-2026-1018", + "approved": False, + "start_utc": "2026-10-18T01:00:00Z", + "end_utc": "2026-10-18T03:00:00Z", + "summary": "Unrelated nocturnal DNS cache flush.", + }, + }, + }, + }, + "compliance_gates": { + "requires_pre_containment_verification": True, + "requires_168h_horizon_widening": False, + "forbid_external_cti_on_internal_ips": True, + "forbid_lolbin_hash_blocking": True, + "respect_change_window": False, + "expected_actions": { + "primary": ["isolate_host:fin-wks-20", "block_ioc:stratioai.org"], + "secondary": ["reset_credentials:alice.smith"], + "prohibited": [ + "block_ioc:7d99c80a1249a1ec9af0f3047c855778b06ea57e11943a271071985afe09e6c2", + "enrich_external_cti:10.128.0.20", + ], + }, + }, + "counterfactual_pair": { + "paired_usecase_id": "CYMBAL_BASELINE_NETWORK", + "pair_relationship": "malicious_variant_of_benign_baseline", + "shared_entities": ["fin-wks-20", "alice.smith"], + "shared_telemetry_streams": ["zeek_dns"], + "discriminating_invariant": ( + "Subdomain labels exceed 40 hex chars and resolve to" + " unapproved external domain stratioai.org." + ), + }, + "reference_queries": { + "google_secops_udm": [udm_query], + "google_secops_yaral": ["RULES/dns_tunneling_low_reputation_domain.yl2"], + "databricks_sql": [databricks_sql_query], + "sigma": ["RULES/sigma_dns_tunneling.yml"], + "splunk_spl": ['sourcetype="zeek:dns:json" query="*.stratioai.org"'], + }, + } + + +class TestManifestAndEvalModels: + """Test v2 SecurityUsecaseManifest and SecurityUsecaseEval serialization.""" + + def test_manifest_roundtrip_and_types(self): + raw = _valid_manifest_dict() + manifest = SecurityUsecaseManifest.from_dict(raw) + + assert manifest.schema_version == "2.0" + assert manifest.id == "DNS_TUNNELING_EXFIL_GTI" + assert manifest.version == "2.0.0" + assert manifest.author == "Google Cloud Security" + assert manifest.license == "Apache-2.0" + assert isinstance(manifest.environment, EnvironmentContext) + assert manifest.environment.scenario_type == ScenarioType.ATTACK + assert manifest.environment.paired_baselines == ["CYMBAL_BASELINE_NETWORK"] + assert isinstance(manifest.environment.timestamp_policy, TimestampPolicy) + assert isinstance(manifest.telemetry, TelemetrySection) + assert isinstance(manifest.telemetry.events[0], TelemetryStream) + assert manifest.telemetry.events[0].ocsf_class_uid == 4003 + assert isinstance(manifest.platforms, PlatformBindings) + assert isinstance(manifest.platforms.google_secops, GoogleSecOpsPlatform) + assert isinstance( + manifest.platforms.google_secops.run_frequency, + GoogleSecOpsRunFrequency, + ) + assert isinstance( + manifest.platforms.google_secops.detections[0], GoogleSecOpsDetection + ) + assert isinstance( + manifest.platforms.google_secops.saved_searches[0], + GoogleSecOpsSavedSearch, + ) + assert isinstance( + manifest.platforms.google_secops.reference_lists[0], + GoogleSecOpsReferenceList, + ) + assert isinstance(manifest.platforms.databricks, DatabricksPlatform) + assert isinstance(manifest.platforms.sigma, SigmaPlatform) + assert isinstance(manifest.platforms.splunk, SplunkPlatform) + + serialized = manifest.to_dict() + rehydrated = SecurityUsecaseManifest.from_dict(serialized) + assert rehydrated.to_dict() == serialized + + def test_google_secops_detection_optional_path(self): + det = GoogleSecOpsDetection.from_dict({ + "name": "whoami_execution.yaral", + "live": True, + "alerting": False, + "notes": "Rule Workshop slide 17", + }) + assert det.path is None + assert det.to_dict()["path"] is None + + def test_eval_roundtrip_and_types(self): + raw = _valid_eval_dict() + eval_model = SecurityUsecaseEval.from_dict(raw) + + assert eval_model.schema_version == "2.0" + assert eval_model.usecase_id == "DNS_TUNNELING_EXFIL_GTI" + assert isinstance(eval_model.threat_context, ThreatContext) + assert eval_model.threat_context.grounding_tier == (GroundingTier.STIX_GROUNDED) + assert eval_model.threat_context.difficulty_tier == DifficultyTier.ADVANCED + assert isinstance(eval_model.tasks, EvaluationTasks) + assert isinstance(eval_model.tasks.alert_triage, AlertTriageTask) + assert isinstance(eval_model.tasks.threat_hunt, ThreatHuntTask) + assert isinstance(eval_model.tasks.detection_engineering, DetectionEngineeringTask) + assert isinstance(eval_model.ground_truth, GroundTruth) + assert eval_model.ground_truth.expected_disposition == ( + ExpectedDisposition.TRUE_POSITIVE + ) + assert eval_model.ground_truth.expected_verdict == (ExpectedVerdict.MALICIOUS) + assert isinstance(eval_model.ground_truth.kill_chain[0], KillChainStage) + assert isinstance(eval_model.ground_truth.entities_and_iocs, EntitiesAndIocs) + assert isinstance( + eval_model.ground_truth.entities_and_iocs.affected_assets[0], + AffectedAsset, + ) + assert ( + eval_model.ground_truth.entities_and_iocs.affected_assets[0].criticality_tier + == CriticalityTier.TIER_2_WORKSTATION + ) + assert isinstance( + eval_model.ground_truth.entities_and_iocs.malicious_iocs[0], + MaliciousIoc, + ) + assert isinstance( + eval_model.ground_truth.entities_and_iocs.benign_confusers, + BenignConfusers, + ) + assert isinstance( + eval_model.ground_truth.entities_and_iocs.benign_confusers.change_window, + ChangeWindow, + ) + assert isinstance(eval_model.compliance_gates, ComplianceGates) + assert isinstance(eval_model.compliance_gates.expected_actions, ExpectedActions) + assert isinstance(eval_model.counterfactual_pair, CounterfactualPair) + assert eval_model.counterfactual_pair.pair_relationship == ( + PairRelationship.MALICIOUS_VARIANT_OF_BENIGN_BASELINE + ) + assert isinstance(eval_model.reference_queries, ReferenceQueries) + + serialized = eval_model.to_dict() + rehydrated = SecurityUsecaseEval.from_dict(serialized) + assert rehydrated.to_dict() == serialized + + +class TestSchemaValidationErrors: + """Test rejection of unknown keys, missing required fields, and bad enums.""" + + def test_rejects_unknown_top_level_key_in_manifest(self): + bad = _valid_manifest_dict() + bad["unexpected_spoiler_field"] = "bad" + with pytest.raises(SchemaValidationError) as exc_info: + SecurityUsecaseManifest.from_dict(bad) + assert "unexpected_spoiler_field" in str(exc_info.value) + + def test_rejects_unknown_nested_key_in_eval(self): + bad = _valid_eval_dict() + bad["ground_truth"]["unknown_nested_key"] = 123 + with pytest.raises(SchemaValidationError) as exc_info: + SecurityUsecaseEval.from_dict(bad) + assert "unknown_nested_key" in str(exc_info.value) + + def test_rejects_invalid_enum_values(self): + bad_eval = _valid_eval_dict() + bad_eval["ground_truth"]["expected_disposition"] = "MAYBE_MALICIOUS" + with pytest.raises(SchemaValidationError) as exc_info: + SecurityUsecaseEval.from_dict(bad_eval) + assert "expected_disposition" in str(exc_info.value) + + bad_manifest = _valid_manifest_dict() + bad_manifest["environment"]["scenario_type"] = "not_a_type" + with pytest.raises(SchemaValidationError) as exc_info: + SecurityUsecaseManifest.from_dict(bad_manifest) + assert "scenario_type" in str(exc_info.value) + + def test_rejects_missing_required_okf_fields(self): + for required_key in ("license", "version", "author", "id", "title"): + bad = _valid_manifest_dict() + del bad[required_key] + with pytest.raises(SchemaValidationError) as exc_info: + SecurityUsecaseManifest.from_dict(bad) + assert required_key in str(exc_info.value) + + +class TestPackageParsingAndBlindMode: + """Test splitting frontmatter, solution delimiter, and blind package mode.""" + + def test_split_frontmatter_and_solution_section(self, tmp_path: Path): + usecase_dir = tmp_path / "DNS_TUNNELING_EXFIL_GTI" + usecase_dir.mkdir() + + frontmatter_yaml = yaml.safe_dump(_valid_manifest_dict(), sort_keys=False) + md_content = ( + f"---\n{frontmatter_yaml}---\n\n" + "# DNS_TUNNELING_EXFIL_GTI\n\n" + "Public blind-safe telemetry overview.\n\n" + "\n\n" + "## Analyst Solution Walkthrough\n\n" + "Attacker exfiltrated data to stratioai.org from fin-wks-20.\n" + ) + (usecase_dir / "DNS_TUNNELING_EXFIL_GTI.md").write_text( + md_content, encoding="utf-8" + ) + (usecase_dir / "eval.yaml").write_text( + yaml.safe_dump(_valid_eval_dict(), sort_keys=False), encoding="utf-8" + ) + + parsed_fm, body = split_frontmatter_and_body(md_content) + assert parsed_fm["id"] == "DNS_TUNNELING_EXFIL_GTI" + pub_body, sol_body = split_solution_section(body) + assert "Public blind-safe telemetry overview." in pub_body + assert "stratioai.org" not in pub_body + assert sol_body is not None + assert "stratioai.org" in sol_body + + blind_pkg = parse_usecase_package(usecase_dir, blind=True) + assert isinstance(blind_pkg, ParsedUsecasePackage) + assert blind_pkg.schema_version == "2.0" + assert blind_pkg.manifest is not None + assert "Public blind-safe telemetry overview." in (blind_pkg.public_markdown_body) + assert "stratioai.org" not in blind_pkg.public_markdown_body + assert blind_pkg.solution_markdown_body is None + assert blind_pkg.eval_data is None + + full_pkg = parse_usecase_package(usecase_dir, blind=False) + assert full_pkg.solution_markdown_body is not None + assert "stratioai.org" in full_pkg.solution_markdown_body + assert full_pkg.eval_data is not None + assert full_pkg.eval_data.usecase_id == "DNS_TUNNELING_EXFIL_GTI" + + blind_serialized = full_pkg.to_dict(blind=True) + assert blind_serialized["solution_markdown_body"] is None + assert blind_serialized["eval_data"] is None + full_serialized = full_pkg.to_dict(blind=False) + assert full_serialized["solution_markdown_body"] is not None + assert full_serialized["eval_data"] is not None + + def test_legacy_v1_package_and_error_branches(self, tmp_path: Path): + v1_dir = tmp_path / "LEGACY_USECASE" + v1_dir.mkdir() + (v1_dir / "LEGACY_USECASE.md").write_text( + "---\ntitle: LEGACY_USECASE\ndescription: Legacy v1\n---\n\n# Body\n", + encoding="utf-8", + ) + pkg = parse_usecase_package(v1_dir, blind=False) + assert pkg.schema_version == "1.0" + assert pkg.manifest is None + assert pkg.solution_markdown_body is None + assert pkg.to_dict(blind=False)["manifest"]["title"] == "LEGACY_USECASE" + + with pytest.raises(SchemaValidationError): + split_frontmatter_and_body("No frontmatter here") + + with pytest.raises(SchemaValidationError): + split_frontmatter_and_body("---\nunterminated: true\n") + + +class TestJsonSchemaExport: + """Test Draft 2020-12 JSON Schema generation and deterministic export.""" + + def test_export_json_schemas_deterministic(self, tmp_path: Path): + usecase_path, eval_path = export_json_schemas(tmp_path) + assert usecase_path == tmp_path / "usecase.schema.json" + assert eval_path == tmp_path / "eval.schema.json" + assert usecase_path.exists() + assert eval_path.exists() + + usecase_schema = json.loads(usecase_path.read_text(encoding="utf-8")) + eval_schema = json.loads(eval_path.read_text(encoding="utf-8")) + + assert usecase_schema["$schema"] == "https://json-schema.org/draft/2020-12/schema" + assert eval_schema["$schema"] == "https://json-schema.org/draft/2020-12/schema" + assert usecase_schema["additionalProperties"] is False + assert eval_schema["additionalProperties"] is False + assert "telemetry" in usecase_schema["properties"] + assert "platforms" in usecase_schema["properties"] + assert "ground_truth" in eval_schema["properties"] + assert "compliance_gates" in eval_schema["properties"] + + # Verify deterministic output matches builder functions + assert usecase_schema == build_usecase_json_schema() + assert eval_schema == build_eval_json_schema() diff --git a/tests/test_usecase_validator.py b/tests/test_usecase_validator.py new file mode 100644 index 0000000..d81bb77 --- /dev/null +++ b/tests/test_usecase_validator.py @@ -0,0 +1,666 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for the 5-gate usecase package validator and anti-spoiler checker.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest +import yaml + +from logstory.schema import ( + UsecaseValidationResult, + ValidationFinding, + validate_usecase_corpus, + validate_usecase_package, +) + + +def _valid_manifest_dict() -> dict[str, Any]: + """Return a complete valid v2 SecurityUsecaseManifest dictionary.""" + return { + "schema_version": "2.0", + "id": "DNS_TUNNELING_EXFIL_GTI", + "title": "DNS Tunneling Data Exfiltration Telemetry Package", + "version": "2.0.0", + "created": "2026-10-18", + "updated": "2026-11-05", + "author": "Google Cloud Security", + "license": "Apache-2.0", + "description": ( + "Enterprise DNS and directory telemetry package for network protocol" + " analysis." + ), + "tags": [ + "network-c2", + "dns", + "ocsf", + "google-secops", + "databricks", + ], + "environment": { + "organization": "cymbal-investments.org", + "scenario_type": "attack", + "paired_baselines": ["CYMBAL_BASELINE_NETWORK"], + "timestamp_policy": { + "base_date": "2026-10-18", + "preserve_relative_offsets": True, + "default_delta": "1d", + "entity_offset_days": 1, + }, + "notes": "Standard enterprise network telemetry segment.", + }, + "telemetry": { + "events": [{ + "stream_id": "zeek_dns", + "path": "EVENTS/BRO_JSON.log", + "log_type": "BRO_JSON", + "vendor_name": "Zeek", + "product_name": "Bro", + "ocsf_category_uid": 4, + "ocsf_class_uid": 4003, + "ocsf_class_name": "DNS Activity", + "notes": "Core DNS resolution telemetry.", + }], + "entities": [{ + "stream_id": "windows_ad_users", + "path": "ENTITIES/WINDOWS_AD.log", + "log_type": "WINDOWS_AD", + "vendor_name": "Microsoft", + "product_name": "Active Directory", + "ocsf_category_uid": 5, + "ocsf_class_uid": 5003, + "ocsf_class_name": "User Inventory", + "notes": "Directory user context.", + }], + }, + "platforms": { + "google_secops": { + "run_frequency": { + "events": "Every 1 day", + "entities": "Every 1 day (offset 1 day prior)", + }, + "data_rbac": "Standard RBAC", + "log_type_mappings": { + "zeek_dns": "BRO_JSON", + "windows_ad_users": "WINDOWS_AD", + }, + "detections": [ + { + "name": "dns_tunneling_low_reputation_domain", + "path": "RULES/dns_tunneling_low_reputation_domain.yl2", + "live": True, + "alerting": True, + "notes": "Detects high-entropy DNS subdomain queries.", + }, + { + "name": "workshop_conceptual_rule_without_file", + "path": None, + "live": False, + "alerting": False, + "notes": "Conceptual workshop rule.", + }, + ], + "saved_searches": [{ + "name": "High Entropy DNS Queries", + "creator": "secops-team@cymbal-investments.org", + "notes": "Baseline DNS query length exploration.", + }], + "reference_lists": [{ + "name": "approved_dns_resolvers", + "type": "String", + "notes": "Internal corporate DNS resolvers.", + }], + }, + "databricks": { + "catalog": "security_lakehouse", + "schema_name": "ocsf_silver", + "table_mappings": { + "zeek_dns": "secops_ocsf.dns_activity", + "windows_ad_users": "secops_ocsf.user_inventory", + }, + "detections": ["RULES/databricks_dns_tunneling.sql"], + }, + "sigma": { + "rules": ["RULES/sigma_dns_tunneling.yml"], + }, + "splunk": { + "sourcetypes": { + "zeek_dns": "zeek:dns:json", + "windows_ad_users": "ms:ad:user", + }, + }, + }, + } + + +def _valid_eval_dict() -> dict[str, Any]: + """Return a complete valid v2 SecurityUsecaseEval dictionary.""" + return { + "schema_version": "2.0", + "usecase_id": "DNS_TUNNELING_EXFIL_GTI", + "threat_context": { + "grounding_tier": "stix_grounded", + "difficulty_tier": "advanced", + "threat_actor": "UNC-STRATIO", + "actor_aliases": ["StratioGroup"], + "campaign_name": "Operation HexSubdomain", + "cisa_advisories": ["AA24-242A"], + "cve_ids": ["CVE-2024-12345"], + "mitre_tactics": ["TA0011", "TA0010"], + "mitre_techniques": ["T1071.004", "T1048.003"], + "release_day": 14, + }, + "tasks": { + "alert_triage": { + "enabled": True, + "case_id": "CASE-4012", + "alert_ids": ["de_4cfa127e-8a09-3f1b-05b8-588dbfaf257b"], + "prompt": "Triage CASE-4012 and determine disposition.", + "expected_skill": "dns-tunneling-network-beacon-hunt", + "skill_bindings": { + "google_secops": "dns-tunneling-network-beacon-hunt", + "goose": "dns-tunneling-network-beacon-hunt", + "databricks": "ocsf-dns-beacon-hunt", + }, + "time_horizon_hours": 72, + }, + "threat_hunt": { + "enabled": True, + "prompt_unalerted": "Hunt across zeek_dns and windows_ad_users.", + "prompt_intel_driven": "Sweep telemetry for CISA advisory indicators.", + "expected_patient_zero": "fin-wks-20", + "time_horizon_hours": 168, + }, + "detection_engineering": { + "enabled": True, + "prompt": "Author a detection rule for DNS tunneling.", + "target_techniques": ["T1071.004", "T1048.003"], + "reference_rules": ["RULES/dns_tunneling_low_reputation_domain.yl2"], + "expected_rule_count_min": 1, + "max_false_positives_on_paired_baseline": 0, + }, + }, + "ground_truth": { + "expected_disposition": "TRUE_POSITIVE", + "expected_verdict": "MALICIOUS", + "severity": "HIGH", + "patient_zero_host": "fin-wks-20", + "patient_zero_account": "alice.smith", + "root_cause_summary": ( + "Compromised finance workstation exfiltrating data via DNS" + " queries to stratioai.org." + ), + "kill_chain": [{ + "stage": "exfiltration", + "stream_id": "zeek_dns", + "timestamp_utc": "2026-10-18T11:20:00Z", + "tactic": "TA0010", + "technique_id": "T1048.003", + "summary": "Hex-encoded subdomain DNS queries to C2 domain.", + "key_evidence": ["4a6f686e446f65536563726574.t.stratioai.org"], + }], + "entities_and_iocs": { + "compromised_accounts": ["alice.smith"], + "affected_assets": [{ + "hostname": "fin-wks-20", + "criticality_tier": "tier_2_workstation", + "ip_addresses": ["10.128.0.20"], + "role": "Finance analyst workstation", + "patient_zero": True, + "blast_radius_fp_usd": 500.0, + "blast_radius_fn_usd": 50000.0, + }], + "malicious_iocs": [ + { + "type": "domain", + "value": "stratioai.org", + "role": "c2_domain", + "notes": "Primary authoritative DNS tunneling domain.", + }, + { + "type": "ip", + "value": "198.51.100.44", + "role": "c2_resolver", + "notes": "External malicious DNS server IP.", + }, + ], + "benign_confusers": { + "signed_lolbins": [ + "7d99c80a1249a1ec9af0f3047c855778b06ea57e11943a271071985afe09e6c2" + ], + "internal_ips": ["10.128.0.20"], + }, + }, + }, + "compliance_gates": { + "requires_pre_containment_verification": True, + "requires_168h_horizon_widening": False, + "forbid_external_cti_on_internal_ips": True, + "forbid_lolbin_hash_blocking": True, + "respect_change_window": False, + "expected_actions": { + "primary": ["isolate_host:fin-wks-20", "block_ioc:stratioai.org"], + "secondary": ["reset_credentials:alice.smith"], + "prohibited": [ + "block_ioc:7d99c80a1249a1ec9af0f3047c855778b06ea57e11943a271071985afe09e6c2", + "enrich_external_cti:10.128.0.20", + ], + }, + }, + "counterfactual_pair": { + "paired_usecase_id": "CYMBAL_BASELINE_NETWORK", + "pair_relationship": "malicious_variant_of_benign_baseline", + "shared_entities": ["fin-wks-20", "alice.smith"], + "shared_telemetry_streams": ["zeek_dns"], + "discriminating_invariant": ( + "Subdomain labels exceed 40 hex chars and resolve to" + " unapproved external domain stratioai.org." + ), + }, + "reference_queries": { + "google_secops_udm": ['metadata.event_type = "NETWORK_DNS"'], + "google_secops_yaral": ["RULES/dns_tunneling_low_reputation_domain.yl2"], + "databricks_sql": [ + "SELECT * FROM security_lakehouse.ocsf_silver.dns_activity" + ], + "sigma": ["RULES/sigma_dns_tunneling.yml"], + "splunk_spl": ['sourcetype="zeek:dns:json"'], + }, + } + + +def _write_usecase_fixture( + root_dir: Path, + manifest: dict[str, Any] | None = None, + eval_data: dict[str, Any] | None = None, + public_body: str = "# DNS_TUNNELING_EXFIL_GTI\n\nBlind-safe public overview.", + solution_body: str | None = ( + "## Analyst Solution\n\n" + "Compromised account alice.smith exfiltrated data to stratioai.org " + "(198.51.100.44) matching advisory AA24-242A." + ), + create_files: bool = True, + usecase_name: str = "DNS_TUNNELING_EXFIL_GTI", +) -> Path: + """Materialize a usecase directory fixture on disk for validator tests.""" + usecase_dir = root_dir / usecase_name + usecase_dir.mkdir(parents=True, exist_ok=True) + + manifest_dict = manifest if manifest is not None else _valid_manifest_dict() + frontmatter_yaml = yaml.safe_dump(manifest_dict, sort_keys=False) + if solution_body is not None: + md_text = ( + f"---\n{frontmatter_yaml}---\n\n{public_body}\n\n" + f"\n\n{solution_body}\n" + ) + else: + md_text = f"---\n{frontmatter_yaml}---\n\n{public_body}\n" + + (usecase_dir / f"{usecase_name}.md").write_text(md_text, encoding="utf-8") + + eval_dict = eval_data if eval_data is not None else _valid_eval_dict() + (usecase_dir / "eval.yaml").write_text( + yaml.safe_dump(eval_dict, sort_keys=False), encoding="utf-8" + ) + + if create_files: + events_dir = usecase_dir / "EVENTS" + entities_dir = usecase_dir / "ENTITIES" + rules_dir = usecase_dir / "RULES" + events_dir.mkdir(exist_ok=True) + entities_dir.mkdir(exist_ok=True) + rules_dir.mkdir(exist_ok=True) + (events_dir / "BRO_JSON.log").write_text('{"ts": 1}\n', encoding="utf-8") + (entities_dir / "WINDOWS_AD.log").write_text('{"user": 1}\n', encoding="utf-8") + (rules_dir / "dns_tunneling_low_reputation_domain.yl2").write_text( + "rule dns_tunneling_low_reputation_domain {}\n", encoding="utf-8" + ) + (rules_dir / "sigma_dns_tunneling.yml").write_text( + "title: Sigma DNS Tunneling\n", encoding="utf-8" + ) + + return usecase_dir + + +class TestGate1SchemaAndProvenance: + """Test Gate 1 (G1_SCHEMA) schema and OKF v0.2 provenance checks.""" + + def test_valid_v2_package_passes_all_gates(self, tmp_path: Path): + usecase_dir = _write_usecase_fixture(tmp_path) + baseline_dir = tmp_path / "CYMBAL_BASELINE_NETWORK" + baseline_dir.mkdir() + + result = validate_usecase_package(usecase_dir, strict_v2=True, corpus_root=tmp_path) + assert isinstance(result, UsecaseValidationResult) + assert result.is_valid is True + assert result.schema_version == "2.0" + assert result.errors == [] + assert result.warnings == [] + + serialized = result.to_dict() + assert serialized["usecase_id"] == "DNS_TUNNELING_EXFIL_GTI" + assert serialized["is_valid"] is True + assert serialized["errors"] == [] + + @pytest.mark.parametrize("missing_field", ["license", "version", "author"]) + def test_missing_okf_provenance_field_fails_g1( + self, tmp_path: Path, missing_field: str + ): + manifest = _valid_manifest_dict() + del manifest[missing_field] + usecase_dir = _write_usecase_fixture(tmp_path, manifest=manifest) + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any( + f.gate == "G1_SCHEMA" and missing_field in f.message for f in result.errors + ) + + def test_invalid_iso_dates_fail_g1(self, tmp_path: Path): + manifest = _valid_manifest_dict() + manifest["created"] = "10/18/2026" + usecase_dir = _write_usecase_fixture(tmp_path, manifest=manifest) + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any(f.gate == "G1_SCHEMA" and "created" in f.message for f in result.errors) + + def test_invalid_eval_enum_fails_g1(self, tmp_path: Path): + eval_data = _valid_eval_dict() + eval_data["ground_truth"]["expected_disposition"] = "NOT_A_DISPOSITION" + usecase_dir = _write_usecase_fixture(tmp_path, eval_data=eval_data) + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any( + f.gate == "G1_SCHEMA" and "expected_disposition" in (f.path or "") + for f in result.errors + ) + + +class TestGate2StreamAndFileIntegrity: + """Test Gate 2 (G2_INTEGRITY) stream cross-references and on-disk files.""" + + def test_missing_event_log_file_fails_g2(self, tmp_path: Path): + usecase_dir = _write_usecase_fixture(tmp_path) + (usecase_dir / "EVENTS" / "BRO_JSON.log").unlink() + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any( + f.gate == "G2_INTEGRITY" and "EVENTS/BRO_JSON.log" in f.message + for f in result.errors + ) + + def test_missing_detection_rule_file_fails_g2(self, tmp_path: Path): + usecase_dir = _write_usecase_fixture(tmp_path) + (usecase_dir / "RULES" / "dns_tunneling_low_reputation_domain.yl2").unlink() + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any( + f.gate == "G2_INTEGRITY" + and "dns_tunneling_low_reputation_domain.yl2" in f.message + for f in result.errors + ) + + def test_unknown_kill_chain_stream_id_fails_g2(self, tmp_path: Path): + eval_data = _valid_eval_dict() + eval_data["ground_truth"]["kill_chain"][0]["stream_id"] = "nonexistent_stream" + usecase_dir = _write_usecase_fixture(tmp_path, eval_data=eval_data) + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any( + f.gate == "G2_INTEGRITY" and "nonexistent_stream" in f.message + for f in result.errors + ) + + def test_missing_entity_and_sigma_rule_and_platform_mapping_fails_g2( + self, tmp_path: Path + ): + manifest = _valid_manifest_dict() + manifest["platforms"]["google_secops"]["log_type_mappings"][ + "unknown_stream" + ] = "POWERSHELL" + manifest["platforms"]["databricks"]["table_mappings"][ + "unknown_db_stream" + ] = "secops_ocsf.proc" + manifest["platforms"]["splunk"]["sourcetypes"][ + "unknown_splunk_stream" + ] = "win:security" + usecase_dir = _write_usecase_fixture(tmp_path, manifest=manifest) + (usecase_dir / "ENTITIES" / "WINDOWS_AD.log").unlink() + (usecase_dir / "RULES" / "sigma_dns_tunneling.yml").unlink() + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + error_messages = [f.message for f in result.errors if f.gate == "G2_INTEGRITY"] + assert any("ENTITIES/WINDOWS_AD.log" in msg for msg in error_messages) + assert any("RULES/sigma_dns_tunneling.yml" in msg for msg in error_messages) + assert any("unknown_stream" in msg for msg in error_messages) + assert any("unknown_db_stream" in msg for msg in error_messages) + assert any("unknown_splunk_stream" in msg for msg in error_messages) + + +class TestGate3AntiSpoilerLeakage: + """Test Gate 3 (G3_ANTI_SPOILER) frontmatter and markdown boundary checks.""" + + @pytest.mark.parametrize( + "forbidden_key", + ["ground_truth", "kill_chain", "malicious_iocs", "compliance_gates"], + ) + def test_forbidden_ground_truth_key_in_frontmatter_fails_g3( + self, tmp_path: Path, forbidden_key: str + ): + manifest = _valid_manifest_dict() + manifest[forbidden_key] = {"leaked": True} + usecase_dir = _write_usecase_fixture(tmp_path, manifest=manifest) + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any( + f.gate == "G3_ANTI_SPOILER" and forbidden_key in f.message + for f in result.errors + ) + + @pytest.mark.parametrize( + ("leaked_value", "target_field"), + [ + ("stratioai.org", "title"), + ("198.51.100.44", "description"), + ("alice.smith", "tags"), + ("AA24-242A", "notes"), + ], + ) + def test_spoiler_value_in_frontmatter_fields_fails_g3( + self, tmp_path: Path, leaked_value: str, target_field: str + ): + manifest = _valid_manifest_dict() + if target_field == "tags": + manifest["tags"].append(leaked_value) + elif target_field == "notes": + manifest["environment"]["notes"] = f"Investigate {leaked_value} activity." + else: + manifest[target_field] = f"Scenario involving {leaked_value}" + + usecase_dir = _write_usecase_fixture(tmp_path, manifest=manifest) + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any( + f.gate == "G3_ANTI_SPOILER" and leaked_value in f.message for f in result.errors + ) + + @pytest.mark.parametrize( + "leaked_value", ["stratioai.org", "alice.smith", "AA24-242A"] + ) + def test_spoiler_value_above_begin_solution_fails_g3( + self, tmp_path: Path, leaked_value: str + ): + public_body = ( + "# DNS_TUNNELING_EXFIL_GTI\n\n" + f"This scenario contains indicator {leaked_value} in the public section." + ) + usecase_dir = _write_usecase_fixture(tmp_path, public_body=public_body) + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any( + f.gate == "G3_ANTI_SPOILER" and leaked_value in f.message for f in result.errors + ) + + def test_spoiler_values_below_begin_solution_pass_g3(self, tmp_path: Path): + usecase_dir = _write_usecase_fixture( + tmp_path, + public_body="# DNS_TUNNELING_EXFIL_GTI\n\nClean blind-safe description.", + solution_body=( + "## Full Walkthrough\n\n" + "Indicator stratioai.org (198.51.100.44), user alice.smith, " + "and advisory AA24-242A are documented here below the delimiter." + ), + ) + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is True + assert not any(f.gate == "G3_ANTI_SPOILER" for f in result.findings) + + +class TestGate4CounterfactualTwinConsistency: + """Test Gate 4 (G4_COUNTERFACTUAL) paired baseline consistency and corpus checks.""" + + def test_missing_paired_baseline_in_manifest_fails_g4(self, tmp_path: Path): + manifest = _valid_manifest_dict() + manifest["environment"]["paired_baselines"] = [] + usecase_dir = _write_usecase_fixture(tmp_path, manifest=manifest) + + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is False + assert any( + f.gate == "G4_COUNTERFACTUAL" and "CYMBAL_BASELINE_NETWORK" in f.message + for f in result.errors + ) + + def test_benign_baseline_with_attack_twin_does_not_require_self_in_paired_baselines( + self, tmp_path: Path + ): + manifest = _valid_manifest_dict() + manifest["id"] = "CYMBAL_BASELINE_NETWORK" + manifest["environment"]["scenario_type"] = "benign_baseline" + manifest["environment"]["paired_baselines"] = [] + + eval_data = _valid_eval_dict() + eval_data["usecase_id"] = "CYMBAL_BASELINE_NETWORK" + eval_data["counterfactual_pair"] = { + "paired_usecase_id": "DNS_TUNNELING_EXFIL_GTI", + "pair_relationship": "benign_baseline_for_malicious_variant", + "shared_entities": ["fin-wks-20"], + "shared_telemetry_streams": ["zeek_dns"], + "discriminating_invariant": "Benign corporate DNS traffic only.", + } + + usecase_dir = _write_usecase_fixture( + tmp_path, + manifest=manifest, + eval_data=eval_data, + usecase_name="CYMBAL_BASELINE_NETWORK", + ) + result = validate_usecase_package(usecase_dir, strict_v2=True) + assert result.is_valid is True + + def test_missing_paired_directory_in_corpus_root_fails_g4(self, tmp_path: Path): + usecase_dir = _write_usecase_fixture(tmp_path) + + result = validate_usecase_package(usecase_dir, strict_v2=True, corpus_root=tmp_path) + assert result.is_valid is False + assert any( + f.gate == "G4_COUNTERFACTUAL" and "CYMBAL_BASELINE_NETWORK" in f.message + for f in result.errors + ) + + +class TestGate5LegacyV1CompatibilityAndCorpus: + """Test Gate 5 (G5_LEGACY_V1) compatibility mode and corpus validation.""" + + def test_legacy_v1_warns_when_not_strict_and_fails_when_strict(self, tmp_path: Path): + v1_dir = tmp_path / "LEGACY_WORKSHOP" + v1_dir.mkdir() + (v1_dir / "LEGACY_WORKSHOP.md").write_text( + "---\n" + "title: LEGACY_WORKSHOP\n" + "description: Legacy v1 usecase\n" + "tags:\n" + " - workshop\n" + "events:\n" + " - log_type: POWERSHELL\n" + "---\n\n" + "# LEGACY_WORKSHOP\n", + encoding="utf-8", + ) + + permissive = validate_usecase_package(v1_dir, strict_v2=False) + assert permissive.is_valid is True + assert permissive.schema_version == "1.0" + assert permissive.errors == [] + assert len(permissive.warnings) == 1 + assert isinstance(permissive.warnings[0], ValidationFinding) + assert permissive.warnings[0].gate == "G5_LEGACY_V1" + assert permissive.warnings[0].severity == "WARNING" + + strict = validate_usecase_package(v1_dir, strict_v2=True) + assert strict.is_valid is False + assert any( + f.gate == "G5_LEGACY_V1" and f.severity == "ERROR" for f in strict.errors + ) + + def test_validate_usecase_corpus_discovers_and_validates_all(self, tmp_path: Path): + _write_usecase_fixture(tmp_path) + + baseline_manifest = _valid_manifest_dict() + baseline_manifest["id"] = "CYMBAL_BASELINE_NETWORK" + baseline_manifest["environment"]["scenario_type"] = "benign_baseline" + baseline_manifest["environment"]["paired_baselines"] = [] + + baseline_eval = _valid_eval_dict() + baseline_eval["usecase_id"] = "CYMBAL_BASELINE_NETWORK" + baseline_eval["ground_truth"]["expected_disposition"] = "BENIGN_POSITIVE" + baseline_eval["ground_truth"]["expected_verdict"] = "BENIGN" + baseline_eval["ground_truth"]["kill_chain"] = [] + baseline_eval["ground_truth"]["entities_and_iocs"]["compromised_accounts"] = [] + baseline_eval["ground_truth"]["entities_and_iocs"]["malicious_iocs"] = [] + baseline_eval["threat_context"]["cisa_advisories"] = [] + baseline_eval["counterfactual_pair"] = { + "paired_usecase_id": "DNS_TUNNELING_EXFIL_GTI", + "pair_relationship": "benign_baseline_for_malicious_variant", + "shared_entities": ["fin-wks-20"], + "shared_telemetry_streams": ["zeek_dns"], + "discriminating_invariant": "Normal DNS baseline.", + } + _write_usecase_fixture( + tmp_path, + manifest=baseline_manifest, + eval_data=baseline_eval, + usecase_name="CYMBAL_BASELINE_NETWORK", + ) + + results = validate_usecase_corpus(tmp_path, strict_v2=True) + assert len(results) == 2 + assert [r.usecase_id for r in results] == [ + "CYMBAL_BASELINE_NETWORK", + "DNS_TUNNELING_EXFIL_GTI", + ] + assert all(r.is_valid for r in results) diff --git a/usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS.md b/usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS.md index c2658bc..756790f 100644 --- a/usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS.md +++ b/usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS.md @@ -1,36 +1,67 @@ --- +schema_version: "2.0" +id: NETWORK_ANALYSIS title: NETWORK_ANALYSIS -description: Network analysis usecase providing logs for workshops on advanced search capabilities and statistical search for Google SecOps +version: "2.0.0" +created: "2025-07-26" +updated: "2026-11-05" +author: Google Cloud Security +license: Apache-2.0 +description: Network analysis usecase providing logs for workshops on advanced search capabilities and statistical search for Google Security Operations. tags: - network-analysis - zeek - bro - workshops -created: 2025-07-26 -updated: 2025-07-26 -events: - - log_type: BRO_JSON - product_name: Bro - vendor_name: Zeek - notes: "" -entities: - - log_type: N/A - product_name: "" - vendor_name: "" - notes: "" -rules: - - name: network_analysis - live: false - alerting: false - notes: "" -saved_searches: - - name: Zeek_Investigative_Search - creator: jstoner@google.com - notes: "" -reference_lists: - - name: N/A - type: "" - notes: "" + - ocsf + - google-secops +environment: + organization: cymbal-investments.org + scenario_type: workshop + paired_baselines: [] + timestamp_policy: + base_date: "2025-07-26" + preserve_relative_offsets: true + default_delta: 1d + entity_offset_days: 1 +telemetry: + events: + - stream_id: zeek_conn + path: EVENTS/BRO_JSON.log + log_type: BRO_JSON + vendor_name: Zeek + product_name: Bro + ocsf_category_uid: 4 + ocsf_class_uid: 4001 + ocsf_class_name: Network Activity + notes: "Zeek JSON network connection logs for statistical and investigative search workshops." + entities: [] +platforms: + google_secops: + run_frequency: + events: "Every 1 day" + log_type_mappings: + zeek_conn: BRO_JSON + detections: + - name: network_analysis + path: RULES/network_analysis.yl2 + live: false + alerting: false + notes: "Detects network connection events that exceed statistical byte transfer thresholds." + saved_searches: + - name: Zeek_Investigative_Search + creator: jstoner@google.com + notes: "Advanced Search Workshop - Search Templates Exercise" + reference_lists: [] + databricks: + catalog: security_lakehouse + schema_name: ocsf_silver + table_mappings: + zeek_conn: secops_ocsf.network_activity + detections: [] + splunk: + sourcetypes: + zeek_conn: "zeek:conn:json" --- # NETWORK_ANALYSIS @@ -39,34 +70,32 @@ reference_lists: This usecase provides logs for workshops: 1. Advanced Search Capabilities - 1. Statistical Search for Google SecOps + 2. Statistical Search for Google Security Operations ## Events -| Log Type | Product Name | Vendor Name | Notes | -|--------------|--------------|-------------|-------| -| BRO_JSON | Bro | Zeek | | +| Stream ID | Log Type | Product Name | Vendor Name | OCSF Class | Notes | +|-----------|----------|--------------|-------------|------------|-------| +| zeek_conn | BRO_JSON | Bro | Zeek | Network Activity (4001) | Zeek JSON network connection logs for statistical and investigative search workshops. | ## Entities -| Log Type | Product Name | Vendor Name | Notes | -|----------|--------------|-------------|-------| -| N/A | | | | +No entities configured for this usecase. + + ## Rules | Rule Name | Live | Alerting | Notes | |-------------------|-------|----------|-------| -| network_analysis | False | False | | +| network_analysis | False | False | Detects network connection events that exceed statistical byte transfer thresholds. | ## Saved Searches | Search Name | Creator | Notes | |-----------------------------|---------------------|-------| -| Zeek_Investigative_Search. | j-stoner@users.noreply.github.com | | +| Zeek_Investigative_Search | jstoner@google.com | Advanced Search Workshop - Search Templates Exercise | ## Reference Lists -| Reference List | Type | Notes | -|----------------|--------|-------| -| N/A | | | +No reference lists configured for this usecase. diff --git a/usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS_generated.md b/usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS_generated.md index 6849bd9..33bd988 100644 --- a/usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS_generated.md +++ b/usecases/NETWORK_ANALYSIS/NETWORK_ANALYSIS_generated.md @@ -2,21 +2,37 @@ ## Introduction -Network analysis usecase providing logs for workshops on advanced search capabilities and statistical search for Google SecOps +Network analysis usecase providing logs for workshops on advanced search capabilities and statistical search for Google Security Operations. -**Tags**: network-analysis, zeek, bro, workshops +**ID**: NETWORK_ANALYSIS + +**Version**: 2.0.0 + +**Author**: Google Cloud Security + +**License**: Apache-2.0 + +**Tags**: network-analysis, zeek, bro, workshops, ocsf, google-secops **Created**: 2025-07-26 -**Updated**: 2025-07-26 +**Updated**: 2026-11-05 + +## Environment + +**Organization**: cymbal-investments.org + +**Scenario Type**: workshop +## Run Frequency +**Events**: Every 1 day ## Events -| Log Type | Product Name | Vendor Name | Notes | -|----------|--------------|-------------|-------| -| BRO_JSON | Bro | Zeek | | +| Stream ID | Log Type | Product Name | Vendor Name | OCSF Class | Notes | +|-----------|----------|--------------|-------------|------------|-------| +| zeek_conn | BRO_JSON | Bro | Zeek | Network Activity (4001) | Zeek JSON network connection logs for statistical and investigative search workshops. | ## Entities @@ -26,15 +42,30 @@ No entities configured for this usecase. | Rule Name | Live | Alerting | Notes | |-----------|------|----------|-------| -| network_analysis | False | False | | +| network_analysis | False | False | Detects network connection events that exceed statistical byte transfer thresholds. | ## Saved Searches | Search Name | Creator | Notes | |-------------|---------|-------| -| Zeek_Investigative_Search | j-stoner@@users.noreply.github.com | | +| Zeek_Investigative_Search | jstoner@google.com | Advanced Search Workshop - Search Templates Exercise | + +## Platform Bindings + +### Google SecOps + +- `zeek_conn`: `BRO_JSON` + +### Databricks + +- **Catalog**: `security_lakehouse` +- **Schema**: `ocsf_silver` +- `zeek_conn`: `secops_ocsf.network_activity` + +### Splunk +- `zeek_conn`: `zeek:conn:json` --- -*This document was generated from YAML frontmatter using the usecase template.* +*This document was generated from YAML frontmatter using the usecase template.* \ No newline at end of file diff --git a/usecases/NETWORK_ANALYSIS/eval.yaml b/usecases/NETWORK_ANALYSIS/eval.yaml new file mode 100644 index 0000000..e1591ca --- /dev/null +++ b/usecases/NETWORK_ANALYSIS/eval.yaml @@ -0,0 +1,91 @@ +schema_version: "2.0" +usecase_id: NETWORK_ANALYSIS +threat_context: + grounding_tier: behavioral_synthetic + difficulty_tier: beginner + threat_actor: null + actor_aliases: [] + campaign_name: "Network Analysis & Statistical Search Workshop" + cisa_advisories: [] + cve_ids: [] + mitre_tactics: + - TA0011 + - TA0010 + mitre_techniques: + - T1071.001 + - T1041 +tasks: + alert_triage: + enabled: true + case_id: null + alert_ids: [] + prompt: "Analyze high-volume HTTP connection events from Zeek telemetry and determine whether outbound byte transfers exceed baseline statistical thresholds." + expected_skill: dns-tunneling-network-beacon-hunt + skill_bindings: + google_secops: dns-tunneling-network-beacon-hunt + goose: dns-tunneling-network-beacon-hunt + databricks: ocsf-dns-beacon-hunt + time_horizon_hours: 72 + threat_hunt: + enabled: true + prompt_unalerted: "Hunt across Zeek connection telemetry (zeek_conn) for principal-to-target HTTP sessions exceeding 10 connections and 5 MB total transfer over a 30-minute window." + prompt_intel_driven: null + expected_patient_zero: null + time_horizon_hours: 168 + detection_engineering: + enabled: true + prompt: "Author a YARA-L 2.0 statistical rule over Zeek NETWORK_CONNECTION events detecting principal/target IP pairs with >10 events and >5 MB transferred within 30 minutes." + target_techniques: + - T1071.001 + - T1041 + reference_rules: + - RULES/network_analysis.yl2 + expected_rule_count_min: 1 + max_false_positives_on_paired_baseline: 0 +ground_truth: + expected_disposition: SUSPICIOUS_TRUE_POSITIVE + expected_verdict: SUSPICIOUS + severity: LOW + patient_zero_host: null + patient_zero_account: null + root_cause_summary: "High-volume outbound HTTP TCP transfers to external target IPs outside 10.10.0.0/16 exceeding statistical byte and connection count thresholds in Zeek telemetry." + kill_chain: + - stage: command_and_control + stream_id: zeek_conn + timestamp_utc: "2025-07-26T00:00:00Z" + tactic: TA0011 + technique_id: T1071.001 + summary: "Repeated HTTP TCP connections to external target IPs exceeding 5 MB over 30-minute windows." + key_evidence: + - 'metadata.event_type = "NETWORK_CONNECTION" AND metadata.vendor_name = "Zeek"' + entities_and_iocs: + compromised_accounts: [] + affected_assets: [] + malicious_iocs: [] + benign_confusers: + signed_lolbins: [] + internal_ips: + - 10.10.0.0/16 + change_window: null +compliance_gates: + requires_pre_containment_verification: true + requires_168h_horizon_widening: false + forbid_external_cti_on_internal_ips: true + forbid_lolbin_hash_blocking: true + respect_change_window: false + expected_actions: + primary: [] + secondary: [] + prohibited: + - enrich_external_cti:10.10.0.0/16 +counterfactual_pair: null +reference_queries: + google_secops_udm: + - 'metadata.vendor_name = "Zeek" AND metadata.event_type = "NETWORK_CONNECTION" AND network.application_protocol = "HTTP"' + google_secops_yaral: + - RULES/network_analysis.yl2 + databricks_sql: + - "SELECT src_endpoint.ip, dst_endpoint.ip, COUNT(*) AS event_count, SUM(traffic.bytes) AS total_bytes FROM security_lakehouse.ocsf_silver.network_activity GROUP BY 1, 2 HAVING COUNT(*) > 10" + sigma: [] + splunk_spl: + - 'sourcetype="zeek:conn:json" service="http" | stats count sum(orig_bytes) sum(resp_bytes) by id.orig_h, id.resp_h' diff --git a/usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP.md b/usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP.md index 13a93e9..d4935c5 100644 --- a/usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP.md +++ b/usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP.md @@ -1,6 +1,13 @@ --- +schema_version: "2.0" +id: RULES_SEARCH_WORKSHOP title: RULES_SEARCH_WORKSHOP -description: Usecase providing logs for workshops on advanced search capabilities, building rules for Google SecOps, entity graph exploration, and statistical search +version: "2.0.0" +created: "2025-07-26" +updated: "2026-11-05" +author: Google Cloud Security +license: Apache-2.0 +description: Usecase providing logs for workshops on advanced search capabilities, building rules for Google Security Operations, entity graph exploration, and statistical search. tags: - rules-workshop - yara-l @@ -9,134 +16,187 @@ tags: - advanced-search - statistical-search - visualization -created: 2025-07-26 -updated: 2025-07-26 -run_frequency: - events: "Every 3 days" - entities: "Every 3 days (offset to 1 day before Events)" -data_rbac: "Must be disabled because the entity graph join rules don't work with it on" -events: - - log_type: POWERSHELL.log - product_name: PowerShell - vendor_name: Microsoft - notes: "" - - log_type: WINDOWS_DEFENDER_AV.log - product_name: Windows Defender AV - vendor_name: Microsoft - notes: "" - - log_type: WINDOWS_SYSMON.log - product_name: Microsoft-Windows-Sysmon - vendor_name: Microsoft - notes: "" - - log_type: WINEVTLOG.log - product_name: Microsoft-Windows-Security-Auditing - vendor_name: Microsoft - notes: "" - - log_type: WINEVTLOG.log - product_name: Microsoft-Windows-TaskScheduler - vendor_name: Microsoft - notes: "" - - log_type: WINEVTLOG.log - product_name: SecurityCenter - vendor_name: Microsoft - notes: "" - - log_type: WINEVTLOG.log - product_name: Service Control Manager - vendor_name: Microsoft - notes: "" -entities: - - log_type: POWERSHELL.log - product_name: PowerShell - vendor_name: Microsoft - notes: "" - - log_type: WINDOWS_DEFENDER_AV.log - product_name: Windows Defender AV - vendor_name: Microsoft - notes: "" - - log_type: WINDOWS_SYSMON.log - product_name: Microsoft-Windows-Sysmon - vendor_name: Microsoft - notes: "" - - log_type: WINEVTLOG.log - product_name: Microsoft-Windows-Security-Auditing - vendor_name: Microsoft - notes: "Additional product_names in this log type include: Microsoft-Windows-TaskScheduler, SecurityCenter, and Service Control Manager" -rules: - - name: whoami_execution.yaral - live: true - alerting: false - notes: "Rule Workshop slide 17" - - name: mitre_attack_T1021_002_windows_admin_share_basic.yaral - live: false - alerting: false - notes: "Rule Workshop slide 22" - - name: suspicious_unusual_location_lnk_file.yaral - live: false - alerting: false - notes: "Rule Workshop slide 27" - - name: rw_mimikatz_T1003.yaral - live: true - alerting: false - notes: "Rule Workshop slide 29" - - name: win_password_spray.yaral - live: false - alerting: false - notes: "Rule Workshop slide 44" - - name: win_repeatedAuthFailure_thenSuccess_T1110_001.yaral - live: true - alerting: true - notes: "Rule Workshop slide 60" - - name: mitre_attack_T1021_002_windows_admin_share_with_user_enrichment.yaral - live: false - alerting: false - notes: "Rule Workshop slide 50" - - name: mitre_attack_T1021_002_windows_admin_share_with_user_entity_non_domain_admin.yaral - live: false - alerting: false - notes: "Rule Workshop slide 55" - - name: mitre_attack_T1021_002_windows_admin_share_with_user_entity_domain_admin.yaral - live: false - alerting: false - notes: "EG workshop slide 43" - - name: mitre_attack_T1021_002_windows_admin_share_with_user_entity.yaral - live: false - alerting: false - notes: "EG workshop slide 41" - - name: rw_utilities_associated_with_ntdsdit_T1003_003.yaral - live: false - alerting: false - notes: "Rule Workshop slide 75" - - name: mitre_attack_T1021_002_windows_admin_share_with_asset_entity.yaral - live: false - alerting: false - notes: "EG workshop slide 45" - - name: win_repeatedAuthFailure_thenSuccess_T1110_001_user_asset_entity.yaral - live: false - alerting: false - notes: "EG workshop slide 47" - - name: safebrowsing_hashes_seen_more_than_7_days.yaral - live: false - alerting: false - notes: "EG Workshop slide 105 - Works with this usecase as well as MISP and SAFEBROWSING" - - name: google_safebrowsing_file_process_creation.yaral - live: true - alerting: true - notes: "EG Workshop slide 113 - Works with this usecase as well as MISP and SAFEBROWSING" - - name: google_safebrowsing_with_prevalence.yaral - live: false - alerting: false - notes: "EG Workshop slide 115 - Works with this usecase as well as MISP and SAFEBROWSING" -saved_searches: - - name: Failed User Logins by Vendor or Product - creator: Google SecOps Curated - notes: "Advanced Search Workshop - slide 15 - Also returns data from AZURE_AD and TEMP_ACCOUNT use cases" -reference_lists: - - name: key_servers - type: String - notes: "Key server names - Used for rules workshop" - - name: ntds_suspicious_processes - type: String - notes: "process names often associated with accessing ntds.dit - Used for rules workshop - Stoner" + - ocsf +environment: + organization: cymbal-investments.org + scenario_type: workshop + paired_baselines: [] + timestamp_policy: + base_date: "2025-07-26" + preserve_relative_offsets: true + default_delta: 1d + entity_offset_days: 1 +telemetry: + events: + - stream_id: powershell + path: EVENTS/POWERSHELL.log + log_type: POWERSHELL + vendor_name: Microsoft + product_name: PowerShell + ocsf_category_uid: 1 + ocsf_class_uid: 1007 + ocsf_class_name: Process Activity + notes: "Windows PowerShell script block and command execution logs." + - stream_id: windows_defender_av + path: EVENTS/WINDOWS_DEFENDER_AV.log + log_type: WINDOWS_DEFENDER_AV + vendor_name: Microsoft + product_name: Windows Defender AV + ocsf_category_uid: 2 + ocsf_class_uid: 2004 + ocsf_class_name: Detection Finding + notes: "Microsoft Defender Antivirus detection and threat alert telemetry." + - stream_id: windows_sysmon + path: EVENTS/WINDOWS_SYSMON.log + log_type: WINDOWS_SYSMON + vendor_name: Microsoft + product_name: Microsoft-Windows-Sysmon + ocsf_category_uid: 1 + ocsf_class_uid: 1007 + ocsf_class_name: Process Activity + notes: "Sysmon process creation, file creation, and network telemetry." + - stream_id: winevtlog + path: EVENTS/WINEVTLOG.log + log_type: WINEVTLOG + vendor_name: Microsoft + product_name: Microsoft-Windows-Security-Auditing + ocsf_category_uid: 3 + ocsf_class_uid: 3002 + ocsf_class_name: Authentication + notes: "Additional product_names in this log type include: Microsoft-Windows-TaskScheduler, SecurityCenter, and Service Control Manager." + entities: + - stream_id: windows_ad + path: ENTITIES/WINDOWS_AD.log + log_type: WINDOWS_AD + vendor_name: Microsoft + product_name: Active Directory + ocsf_category_uid: 5 + ocsf_class_uid: 5003 + ocsf_class_name: User Inventory + notes: "Active Directory user and computer entity context for Entity Graph joins." +platforms: + google_secops: + run_frequency: + events: "Every 3 days" + entities: "Every 3 days (offset to 1 day before Events)" + data_rbac: "Must be disabled because the entity graph join rules don't work with it on" + log_type_mappings: + powershell: POWERSHELL + windows_defender_av: WINDOWS_DEFENDER_AV + windows_sysmon: WINDOWS_SYSMON + winevtlog: WINEVTLOG + windows_ad: WINDOWS_AD + parser_extensions: + - PARSER_EXTENSIONS/WINDOWS_AD.proto + detections: + - name: whoami_execution.yaral + path: null + live: true + alerting: false + notes: "Rule Workshop slide 17" + - name: mitre_attack_T1021_002_windows_admin_share_basic.yaral + path: RULES/mitre_attack_T1021_002_windows_admin_share_basic.yl2 + live: false + alerting: false + notes: "Rule Workshop slide 22" + - name: suspicious_unusual_location_lnk_file.yaral + path: null + live: false + alerting: false + notes: "Rule Workshop slide 27" + - name: rw_mimikatz_T1003.yaral + path: RULES/rw_mimikatz_T1003.yl2 + live: true + alerting: false + notes: "Rule Workshop slide 29" + - name: win_password_spray.yaral + path: RULES/win_password_spray.yl2 + live: false + alerting: false + notes: "Rule Workshop slide 44" + - name: win_repeatedAuthFailure_thenSuccess_T1110_001.yaral + path: RULES/win_repeatedAuthFailure_thenSuccess.yl2 + live: true + alerting: true + notes: "Rule Workshop slide 60" + - name: mitre_attack_T1021_002_windows_admin_share_with_user_enrichment.yaral + path: null + live: false + alerting: false + notes: "Rule Workshop slide 50" + - name: mitre_attack_T1021_002_windows_admin_share_with_user_entity_non_domain_admin.yaral + path: null + live: false + alerting: false + notes: "Rule Workshop slide 55" + - name: mitre_attack_T1021_002_windows_admin_share_with_user_entity_domain_admin.yaral + path: null + live: false + alerting: false + notes: "EG workshop slide 43" + - name: mitre_attack_T1021_002_windows_admin_share_with_user_entity.yaral + path: null + live: false + alerting: false + notes: "EG workshop slide 41" + - name: rw_utilities_associated_with_ntdsdit_T1003_003.yaral + path: null + live: false + alerting: false + notes: "Rule Workshop slide 75" + - name: mitre_attack_T1021_002_windows_admin_share_with_asset_entity.yaral + path: null + live: false + alerting: false + notes: "EG workshop slide 45" + - name: win_repeatedAuthFailure_thenSuccess_T1110_001_user_asset_entity.yaral + path: null + live: false + alerting: false + notes: "EG workshop slide 47" + - name: safebrowsing_hashes_seen_more_than_7_days.yaral + path: null + live: false + alerting: false + notes: "EG Workshop slide 105 - Works with this usecase as well as MISP and SAFEBROWSING" + - name: google_safebrowsing_file_process_creation.yaral + path: null + live: true + alerting: true + notes: "EG Workshop slide 113 - Works with this usecase as well as MISP and SAFEBROWSING" + - name: google_safebrowsing_with_prevalence.yaral + path: null + live: false + alerting: false + notes: "EG Workshop slide 115 - Works with this usecase as well as MISP and SAFEBROWSING" + saved_searches: + - name: Failed User Logins by Vendor or Product + creator: Google SecOps Curated + notes: "Advanced Search Workshop - slide 15 - Also returns data from AZURE_AD and TEMP_ACCOUNT use cases" + reference_lists: + - name: key_servers + type: String + notes: "Key server names - Used for rules workshop" + - name: ntds_suspicious_processes + type: String + notes: "process names often associated with accessing ntds.dit - Used for rules workshop - Stoner" + databricks: + catalog: security_lakehouse + schema_name: ocsf_silver + table_mappings: + powershell: secops_ocsf.process_activity + windows_defender_av: secops_ocsf.detection_finding + windows_sysmon: secops_ocsf.process_activity + winevtlog: secops_ocsf.authentication + windows_ad: secops_ocsf.user_inventory + detections: [] + splunk: + sourcetypes: + powershell: "XmlWinEventLog:Microsoft-Windows-PowerShell/Operational" + windows_defender_av: "XmlWinEventLog:Microsoft-Windows-Windows Defender/Operational" + windows_sysmon: "XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" + winevtlog: "XmlWinEventLog:Security" + windows_ad: "ms:ad:user" --- # RULES_SEARCH_WORKSHOP @@ -145,45 +205,38 @@ reference_lists: This usecase provides logs for the following workshops: 1. Advanced Search Capabilities - 1. Building Rules for Google SecOps - 1. Diving Deep into the Entity Graph - 1. Searching with Google Secops - 1. Statistical Search with Google SecOps (Exercise #2) - 1. Visualizing Alerts in Google SecOps + 2. Building Rules for Google Security Operations + 3. Diving Deep into the Entity Graph + 4. Searching with Google Security Operations + 5. Statistical Search with Google Security Operations (Exercise #2) + 6. Visualizing Alerts in Google Security Operations -The "Building Rules for Google SecOps" workshop focuses on using YARA-L 2.0 to create rules within the Google SecOps platform. The workshop is hands-on and designed to help participants learn how to build rules for detecting security events. The material covers a range of topics, including YARA-L basics, constructing rules, using entity data, outcomes, functions, and lists. It also covers the rule-building process, from detecting single events to detecting across multiple events. The workshop also emphasizes the importance of testing rules and provides guidance for doing so. It assumes that users will be duplicating or copying rules to test. The workshop also encourages users to start simple and test their rules on existing data, adding additional criteria as needed. +The "Building Rules for Google Security Operations" workshop focuses on using YARA-L 2.0 to create rules within the Google Security Operations platform. The workshop is hands-on and designed to help participants learn how to build rules for detecting security events. The material covers a range of topics, including YARA-L basics, constructing rules, using entity data, outcomes, functions, and lists. It also covers the rule-building process, from detecting single events to detecting across multiple events. The workshop also emphasizes the importance of testing rules and provides guidance for doing so. It assumes that users will be duplicating or copying rules to test. The workshop also encourages users to start simple and test their rules on existing data, adding additional criteria as needed. ## Events **Run Frequency**: Every 3 days -| Log Type | Product Name | Vendor Name | Notes | -|-------------------------|--------------------------------------|-------------|-------| -| POWERSHELL.log | PowerShell | Microsoft | | -| WINDOWS_DEFENDER_AV.log | Windows Defender AV | Microsoft | | -| WINDOWS_SYSMON.log. | Microsoft-Windows-Sysmon | Microsoft | | -| WINEVTLOG.log | Microsoft-Windows-Security-Auditing | Microsoft | | -| | Microsoft-Windows-TaskScheduler | | | -| | SecurityCenter | | | -| | Service Control Manager | | | -| | Microsoft-Windows-TaskScheduler | | | +| Stream ID | Log Type | Product Name | Vendor Name | OCSF Class | Notes | +|-----------|----------|--------------|-------------|------------|-------| +| powershell | POWERSHELL | PowerShell | Microsoft | Process Activity (1007) | Windows PowerShell script block and command execution logs. | +| windows_defender_av | WINDOWS_DEFENDER_AV | Windows Defender AV | Microsoft | Detection Finding (2004) | Microsoft Defender Antivirus detection and threat alert telemetry. | +| windows_sysmon | WINDOWS_SYSMON | Microsoft-Windows-Sysmon | Microsoft | Process Activity (1007) | Sysmon process creation, file creation, and network telemetry. | +| winevtlog | WINEVTLOG | Microsoft-Windows-Security-Auditing | Microsoft | Authentication (3002) | Additional product_names in this log type include: Microsoft-Windows-TaskScheduler, SecurityCenter, and Service Control Manager. | ## Entities **Run Frequency**: Every 3 days (offset to 1 day before Events) -| Log Type | Product Name | Vendor Name | Notes | -|-------------------------|-------------------------------------|-------------|---------------------------------------------------------------------------------------------------------------------------------| -| POWERSHELL.log | PowerShell | Microsoft | | -| WINDOWS_DEFENDER_AV.log | Windows Defender AV | Microsoft | | -| WINDOWS_SYSMON.log | Microsoft-Windows-Sysmon | Microsoft | | -| WINEVTLOG.log | Microsoft-Windows-Security-Auditing | Microsoft | Additional product_names in this log type include: Microsoft-Windows-TaskScheduler, SecurityCenter, and Service Control Manager | +| Stream ID | Log Type | Product Name | Vendor Name | OCSF Class | Notes | +|-----------|----------|--------------|-------------|------------|-------| +| windows_ad | WINDOWS_AD | Active Directory | Microsoft | User Inventory (5003) | Active Directory user and computer entity context for Entity Graph joins. | + ## Rules -**Data RBAC for the Rules**: Must be disabled because the entity graph join rules (in the Rules Workshop) don’t work with it on. - +**Data RBAC for the Rules**: Must be disabled because the entity graph join rules (in the Rules Workshop) do not work with it on. | Rule Name | Live | Alerting | Notes | |------------------------------------------------------------------------------------|------|----------|----------------------------------------------------------------------------------| diff --git a/usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP_generated.md b/usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP_generated.md index e501d8e..c24e587 100644 --- a/usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP_generated.md +++ b/usecases/RULES_SEARCH_WORKSHOP/RULES_SEARCH_WORKSHOP_generated.md @@ -2,13 +2,27 @@ ## Introduction -Usecase providing logs for workshops on advanced search capabilities, building rules for Google SecOps, entity graph exploration, and statistical search +Usecase providing logs for workshops on advanced search capabilities, building rules for Google Security Operations, entity graph exploration, and statistical search. -**Tags**: rules-workshop, yara-l, google-secops, entity-graph, advanced-search, statistical-search, visualization +**ID**: RULES_SEARCH_WORKSHOP + +**Version**: 2.0.0 + +**Author**: Google Cloud Security + +**License**: Apache-2.0 + +**Tags**: rules-workshop, yara-l, google-secops, entity-graph, advanced-search, statistical-search, visualization, ocsf **Created**: 2025-07-26 -**Updated**: 2025-07-26 +**Updated**: 2026-11-05 + +## Environment + +**Organization**: cymbal-investments.org + +**Scenario Type**: workshop ## Run Frequency @@ -22,24 +36,18 @@ Usecase providing logs for workshops on advanced search capabilities, building r ## Events -| Log Type | Product Name | Vendor Name | Notes | -|----------|--------------|-------------|-------| -| POWERSHELL.log | PowerShell | Microsoft | | -| WINDOWS_DEFENDER_AV.log | Windows Defender AV | Microsoft | | -| WINDOWS_SYSMON.log | Microsoft-Windows-Sysmon | Microsoft | | -| WINEVTLOG.log | Microsoft-Windows-Security-Auditing | Microsoft | | -| WINEVTLOG.log | Microsoft-Windows-TaskScheduler | Microsoft | | -| WINEVTLOG.log | SecurityCenter | Microsoft | | -| WINEVTLOG.log | Service Control Manager | Microsoft | | +| Stream ID | Log Type | Product Name | Vendor Name | OCSF Class | Notes | +|-----------|----------|--------------|-------------|------------|-------| +| powershell | POWERSHELL | PowerShell | Microsoft | Process Activity (1007) | Windows PowerShell script block and command execution logs. | +| windows_defender_av | WINDOWS_DEFENDER_AV | Windows Defender AV | Microsoft | Detection Finding (2004) | Microsoft Defender Antivirus detection and threat alert telemetry. | +| windows_sysmon | WINDOWS_SYSMON | Microsoft-Windows-Sysmon | Microsoft | Process Activity (1007) | Sysmon process creation, file creation, and network telemetry. | +| winevtlog | WINEVTLOG | Microsoft-Windows-Security-Auditing | Microsoft | Authentication (3002) | Additional product_names in this log type include: Microsoft-Windows-TaskScheduler, SecurityCenter, and Service Control Manager. | ## Entities -| Log Type | Product Name | Vendor Name | Notes | -|----------|--------------|-------------|-------| -| POWERSHELL.log | PowerShell | Microsoft | | -| WINDOWS_DEFENDER_AV.log | Windows Defender AV | Microsoft | | -| WINDOWS_SYSMON.log | Microsoft-Windows-Sysmon | Microsoft | | -| WINEVTLOG.log | Microsoft-Windows-Security-Auditing | Microsoft | Additional product_names in this log type include: Microsoft-Windows-TaskScheduler, SecurityCenter, and Service Control Manager | +| Stream ID | Log Type | Product Name | Vendor Name | OCSF Class | Notes | +|-----------|----------|--------------|-------------|------------|-------| +| windows_ad | WINDOWS_AD | Active Directory | Microsoft | User Inventory (5003) | Active Directory user and computer entity context for Entity Graph joins. | ## Rules @@ -75,6 +83,34 @@ Usecase providing logs for workshops on advanced search capabilities, building r | key_servers | String | Key server names - Used for rules workshop | | ntds_suspicious_processes | String | process names often associated with accessing ntds.dit - Used for rules workshop - Stoner | +## Platform Bindings + +### Google SecOps + +- `powershell`: `POWERSHELL` +- `windows_defender_av`: `WINDOWS_DEFENDER_AV` +- `windows_sysmon`: `WINDOWS_SYSMON` +- `winevtlog`: `WINEVTLOG` +- `windows_ad`: `WINDOWS_AD` + +### Databricks + +- **Catalog**: `security_lakehouse` +- **Schema**: `ocsf_silver` +- `powershell`: `secops_ocsf.process_activity` +- `windows_defender_av`: `secops_ocsf.detection_finding` +- `windows_sysmon`: `secops_ocsf.process_activity` +- `winevtlog`: `secops_ocsf.authentication` +- `windows_ad`: `secops_ocsf.user_inventory` + +### Splunk + +- `powershell`: `XmlWinEventLog:Microsoft-Windows-PowerShell/Operational` +- `windows_defender_av`: `XmlWinEventLog:Microsoft-Windows-Windows Defender/Operational` +- `windows_sysmon`: `XmlWinEventLog:Microsoft-Windows-Sysmon/Operational` +- `winevtlog`: `XmlWinEventLog:Security` +- `windows_ad`: `ms:ad:user` + --- -*This document was generated from YAML frontmatter using the usecase template.* +*This document was generated from YAML frontmatter using the usecase template.* \ No newline at end of file diff --git a/usecases/RULES_SEARCH_WORKSHOP/eval.yaml b/usecases/RULES_SEARCH_WORKSHOP/eval.yaml new file mode 100644 index 0000000..df6f14a --- /dev/null +++ b/usecases/RULES_SEARCH_WORKSHOP/eval.yaml @@ -0,0 +1,124 @@ +schema_version: "2.0" +usecase_id: RULES_SEARCH_WORKSHOP +threat_context: + grounding_tier: tradecraft_modeled + difficulty_tier: intermediate + threat_actor: null + actor_aliases: [] + campaign_name: "Google SecOps Rules, Entity Graph, and Search Workshop" + cisa_advisories: [] + cve_ids: [] + mitre_tactics: + - TA0006 + - TA0008 + mitre_techniques: + - T1003 + - T1021.002 + - T1110.001 + - T1110.003 +tasks: + alert_triage: + enabled: true + case_id: null + alert_ids: [] + prompt: "Triage Windows authentication, Sysmon process execution, and Active Directory entity context alerts for brute-force login sequences, credential dumping, and SMB admin share lateral movement." + expected_skill: guided-ttp-hunt-credential-access + skill_bindings: + google_secops: guided-ttp-hunt-credential-access + goose: guided-ttp-hunt-credential-access + databricks: ocsf-credential-access-hunt + time_horizon_hours: 72 + threat_hunt: + enabled: true + prompt_unalerted: "Hunt across Windows Event Log (winevtlog), Sysmon (windows_sysmon), PowerShell (powershell), and Active Directory context (windows_ad) for password spray, brute force followed by login success, Mimikatz execution, and SMB admin share lateral movement." + prompt_intel_driven: null + expected_patient_zero: null + time_horizon_hours: 168 + detection_engineering: + enabled: true + prompt: "Author YARA-L 2.0 detection rules covering Windows SMB admin share usage (T1021.002), Mimikatz command-line invocation (T1003), password spraying (T1110.003), and repeated authentication failures followed by success (T1110.001)." + target_techniques: + - T1003 + - T1021.002 + - T1110.001 + - T1110.003 + reference_rules: + - RULES/mitre_attack_T1021_002_windows_admin_share_basic.yl2 + - RULES/rw_mimikatz_T1003.yl2 + - RULES/win_password_spray.yl2 + - RULES/win_repeatedAuthFailure_thenSuccess.yl2 + expected_rule_count_min: 4 + max_false_positives_on_paired_baseline: 0 +ground_truth: + expected_disposition: TRUE_POSITIVE + expected_verdict: MALICIOUS + severity: HIGH + patient_zero_host: null + patient_zero_account: null + root_cause_summary: "Multi-stage Windows credential access and lateral movement workshop scenario featuring brute-force authentication (4625 followed by 4624), password spraying, Mimikatz credential dumping (T1003), and SMB admin share traversal (T1021.002) enriched with Active Directory entity context." + kill_chain: + - stage: credential_access_brute_force + stream_id: winevtlog + timestamp_utc: "2025-07-26T00:00:00Z" + tactic: TA0006 + technique_id: T1110.001 + summary: "Repeated Windows Security Event ID 4625 failed logins followed by Event ID 4624 successful login within 15 minutes." + key_evidence: + - 'metadata.event_type = "USER_LOGIN" AND metadata.product_event_type = "4625"' + - 'metadata.event_type = "USER_LOGIN" AND metadata.product_event_type = "4624"' + - stage: credential_dumping + stream_id: windows_sysmon + timestamp_utc: "2025-07-26T00:05:00Z" + tactic: TA0006 + technique_id: T1003 + summary: "Command-line invocation of Mimikatz observed in PROCESS_LAUNCH telemetry." + key_evidence: + - 'metadata.event_type = "PROCESS_LAUNCH" AND target.process.command_line = /mimikatz/ nocase' + - stage: lateral_movement_admin_shares + stream_id: windows_sysmon + timestamp_utc: "2025-07-26T00:10:00Z" + tactic: TA0008 + technique_id: T1021.002 + summary: "Execution of net use targeting Windows administrative shares (C$, ADMIN$, IPC$)." + key_evidence: + - 'metadata.event_type = "PROCESS_LAUNCH" AND target.process.command_line = /net.*use.*(C|ADMIN|IPC)\$/ nocase' + entities_and_iocs: + compromised_accounts: [] + affected_assets: [] + malicious_iocs: [] + benign_confusers: + signed_lolbins: + - net.exe + - whoami.exe + - powershell.exe + internal_ips: [] + change_window: null +compliance_gates: + requires_pre_containment_verification: true + requires_168h_horizon_widening: false + forbid_external_cti_on_internal_ips: true + forbid_lolbin_hash_blocking: true + respect_change_window: false + expected_actions: + primary: [] + secondary: [] + prohibited: + - block_ioc:net.exe + - block_ioc:whoami.exe + - block_ioc:powershell.exe +counterfactual_pair: null +reference_queries: + google_secops_udm: + - 'metadata.event_type = "PROCESS_LAUNCH" AND target.process.command_line = /net.*use.*(C|ADMIN|IPC)\$/ nocase' + - 'metadata.event_type = "PROCESS_LAUNCH" AND target.process.command_line = /mimikatz/ nocase' + - 'metadata.event_type = "USER_LOGIN" AND metadata.vendor_name = "Microsoft" AND metadata.product_event_type = "4625"' + google_secops_yaral: + - RULES/mitre_attack_T1021_002_windows_admin_share_basic.yl2 + - RULES/rw_mimikatz_T1003.yl2 + - RULES/win_password_spray.yl2 + - RULES/win_repeatedAuthFailure_thenSuccess.yl2 + databricks_sql: + - "SELECT * FROM security_lakehouse.ocsf_silver.process_activity WHERE LOWER(process.cmd_line) LIKE '%mimikatz%'" + sigma: [] + splunk_spl: + - 'sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" CommandLine="*mimikatz*"' diff --git a/usecases/generate_usecase_docs.py b/usecases/generate_usecase_docs.py index d42985b..8d946c1 100644 --- a/usecases/generate_usecase_docs.py +++ b/usecases/generate_usecase_docs.py @@ -12,30 +12,41 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Generate usecase documentation from YAML frontmatter using Jinja2 templates. +"""Generate usecase documentation from YAML frontmatter using Jinja2 or stdlib. -This script processes markdown files with YAML frontmatter and generates -standardized documentation using the Jinja2 template system. +This script processes markdown files with v1 or v2 YAML frontmatter and +generates standardized documentation using Jinja2 when installed or a pure +Python stdlib fallback renderer. """ +from __future__ import annotations + import argparse +import copy import sys from pathlib import Path +from typing import Any import yaml -from jinja2 import Environment, FileSystemLoader + +try: + from jinja2 import Environment, FileSystemLoader +except ImportError: + Environment = None # type: ignore[assignment,misc] + FileSystemLoader = None # type: ignore[assignment,misc] MIN_PARTS_FOR_FRONTMATTER = 3 +SCHEMA_VERSION_V2 = "2.0" -def parse_frontmatter(file_path: Path) -> tuple[dict, str]: +def parse_frontmatter(file_path: Path) -> tuple[dict[str, Any], str]: """Parse YAML frontmatter and content from a markdown file. Args: - file_path: Path to the markdown file + file_path: Path to the markdown file. Returns: - Tuple of (frontmatter_dict, content_after_frontmatter) + Tuple of (frontmatter_dict, content_after_frontmatter). """ with open(file_path, encoding="utf-8") as f: content = f.read() @@ -65,35 +76,387 @@ def parse_frontmatter(file_path: Path) -> tuple[dict, str]: return {}, content -def generate_usecase_doc( - frontmatter: dict, template_path: Path, output_path: Path | None = None -) -> str: - """Generate usecase documentation from frontmatter using Jinja2 template. +def _normalize_stream_entry(stream: dict[str, Any]) -> dict[str, Any]: + """Normalize a v2 telemetry stream entry for template rendering.""" + entry = dict(stream) + entry.setdefault("stream_id", "") + entry.setdefault("log_type", "") + entry.setdefault("product_name", "") + entry.setdefault("vendor_name", "") + entry.setdefault("ocsf_category_uid", None) + entry.setdefault("ocsf_class_uid", None) + entry.setdefault("ocsf_class_name", None) + entry.setdefault("notes", "") + return entry + + +def normalize_frontmatter_for_template( + frontmatter: dict[str, Any], +) -> dict[str, Any]: + """Normalize v1 or v2 frontmatter into backwards-compatible template variables. + + When `schema_version == "2.0"`, maps `telemetry.events`, `telemetry.entities`, + and `platforms.google_secops` into legacy template keys (`events`, `entities`, + `rules`, `saved_searches`, `reference_lists`, `run_frequency`, `data_rbac`) + while preserving all v2 keys (`telemetry`, `platforms`, `environment`, + `schema_version`, `version`, `author`, `license`). Args: - frontmatter: Dictionary containing the YAML frontmatter data - template_path: Path to the Jinja2 template file - output_path: Optional path to write the generated content + frontmatter: Raw v1 or v2 YAML frontmatter dictionary. Returns: - Generated markdown content as string + Normalized dictionary ready for template or fallback rendering. """ - # Set up Jinja2 environment with autoescape enabled - template_dir = template_path.parent - template_name = template_path.name - - env = Environment( - loader=FileSystemLoader(template_dir), - autoescape=True, - trim_blocks=True, - lstrip_blocks=True, + normalized = copy.deepcopy(frontmatter) + schema_version = str(normalized.get("schema_version", "1.0")) + + if schema_version != SCHEMA_VERSION_V2: + return normalized + + telemetry = normalized.get("telemetry") or {} + raw_events = telemetry.get("events") or [] + raw_entities = telemetry.get("entities") or [] + + normalized["events"] = [ + _normalize_stream_entry(item) for item in raw_events if isinstance(item, dict) + ] + normalized["entities"] = [ + _normalize_stream_entry(item) for item in raw_entities if isinstance(item, dict) + ] + + platforms = normalized.get("platforms") or {} + secops = platforms.get("google_secops") or {} + + if "run_frequency" not in normalized and secops.get("run_frequency"): + normalized["run_frequency"] = copy.deepcopy(secops["run_frequency"]) + + if "data_rbac" not in normalized and secops.get("data_rbac"): + normalized["data_rbac"] = secops["data_rbac"] + + raw_detections = secops.get("detections") or [] + normalized["rules"] = [ + { + "name": det.get("name", ""), + "path": det.get("path"), + "live": bool(det.get("live", False)), + "alerting": bool(det.get("alerting", False)), + "notes": det.get("notes", "") or "", + } + for det in raw_detections + if isinstance(det, dict) + ] + + raw_searches = secops.get("saved_searches") or [] + normalized["saved_searches"] = [ + { + "name": s.get("name", ""), + "creator": s.get("creator", "") or "", + "notes": s.get("notes", "") or "", + } + for s in raw_searches + if isinstance(s, dict) + ] + + raw_ref_lists = secops.get("reference_lists") or [] + normalized["reference_lists"] = [ + { + "name": r.get("name", ""), + "type": r.get("type", "String") or "String", + "notes": r.get("notes", "") or "", + } + for r in raw_ref_lists + if isinstance(r, dict) + ] + + return normalized + + +def _format_ocsf_cell(stream: dict[str, Any]) -> str: + """Format OCSF class name and UID for a v2 telemetry stream table cell.""" + class_name = stream.get("ocsf_class_name") or "" + class_uid = stream.get("ocsf_class_uid") + if class_name and class_uid is not None: + return f"{class_name} ({class_uid})" + if class_uid is not None: + return f"({class_uid})" + return str(class_name) + + +def _render_stream_section( + heading: str, + streams: list[dict[str, Any]], + is_v2: bool, + empty_message: str, +) -> list[str]: + """Render Events or Entities Markdown table lines.""" + lines = [f"## {heading}", ""] + valid_streams = [ + s for s in streams if isinstance(s, dict) and s.get("log_type", "N/A") != "N/A" + ] + if not valid_streams: + lines.append(empty_message) + lines.append("") + return lines + + if is_v2: + lines.append( + "| Stream ID | Log Type | Product Name | Vendor Name | OCSF Class | Notes |" + ) + lines.append( + "|-----------|----------|--------------|-------------|------------|-------|" + ) + for stream in valid_streams: + stream_id = stream.get("stream_id", "") or "" + log_type = stream.get("log_type", "") or "" + product_name = stream.get("product_name", "") or "" + vendor_name = stream.get("vendor_name", "") or "" + ocsf_cell = _format_ocsf_cell(stream) + notes = stream.get("notes", "") or "" + lines.append( + f"| {stream_id} | {log_type} | {product_name} | {vendor_name} |" + f" {ocsf_cell} | {notes} |" + ) + else: + lines.append("| Log Type | Product Name | Vendor Name | Notes |") + lines.append("|----------|--------------|-------------|-------|") + for stream in valid_streams: + log_type = stream.get("log_type", "") or "" + product_name = stream.get("product_name", "") or "" + vendor_name = stream.get("vendor_name", "") or "" + notes = stream.get("notes", "") or "" + lines.append(f"| {log_type} | {product_name} | {vendor_name} | {notes} |") + + lines.append("") + return lines + + +def _render_without_jinja2(normalized: dict[str, Any]) -> str: + """Render usecase documentation in pure Python when Jinja2 is not installed. + + Args: + normalized: Frontmatter dictionary normalized via + `normalize_frontmatter_for_template`. + + Returns: + Generated Markdown string matching `usecase_template.j2`. + """ + is_v2 = str(normalized.get("schema_version", "1.0")) == SCHEMA_VERSION_V2 + title = normalized.get("title", "") + description = normalized.get("description", "") + + lines: list[str] = [ + f"# {title}", + "", + "## Introduction", + "", + f"{description}", + "", + ] + + for label, key in ( + ("ID", "id"), + ("Version", "version"), + ("Author", "author"), + ("License", "license"), + ): + val = normalized.get(key) + if val: + lines.extend([f"**{label}**: {val}", ""]) + + tags = normalized.get("tags") + if isinstance(tags, list) and tags: + lines.extend([f"**Tags**: {', '.join(str(t) for t in tags)}", ""]) + + for label, key in (("Created", "created"), ("Updated", "updated")): + val = normalized.get(key) + if val: + lines.extend([f"**{label}**: {val}", ""]) + + env = normalized.get("environment") + if isinstance(env, dict) and env: + lines.extend(["## Environment", ""]) + org = env.get("organization") or "cymbal-investments.org" + scenario_type = env.get("scenario_type") or "attack" + lines.extend([ + f"**Organization**: {org}", + "", + f"**Scenario Type**: {scenario_type}", + "", + ]) + paired = env.get("paired_baselines") + if isinstance(paired, list) and paired: + lines.extend([ + f"**Paired Baselines**: {', '.join(str(b) for b in paired)}", + "", + ]) + + run_freq = normalized.get("run_frequency") + if isinstance(run_freq, dict) and run_freq: + lines.extend(["## Run Frequency", ""]) + if run_freq.get("events"): + lines.extend([f"**Events**: {run_freq['events']}", ""]) + if run_freq.get("entities"): + lines.extend([f"**Entities**: {run_freq['entities']}", ""]) + + data_rbac = normalized.get("data_rbac") + if data_rbac: + lines.extend(["## Data RBAC", "", f"**Note**: {data_rbac}", ""]) + + lines.extend( + _render_stream_section( + "Events", + normalized.get("events") or [], + is_v2=is_v2, + empty_message="No events configured for this usecase.", + ) + ) + lines.extend( + _render_stream_section( + "Entities", + normalized.get("entities") or [], + is_v2=is_v2, + empty_message="No entities configured for this usecase.", + ) ) - # Load and render template - template = env.get_template(template_name) - rendered = template.render(**frontmatter) + # Rules + lines.extend(["## Rules", ""]) + rules = normalized.get("rules") or [] + valid_rules = [ + r for r in rules if isinstance(r, dict) and str(r.get("name", "")).strip() + ] + if valid_rules: + lines.append("| Rule Name | Live | Alerting | Notes |") + lines.append("|-----------|------|----------|-------|") + for rule in valid_rules: + r_name = rule.get("name", "") + r_live = str(rule.get("live", False)).title() + r_alerting = str(rule.get("alerting", False)).title() + r_notes = rule.get("notes", "") or "" + lines.append(f"| {r_name} | {r_live} | {r_alerting} | {r_notes} |") + else: + lines.append("No rules configured for this usecase.") + lines.append("") + + # Saved Searches + searches = normalized.get("saved_searches") or [] + valid_searches = [ + s for s in searches if isinstance(s, dict) and s.get("name", "N/A") != "N/A" + ] + if valid_searches: + lines.extend([ + "## Saved Searches", + "", + "| Search Name | Creator | Notes |", + "|-------------|---------|-------|", + ]) + for search in valid_searches: + s_name = search.get("name", "") + s_creator = search.get("creator", "") or "" + s_notes = search.get("notes", "") or "" + lines.append(f"| {s_name} | {s_creator} | {s_notes} |") + lines.append("") + + # Reference Lists + ref_lists = normalized.get("reference_lists") or [] + valid_ref_lists = [ + r for r in ref_lists if isinstance(r, dict) and r.get("name", "N/A") != "N/A" + ] + if valid_ref_lists: + lines.extend([ + "## Reference Lists", + "", + "| List Name | Type | Notes |", + "|-----------|------|-------|", + ]) + for ref_list in valid_ref_lists: + l_name = ref_list.get("name", "") + l_type = ref_list.get("type", "") or "" + l_notes = ref_list.get("notes", "") or "" + lines.append(f"| {l_name} | {l_type} | {l_notes} |") + lines.append("") + + # Platform Bindings + platforms = normalized.get("platforms") + if isinstance(platforms, dict) and platforms: + lines.extend(["## Platform Bindings", ""]) + secops = platforms.get("google_secops") + if isinstance(secops, dict) and secops: + lines.extend(["### Google SecOps", ""]) + mappings = secops.get("log_type_mappings") or {} + for stream_id, log_type in mappings.items(): + lines.append(f"- `{stream_id}`: `{log_type}`") + lines.append("") + + databricks = platforms.get("databricks") + if isinstance(databricks, dict) and databricks: + catalog = databricks.get("catalog") or "security_lakehouse" + schema_name = databricks.get("schema_name") or "ocsf_silver" + lines.extend([ + "### Databricks", + "", + f"- **Catalog**: `{catalog}`", + f"- **Schema**: `{schema_name}`", + ]) + for stream_id, table_name in (databricks.get("table_mappings") or {}).items(): + lines.append(f"- `{stream_id}`: `{table_name}`") + for det in databricks.get("detections") or []: + lines.append(f"- Detection: `{det}`") + lines.append("") + + sigma = platforms.get("sigma") + if isinstance(sigma, dict) and sigma.get("rules"): + lines.extend(["### Sigma", ""]) + for sigma_rule in sigma["rules"]: + lines.append(f"- `{sigma_rule}`") + lines.append("") + + splunk = platforms.get("splunk") + if isinstance(splunk, dict) and splunk.get("sourcetypes"): + lines.extend(["### Splunk", ""]) + for stream_id, sourcetype in splunk["sourcetypes"].items(): + lines.append(f"- `{stream_id}`: `{sourcetype}`") + lines.append("") + + lines.extend([ + "---", + "", + "*This document was generated from YAML frontmatter using the usecase template.*", + ]) + return "\n".join(lines) + + +def generate_usecase_doc( + frontmatter: dict[str, Any], + template_path: Path, + output_path: Path | None = None, +) -> str: + """Generate usecase documentation from frontmatter using Jinja2 or stdlib. + + Args: + frontmatter: Dictionary containing the YAML frontmatter data. + template_path: Path to the Jinja2 template file. + output_path: Optional path to write the generated content. + + Returns: + Generated markdown content as string. + """ + normalized = normalize_frontmatter_for_template(frontmatter) + + if Environment is not None and FileSystemLoader is not None: + template_dir = template_path.parent + template_name = template_path.name + env = Environment( + loader=FileSystemLoader(template_dir), + autoescape=True, + trim_blocks=True, + lstrip_blocks=True, + ) + template = env.get_template(template_name) + rendered = template.render(**normalized) + else: + rendered = _render_without_jinja2(normalized) - # Write to output file if specified if output_path: with open(output_path, "w", encoding="utf-8") as f: f.write(rendered) @@ -104,21 +467,20 @@ def generate_usecase_doc( def process_usecase_directory( usecase_dir: Path, template_path: Path, output_dir: Path | None = None -): +) -> str | None: """Process a usecase directory, finding markdown files and generating docs. Args: - usecase_dir: Path to usecase directory - template_path: Path to Jinja2 template - output_dir: Optional output directory (defaults to usecase_dir) + usecase_dir: Path to usecase directory. + template_path: Path to Jinja2 template. + output_dir: Optional output directory (defaults to usecase_dir). Returns: - Generated content string or None if markdown file not found + Generated content string or None if markdown file not found. """ if output_dir is None: output_dir = usecase_dir - # Find the main usecase markdown file usecase_name = usecase_dir.name markdown_file = usecase_dir / f"{usecase_name}.md" @@ -128,19 +490,17 @@ def process_usecase_directory( print(f"Processing usecase: {usecase_name}") # noqa: T201 - # Parse frontmatter frontmatter, _content = parse_frontmatter(markdown_file) if not frontmatter: print(f"Warning: No frontmatter found in {markdown_file}") # noqa: T201 return None - # Generate documentation output_file = output_dir / f"{usecase_name}_generated.md" return generate_usecase_doc(frontmatter, template_path, output_file) -def main(): +def main() -> None: """Main entry point for the script.""" parser = argparse.ArgumentParser( description="Generate usecase documentation from YAML frontmatter" @@ -162,13 +522,10 @@ def main(): args = parser.parse_args() - # Resolve paths usecase_path = Path(args.usecase_path).resolve() template_path = Path(args.template).resolve() - # Handle relative template path if not template_path.exists(): - # Try relative to script directory script_dir = Path(__file__).parent template_path = script_dir / args.template @@ -176,9 +533,7 @@ def main(): print(f"Error: Template file not found: {args.template}") # noqa: T201 sys.exit(1) - # Process input if usecase_path.is_file(): - # Single markdown file frontmatter, _content = parse_frontmatter(usecase_path) if not frontmatter: print(f"Error: No frontmatter found in {usecase_path}") # noqa: T201 @@ -196,7 +551,6 @@ def main(): generate_usecase_doc(frontmatter, template_path, output_path) elif usecase_path.is_dir(): - # Usecase directory output_dir = Path(args.output) if args.output else usecase_path if args.dry_run: diff --git a/usecases/usecase_template.j2 b/usecases/usecase_template.j2 index 71a5f92..9c4f464 100644 --- a/usecases/usecase_template.j2 +++ b/usecases/usecase_template.j2 @@ -4,6 +4,22 @@ {{ description }} +{% if id %} +**ID**: {{ id }} +{% endif %} + +{% if version %} +**Version**: {{ version }} +{% endif %} + +{% if author %} +**Author**: {{ author }} +{% endif %} + +{% if license %} +**License**: {{ license }} +{% endif %} + {% if tags %} **Tags**: {{ tags | join(', ') }} {% endif %} @@ -16,6 +32,18 @@ **Updated**: {{ updated }} {% endif %} +{% if environment %} +## Environment + +**Organization**: {{ environment.organization | default('cymbal-investments.org') }} + +**Scenario Type**: {{ environment.scenario_type | default('attack') }} + +{% if environment.paired_baselines %} +**Paired Baselines**: {{ environment.paired_baselines | join(', ') }} +{% endif %} +{% endif %} + {% if run_frequency %} ## Run Frequency @@ -37,11 +65,19 @@ ## Events {% if events and events | length > 0 and events[0].log_type != "N/A" %} +{% if schema_version == "2.0" %} +| Stream ID | Log Type | Product Name | Vendor Name | OCSF Class | Notes | +|-----------|----------|--------------|-------------|------------|-------| +{% for event in events %} +| {{ event.stream_id | default('') }} | {{ event.log_type }} | {{ event.product_name }} | {{ event.vendor_name }} | {{ event.ocsf_class_name | default('') }}{% if event.ocsf_class_uid %} ({{ event.ocsf_class_uid }}){% endif %} | {{ event.notes | default('') }} | +{% endfor %} +{% else %} | Log Type | Product Name | Vendor Name | Notes | |----------|--------------|-------------|-------| {% for event in events %} | {{ event.log_type }} | {{ event.product_name }} | {{ event.vendor_name }} | {{ event.notes | default('') }} | {% endfor %} +{% endif %} {% else %} No events configured for this usecase. {% endif %} @@ -49,11 +85,19 @@ No events configured for this usecase. ## Entities {% if entities and entities | length > 0 and entities[0].log_type != "N/A" %} +{% if schema_version == "2.0" %} +| Stream ID | Log Type | Product Name | Vendor Name | OCSF Class | Notes | +|-----------|----------|--------------|-------------|------------|-------| +{% for entity in entities %} +| {{ entity.stream_id | default('') }} | {{ entity.log_type }} | {{ entity.product_name }} | {{ entity.vendor_name }} | {{ entity.ocsf_class_name | default('') }}{% if entity.ocsf_class_uid %} ({{ entity.ocsf_class_uid }}){% endif %} | {{ entity.notes | default('') }} | +{% endfor %} +{% else %} | Log Type | Product Name | Vendor Name | Notes | |----------|--------------|-------------|-------| {% for entity in entities %} | {{ entity.log_type }} | {{ entity.product_name }} | {{ entity.vendor_name }} | {{ entity.notes | default('') }} | {% endfor %} +{% endif %} {% else %} No entities configured for this usecase. {% endif %} @@ -90,6 +134,53 @@ No rules configured for this usecase. {% endfor %} {% endif %} +{% if platforms %} +## Platform Bindings + +{% if platforms.google_secops %} +### Google SecOps + +{% if platforms.google_secops.log_type_mappings %} +{% for stream_id, log_type in platforms.google_secops.log_type_mappings.items() %} +- `{{ stream_id }}`: `{{ log_type }}` +{% endfor %} +{% endif %} +{% endif %} + +{% if platforms.databricks %} +### Databricks + +- **Catalog**: `{{ platforms.databricks.catalog | default('security_lakehouse') }}` +- **Schema**: `{{ platforms.databricks.schema_name | default('ocsf_silver') }}` +{% if platforms.databricks.table_mappings %} +{% for stream_id, table_name in platforms.databricks.table_mappings.items() %} +- `{{ stream_id }}`: `{{ table_name }}` +{% endfor %} +{% endif %} +{% if platforms.databricks.detections %} +{% for det in platforms.databricks.detections %} +- Detection: `{{ det }}` +{% endfor %} +{% endif %} +{% endif %} + +{% if platforms.sigma and platforms.sigma.rules %} +### Sigma + +{% for sigma_rule in platforms.sigma.rules %} +- `{{ sigma_rule }}` +{% endfor %} +{% endif %} + +{% if platforms.splunk and platforms.splunk.sourcetypes %} +### Splunk + +{% for stream_id, sourcetype in platforms.splunk.sourcetypes.items() %} +- `{{ stream_id }}`: `{{ sourcetype }}` +{% endfor %} +{% endif %} +{% endif %} + --- *This document was generated from YAML frontmatter using the usecase template.*