diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..c77bcdbd --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,129 @@ +# AGENTS.md — stellar-horizon + +## Overview +Horizon is the client-facing HTTP API for the Stellar blockchain network. Ingests ledger data from Stellar Core, serves REST/SSE endpoints, persists to PostgreSQL. + +## Architecture +``` +main.go → cmd/root.go (cobra CLI) → internal/flags.go (NewAppFromFlags) + → internal/app.go (App) → internal/httpx/server.go (HTTP server) + → internal/ingest/ (FSM-based ledger processor) + → internal/txsub/ (transaction submission) +``` + +### Core Subsystems +| Package | Purpose | Entry | +|---------|---------|-------| +| `internal/ingest/` | FSM ledger ingestion. States: start→build→resume loop | `main.go`, `fsm.go` | +| `internal/db2/history/` | DB models + queries. `Q` struct wraps all history ops | `main.go` | +| `internal/actions/` | HTTP handlers. One file per resource type | `account.go`, `transaction.go`, etc. | +| `internal/txsub/` | Tx submission to Core, result tracking | `submitter.go` | +| `internal/httpx/` | Router setup, middleware, server lifecycle | `server.go`, `router.go` | + +### Key Types +- `App` (internal/app.go): Main application container. Holds web server, ingester, submitter, DB sessions +- `Q` (internal/db2/history/main.go): Query builder for all history tables. Wraps squirrel +- `system` (internal/ingest/main.go): Ingestion coordinator. Runs FSM, holds processors + +## Build & Run +```bash +# Build +go build -o stellar-horizon -trimpath -v . + +# Local dev (requires Docker) +./docker/start.sh standalone # or: testnet, pubnet + +# CLI commands +./stellar-horizon serve # Start API server +./stellar-horizon db migrate up +./stellar-horizon ingest verify-range 1000 2000 +``` + +## Testing +```bash +go test ./... # Unit tests (co-located *_test.go) +HORIZON_INTEGRATION_TESTS_ENABLED=true \ # Integration tests + go test -v ./internal/integration/... + +# Integration test setup requires: +# - PostgreSQL (HORIZON_INTEGRATION_TESTS_DOCKER_IMG or local) +# - Stellar Core binary (STELLAR_CORE_BINARY_PATH) +``` +DO NOT use `internal/test/integration/` framework — deprecated, will be EOL. + +## Code Style +**Linting**: golangci-lint with `.golangci.yml` +- Max line length: 140 chars +- Max function length: 100 lines, 50 statements +- Cyclomatic complexity: 15 +- Run: `golangci-lint run` or `./gofmt.sh && ./govet.sh && ./staticcheck.sh` + +**Formatting**: gofmt + goimports (local prefix: `github.com/stellar/stellar-horizon`) + +## Database +PostgreSQL. Migrations in `internal/db2/schema/migrations/` (goose-style numbered SQL). + +Two table categories: +1. **History tables**: Time-series data (`history_transactions`, `history_operations`, `history_effects`) +2. **State tables**: Current ledger state (`accounts`, `offers`, `trust_lines`, `claimable_balances`) + +Generated bindata: `internal/db2/schema/bindata.go` — DO NOT EDIT manually. + +## Ingest FSM (internal/ingest/) +State machine for ledger processing: +``` +start → build (checkpoint) → resume (ledger-by-ledger) ↺ + → historyRange (backfill gaps) + → waitForCheckpoint (if ahead of archives) +``` +- Checkpoint ledger: `(ledger# + 1) mod 64 == 0` +- `lastIngestedLedger`: cumulative + time-series data complete +- `lastHistoryLedger`: time-series only (from `history_ledgers` table) + +Critical: Only ONE instance should write to DB at a time globally. + +## API Endpoints (internal/actions/) +Handlers return JSON or SSE streams. Pattern: +```go +func GetAccount(w http.ResponseWriter, r *http.Request) { + // 1. Parse/validate params via helpers.go + // 2. Query via historyQ (db2/history) + // 3. Adapt to resource via resourceadapter/ + // 4. Render JSON or stream SSE +} +``` + +## Dependencies (go.mod) +| Package | Use | +|---------|-----| +| `github.com/stellar/go-stellar-sdk` | XDR, ingest, history archives | +| `github.com/go-chi/chi` | HTTP router | +| `github.com/Masterminds/squirrel` | SQL query builder | +| `github.com/spf13/cobra` | CLI framework | +| `github.com/prometheus/client_golang` | Metrics | + +## Key Files +| File | Purpose | +|------|---------| +| `internal/flags.go` | All CLI flags, env vars. DEPRECATED flags documented inline | +| `internal/app.go` | App lifecycle: init, serve, tick, shutdown | +| `internal/ingest/fsm.go` | FSM state definitions and transitions | +| `internal/db2/history/main.go` | All DB types (1295 lines). Central data model | +| `internal/httpx/router.go` | Route definitions, middleware chain | + +## Conventions +- Errors: Use `github.com/stellar/go-stellar-sdk/support/errors` for wrapping +- Logging: `github.com/stellar/go-stellar-sdk/support/log` (structured) +- Problems: RFC 7807 via `internal/render/problem/` and `support/render/problem` +- Context: Pass `context.Context` first param, respect cancellation + +## Anti-patterns (from codebase) +- NEVER run `db reingest` on production DB without planning for endpoint unavailability +- DO NOT change key_value store keys in `internal/db2/history/key_value.go` — migration-sensitive +- DO NOT modify `bindata.go` files — auto-generated + +## See Also +- `DEVELOPING.md`: Full dev environment setup +- `CONTRIBUTING.md`: PR workflow +- `internal/docs/TESTING_NOTES.md`: Test patterns and gotchas +- `internal/ingest/README.md`: FSM deep dive with state diagram diff --git a/internal/actions/AGENTS.md b/internal/actions/AGENTS.md new file mode 100644 index 00000000..c7afafe9 --- /dev/null +++ b/internal/actions/AGENTS.md @@ -0,0 +1,74 @@ +# AGENTS.md — internal/actions + +## Purpose +HTTP handlers for Horizon REST API. One file per resource type. + +## Handler Pattern +```go +// account.go +func GetAccountByID(w http.ResponseWriter, r *http.Request) { + // 1. Parse params + accountID, err := getAccountID(r, "account_id") + + // 2. Query DB + historyQ, _ := horizonContext.HistoryQFromRequest(r) + account, err := historyQ.Accounts().ForAccounts(ctx, []string{accountID}).Select(ctx) + + // 3. Adapt to resource + resource := resourceadapter.NewAccount(account) + + // 4. Render + httpjson.Render(w, resource, httpjson.HALJSON) +} +``` + +## File Organization +| File | Endpoints | +|------|-----------| +| `account.go` | `/accounts/{id}`, `/accounts` | +| `transaction.go` | `/transactions/{hash}`, `/transactions` | +| `operation.go` | `/operations/{id}`, `/operations` | +| `ledger.go` | `/ledgers/{seq}`, `/ledgers` | +| `offer.go` | `/offers/{id}`, `/accounts/{id}/offers` | +| `trade.go` | `/trades`, `/accounts/{id}/trades` | +| `path.go` | `/paths/strict-receive`, `/paths/strict-send` | +| `submit_transaction.go` | `POST /transactions` | +| `submit_transaction_async.go` | `POST /transactions_async` | + +## Helper Files +| File | Purpose | +|------|---------| +| `helpers.go` | Param parsing: `getString()`, `getAccountID()`, `getCursor()` | +| `validators.go` | Input validation | +| `query_params.go` | URL query struct definitions | +| `main.go` | Shared types, `ActionContext` | + +## Response Formats +- **JSON**: Standard responses via `httpjson.Render()` +- **SSE**: Streaming via `sse.Stream()` for `/stream` suffix endpoints + +## Pagination Pattern +```go +// GetPageQuery returns cursor, order, limit +pq, _ := GetPageQuery(r, opts...) +query := q.Transactions().Page(pq) +``` + +## Context Access +```go +historyQ, _ := horizonContext.HistoryQFromRequest(r) // DB queries +ledgerState := r.Context().Value(&ledger.State{}) // Current ledger +app := r.Context().Value(&App{}) // Full app access +``` + +## Error Handling +Use `support/render/problem` for RFC 7807 responses: +```go +problem.Render(ctx, w, problem.NotFound) +problem.Render(ctx, w, hProblem.StaleHistory) // Horizon-specific +``` + +## Anti-patterns +- Raw SQL in handlers → use `historyQ` methods +- Blocking in SSE handlers → respect context cancellation +- Ignoring `Latest-Ledger` header → always set via `SetLastLedgerHeader()` diff --git a/internal/db2/history/AGENTS.md b/internal/db2/history/AGENTS.md new file mode 100644 index 00000000..6e0844f6 --- /dev/null +++ b/internal/db2/history/AGENTS.md @@ -0,0 +1,52 @@ +# AGENTS.md — internal/db2/history + +## Purpose +Data access layer for all Horizon DB operations. `Q` struct wraps all queries via squirrel builder. + +## File Organization +| Pattern | Purpose | +|---------|---------| +| `{entity}.go` | Query methods: `Accounts()`, `Select()`, `InsertX()` | +| `{entity}_batch_insert_builder.go` | Bulk insert interfaces + impls | +| `{entity}_loader.go` | Foreign key lookup caching (AccountLoader, AssetLoader) | +| `mock_*.go` | Test doubles for interfaces | +| `main.go` | Core types: `Q`, EffectType constants, all DB struct definitions | + +## Key Types (main.go) +- `Q`: Central query struct. Wraps `db.Session` from go-stellar-sdk +- `EffectType`: 80+ constants for effect categorization (EffectAccountCreated=0, EffectTrade=33, etc.) +- `*BatchInsertBuilder`: Interfaces for bulk upserts during ingestion + +## Query Patterns +```go +// Always use Q methods, not raw SQL +q.Accounts().ForAccounts(ctx, addresses) // Returns AccountsQuery builder +q.Transactions().ForAccount(ctx, address) // Chainable +q.Select(ctx, &dest, query) // Execute and scan +``` + +## Table Categories +| Category | Tables | Query File | +|----------|--------|------------| +| History (time-series) | `history_transactions`, `history_operations`, `history_effects` | `transaction.go`, `operation.go`, `effect.go` | +| State (cumulative) | `accounts`, `offers`, `trust_lines`, `claimable_balances` | `accounts.go`, `offers.go`, etc. | +| Lookup | `history_accounts`, `history_assets` | `account.go`, `asset.go` | + +## Loaders (FK caching) +Used during ingestion to batch-resolve FKs: +```go +loader := history.NewAccountLoader() +future := loader.GetFuture(address) // Queue lookup +loader.Exec(ctx, session) // Bulk resolve +id := future.Value() // Get cached ID +``` + +## Critical Constraints +- DO NOT change `key_value.go` constants (`exp_ingest_last_ledger`, `exp_ingest_version`) — migration-sensitive, distributed locking relies on exact key names +- `GetLastLedgerIngest()` uses `SELECT ... FOR UPDATE` — intentional for distributed lock +- Batch insert builders: call `Add()` then `Exec()` — not calling Exec = data loss + +## Anti-patterns +- Never bypass `Q` methods with raw queries +- Never modify `bindata.go` — auto-generated from migrations +- Avoid `Q.Clone()` unless you need independent transaction scope diff --git a/internal/ingest/AGENTS.md b/internal/ingest/AGENTS.md new file mode 100644 index 00000000..a28d3e9e --- /dev/null +++ b/internal/ingest/AGENTS.md @@ -0,0 +1,57 @@ +# AGENTS.md — internal/ingest + +## Purpose +FSM-based ledger ingestion from Stellar Core into Horizon's PostgreSQL. + +## Architecture +``` +system (main.go) → FSM (fsm.go) → states → processor_runner.go → processors/ + → db2/history/ +``` + +## FSM States (fsm.go) +| State | Purpose | Next | +|-------|---------|------| +| `start` | Entry point. Checks DB version, determines path | build, resume, historyRange, waitForCheckpoint | +| `build` | Initial state from checkpoint archive | resume (success), start (failure) | +| `resume` | Steady-state ledger-by-ledger ingestion | resume (loop), start (error) | +| `historyRange` | Backfill missing time-series data | start | +| `waitForCheckpoint` | Sleep until next 64-ledger checkpoint | start | + +## Key Definitions +- **Checkpoint ledger**: `(ledger# + 1) mod 64 == 0` — when history archives publish +- **lastIngestedLedger**: Both cumulative AND time-series data complete +- **lastHistoryLedger**: Time-series only (from `history_ledgers`) +- FSM ensures these stay in sync + +## Critical Files +| File | Purpose | +|------|---------| +| `main.go` | `system` struct, `Run()` loop | +| `fsm.go` | State definitions, `stateMachineNode` interface | +| `processor_runner.go` | Runs all processors for a ledger range | +| `verify_range_state.go` | Data integrity verification | + +## Running Processors +```go +// processor_runner.go orchestrates +runner.RunAllProcessorsOnLedger(ledger) // Change + Transaction processors +runner.RunTransactionProcessorsOnLedger(ledger) // Time-series only +``` + +## Concurrency Warning +**Only ONE Horizon instance should write to DB globally.** FSM uses `SELECT FOR UPDATE` on `key_value` table for distributed locking. + +## Reingestion +```bash +# Backfill range (offline operation) +./stellar-horizon db reingest range 1000 2000 + +# DO NOT run on production while serving traffic +# Endpoints become unavailable until rebuild complete +``` + +## Anti-patterns +- Never run multiple ingestion instances against same DB +- Never skip `verify-range` after reingest +- State machine errors → fix root cause, don't restart blindly diff --git a/internal/ingest/processors/AGENTS.md b/internal/ingest/processors/AGENTS.md new file mode 100644 index 00000000..11f7f0ac --- /dev/null +++ b/internal/ingest/processors/AGENTS.md @@ -0,0 +1,62 @@ +# AGENTS.md — internal/ingest/processors + +## Purpose +Transform ledger data into database rows. Two processor types: Change (state) and Transaction (time-series). + +## Processor Types +| Interface | Input | Output Tables | +|-----------|-------|---------------| +| `ChangeProcessor` | `ingest.Change` (state delta) | `accounts`, `offers`, `trust_lines`, etc. | +| `LedgerTransactionProcessor` | `ingest.LedgerTransaction` | `history_*` tables | + +## Naming Convention +| Suffix | Type | Example | +|--------|------|---------| +| `*_processor.go` | Core processor impl | `accounts_processor.go` | +| `*_change_processor.go` | State table processor | `liquidity_pools_change_processor.go` | +| `*_transaction_processor.go` | History table processor | `claimable_balances_transaction_processor.go` | + +## Processor Pattern +```go +type FooProcessor struct { + batch history.FooBatchInsertBuilder +} + +func (p *FooProcessor) ProcessTransaction(lcm xdr.LedgerCloseMeta, tx ingest.LedgerTransaction) error { + // Extract data from tx + // p.batch.Add(row) + return nil +} + +func (p *FooProcessor) Flush(ctx context.Context, session db.SessionInterface) error { + return p.batch.Exec(ctx, session) // MUST call or data lost +} +``` + +## Key Processors +| Processor | Tables | Notes | +|-----------|--------|-------| +| `effects_processor.go` | `history_effects` | 80+ effect types, largest processor (~1500 lines) | +| `operations_processor.go` | `history_operations` | One row per operation | +| `transactions_processor.go` | `history_transactions` | Fee bumps handled specially | +| `accounts_processor.go` | `accounts` | State table, change processor | +| `asset_stats_processor.go` | `asset_stats` | Aggregated per-asset metrics | + +## Batch Insert Pattern +All processors use loaders for FK resolution: +```go +accountLoader.GetFuture(address) // Queue +assetLoader.GetFuture(asset) // Queue +// ... after all txs processed ... +accountLoader.Exec(ctx, session) // Bulk resolve +``` + +## Adding New Processor +1. Implement `ChangeProcessor` or `LedgerTransactionProcessor` +2. Register in `processor_runner.go` (change or transaction list) +3. Ensure `Flush()` calls batch builder's `Exec()` + +## Anti-patterns +- Forgetting to call `Flush()` → silent data loss +- Processing failed transactions for effects → effects only exist for successful txs +- Modifying processor order without understanding dependencies