From 003dc48ef5a94d429769f6c17798b1b6ab1a67ef Mon Sep 17 00:00:00 2001 From: Chubin Date: Wed, 19 Aug 2026 11:20:31 -0600 Subject: [PATCH 1/2] Port transformer decomposition and POM subtype-inheritance DB write Rebased on top of psse-parser-consolidation (single PowerModels parse path, no PowerFlowData dep). Introduces the POM-target intermediate layer and the SQLite writer: - ParsedOpenAPIObjects: typed containers for each POM component built from PowerModelsData dicts. - Transformer decomposition to POM shape: TransformerCircuit rows shared by TwoWindingTransformer / ThreeWindingTransformer holders, replacing the old Transformer2W / TapTransformer / PhaseShiftingTransformer triplet. - Adopt POM.ImpedanceCorrectionData (dropped the local shim it duplicated). - src/dbinterface: SQLite schema (schema.sql, triggers.sql), TABLE_SCHEMAS / OPENAPI_FIELDS_TO_DB routing, and db_write.jl to send OpenAPI-typed rows to the DB, ordered so topology/arcs land before FK-dependent rows. - Subtype inheritance in the DB write: the 7 previously-skipped POM types (TransformerCircuit, TwoWindingTransformer, ThreeWindingTransformer, DiscreteControlledACBranch, TwoTerminalLCCLine, TwoTerminalVSCLine, FACTSControlDevice, SwitchedAdmittance, SynchronousCondenser, InterruptibleStandardLoad) share their parent's table with entity_type preserving the concrete class. - src/pm_io/psse.jl: emit Dicts (not NamedTuples) for the nested MinMax limit fields on LCC / VSC / DiscreteControlledACBranch so OpenAPI.from_json can build them. - Consolidate constants into definitions.jl (matching the rebase base's convention; merged our PM/OpenAPI helpers, IDGenerator, and _MAKE_DATABASE_TYPE_ORDER into it. Full state documented in .claude/HANDOFF.md. EOF ) --- .claude/HANDOFF.md | 309 +++ .gitignore | 1 + Project.toml | 18 +- src/PowerFlowFileParser.jl | 13 + src/db_write.jl | 308 +++ src/dbinterface/db_helpers.jl | 110 ++ src/dbinterface/db_schema.jl | 791 ++++++++ src/dbinterface/schema.sql | 574 ++++++ src/dbinterface/triggers.sql | 514 +++++ src/definitions.jl | 342 ++++ src/generator_mapping_pm.yaml | 33 + src/pm_io/data.jl | 2906 ++++++--------------------- src/pm_io/psse.jl | 54 +- src/power_models_data.jl | 3481 ++++++++++++++++++++++++++++++++- 14 files changed, 7132 insertions(+), 2322 deletions(-) create mode 100644 .claude/HANDOFF.md create mode 100644 src/db_write.jl create mode 100644 src/dbinterface/db_helpers.jl create mode 100644 src/dbinterface/db_schema.jl create mode 100644 src/dbinterface/schema.sql create mode 100644 src/dbinterface/triggers.sql create mode 100644 src/generator_mapping_pm.yaml diff --git a/.claude/HANDOFF.md b/.claude/HANDOFF.md new file mode 100644 index 0000000..4cee9ad --- /dev/null +++ b/.claude/HANDOFF.md @@ -0,0 +1,309 @@ +# PowerFlowFileParser.jl — Handoff (branch: `hc/fix_removePSY_breaks`) + +## Status + +End-to-end file → SQLite database works on the `modified_14bus_system.raw` +fixture. All 15 OpenAPI types the parser emits are written to the DB; +concrete-type distinctions are preserved in `entities.entity_type` even +when subtypes ride along on a parent table. `using PowerFlowFileParser` +precompiles cleanly. Test suite is broken (see gaps). + +## Pipeline overview + +Three stages: + +1. **Readers** (`src/power_models_data.jl`, ~3400 lines) produce + OpenAPI-shaped `Dict{String, Any}`s. Entry point: + `parse_to_openapi_dicts(pm_data::PowerModelsData; kwargs...)`. Defined + in the module but **not exported** — reach via qualified access for + debugging. +2. **Typed layer** — `parse_to_openapi_objects(pm_data; kwargs...) → + ParsedOpenAPIObjects` runs `OpenAPI.from_json` per output collection, + validating required fields and enum membership. Returns a struct of + typed `Vector{T}` of POM structs. +3. **DB write** — `make_database(pm_data; path=":memory:", kwargs...) → + SQLite.DB` calls `parse_to_openapi_objects` first (so schema + validation fails before any disk write), initializes the schema via + `make_sqlite!(db)`, and walks the type order. `save_database(db, path)` + persists an in-memory DB to a file. + +**No PSY dependency in the parsing code.** Downstream PSY6 System +construction is a separate consumer (not in this repo). + +## OpenAPI target + +`PowerOperationsOpenAPIModels` (POM) — resolved from +`~/SiennaRepos/InactiveRepos/PowerOpenAPIModels/PowerOperationsOpenAPIModels.jl`. +POM ships the OpenAPI structs; it does **not** ship a DB schema. The +schema lives locally under `src/dbinterface/`. + +Types emitted by our readers, all POM: +`ACBus`, `Area`, `AreaInterchange`, `LoadZone`, `PowerLoad`, +`StandardLoad`, `InterruptibleStandardLoad`, `FixedAdmittance`, +`SwitchedAdmittance`, `ThermalStandard`, `HydroDispatch`, +`RenewableDispatch`, `RenewableNonDispatch`, `SynchronousCondenser`, +`EnergyReservoirStorage`, `Line`, `TwoWindingTransformer`, +`ThreeWindingTransformer`, `TransformerCircuit`, +`DiscreteControlledACBranch`, `TwoTerminalLCCLine`, +`TwoTerminalGenericHVDCLine`, `TwoTerminalVSCLine`, `FACTSControlDevice`, +`Arc`, `ImpedanceCorrectionData`. + +## Reader outputs (`ParsedOpenAPIDicts` / `ParsedOpenAPIObjects`) + +| Reader | Output | POM type(s) | +|---|---|---| +| `read_bus!` | `Dict{Int, Dict}` by bus_number | `ACBus` | +| `read_area!` | `Dict{Int, Dict}` by area_number | `Area` | +| `read_area_interchange!` | `Dict{String, Dict}` | `AreaInterchange` | +| `read_loadzones!` | `Dict{Int, Dict}` by zone_number | `LoadZone` | +| `read_loads!` | NamedTuple | `power_load` / `standard_load` / `interruptible_standard_load` | +| `read_switched_shunt!` | `Dict{String, Dict}` | `SwitchedAdmittance` | +| `read_shunt!` | `Dict{String, Dict}` | `FixedAdmittance` | +| `read_gen!` | NamedTuple | `thermal_standard` / `hydro_dispatch` / `renewable_dispatch` / `renewable_non_dispatch` / `synchronous_condenser` | +| `read_storage!` | `Dict{String, Dict}` | `EnergyReservoirStorage` | +| `read_branch!` | NamedTuple | `line` / `two_winding_transformer` / `discrete_controlled_ac_branch` (also mutates `transformer_circuits`) | +| `read_3w_transformer!` | NamedTuple | `three_winding_transformer` (also mutates `transformer_circuits`) | +| `read_switch_breaker!` | `Dict{String, Dict}` | `DiscreteControlledACBranch` | +| `read_dcline!` | NamedTuple | `two_terminal_lcc_line` / `two_terminal_generic_hvdc_line` | +| `read_vscline!` | `Dict{String, Dict}` | `TwoTerminalVSCLine` | +| `read_facts!` | `Dict{String, Dict}` | `FACTSControlDevice` | +| `read_impedance_correction!` | `Dict{Tuple{Int,String}, Dict}` | `ImpedanceCorrectionData` | + +## Transformer decomposition (POM design) + +POM decomposes transformers into **holder + per-winding circuit**: + +- `TwoWindingTransformer` holder: `id, name, circuit(FK), admittance_units, magnetizing_shunt, shunt_location`. +- `ThreeWindingTransformer` holder: `id, name, primary_circuit(FK), secondary_circuit(FK), tertiary_circuit(FK), star_bus, r_12, x_12, r_23, x_23, r_31, x_31, base_power_12, base_power_23, base_power_31, admittance_units, magnetizing_shunt, shunt_location`. +- `TransformerCircuit` per winding: `id, available, arc, tap, alpha, r, x, control_objective, rating(_b,_c), active/reactive_power_flow, base_power, base_voltage_primary/secondary`. + +Three PSS/E-side variants (`:TwoWindingTransformer`, `:TapTransformer`, +`:PhaseShiftingTransformer` from the branch-type dispatcher) all emit a +`TwoWindingTransformer` holder — the variant only determines which +circuit fields (tap, alpha, control_objective) get populated. Same for 3W +plain vs. phase-shifting. + +Circuits are threaded via a shared `transformer_circuits::Dict{Int, Dict}` +accumulator (mirrors the `arcs` pattern). Populated by both `read_branch!` +and `read_3w_transformer!`; consumed by `_extract_type_dicts/_objects` at +the `:TransformerCircuit` case. + +## ID generation + +`IDGenerator()` — per-parse int counter (`nextid=1`) + cache keyed by +`(type_tag::Symbol, natural_key)`. + +- `getid!(ids, type_tag, natural_key)` returns cached id or mints a new one. +- Passing `nothing` as the natural key returns `nothing` (for optional FKs). +- Cross-references between readers collapse via the cache (whichever + reader hits a given key second gets the same id). +- Composite keys where a single index would collide: + - `:DiscreteControlledACBranch` — `("branch"|"switch"|"breaker", index)` + - `:ImpedanceCorrectionData` — `(table_number, winding_string)` + - `:TransformerCircuit` — `(:TwoWinding, index)` or `(:ThreeWinding, index, :primary|:secondary|:tertiary)` + +## Accumulators threaded through readers + +| Accumulator | Type | Mutated by | +|---|---|---| +| `arcs` | `Dict{Int, Dict}` keyed by Arc id | branch/3W/switch-breaker/dcline/vscline readers | +| `transformer_circuits` | `Dict{Int, Dict}` keyed by circuit id | branch (2W) + 3W readers | +| `supplemental_attribute_associations` | `Vector{Dict}` of `{attribute_id, entity_id}` | branch (2W ICT) + 3W (per-winding ICT) | + +## DB layer + +`src/dbinterface/`: + +- `schema.sql` — CREATE TABLE statements. `two_winding_transformers`, + `three_winding_transformers`, `transformer_circuits` are our + decomposition. Rest matches SOM's schema layout byte-for-byte. +- `triggers.sql`, `db_schema.jl` — table registration + (`TABLE_SCHEMAS`, `OPENAPI_FIELDS_TO_DB`, `JSON_COLUMNS`, entity_types + lists), `make_sqlite!(db)`. +- `db_helpers.jl` — `get_row_field`, `insert_attributes!`, + `insert_uuid!`, statement prep helpers. Direct import of SOM's helper + shape — operates purely on OpenAPI structs. + +`src/db_write.jl`: + +- `_POAM_TYPE_TO_TABLE` — concrete OpenAPI type → target table. Subtypes + ride along on parent tables (see "Subtype inheritance" below). +- `send_openapi_table_to_db!(T, db, components)` — generic per-type + writer; uses `nameof(T)` as `entities.entity_type` so concrete types + are recorded even when sharing a parent's table. +- `send_openapi_table_to_db!(::Type{AreaInterchange}, …)` — inline arc + minting specialization. +- `write_arcs_to_db!`, `write_supplemental_attributes_to_db!`, + `write_supplemental_attribute_associations_to_db!` — helpers for topology + arcs and ICT SAs. + +`src/common.jl`: + +- `_MAKE_DATABASE_TYPE_ORDER` — insertion order. Topology first, then + Line/TransformerCircuit before the 2W/3W holders (FK ordering). + +## `make_database` flow + +``` +parse_to_openapi_objects(pm_data) # from_json validation happens here + ↓ +SQLite.DB open + make_sqlite!(db) # schema + entity_types + prime_mover_types + fuels seed + ↓ +Write topology (Area, LoadZone, ACBus) so `entities` has bus ids + ↓ +write_arcs_to_db! # arcs.from_id/to_id FK to entities + ↓ +write_supplemental_attributes_to_db!(ImpedanceCorrectionData, ...) # ICT SAs + ↓ +Loop through _MAKE_DATABASE_TYPE_ORDER (skipping topology, already done): + for each POM type T: + components = _extract_type_objects(parsed, T) + isempty(components) && continue + send_openapi_table_to_db!(T, db, components) + ↓ +write_supplemental_attribute_associations_to_db! # transformer↔ICT edges +``` + +Synthesized UUIDs (via `UUIDs.uuid4()`) are written into `attributes` +at the `uuid` name slot. One-way parser → DB workflow — no round-trip +back to PSY components. + +## Subtype inheritance in the DB write + +Seven POM types share tables with their parents. `entities.entity_type` +records the concrete OpenAPI type; subtype-specific fields land in the +generic `attributes` table via `insert_attributes!` (which walks +`OpenAPI.to_json(c)` and pushes any key not in the parent's +`TABLE_SCHEMAS[table_name].names`). + +| POM type | Shared table | Reason | +|---|---|---| +| `DiscreteControlledACBranch` | `transmission_lines` | Arc-based AC branch | +| `TwoTerminalLCCLine` | `transmission_lines` | DC line, arc-based | +| `TwoTerminalVSCLine` | `transmission_lines` | DC line, arc-based | +| `FACTSControlDevice` | `loads` | Single-bus injection | +| `SwitchedAdmittance` | `loads` | Single-bus shunt, same category as FixedAdmittance | +| `SynchronousCondenser` | `loads` | Single-bus, avoids required thermal_generators columns (fuel/prime_mover/active_power_limits) that SC lacks | +| `InterruptibleStandardLoad` | `loads` | Follows PowerLoad / StandardLoad grouping | + +## Public API + +```julia +using PowerFlowFileParser + +pm_data = PowerModelsData("path/to/case.raw") # or .m + +# Path 1: raw → SQLite DB +db = make_database(pm_data; path = ":memory:") # or a file path + +# Path 2: raw → typed POM structs (for JSON export or downstream System build) +parsed = parse_to_openapi_objects(pm_data) +parsed.buses # Vector{POM.ACBus} +parsed.transformer_circuits # Vector{POM.TransformerCircuit} +parsed.branches.two_winding_transformer # Vector{POM.TwoWindingTransformer} +parsed.xfrm_3w.three_winding_transformer # Vector{POM.ThreeWindingTransformer} +# … etc. +``` + +Reader kwargs (all silently ignored by readers that don't need them): +`bus_name_formatter`, `gen_name_formatter`, `generator_mapping`, +`branch_name_formatter`, `xfrm_3w_name_formatter`, +`transformer_control_objective_formatter`. + +## Session history (what this branch consolidated) + +- **PSY dependency removed** from `pm_io/`, `im_io/`, and the top-level module. +- **Swap SOM → POM.** Direct dep on `SiennaOpenAPIModels` dropped in favor + of `PowerCoreOpenAPIModels` + `PowerOperationsOpenAPIModels`. +- **Transformer decomposition** to POM's holder + TransformerCircuit + model. Five OpenAPI type Symbols (`:Transformer2W`, `:TapTransformer`, + `:PhaseShiftingTransformer`, `:Transformer3W`, `:PhaseShiftingTransformer3W`) + collapsed to two OUTPUT types (`:TwoWindingTransformer`, + `:ThreeWindingTransformer`) plus a shared circuit collection. +- **ICT shim removed** in favor of POM's `ImpedanceCorrectionData` — the + field/enum layout is identical. `dbinterface/local_types.jl` deleted. +- **Subtype table sharing** wired up. 7 previously-skipped types now + write to shared parent tables with concrete type preserved in + `entities.entity_type`. +- **`make_database` reordered** — topology → arcs → ICTs → everything + else, so `entities` FK from arcs resolves. +- **DC-line NamedTuple → Dict** in `src/pm_io/psse.jl` so + `OpenAPI.from_json` can build MinMax sub-objects. +- **Arc field rename**: `arc.from`/`arc.to` → `arc.from_id`/`arc.to_id` + matching POM's Arc. + +## Known gaps + +1. **Tests broken.** `test/runtests.jl` needs `Logging` in + `test/Project.toml`; `test/test_parse_psse.jl` and + `test/test_parse_matpower.jl` reference `PSY.System(pm_data)` which + no longer exists in this branch. Needs rewrite against + `parse_to_openapi_objects` / `make_database`. +2. **`PowerFlowDataNetwork` workflow not ported** — no + `parse_to_openapi_objects` / `make_database` methods for the + `PowerFlowData.Network` path. Rebasing onto + `origin/psse-parser-consolidation` drops this concern entirely + (that branch removes the PowerFlowData path upstream). +3. **Stale mappings in `OPENAPI_FIELDS_TO_DB`:** `("arcs", "from") => + "from_id"` and `("arcs", "to") => "to_id"` are no-ops now that POM's + Arc has `from_id`/`to_id` directly. Harmless but cleanup-worthy. + +## Deferred design decisions (HANDOFF originals still open) + +- **`bustype` type 3** — currently `"REF"`. Confirm downstream doesn't + need `"SLACK"` instead. +- **`ext` field data loss** — settled as accepted loss (Sienna doesn't + use these fields per user directive). +- **FACTS `control_mode` code 3** — PSY accepts 0-3, POM enum has only + `{"OOS", "NML", "BYP"}`. We throw on code 3 (`_normalize_facts_control_mode`). + Either widen the enum or document a coercion rule. +- **Synthetic UUIDs** — `UUIDs.uuid4()` in the DB `attributes` table. + Fine for one-way parser → DB; no round-trip to PSY components. +- **Single-point piecewise cost curves** — `_thermal_variable_cost_and_fixed` + will `BoundsError` on 1-point piecewise (matches PSY's behavior). + +## File layout + +``` +src/ +├── PowerFlowFileParser.jl # module: exports, imports, includes +├── common.jl # shared constants, _MAKE_DATABASE_TYPE_ORDER +├── db_write.jl # per-type writer + _POAM_TYPE_TO_TABLE +├── power_models_data.jl # readers + parse_to_openapi_dicts/objects + make_database +├── powerflowdata_data.jl # PowerFlowDataNetwork wrapper (no OpenAPI methods yet) +├── generator_mapping_pm.yaml # fuel/prime-mover mapping +├── dbinterface/ +│ ├── db_schema.jl # TABLE_SCHEMAS, OPENAPI_FIELDS_TO_DB, entity_types seed, make_sqlite! +│ ├── db_helpers.jl # get_row_field, insert_attributes!, statement prep +│ ├── schema.sql # CREATE TABLE statements +│ └── triggers.sql # trigger definitions +├── pm_io/ # upstream PowerModels-style parsers (PSY-free) +│ ├── common.jl, data.jl, matpower.jl, psse.jl, pti.jl, LICENSE.md +├── pm_io.jl # pm_io/ includes +├── im_io/ # InfrastructureModels-style parsers (PSY-free) +│ ├── common.jl, data.jl, matlab.jl, LICENSE.md +└── im_io.jl # im_io/ includes +``` + +## Reference paths + +- **Upstream PSY parser** (historical reference for reader field names): + `~/SiennaRepos/Extra-unused-PowerSystems.jl/src/parsers/power_models_data.jl` +- **POM checkout** (target OpenAPI schema): + `~/SiennaRepos/InactiveRepos/PowerOpenAPIModels/PowerOperationsOpenAPIModels.jl/` +- **SOM checkout** (companion — DB helper shape lives here, transformer + decomposition NOT yet reflected in SOM's main branch): + `~/SiennaRepos/PowerSystemSchemas/SiennaOpenAPIModels.jl/` +- **Sibling branch to consider rebasing onto**: + `origin/psse-parser-consolidation` — free-format PTI, v30 native path, + drops PowerFlowData entirely. + +## Related branches in this repo + +| Branch | Direction | Relation | +|---|---|---| +| `main` | Baseline (still uses PSY) | Ancestor of everything below | +| `origin/psse-parser-consolidation` | Parser-only fixes (v30 native, free-format PTI, drop PowerFlowData) | Contains fixes we don't have | +| `origin/jd/openapi-json-export` | Builds `src/openapi/` SOM emit layer on top of psse-parser-consolidation | Opposite direction from this branch | +| `origin/psy6` | jd/openapi-json-export + HVDC + oneOf wrap | Descendant of jd/openapi-json-export | +| `hc/fix_removePSY_breaks` (this one) | Removes PSY, inlines DB schema, targets POM | Diverged directly from main | diff --git a/.gitignore b/.gitignore index 45464e7..27151bb 100644 --- a/.gitignore +++ b/.gitignore @@ -39,6 +39,7 @@ docs/site/ Manifest.toml .vscode *.h5 +.claude/HANDOFF.md ################################################################################ # Operating systems # diff --git a/Project.toml b/Project.toml index 8a9606b..c58e602 100644 --- a/Project.toml +++ b/Project.toml @@ -4,18 +4,34 @@ version = "0.2.0" authors = ["Sienna Team"] [deps] +DBInterface = "a10d1c49-ce27-4219-8d33-6db1a4562965" DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" DocStringExtensions = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae" InfrastructureSystems = "2cd47ed4-ca9b-11e9-27f2-ab636a7671f1" +JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6" LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" +OpenAPI = "d5e62ea6-ddf3-4d43-8e4c-ad5e6c8bfd7d" +PowerCoreOpenAPIModels = "b7b40286-e793-417d-a9a0-b1583e4da1cb" +PowerOperationsOpenAPIModels = "a372b6d7-45a2-44c2-8199-6a724b72e8ff" +SQLite = "0aa819cd-b072-5ff4-a722-6bc24af294d9" +Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" +UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" Unicode = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" YAML = "ddb6d928-2868-570f-bddf-ab3f9cf99eb6" [compat] +DBInterface = "^2.6" DataStructures = "0.19.3" DocStringExtensions = "~0.8, ~0.9" -InfrastructureSystems = "^3.2" +InfrastructureSystems = "^3.6" +JSON = "0.21, 1" LinearAlgebra = "1" +OpenAPI = "0.2.6" +PowerCoreOpenAPIModels = "0.1" +PowerOperationsOpenAPIModels = "0.1" +SQLite = "^1.6" +Tables = "1.13.0" +UUIDs = "1" Unicode = "1" YAML = "0.4.16" julia = "^1.10" diff --git a/src/PowerFlowFileParser.jl b/src/PowerFlowFileParser.jl index 4c73eca..9be1358 100644 --- a/src/PowerFlowFileParser.jl +++ b/src/PowerFlowFileParser.jl @@ -7,12 +7,22 @@ module PowerFlowFileParser export PowerModelsData export parse_file +export IDGenerator +export ParsedOpenAPIObjects +export parse_to_openapi_objects +export make_database +export save_database +export send_openapi_table_to_db! ################################################################################# # Imports import LinearAlgebra import DataStructures: SortedDict +import PowerCoreOpenAPIModels +import PowerOperationsOpenAPIModels +import PowerOperationsOpenAPIModels: ImpedanceCorrectionData +import SQLite import Unicode: normalize import YAML @@ -27,6 +37,9 @@ import InfrastructureSystems: # Includes include("definitions.jl") +include("dbinterface/db_schema.jl") +include("dbinterface/db_helpers.jl") +include("db_write.jl") include("power_models_data.jl") include("im_io.jl") include("pm_io.jl") diff --git a/src/db_write.jl b/src/db_write.jl new file mode 100644 index 0000000..17e850a --- /dev/null +++ b/src/db_write.jl @@ -0,0 +1,308 @@ +# ============================================================================= +# Parser-agnostic DB-write layer. +# Materializes OpenAPI structs from our dicts and inserts them into an +# SQLite database whose schema is created by `make_sqlite!` (see +# `dbinterface/db_schema.jl`). +# +# Most of the heavy lifting is delegated to helpers duplicated from +# `SiennaOpenAPIModels.jl/src/dbinterface/sqlite.jl` (`get_row_field`, +# `insert_attributes!`, `insert_uuid!`, `prepare_*_insert`, `TABLE_SCHEMAS`) +# — they operate purely on `OpenAPI.APIModel` structs, so they work +# equally well against `PowerOperationsOpenAPIModels` structs. We only +# contribute a thin loop that supplies OpenAPI structs in place of the +# PSY components SOM's `sys2db!` flow expects. The type → table map is +# kept local (`_POAM_TYPE_TO_TABLE`) so the DB writers key off POM +# structs rather than a PSY-keyed dict. +# +# Where PSY writes `IS.get_uuid(component)` to record the round-trip UUID, +# we synthesize via `UUIDs.uuid4()` — the dict pipeline has no PSY UUID to +# preserve. This forfeits round-trip-to-PSY ability in the DB but is a +# one-way parser → DB workflow concern documented in HANDOFF.md (open +# question #11). +# ============================================================================= + +import SQLite +import DBInterface +import JSON +import UUIDs +using Tables +import OpenAPI +import PowerOperationsOpenAPIModels + +# Local POM-keyed type→table map. Mirrors the SOM-keyed `TYPE_TO_TABLE` at +# `SiennaOpenAPIModels.jl/src/dbinterface/translation_constants.jl:19-57` +# but keyed by `PowerOperationsOpenAPIModels` structs so the DB writers +# don't depend on SOM's PSY-keyed dict. Only the entries used by our +# `_MAKE_DATABASE_TYPE_ORDER` walk are included. +const _POAM_TYPE_TO_TABLE = Dict{DataType, String}( + PowerOperationsOpenAPIModels.Area => "planning_regions", + PowerOperationsOpenAPIModels.LoadZone => "balancing_topologies", + PowerOperationsOpenAPIModels.ACBus => "balancing_topologies", + PowerOperationsOpenAPIModels.AreaInterchange => "transmission_interchanges", + PowerOperationsOpenAPIModels.Line => "transmission_lines", + PowerOperationsOpenAPIModels.TransformerCircuit => "transformer_circuits", + PowerOperationsOpenAPIModels.TwoWindingTransformer => "two_winding_transformers", + PowerOperationsOpenAPIModels.TwoTerminalGenericHVDCLine => "transmission_lines", + PowerOperationsOpenAPIModels.ThreeWindingTransformer => "three_winding_transformers", + PowerOperationsOpenAPIModels.PowerLoad => "loads", + PowerOperationsOpenAPIModels.StandardLoad => "loads", + PowerOperationsOpenAPIModels.FixedAdmittance => "loads", + PowerOperationsOpenAPIModels.ThermalStandard => "thermal_generators", + PowerOperationsOpenAPIModels.RenewableDispatch => "renewable_generators", + PowerOperationsOpenAPIModels.RenewableNonDispatch => "renewable_generators", + PowerOperationsOpenAPIModels.HydroDispatch => "hydro_generators", + PowerOperationsOpenAPIModels.EnergyReservoirStorage => "storage_units", + # Subtypes that ride along on the parent's table. Concrete OpenAPI type + # fields land in typed columns where names match; everything else falls + # through to the generic `attributes` table via `insert_attributes!`. + # `entities.entity_type` records the concrete OpenAPI type name, so + # readback via `db2openapi_json` can reconstruct the right struct. + PowerOperationsOpenAPIModels.DiscreteControlledACBranch => "transmission_lines", + PowerOperationsOpenAPIModels.TwoTerminalLCCLine => "transmission_lines", + PowerOperationsOpenAPIModels.TwoTerminalVSCLine => "transmission_lines", + PowerOperationsOpenAPIModels.FACTSControlDevice => "loads", + PowerOperationsOpenAPIModels.SwitchedAdmittance => "loads", + PowerOperationsOpenAPIModels.SynchronousCondenser => "loads", + PowerOperationsOpenAPIModels.InterruptibleStandardLoad => "loads", +) + +""" +Per-type generic DB writer. Materializes a row from each OpenAPI struct +via `get_row_field` (pure OpenAPI struct access — no PSY needed) and runs +the entity / table / attribute / uuid inserts in the same order PSY's +`add_components_to_tables!` does. + +Mirrors `add_components_to_tables!` at +`SiennaOpenAPIModels.jl/src/dbinterface/sqlite.jl:106-133`, with the +PSY-specific lines (`IS.get_uuid(component)` and +`sienna2openapi(component, ids)`) removed: components are already OpenAPI +structs, and the UUID is synthesized. +""" +function _add_openapi_components_to_tables!( + ::Type{T}, + table_name::AbstractString, + schema::Tables.Schema, + table_statement::DBInterface.Statement, + entity_statement::DBInterface.Statement, + attribute_statement::DBInterface.Statement, + components, +) where {T <: OpenAPI.APIModel} + for c in components + row = tuple( + (get_row_field(c, table_name, col_name) for col_name in schema.names)..., + ) + try + DBInterface.execute(entity_statement, (c.id,)) + DBInterface.execute(table_statement, row) + catch e + if isa(e, SQLite.SQLiteException) + error("Failed to insert into $(table_name): $(e.msg) with values $(row)") + else + rethrow(e) + end + end + insert_attributes!(T, table_name, schema, attribute_statement, c) + insert_uuid!(attribute_statement, table_name, c.id, UUIDs.uuid4()) + end +end + +""" +Write every component of OpenAPI type `T` to its corresponding DB table. +Resolves the table name via `_POAM_TYPE_TO_TABLE[T]` and the schema via +`TABLE_SCHEMAS[table_name]`, then dispatches to +[`_add_openapi_components_to_tables!`]. + +Mirrors `send_table_to_db!` at +`SiennaOpenAPIModels.jl/src/dbinterface/sqlite.jl:240-254`. +""" +function send_openapi_table_to_db!( + ::Type{T}, + db, + components, +) where {T <: OpenAPI.APIModel} + table_name = _POAM_TYPE_TO_TABLE[T] + obj_type = string(nameof(T)) + schema = TABLE_SCHEMAS[table_name] + _add_openapi_components_to_tables!( + T, + table_name, + schema, + prepare_schema_insert(db, table_name, schema), + prepare_entity_insert(db, table_name, obj_type), + prepare_attributes_insert(db), + components, + ) +end + +""" +Specialization for `AreaInterchange` — the DB row references an arc id +that's synthesized at write time (PSY's AreaInterchange holds +from_area/to_area Area refs, and a fresh arc per interchange is created +inline). Mirrors the same logic in +`SiennaOpenAPIModels.jl/src/dbinterface/sqlite.jl:157-179`, with `IS.get_uuid` +removed. +""" +function send_openapi_table_to_db!( + ::Type{PowerOperationsOpenAPIModels.AreaInterchange}, + db, + components, +) + table_name = "transmission_interchanges" + obj_type = "AreaInterchange" + schema = TABLE_SCHEMAS[table_name] + table_statement = prepare_schema_insert(db, table_name, schema) + arc_statement = prepare_schema_insert(db, "arcs", TABLE_SCHEMAS["arcs"]) + arc_entity_statement = prepare_entity_insert(db, "arcs", "Arc") + attribute_statement = prepare_attributes_insert(db) + entity_statement = prepare_entity_insert(db, table_name, obj_type) + + for c in components + # PSY's `getid!(ids, UUIDs.uuid4())` minted a fresh int id from a + # random UUID. Here we just take the int id directly from a fresh + # uuid4 hash; the only constraint is uniqueness within the DB. + new_arc_id = Int(rand(Int32)) + DBInterface.execute(arc_entity_statement, (new_arc_id,)) + DBInterface.execute(arc_statement, (new_arc_id, c.from_area, c.to_area)) + row = (c.id, c.name, new_arc_id, c.flow_limits.to_from, c.flow_limits.from_to) + DBInterface.execute(entity_statement, (c.id,)) + DBInterface.execute(table_statement, row) + insert_attributes!( + PowerOperationsOpenAPIModels.AreaInterchange, + table_name, + schema, + attribute_statement, + c, + ) + insert_uuid!(attribute_statement, table_name, c.id, UUIDs.uuid4()) + end +end + +""" +Write every Arc struct in `arcs` to the `arcs` table. Each Arc carries +`id`/`from`/`to` fields populated upstream by `_get_or_mint_arc_id!`. Each +Arc gets an `entities` row tagged `(entity_table='arcs', entity_type='Arc')`. + +Accepts any iterable of `PowerOperationsOpenAPIModels.Arc` structs. + +Run this **before** any branch-like type that references arc ids +(`Line`, `TwoWindingTransformer`, `TapTransformer`, `PhaseShiftingTransformer`, +`DiscreteControlledACBranch`, `TwoTerminalLCCLine`, +`TwoTerminalGenericHVDCLine`, `TwoTerminalVSCLine`, etc.) so the FK +references the existing arc row. +""" +function write_arcs_to_db!(db, arcs) + schema = TABLE_SCHEMAS["arcs"] + arc_statement = prepare_schema_insert(db, "arcs", schema) + entity_statement = prepare_entity_insert(db, "arcs", "Arc") + for arc in arcs + DBInterface.execute(entity_statement, (arc.id,)) + DBInterface.execute(arc_statement, (arc.id, arc.from_id, arc.to_id)) + end +end + +""" +Write a homogeneous collection of supplemental-attribute structs to the +`supplemental_attributes` table. Mirrors +`SiennaOpenAPIModels.jl/src/dbinterface/sqlite.jl:256-287` (`serialize_supplemental_attributes!`), +with PSY-iteration replaced by direct iteration over our typed Vector. + +Currently used for `ImpedanceCorrectionData` (PSS/E only). Generalizes +trivially to other supplemental-attribute types by passing a different +`T` and a different `Vector{T}` collection. +""" +function write_supplemental_attributes_to_db!( + ::Type{T}, + db, + sa_objects, +) where {T <: OpenAPI.APIModel} + type_name = string(nameof(T)) + entity_stmt = DBInterface.prepare( + db, + "INSERT INTO entities (id, entity_table, entity_type) VALUES (?, 'supplemental_attributes', ?)", + ) + sa_stmt = DBInterface.prepare( + db, + "INSERT INTO supplemental_attributes (id, TYPE, value) VALUES (?, ?, json(?))", + ) + attr_stmt = prepare_attributes_insert(db) + + for sa in sa_objects + DBInterface.execute(entity_stmt, (sa.id, type_name)) + DBInterface.execute(sa_stmt, (sa.id, type_name, JSON.json(sa))) + insert_uuid!(attr_stmt, "supplemental_attributes", sa.id, UUIDs.uuid4()) + end +end + +""" +Write the `supplemental_attribute_associations` table from the accumulator +of `{"attribute_id", "entity_id"}` dicts collected during the parse. +Mirrors `serialize_supplemental_attribute_associations!` at +`SiennaOpenAPIModels.jl/src/dbinterface/sqlite.jl:289-310`. + +Run this **after** both the referenced supplemental attribute rows and the +referenced component rows are in the DB (so the FK references exist). +""" +function write_supplemental_attribute_associations_to_db!( + db, + associations::Vector{Dict{String, Any}}, +) + stmt = DBInterface.prepare( + db, + "INSERT INTO supplemental_attributes_association (attribute_id, entity_id) VALUES (?, ?)", + ) + for assoc in associations + DBInterface.execute(stmt, (assoc["attribute_id"], assoc["entity_id"])) + end +end + +""" +Copy an open SQLite database to a self-contained file at `path` using +SQLite's `VACUUM INTO` command. Works for both in-memory and file-backed +source databases; the resulting file is a complete, ready-to-share SQLite +database (no WAL or journal companions needed). + +Useful as the second half of the "build in-memory, persist later" workflow: + +```julia +db = make_database(pm_data) # in-memory, default +# …inspect, query, mutate in-process… +save_database(db, "case_30bus.db") # writes a portable file +``` + +# Arguments + +- `db`: an open `SQLite.DB` connection (in-memory or file-backed). +- `path::AbstractString`: destination file path. + +# Keyword Arguments + +- `overwrite::Bool = false`: if `true`, remove an existing file at `path` + before writing. SQLite's `VACUUM INTO` itself refuses to overwrite, so + without this we'd surface the raw "output file already exists" error. + +# Returns + +The destination `path` (for chaining). + +# Throws + +`ArgumentError` if `path` already exists and `overwrite=false`. +Whatever `SQLite.SQLiteException` `VACUUM INTO` raises on permission, +disk-full, or invalid-path errors. +""" +function save_database(db::SQLite.DB, path::AbstractString; overwrite::Bool = false) + if isfile(path) + overwrite || throw( + ArgumentError( + "File $path already exists; pass `overwrite=true` to replace it.", + ), + ) + rm(path) + end + # SQL string literal: escape single quotes by doubling them, per SQL92. + # The path is part of the VACUUM INTO statement itself, not a bind + # parameter, so we can't use DBInterface parameter binding here. + escaped = replace(String(path), "'" => "''") + DBInterface.execute(db, "VACUUM INTO '$escaped'") + return path +end diff --git a/src/dbinterface/db_helpers.jl b/src/dbinterface/db_helpers.jl new file mode 100644 index 0000000..317ef53 --- /dev/null +++ b/src/dbinterface/db_helpers.jl @@ -0,0 +1,110 @@ +# ============================================================================= +# Per-row / per-attribute DB write helpers. +# +# Duplicated from `SiennaOpenAPIModels.jl/src/dbinterface/sqlite.jl` — only +# the writer helpers that operate on `OpenAPI.APIModel` structs are kept. +# The PSY-typed `get_row(::PSY.Component)` variants, the sys2db!/db2sys! +# orchestrators, and the HydroReservoir _ignoreattribute specialization +# are dropped (PFFP never emits PSY components or HydroReservoirs). +# ============================================================================= + +import DBInterface +import JSON +using Tables +import OpenAPI + +""" +Fetch column `col_name` from the OpenAPI struct `c`, honoring the +(table, db_column) → openapi_field renames in `DB_TO_OPENAPI_FIELDS` and +JSON-serializing any column listed in `JSON_COLUMNS` (except the +`thermal_generators.fuel` FK, which is a plain string). +Returns `nothing` for missing properties. +""" +function get_row_field(c::OpenAPI.APIModel, table_name::AbstractString, col_name::Symbol) + col_str = string(col_name) + k = Symbol(get(DB_TO_OPENAPI_FIELDS, (table_name, col_str), col_name)) + + if !hasproperty(c, k) + return nothing + end + + val = getproperty(c, k) + + # Serialize JSON columns (skip fuel for thermal_generators — plain FK, not JSON). + if col_str in JSON_COLUMNS && val !== nothing + if col_str == "fuel" && table_name == "thermal_generators" + return val + end + return JSON.json(val) + end + + return val +end + +""" +Return `true` if OpenAPI field `k` for type `T` (table `table_name`) +already maps to a typed column in `schema` — meaning it should be +skipped when walking the JSON representation to populate the generic +`attributes` table. +""" +function _ignoreattribute( + ::Type{T}, + table_name::AbstractString, + schema::Tables.Schema, + k::AbstractString, +) where {T <: OpenAPI.APIModel} + col_name = get(OPENAPI_FIELDS_TO_DB, (table_name, k), k) + return in(Symbol(col_name), schema.names) +end + +""" +Write every OpenAPI field of `c` that doesn't already live in a typed +column of `table_name` into the generic `attributes` table +(`entity_id`, `type='FromSienna'`, `name`, `value` as JSON). +""" +function insert_attributes!( + ::Type{T}, + table_name::AbstractString, + schema::Tables.Schema, + attribute_statement, + c::OpenAPI.APIModel, +) where {T <: OpenAPI.APIModel} + for (k, v) in JSON.parse(OpenAPI.to_json(c)) + if !_ignoreattribute(T, table_name, schema, k) + DBInterface.execute(attribute_statement, (c.id, "FromSienna", k, JSON.json(v))) + end + end +end + +""" +Record the round-trip UUID for a row by writing an `attributes` row with +`name='uuid'` and a JSON-serialized UUID string. +""" +function insert_uuid!(attribute_statement, table_name, id, uuid) + DBInterface.execute( + attribute_statement, + (id, table_name, "uuid", JSON.json(string(uuid))), + ) +end + +function prepare_schema_insert(db, table_name::AbstractString, schema::Tables.Schema) + return DBInterface.prepare( + db, + """INSERT INTO $table_name ($(join(schema.names, ", "))) + VALUES ($(join(repeat("?", length(schema.names)), ", ")))""", + ) +end + +function prepare_entity_insert(db, table_name::AbstractString, obj_type::AbstractString) + return DBInterface.prepare( + db, + "INSERT INTO entities (id, entity_table, entity_type) VALUES (?, '$table_name', '$obj_type')", + ) +end + +function prepare_attributes_insert(db) + return DBInterface.prepare( + db, + "INSERT INTO attributes (entity_id, type, name, value) VALUES (?, ?, ?, json(?))", + ) +end diff --git a/src/dbinterface/db_schema.jl b/src/dbinterface/db_schema.jl new file mode 100644 index 0000000..0fcf4cd --- /dev/null +++ b/src/dbinterface/db_schema.jl @@ -0,0 +1,791 @@ +# ============================================================================= +# DB schema definition and initialization. +# +# Duplicated from `SiennaOpenAPIModels.jl/src/dbinterface/db_definition.jl` +# so the parser can build and populate the Sienna SQLite schema without +# taking a (transitive) PowerSystems.jl dependency. The SQL text and the +# `TABLE_SCHEMAS` table map are kept byte-identical to SOM; only +# `make_sqlite!` is trimmed — it iterates a local list of the OpenAPI type +# names PFFP actually writes, in place of SOM's PSY-keyed type registries. +# ============================================================================= + +import SQLite +import DBInterface +using Tables + +function _read_sql_statements(filepath::AbstractString) + sql_content = read(filepath, String) + statements = split(sql_content, ';') + cleaned_statements = [strip(s) for s in statements if !isempty(strip(s))] + return cleaned_statements +end + +# Track the SQL files as precompilation dependencies so editing them +# (without touching any .jl) correctly invalidates the precompile cache. +include_dependency(joinpath(@__DIR__, "schema.sql")) +include_dependency(joinpath(@__DIR__, "triggers.sql")) + +const SQLITE_CREATE_STR = _read_sql_statements(joinpath(@__DIR__, "schema.sql")) +const SQLITE_TRIGGERS_STR = [read(joinpath(@__DIR__, "triggers.sql"), String)] + +# OpenAPI field name → DB column name overrides, keyed by (table, openapi_field). +# Used by `insert_attributes!` and `get_row_field` to detect which OpenAPI +# fields land in typed columns vs. the generic `attributes` table. +const OPENAPI_FIELDS_TO_DB = Dict( + ("transmission_lines", "arc") => "arc_id", + ("transformer_circuits", "arc") => "arc_id", + ("two_winding_transformers", "circuit") => "circuit_id", + ("three_winding_transformers", "primary_circuit") => "primary_circuit_id", + ("three_winding_transformers", "secondary_circuit") => "secondary_circuit_id", + ("three_winding_transformers", "tertiary_circuit") => "tertiary_circuit_id", + ("thermal_generators", "bus") => "balancing_topology", + ("renewable_generators", "bus") => "balancing_topology", + ("hydro_generators", "bus") => "balancing_topology", + ("storage_units", "bus") => "balancing_topology", + ("loads", "bus") => "balancing_topology", + ("arcs", "from") => "from_id", + ("arcs", "to") => "to_id", + ("transmission_lines", "rating") => "continuous_rating", +) + +const DB_TO_OPENAPI_FIELDS = Dict((s[1], t) => s[2] for (s, t) in OPENAPI_FIELDS_TO_DB) + +# Columns whose values are stored as JSON strings in SQLite. Consulted by +# `get_row_field` when serializing rows out of OpenAPI structs and by the +# db-read path when reconstituting dicts from rows. +const JSON_COLUMNS = Set([ + "operation_cost", + "active_power_limits", + "reactive_power_limits", + "ramp_limits", + "time_limits", + "outflow_limits", + "storage_level_limits", + "input_active_power_limits", + "output_active_power_limits", + "efficiency", + "spillage_limits", + "head_to_volume_factor", + "region", + "fuel", + "capacity_limits", + "capacity_limits_charge", + "capacity_limits_discharge", + "capacity_limits_energy", + "co2", + "capital_costs", + "capital_costs_charge", + "capital_costs_discharge", + "capital_costs_energy", + "operation_costs", + "cofire_start_limits", + "cofire_level_limits", + "financial_data", + "unserved_demand_curve", + "duration_limits", + "features", +]) + +const TABLE_SCHEMAS = Dict( + "entities" => + Tables.Schema(["id", "entity_table", "entity_type"], [Int64, String, String]), + "entity_types" => Tables.Schema(["name", "is_topology"], [String, Bool]), + "prime_mover_types" => Tables.Schema( + ["id", "name", "description"], + [Int64, String, Union{String, Nothing}], + ), + "fuels" => Tables.Schema( + ["id", "name", "description"], + [Int64, String, Union{String, Nothing}], + ), + "planning_regions" => Tables.Schema( + ["id", "name", "description"], + [Int64, String, Union{String, Nothing}], + ), + "balancing_topologies" => Tables.Schema( + ["id", "name", "area", "description"], + [Int64, String, Union{Int64, Nothing}, Union{String, Nothing}], + ), + "arcs" => Tables.Schema(["id", "from_id", "to_id"], [Int64, Int64, Int64]), + "transmission_lines" => Tables.Schema( + [ + "id", + "name", + "arc_id", + "continuous_rating", + "ste_rating", + "lte_rating", + "line_length", + ], + [ + Int64, + String, + Int64, + Float64, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + ], + ), + "transmission_interchanges" => Tables.Schema( + ["id", "name", "arc_id", "max_flow_from", "max_flow_to"], + [Int64, String, Int64, Float64, Float64], + ), + # Per-winding electricals for both 2W and 3W transformers. Mirrors POM's + # TransformerCircuit. Column order MUST match the CREATE TABLE in schema.sql. + "transformer_circuits" => Tables.Schema( + [ + "id", + "available", + "arc_id", + "tap", + "alpha", + "parameter_units", + "r", + "x", + "control_objective", + "regulated_bus_number", + "number_of_tap_positions", + "rating", + "rating_b", + "rating_c", + "active_power_flow", + "reactive_power_flow", + "base_power", + "base_voltage_primary", + "base_voltage_secondary", + ], + [ + Int64, + Bool, + Int64, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{String, Nothing}, + Float64, + Float64, + Union{String, Nothing}, + Union{Int64, Nothing}, + Union{Int64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + ], + ), + # TwoWindingTransformer holder — electricals live on the referenced + # TransformerCircuit row. + "two_winding_transformers" => Tables.Schema( + ["id", "name", "circuit_id", "admittance_units", "shunt_location"], + [ + Int64, + String, + Int64, + Union{String, Nothing}, + Union{String, Nothing}, + ], + ), + # ThreeWindingTransformer holder — per-winding electricals live on the + # three referenced TransformerCircuit rows; pairwise mutual impedances / + # base powers stay here to match POM's ThreeWindingTransformer struct. + "three_winding_transformers" => Tables.Schema( + [ + "id", + "name", + "primary_circuit_id", + "secondary_circuit_id", + "tertiary_circuit_id", + "star_bus", + "parameter_units", + "r_12", + "x_12", + "r_23", + "x_23", + "r_31", + "x_31", + "base_power_12", + "base_power_23", + "base_power_31", + "admittance_units", + "shunt_location", + ], + [ + Int64, + String, + Int64, + Int64, + Int64, + Int64, + Union{String, Nothing}, + Float64, + Float64, + Float64, + Float64, + Float64, + Float64, + Float64, + Float64, + Float64, + Union{String, Nothing}, + Union{String, Nothing}, + ], + ), + "thermal_generators" => Tables.Schema( + [ + "id", + "name", + "prime_mover_type", + "fuel", + "balancing_topology", + "rating", + "base_power", + "active_power_limits", + "reactive_power_limits", + "ramp_limits", + "time_limits", + "must_run", + "available", + "status", + "active_power", + "reactive_power", + "operation_cost", + ], + [ + Int64, + String, + String, + String, + Int64, + Float64, + Float64, + String, # JSON: {"min": ..., "max": ...} + Union{String, Nothing}, # JSON: {"min": ..., "max": ...} + Union{String, Nothing}, # JSON: {"up": ..., "down": ...} + Union{String, Nothing}, # JSON: {"up": ..., "down": ...} + Bool, + Bool, + Bool, + Float64, + Float64, + String, # JSON stored as String + ], + ), + "renewable_generators" => Tables.Schema( + [ + "id", + "name", + "prime_mover_type", + "balancing_topology", + "rating", + "base_power", + "power_factor", + "reactive_power_limits", + "available", + "active_power", + "reactive_power", + "operation_cost", + ], + [ + Int64, + String, + String, + Int64, + Float64, + Float64, + Float64, + Union{String, Nothing}, # JSON: {"min": ..., "max": ...} + Bool, + Float64, + Float64, + Union{String, Nothing}, # JSON stored as String, NULL for RenewableNonDispatch + ], + ), + "hydro_generators" => Tables.Schema( + [ + "id", + "name", + "prime_mover_type", + "balancing_topology", + "rating", + "base_power", + "active_power_limits", + "reactive_power_limits", + "ramp_limits", + "time_limits", + "available", + "active_power", + "reactive_power", + "powerhouse_elevation", + "outflow_limits", + "conversion_factor", + "travel_time", + "operation_cost", + ], + [ + Int64, + String, + String, + Int64, + Float64, + Float64, + String, # JSON: {"min": ..., "max": ...} + Union{String, Nothing}, # JSON: {"min": ..., "max": ...} + Union{String, Nothing}, # JSON: {"up": ..., "down": ...} + Union{String, Nothing}, # JSON: {"up": ..., "down": ...} + Bool, + Float64, + Float64, + Union{Float64, Nothing}, + Union{String, Nothing}, # JSON: {"min": ..., "max": ...} + Union{Float64, Nothing}, + Union{Float64, Nothing}, + String, # JSON stored as String + ], + ), + "storage_units" => Tables.Schema( + [ + "id", + "name", + "prime_mover_type", + "storage_technology_type", + "balancing_topology", + "rating", + "base_power", + "storage_capacity", + "storage_level_limits", + "initial_storage_capacity_level", + "input_active_power_limits", + "output_active_power_limits", + "efficiency", + "reactive_power_limits", + "active_power", + "reactive_power", + "available", + "conversion_factor", + "storage_target", + "cycle_limits", + "operation_cost", + ], + [ + Int64, + String, + String, + String, + Int64, + Float64, + Float64, + Float64, + String, # JSON: {"min": ..., "max": ...} + Float64, + String, # JSON: {"min": ..., "max": ...} + String, # JSON: {"min": ..., "max": ...} + String, # JSON: {"in": ..., "out": ...} + Union{String, Nothing}, # JSON: {"min": ..., "max": ...} + Float64, + Float64, + Bool, + Float64, + Float64, + Int64, + Union{String, Nothing}, # JSON stored as String, nullable + ], + ), + "hydro_reservoirs" => Tables.Schema( + [ + "id", + "name", + "available", + "storage_level_limits", + "initial_level", + "spillage_limits", + "inflow", + "outflow", + "level_targets", + "intake_elevation", + "head_to_volume_factor", + "operation_cost", + "level_data_type", + ], + [ + Int64, + String, + Bool, + String, # JSON + Float64, + Union{String, Nothing}, # JSON, nullable + Float64, + Float64, + Union{Float64, Nothing}, + Float64, + String, # JSON + String, # JSON + String, + ], + ), + "hydro_reservoir_connections" => + Tables.Schema(["source_id", "sink_id"], [Int64, Int64]), + "supply_technologies" => Tables.Schema( + [ + "id", + "name", + "prime_mover_type", + "region", + "power_systems_type", + "lifetime", + "unit_size", + "capacity_limits", + "fuel", + "start_fuel_mmbtu_per_mwh", + "cofire_level_limits", + "cofire_start_limits", + "co2", + "available", + "ramp_limits", + "time_limits", + "outage_factor", + "min_generation_fraction", + "capital_costs", + "operation_costs", + "financial_data", + ], + [ + Int64, + String, + String, + String, + String, + Union{Int64, Nothing}, + Union{Float64, Nothing}, + Union{String, Nothing}, + String, + Union{Float64, Nothing}, + Union{String, Nothing}, + Union{String, Nothing}, + Union{String, Nothing}, + Bool, + Union{String, Nothing}, + Union{String, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + String, + String, + String, + ], + ), + "storage_technologies" => Tables.Schema( + [ + "id", + "name", + "prime_mover_type", + "storage_tech", + "region", + "power_systems_type", + "lifetime", + "unit_size_charge", + "unit_size_discharge", + "unit_size_energy", + "capacity_limits_charge", + "capacity_limits_discharge", + "capacity_limits_energy", + "available", + "duration_limits", + "efficiency", + "min_discharge_fraction", + "losses", + "capital_costs_charge", + "capital_costs_discharge", + "capital_costs_energy", + "operation_costs", + "financial_data", + ], + [ + Int64, + String, + String, + String, + String, + String, + Union{Int64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{String, Nothing}, + Union{String, Nothing}, + Union{String, Nothing}, + Bool, + Union{String, Nothing}, + Union{String, Nothing}, + Union{Float64, Nothing}, + Union{Float64, Nothing}, + Union{String, Nothing}, + String, + String, + String, + String, + ], + ), + "transport_technologies" => Tables.Schema( + [ + "id", + "name", + "power_systems_type", + "available", + "capital_costs", + "financial_data", + "unit_size", + ], + [Int64, String, String, Bool, String, String, Union{Float64, Nothing}], + ), + "demand_technologies" => Tables.Schema( + ["id", "name", "available", "region", "power_systems_type"], + [Int64, String, Bool, String, String], + ), + "attributes" => Tables.Schema( + ["id", "entity_id", "TYPE", "name", "value"], + # Note: json_type is a generated column, not included here + [Int64, Int64, String, String, String], + ), + "supplemental_attributes" => Tables.Schema( + ["id", "TYPE", "value"], + # Note: json_type is a generated column, not included here + [Int64, String, String], + ), + "supplemental_attributes_association" => + Tables.Schema(["attribute_id", "entity_id"], [Int64, Int64]), + "plants" => Tables.Schema( + ["id", "name", "TYPE", "value"], + # Note: json_type is a generated column, not included here + [Int64, String, String, Union{String, Nothing}], + ), + "plant_associations" => + Tables.Schema(["plant_id", "entity_id", "group_index"], [Int64, Int64, Int64]), + "combined_cycle_associations" => Tables.Schema( + ["plant_id", "entity_id", "role", "hrsg_index"], + [Int64, Int64, String, Int64], + ), + "time_series_associations" => Tables.Schema( + [ + "id", + "time_series_uuid", + "time_series_type", + "initial_timestamp", + "resolution", + "horizon", + "interval", + "window_count", + "length", + "name", + "owner_id", + "owner_type", + "owner_category", + "features", + "scaling_factor_multiplier", + "metadata_uuid", + "units", + ], + [ + Int64, + String, + String, + String, + Int64, + Union{Int64, Nothing}, + Union{Int64, Nothing}, + Union{Int64, Nothing}, + Union{Int64, Nothing}, + Union{String, Nothing}, + String, + Int64, + Union{String, Nothing}, + Union{String, Nothing}, + Union{String, Nothing}, + Union{String, Nothing}, + Union{String, Nothing}, + ], + ), + "loads" => Tables.Schema( + ["id", "name", "balancing_topology", "base_power"], + [Int64, String, Int64, Union{Float64, Nothing}], + ), + "static_time_series" => + Tables.Schema(["id", "uuid", "idx", "value"], [Int64, String, Int64, Float64]), +) + +# OpenAPI type names PFFP writes into the DB. Populated into `entity_types` +# by `make_sqlite!`. Mirrors the union of SOM's TYPE_NAMES / SA_TYPE_NAMES +# for the subset PFFP actually emits — the PSIP and plant registries are +# omitted because PFFP never produces those. Two arrays because SOM keys the +# topology flag on whether the type maps to `planning_regions` / +# `balancing_topologies`; here we hard-code that distinction. +const _TOPOLOGY_TYPE_NAMES = [ + "Area", + "LoadZone", + "ACBus", +] + +const _NON_TOPOLOGY_TYPE_NAMES = [ + "Arc", + "AreaInterchange", + "Line", + "TransformerCircuit", + "TwoWindingTransformer", + "TwoTerminalGenericHVDCLine", + "ThreeWindingTransformer", + "PowerLoad", + "StandardLoad", + "FixedAdmittance", + "ThermalStandard", + "RenewableDispatch", + "RenewableNonDispatch", + "HydroDispatch", + "EnergyReservoirStorage", + "ImpedanceCorrectionData", + # Subtypes that share tables with their parents. Registered so + # `entities.entity_type` can record the concrete OpenAPI type. + "DiscreteControlledACBranch", + "TwoTerminalLCCLine", + "TwoTerminalVSCLine", + "FACTSControlDevice", + "SwitchedAdmittance", + "SynchronousCondenser", + "InterruptibleStandardLoad", +] + +""" +Initialize a fresh SQLite database with the Sienna schema: creates all +tables, indexes, and triggers, then seeds the metadata tables +(`entity_types`, `prime_mover_types`, `fuels`, `storage_technology_types`) +with the default enum populations. + +Duplicated from `SiennaOpenAPIModels.jl/src/dbinterface/db_definition.jl:543`. +The only substantive change: `entity_types` is populated from the local +`_TOPOLOGY_TYPE_NAMES` + `_NON_TOPOLOGY_TYPE_NAMES` lists rather than SOM's +PSY-keyed `TYPE_NAMES` / `SA_TYPE_NAMES` / `SA_TYPE_NAMES_PSIP` / +`PLANT_TYPE_NAMES` registries. +""" +function make_sqlite!(db) + for table in SQLITE_CREATE_STR + DBInterface.execute(db, table) + end + for table in SQLITE_TRIGGERS_STR + DBInterface.execute(db, table) + end + + entity_type_stmt = DBInterface.prepare( + db, + "INSERT INTO entity_types (name, is_topology) VALUES (?, ?)", + ) + for type_name in _TOPOLOGY_TYPE_NAMES + DBInterface.execute(entity_type_stmt, (type_name, true)) + end + for type_name in _NON_TOPOLOGY_TYPE_NAMES + DBInterface.execute(entity_type_stmt, (type_name, false)) + end + + # Default prime mover types (derived from PowerSystems.PrimeMovers enums). + pm_stmt = DBInterface.prepare( + db, + "INSERT INTO prime_mover_types (id, name, description) VALUES (?, ?, ?)", + ) + default_prime_movers = [ + (1, "BA", "Battery Energy Storage"), + (2, "BT", "Binary Cycle Turbine"), + (3, "CA", "Compressed Air Energy Storage"), + (4, "CC", "Combined Cycle"), + (5, "CE", "Reciprocating Engine"), + (6, "CP", "Concentrated Solar Power"), + (7, "CS", "Combined Cycle Steam"), + (8, "CT", "Combustion (Gas) Turbine"), + (9, "ES", "Energy Storage"), + (10, "FC", "Fuel Cell"), + (11, "FW", "Flywheel Energy Storage"), + (12, "GT", "Gas Turbine"), + (13, "HA", "Hydro Francis"), + (14, "HB", "Hydro Bulb"), + (15, "HK", "Hydro Kaplan"), + (16, "HY", "Hydro"), + (17, "IC", "Internal Combustion Engine"), + (18, "OT", "Other"), + (19, "PS", "Pumped Storage"), + (20, "PVe", "Photovoltaic"), + (21, "ST", "Steam Turbine"), + (22, "WS", "Wind Offshore"), + (23, "WT", "Wind Onshore"), + ] + for (id, name, desc) in default_prime_movers + DBInterface.execute(pm_stmt, (id, name, desc)) + end + + # Default fuels (derived from PowerSystems.ThermalFuels enums). + fuel_stmt = DBInterface.prepare( + db, + "INSERT INTO fuels (id, name, description) VALUES (?, ?, ?)", + ) + default_fuels = [ + (1, "COAL", "Coal"), + (2, "ANTHRACITE_COAL", "Anthracite Coal"), + (3, "BITUMINOUS_COAL", "Bituminous Coal"), + (4, "LIGNITE_COAL", "Lignite Coal"), + (5, "SUBBITUMINOUS_COAL", "Subbituminous Coal"), + (6, "WASTE_COAL", "Waste Coal"), + (7, "REFINED_COAL", "Refined Coal"), + (8, "SYNTHESIS_GAS_COAL", "Synthesis Gas Coal"), + (9, "DISTILLATE_FUEL_OIL", "Distillate Fuel Oil"), + (10, "JET_FUEL", "Jet Fuel"), + (11, "KEROSENE", "Kerosene"), + (12, "PETROLEUM_COKE", "Petroleum Coke"), + (13, "RESIDUAL_FUEL_OIL", "Residual Fuel Oil"), + (14, "PROPANE", "Propane"), + (15, "SYNTHESIS_GAS_PETROLEUM_COKE", "Synthesis Gas Petroleum Coke"), + (16, "WASTE_OIL", "Waste Oil"), + (17, "BLASTE_FURNACE_GAS", "Blaste Furnace Gas"), + (18, "NATURAL_GAS", "Natural Gas"), + (19, "OTHER_GAS", "Other Gas"), + (20, "NUCLEAR", "Nuclear"), + (21, "AG_BYPRODUCT", "Ag Byproduct"), + (22, "MUNICIPAL_WASTE", "Municipal Waste"), + (23, "OTHER_BIOMASS_SOLIDS", "Other Biomass Solids"), + (24, "WOOD_WASTE_SOLIDS", "Wood Waste Solids"), + (26, "OTHER_BIOMASS_LIQUIDS", "Other Biomass Liquids"), + (27, "SLUDGE_WASTE", "Sludge Waste"), + (28, "BLACK_LIQUOR", "Black Liquor"), + (29, "WOOD_WASTE_LIQUIDS", "Wood Waste Liquids"), + (30, "LANDFILL_GAS", "Landfill Gas"), + (31, "OTHEHR_BIOMASS_GAS", "Other Biomass Gas"), + (32, "GEOTHERMAL", "Geothermal"), + (33, "WASTE_HEAT", "Waste Heat"), + (34, "TIREDERIVED_FUEL", "Tirederived Fuel"), + (35, "OTHER", "Other"), + (36, "WIND", "Wind"), + (37, "SOLAR", "Solar"), + ] + for (id, name, desc) in default_fuels + DBInterface.execute(fuel_stmt, (id, name, desc)) + end + + # Default storage technology types (derived from PowerSystems.StorageTech enums). + st_stmt = DBInterface.prepare( + db, + "INSERT INTO storage_technology_types (id, name, description) VALUES (?, ?, ?)", + ) + default_storage_techs = [ + (1, "PTES", "Pumped Thermal Energy Storage"), + (2, "LIB", "Lithium-Ion Battery"), + (3, "LAB", "Lead Acid Battery"), + (4, "FLWB", "Redox Flow Battery"), + (5, "SIB", "Sodium Ion Battery"), + (6, "ZIB", "Zinc Ion Battery"), + (7, "HGS", "Hydrogen Gas Storage"), + (8, "LAES", "Liquid Air Energy Storage"), + (9, "OTHER_CHEM", "Chemical Storage"), + (10, "OTHER_MECH", "Mechanical Storage"), + (11, "OTHER_THERM", "Thermal Storage"), + ] + for (id, name, desc) in default_storage_techs + DBInterface.execute(st_stmt, (id, name, desc)) + end +end diff --git a/src/dbinterface/schema.sql b/src/dbinterface/schema.sql new file mode 100644 index 0000000..f1be6b6 --- /dev/null +++ b/src/dbinterface/schema.sql @@ -0,0 +1,574 @@ +-- DISCLAIMER +-- The current version of this schema only works for SQLITE >=3.45 +-- When adding new functionality, think about the following: +-- 1. Simplicity and ease of use over complexity, +-- 2. Clear, consice and strict fields but allow for extensability, +-- 3. User friendly over peformance, but consider performance always, +-- WARNING: This script should only be used while testing the schema and should not +-- be applied to existing dataset since it drops all the information it has. +DROP TABLE IF EXISTS thermal_generators; + +DROP TABLE IF EXISTS renewable_generators; + +DROP TABLE IF EXISTS hydro_generators; + +DROP TABLE IF EXISTS storage_units; + +DROP TABLE IF EXISTS prime_mover_types; + +DROP TABLE IF EXISTS balancing_topologies; + +DROP TABLE IF EXISTS supply_technologies; + +DROP TABLE IF EXISTS storage_technology_types; + +DROP TABLE IF EXISTS storage_technologies; + +DROP TABLE IF EXISTS demand_technologies; + +DROP TABLE IF EXISTS transmission_lines; + +DROP TABLE IF EXISTS two_winding_transformers; + +DROP TABLE IF EXISTS three_winding_transformers; + +DROP TABLE IF EXISTS transformer_circuits; + +DROP TABLE IF EXISTS planning_regions; + +DROP TABLE IF EXISTS transmission_interchanges; + +DROP TABLE IF EXISTS entities; + +DROP TABLE IF EXISTS time_series_associations; + +DROP TABLE IF EXISTS attributes; + +DROP TABLE IF EXISTS loads; + +DROP TABLE IF EXISTS static_time_series; + +DROP TABLE IF EXISTS entity_types; + +DROP TABLE IF EXISTS supplemental_attributes; + +DROP TABLE IF EXISTS arcs; + +DROP TABLE IF EXISTS hydro_reservoirs; + +DROP TABLE IF EXISTS hydro_reservoir_connections; + +DROP TABLE IF EXISTS fuels; + +DROP TABLE IF EXISTS supplemental_attributes_association; + +DROP TABLE IF EXISTS transport_technologies; + +PRAGMA foreign_keys = ON; + +-- NOTE: This table should not be interacted directly since it gets populated +-- automatically. +-- Table of certain entities of griddb schema. +CREATE TABLE entities ( + id INTEGER PRIMARY KEY, + entity_table TEXT NOT NULL, + entity_type TEXT NOT NULL, + FOREIGN KEY (entity_type) REFERENCES entity_types (name) +) strict; + +-- Table of possible entity types +CREATE TABLE entity_types ( + name TEXT PRIMARY KEY, + is_topology BOOLEAN NOT NULL DEFAULT FALSE +); + +-- NOTE: Sienna-griddb follows the convention of the EIA prime mover where we +-- have a `prime_mover` and `fuel` to classify generators/storage units. +-- However, users could use any combination of `prime_mover` and `fuel` for +-- their own application. The only constraint is that the uniqueness is enforced +-- by the combination of (prime_mover, fuel) +-- Categories to classify generating units and supply technologies +CREATE TABLE prime_mover_types ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + description TEXT NULL +) strict; + +CREATE TABLE fuels ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + description TEXT NULL +) strict; + +CREATE TABLE storage_technology_types ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + description TEXT NULL +) strict; + +-- Investment regions +CREATE TABLE planning_regions ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + description TEXT NULL +) strict; + +-- Balancing topologies for the system. Could be either buses, or larger +-- aggregated regions. +CREATE TABLE balancing_topologies ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + area INTEGER NULL REFERENCES planning_regions (id) ON DELETE + SET NULL, + description TEXT NULL +) strict; + +-- NOTE: The purpose of this table is to provide links different entities that +-- naturally have a relantionship not model dependent (e.g., transmission lines, +-- transmission interchanges, etc.). +-- Physical connection between entities. +CREATE TABLE arcs ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + from_id INTEGER NOT NULL, + to_id INTEGER NOT NULL, + CHECK (from_id <> to_id), + FOREIGN KEY (from_id) REFERENCES entities (id) ON DELETE CASCADE, + FOREIGN KEY (to_id) REFERENCES entities (id) ON DELETE CASCADE +) strict; + +-- Existing transmission lines +CREATE TABLE transmission_lines ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + arc_id INTEGER, + continuous_rating REAL NULL CHECK (continuous_rating >= 0), + ste_rating REAL NULL CHECK (ste_rating >= 0), + lte_rating REAL NULL CHECK (lte_rating >= 0), + line_length REAL NULL CHECK (line_length >= 0), + FOREIGN KEY (arc_id) REFERENCES arcs (id) ON DELETE CASCADE +) strict; + +-- NOTE: The purpose of this table is to provide physical limits to flows +-- between areas or balancing topologies. In contrast with the transmission +-- lines, this entities are used to enforce given physical limits of certain +-- markets. +-- Transmission interchanges between two balancing topologies or areas +CREATE TABLE transmission_interchanges ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + arc_id INTEGER REFERENCES arcs(id) ON DELETE CASCADE, + max_flow_from REAL NOT NULL, + max_flow_to REAL NOT NULL +) strict; + +-- Per-winding electrical/control state for both TwoWindingTransformer and +-- ThreeWindingTransformer, mirroring POM's TransformerCircuit struct. +-- A TwoWindingTransformer references exactly one row here, and a +-- ThreeWindingTransformer references three (primary/secondary/tertiary). +-- Type-specific fields (control_limits, controlled_quantity_limits) fall +-- through to the generic `attributes` table as JSON. +CREATE TABLE transformer_circuits ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + available INTEGER NOT NULL, + arc_id INTEGER NOT NULL REFERENCES arcs (id) ON DELETE CASCADE, + tap REAL, + alpha REAL, + parameter_units TEXT, + r REAL NOT NULL, + x REAL NOT NULL, + control_objective TEXT, + regulated_bus_number INTEGER, + number_of_tap_positions INTEGER, + rating REAL, + rating_b REAL, + rating_c REAL, + active_power_flow REAL, + reactive_power_flow REAL, + base_power REAL, + base_voltage_primary REAL, + base_voltage_secondary REAL +) strict; + +-- TwoWindingTransformer holder rows. All electrical state lives on the +-- referenced `transformer_circuits` row -- the holder just carries name, +-- shunt, and admittance metadata (matches POM's TwoWindingTransformer). +CREATE TABLE two_winding_transformers ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + circuit_id INTEGER NOT NULL REFERENCES transformer_circuits (id) ON DELETE CASCADE, + admittance_units TEXT, + shunt_location TEXT +) strict; + +-- ThreeWindingTransformer holder rows. Per-winding electricals live on the +-- three referenced `transformer_circuits` rows -- the pairwise mutual +-- impedances / base powers stay here to match POM's ThreeWindingTransformer. +CREATE TABLE three_winding_transformers ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL, + primary_circuit_id INTEGER NOT NULL REFERENCES transformer_circuits (id) ON DELETE CASCADE, + secondary_circuit_id INTEGER NOT NULL REFERENCES transformer_circuits (id) ON DELETE CASCADE, + tertiary_circuit_id INTEGER NOT NULL REFERENCES transformer_circuits (id) ON DELETE CASCADE, + star_bus INTEGER NOT NULL REFERENCES entities (id) ON DELETE CASCADE, + parameter_units TEXT, + r_12 REAL NOT NULL, + x_12 REAL NOT NULL, + r_23 REAL NOT NULL, + x_23 REAL NOT NULL, + r_31 REAL NOT NULL, + x_31 REAL NOT NULL, + base_power_12 REAL NOT NULL, + base_power_23 REAL NOT NULL, + base_power_31 REAL NOT NULL, + admittance_units TEXT, + shunt_location TEXT +) strict; + +-- NOTE: The purpose of these tables is to capture data of **existing units only**. +-- Table of thermal generation units (ThermalStandard, ThermalMultiStart) +CREATE TABLE thermal_generators ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + prime_mover_type TEXT NOT NULL REFERENCES prime_mover_types(name), + fuel TEXT NOT NULL DEFAULT 'OTHER' REFERENCES fuels(name), + balancing_topology INTEGER NOT NULL REFERENCES balancing_topologies (id) ON DELETE CASCADE, + rating REAL NOT NULL CHECK (rating >= 0), + base_power REAL NOT NULL CHECK (base_power > 0), + -- Power limits (JSON: {"min": ..., "max": ...}): + active_power_limits JSON NOT NULL, + reactive_power_limits JSON NULL, + -- Ramp limits (JSON: {"up": ..., "down": ...}, MW/min): + ramp_limits JSON NULL, + -- Time limits (JSON: {"up": ..., "down": ...}, hours): + time_limits JSON NULL, + -- Operational flags: + must_run BOOLEAN NOT NULL DEFAULT FALSE, + available BOOLEAN NOT NULL DEFAULT TRUE, + "status" BOOLEAN NOT NULL DEFAULT FALSE, + -- Initial setpoints: + active_power REAL NOT NULL DEFAULT 0.0, + reactive_power REAL NOT NULL DEFAULT 0.0, + -- Cost (complex structure, stored as JSON): + operation_cost JSON NOT NULL DEFAULT '{"cost_type": "THERMAL", "fixed": 0, "shut_down": 0, "start_up": 0, "variable": {"variable_cost_type": "COST", "power_units": "NATURAL_UNITS", "value_curve": {"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}, "vom_cost": {"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}}}' +); + +-- Table of renewable generation units (RenewableDispatch, RenewableNonDispatch) +CREATE TABLE renewable_generators ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + prime_mover_type TEXT NOT NULL REFERENCES prime_mover_types(name), + balancing_topology INTEGER NOT NULL REFERENCES balancing_topologies (id) ON DELETE CASCADE, + rating REAL NOT NULL CHECK (rating >= 0), + base_power REAL NOT NULL CHECK (base_power > 0), + -- Renewable-specific: + power_factor REAL NOT NULL DEFAULT 1.0 CHECK ( + power_factor > 0 + AND power_factor <= 1.0 + ), + -- Power limits (JSON: {"min": ..., "max": ...}): + reactive_power_limits JSON NULL, + -- Operational flags: + available BOOLEAN NOT NULL DEFAULT TRUE, + -- Initial setpoints: + active_power REAL NOT NULL DEFAULT 0.0, + reactive_power REAL NOT NULL DEFAULT 0.0, + -- Cost (NULL for RenewableNonDispatch): + operation_cost JSON NULL DEFAULT '{"cost_type":"RENEWABLE","fixed":0,"variable":{"variable_cost_type":"COST","power_units":"NATURAL_UNITS","value_curve":{"curve_type":"INPUT_OUTPUT","function_data":{"function_type":"LINEAR","proportional_term":0,"constant_term":0}},"vom_cost":{"curve_type":"INPUT_OUTPUT","function_data":{"function_type":"LINEAR","proportional_term":0,"constant_term":0}}},"curtailment_cost":{"variable_cost_type":"COST","power_units":"NATURAL_UNITS","value_curve":{"curve_type":"INPUT_OUTPUT","function_data":{"function_type":"LINEAR","proportional_term":0,"constant_term":0}},"vom_cost":{"curve_type":"INPUT_OUTPUT","function_data":{"function_type":"LINEAR","proportional_term":0,"constant_term":0}}}}' +); + +-- Table of hydro generation units (HydroDispatch, HydroTurbine, HydroPumpTurbine) +CREATE TABLE hydro_generators ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + prime_mover_type TEXT NOT NULL DEFAULT 'HY' REFERENCES prime_mover_types(name), + balancing_topology INTEGER NOT NULL REFERENCES balancing_topologies (id) ON DELETE CASCADE, + rating REAL NOT NULL CHECK (rating >= 0), + base_power REAL NOT NULL CHECK (base_power > 0), + -- Power limits (JSON: {"min": ..., "max": ...}): + active_power_limits JSON NOT NULL, + reactive_power_limits JSON NULL, + -- Ramp limits (JSON: {"up": ..., "down": ...}, MW/min): + ramp_limits JSON NULL, + -- Time limits (JSON: {"up": ..., "down": ...}, hours): + time_limits JSON NULL, + -- Operational flags: + available BOOLEAN NOT NULL DEFAULT TRUE, + -- Initial setpoints: + active_power REAL NOT NULL DEFAULT 0.0, + reactive_power REAL NOT NULL DEFAULT 0.0, + -- HydroTurbine/HydroPumpTurbine fields (nullable for HydroDispatch): + powerhouse_elevation REAL NULL DEFAULT 0.0 CHECK (powerhouse_elevation >= 0), + -- Outflow limits (JSON: {"min": ..., "max": ...}): + outflow_limits JSON NULL, + conversion_factor REAL NULL DEFAULT 1.0 CHECK (conversion_factor > 0), + travel_time REAL NULL CHECK (travel_time >= 0), + -- Cost: + operation_cost JSON NOT NULL DEFAULT '{"cost_type": "HYDRO_GEN", "fixed": 0.0, "variable": {"variable_cost_type": "COST", "power_units": "NATURAL_UNITS", "value_curve": {"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}, "vom_cost": {"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}}}' -- Note: efficiency (varies by type), turbine_type, and HydroPumpTurbine-specific + -- fields (active_power_limits_pump, etc.) are stored in the attributes table +); + +-- NOTE: The purpose of this table is to capture data of **existing storage units only**. +-- Table of energy storage units (including PHES or other kinds), +CREATE TABLE storage_units ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + prime_mover_type TEXT NOT NULL REFERENCES prime_mover_types(name), + storage_technology_type TEXT NOT NULL REFERENCES storage_technology_types(name), + balancing_topology INTEGER NOT NULL REFERENCES balancing_topologies (id) ON DELETE CASCADE, + rating REAL NOT NULL CHECK (rating >= 0), + base_power REAL NOT NULL CHECK (base_power > 0), + -- Storage capacity and limits (JSON: {"min": ..., "max": ...}): + storage_capacity REAL NOT NULL CHECK (storage_capacity >= 0), + storage_level_limits JSON NOT NULL, + initial_storage_capacity_level REAL NOT NULL CHECK (initial_storage_capacity_level >= 0), + -- Power limits (JSON: {"min": ..., "max": ...}, input = charging, output = discharging): + input_active_power_limits JSON NOT NULL, + output_active_power_limits JSON NOT NULL, + -- Efficiency (JSON: {"in": ..., "out": ...}): + efficiency JSON NOT NULL, + -- Reactive power (JSON: {"min": ..., "max": ...}): + reactive_power_limits JSON NULL, + -- Initial setpoints: + active_power REAL NOT NULL DEFAULT 0.0, + reactive_power REAL NOT NULL DEFAULT 0.0, + -- Status: + available BOOLEAN NOT NULL DEFAULT TRUE, + -- Storage-specific with defaults: + conversion_factor REAL NOT NULL DEFAULT 1.0 CHECK (conversion_factor > 0), + storage_target REAL NOT NULL DEFAULT 0.0, + cycle_limits INTEGER NOT NULL DEFAULT 10000 CHECK (cycle_limits > 0), + -- Cost: + operation_cost JSON NULL +); + +-- Topological hydro reservoirs +CREATE TABLE hydro_reservoirs ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + available BOOLEAN NOT NULL DEFAULT TRUE, + -- Storage level limits (JSON: {"min": ..., "max": ...}): + storage_level_limits JSON NOT NULL, + initial_level REAL NOT NULL, + -- Spillage limits (JSON: {"min": ..., "max": ...}, nullable): + spillage_limits JSON NULL, + inflow REAL NOT NULL DEFAULT 0.0, + outflow REAL NOT NULL DEFAULT 0.0, + level_targets REAL NULL, + intake_elevation REAL NOT NULL DEFAULT 0.0, + -- Head to volume relationship (JSON ValueCurve): + head_to_volume_factor JSON NOT NULL, + -- Cost (HydroReservoirCost): + operation_cost JSON NOT NULL DEFAULT '{"cost_type": "HYDRO_RES", "level_shortage_cost": 0.0, "level_surplus_cost": 0.0, "spillage_cost": 0.0}', + level_data_type TEXT NOT NULL DEFAULT 'USABLE_VOLUME' +); + +CREATE TABLE hydro_reservoir_connections ( + source_id INTEGER NOT NULL REFERENCES entities(id) ON DELETE CASCADE, + sink_id INTEGER NOT NULL REFERENCES entities(id) ON DELETE CASCADE, + CHECK (source_id <> sink_id), + PRIMARY KEY (source_id, sink_id) +) strict; +-- investment for expansion problems. +-- Investment technology options for expansion problems +CREATE TABLE supply_technologies ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + prime_mover_type TEXT NOT NULL REFERENCES prime_mover_types(name), + region JSON NOT NULL, + power_systems_type TEXT NOT NULL, + lifetime INTEGER NULL, + unit_size REAL NULL, + -- Capacity limits (JSON: {"min": ..., "max": ...}, MW): + capacity_limits JSON NULL, + -- Fuel information: + fuel TEXT NOT NULL DEFAULT '["OTHER"]', + start_fuel_mmbtu_per_mwh REAL NULL, + -- Fuel cofire limits (JSON: {"fuel1": {"min": ..., "max": ...}, "fuel2": {"min": ..., "max": ...}}): + cofire_level_limits JSON NULL, + -- Fuel cofire start limits (JSON: {"fuel1": ..., "fuel2": ...}): + cofire_start_limits JSON NULL, + -- CO2 emissions (JSON: {"fuel1": ..., "fuel2": ...}, tons per MMBTU): + co2 JSON NULL, + -- Operational information: + available BOOLEAN NOT NULL DEFAULT TRUE, + -- Ramp limits (JSON: {"up": ..., "down": ...}, MW/min): + ramp_limits JSON NULL, + -- Time limits (JSON: {"up": ..., "down": ...}, hours): + time_limits JSON NULL, + outage_factor REAL NULL, + min_generation_fraction REAL NULL, + -- Financial data: + -- Capital cost (complex structure, stored as JSON): + capital_costs JSON NOT NULL DEFAULT '{"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}', + -- Cost (complex structure, stored as JSON): + operation_costs JSON NOT NULL DEFAULT '{"cost_type": "THERMAL", "fixed": 0, "shut_down": 0, "start_up": 0, "variable": {"variable_cost_type": "COST", "power_units": "NATURAL_UNITS", "value_curve": {"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}, "vom_cost": {"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}}}', + -- Other financial parameters (complex structure, stored as JSON): + financial_data JSON NOT NULL +); + +CREATE TABLE storage_technologies ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + prime_mover_type TEXT NOT NULL REFERENCES prime_mover_types(name), + storage_tech TEXT NOT NULL DEFAULT '["OTHER"]', + region JSON NOT NULL, + power_systems_type TEXT NOT NULL, + lifetime INTEGER NULL, + unit_size_charge REAL NULL, + unit_size_discharge REAL NULL, + unit_size_energy REAL NULL, + -- Capacity limits (JSON: {"min": ..., "max": ...}, MW): + capacity_limits_charge JSON NULL, + capacity_limits_discharge JSON NULL, + capacity_limits_energy JSON NULL, + -- Operational information: + available BOOLEAN NOT NULL DEFAULT TRUE, + -- Duration limits (JSON: {"min": ..., "max": ...}, hours): + duration_limits JSON NULL, + -- Efficiency (JSON: {"in": ..., "out": ...}, fraction): + efficiency JSON NULL, + min_discharge_fraction REAL NULL, + losses REAL NULL, + -- Financial data: + -- Capital cost (complex structure, stored as JSON): + capital_costs_charge JSON NULL, + capital_costs_discharge JSON NOT NULL DEFAULT '{"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}', + capital_costs_energy JSON NOT NULL DEFAULT '{"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}', + -- Cost (complex structure, stored as JSON): + operation_costs JSON NOT NULL DEFAULT '{"cost_type": "THERMAL", "fixed": 0, "shut_down": 0, "start_up": 0, "variable": {"variable_cost_type": "COST", "power_units": "NATURAL_UNITS", "value_curve": {"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}, "vom_cost": {"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}}}', + -- Other financial parameters (complex structure, stored as JSON): + financial_data JSON NOT NULL +); + +CREATE TABLE transport_technologies ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + power_systems_type TEXT NOT NULL, + available BOOLEAN NOT NULL DEFAULT TRUE, + capital_costs JSON NOT NULL DEFAULT '{"curve_type": "INPUT_OUTPUT", "function_data": {"function_type": "LINEAR", "proportional_term": 0, "constant_term": 0}}', + financial_data JSON NOT NULL, + unit_size REAL NULL +); + +CREATE TABLE demand_technologies ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + available BOOLEAN NOT NULL DEFAULT TRUE, + region TEXT NOT NULL, + power_systems_type TEXT NOT NULL +); + +-- NOTE: Attributes are additional parameters that can be linked to entities. +-- The main purpose of this is when there is an important field that is not +-- capture on the entity table that should exist on the model. Example of this +-- fields are variable or fixed operation and maintenance cost or any other +-- field that its representation is hard to fit into a `integer`, `real` or +-- `text`. It must not be used for operational details since most of the should +-- be included in the `operational_data` table. +CREATE TABLE attributes ( + id INTEGER PRIMARY KEY, + entity_id INTEGER NOT NULL, + TYPE TEXT NOT NULL, + name TEXT NOT NULL, + value JSON NOT NULL, + json_type TEXT generated always AS (json_type(value)) virtual, + FOREIGN KEY (entity_id) REFERENCES entities (id) ON DELETE CASCADE, + UNIQUE(entity_id, name) +); + +-- NOTE: Supplemental are optional parameters that can be linked to entities. +-- The main purpose of this is to provide a way to save relevant information +-- but that could or could not be used for modeling. not `text`. Examples of +-- this field are geolocation (e.g., lat, long), outages, etc.) +CREATE TABLE supplemental_attributes ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + TYPE TEXT NOT NULL, + value JSON NOT NULL, + json_type TEXT generated always AS (json_type (value)) virtual +); + +CREATE TABLE supplemental_attributes_association ( + attribute_id INTEGER NOT NULL, + entity_id INTEGER NOT NULL, + FOREIGN KEY (entity_id) REFERENCES entities (id) ON DELETE CASCADE, + FOREIGN KEY (attribute_id) REFERENCES supplemental_attributes (id) ON DELETE CASCADE, + PRIMARY KEY (attribute_id, entity_id) +) strict; + +CREATE TABLE plants ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + TYPE TEXT NOT NULL, + value JSON, + json_type TEXT generated always AS (json_type (value)) virtual +); + +CREATE TABLE plant_associations ( + plant_id INTEGER NOT NULL, + entity_id INTEGER NOT NULL, + group_index INTEGER NOT NULL, + FOREIGN KEY (plant_id) REFERENCES plants (id) ON DELETE CASCADE, + FOREIGN KEY (entity_id) REFERENCES entities (id) ON DELETE CASCADE, + PRIMARY KEY (plant_id, entity_id) +) strict; + +-- CombinedCycleBlock CT/CA <-> HRSG associations are n-to-m: a CT or CA can +-- feed multiple HRSGs and an HRSG can have multiple CTs/CAs. Kept in its own +-- table so (plant, entity) is not unique. +CREATE TABLE combined_cycle_associations ( + plant_id INTEGER NOT NULL, + entity_id INTEGER NOT NULL, + role TEXT NOT NULL CHECK (role IN ('CT', 'CA')), + hrsg_index INTEGER NOT NULL, + FOREIGN KEY (plant_id) REFERENCES plants (id) ON DELETE CASCADE, + FOREIGN KEY (entity_id) REFERENCES entities (id) ON DELETE CASCADE, + PRIMARY KEY (plant_id, entity_id, hrsg_index) +) strict; + +CREATE TABLE time_series_associations( + id INTEGER PRIMARY KEY, + time_series_uuid TEXT NOT NULL, + time_series_type TEXT NOT NULL, + initial_timestamp TEXT NOT NULL, + resolution TEXT NOT NULL, + horizon TEXT, + "interval" TEXT, + window_count INTEGER, + length INTEGER, + name TEXT NOT NULL, + owner_id INTEGER NOT NULL REFERENCES entities(id) ON DELETE CASCADE, + owner_type TEXT NOT NULL, + owner_category TEXT NOT NULL, + features TEXT NOT NULL, + scaling_factor_multiplier TEXT NULL, + metadata_uuid TEXT NOT NULL, + units TEXT NULL +); +CREATE UNIQUE INDEX uq_time_series_assoc_owner_type_name_res_feat ON time_series_associations ( + owner_id, + time_series_type, + name, + resolution, + features +); +CREATE INDEX idx_time_series_assoc_uuid ON time_series_associations (time_series_uuid); + + +CREATE TABLE loads ( + id INTEGER PRIMARY KEY REFERENCES entities (id) ON DELETE CASCADE, + name TEXT NOT NULL UNIQUE, + balancing_topology INTEGER NOT NULL, + base_power REAL, + FOREIGN KEY(balancing_topology) REFERENCES balancing_topologies (id) ON DELETE CASCADE +); + +CREATE TABLE static_time_series ( + id INTEGER PRIMARY KEY, + uuid TEXT NOT NULL, + idx INTEGER NOT NULL, + value REAL NOT NULL +) strict; + +CREATE INDEX idx_static_time_series_uuid_idx ON static_time_series (uuid, idx); +CREATE INDEX idx_arcs_from ON arcs (from_id); +CREATE INDEX idx_arcs_to ON arcs (to_id); diff --git a/src/dbinterface/triggers.sql b/src/dbinterface/triggers.sql new file mode 100644 index 0000000..dd72c0f --- /dev/null +++ b/src/dbinterface/triggers.sql @@ -0,0 +1,514 @@ +CREATE TRIGGER IF NOT EXISTS check_planning_regions_entity_exists BEFORE +INSERT ON planning_regions + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'planning_regions' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table planning_regions before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_balancing_topologies_entity_exists BEFORE +INSERT ON balancing_topologies + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'balancing_topologies' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table balancing_topologies before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_arcs_entity_exists BEFORE +INSERT ON arcs + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'arcs' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table arcs before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_transmission_lines_entity_exists BEFORE +INSERT ON transmission_lines + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'transmission_lines' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table transmission_lines before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_three_winding_transformers_entity_exists BEFORE +INSERT ON three_winding_transformers + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'three_winding_transformers' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table three_winding_transformers before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_transmission_interchanges_entity_exists BEFORE +INSERT ON transmission_interchanges + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'transmission_interchanges' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table transmission_interchanges before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_thermal_generators_entity_exists BEFORE +INSERT ON thermal_generators + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'thermal_generators' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table thermal_generators before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_renewable_generators_entity_exists BEFORE +INSERT ON renewable_generators + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'renewable_generators' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table renewable_generators before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_hydro_generators_entity_exists BEFORE +INSERT ON hydro_generators + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'hydro_generators' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table hydro_generators before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_storage_units_entity_exists BEFORE +INSERT ON storage_units + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'storage_units' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table storage_units before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_hydro_reservoirs_entity_exists BEFORE +INSERT ON hydro_reservoirs + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'hydro_reservoirs' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table hydro_reservoirs before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_supply_technologies_entity_exists BEFORE +INSERT ON supply_technologies + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'supply_technologies' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table supply_technologies before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_transport_technologies_entity_exists BEFORE +INSERT ON transport_technologies + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'transport_technologies' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table transport_technologies before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_supplemental_attributes_entity_exists BEFORE +INSERT ON supplemental_attributes + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'supplemental_attributes' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table supplemental_attributes before insertion' + ); +END; + +CREATE TRIGGER IF NOT EXISTS check_loads_entity_exists BEFORE +INSERT ON loads + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.id + AND entity_table = 'loads' + ) BEGIN +SELECT RAISE( + ABORT, + 'Entity ID must exist in entities table with entity_table loads before insertion' + ); +END; + + +-- Business Logic Validation Triggers +CREATE TRIGGER enforce_arc_entity_types_insert +AFTER +INSERT ON arcs BEGIN +SELECT CASE + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.from_id + ) THEN RAISE(ABORT, 'from_id entity does not exist') + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.to_id + ) THEN RAISE(ABORT, 'to_id entity does not exist') + WHEN ( + SELECT et.is_topology + FROM entities e + JOIN entity_types et ON e.entity_type = et.name + WHERE e.id = NEW.from_id + ) = 0 THEN RAISE( + ABORT, + 'Invalid from_id entity type: must be a topology type (entity_types.is_topology = 1)' + ) + WHEN ( + SELECT et.is_topology + FROM entities e + JOIN entity_types et ON e.entity_type = et.name + WHERE e.id = NEW.to_id + ) = 0 THEN RAISE( + ABORT, + 'Invalid to_id entity type: must be a topology type (entity_types.is_topology = 1)' + ) + END; +END; + +CREATE TRIGGER enforce_arc_entity_types_update +AFTER UPDATE OF from_id, to_id ON arcs BEGIN +SELECT CASE + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.from_id + ) THEN RAISE(ABORT, 'from_id entity does not exist') + WHEN NOT EXISTS ( + SELECT 1 + FROM entities + WHERE id = NEW.to_id + ) THEN RAISE(ABORT, 'to_id entity does not exist') + WHEN ( + SELECT et.is_topology + FROM entities e + JOIN entity_types et ON e.entity_type = et.name + WHERE e.id = NEW.from_id + ) = 0 THEN RAISE( + ABORT, + 'Invalid from_id entity type: must be a topology type (entity_types.is_topology = 1)' + ) + WHEN ( + SELECT et.is_topology + FROM entities e + JOIN entity_types et ON e.entity_type = et.name + WHERE e.id = NEW.to_id + ) = 0 THEN RAISE( + ABORT, + 'Invalid to_id entity type: must be a topology type (entity_types.is_topology = 1)' + ) + END; +END; + +-- Enforce that a turbine can have at most 1 upstream reservoir +-- (i.e., at most 1 row where sink is a turbine and source is a reservoir) +CREATE TRIGGER IF NOT EXISTS enforce_turbine_single_upstream_reservoir BEFORE +INSERT ON hydro_reservoir_connections + WHEN ( + -- Check if sink is a turbine (hydro_generators or storage_units) + SELECT entity_table + FROM entities + WHERE id = NEW.sink_id + ) IN ('hydro_generators', 'storage_units') + AND ( + -- Check if source is a reservoir + SELECT entity_table + FROM entities + WHERE id = NEW.source_id + ) = 'hydro_reservoirs' BEGIN +SELECT CASE + WHEN EXISTS ( + SELECT 1 + FROM hydro_reservoir_connections hrc + JOIN entities e_source ON hrc.source_id = e_source.id + WHERE hrc.sink_id = NEW.sink_id + AND e_source.entity_table = 'hydro_reservoirs' + ) THEN RAISE( + ABORT, + 'Turbine already has an upstream reservoir. Each turbine can have at most 1 upstream reservoir.' + ) + END; +END; + +CREATE TRIGGER IF NOT EXISTS enforce_turbine_single_upstream_reservoir_update +BEFORE UPDATE OF source_id, sink_id ON hydro_reservoir_connections + WHEN ( + SELECT entity_table + FROM entities + WHERE id = NEW.sink_id + ) IN ('hydro_generators', 'storage_units') + AND ( + SELECT entity_table + FROM entities + WHERE id = NEW.source_id + ) = 'hydro_reservoirs' BEGIN +SELECT CASE + WHEN EXISTS ( + SELECT 1 + FROM hydro_reservoir_connections hrc + JOIN entities e_source ON hrc.source_id = e_source.id + WHERE hrc.sink_id = NEW.sink_id + AND e_source.entity_table = 'hydro_reservoirs' + AND hrc.rowid != OLD.rowid + ) THEN RAISE( + ABORT, + 'Turbine already has an upstream reservoir. Each turbine can have at most 1 upstream reservoir.' + ) + END; +END; + +-- Enforce that a turbine can have at most 1 downstream reservoir +-- (i.e., at most 1 row where source is a turbine and sink is a reservoir) +CREATE TRIGGER IF NOT EXISTS enforce_turbine_single_downstream_reservoir BEFORE +INSERT ON hydro_reservoir_connections + WHEN ( + -- Check if source is a turbine (hydro_generators or storage_units) + SELECT entity_table + FROM entities + WHERE id = NEW.source_id + ) IN ('hydro_generators', 'storage_units') + AND ( + -- Check if sink is a reservoir + SELECT entity_table + FROM entities + WHERE id = NEW.sink_id + ) = 'hydro_reservoirs' BEGIN +SELECT CASE + WHEN EXISTS ( + SELECT 1 + FROM hydro_reservoir_connections hrc + JOIN entities e_sink ON hrc.sink_id = e_sink.id + WHERE hrc.source_id = NEW.source_id + AND e_sink.entity_table = 'hydro_reservoirs' + ) THEN RAISE( + ABORT, + 'Turbine already has a downstream reservoir. Each turbine can have at most 1 downstream reservoir.' + ) + END; +END; + +CREATE TRIGGER IF NOT EXISTS enforce_turbine_single_downstream_reservoir_update +BEFORE UPDATE OF source_id, sink_id ON hydro_reservoir_connections + WHEN ( + SELECT entity_table + FROM entities + WHERE id = NEW.source_id + ) IN ('hydro_generators', 'storage_units') + AND ( + SELECT entity_table + FROM entities + WHERE id = NEW.sink_id + ) = 'hydro_reservoirs' BEGIN +SELECT CASE + WHEN EXISTS ( + SELECT 1 + FROM hydro_reservoir_connections hrc + JOIN entities e_sink ON hrc.sink_id = e_sink.id + WHERE hrc.source_id = NEW.source_id + AND e_sink.entity_table = 'hydro_reservoirs' + AND hrc.rowid != OLD.rowid + ) THEN RAISE( + ABORT, + 'Turbine already has a downstream reservoir. Each turbine can have at most 1 downstream reservoir.' + ) + END; +END; + +-- Reverse cascade triggers: delete from entities when child table row is deleted +CREATE TRIGGER IF NOT EXISTS delete_planning_regions_entity +AFTER DELETE ON planning_regions +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_balancing_topologies_entity +AFTER DELETE ON balancing_topologies +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_arcs_entity +AFTER DELETE ON arcs +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_transmission_lines_entity +AFTER DELETE ON transmission_lines +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_transmission_interchanges_entity +AFTER DELETE ON transmission_interchanges +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_three_winding_transformers_entity +AFTER DELETE ON three_winding_transformers +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_thermal_generators_entity +AFTER DELETE ON thermal_generators +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_renewable_generators_entity +AFTER DELETE ON renewable_generators +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_hydro_generators_entity +AFTER DELETE ON hydro_generators +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_storage_units_entity +AFTER DELETE ON storage_units +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_hydro_reservoirs_entity +AFTER DELETE ON hydro_reservoirs +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_supply_technologies_entity +AFTER DELETE ON supply_technologies +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_transport_technologies_entity +AFTER DELETE ON transport_technologies +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_storage_technologies_entity +AFTER DELETE ON storage_technologies +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_demand_technologies_entity +AFTER DELETE ON demand_technologies +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_supplemental_attributes_entity +AFTER DELETE ON supplemental_attributes +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; + +CREATE TRIGGER IF NOT EXISTS delete_loads_entity +AFTER DELETE ON loads +FOR EACH ROW +BEGIN + DELETE FROM entities WHERE id = OLD.id; +END; diff --git a/src/definitions.jl b/src/definitions.jl index 9458fb5..65e6a38 100644 --- a/src/definitions.jl +++ b/src/definitions.jl @@ -24,7 +24,349 @@ const WINDING_NAMES = Dict( 3 => "tertiary", ) +const WINDING_NAMES_PARSING = [ + "PRIMARY_WINDING", + "SECONDARY_WINDING", + "TERTIARY_WINDING", +] + const TRANSFORMER3W_PARAMETER_NAMES = [ "COD", "CONT", "NOMV", "WINDV", "RMA", "RMI", "NTP", "VMA", "VMI", "RATA", "RATB", "RATC", ] + +################################################# + +const _PM_BUS_TYPE_ENUM = Dict( + 1 => "PQ", + 2 => "PV", + 3 => "REF", + 4 => "ISOLATED", +) + +const _LOAD_CONFORMITY_ENUM = Dict( + 0 => "NON_CONFORMING", + 1 => "CONFORMING", + 2 => "UNDEFINED", +) +_loadconformity_string(val::Integer) = get(_LOAD_CONFORMITY_ENUM, Int(val), "UNDEFINED") + +# Render a `ComplexF64` as the OpenAPI `ComplexNumber` dict shape +# (`{"real": ..., "imag": ...}`). Mirrors +# `sienna_to_json/common.jl:get_complex_number`. +_complex_to_dict(z::Complex) = + Dict{String, Any}("real" => real(z), "imag" => imag(z)) + +const _GENERATOR_MAPPING_FILE = joinpath(@__DIR__, "generator_mapping_pm.yaml") + +const _PRIME_MOVER_CANONICAL = ( + "BA", "BT", "CA", "CC", "CE", "CP", "CS", "CT", "ES", "FC", "FW", "GT", + "HA", "HB", "HK", "HY", "IC", "PS", "OT", "ST", "PVe", "WT", "WS", +) + +const _PRIME_MOVER_ALIASES = Dict( + "w2" => "WT", + "wind" => "WT", + "pv" => "PVe", + "solar" => "PVe", + "rtpv" => "PVe", + "nb" => "ST", + "steam" => "ST", + "hydro" => "HY", + "ror" => "HY", + "pump" => "PS", + "pumped_hydro" => "PS", + "nuclear" => "ST", + "sync_cond" => "OT", + "csp" => "CP", + "un" => "OT", + "storage" => "BA", + "ice" => "IC", +) + +const _STRING2PRIMEMOVER = let d = Dict{String, String}() + for canonical in _PRIME_MOVER_CANONICAL + d[lowercase(canonical)] = canonical + end + merge!(d, _PRIME_MOVER_ALIASES) + d +end + +const _FUEL_CANONICAL = ( + "ANTHRACITE_COAL", "BITUMINOUS_COAL", "LIGNITE_COAL", "SUBBITUMINOUS_COAL", + "WASTE_COAL", "REFINED_COAL", "SYNTHESIS_GAS_COAL", "DISTILLATE_FUEL_OIL", + "JET_FUEL", "KEROSENE", "PETROLEUM_COKE", "RESIDUAL_FUEL_OIL", "PROPANE", + "SYNTHESIS_GAS_PETROLEUM_COKE", "WASTE_OIL", "BLASTE_FURNACE_GAS", + "NATURAL_GAS", "OTHER_GAS", "AG_BYPRODUCT", "MUNICIPAL_WASTE", + "OTHER_BIOMASS_SOLIDS", "WOOD_WASTE_SOLIDS", "OTHER_BIOMASS_LIQUIDS", + "SLUDGE_WASTE", "BLACK_LIQUOR", "WOOD_WASTE_LIQUIDS", "LANDFILL_GAS", + "OTHEHR_BIOMASS_GAS", "NUCLEAR", "WASTE_HEAT", "TIREDERIVED_FUEL", + "COAL", "GEOTHERMAL", "OTHER", +) + +const _FUEL_ALIASES = Dict( + "ng" => "NATURAL_GAS", + "nuc" => "NUCLEAR", + "gas" => "NATURAL_GAS", + "oil" => "DISTILLATE_FUEL_OIL", + "dfo" => "DISTILLATE_FUEL_OIL", + "sync_cond" => "OTHER", +) + +const _STRING2FUEL = let d = Dict{String, String}() + for canonical in _FUEL_CANONICAL + d[lowercase(canonical)] = canonical + end + merge!(d, _FUEL_ALIASES) + d +end + +const _SHIFT_TO_GROUP_MAP = Dict{Float64, String}( + 0.0 => "GROUP_0", + -30.0 => "GROUP_1", + -150.0 => "GROUP_5", + 180.0 => "GROUP_6", + 150.0 => "GROUP_7", + 30.0 => "GROUP_11", +) + +const _CONTROL_OBJECTIVE_MAP = Dict{Int, String}( + 0 => "FIXED", + 1 => "REACTIVE_POWER_FLOW", + -1 => "REACTIVE_POWER_FLOW_DISABLED", + 2 => "VOLTAGE", + -2 => "VOLTAGE_DISABLED", + 3 => "ACTIVE_POWER_FLOW", + -3 => "ACTIVE_POWER_FLOW_DISABLED", + 4 => "CONTROL_OF_DC_LINE", + -4 => "CONTROL_OF_DC_LINE_DISABLED", + 5 => "ASYMETRIC_ACTIVE_POWER_FLOW", + -5 => "ASYMETRIC_ACTIVE_POWER_FLOW_DISABLED", + -99 => "UNDEFINED", +) + +_normalize_control_objective(val::Integer) = + get(_CONTROL_OBJECTIVE_MAP, Int(val), "UNDEFINED") + +const _DISCRETE_BRANCH_TYPE_MAP = Dict{Int, String}( + 0 => "SWITCH", + 1 => "BREAKER", + 2 => "OTHER", +) + +const _BRANCH_STATUS_MAP = Dict{Int, String}( + 0 => "OPEN", + 1 => "CLOSED", +) + +# FACTS control_mode: the PSS/E POM v33 manual and PSY's `FACTSOperationModes` +# enum both define exactly three modes — OOS=0, NML=1, BYP=2. The OpenAPI +# `FACTSControlDevice.control_mode` enum covers the same set. +const _FACTS_CONTROL_MODE_MAP = Dict{Int, String}( + 0 => "OOS", + 1 => "NML", + 2 => "BYP", +) + +const _ICT_WINDING_CATEGORIES = + ("TR2W_WINDING", "PRIMARY_WINDING", "SECONDARY_WINDING", "TERTIARY_WINDING") + +const _ICT_3W_WINDING_KEYS = ( + ("primary", "PRIMARY_WINDING"), + ("secondary", "SECONDARY_WINDING"), + ("tertiary", "TERTIARY_WINDING"), +) + +# Insertion order for the OpenAPI types: matches `ALL_DESERIALIZABLE_TYPES` +# / `ALL_PSY_TYPES` in `SiennaOpenAPIModels.jl/src/dbinterface/translation_constants.jl:59-86`, +# so FK references (e.g. ThermalStandard.bus → ACBus.id) resolve in order. +# Topology and arcs first, then branches, then injections. +const _MAKE_DATABASE_TYPE_ORDER = ( + :Area, + :LoadZone, + :ACBus, + # Arcs are inserted by `write_arcs_to_db!` before the branch-like types + # below, not via send_openapi_table_to_db!. + :AreaInterchange, + :Line, + # TransformerCircuit rows must exist before the 2W/3W holders that FK + # into them. + :TransformerCircuit, + :TwoWindingTransformer, + :ThreeWindingTransformer, + :TwoTerminalLCCLine, + :TwoTerminalGenericHVDCLine, + :TwoTerminalVSCLine, + :DiscreteControlledACBranch, + :FACTSControlDevice, + :PowerLoad, + :StandardLoad, + :InterruptibleStandardLoad, + :FixedAdmittance, + :SwitchedAdmittance, + :ThermalStandard, + :RenewableDispatch, + :RenewableNonDispatch, + :HydroDispatch, + :SynchronousCondenser, + :EnergyReservoirStorage, +) + +""" +Translate a PSS/E FACTS control_mode integer to the OpenAPI enum string +(0/1/2 → "OOS"/"NML"/"BYP"). Throws on any other integer. +""" +function _normalize_facts_control_mode(val::Integer) + haskey(_FACTS_CONTROL_MODE_MAP, Int(val)) || throw( + DataFormatError( + "FACTS control_mode $val has no corresponding OpenAPI enum string", + ), + ) + return _FACTS_CONTROL_MODE_MAP[Int(val)] +end + +_min_max_dict(min_val, max_val) = + Dict{String, Any}("min" => min_val, "max" => max_val) + +_up_down_dict(up_val, down_val) = + Dict{String, Any}("up" => up_val, "down" => down_val) + +_in_out_dict(in_val, out_val) = + Dict{String, Any}("in" => in_val, "out" => out_val) + +""" +Look up the canonical OpenAPI `prime_mover_type` enum string for a raw input +string (e.g. `"WIND"` → `"WT"`). +""" +function _normalize_prime_mover(raw::AbstractString) + key = lowercase(strip(String(raw))) + if haskey(_STRING2PRIMEMOVER, key) + return _STRING2PRIMEMOVER[key] + end + @warn "Unrecognized prime mover string $(repr(raw)); falling back to \"OT\"" + return "OT" +end + +""" +Look up the canonical OpenAPI `fuel_type` enum string for a raw input string +(e.g. `"NG"` → `"NATURAL_GAS"`). +""" +function _normalize_fuel(raw::AbstractString) + key = lowercase(strip(String(raw))) + if haskey(_STRING2FUEL, key) + return _STRING2FUEL[key] + end + @warn "Unrecognized fuel string $(repr(raw)); falling back to \"OTHER\"" + return "OTHER" +end + +""" +Resolve a generator's per-unit base power. +""" +function _resolve_mbase(d::Dict, sys_mbase::Float64, gen_name::AbstractString) + if d["mbase"] != 0.0 + return float(d["mbase"]) + end + @warn "Generator $gen_name has base power equal to zero: $(d["mbase"]). + Changing it to system base: $sys_mbase" + return sys_mbase +end + +# Functions for calculating generator values. + +function _calculate_gen_rating(pmax::Float64, qmax::Float64, base_conversion::Float64) + rating = sqrt(pmax^2 + qmax^2) + if rating == 0.0 + @warn "Rating calculation returned 0.0. Changing to 1.0 in the p.u. of the device." + return 1.0 + end + return rating * base_conversion +end + +function _calculate_ramp_limit_dict(d::Dict, gen_name::AbstractString) + if haskey(d, "ramp_agc") + return _up_down_dict(d["ramp_agc"], d["ramp_agc"]) + end + if haskey(d, "ramp_10") + return _up_down_dict(d["ramp_10"], d["ramp_10"]) + end + if haskey(d, "ramp_30") + return _up_down_dict(d["ramp_30"], d["ramp_30"]) + end + if abs(d["pmax"]) > 0.0 + @debug "No ramp limits found for generator $gen_name. Using pmax as ramp limit." + return _up_down_dict(abs(d["pmax"]), abs(d["pmax"])) + end + @warn "Not enough information to determine ramp limit for generator $gen_name. Returning nothing" + return nothing +end + +""" +Load the YAML mapping from `(fuel, unit_type)` tuples to OpenAPI generator +type tags. +""" +function _get_generator_mapping(filename::AbstractString) + genmap = open(YAML.load, filename) + mappings = Dict{NamedTuple{(:fuel, :unit_type), Tuple{Any, Any}}, Symbol}() + for (gen_type, vals) in genmap + tag = Symbol(gen_type) + for val in vals + key = (fuel = val["fuel"], unit_type = val["type"]) + if haskey(mappings, key) + error("duplicate generator mappings: $tag $(key.fuel) $(key.unit_type)") + end + mappings[key] = tag + end + end + return mappings +end + +""" +Look up the OpenAPI generator type tag for a given `(fuel, unit_type)` pair. +""" +function _get_generator_type(fuel, unit_type, mappings) + fuel_str = isnothing(fuel) ? "" : uppercase(String(fuel)) + unit_type_str = uppercase(String(unit_type)) + for ut in (unit_type_str, nothing), fu in (fuel_str, nothing) + key = (fuel = fu, unit_type = ut) + if haskey(mappings, key) + return mappings[key] + end + end + @error "No mapping for generator fuel=$fuel_str unit_type=$unit_type_str" + return nothing +end + +""" +Extract proportional/constant terms from an `IS.LinearCurve` and wrap as +an OpenAPI `InputOutputCurve` dict. +""" +_linear_curve_to_io_dict(curve) = + _linear_io_curve_dict(IS.get_proportional_term(curve), IS.get_constant_term(curve)) + +""" +Monotonic id minter for OpenAPI-shaped component dicts. Guarantees +cross-component-type uniqueness. +""" +mutable struct IDGenerator + nextid::Int64 + key2int::Dict{Tuple{Symbol, Any}, Int64} +end + +IDGenerator(nextid::Int64 = 1) = IDGenerator(nextid, Dict{Tuple{Symbol, Any}, Int64}()) + +""" +Return the int id for `(type_tag, natural_key)`, minting one from the +generator's counter if this is the first time the key has been seen. +""" +function getid!(ids::IDGenerator, type_tag::Symbol, natural_key) + key = (type_tag, natural_key) + if haskey(ids.key2int, key) + return ids.key2int[key] + end + ids.key2int[key] = ids.nextid + ids.nextid += 1 + return ids.key2int[key] +end + +getid!(::IDGenerator, ::Symbol, ::Nothing) = nothing diff --git a/src/generator_mapping_pm.yaml b/src/generator_mapping_pm.yaml new file mode 100644 index 0000000..4ed6851 --- /dev/null +++ b/src/generator_mapping_pm.yaml @@ -0,0 +1,33 @@ +# Parsing code ignores type=null. + +HydroTurbine: +- {fuel: HYDRO, type: null} +- {fuel: HYDRO, type: HYDRO} + +HydroDispatch: +- {fuel: HYDRO, type: ROR} + +RenewableDispatch: +- {fuel: SOLAR, type: PV} +- {fuel: SOLAR, type: UN} +- {fuel: WIND, type: WIND} +- {fuel: WIND, type: null} +- {fuel: SOLAR, type: CSP} # TODO: may need a new struct + +RenewableNonDispatch: +- {fuel: SOLAR, type: RTPV} + +ThermalStandard: +- {fuel: OIL, type: null} +- {fuel: COAL, type: null} +- {fuel: NG, type: null} +- {fuel: GAS, type: null} +- {fuel: NUCLEAR, type: null} +- {fuel: NUC, type: null} +- {fuel: OTHER, type: OT} + +SynchronousCondenser: +- {fuel: SYNC_COND, type: SYNC_COND} + +EnergyReservoirStorage: +- {fuel: STORAGE, type: null} diff --git a/src/pm_io/data.jl b/src/pm_io/data.jl index 9335fc7..fc950d1 100644 --- a/src/pm_io/data.jl +++ b/src/pm_io/data.jl @@ -1,121 +1,5 @@ # tools for working with a PowerModels data dict structure -"" -function calc_branch_t(branch::Dict{String, <:Any}) - tap_ratio = branch["tap"] - angle_shift = branch["shift"] - - tr = tap_ratio .* cos.(angle_shift) - ti = tap_ratio .* sin.(angle_shift) - - return tr, ti -end - -"" -function calc_branch_y(branch::Dict{String, <:Any}) - y = LinearAlgebra.pinv(branch["br_r"] + im * branch["br_x"]) - g, b = real(y), imag(y) - return g, b -end - -"" -function calc_theta_delta_bounds(data::Dict{String, <:Any}) - bus_count = length(data["bus"]) - branches = [branch for branch in values(data["branch"])] - if haskey(data, "ne_branch") - append!(branches, values(data["ne_branch"])) - end - - angle_min = Real[] - angle_max = Real[] - - conductors = 1 - if haskey(data, "conductors") - conductors = data["conductors"] - end - conductor_ids = 1:conductors - - for c in conductor_ids - angle_mins = [branch["angmin"][c] for branch in branches] - angle_maxs = [branch["angmax"][c] for branch in branches] - - sort!(angle_mins) - sort!(angle_maxs; rev = true) - - if length(angle_mins) > 1 - # note that, this can occur when dclines are present - angle_count = min(bus_count - 1, length(branches)) - - angle_min_val = sum(angle_mins[1:angle_count]) - angle_max_val = sum(angle_maxs[1:angle_count]) - else - angle_min_val = angle_mins[1] - angle_max_val = angle_maxs[1] - end - - push!(angle_min, angle_min_val) - push!(angle_max, angle_max_val) - end - - if haskey(data, "conductors") - error("Multiconductor Not Supported in PowerSystems") - #amin = MultiConductorVector(angle_min) - #amax = MultiConductorVector(angle_max) - #return amin, amax - else - return angle_min[1], angle_max[1] - end -end - -"" -function calc_max_cost_index(data::Dict{String, Any}) - if ismultinetwork(data) # ismultinetwork is in im_io/data.jl - max_index = 0 - for (i, nw_data) in data["nw"] - nw_max_index = _calc_max_cost_index(nw_data) - max_index = max(max_index, nw_max_index) - end - return max_index - else - return _calc_max_cost_index(data) - end -end - -"" -function _calc_max_cost_index(data::Dict{String, <:Any}) - max_index = 0 - - for (i, gen) in data["gen"] - if haskey(gen, "model") - if gen["model"] == 2 - if haskey(gen, "cost") - max_index = max(max_index, length(gen["cost"])) - end - else - @info( - "skipping cost generator $(i) cost model in calc_cost_order, only model 2 is supported." - ) - end - end - end - - for (i, dcline) in data["dcline"] - if haskey(dcline, "model") - if dcline["model"] == 2 - if haskey(dcline, "cost") - max_index = max(max_index, length(dcline["cost"])) - end - else - @info( - "skipping cost dcline $(i) cost model in calc_cost_order, only model 2 is supported." - ) - end - end - end - - return max_index -end - "maps component types to status parameters" const pm_component_status = Dict( "bus" => "bus_type", @@ -236,23 +120,7 @@ const _pm_component_parameter_order = Dict( const _pm_component_status_parameters = Set(["status", "gen_status", "br_status"]) -#component_table(data::Dict{String,Any}, component::String, args...) = component_table(data, component, args...) -#= -"recursively applies new_data to data, overwriting information" -function update_data!(data::Dict{String,<:Any}, new_data::Dict{String,<:Any}) - if haskey(data, "conductors") && haskey(new_data, "conductors") - if data["conductors"] != new_data["conductors"] - error("update_data requires datasets with the same number of conductors") - end - else - if (haskey(data, "conductors") && !haskey(new_data, "conductors")) || (!haskey(data, "conductors") && haskey(new_data, "conductors")) - @info("running update_data with missing onductors fields, conductors may be incorrect") - end - end - update_data!(data, new_data) -end -=# """ Turns in given single network data in multinetwork data with a `count` replicate of the given network. Note that this function performs a deepcopy @@ -260,16 +128,6 @@ of the network data. Significant multinetwork space savings can often be achieved by building application specific methods of building multinetwork with minimal data replication. """ -function replicate( - sn_data::Dict{String, <:Any}, - count::Int; - global_keys::Set{String} = Set{String}(), -) - pm_global_keys = Set(["baseMVA", "per_unit"]) - return im_replicate(sn_data, count, union(global_keys, pm_global_keys)) -end - -"" function _apply_func!(data::Dict{String, <:Any}, key::String, func) if haskey(data, key) data[key] = func(data[key]) # multiconductor not supported in PowerSystems @@ -435,2430 +293,924 @@ function _make_per_unit!(data::Dict{String, <:Any}, mva_base::Real) end end -"Transforms network data into mixed-units (inverse of per-unit)" -function make_mixed_units!(data::Dict{String, <:Any}) - if haskey(data, "per_unit") && data["per_unit"] == true - data["per_unit"] = false - mva_base = data["baseMVA"] - if ismultinetwork(data) - for (i, nw_data) in data["nw"] - _make_mixed_units!(nw_data, mva_base) +"" +function _rescale_cost_model!(comp::Dict{String, <:Any}, scale::Real) + if "model" in keys(comp) && "cost" in keys(comp) + if comp["model"] == 1 + for i in 1:2:length(comp["cost"]) + comp["cost"][i] = comp["cost"][i] / scale + end + elseif comp["model"] == 2 + degree = length(comp["cost"]) + for (i, item) in enumerate(comp["cost"]) + comp["cost"][i] = item * (scale^(degree - i)) end else - _make_mixed_units!(data, mva_base) + @info("Skipping cost model of type $(comp["model"]) in per unit transformation") end end end "" -function _make_mixed_units!(data::Dict{String, <:Any}) - mva_base = data["baseMVA"] - - # to be consistent with matpower's opf.flow_lim= 'I' with current magnitude - # limit defined in MVA at 1 p.u. voltage - ka_base = mva_base - - rescale = x -> x * mva_base - rescale_dual = x -> x / mva_base - rescale_ampere = x -> x * ka_base - - if haskey(data, "bus") - for (i, bus) in data["bus"] - _apply_func!(bus, "va", rad2deg) - - _apply_func!(bus, "lam_kcl_r", rescale_dual) - _apply_func!(bus, "lam_kcl_i", rescale_dual) +function check_conductors(data::Dict{String, <:Any}) + if ismultinetwork(data) + for (i, nw_data) in data["nw"] + _check_conductors(nw_data) end + else + _check_conductors(data) end +end - if haskey(data, "load") - for (i, load) in data["load"] - _apply_func!(load, "pd", rescale) - _apply_func!(load, "qd", rescale) - end +"" +function _check_conductors(data::Dict{String, <:Any}) + if haskey(data, "conductors") && data["conductors"] < 1 + error("conductor values must be positive integers, given $(data["conductors"])") end +end - if haskey(data, "shunt") - for (i, shunt) in data["shunt"] - _apply_func!(shunt, "gs", rescale) - _apply_func!(shunt, "bs", rescale) - end +"checks that voltage angle differences are within 90 deg., if not tightens" +function correct_voltage_angle_differences!(data::Dict{String, <:Any}, default_pad = 1.0472) + if ismultinetwork(data) + error("check_voltage_angle_differences does not yet support multinetwork data") end - if haskey(data, "gen") - for (i, gen) in data["gen"] - _apply_func!(gen, "pg", rescale) - _apply_func!(gen, "qg", rescale) + @assert("per_unit" in keys(data) && data["per_unit"]) + default_pad_deg = round(rad2deg(default_pad); digits = 2) - _apply_func!(gen, "pmax", rescale) - _apply_func!(gen, "pmin", rescale) + modified = Set{Int}() - _apply_func!(gen, "qmax", rescale) - _apply_func!(gen, "qmin", rescale) + for c in 1:get(data, "conductors", 1) + cnd_str = haskey(data, "conductors") ? ", conductor $(c)" : "" + for (i, branch) in data["branch"] + angmin = branch["angmin"][c] + angmax = branch["angmax"][c] - _apply_func!(gen, "ramp_agc", rescale) - _apply_func!(gen, "ramp_10", rescale) - _apply_func!(gen, "ramp_30", rescale) - _apply_func!(gen, "ramp_q", rescale) + if angmin <= -pi / 2 + @info "this code only supports angmin values in -90 deg. to 90 deg., tightening the value on branch $i$(cnd_str) from $(rad2deg(angmin)) to -$(default_pad_deg) deg." maxlog = + PS_MAX_LOG + if haskey(data, "conductors") + branch["angmin"][c] = -default_pad + else + branch["angmin"] = -default_pad + end + push!(modified, branch["index"]) + end - _rescale_cost_model!(gen, 1.0 / mva_base) - end - end + if angmax >= pi / 2 + @info "this code only supports angmax values in -90 deg. to 90 deg., tightening the value on branch $i$(cnd_str) from $(rad2deg(angmax)) to $(default_pad_deg) deg." maxlog = + PS_MAX_LOG + if haskey(data, "conductors") + branch["angmax"][c] = default_pad + else + branch["angmax"] = default_pad + end + push!(modified, branch["index"]) + end - if haskey(data, "storage") - for (i, strg) in data["storage"] - _apply_func!(strg, "energy", rescale) - _apply_func!(strg, "energy_rating", rescale) - _apply_func!(strg, "charge_rating", rescale) - _apply_func!(strg, "discharge_rating", rescale) - _apply_func!(strg, "thermal_rating", rescale) - _apply_func!(strg, "current_rating", rescale) - _apply_func!(strg, "qmin", rescale) - _apply_func!(strg, "qmax", rescale) - _apply_func!(strg, "p_loss", rescale) - _apply_func!(strg, "q_loss", rescale) + if angmin == 0.0 && angmax == 0.0 + @info "angmin and angmax values are 0, widening these values on branch $i$(cnd_str) to +/- $(default_pad_deg) deg." maxlog = + PS_MAX_LOG + if haskey(data, "conductors") + branch["angmin"][c] = -default_pad + branch["angmax"][c] = default_pad + else + branch["angmin"] = -default_pad + branch["angmax"] = default_pad + end + push!(modified, branch["index"]) + end end end - if haskey(data, "switch") - for (i, switch) in data["switch"] - _apply_func!(switch, "psw", rescale) - _apply_func!(switch, "qsw", rescale) - _apply_func!(switch, "thermal_rating", rescale) - _apply_func!(switch, "current_rating", rescale) - end - end + return modified +end - branches = [] - if haskey(data, "branch") - append!(branches, values(data["branch"])) +"checks that each branch has a reasonable thermal rating-a, if not computes one" +function correct_thermal_limits!(data::Dict{String, <:Any}) + if ismultinetwork(data) + error("correct_thermal_limits! does not yet support multinetwork data") end + @assert("per_unit" in keys(data) && data["per_unit"]) + mva_base = data["baseMVA"] + + modified = Set{Int}() + + branches = [branch for branch in values(data["branch"])] if haskey(data, "ne_branch") append!(branches, values(data["ne_branch"])) end for branch in branches - _apply_func!(branch, "rate_a", rescale) - _apply_func!(branch, "rate_b", rescale) - _apply_func!(branch, "rate_c", rescale) + if !haskey(branch, "rate_a") + if haskey(data, "conductors") + error("Multiconductor Not Supported in PowerSystems") + else + branch["rate_a"] = 0.0 + end + end - _apply_func!(branch, "c_rating_a", rescale_ampere) - _apply_func!(branch, "c_rating_b", rescale_ampere) - _apply_func!(branch, "c_rating_c", rescale_ampere) + for c in 1:get(data, "conductors", 1) + cnd_str = haskey(data, "conductors") ? ", conductor $(c)" : "" + if branch["rate_a"][c] <= 0.0 + theta_max = max(abs(branch["angmin"][c]), abs(branch["angmax"][c])) - _apply_func!(branch, "shift", rad2deg) - _apply_func!(branch, "angmax", rad2deg) - _apply_func!(branch, "angmin", rad2deg) + r = branch["br_r"] + x = branch["br_x"] + z = r + im * x + y = LinearAlgebra.pinv(z) + y_mag = abs.(y[c, c]) - _apply_func!(branch, "pf", rescale) - _apply_func!(branch, "pt", rescale) - _apply_func!(branch, "qf", rescale) - _apply_func!(branch, "qt", rescale) + fr_vmax = data["bus"][branch["f_bus"]]["vmax"][c] + to_vmax = data["bus"][branch["t_bus"]]["vmax"][c] + m_vmax = max(fr_vmax, to_vmax) - _apply_func!(branch, "mu_sm_fr", rescale_dual) - _apply_func!(branch, "mu_sm_to", rescale_dual) + c_max = sqrt(fr_vmax^2 + to_vmax^2 - 2 * fr_vmax * to_vmax * cos(theta_max)) - _apply_func!(branch, "ta", rad2deg) - end + new_rate = y_mag * m_vmax * c_max - if haskey(data, "dcline") - for (i, dcline) in data["dcline"] - _apply_func!(dcline, "loss0", rescale) - _apply_func!(dcline, "pf", rescale) - _apply_func!(dcline, "pt", rescale) - _apply_func!(dcline, "qf", rescale) - _apply_func!(dcline, "qt", rescale) - _apply_func!(dcline, "pmaxt", rescale) - _apply_func!(dcline, "pmint", rescale) - _apply_func!(dcline, "pmaxf", rescale) - _apply_func!(dcline, "pminf", rescale) - _apply_func!(dcline, "qmaxt", rescale) - _apply_func!(dcline, "qmint", rescale) - _apply_func!(dcline, "qmaxf", rescale) - _apply_func!(dcline, "qminf", rescale) + if haskey(branch, "c_rating_a") && branch["c_rating_a"][c] > 0.0 + new_rate = min(new_rate, branch["c_rating_a"][c] * m_vmax) + end - _rescale_cost_model!(dcline, 1.0 / mva_base) - end - end -end + @info "this code only supports positive rate_a values, changing the value on branch $(branch["index"])$(cnd_str) to $(round(mva_base*new_rate, digits=4))" maxlog = + PS_MAX_LOG -"" -function _rescale_cost_model!(comp::Dict{String, <:Any}, scale::Real) - if "model" in keys(comp) && "cost" in keys(comp) - if comp["model"] == 1 - for i in 1:2:length(comp["cost"]) - comp["cost"][i] = comp["cost"][i] / scale - end - elseif comp["model"] == 2 - degree = length(comp["cost"]) - for (i, item) in enumerate(comp["cost"]) - comp["cost"][i] = item * (scale^(degree - i)) + if haskey(data, "conductors") + branch["rate_a"][c] = new_rate + else + branch["rate_a"] = new_rate + end + + push!(modified, branch["index"]) end - else - @info("Skipping cost model of type $(comp["model"]) in per unit transformation") end end -end -"computes the generator cost from given network data" -function calc_gen_cost(data::Dict{String, <:Any}) - @assert("per_unit" in keys(data) && data["per_unit"]) - @assert(!haskey(data, "conductors")) + return modified +end +"checks that all parallel branches have the same orientation" +function correct_branch_directions!(data::Dict{String, <:Any}) if ismultinetwork(data) - nw_costs = Dict{String, Any}() - for (i, nw_data) in data["nw"] - nw_costs[i] = _calc_gen_cost(nw_data) - end - return sum(nw_cost for (i, nw_cost) in nw_costs) - else - return _calc_gen_cost(data) + error("correct_branch_directions! does not yet support multinetwork data") end -end -function _calc_gen_cost(data::Dict{String, <:Any}) - cost = 0.0 - for (i, gen) in data["gen"] - if gen["gen_status"] == 1 - if haskey(gen, "model") - if gen["model"] == 1 - cost += _calc_cost_pwl(gen, "pg") - elseif gen["model"] == 2 - cost += _calc_cost_polynomial(gen, "pg") - else - @info "generator $(i) has an unknown cost model $(gen["model"])" maxlog = - PS_MAX_LOG - end - else - @info "generator $(i) does not have a cost model" maxlog = PS_MAX_LOG - end - end - end - return cost -end + modified = Set{Int}() -"computes the dcline cost from given network data" -function calc_dcline_cost(data::Dict{String, <:Any}) - @assert("per_unit" in keys(data) && data["per_unit"]) - @assert(!haskey(data, "conductors")) + orientations = Set() + for (i, branch) in data["branch"] + orientation = (branch["f_bus"], branch["t_bus"]) + orientation_rev = (branch["t_bus"], branch["f_bus"]) - if ismultinetwork(data) - nw_costs = Dict{String, Any}() - for (i, nw_data) in data["nw"] - nw_costs[i] = _calc_dcline_cost(nw_data) - end - return sum(nw_cost for (i, nw_cost) in nw_costs) - else - return _calc_dcline_cost(data) - end -end + if in(orientation_rev, orientations) + @info( + "reversing the orientation of branch $(i) $(orientation) to be consistent with other parallel branches" + ) + branch_orginal = copy(branch) + branch["f_bus"] = branch_orginal["t_bus"] + branch["t_bus"] = branch_orginal["f_bus"] + branch["g_to"] = branch_orginal["g_fr"] .* branch_orginal["tap"]' .^ 2 + branch["b_to"] = branch_orginal["b_fr"] .* branch_orginal["tap"]' .^ 2 + branch["g_fr"] = branch_orginal["g_to"] ./ branch_orginal["tap"]' .^ 2 + branch["b_fr"] = branch_orginal["b_to"] ./ branch_orginal["tap"]' .^ 2 + branch["tap"] = 1 ./ branch_orginal["tap"] + branch["br_r"] = branch_orginal["br_r"] .* branch_orginal["tap"]' .^ 2 + branch["br_x"] = branch_orginal["br_x"] .* branch_orginal["tap"]' .^ 2 + branch["shift"] = -branch_orginal["shift"] + branch["angmin"] = -branch_orginal["angmax"] + branch["angmax"] = -branch_orginal["angmin"] -function _calc_dcline_cost(data::Dict{String, <:Any}) - cost = 0.0 - for (i, dcline) in data["dcline"] - if dcline["br_status"] == 1 - if haskey(dcline, "model") - if dcline["model"] == 1 - cost += _calc_cost_pwl(dcline, "pf") - elseif dcline["model"] == 2 - cost += _calc_cost_polynomial(dcline, "pf") - else - @info "dcline $(i) has an unknown cost model $(dcline["model"])" maxlog = - PS_MAX_LOG - end - else - @info "dcline $(i) does not have a cost model" maxlog = PS_MAX_LOG - end + push!(modified, branch["index"]) + else + push!(orientations, orientation) end end - return cost -end - -""" -compute lines in m and b from from pwl cost models data is a list of components. -Can be run on data or ref data structures -""" -function calc_cost_pwl_lines(comp_dict::Dict) - lines = Dict() - for (i, comp) in comp_dict - lines[i] = _calc_comp_lines(comp) - end - return lines + return modified end -""" -compute lines in m and b from from pwl cost models -""" -function _calc_comp_lines(component::Dict{String, <:Any}) - @assert component["model"] == 1 - points = component["cost"] - - line_data = [] - for i in 3:2:length(points) - x1 = points[i - 2] - y1 = points[i - 1] - x2 = points[i - 0] - y2 = points[i + 1] - - m = (y2 - y1) / (x2 - x1) - b = y1 - m * x1 - - push!(line_data, (slope = m, intercept = b)) +"checks that all branches connect two distinct buses" +function check_branch_loops(data::Dict{String, <:Any}) + if ismultinetwork(data) + error("check_branch_loops does not yet support multinetwork data") end - for i in 2:length(line_data) - if line_data[i - 1].slope > line_data[i].slope - @info "non-convex pwl function found in points $(component["cost"])\nlines: $(line_data)" maxlog = - PS_MAX_LOG + for (i, branch) in data["branch"] + if branch["f_bus"] == branch["t_bus"] + throw( + DataFormatError( + "both sides of branch $(i) connect to bus $(branch["f_bus"])", + ), + ) end end - - return line_data end -function _calc_cost_pwl(component::Dict{String, <:Any}, setpoint_id) - comp_lines = _calc_comp_lines(component) - - setpoint = component[setpoint_id] - cost = -Inf - for line in comp_lines - cost = max(cost, line.slope * setpoint + line.intercept) +"checks that all buses are unique and other components link to valid buses" +function check_connectivity(data::Dict{String, <:Any}) + if ismultinetwork(data) + error("check_connectivity does not yet support multinetwork data") end - return cost -end - -function _calc_cost_polynomial(component::Dict{String, <:Any}, setpoint_id) - cost_terms_rev = reverse(component["cost"]) - - setpoint = component[setpoint_id] + bus_ids = Set(bus["index"] for (i, bus) in data["bus"]) + @assert(length(bus_ids) == length(data["bus"])) # if this is not true something very bad is going on - if length(cost_terms_rev) == 0 - cost = 0.0 - elseif length(cost_terms_rev) == 1 - cost = cost_terms_rev[1] - elseif length(cost_terms_rev) == 2 - cost = cost_terms_rev[1] + cost_terms_rev[2] * setpoint - else - cost_terms_rev_high = cost_terms_rev[3:end] - cost = - cost_terms_rev[1] + - cost_terms_rev[2] * setpoint + - sum(v * setpoint^(d + 1) for (d, v) in enumerate(cost_terms_rev_high)) + for (i, load) in data["load"] + if !(load["load_bus"] in bus_ids) + throw(DataFormatError("bus $(load["load_bus"]) in load $(i) is not defined")) + end end - return cost -end - -"assumes a vaild ac solution is included in the data and computes the branch flow values" -function calc_branch_flow_ac(data::Dict{String, <:Any}) - @assert("per_unit" in keys(data) && data["per_unit"]) - @assert(!haskey(data, "conductors")) - - if ismultinetwork(data) - nws = Dict{String, Any}() - for (i, nw_data) in data["nw"] - nws[i] = _calc_branch_flow_ac(nw_data) + for (i, shunt) in data["shunt"] + if !(shunt["shunt_bus"] in bus_ids) + throw(DataFormatError("bus $(shunt["shunt_bus"]) in shunt $(i) is not defined")) end - return Dict{String, Any}( - "nw" => nws, - "per_unit" => data["per_unit"], - "baseMVA" => data["baseMVA"], - ) - else - flows = _calc_branch_flow_ac(data) - flows["per_unit"] = data["per_unit"] - flows["baseMVA"] = data["baseMVA"] - return flows end -end - -"helper function for calc_branch_flow_ac" -function _calc_branch_flow_ac(data::Dict{String, <:Any}) - vm = Dict(bus["index"] => bus["vm"] for (i, bus) in data["bus"]) - va = Dict(bus["index"] => bus["va"] for (i, bus) in data["bus"]) - - flows = Dict{String, Any}() - for (i, branch) in data["branch"] - if branch["br_status"] != 0 - f_bus = branch["f_bus"] - t_bus = branch["t_bus"] - - g, b = calc_branch_y(branch) - tr, ti = calc_branch_t(branch) - g_fr = branch["g_fr"] - b_fr = branch["b_fr"] - g_to = branch["g_to"] - b_to = branch["b_to"] - - tm = branch["tap"] - - vm_fr = vm[f_bus] - vm_to = vm[t_bus] - va_fr = va[f_bus] - va_to = va[t_bus] - - p_fr = - (g + g_fr) / tm^2 * vm_fr^2 + - (-g * tr + b * ti) / tm^2 * (vm_fr * vm_to * cos(va_fr - va_to)) + - (-b * tr - g * ti) / tm^2 * (vm_fr * vm_to * sin(va_fr - va_to)) - q_fr = - -(b + b_fr) / tm^2 * vm_fr^2 - - (-b * tr - g * ti) / tm^2 * (vm_fr * vm_to * cos(va_fr - va_to)) + - (-g * tr + b * ti) / tm^2 * (vm_fr * vm_to * sin(va_fr - va_to)) - - p_to = - (g + g_to) * vm_to^2 + - (-g * tr - b * ti) / tm^2 * (vm_to * vm_fr * cos(va_to - va_fr)) + - (-b * tr + g * ti) / tm^2 * (vm_to * vm_fr * sin(va_to - va_fr)) - q_to = - -(b + b_to) * vm_to^2 - - (-b * tr + g * ti) / tm^2 * (vm_to * vm_fr * cos(va_to - va_fr)) + - (-g * tr - b * ti) / tm^2 * (vm_to * vm_fr * sin(va_to - va_fr)) - else - p_fr = NaN - q_fr = NaN - p_to = NaN - q_to = NaN + for (i, gen) in data["gen"] + if !(gen["gen_bus"] in bus_ids) + throw(DataFormatError("bus $(gen["gen_bus"]) in generator $(i) is not defined")) end - - flows[i] = Dict("pf" => p_fr, "qf" => q_fr, "pt" => p_to, "qt" => q_to) end - return Dict{String, Any}("branch" => flows) -end + for (i, strg) in data["storage"] + if !(strg["storage_bus"] in bus_ids) + throw( + DataFormatError( + "bus $(strg["storage_bus"]) in storage unit $(i) is not defined", + ), + ) + end + end -"assumes a vaild dc solution is included in the data and computes the branch flow values" -function calc_branch_flow_dc(data::Dict{String, <:Any}) - @assert("per_unit" in keys(data) && data["per_unit"]) - @assert(!haskey(data, "conductors")) + if haskey(data, "switch") + for (i, switch) in data["switch"] + if !(switch["f_bus"] in bus_ids) + throw( + DataFormatError( + "from bus $(branch["f_bus"]) in switch $(i) is not defined", + ), + ) + end - if ismultinetwork(data) - nws = Dict{String, Any}() - for (i, nw_data) in data["nw"] - nws[i] = _calc_branch_flow_dc(nw_data) + if !(switch["t_bus"] in bus_ids) + throw( + DataFormatError( + "to bus $(branch["t_bus"]) in switch $(i) is not defined", + ), + ) + end end - return Dict{String, Any}( - "nw" => nws, - "per_unit" => data["per_unit"], - "baseMVA" => data["baseMVA"], - ) - else - flows = _calc_branch_flow_dc(data) - flows["per_unit"] = data["per_unit"] - flows["baseMVA"] = data["baseMVA"] - return flows end -end -"helper function for calc_branch_flow_dc" -function _calc_branch_flow_dc(data::Dict{String, <:Any}) - vm = Dict(bus["index"] => bus["vm"] for (i, bus) in data["bus"]) - va = Dict(bus["index"] => bus["va"] for (i, bus) in data["bus"]) - - flows = Dict{String, Any}() for (i, branch) in data["branch"] - if branch["br_status"] != 0 - f_bus = branch["f_bus"] - t_bus = branch["t_bus"] - - g, b = calc_branch_y(branch) - - p_fr = -b * (va[f_bus] - va[t_bus]) - else - p_fr = NaN + if !(branch["f_bus"] in bus_ids) + throw( + DataFormatError( + "from bus $(branch["f_bus"]) in branch $(i) is not defined", + ), + ) end - flows[i] = Dict("pf" => p_fr, "qf" => NaN, "pt" => -p_fr, "qt" => NaN) + if !(branch["t_bus"] in bus_ids) + throw( + DataFormatError("to bus $(branch["t_bus"]) in branch $(i) is not defined"), + ) + end end - return Dict{String, Any}("branch" => flows) -end - -"assumes a vaild solution is included in the data and computes the power balance at each bus" -function calc_power_balance(data::Dict{String, <:Any}) - @assert("per_unit" in keys(data) && data["per_unit"]) # may not be strictly required - @assert(!haskey(data, "conductors")) + for (i, dcline) in data["dcline"] + if !(dcline["f_bus"] in bus_ids) + throw( + DataFormatError( + "from bus $(dcline["f_bus"]) in dcline $(i) is not defined", + ), + ) + end - if ismultinetwork(data) - nws = Dict{String, Any}() - for (i, nw_data) in data["nw"] - nws[i] = _calc_power_balance(nw_data) + if !(dcline["t_bus"] in bus_ids) + throw( + DataFormatError("to bus $(dcline["t_bus"]) in dcline $(i) is not defined"), + ) end - return Dict{String, Any}( - "nw" => nws, - "per_unit" => data["per_unit"], - "baseMVA" => data["baseMVA"], - ) - else - flows = _calc_power_balance(data) - flows["per_unit"] = data["per_unit"] - flows["baseMVA"] = data["baseMVA"] - return flows end end -"helper function for calc_power_balance" -function _calc_power_balance(data::Dict{String, <:Any}) - bus_values = Dict(bus["index"] => Dict{String, Float64}() for (i, bus) in data["bus"]) - for (i, bus) in data["bus"] - bvals = bus_values[bus["index"]] - bvals["vm"] = bus["vm"] - - bvals["pd"] = 0.0 - bvals["qd"] = 0.0 - - bvals["gs"] = 0.0 - bvals["bs"] = 0.0 - - bvals["ps"] = 0.0 - bvals["qs"] = 0.0 - - bvals["pg"] = 0.0 - bvals["qg"] = 0.0 - - bvals["p"] = 0.0 - bvals["q"] = 0.0 - - bvals["psw"] = 0.0 - bvals["qsw"] = 0.0 - - bvals["p_dc"] = 0.0 - bvals["q_dc"] = 0.0 +"checks that active components are not connected to inactive buses, otherwise prints warnings" +function check_status(data::Dict{String, <:Any}) + if ismultinetwork(data) + error("check_status does not yet support multinetwork data") end + active_bus_ids = Set(bus["index"] for (i, bus) in data["bus"] if bus["bus_type"] != 4) + for (i, load) in data["load"] - if load["status"] != 0 - bvals = bus_values[load["load_bus"]] - bvals["pd"] += load["pd"] - bvals["qd"] += load["qd"] + if load["status"] != 0 && !(load["load_bus"] in active_bus_ids) + @warn("active load $(i) is connected to inactive bus $(load["load_bus"])") end end for (i, shunt) in data["shunt"] - if shunt["status"] != 0 - bvals = bus_values[shunt["shunt_bus"]] - bvals["gs"] += shunt["gs"] - bvals["bs"] += shunt["bs"] - end - end - - for (i, storage) in data["storage"] - if storage["status"] != 0 - bvals = bus_values[storage["storage_bus"]] - bvals["ps"] += storage["ps"] - bvals["qs"] += storage["qs"] + if shunt["status"] != 0 && !(shunt["shunt_bus"] in active_bus_ids) + @warn("active shunt $(i) is connected to inactive bus $(shunt["shunt_bus"])") end end for (i, gen) in data["gen"] - if gen["gen_status"] != 0 - bvals = bus_values[gen["gen_bus"]] - bvals["pg"] += gen["pg"] - bvals["qg"] += gen["qg"] + if gen["gen_status"] != 0 && !(gen["gen_bus"] in active_bus_ids) + @warn("active generator $(i) is connected to inactive bus $(gen["gen_bus"])") end end - for (i, switch) in data["switch"] - if switch["status"] != 0 - bus_fr = switch["f_bus"] - bvals_fr = bus_values[bus_fr] - bvals_fr["psw"] += switch["psw"] - bvals_fr["qsw"] += switch["qsw"] - - bus_to = switch["t_bus"] - bvals_to = bus_values[bus_to] - bvals_to["psw"] -= switch["psw"] - bvals_to["qsw"] -= switch["qsw"] + for (i, strg) in data["storage"] + if strg["status"] != 0 && !(strg["storage_bus"] in active_bus_ids) + @warn( + "active storage unit $(i) is connected to inactive bus $(strg["storage_bus"])" + ) end end for (i, branch) in data["branch"] - if branch["br_status"] != 0 - bus_fr = branch["f_bus"] - bvals_fr = bus_values[bus_fr] - bvals_fr["p"] += branch["pf"] - bvals_fr["q"] += branch["qf"] + if branch["br_status"] != 0 && !(branch["f_bus"] in active_bus_ids) + @warn("active branch $(i) is connected to inactive bus $(branch["f_bus"])") + end - bus_to = branch["t_bus"] - bvals_to = bus_values[bus_to] - bvals_to["p"] += branch["pt"] - bvals_to["q"] += branch["qt"] + if branch["br_status"] != 0 && !(branch["t_bus"] in active_bus_ids) + @warn("active branch $(i) is connected to inactive bus $(branch["t_bus"])") end end for (i, dcline) in data["dcline"] - if dcline["br_status"] != 0 - bus_fr = dcline["f_bus"] - bvals_fr = bus_values[bus_fr] - bvals_fr["p_dc"] += dcline["pf"] - bvals_fr["q_dc"] += dcline["qf"] - - bus_to = dcline["t_bus"] - bvals_to = bus_values[bus_to] - bvals_to["p_dc"] += dcline["pt"] - bvals_to["q_dc"] += dcline["qt"] - end - end - - deltas = Dict{String, Any}() - for (i, bus) in data["bus"] - if bus["bus_type"] != 4 - bvals = bus_values[bus["index"]] - p_delta = - bvals["p"] + bvals["p_dc"] + bvals["psw"] - bvals["pg"] + - bvals["ps"] + - bvals["pd"] + - bvals["gs"] * (bvals["vm"]^2) - q_delta = - bvals["q"] + bvals["q_dc"] + bvals["qsw"] - bvals["qg"] + - bvals["qs"] + - bvals["qd"] - bvals["bs"] * (bvals["vm"]^2) - else - p_delta = NaN - q_delta = NaN + if dcline["br_status"] != 0 && !(dcline["f_bus"] in active_bus_ids) + @warn("active dcline $(i) is connected to inactive bus $(dcline["f_bus"])") end - deltas[i] = Dict("p_delta" => p_delta, "q_delta" => q_delta) + if dcline["br_status"] != 0 && !(dcline["t_bus"] in active_bus_ids) + @warn("active dcline $(i) is connected to inactive bus $(dcline["t_bus"])") + end end - - return Dict{String, Any}("bus" => deltas) end -"" -function check_conductors(data::Dict{String, <:Any}) +"checks that contains at least one refrence bus" +function check_reference_bus(data::Dict{String, <:Any}) if ismultinetwork(data) - for (i, nw_data) in data["nw"] - _check_conductors(nw_data) - end - else - _check_conductors(data) + error("check_reference_bus does not yet support multinetwork data") end -end -"" -function _check_conductors(data::Dict{String, <:Any}) - if haskey(data, "conductors") && data["conductors"] < 1 - error("conductor values must be positive integers, given $(data["conductors"])") + ref_buses = Dict{Int, Any}() + for (k, v) in data["bus"] + if v["bus_type"] == 3 + ref_buses[k] = v + end end -end -"checks that voltage angle differences are within 90 deg., if not tightens" -function correct_voltage_angle_differences!(data::Dict{String, <:Any}, default_pad = 1.0472) - if ismultinetwork(data) - error("check_voltage_angle_differences does not yet support multinetwork data") + if length(ref_buses) == 0 + if length(data["gen"]) > 0 + big_gen = _biggest_generator(data["gen"]) + gen_bus = big_gen["gen_bus"] + ref_bus = data["bus"][gen_bus] + ref_bus["bus_type"] = 3 + @warn( + "no reference bus found, setting bus $(gen_bus) as reference based on generator $(big_gen["index"])" + ) + else + (bus_item, state) = Base.iterate(values(data["bus"])) + bus_item["bus_type"] = 3 + @warn( + "no reference bus found, setting bus $(bus_item["index"]) as reference" + ) + end end + return +end - @assert("per_unit" in keys(data) && data["per_unit"]) - default_pad_deg = round(rad2deg(default_pad); digits = 2) - - modified = Set{Int}() - - for c in 1:get(data, "conductors", 1) - cnd_str = haskey(data, "conductors") ? ", conductor $(c)" : "" - for (i, branch) in data["branch"] - angmin = branch["angmin"][c] - angmax = branch["angmax"][c] - - if angmin <= -pi / 2 - @info "this code only supports angmin values in -90 deg. to 90 deg., tightening the value on branch $i$(cnd_str) from $(rad2deg(angmin)) to -$(default_pad_deg) deg." maxlog = - PS_MAX_LOG - if haskey(data, "conductors") - branch["angmin"][c] = -default_pad - else - branch["angmin"] = -default_pad - end - push!(modified, branch["index"]) - end - - if angmax >= pi / 2 - @info "this code only supports angmax values in -90 deg. to 90 deg., tightening the value on branch $i$(cnd_str) from $(rad2deg(angmax)) to $(default_pad_deg) deg." maxlog = - PS_MAX_LOG - if haskey(data, "conductors") - branch["angmax"][c] = default_pad - else - branch["angmax"] = default_pad - end - push!(modified, branch["index"]) - end - - if angmin == 0.0 && angmax == 0.0 - @info "angmin and angmax values are 0, widening these values on branch $i$(cnd_str) to +/- $(default_pad_deg) deg." maxlog = - PS_MAX_LOG - if haskey(data, "conductors") - branch["angmin"][c] = -default_pad - branch["angmax"][c] = default_pad - else - branch["angmin"] = -default_pad - branch["angmax"] = default_pad - end - push!(modified, branch["index"]) - end +"find the largest active generator in the network" +function _biggest_generator(gens) + biggest_gen = nothing + biggest_value = -Inf + for (k, gen) in gens + pmax = maximum(gen["pmax"]) + if pmax > biggest_value + biggest_gen = gen + biggest_value = pmax end end - - return modified + @assert(biggest_gen !== nothing) + return biggest_gen end -"checks that each branch has a reasonable thermal rating-a, if not computes one" -function correct_thermal_limits!(data::Dict{String, <:Any}) +""" +checks that each branch has a reasonable transformer parameters + +this is important because setting tap == 0.0 leads to NaN computations, which are hard to debug +""" +function correct_transformer_parameters!(data::Dict{String, <:Any}) if ismultinetwork(data) - error("correct_thermal_limits! does not yet support multinetwork data") + error("check_transformer_parameters does not yet support multinetwork data") end @assert("per_unit" in keys(data) && data["per_unit"]) - mva_base = data["baseMVA"] modified = Set{Int}() - branches = [branch for branch in values(data["branch"])] - if haskey(data, "ne_branch") - append!(branches, values(data["ne_branch"])) - end - - for branch in branches - if !haskey(branch, "rate_a") + for (i, branch) in data["branch"] + if !haskey(branch, "tap") + @info "branch found without tap value, setting a tap to 1.0" maxlog = PS_MAX_LOG if haskey(data, "conductors") error("Multiconductor Not Supported in PowerSystems") else - branch["rate_a"] = 0.0 + branch["tap"] = 1.0 end - end - - for c in 1:get(data, "conductors", 1) - cnd_str = haskey(data, "conductors") ? ", conductor $(c)" : "" - if branch["rate_a"][c] <= 0.0 - theta_max = max(abs(branch["angmin"][c]), abs(branch["angmax"][c])) - - r = branch["br_r"] - x = branch["br_x"] - z = r + im * x - y = LinearAlgebra.pinv(z) - y_mag = abs.(y[c, c]) - - fr_vmax = data["bus"][branch["f_bus"]]["vmax"][c] - to_vmax = data["bus"][branch["t_bus"]]["vmax"][c] - m_vmax = max(fr_vmax, to_vmax) - - c_max = sqrt(fr_vmax^2 + to_vmax^2 - 2 * fr_vmax * to_vmax * cos(theta_max)) - - new_rate = y_mag * m_vmax * c_max - - if haskey(branch, "c_rating_a") && branch["c_rating_a"][c] > 0.0 - new_rate = min(new_rate, branch["c_rating_a"][c] * m_vmax) - end - - @info "this code only supports positive rate_a values, changing the value on branch $(branch["index"])$(cnd_str) to $(round(mva_base*new_rate, digits=4))" maxlog = - PS_MAX_LOG - - if haskey(data, "conductors") - branch["rate_a"][c] = new_rate - else - branch["rate_a"] = new_rate + push!(modified, branch["index"]) + else + for c in 1:get(data, "conductors", 1) + cnd_str = haskey(data, "conductors") ? " on conductor $(c)" : "" + if branch["tap"][c] <= 0.0 + @info( + "branch found with non-positive tap value of $(branch["tap"][c]), setting a tap to 1.0$(cnd_str)" + ) + if haskey(data, "conductors") + branch["tap"][c] = 1.0 + else + branch["tap"] = 1.0 + end + push!(modified, branch["index"]) end - - push!(modified, branch["index"]) end end - end - - return modified -end - -"checks that each branch has a reasonable current rating-a, if not computes one" -function correct_current_limits!(data::Dict{String, <:Any}) - if ismultinetwork(data) - error("correct_current_limits! does not yet support multinetwork data") - end - - @assert("per_unit" in keys(data) && data["per_unit"]) - mva_base = data["baseMVA"] - - modified = Set{Int}() - - branches = [branch for branch in values(data["branch"])] - if haskey(data, "ne_branch") - append!(branches, values(data["ne_branch"])) - end - - for branch in branches - if !haskey(branch, "c_rating_a") + if !haskey(branch, "shift") + @info("branch found without shift value, setting a shift to 0.0") if haskey(data, "conductors") error("Multiconductor Not Supported in PowerSystems") else - branch["c_rating_a"] = 0.0 - end - end - - for c in 1:get(data, "conductors", 1) - cnd_str = haskey(data, "conductors") ? ", conductor $(c)" : "" - if branch["c_rating_a"][c] <= 0.0 - theta_max = max(abs(branch["angmin"][c]), abs(branch["angmax"][c])) - - r = branch["br_r"] - x = branch["br_x"] - z = r + im * x - y = LinearAlgebra.pinv(z) - y_mag = abs.(y[c, c]) - - fr_vmax = data["bus"][string(branch["f_bus"])]["vmax"][c] - to_vmax = data["bus"][string(branch["t_bus"])]["vmax"][c] - m_vmax = max(fr_vmax, to_vmax) - - new_c_rating = - y_mag * - sqrt(fr_vmax^2 + to_vmax^2 - 2 * fr_vmax * to_vmax * cos(theta_max)) - - if haskey(branch, "rate_a") && branch["rate_a"][c] > 0.0 - fr_vmin = data["bus"][string(branch["f_bus"])]["vmin"][c] - to_vmin = data["bus"][string(branch["t_bus"])]["vmin"][c] - vm_min = min(fr_vmin, to_vmin) - - new_c_rating = min(new_c_rating, branch["rate_a"] / vm_min) - end - - @info( - "this code only supports positive c_rating_a values, changing the value on branch $(branch["index"])$(cnd_str) to $(mva_base*new_c_rating)" - ) - if haskey(data, "conductors") - branch["c_rating_a"][c] = new_c_rating - else - branch["c_rating_a"] = new_c_rating - end - - push!(modified, branch["index"]) + branch["shift"] = 0.0 end + push!(modified, branch["index"]) end end return modified end -"checks that all parallel branches have the same orientation" -function correct_branch_directions!(data::Dict{String, <:Any}) +""" +checks that each storage unit has a reasonable parameters +""" +function check_storage_parameters(data::Dict{String, Any}) if ismultinetwork(data) - error("correct_branch_directions! does not yet support multinetwork data") + error("check_storage_parameters does not yet support multinetwork data") end - modified = Set{Int}() - - orientations = Set() - for (i, branch) in data["branch"] - orientation = (branch["f_bus"], branch["t_bus"]) - orientation_rev = (branch["t_bus"], branch["f_bus"]) - - if in(orientation_rev, orientations) - @info( - "reversing the orientation of branch $(i) $(orientation) to be consistent with other parallel branches" + for (i, strg) in data["storage"] + if strg["energy"] < 0.0 + throw( + DataFormatError( + "storage unit $(strg["index"]) has a non-positive energy level $(strg["energy"])", + ), ) - branch_orginal = copy(branch) - branch["f_bus"] = branch_orginal["t_bus"] - branch["t_bus"] = branch_orginal["f_bus"] - branch["g_to"] = branch_orginal["g_fr"] .* branch_orginal["tap"]' .^ 2 - branch["b_to"] = branch_orginal["b_fr"] .* branch_orginal["tap"]' .^ 2 - branch["g_fr"] = branch_orginal["g_to"] ./ branch_orginal["tap"]' .^ 2 - branch["b_fr"] = branch_orginal["b_to"] ./ branch_orginal["tap"]' .^ 2 - branch["tap"] = 1 ./ branch_orginal["tap"] - branch["br_r"] = branch_orginal["br_r"] .* branch_orginal["tap"]' .^ 2 - branch["br_x"] = branch_orginal["br_x"] .* branch_orginal["tap"]' .^ 2 - branch["shift"] = -branch_orginal["shift"] - branch["angmin"] = -branch_orginal["angmax"] - branch["angmax"] = -branch_orginal["angmin"] - - push!(modified, branch["index"]) - else - push!(orientations, orientation) end - end - - return modified -end - -"checks that all branches connect two distinct buses" -function check_branch_loops(data::Dict{String, <:Any}) - if ismultinetwork(data) - error("check_branch_loops does not yet support multinetwork data") - end - - for (i, branch) in data["branch"] - if branch["f_bus"] == branch["t_bus"] + if strg["energy_rating"] < 0.0 throw( DataFormatError( - "both sides of branch $(i) connect to bus $(branch["f_bus"])", + "storage unit $(strg["index"]) has a non-positive energy rating $(strg["energy_rating"])", ), ) end - end -end - -"checks that all buses are unique and other components link to valid buses" -function check_connectivity(data::Dict{String, <:Any}) - if ismultinetwork(data) - error("check_connectivity does not yet support multinetwork data") - end - - bus_ids = Set(bus["index"] for (i, bus) in data["bus"]) - @assert(length(bus_ids) == length(data["bus"])) # if this is not true something very bad is going on - - for (i, load) in data["load"] - if !(load["load_bus"] in bus_ids) - throw(DataFormatError("bus $(load["load_bus"]) in load $(i) is not defined")) - end - end - - for (i, shunt) in data["shunt"] - if !(shunt["shunt_bus"] in bus_ids) - throw(DataFormatError("bus $(shunt["shunt_bus"]) in shunt $(i) is not defined")) - end - end - - for (i, gen) in data["gen"] - if !(gen["gen_bus"] in bus_ids) - throw(DataFormatError("bus $(gen["gen_bus"]) in generator $(i) is not defined")) - end - end - - for (i, strg) in data["storage"] - if !(strg["storage_bus"] in bus_ids) + if strg["charge_rating"] < 0.0 throw( DataFormatError( - "bus $(strg["storage_bus"]) in storage unit $(i) is not defined", + "storage unit $(strg["index"]) has a non-positive charge rating $(strg["energy_rating"])", ), ) end - end - - if haskey(data, "switch") - for (i, switch) in data["switch"] - if !(switch["f_bus"] in bus_ids) - throw( - DataFormatError( - "from bus $(branch["f_bus"]) in switch $(i) is not defined", - ), - ) - end - - if !(switch["t_bus"] in bus_ids) - throw( - DataFormatError( - "to bus $(branch["t_bus"]) in switch $(i) is not defined", - ), - ) - end - end - end - - for (i, branch) in data["branch"] - if !(branch["f_bus"] in bus_ids) + if strg["discharge_rating"] < 0.0 throw( DataFormatError( - "from bus $(branch["f_bus"]) in branch $(i) is not defined", + "storage unit $(strg["index"]) has a non-positive discharge rating $(strg["energy_rating"])", ), ) end - if !(branch["t_bus"] in bus_ids) + if strg["r"] < 0.0 throw( - DataFormatError("to bus $(branch["t_bus"]) in branch $(i) is not defined"), + DataFormatError( + "storage unit $(strg["index"]) has a non-positive resistance $(strg["r"])", + ), ) end - end - - for (i, dcline) in data["dcline"] - if !(dcline["f_bus"] in bus_ids) + if strg["x"] < 0.0 throw( DataFormatError( - "from bus $(dcline["f_bus"]) in dcline $(i) is not defined", + "storage unit $(strg["index"]) has a non-positive reactance $(strg["x"])", ), ) end - - if !(dcline["t_bus"] in bus_ids) + if haskey(strg, "thermal_rating") && strg["thermal_rating"] < 0.0 throw( - DataFormatError("to bus $(dcline["t_bus"]) in dcline $(i) is not defined"), - ) - end - end -end - -"checks that active components are not connected to inactive buses, otherwise prints warnings" -function check_status(data::Dict{String, <:Any}) - if ismultinetwork(data) - error("check_status does not yet support multinetwork data") - end - - active_bus_ids = Set(bus["index"] for (i, bus) in data["bus"] if bus["bus_type"] != 4) - - for (i, load) in data["load"] - if load["status"] != 0 && !(load["load_bus"] in active_bus_ids) - @warn("active load $(i) is connected to inactive bus $(load["load_bus"])") - end - end - - for (i, shunt) in data["shunt"] - if shunt["status"] != 0 && !(shunt["shunt_bus"] in active_bus_ids) - @warn("active shunt $(i) is connected to inactive bus $(shunt["shunt_bus"])") - end - end - - for (i, gen) in data["gen"] - if gen["gen_status"] != 0 && !(gen["gen_bus"] in active_bus_ids) - @warn("active generator $(i) is connected to inactive bus $(gen["gen_bus"])") - end - end - - for (i, strg) in data["storage"] - if strg["status"] != 0 && !(strg["storage_bus"] in active_bus_ids) - @warn( - "active storage unit $(i) is connected to inactive bus $(strg["storage_bus"])" + DataFormatError( + "storage unit $(strg["index"]) has a non-positive thermal rating $(strg["thermal_rating"])", + ), ) end - end - - for (i, branch) in data["branch"] - if branch["br_status"] != 0 && !(branch["f_bus"] in active_bus_ids) - @warn("active branch $(i) is connected to inactive bus $(branch["f_bus"])") - end - - if branch["br_status"] != 0 && !(branch["t_bus"] in active_bus_ids) - @warn("active branch $(i) is connected to inactive bus $(branch["t_bus"])") - end - end - - for (i, dcline) in data["dcline"] - if dcline["br_status"] != 0 && !(dcline["f_bus"] in active_bus_ids) - @warn("active dcline $(i) is connected to inactive bus $(dcline["f_bus"])") - end - - if dcline["br_status"] != 0 && !(dcline["t_bus"] in active_bus_ids) - @warn("active dcline $(i) is connected to inactive bus $(dcline["t_bus"])") - end - end -end - -"checks that contains at least one refrence bus" -function check_reference_bus(data::Dict{String, <:Any}) - if ismultinetwork(data) - error("check_reference_bus does not yet support multinetwork data") - end - - ref_buses = Dict{Int, Any}() - for (k, v) in data["bus"] - if v["bus_type"] == 3 - ref_buses[k] = v - end - end - - if length(ref_buses) == 0 - if length(data["gen"]) > 0 - big_gen = _biggest_generator(data["gen"]) - gen_bus = big_gen["gen_bus"] - ref_bus = data["bus"][gen_bus] - ref_bus["bus_type"] = 3 - @warn( - "no reference bus found, setting bus $(gen_bus) as reference based on generator $(big_gen["index"])" - ) - else - (bus_item, state) = Base.iterate(values(data["bus"])) - bus_item["bus_type"] = 3 - @warn( - "no reference bus found, setting bus $(bus_item["index"]) as reference" - ) - end - end - return -end - -"find the largest active generator in the network" -function _biggest_generator(gens) - biggest_gen = nothing - biggest_value = -Inf - for (k, gen) in gens - pmax = maximum(gen["pmax"]) - if pmax > biggest_value - biggest_gen = gen - biggest_value = pmax - end - end - @assert(biggest_gen !== nothing) - return biggest_gen -end - -""" -checks that each branch has a reasonable transformer parameters - -this is important because setting tap == 0.0 leads to NaN computations, which are hard to debug -""" -function correct_transformer_parameters!(data::Dict{String, <:Any}) - if ismultinetwork(data) - error("check_transformer_parameters does not yet support multinetwork data") - end - - @assert("per_unit" in keys(data) && data["per_unit"]) - - modified = Set{Int}() - - for (i, branch) in data["branch"] - if !haskey(branch, "tap") - @info "branch found without tap value, setting a tap to 1.0" maxlog = PS_MAX_LOG - if haskey(data, "conductors") - error("Multiconductor Not Supported in PowerSystems") - else - branch["tap"] = 1.0 - end - push!(modified, branch["index"]) - else - for c in 1:get(data, "conductors", 1) - cnd_str = haskey(data, "conductors") ? " on conductor $(c)" : "" - if branch["tap"][c] <= 0.0 - @info( - "branch found with non-positive tap value of $(branch["tap"][c]), setting a tap to 1.0$(cnd_str)" - ) - if haskey(data, "conductors") - branch["tap"][c] = 1.0 - else - branch["tap"] = 1.0 - end - push!(modified, branch["index"]) - end - end - end - if !haskey(branch, "shift") - @info("branch found without shift value, setting a shift to 0.0") - if haskey(data, "conductors") - error("Multiconductor Not Supported in PowerSystems") - else - branch["shift"] = 0.0 - end - push!(modified, branch["index"]) - end - end - - return modified -end - -""" -checks that each storage unit has a reasonable parameters -""" -function check_storage_parameters(data::Dict{String, Any}) - if ismultinetwork(data) - error("check_storage_parameters does not yet support multinetwork data") - end - - for (i, strg) in data["storage"] - if strg["energy"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive energy level $(strg["energy"])", - ), - ) - end - if strg["energy_rating"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive energy rating $(strg["energy_rating"])", - ), - ) - end - if strg["charge_rating"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive charge rating $(strg["energy_rating"])", - ), - ) - end - if strg["discharge_rating"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive discharge rating $(strg["energy_rating"])", - ), - ) - end - - if strg["r"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive resistance $(strg["r"])", - ), - ) - end - if strg["x"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive reactance $(strg["x"])", - ), - ) - end - if haskey(strg, "thermal_rating") && strg["thermal_rating"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive thermal rating $(strg["thermal_rating"])", - ), - ) - end - if haskey(strg, "current_rating") && strg["current_rating"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive current rating $(strg["thermal_rating"])", - ), + if haskey(strg, "current_rating") && strg["current_rating"] < 0.0 + throw( + DataFormatError( + "storage unit $(strg["index"]) has a non-positive current rating $(strg["thermal_rating"])", + ), ) end if !isapprox(strg["x"], 0.0; atol = 1e-6, rtol = 1e-6) throw( DataFormatError( - "storage unit $(strg["index"]) has a non-zero reactance $(strg["x"]), which is currently ignored", - ), - ) - end - - if strg["charge_efficiency"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive charge efficiency of $(strg["charge_efficiency"])", - ), - ) - end - if strg["charge_efficiency"] <= 0.0 || strg["charge_efficiency"] > 1.0 - @info "storage unit $(strg["index"]) charge efficiency of $(strg["charge_efficiency"]) is out of the valid range (0.0. 1.0]" maxlog = - PS_MAX_LOG - end - if strg["discharge_efficiency"] < 0.0 - throw( - DataFormatError( - "storage unit $(strg["index"]) has a non-positive discharge efficiency of $(strg["discharge_efficiency"])", - ), - ) - end - if strg["discharge_efficiency"] <= 0.0 || strg["discharge_efficiency"] > 1.0 - @info "storage unit $(strg["index"]) discharge efficiency of $(strg["discharge_efficiency"]) is out of the valid range (0.0. 1.0]" maxlog = - PS_MAX_LOG - end - - if strg["p_loss"] > 0.0 && strg["energy"] <= 0.0 - @info "storage unit $(strg["index"]) has positive active power losses but zero initial energy. This can lead to model infeasiblity." maxlog = - PS_MAX_LOG - end - if strg["q_loss"] > 0.0 && strg["energy"] <= 0.0 - @info "storage unit $(strg["index"]) has positive reactive power losses but zero initial energy. This can lead to model infeasiblity." maxlog = - PS_MAX_LOG - end - end -end - -""" -checks that each switch has a reasonable parameters -""" -function check_switch_parameters(data::Dict{String, <:Any}) - if ismultinetwork(data) - error("check_switch_parameters does not yet support multinetwork data") - end - - for (i, switch) in data["switch"] - if switch["state"] <= 0.0 && - (!isapprox(switch["psw"], 0.0) || !isapprox(switch["qsw"], 0.0)) - @info "switch $(switch["index"]) is open with non-zero power values $(switch["psw"]), $(switch["qsw"])" maxlog = - PS_MAX_LOG - end - if haskey(switch, "thermal_rating") && switch["thermal_rating"] < 0.0 - throw( - DataFormatError( - "switch $(switch["index"]) has a non-positive thermal_rating $(switch["thermal_rating"])", - ), - ) - end - if haskey(switch, "current_rating") && switch["current_rating"] < 0.0 - throw( - DataFormatError( - "switch $(switch["index"]) has a non-positive current_rating $(switch["current_rating"])", - ), - ) - end - end -end - -"checks that parameters for dc lines are reasonable" -function correct_dcline_limits!(data::Dict{String, Any}) - if ismultinetwork(data) - error("check_dcline_limits does not yet support multinetwork data") - end - - @assert("per_unit" in keys(data) && data["per_unit"]) - mva_base = data["baseMVA"] - - modified = Set{Int}() - - for c in 1:get(data, "conductors", 1) - cnd_str = haskey(data, "conductors") ? ", conductor $(c)" : "" - for (i, dcline) in data["dcline"] - if dcline["loss0"][c] < 0.0 - new_rate = 0.0 - @info "this code only supports positive loss0 values, changing the value on dcline $(dcline["index"])$(cnd_str) from $(mva_base*dcline["loss0"][c]) to $(mva_base*new_rate)" maxlog = - PS_MAX_LOG - if haskey(data, "conductors") - dcline["loss0"][c] = new_rate - else - dcline["loss0"] = new_rate - end - push!(modified, dcline["index"]) - end - - if dcline["loss0"][c] >= - dcline["pmaxf"][c] * (1 - dcline["loss1"][c]) + dcline["pmaxt"][c] - new_rate = 0.0 - @info "this code only supports loss0 values which are consistent with the line flow bounds, changing the value on dcline $(dcline["index"])$(cnd_str) from $(mva_base*dcline["loss0"][c]) to $(mva_base*new_rate)" maxlog = - PS_MAX_LOG - if haskey(data, "conductors") - dcline["loss0"][c] = new_rate - else - dcline["loss0"] = new_rate - end - push!(modified, dcline["index"]) - end - - if dcline["loss1"][c] < 0.0 - new_rate = 0.0 - @info "this code only supports positive loss1 values, changing the value on dcline $(dcline["index"])$(cnd_str) from $(dcline["loss1"][c]) to $(new_rate)" maxlog = - PS_MAX_LOG - if haskey(data, "conductors") - dcline["loss1"][c] = new_rate - else - dcline["loss1"] = new_rate - end - push!(modified, dcline["index"]) - end - - if dcline["loss1"][c] >= 1.0 - new_rate = 0.0 - @info "this code only supports loss1 values < 1, changing the value on dcline $(dcline["index"])$(cnd_str) from $(dcline["loss1"][c]) to $(new_rate)" maxlog = - PS_MAX_LOG - if haskey(data, "conductors") - dcline["loss1"][c] = new_rate - else - dcline["loss1"] = new_rate - end - push!(modified, dcline["index"]) - end - - if dcline["pmint"][c] < 0.0 && dcline["loss1"][c] > 0.0 - #new_rate = 0.0 - @info "the dc line model is not meant to be used bi-directionally when loss1 > 0, be careful interpreting the results as the dc line losses can now be negative. change loss1 to 0 to avoid this warning" maxlog = - PS_MAX_LOG - #dcline["loss0"] = new_rate - end - end - end - - return modified -end - -"throws warnings if generator and dc line voltage setpoints are not consistent with the bus voltage setpoint" -function check_voltage_setpoints(data::Dict{String, <:Any}) - if ismultinetwork(data) - error("check_voltage_setpoints does not yet support multinetwork data") - end - - for c in 1:get(data, "conductors", 1) - cnd_str = haskey(data, "conductors") ? "conductor $(c) " : "" - for (i, gen) in data["gen"] - bus_id = gen["gen_bus"] - bus = data["bus"][bus_id] - if gen["vg"][c] != bus["vm"][c] - @info "the $(cnd_str)voltage setpoint on generator $(i) does not match the value at bus $(bus_id)" maxlog = - PS_MAX_LOG - end - end - - for (i, dcline) in data["dcline"] - bus_fr_id = dcline["f_bus"] - bus_to_id = dcline["t_bus"] - - bus_fr = data["bus"][bus_fr_id] - bus_to = data["bus"][bus_to_id] - - if dcline["vf"][c] != bus_fr["vm"][c] - @info( - "the $(cnd_str)from bus voltage setpoint on dc line $(i) does not match the value at bus $(bus_fr_id)" - ) - end - - if dcline["vt"][c] != bus_to["vm"][c] - @info( - "the $(cnd_str)to bus voltage setpoint on dc line $(i) does not match the value at bus $(bus_to_id)" - ) - end - end - end -end - -"throws warnings if cost functions are malformed" -function correct_cost_functions!(data::Dict{String, <:Any}) - if ismultinetwork(data) - error("check_cost_functions does not yet support multinetwork data") - end - - modified_gen = Set{Int}() - for (i, gen) in data["gen"] - if _correct_cost_function!(i, gen, "generator") - push!(modified_gen, gen["index"]) - end - end - - modified_dcline = Set{Int}() - for (i, dcline) in data["dcline"] - if _correct_cost_function!(i, dcline, "dcline") - push!(modified_dcline, dcline["index"]) - end - end - - return (modified_gen, modified_dcline) -end - -"" -function _correct_cost_function!(id, comp, type_name) - modified = false - - if "model" in keys(comp) && "cost" in keys(comp) - if comp["model"] == 1 - if length(comp["cost"]) != 2 * comp["ncost"] - error( - "ncost of $(comp["ncost"]) not consistent with $(length(comp["cost"])) cost values on $(type_name) $(id)", - ) - end - if length(comp["cost"]) < 4 - error( - "cost includes $(comp["ncost"]) points, but at least two points are required on $(type_name) $(id)", - ) - end - - modified = _remove_pwl_cost_duplicates!(id, comp, type_name) - - for i in 3:2:length(comp["cost"]) - if comp["cost"][i - 2] >= comp["cost"][i] - error("non-increasing x values in pwl cost model on $(type_name) $(id)") - end - end - if "pmin" in keys(comp) && "pmax" in keys(comp) - pmin = sum(comp["pmin"]) # sum supports multi-conductor case - pmax = sum(comp["pmax"]) - for i in 3:2:length(comp["cost"]) - if comp["cost"][i] < pmin || comp["cost"][i] > pmax - @info( - "pwl x value $(comp["cost"][i]) is outside the bounds $(pmin)-$(pmax) on $(type_name) $(id)" - ) - end - end - end - modified |= _simplify_pwl_cost!(id, comp, type_name) - elseif comp["model"] == 2 - if length(comp["cost"]) != comp["ncost"] - error( - "ncost of $(comp["ncost"]) not consistent with $(length(comp["cost"])) cost values on $(type_name) $(id)", - ) - end - else - @info "Unknown cost model of type $(comp["model"]) on $(type_name) $(id)" maxlog = - PS_MAX_LOG - end - end - - return modified -end - -"checks that each point in the a pwl function is unique, simplifies the function if duplicates appear" -function _remove_pwl_cost_duplicates!(id, comp, type_name, tolerance = 1e-2) - @assert comp["model"] == 1 - - unique_costs = Float64[comp["cost"][1], comp["cost"][2]] - for i in 3:2:length(comp["cost"]) - x1 = unique_costs[end - 1] - y1 = unique_costs[end] - x2 = comp["cost"][i + 0] - y2 = comp["cost"][i + 1] - if !(isapprox(x1, x2) && isapprox(y1, y2)) - push!(unique_costs, x2) - push!(unique_costs, y2) - end - end - - # in the event that all of the given points are the same - # this code ensures that at least two of the points remain - if length(unique_costs) <= 2 - push!(unique_costs, comp["cost"][end - 1]) - push!(unique_costs, comp["cost"][end]) - end - - if length(unique_costs) < length(comp["cost"]) - @info "removing duplicate points from pwl cost on $(type_name) $(id), $(comp["cost"]) -> $(unique_costs)" maxlog = - PS_MAX_LOG - comp["cost"] = unique_costs - comp["ncost"] = length(unique_costs) / 2 - return true - end - - return false -end - -"checks the slope of each segment in a pwl function, simplifies the function if the slope changes is below a tolerance" -function _simplify_pwl_cost!(id, comp, type_name, tolerance = 1e-2) - @assert comp["model"] == 1 - - slopes = Float64[] - smpl_cost = Float64[] - prev_slope = nothing - - x2, y2 = 0.0, 0.0 - - for i in 3:2:length(comp["cost"]) - x1 = comp["cost"][i - 2] - y1 = comp["cost"][i - 1] - x2 = comp["cost"][i - 0] - y2 = comp["cost"][i + 1] - - m = (y2 - y1) / (x2 - x1) - - if prev_slope === nothing || (abs(prev_slope - m) > tolerance) - push!(smpl_cost, x1) - push!(smpl_cost, y1) - prev_slope = m - end - - push!(slopes, m) - end - - push!(smpl_cost, x2) - push!(smpl_cost, y2) - - if length(smpl_cost) < length(comp["cost"]) - @info "simplifying pwl cost on $(type_name) $(id), $(comp["cost"]) -> $(smpl_cost)" maxlog = - PS_MAX_LOG - comp["cost"] = smpl_cost - comp["ncost"] = length(smpl_cost) / 2 - return true - end - return false -end - -"trims zeros from higher order cost terms" -function simplify_cost_terms!(data::Dict{String, <:Any}) - if ismultinetwork(data) - networks = data["nw"] - else - networks = [("0", data)] - end - - modified_gen = Set{Int}() - modified_dcline = Set{Int}() - - for (i, network) in networks - if haskey(network, "gen") - for (i, gen) in network["gen"] - if haskey(gen, "model") && gen["model"] == 2 - ncost = length(gen["cost"]) - for j in 1:ncost - if gen["cost"][1] == 0.0 - gen["cost"] = gen["cost"][2:end] - else - break - end - end - if length(gen["cost"]) != ncost - gen["ncost"] = length(gen["cost"]) - @info "removing $(ncost - gen["ncost"]) cost terms from generator $(i): $(gen["cost"])" maxlog = - PS_MAX_LOG - push!(modified_gen, gen["index"]) - end - end - end - end - - if haskey(network, "dcline") - for (i, dcline) in network["dcline"] - if haskey(dcline, "model") && dcline["model"] == 2 - ncost = length(dcline["cost"]) - for j in 1:ncost - if dcline["cost"][1] == 0.0 - dcline["cost"] = dcline["cost"][2:end] - else - break - end - end - if length(dcline["cost"]) != ncost - dcline["ncost"] = length(dcline["cost"]) - @info "removing $(ncost - dcline["ncost"]) cost terms from dcline $(i): $(dcline["cost"])" maxlog = - PS_MAX_LOG - push!(modified_dcline, dcline["index"]) - end - end - end - end - end - - return (modified_gen, modified_dcline) -end - -"ensures all polynomial costs functions have the same number of terms" -function standardize_cost_terms!(data::Dict{String, <:Any}; order = -1) - comp_max_order = 1 - - if ismultinetwork(data) - networks = data["nw"] - else - networks = [("0", data)] - end - - for (i, network) in networks - if haskey(network, "gen") - for (i, gen) in network["gen"] - if haskey(gen, "model") && gen["model"] == 2 - max_nonzero_index = 1 - for i in 1:length(gen["cost"]) - max_nonzero_index = i - if gen["cost"][i] != 0.0 - break - end - end - - max_oder = length(gen["cost"]) - max_nonzero_index + 1 - - comp_max_order = max(comp_max_order, max_oder) - end - end - end - - if haskey(network, "dcline") - for (i, dcline) in network["dcline"] - if haskey(dcline, "model") && dcline["model"] == 2 - max_nonzero_index = 1 - for i in 1:length(dcline["cost"]) - max_nonzero_index = i - if dcline["cost"][i] != 0.0 - break - end - end - - max_oder = length(dcline["cost"]) - max_nonzero_index + 1 - - comp_max_order = max(comp_max_order, max_oder) - end - end - end - end - - if comp_max_order <= order + 1 - comp_max_order = order + 1 - else - if order != -1 # if not the default - @info( - "a standard cost order of $(order) was requested but the given data requires an order of at least $(comp_max_order-1)" - ) - end - end - - for (i, network) in networks - if haskey(network, "gen") - _standardize_cost_terms!(network["gen"], comp_max_order, "generator") - end - if haskey(network, "dcline") - _standardize_cost_terms!(network["dcline"], comp_max_order, "dcline") - end - end -end - -"ensures all polynomial costs functions have at exactly comp_order terms" -function _standardize_cost_terms!( - components::Dict{String, <:Any}, - comp_order::Int, - cost_comp_name::String, -) - modified = Set{Int}() - for (i, comp) in components - if haskey(comp, "model") && comp["model"] == 2 && length(comp["cost"]) != comp_order - std_cost = [0.0 for i in 1:comp_order] - current_cost = reverse(comp["cost"]) - #println("gen cost: $(comp["cost"])") - for i in 1:min(comp_order, length(current_cost)) - std_cost[i] = current_cost[i] - end - comp["cost"] = reverse(std_cost) - comp["ncost"] = comp_order - #println("std gen cost: $(comp["cost"])") - - @info "Updated $(cost_comp_name) $(comp["index"]) cost function with order $(length(current_cost)) to a function of order $(comp_order): $(comp["cost"])" maxlog = - PS_MAX_LOG - push!(modified, comp["index"]) - end - end - return modified -end - -""" -finds active network buses and branches that are not necessary for the -computation and sets their status to off. - -Works on a PowerModels data dict, so that a it can be used without a GenericPowerModel object - -Warning: this implementation has quadratic complexity, in the worst case -""" -function propagate_topology_status!(data::Dict{String, <:Any}) - if ismultinetwork(data) - for (i, nw_data) in data["nw"] - _propagate_topology_status!(nw_data) - end - else - _propagate_topology_status!(data) - end -end - -"" -function _propagate_topology_status!(data::Dict{String, <:Any}) - buses = Dict(bus["bus_i"] => bus for (i, bus) in data["bus"]) - - for (i, load) in data["load"] - if load["status"] != 0 && all(load["pd"] .== 0.0) && all(load["qd"] .== 0.0) - @info("deactivating load $(load["index"]) due to zero pd and qd") - load["status"] = 0 + "storage unit $(strg["index"]) has a non-zero reactance $(strg["x"]), which is currently ignored", + ), + ) end - end - for (i, shunt) in data["shunt"] - if shunt["status"] != 0 && all(shunt["gs"] .== 0.0) && all(shunt["bs"] .== 0.0) - @info("deactivating shunt $(shunt["index"]) due to zero gs and bs") - shunt["status"] = 0 + if strg["charge_efficiency"] < 0.0 + throw( + DataFormatError( + "storage unit $(strg["index"]) has a non-positive charge efficiency of $(strg["charge_efficiency"])", + ), + ) end - end - - # compute what active components are incident to each bus - incident_load = bus_load_lookup(data["load"], data["bus"]) - incident_active_load = Dict() - for (i, load_list) in incident_load - incident_active_load[i] = [load for load in load_list if load["status"] != 0] - end - - incident_shunt = bus_shunt_lookup(data["shunt"], data["bus"]) - incident_active_shunt = Dict() - for (i, shunt_list) in incident_shunt - incident_active_shunt[i] = [shunt for shunt in shunt_list if shunt["status"] != 0] - end - - incident_gen = bus_gen_lookup(data["gen"], data["bus"]) - incident_active_gen = Dict() - for (i, gen_list) in incident_gen - incident_active_gen[i] = [gen for gen in gen_list if gen["gen_status"] != 0] - end - - incident_strg = bus_storage_lookup(data["storage"], data["bus"]) - incident_active_strg = Dict() - for (i, strg_list) in incident_strg - incident_active_strg[i] = [strg for strg in strg_list if strg["status"] != 0] - end - - incident_branch = Dict(bus["bus_i"] => [] for (i, bus) in data["bus"]) - for (i, branch) in data["branch"] - push!(incident_branch[branch["f_bus"]], branch) - push!(incident_branch[branch["t_bus"]], branch) - end - - incident_dcline = Dict(bus["bus_i"] => [] for (i, bus) in data["bus"]) - for (i, dcline) in data["dcline"] - push!(incident_dcline[dcline["f_bus"]], dcline) - push!(incident_dcline[dcline["t_bus"]], dcline) - end - - incident_switch = Dict(bus["bus_i"] => [] for (i, bus) in data["bus"]) - for (i, switch) in data["switch"] - push!(incident_switch[switch["f_bus"]], switch) - push!(incident_switch[switch["t_bus"]], switch) - end - - revised = false - - for (i, branch) in data["branch"] - if branch["br_status"] != 0 - f_bus = buses[branch["f_bus"]] - t_bus = buses[branch["t_bus"]] - - if f_bus["bus_type"] == 4 || t_bus["bus_type"] == 4 - @info "deactivating branch $(i):($(branch["f_bus"]),$(branch["t_bus"])) due to connecting bus status" maxlog = - PS_MAX_LOG - branch["br_status"] = 0 - revised = true - end + if strg["charge_efficiency"] <= 0.0 || strg["charge_efficiency"] > 1.0 + @info "storage unit $(strg["index"]) charge efficiency of $(strg["charge_efficiency"]) is out of the valid range (0.0. 1.0]" maxlog = + PS_MAX_LOG end - end - - for (i, dcline) in data["dcline"] - if dcline["br_status"] != 0 - f_bus = buses[dcline["f_bus"]] - t_bus = buses[dcline["t_bus"]] - - if f_bus["bus_type"] == 4 || t_bus["bus_type"] == 4 - @info "deactivating dcline $(i):($(dcline["f_bus"]),$(dcline["t_bus"])) due to connecting bus status" maxlog = - PS_MAX_LOG - dcline["br_status"] = 0 - revised = true - end + if strg["discharge_efficiency"] < 0.0 + throw( + DataFormatError( + "storage unit $(strg["index"]) has a non-positive discharge efficiency of $(strg["discharge_efficiency"])", + ), + ) end - end - - for (i, switch) in data["switch"] - if switch["status"] != 0 - f_bus = buses[switch["f_bus"]] - t_bus = buses[switch["t_bus"]] - - if f_bus["bus_type"] == 4 || t_bus["bus_type"] == 4 - @info "deactivating switch $(i):($(switch["f_bus"]),$(switch["t_bus"])) due to connecting bus status" maxlog = - PS_MAX_LOG - switch["status"] = 0 - revised = true - end + if strg["discharge_efficiency"] <= 0.0 || strg["discharge_efficiency"] > 1.0 + @info "storage unit $(strg["index"]) discharge efficiency of $(strg["discharge_efficiency"]) is out of the valid range (0.0. 1.0]" maxlog = + PS_MAX_LOG end - end - - for (i, bus) in buses - if bus["bus_type"] == 4 - for load in incident_active_load[i] - if load["status"] != 0 - @info "deactivating load $(load["index"]) due to inactive bus $(i)" maxlog = - PS_MAX_LOG - load["status"] = 0 - revised = true - end - end - - for shunt in incident_active_shunt[i] - if shunt["status"] != 0 - @info "deactivating shunt $(shunt["index"]) due to inactive bus $(i)" maxlog = - PS_MAX_LOG - shunt["status"] = 0 - revised = true - end - end - - for gen in incident_active_gen[i] - if gen["gen_status"] != 0 - @info "deactivating generator $(gen["index"]) due to inactive bus $(i)" maxlog = - PS_MAX_LOG - gen["gen_status"] = 0 - revised = true - end - end - for strg in incident_active_strg[i] - if strg["status"] != 0 - @info "deactivating storage $(strg["index"]) due to inactive bus $(i)" maxlog = - PS_MAX_LOG - strg["status"] = 0 - revised = true - end - end + if strg["p_loss"] > 0.0 && strg["energy"] <= 0.0 + @info "storage unit $(strg["index"]) has positive active power losses but zero initial energy. This can lead to model infeasiblity." maxlog = + PS_MAX_LOG + end + if strg["q_loss"] > 0.0 && strg["energy"] <= 0.0 + @info "storage unit $(strg["index"]) has positive reactive power losses but zero initial energy. This can lead to model infeasiblity." maxlog = + PS_MAX_LOG end end - - return revised end """ -removes buses with single branch connections and without any other attached -components. Also removes connected components without suffuceint generation -or loads. - -also deactivates 0 valued loads and shunts. +checks that each switch has a reasonable parameters """ -function deactivate_isolated_components!(data::Dict{String, <:Any}) - revised = false - pm_data = get_pm_data(data) - - if _IM.ismultinetwork(pm_data) - for (i, pm_nw_data) in pm_data["nw"] - revised |= _deactivate_isolated_components!(pm_nw_data) - end - else - revised = _deactivate_isolated_components!(pm_data) +function check_switch_parameters(data::Dict{String, <:Any}) + if ismultinetwork(data) + error("check_switch_parameters does not yet support multinetwork data") end - return revised -end - -"" -function _deactivate_isolated_components!(data::Dict{String, <:Any}) - buses = Dict(bus["bus_i"] => bus for (i, bus) in data["bus"]) - - revised = false - - for (i, load) in data["load"] - if load["status"] != 0 && all(load["pd"] .== 0.0) && all(load["qd"] .== 0.0) - @info "deactivating load $(load["index"]) due to zero pd and qd" maxlog = + for (i, switch) in data["switch"] + if switch["state"] <= 0.0 && + (!isapprox(switch["psw"], 0.0) || !isapprox(switch["qsw"], 0.0)) + @info "switch $(switch["index"]) is open with non-zero power values $(switch["psw"]), $(switch["qsw"])" maxlog = PS_MAX_LOG - load["status"] = 0 - revised = true end - end - - for (i, shunt) in data["shunt"] - if shunt["status"] != 0 && all(shunt["gs"] .== 0.0) && all(shunt["bs"] .== 0.0) - @info "deactivating shunt $(shunt["index"]) due to zero gs and bs" maxlog = - PS_MAX_LOG - shunt["status"] = 0 - revised = true + if haskey(switch, "thermal_rating") && switch["thermal_rating"] < 0.0 + throw( + DataFormatError( + "switch $(switch["index"]) has a non-positive thermal_rating $(switch["thermal_rating"])", + ), + ) + end + if haskey(switch, "current_rating") && switch["current_rating"] < 0.0 + throw( + DataFormatError( + "switch $(switch["index"]) has a non-positive current_rating $(switch["current_rating"])", + ), + ) end end +end - # compute what active components are incident to each bus - incident_load = bus_load_lookup(data["load"], data["bus"]) - incident_active_load = Dict() - for (i, load_list) in incident_load - incident_active_load[i] = [load for load in load_list if load["status"] != 0] - end - - incident_shunt = bus_shunt_lookup(data["shunt"], data["bus"]) - incident_active_shunt = Dict() - for (i, shunt_list) in incident_shunt - incident_active_shunt[i] = [shunt for shunt in shunt_list if shunt["status"] != 0] - end - - incident_gen = bus_gen_lookup(data["gen"], data["bus"]) - incident_active_gen = Dict() - for (i, gen_list) in incident_gen - incident_active_gen[i] = [gen for gen in gen_list if gen["gen_status"] != 0] - end - - incident_strg = bus_storage_lookup(data["storage"], data["bus"]) - incident_active_strg = Dict() - for (i, strg_list) in incident_strg - incident_active_strg[i] = [strg for strg in strg_list if strg["status"] != 0] - end - - incident_branch = Dict(bus["bus_i"] => [] for (i, bus) in data["bus"]) - for (i, branch) in data["branch"] - push!(incident_branch[branch["f_bus"]], branch) - push!(incident_branch[branch["t_bus"]], branch) +"checks that parameters for dc lines are reasonable" +function correct_dcline_limits!(data::Dict{String, Any}) + if ismultinetwork(data) + error("check_dcline_limits does not yet support multinetwork data") end - incident_dcline = Dict(bus["bus_i"] => [] for (i, bus) in data["bus"]) - for (i, dcline) in data["dcline"] - push!(incident_dcline[dcline["f_bus"]], dcline) - push!(incident_dcline[dcline["t_bus"]], dcline) - end + @assert("per_unit" in keys(data) && data["per_unit"]) + mva_base = data["baseMVA"] - incident_switch = Dict(bus["bus_i"] => [] for (i, bus) in data["bus"]) - for (i, switch) in data["switch"] - push!(incident_switch[switch["f_bus"]], switch) - push!(incident_switch[switch["t_bus"]], switch) - end - - changed = true - while changed - changed = false - - for (i, bus) in buses - if bus["bus_type"] != 4 - incident_active_edge = 0 - if length(incident_branch[i]) + - length(incident_dcline[i]) + - length(incident_switch[i]) > 0 - incident_branch_count = - sum([0; [branch["br_status"] for branch in incident_branch[i]]]) - incident_dcline_count = - sum([0; [dcline["br_status"] for dcline in incident_dcline[i]]]) - incident_switch_count = - sum([0; [switch["status"] for switch in incident_switch[i]]]) - incident_active_edge = - incident_branch_count + - incident_dcline_count + - incident_switch_count - end + modified = Set{Int}() - if incident_active_edge == 1 && - length(incident_active_gen[i]) == 0 && - length(incident_active_load[i]) == 0 && - length(incident_active_shunt[i]) == 0 && - length(incident_active_strg[i]) == 0 - @info "deactivating bus $(i) due to dangling bus without generation, load, or storage" maxlog = - PS_MAX_LOG - bus["bus_type"] = 4 - revised = true - changed = true + for c in 1:get(data, "conductors", 1) + cnd_str = haskey(data, "conductors") ? ", conductor $(c)" : "" + for (i, dcline) in data["dcline"] + if dcline["loss0"][c] < 0.0 + new_rate = 0.0 + @info "this code only supports positive loss0 values, changing the value on dcline $(dcline["index"])$(cnd_str) from $(mva_base*dcline["loss0"][c]) to $(mva_base*new_rate)" maxlog = + PS_MAX_LOG + if haskey(data, "conductors") + dcline["loss0"][c] = new_rate + else + dcline["loss0"] = new_rate end + push!(modified, dcline["index"]) end - end - if changed - for (i, branch) in data["branch"] - if branch["br_status"] != 0 - f_bus = buses[branch["f_bus"]] - t_bus = buses[branch["t_bus"]] - - if f_bus["bus_type"] == 4 || t_bus["bus_type"] == 4 - @info "deactivating branch $(i):($(branch["f_bus"]),$(branch["t_bus"])) due to connecting bus status" maxlog = - PS_MAX_LOG - branch["br_status"] = 0 - end + if dcline["loss0"][c] >= + dcline["pmaxf"][c] * (1 - dcline["loss1"][c]) + dcline["pmaxt"][c] + new_rate = 0.0 + @info "this code only supports loss0 values which are consistent with the line flow bounds, changing the value on dcline $(dcline["index"])$(cnd_str) from $(mva_base*dcline["loss0"][c]) to $(mva_base*new_rate)" maxlog = + PS_MAX_LOG + if haskey(data, "conductors") + dcline["loss0"][c] = new_rate + else + dcline["loss0"] = new_rate end + push!(modified, dcline["index"]) end - for (i, dcline) in data["dcline"] - if dcline["br_status"] != 0 - f_bus = buses[dcline["f_bus"]] - t_bus = buses[dcline["t_bus"]] - - if f_bus["bus_type"] == 4 || t_bus["bus_type"] == 4 - @info "deactivating dcline $(i):($(dcline["f_bus"]),$(dcline["t_bus"])) due to connecting bus status" maxlog = - PS_MAX_LOG - dcline["br_status"] = 0 - end + if dcline["loss1"][c] < 0.0 + new_rate = 0.0 + @info "this code only supports positive loss1 values, changing the value on dcline $(dcline["index"])$(cnd_str) from $(dcline["loss1"][c]) to $(new_rate)" maxlog = + PS_MAX_LOG + if haskey(data, "conductors") + dcline["loss1"][c] = new_rate + else + dcline["loss1"] = new_rate end + push!(modified, dcline["index"]) end - for (i, switch) in data["switch"] - if switch["status"] != 0 - f_bus = buses[switch["f_bus"]] - t_bus = buses[switch["t_bus"]] - - if f_bus["bus_type"] == 4 || t_bus["bus_type"] == 4 - @info "deactivating switch $(i):($(switch["f_bus"]),$(switch["t_bus"])) due to connecting bus status" maxlog = - PS_MAX_LOG - switch["status"] = 0 - end + if dcline["loss1"][c] >= 1.0 + new_rate = 0.0 + @info "this code only supports loss1 values < 1, changing the value on dcline $(dcline["index"])$(cnd_str) from $(dcline["loss1"][c]) to $(new_rate)" maxlog = + PS_MAX_LOG + if haskey(data, "conductors") + dcline["loss1"][c] = new_rate + else + dcline["loss1"] = new_rate end + push!(modified, dcline["index"]) end - end - end - - ccs = calc_connected_components(data) - - for cc in ccs - cc_active_loads = [0] - cc_active_shunts = [0] - cc_active_gens = [0] - cc_active_strg = [0] - - for i in cc - cc_active_loads = push!(cc_active_loads, length(incident_active_load[i])) - cc_active_shunts = push!(cc_active_shunts, length(incident_active_shunt[i])) - cc_active_gens = push!(cc_active_gens, length(incident_active_gen[i])) - end - - active_load_count = sum(cc_active_loads) - active_shunt_count = sum(cc_active_shunts) - active_gen_count = sum(cc_active_gens) - - if (active_load_count == 0 && active_shunt_count == 0 && active_strg_count == 0) || - active_gen_count == 0 - @info "deactivating connected component $(cc) due to isolation without generation and load" maxlog = - PS_MAX_LOG - for i in cc - buses[i]["bus_type"] = 4 - end - revised = true - end - end - - return revised -end - -""" -attempts to deactive components that are not needed in the network by repeated -calls to `propagate_topology_status!` and `deactivate_isolated_components!` - -warning: this implementation has quadratic complexity, in the worst case -""" -function simplify_network!(data::Dict{String, <:Any}) - revised = true - iteration = 0 - - while revised - iteration += 1 - revised = false - revised |= propagate_topology_status!(data) - revised |= deactivate_isolated_components!(data) - end - - @info "network simplification fixpoint reached in $(iteration) rounds" maxlog = - PS_MAX_LOG - return revised -end - -""" -determines the largest connected component of the network and turns everything else off -""" -function select_largest_component(data::Dict{String, Any}) - if ismultinetwork(data) - for (i, nw_data) in data["nw"] - _select_largest_component(nw_data) - end - else - _select_largest_component(data) - end -end -"" -function _select_largest_component!(data::Dict{String, <:Any}) - ccs = calc_connected_components(data) - @info "found $(length(ccs)) components" maxlog = PS_MAX_LOG - - if length(ccs) > 1 - ccs_order = sort(collect(ccs); by = length) - largest_cc = ccs_order[end] - - @info "largest component has $(length(largest_cc)) buses" maxlog = PS_MAX_LOG - - for (i, bus) in data["bus"] - if bus["bus_type"] != 4 && !(bus["index"] in largest_cc) - bus["bus_type"] = 4 - @info "deactivating bus $(i) due to small connected component" maxlog = + if dcline["pmint"][c] < 0.0 && dcline["loss1"][c] > 0.0 + #new_rate = 0.0 + @info "the dc line model is not meant to be used bi-directionally when loss1 > 0, be careful interpreting the results as the dc line losses can now be negative. change loss1 to 0 to avoid this warning" maxlog = PS_MAX_LOG + #dcline["loss0"] = new_rate end end - - correct_reference_buses!(data) end -end -""" -checks that each connected components has a reference bus, if not, adds one -""" -function check_reference_buses(data::Dict{String, Any}) - if ismultinetwork(data) - for (i, nw_data) in data["nw"] - _correct_reference_buses!(nw_data) - end - else - _correct_reference_buses!(data) - end + return modified end -"" -function _correct_reference_buses!(data::Dict{String, <:Any}) - bus_lookup = Dict(bus["bus_i"] => bus for (i, bus) in data["bus"]) - bus_gen = bus_gen_lookup(data["gen"], data["bus"]) - - ccs = calc_connected_components(data) - ccs_order = sort(collect(ccs); by = length) - - bus_to_cc = Dict() - for (i, cc) in enumerate(ccs_order) - for bus_i in cc - bus_to_cc[bus_i] = i - end - end - - cc_gens = Dict(i => Dict() for (i, cc) in enumerate(ccs_order)) - for (i, gen) in data["gen"] - bus_id = gen["gen_bus"] - if haskey(bus_to_cc, bus_id) - cc_id = bus_to_cc[bus_id] - cc_gens[cc_id][i] = gen - end - end - - for (i, cc) in enumerate(ccs_order) - correct_component_refrence_bus!(cc, bus_lookup, cc_gens[i]) +"throws warnings if generator and dc line voltage setpoints are not consistent with the bus voltage setpoint" +function check_voltage_setpoints(data::Dict{String, <:Any}) + if ismultinetwork(data) + error("check_voltage_setpoints does not yet support multinetwork data") end -end -""" -checks that a connected component has a reference bus, if not, tries to add one -""" -function correct_component_refrence_bus!(component_bus_ids, bus_lookup, component_gens) - refrence_buses = Set() - for bus_id in component_bus_ids - bus = bus_lookup[bus_id] - if bus["bus_type"] == 3 - push!(refrence_buses, bus_id) + for c in 1:get(data, "conductors", 1) + cnd_str = haskey(data, "conductors") ? "conductor $(c) " : "" + for (i, gen) in data["gen"] + bus_id = gen["gen_bus"] + bus = data["bus"][bus_id] + if gen["vg"][c] != bus["vm"][c] + @info "the $(cnd_str)voltage setpoint on generator $(i) does not match the value at bus $(bus_id)" maxlog = + PS_MAX_LOG + end end - end - if length(refrence_buses) == 0 - @info("no reference bus found in connected component $(component_bus_ids)") - component_gens_active = - Dict(k => v for (k, v) in component_gens if v["gen_status"] != 0) + for (i, dcline) in data["dcline"] + bus_fr_id = dcline["f_bus"] + bus_to_id = dcline["t_bus"] - if length(component_gens_active) > 0 - big_gen = _biggest_generator(component_gens_active) - gen_bus = bus_lookup[big_gen["gen_bus"]] - gen_bus["bus_type"] = 3 - @info( - "setting bus $(gen_bus["index"]) as reference bus in connected component $(component_bus_ids), based on generator $(big_gen["index"])" - ) - else - @info( - "no generators found in connected component $(component_bus_ids), try running propagate_topology_status" - ) - end - end -end + bus_fr = data["bus"][bus_fr_id] + bus_to = data["bus"][bus_to_id] -"builds a lookup list of what generators are connected to a given bus" -function bus_gen_lookup(gen_data::Dict{String, <:Any}, bus_data::Dict{String, <:Any}) - bus_gen = Dict(bus["bus_i"] => [] for (i, bus) in bus_data) - for (i, gen) in gen_data - push!(bus_gen[gen["gen_bus"]], gen) - end - return bus_gen -end + if dcline["vf"][c] != bus_fr["vm"][c] + @info( + "the $(cnd_str)from bus voltage setpoint on dc line $(i) does not match the value at bus $(bus_fr_id)" + ) + end -"builds a lookup list of what loads are connected to a given bus" -function bus_load_lookup(load_data::Dict{String, <:Any}, bus_data::Dict{String, <:Any}) - bus_load = Dict(bus["bus_i"] => [] for (i, bus) in bus_data) - for (i, load) in load_data - push!(bus_load[load["load_bus"]], load) + if dcline["vt"][c] != bus_to["vm"][c] + @info( + "the $(cnd_str)to bus voltage setpoint on dc line $(i) does not match the value at bus $(bus_to_id)" + ) + end + end end - return bus_load end -"builds a lookup list of what shunts are connected to a given bus" -function bus_shunt_lookup(shunt_data::Dict{String, <:Any}, bus_data::Dict{String, <:Any}) - bus_shunt = Dict(bus["bus_i"] => [] for (i, bus) in bus_data) - for (i, shunt) in shunt_data - push!(bus_shunt[shunt["shunt_bus"]], shunt) +"throws warnings if cost functions are malformed" +function correct_cost_functions!(data::Dict{String, <:Any}) + if ismultinetwork(data) + error("check_cost_functions does not yet support multinetwork data") end - return bus_shunt -end -"builds a lookup list of what storage is connected to a given bus" -function bus_storage_lookup( - storage_data::Dict{String, <:Any}, - bus_data::Dict{String, <:Any}, -) - bus_storage = Dict(bus["bus_i"] => [] for (i, bus) in bus_data) - for (i, storage) in storage_data - push!(bus_storage[storage["storage_bus"]], storage) + modified_gen = Set{Int}() + for (i, gen) in data["gen"] + if _correct_cost_function!(i, gen, "generator") + push!(modified_gen, gen["index"]) + end end - return bus_storage -end -""" -computes the connected components of the network graph -returns a set of sets of bus ids, each set is a connected component -""" -function calc_connected_components( - pm_data::Dict{String, <:Any}; - edges = ["branch", "dcline", "switch"], -) - if ismultinetwork(pm_data) - error("connected_components does not yet support multinetwork data") - end - - active_bus = Dict(x for x in pm_data["bus"] if x.second["bus_type"] != 4) - active_bus_ids = Set{Int64}([bus["bus_i"] for (i, bus) in active_bus]) - - neighbors = Dict(i => Int[] for i in active_bus_ids) - for comp_type in edges - status_key = get(pm_component_status, comp_type, "status") - status_inactive = get(pm_component_status_inactive, comp_type, 0) - for edge in values(get(pm_data, comp_type, Dict())) - if get(edge, status_key, 1) != status_inactive && - edge["f_bus"] in active_bus_ids && - edge["t_bus"] in active_bus_ids - push!(neighbors[edge["f_bus"]], edge["t_bus"]) - push!(neighbors[edge["t_bus"]], edge["f_bus"]) - end + modified_dcline = Set{Int}() + for (i, dcline) in data["dcline"] + if _correct_cost_function!(i, dcline, "dcline") + push!(modified_dcline, dcline["index"]) end end - component_lookup = Dict(i => Set{Int}([i]) for i in active_bus_ids) - touched = Set{Int64}() + return (modified_gen, modified_dcline) +end - for i in active_bus_ids - if !(i in touched) - _cc_dfs(i, neighbors, component_lookup, touched) - end - end +"" +function _correct_cost_function!(id, comp, type_name) + modified = false - ccs = (Set(values(component_lookup))) + if "model" in keys(comp) && "cost" in keys(comp) + if comp["model"] == 1 + if length(comp["cost"]) != 2 * comp["ncost"] + error( + "ncost of $(comp["ncost"]) not consistent with $(length(comp["cost"])) cost values on $(type_name) $(id)", + ) + end + if length(comp["cost"]) < 4 + error( + "cost includes $(comp["ncost"]) points, but at least two points are required on $(type_name) $(id)", + ) + end - return ccs -end + modified = _remove_pwl_cost_duplicates!(id, comp, type_name) -""" -DFS on a graph -""" -function _cc_dfs(i, neighbors, component_lookup, touched) - push!(touched, i) - for j in neighbors[i] - if !(j in touched) - for k in component_lookup[j] - push!(component_lookup[i], k) + for i in 3:2:length(comp["cost"]) + if comp["cost"][i - 2] >= comp["cost"][i] + error("non-increasing x values in pwl cost model on $(type_name) $(id)") + end + end + if "pmin" in keys(comp) && "pmax" in keys(comp) + pmin = sum(comp["pmin"]) # sum supports multi-conductor case + pmax = sum(comp["pmax"]) + for i in 3:2:length(comp["cost"]) + if comp["cost"][i] < pmin || comp["cost"][i] > pmax + @info( + "pwl x value $(comp["cost"][i]) is outside the bounds $(pmin)-$(pmax) on $(type_name) $(id)" + ) + end + end end - for k in component_lookup[j] - component_lookup[k] = component_lookup[i] + modified |= _simplify_pwl_cost!(id, comp, type_name) + elseif comp["model"] == 2 + if length(comp["cost"]) != comp["ncost"] + error( + "ncost of $(comp["ncost"]) not consistent with $(length(comp["cost"])) cost values on $(type_name) $(id)", + ) end - _cc_dfs(j, neighbors, component_lookup, touched) + else + @info "Unknown cost model of type $(comp["model"]) on $(type_name) $(id)" maxlog = + PS_MAX_LOG end end + + return modified end -""" -given a network data dict and a mapping of current-bus-ids to new-bus-ids -modifies the data dict to reflect the proposed new bus ids. -""" -function update_bus_ids!( - data::Dict{String, <:Any}, - bus_id_map::Dict{Int, Int}; - injective = true, -) - data_it = ismultiinfrastructure(data) ? data["it"][pm_it_name] : data +"checks that each point in the a pwl function is unique, simplifies the function if duplicates appear" +function _remove_pwl_cost_duplicates!(id, comp, type_name, tolerance = 1e-2) + @assert comp["model"] == 1 - if _IM.ismultinetwork(data_it) && apply_to_subnetworks - for (nw, nw_data) in data_it["nw"] - _update_bus_ids!(nw_data, bus_id_map; injective = injective) + unique_costs = Float64[comp["cost"][1], comp["cost"][2]] + for i in 3:2:length(comp["cost"]) + x1 = unique_costs[end - 1] + y1 = unique_costs[end] + x2 = comp["cost"][i + 0] + y2 = comp["cost"][i + 1] + if !(isapprox(x1, x2) && isapprox(y1, y2)) + push!(unique_costs, x2) + push!(unique_costs, y2) end - else - _update_bus_ids!(data_it, bus_id_map; injective = injective) end -end -function _update_bus_ids!( - data::Dict{String, <:Any}, - bus_id_map::Dict{Int, Int}; - injective = true, -) - # verify bus id map is injective - if injective - new_bus_ids = Set{Int}() - for (i, bus) in data["bus"] - new_id = get(bus_id_map, bus["index"], bus["index"]) - if !(new_id in new_bus_ids) - push!(new_bus_ids, new_id) - else - throw( - error( - "bus id mapping given to update_bus_ids has an id clash on new bus id $(new_id)", - ), - ) - end - end + # in the event that all of the given points are the same + # this code ensures that at least two of the points remain + if length(unique_costs) <= 2 + push!(unique_costs, comp["cost"][end - 1]) + push!(unique_costs, comp["cost"][end]) end - # start renumbering process - renumbered_bus_dict = Dict{String, Any}() - - for (i, bus) in data["bus"] - new_id = get(bus_id_map, bus["index"], bus["index"]) - bus["index"] = new_id - bus["bus_i"] = new_id - renumbered_bus_dict["$new_id"] = bus + if length(unique_costs) < length(comp["cost"]) + @info "removing duplicate points from pwl cost on $(type_name) $(id), $(comp["cost"]) -> $(unique_costs)" maxlog = + PS_MAX_LOG + comp["cost"] = unique_costs + comp["ncost"] = length(unique_costs) / 2 + return true end - data["bus"] = renumbered_bus_dict + return false +end - # update bus numbering in dependent components - for (i, load) in data["load"] - load["load_bus"] = get(bus_id_map, load["load_bus"], load["load_bus"]) - end +"checks the slope of each segment in a pwl function, simplifies the function if the slope changes is below a tolerance" +function _simplify_pwl_cost!(id, comp, type_name, tolerance = 1e-2) + @assert comp["model"] == 1 - for (i, shunt) in data["shunt"] - shunt["shunt_bus"] = get(bus_id_map, shunt["shunt_bus"], shunt["shunt_bus"]) - end + slopes = Float64[] + smpl_cost = Float64[] + prev_slope = nothing - for (i, gen) in data["gen"] - gen["gen_bus"] = get(bus_id_map, gen["gen_bus"], gen["gen_bus"]) - end + x2, y2 = 0.0, 0.0 - for (i, strg) in data["storage"] - strg["storage_bus"] = get(bus_id_map, strg["storage_bus"], strg["storage_bus"]) - end + for i in 3:2:length(comp["cost"]) + x1 = comp["cost"][i - 2] + y1 = comp["cost"][i - 1] + x2 = comp["cost"][i - 0] + y2 = comp["cost"][i + 1] - for (i, switch) in data["switch"] - switch["f_bus"] = get(bus_id_map, switch["f_bus"], switch["f_bus"]) - switch["t_bus"] = get(bus_id_map, switch["t_bus"], switch["t_bus"]) - end + m = (y2 - y1) / (x2 - x1) - branches = [] - if haskey(data, "branch") - append!(branches, values(data["branch"])) - end + if prev_slope === nothing || (abs(prev_slope - m) > tolerance) + push!(smpl_cost, x1) + push!(smpl_cost, y1) + prev_slope = m + end - if haskey(data, "ne_branch") - append!(branches, values(data["ne_branch"])) + push!(slopes, m) end - for branch in branches - branch["f_bus"] = get(bus_id_map, branch["f_bus"], branch["f_bus"]) - branch["t_bus"] = get(bus_id_map, branch["t_bus"], branch["t_bus"]) - end + push!(smpl_cost, x2) + push!(smpl_cost, y2) - for (i, dcline) in data["dcline"] - dcline["f_bus"] = get(bus_id_map, dcline["f_bus"], dcline["f_bus"]) - dcline["t_bus"] = get(bus_id_map, dcline["t_bus"], dcline["t_bus"]) + if length(smpl_cost) < length(comp["cost"]) + @info "simplifying pwl cost on $(type_name) $(id), $(comp["cost"]) -> $(smpl_cost)" maxlog = + PS_MAX_LOG + comp["cost"] = smpl_cost + comp["ncost"] = length(smpl_cost) / 2 + return true end + return false end -""" -given a network data dict merges buses that are connected by closed switches -converting the dataset into a pure bus-branch model. -""" -function resolve_swithces!(data::Dict{String, <:Any}) +"trims zeros from higher order cost terms" +function simplify_cost_terms!(data::Dict{String, <:Any}) if ismultinetwork(data) - for (i, nw_data) in data["nw"] - _resolve_swithces!(nw_data, mva_base) - end + networks = data["nw"] else - _resolve_swithces!(data, mva_base) - end -end - -"" -function _resolve_swithces!(data::Dict{String, <:Any}) - if length(data["switch"]) <= 0 - return + networks = [("0", data)] end - bus_sets = Dict{Int, Set{Int}}() - - switch_status_key = pm_component_status["switch"] - switch_status_value = pm_component_status_inactive["switch"] + modified_gen = Set{Int}() + modified_dcline = Set{Int}() - for (i, switch) in data["switch"] - if switch[switch_status_key] != switch_status_value && switch["state"] == 1 - if !haskey(bus_sets, switch["f_bus"]) - bus_sets[switch["f_bus"]] = Set{Int}([switch["f_bus"]]) - end - if !haskey(bus_sets, switch["t_bus"]) - bus_sets[switch["t_bus"]] = Set{Int}([switch["t_bus"]]) + for (i, network) in networks + if haskey(network, "gen") + for (i, gen) in network["gen"] + if haskey(gen, "model") && gen["model"] == 2 + ncost = length(gen["cost"]) + for j in 1:ncost + if gen["cost"][1] == 0.0 + gen["cost"] = gen["cost"][2:end] + else + break + end + end + if length(gen["cost"]) != ncost + gen["ncost"] = length(gen["cost"]) + @info "removing $(ncost - gen["ncost"]) cost terms from generator $(i): $(gen["cost"])" maxlog = + PS_MAX_LOG + push!(modified_gen, gen["index"]) + end + end end - - merged_set = - Set{Int}([bus_sets[switch["f_bus"]]..., bus_sets[switch["t_bus"]]...]) - bus_sets[switch["f_bus"]] = merged_set - bus_sets[switch["t_bus"]] = merged_set end - end - bus_id_map = Dict{Int, Int}() - for bus_set in Set(values(bus_sets)) - bus_min = minimum(bus_set) - @info "merged buses $(join(bus_set, ",")) in to bus $(bus_min) based on switch status" maxlog = - PS_MAX_LOG - for i in bus_set - if i != bus_min - bus_id_map[i] = bus_min + if haskey(network, "dcline") + for (i, dcline) in network["dcline"] + if haskey(dcline, "model") && dcline["model"] == 2 + ncost = length(dcline["cost"]) + for j in 1:ncost + if dcline["cost"][1] == 0.0 + dcline["cost"] = dcline["cost"][2:end] + else + break + end + end + if length(dcline["cost"]) != ncost + dcline["ncost"] = length(dcline["cost"]) + @info "removing $(ncost - dcline["ncost"]) cost terms from dcline $(i): $(dcline["cost"])" maxlog = + PS_MAX_LOG + push!(modified_dcline, dcline["index"]) + end + end end end end - update_bus_ids!(data, bus_id_map; injective = false) - - for (i, branch) in data["branch"] - if branch["f_bus"] == branch["t_bus"] - @warn "switch removal resulted in both sides of branch $(i) connect to bus $(branch["f_bus"]), deactivating branch" - branch[pm_component_status["branch"]] = pm_component_status_inactive["branch"] - end - end - - for (i, dcline) in data["dcline"] - if dcline["f_bus"] == dcline["t_bus"] - @warn "switch removal resulted in both sides of dcline $(i) connect to bus $(branch["f_bus"]), deactivating dcline" - branch[pm_component_status["dcline"]] = pm_component_status_inactive["dcline"] - end - end - - @info "removed $(length(data["switch"])) switch components" - data["switch"] = Dict{String, Any}() - return + return (modified_gen, modified_dcline) end """ diff --git a/src/pm_io/psse.jl b/src/pm_io/psse.jl index 4253721..01cb114 100644 --- a/src/pm_io/psse.jl +++ b/src/pm_io/psse.jl @@ -604,8 +604,19 @@ function _psse2pm_load!(pm_data::Dict, pti_data::Dict, import_all::Bool) for load in pti_data["LOAD"] sub_data = Dict{String, Any}() sub_data["load_bus"] = pop!(load, "I") - sub_data["pd"] = pop!(load, "PL") - sub_data["qd"] = pop!(load, "QL") + dgenp = 0.0 + dgenq = 0.0 + dgenm = 0.0 + if pm_data["source_version"] == "35" + dgenp = pop!(load, "DGENP", 0.0) + dgenq = pop!(load, "DGENQ", 0.0) + dgenm = pop!(load, "DGENM", 0.0) + end + + # PSS(R)E models distributed generation as negative demand on the load record. + # Net active/reactive demand seen by PF should be gross load minus DGEN. + sub_data["pd"] = pop!(load, "PL") - dgenp + sub_data["qd"] = pop!(load, "QL") - dgenq sub_data["pi"] = pop!(load, "IP") sub_data["qi"] = pop!(load, "IQ") sub_data["py"] = pop!(load, "YP") @@ -622,6 +633,9 @@ function _psse2pm_load!(pm_data::Dict, pti_data::Dict, import_all::Bool) sub_data["ext"]["LOADTYPE"] = "" elseif pm_data["source_version"] == "35" sub_data["ext"]["LOADTYPE"] = pop!(load, "LOADTYPE", "") + sub_data["ext"]["DGENP"] = dgenp + sub_data["ext"]["DGENQ"] = dgenq + sub_data["ext"]["DGENM"] = dgenm else error("Unsupported PSS(R)E source version: $(pm_data["source_version"])") end @@ -710,9 +724,9 @@ function _psse2pm_shunt!(pm_data::Dict, pti_data::Dict, import_all::Bool) sort(collect(keys(step_numbers)); by = x -> parse(Int, x[2:end])) sub_data["step_number"] = [step_numbers[k] for k in step_numbers_sorted] sub_data["step_number"] = sub_data["step_number"][sub_data["step_number"] .!= 0] - + modsw = switched_shunt["MODSW"] sub_data["ext"] = Dict{String, Any}( - "MODSW" => switched_shunt["MODSW"], + "MODSW" => modsw, "ADJM" => switched_shunt["ADJM"], "RMPCT" => switched_shunt["RMPCT"], "RMIDNT" => switched_shunt["RMIDNT"], @@ -749,6 +763,13 @@ function _psse2pm_shunt!(pm_data::Dict, pti_data::Dict, import_all::Bool) error("Unsupported PSS(R)E source version: $(pm_data["source_version"])") end + if modsw ∈ (0, 1, 2) + # For fixed/discrete/continuous modes used for PF comparison, + # BINIT is treated as the total shunt admittance. + # Keep Y_increase but zero all initial states to avoid double counting. + sub_data["initial_status"] = zeros(Int, length(sub_data["y_increment"])) + end + sub_data["index"] = length(pm_data["switched_shunt"]) + 1 sub_data["source_id"] = ["switched shunt", sub_data["shunt_bus"], sub_data["index"]] @@ -1541,7 +1562,7 @@ function _psse2pm_transformer!(pm_data::Dict, pti_data::Dict, import_all::Bool) ) for prefix in TRANSFORMER3W_PARAMETER_NAMES - for i in 1:length(WINDING_NAMES) + for i in 1:length(WINDING_NAMES_PARSING) key = "$prefix$i" if pm_data["source_version"] ∈ ("30", "32", "33") sub_data["ext"][key] = transformer[key] @@ -1676,12 +1697,14 @@ function _psse2pm_dcline!(pm_data::Dict, pti_data::Dict, import_all::Bool) sub_data["rectifier_transformer_ratio"] = dcline["TRR"] sub_data["rectifier_tap_setting"] = dcline["TAPR"] - sub_data["rectifier_tap_limits"] = (min = dcline["TMNR"], max = dcline["TMXR"]) + sub_data["rectifier_tap_limits"] = + Dict{String, Any}("min" => dcline["TMNR"], "max" => dcline["TMXR"]) sub_data["rectifier_tap_step"] = dcline["STPR"] sub_data["inverter_transformer_ratio"] = dcline["TRI"] sub_data["inverter_tap_setting"] = dcline["TAPI"] - sub_data["inverter_tap_limits"] = (min = dcline["TMNI"], max = dcline["TMXI"]) + sub_data["inverter_tap_limits"] = + Dict{String, Any}("min" => dcline["TMNI"], "max" => dcline["TMXI"]) sub_data["inverter_tap_step"] = dcline["STPI"] sub_data["loss0"] = 0.0 @@ -1709,10 +1732,14 @@ function _psse2pm_dcline!(pm_data::Dict, pti_data::Dict, import_all::Bool) @info("$key outside reasonable limits, setting to 0 degress") end end - sub_data["rectifier_delay_angle_limits"] = - (min = deg2rad(anmn[1]), max = deg2rad(dcline["ANMXR"])) - sub_data["inverter_extinction_angle_limits"] = - (min = deg2rad(anmn[2]), max = deg2rad(dcline["ANMXI"])) + sub_data["rectifier_delay_angle_limits"] = Dict{String, Any}( + "min" => deg2rad(anmn[1]), + "max" => deg2rad(dcline["ANMXR"]), + ) + sub_data["inverter_extinction_angle_limits"] = Dict{String, Any}( + "min" => deg2rad(anmn[2]), + "max" => deg2rad(dcline["ANMXI"]), + ) sub_data["rectifier_delay_angle"] = deg2rad(anmn[1]) sub_data["inverter_extinction_angle"] = deg2rad(anmn[2]) @@ -1813,14 +1840,15 @@ function _psse2pm_dcline!(pm_data::Dict, pti_data::Dict, import_all::Bool) sub_data["ac_setpoint_to"] = to_bus["ACSET"] # ALOSS, MINLOSS in kW, and BLOSS in kW/A. Divide by a 1000 to transform into MW, and divide by baseMVA to normalize to per-unit. - sub_data["converter_loss_from"] = LinearCurve( + sub_data["converter_loss_from"] = IS.LinearCurve( from_bus["BLOSS"] / (1000.0 * baseMVA), (from_bus["ALOSS"] + from_bus["MINLOSS"]) / (1000.0 * baseMVA), ) - sub_data["converter_loss_to"] = LinearCurve( + sub_data["converter_loss_to"] = IS.LinearCurve( to_bus["BLOSS"] / (1000.0 * baseMVA), (to_bus["ALOSS"] + to_bus["MINLOSS"]) / (1000.0 * baseMVA), ) + # since IS is an allowed dep in this repo, using IS.LinearCurve sub_data["pf"] = 0.0 sub_data["pt"] = 0.0 diff --git a/src/power_models_data.jl b/src/power_models_data.jl index 0eb1e3d..aa75ab7 100644 --- a/src/power_models_data.jl +++ b/src/power_models_data.jl @@ -1,16 +1,5 @@ """ Container for data parsed by PowerModels. - -# Fields -- `data::Dict{String, Any}`: Dictionary containing the parsed power system data - -# Example -```julia -pm_data = PowerModelsData("case5.m") -# Access the data dictionary -baseMVA = pm_data.data["baseMVA"] -buses = pm_data.data["bus"] -``` """ struct PowerModelsData data::Dict{String, Any} @@ -19,20 +8,6 @@ end """ Constructs PowerModelsData from a raw file. Currently Supports MATPOWER and PSSE data files parsed by PowerModels. - -# Arguments -- `file::Union{String, IO}`: Path to the file or IO stream to parse - -# Keyword Arguments -- `pm_data_corrections::Bool=true`: Run PowerModels data corrections (validation) -- `import_all::Bool=false`: Import all fields from PTI files -- `correct_branch_rating::Bool=true`: Correct branch ratings during parsing - -# Example -```julia -pm_data = PowerModelsData("case5.m") -pm_data = PowerModelsData("system.raw"; import_all=true) -``` """ function PowerModelsData(file::Union{String, IO}; kwargs...) validate = get(kwargs, :pm_data_corrections, true) @@ -50,14 +25,727 @@ function PowerModelsData(file::Union{String, IO}; kwargs...) end """ -Corrects transformer status in PowerModelsData based on bus voltage differences. +Container holding every output collection produced by a single +[`parse_to_openapi_dicts`] run. + +# Fields — homogeneous reader outputs + +- `bus_dicts::Dict{Int, Dict{String, Any}}` — keyed by bus number +- `area_dicts::Dict{Int, Dict{String, Any}}` — keyed by area number +- `area_interchange_dicts::Dict{String, Dict{String, Any}}` — keyed by name +- `loadzone_dicts::Dict{Int, Dict{String, Any}}` — keyed by zone number +- `switched_shunt_dicts::Dict{String, Dict{String, Any}}` — keyed by name +- `shunt_dicts::Dict{String, Dict{String, Any}}` — keyed by name +- `storage_dicts::Dict{String, Dict{String, Any}}` — keyed by name +- `switch_dicts::Dict{String, Dict{String, Any}}` — from `data["switch"]` +- `breaker_dicts::Dict{String, Dict{String, Any}}` — from `data["breaker"]` +- `vscline_dicts::Dict{String, Dict{String, Any}}` — keyed by name +- `facts_dicts::Dict{String, Dict{String, Any}}` — keyed by `"_"` +- `ict_instances::Dict{Tuple{Int, String}, Dict{String, Any}}` — keyed by + `(table_number, transformer_winding)` + +# Fields — heterogeneous reader outputs (NamedTuples per OpenAPI type) + +- `loads` — `(; power_load, standard_load, interruptible_standard_load)` +- `gens` — `(; thermal_standard, hydro_dispatch, renewable_dispatch, + renewable_non_dispatch, synchronous_condenser)` +- `branches` — `(; line, two_winding_transformer, discrete_controlled_ac_branch)` +- `xfrm_3w` — `(; three_winding_transformer,)` +- `dclines` — `(; two_terminal_lcc_line, two_terminal_generic_hvdc_line)` + +# Fields — accumulators shared across readers + +- `arcs::Dict{Int, Dict{String, Any}}` — Arc dicts referenced by every + 2-terminal branch + 3W star arc + DC/VSC line + switch/breaker. +- `supplemental_attribute_associations::Vector{Dict{String, Any}}` — + `{"attribute_id", "entity_id"}` rows linking transformers to their + attached ImpedanceCorrectionData. +- `ids::IDGenerator` — the id minter threaded through every reader. + Retained so callers can mint additional ids consistent with the parsed + output. +""" +struct ParsedOpenAPIDicts + bus_dicts::Dict{Int, Dict{String, Any}} + area_dicts::Dict{Int, Dict{String, Any}} + area_interchange_dicts::Dict{String, Dict{String, Any}} + loadzone_dicts::Dict{Int, Dict{String, Any}} + loads::NamedTuple + switched_shunt_dicts::Dict{String, Dict{String, Any}} + shunt_dicts::Dict{String, Dict{String, Any}} + gens::NamedTuple + storage_dicts::Dict{String, Dict{String, Any}} + branches::NamedTuple + xfrm_3w::NamedTuple + transformer_circuits::Dict{Int, Dict{String, Any}} + switch_dicts::Dict{String, Dict{String, Any}} + breaker_dicts::Dict{String, Dict{String, Any}} + dclines::NamedTuple + vscline_dicts::Dict{String, Dict{String, Any}} + facts_dicts::Dict{String, Dict{String, Any}} + ict_instances::Dict{Tuple{Int, String}, Dict{String, Any}} + arcs::Dict{Int, Dict{String, Any}} + supplemental_attribute_associations::Vector{Dict{String, Any}} + ids::IDGenerator +end + +""" +Parse a [`PowerModelsData`] into OpenAPI-shaped dicts in one call. +Bundles the 16 `read_*!` calls behind a single entry point. + +Output is a [`ParsedOpenAPIDicts`] holding every collection plus the three +shared accumulators (`arcs`, `ict_instances`, +`supplemental_attribute_associations`) and the `IDGenerator` used. +Heterogeneous reader outputs (`loads`, `gens`, `branches`, `xfrm_3w`, +`dclines`) are NamedTuples of per-OpenAPI-type sub-collections. + +# Arguments + +- `pm_data::PowerModelsData`: parsed PowerModels data. + +# Keyword Arguments + +All kwargs are forwarded to every underlying reader. Each reader silently +ignores kwargs it doesn't recognize, so callers can pass any subset of the +following: + +- `bus_name_formatter::Function` +- `load_name_formatter::Function` +- `loadzone_name_formatter::Function` +- `area_name_formatter::Function` +- `gen_name_formatter::Function` +- `generator_mapping::Union{AbstractString, Dict}` — override the YAML + dispatch table path or the prebuilt mapping +- `branch_name_formatter::Function` +- `xfrm_3w_name_formatter::Function` +- `transformer_control_objective_formatter::Function` +- `dcline_name_formatter::Function` +- `vsc_line_name_formatter::Function` +- `switched_shunt_name_formatter::Function` +- `shunt_name_formatter::Function` + +# Throws + +`DataFormatError` if `pm_data` has no buses. + +# Example + +```julia +pm_data = PowerModelsData("test/path/to/case.raw") +parsed = parse_to_openapi_dicts( + pm_data; + bus_name_formatter = b -> "BUS_" * string(b["bus_i"]), + gen_name_formatter = g -> strip(join(g["source_id"], "_")), +) + +parsed.bus_dicts # Dict{Int, Dict{String, Any}} +parsed.loads.power_load # Dict{String, Dict{String, Any}} +parsed.gens.thermal_standard # ... +parsed.branches.line # ... +parsed.arcs # Arc accumulator +parsed.ict_instances # ICT dicts (PSS/E only) +parsed.supplemental_attribute_associations +``` +""" +function parse_to_openapi_dicts(pm_data::PowerModelsData; kwargs...) + data = pm_data.data + if !haskey(data, "bus") || isempty(data["bus"]) + throw(DataFormatError("There are no buses in this file.")) + end + + @info "Building OpenAPI dicts from PowerModelsData" source_type = + get(data, "source_type", "") + + ids = IDGenerator() + arcs = Dict{Int, Dict{String, Any}}() + transformer_circuits = Dict{Int, Dict{String, Any}}() + supplemental_attribute_associations = Dict{String, Any}[] + + # ICT lookup is built first so it's available to read_branch! / + # read_3w_transformer! (PSS/E-only; empty for MATPOWER files). + # read_impedance_correction! has no kwargs. + ict_instances = read_impedance_correction!(pm_data, ids) + + bus_dicts = read_bus!(pm_data, ids; kwargs...) + area_dicts = read_area!(pm_data, ids; kwargs...) + area_interchange_dicts = + read_area_interchange!(pm_data, ids, area_dicts; kwargs...) + loadzone_dicts = read_loadzones!(pm_data, ids; kwargs...) + loads = read_loads!(pm_data, ids; kwargs...) + switched_shunt_dicts = read_switched_shunt!(pm_data, ids; kwargs...) + shunt_dicts = read_shunt!(pm_data, ids; kwargs...) + gens = read_gen!(pm_data, ids; kwargs...) + storage_dicts = read_storage!(pm_data, ids, bus_dicts; kwargs...) + branches = read_branch!( + pm_data, + ids, + bus_dicts, + arcs, + transformer_circuits; + ict_instances = ict_instances, + supplemental_attribute_associations = supplemental_attribute_associations, + kwargs..., + ) + xfrm_3w = read_3w_transformer!( + pm_data, + ids, + bus_dicts, + arcs, + transformer_circuits; + ict_instances = ict_instances, + supplemental_attribute_associations = supplemental_attribute_associations, + kwargs..., + ) + switch_dicts = + read_switch_breaker!(pm_data, ids, bus_dicts, arcs, "switch"; kwargs...) + breaker_dicts = + read_switch_breaker!(pm_data, ids, bus_dicts, arcs, "breaker"; kwargs...) + dclines = read_dcline!(pm_data, ids, bus_dicts, arcs; kwargs...) + vscline_dicts = read_vscline!(pm_data, ids, bus_dicts, arcs; kwargs...) + facts_dicts = read_facts!(pm_data, ids, bus_dicts; kwargs...) + + return ParsedOpenAPIDicts( + bus_dicts, + area_dicts, + area_interchange_dicts, + loadzone_dicts, + loads, + switched_shunt_dicts, + shunt_dicts, + gens, + storage_dicts, + branches, + xfrm_3w, + transformer_circuits, + switch_dicts, + breaker_dicts, + dclines, + vscline_dicts, + facts_dicts, + ict_instances, + arcs, + supplemental_attribute_associations, + ids, + ) +end + +""" +Parse `pm_data` and write everything into a fresh SQLite database whose +schema is initialized by [`make_sqlite!`]. Returns the DB connection. +Parallels PSY's depreciated `System(pm_data::PowerModelsData)`, instead +of returning an in-memory typed System, it lands a populated database +directly. + +The internal flow is: +1. Open an SQLite DB (in-memory by default, or at `path`). +2. Initialize the full schema via [`make_sqlite!`]`(db)`. +3. Call [`parse_to_openapi_dicts(pm_data; kwargs...)`](@ref) to get the + OpenAPI-shaped dicts. +4. Write Arcs first (every branch-like row references an arc id). +5. Write ImpedanceCorrectionData supplemental attributes (rows must exist + before the transformer↔ICT association rows are inserted). +6. Walk [`_MAKE_DATABASE_TYPE_ORDER`] — for each OpenAPI type, materialize + structs via `OpenAPI.from_json` and call [`send_openapi_table_to_db!`]. + Insertion order matches `ALL_DESERIALIZABLE_TYPES` so FK references + (e.g. `ThermalStandard.bus` → `ACBus.id`) resolve in order. +7. Write the transformer↔ICT associations into + `supplemental_attributes_association`. + +The synthesized UUIDs written into the `attributes` table at the +`get_uuid` slot are random (`UUIDs.uuid4()`); this DB is one-way (parser → +DB) and cannot round-trip back to PSY components without an additional +mapping layer. + +# Arguments + +- `pm_data::PowerModelsData`: parsed PowerModels data. + +# Keyword Arguments + +- `path::AbstractString = ":memory:"` — SQLite destination. `":memory:"` + yields an in-memory DB; any other string opens a file at that path + (created if absent, truncated if present). + +# Returns + +`SQLite.DB` — the open connection. The caller is responsible for closing +it (`SQLite.close!` / `DBInterface.close!`) if the database needs to be +flushed to disk and reopened in another process. + +# Example + +```julia +pm_data = PowerModelsData("test/path/to/case.raw") +db = make_database(pm_data; path = "/tmp/case.db", + bus_name_formatter = b -> "BUS_" * string(b["bus_i"]), +) +# inspect: +DBInterface.execute(db, "SELECT COUNT(*) FROM buses") |> first +``` +""" +function make_database( + pm_data::PowerModelsData; + path::AbstractString = ":memory:", + kwargs..., +) + parsed = parse_to_openapi_objects(pm_data; kwargs...) + + db = SQLite.DB(String(path)) + make_sqlite!(db) + + # Split the type-order walk in half: topology types (Area / LoadZone / + # ACBus) go first so their `entities` rows exist before we insert + # arcs (whose `from_id`/`to_id` FK back into `entities`). Then arcs, + # then ICT supplemental attributes, then everything else. + _TOPOLOGY = (:Area, :LoadZone, :ACBus) + _is_topology(t::Symbol) = t in _TOPOLOGY + + for type_sym in _MAKE_DATABASE_TYPE_ORDER + _is_topology(type_sym) || continue + T = getfield(PowerOperationsOpenAPIModels, type_sym) + components = _extract_type_objects(parsed, type_sym) + isempty(components) && continue + if !haskey(_POAM_TYPE_TO_TABLE, T) && + T !== PowerOperationsOpenAPIModels.AreaInterchange + # Types not yet mapped to a table (TwoTerminalLCCLine, + # DiscreteControlledACBranch, FACTSControlDevice, etc.) are + # parsed but not persisted. Log and skip. + @debug "No DB table mapped for $type_sym; skipping ($(length(components)) objects dropped)" + continue + end + send_openapi_table_to_db!(T, db, components) + end + + # Arcs after topology so `from_id`/`to_id` FKs resolve. + write_arcs_to_db!(db, parsed.arcs) + + # Supplemental attributes (ICTs) must exist before the transformer↔ICT + # association rows reference them. ICT dicts are materialized to typed + # structs here rather than up-front in `_dicts_to_objects` so schema + # validation happens right before the DB write. + if !isempty(parsed.ict_instances) + ict_structs = [ + OpenAPI.from_json(ImpedanceCorrectionData, d) for d in parsed.ict_instances + ] + write_supplemental_attributes_to_db!(ImpedanceCorrectionData, db, ict_structs) + end + + # Everything else (branches, transformers, loads, gens, …). Order in + # `_MAKE_DATABASE_TYPE_ORDER` still governs FK dependencies among these. + for type_sym in _MAKE_DATABASE_TYPE_ORDER + _is_topology(type_sym) && continue + T = getfield(PowerOperationsOpenAPIModels, type_sym) + components = _extract_type_objects(parsed, type_sym) + isempty(components) && continue + if !haskey(_POAM_TYPE_TO_TABLE, T) && + T !== PowerOperationsOpenAPIModels.AreaInterchange + # Types not yet mapped to a table (TwoTerminalLCCLine, + # DiscreteControlledACBranch, FACTSControlDevice, etc.) are + # parsed but not persisted. Log and skip. + @debug "No DB table mapped for $type_sym; skipping ($(length(components)) objects dropped)" + continue + end + send_openapi_table_to_db!(T, db, components) + end + + # Transformer↔ICT association rows last, after both ICTs and Transformer + # rows are in the DB. + write_supplemental_attribute_associations_to_db!( + db, + parsed.supplemental_attribute_associations, + ) + + return db +end + +""" +Map an OpenAPI type Symbol to its corresponding sub-collection inside a +[`ParsedOpenAPIDicts`]. Encapsulates the heterogeneous-reader unwrapping +(`parsed.gens.thermal_standard` for `:ThermalStandard`, etc.) plus the +`DiscreteControlledACBranch` three-source merge. + +Returns an `AbstractDict` — could be an alias to one of the existing +sub-collections, or a freshly-merged dict (the DCAB case). +""" +function _extract_type_dicts(parsed::ParsedOpenAPIDicts, type_sym::Symbol) + if type_sym === :ACBus + return parsed.bus_dicts + elseif type_sym === :Area + return parsed.area_dicts + elseif type_sym === :AreaInterchange + return parsed.area_interchange_dicts + elseif type_sym === :LoadZone + return parsed.loadzone_dicts + elseif type_sym === :PowerLoad + return parsed.loads.power_load + elseif type_sym === :StandardLoad + return parsed.loads.standard_load + elseif type_sym === :InterruptibleStandardLoad + return parsed.loads.interruptible_standard_load + elseif type_sym === :SwitchedAdmittance + return parsed.switched_shunt_dicts + elseif type_sym === :FixedAdmittance + return parsed.shunt_dicts + elseif type_sym === :ThermalStandard + return parsed.gens.thermal_standard + elseif type_sym === :HydroDispatch + return parsed.gens.hydro_dispatch + elseif type_sym === :RenewableDispatch + return parsed.gens.renewable_dispatch + elseif type_sym === :RenewableNonDispatch + return parsed.gens.renewable_non_dispatch + elseif type_sym === :SynchronousCondenser + return parsed.gens.synchronous_condenser + elseif type_sym === :EnergyReservoirStorage + return parsed.storage_dicts + elseif type_sym === :Line + return parsed.branches.line + elseif type_sym === :TransformerCircuit + return parsed.transformer_circuits + elseif type_sym === :TwoWindingTransformer + return parsed.branches.two_winding_transformer + elseif type_sym === :ThreeWindingTransformer + return parsed.xfrm_3w.three_winding_transformer + elseif type_sym === :DiscreteControlledACBranch + # Three sources (zero-impedance branches + switches + breakers) + # collapse to one OpenAPI type. Composite id keys prevent + # collisions, so the merge is safe. + return merge( + parsed.branches.discrete_controlled_ac_branch, + parsed.switch_dicts, + parsed.breaker_dicts, + ) + elseif type_sym === :TwoTerminalLCCLine + return parsed.dclines.two_terminal_lcc_line + elseif type_sym === :TwoTerminalGenericHVDCLine + return parsed.dclines.two_terminal_generic_hvdc_line + elseif type_sym === :TwoTerminalVSCLine + return parsed.vscline_dicts + elseif type_sym === :FACTSControlDevice + return parsed.facts_dicts + else + error("Unknown OpenAPI type symbol: $type_sym") + end +end + +# ============================================================================= +# Typed OpenAPI objects — public end-user entry point. +# +# The `ParsedOpenAPIObjects` container holds *materialized* OpenAPI structs +# (validated via `OpenAPI.from_json`) instead of the raw dicts. This is the +# shape downstream consumers want — both [`make_database`] (which writes +# them directly into SQLite) and any external PSY-System constructor that +# wants to consume OpenAPI structs (e.g. +# `SiennaOpenAPIModels.jl/src/json_to_sienna/`). +# +# `ParsedOpenAPIDicts` is kept as the under-the-hood intermediate +# representation — readers still produce dicts, then `_dicts_to_objects` +# converts. Direct dict access remains available via +# `PowerFlowFileParser.parse_to_openapi_dicts(pm_data)` (qualified, not +# exported). +# ============================================================================= + +""" +Container holding every output collection produced by a single +[`parse_to_openapi_objects`] run, with each collection holding +**materialized** OpenAPI structs (validated by `OpenAPI.from_json`) +rather than the dicts that come out of the readers. + +# Fields — homogeneous reader outputs (`Vector{T}` for the corresponding T) + +- `buses::Vector{PowerOperationsOpenAPIModels.ACBus}` +- `areas::Vector{PowerOperationsOpenAPIModels.Area}` +- `area_interchanges::Vector{PowerOperationsOpenAPIModels.AreaInterchange}` +- `loadzones::Vector{PowerOperationsOpenAPIModels.LoadZone}` +- `switched_shunts::Vector{PowerOperationsOpenAPIModels.SwitchedAdmittance}` +- `shunts::Vector{PowerOperationsOpenAPIModels.FixedAdmittance}` +- `storage::Vector{PowerOperationsOpenAPIModels.EnergyReservoirStorage}` +- `switches::Vector{PowerOperationsOpenAPIModels.DiscreteControlledACBranch}` — from + `data["switch"]` +- `breakers::Vector{PowerOperationsOpenAPIModels.DiscreteControlledACBranch}` — + from `data["breaker"]` +- `vsclines::Vector{PowerOperationsOpenAPIModels.TwoTerminalVSCLine}` +- `facts::Vector{PowerOperationsOpenAPIModels.FACTSControlDevice}` +- `ict_instances::Vector{Dict{String, Any}}` — raw ICT dicts, not + materialized here. [`make_database`] materializes them to + `PowerOperationsOpenAPIModels.ImpedanceCorrectionData` structs at write + time. + +# Fields — heterogeneous reader outputs (NamedTuples per OpenAPI type) -Identifies branches that should be transformers by comparing voltage levels at -connected buses. If the voltage difference exceeds the threshold -(BRANCH_BUS_VOLTAGE_DIFFERENCE_TOL), the branch is converted to a transformer. +- `loads` — `(; power_load, standard_load, interruptible_standard_load)`, + each a `Vector` of its corresponding OpenAPI type +- `gens` — `(; thermal_standard, hydro_dispatch, renewable_dispatch, + renewable_non_dispatch, synchronous_condenser)` +- `branches` — `(; line, two_winding_transformer, discrete_controlled_ac_branch)` +- `xfrm_3w` — `(; three_winding_transformer,)` +- `dclines` — `(; two_terminal_lcc_line, two_terminal_generic_hvdc_line)` + +# Fields — accumulators + +- `arcs::Vector{PowerOperationsOpenAPIModels.Arc}` — Arc structs referenced by + every 2-terminal branch + 3W star arc + DC/VSC line + switch/breaker. +- `supplemental_attribute_associations::Vector{Dict{String, Any}}` — + `{"attribute_id", "entity_id"}` rows linking transformers to their + attached ImpedanceCorrectionData. Kept as Dicts — they're FK edges, not + OpenAPI components. +- `ids::IDGenerator` — the id minter threaded through every reader. +""" +struct ParsedOpenAPIObjects + buses::Vector{PowerOperationsOpenAPIModels.ACBus} + areas::Vector{PowerOperationsOpenAPIModels.Area} + area_interchanges::Vector{PowerOperationsOpenAPIModels.AreaInterchange} + loadzones::Vector{PowerOperationsOpenAPIModels.LoadZone} + loads::NamedTuple + switched_shunts::Vector{PowerOperationsOpenAPIModels.SwitchedAdmittance} + shunts::Vector{PowerOperationsOpenAPIModels.FixedAdmittance} + gens::NamedTuple + storage::Vector{PowerOperationsOpenAPIModels.EnergyReservoirStorage} + branches::NamedTuple + xfrm_3w::NamedTuple + transformer_circuits::Vector{PowerOperationsOpenAPIModels.TransformerCircuit} + switches::Vector{PowerOperationsOpenAPIModels.DiscreteControlledACBranch} + breakers::Vector{PowerOperationsOpenAPIModels.DiscreteControlledACBranch} + dclines::NamedTuple + vsclines::Vector{PowerOperationsOpenAPIModels.TwoTerminalVSCLine} + facts::Vector{PowerOperationsOpenAPIModels.FACTSControlDevice} + ict_instances::Vector{Dict{String, Any}} + arcs::Vector{PowerOperationsOpenAPIModels.Arc} + supplemental_attribute_associations::Vector{Dict{String, Any}} + ids::IDGenerator +end + +""" +Convert a homogeneous collection of OpenAPI-shaped dicts to a typed +`Vector{T}`. Each dict is passed through `OpenAPI.from_json(T, d)`, which +validates required fields and enum values per the schema. + +Accepts any iterable of dicts (Dict-of-dicts and NamedTuple sub-fields +both work — we call `values(…)` to drop keys uniformly). +""" +_convert_dicts(::Type{T}, dicts) where {T <: OpenAPI.APIModel} = + T[OpenAPI.from_json(T, d) for d in values(dicts)] + +""" +Convert a [`ParsedOpenAPIDicts`] into a [`ParsedOpenAPIObjects`] by +calling `OpenAPI.from_json` once per dict per OpenAPI type. Centralizes +all the from_json work in one place so failures (required-field misses, +enum mismatches) surface here rather than during a downstream loop. + +Heterogeneous reader outputs are converted in lockstep — each NamedTuple +sub-field becomes a typed Vector under the same name. +""" +function _dicts_to_objects(parsed::ParsedOpenAPIDicts) + POAM = PowerOperationsOpenAPIModels # local alias for brevity + + loads = (; + power_load = _convert_dicts(POAM.PowerLoad, parsed.loads.power_load), + standard_load = _convert_dicts(POAM.StandardLoad, parsed.loads.standard_load), + interruptible_standard_load = _convert_dicts( + POAM.InterruptibleStandardLoad, + parsed.loads.interruptible_standard_load, + ), + ) + gens = (; + thermal_standard = _convert_dicts( + POAM.ThermalStandard, + parsed.gens.thermal_standard, + ), + hydro_dispatch = _convert_dicts(POAM.HydroDispatch, parsed.gens.hydro_dispatch), + renewable_dispatch = _convert_dicts( + POAM.RenewableDispatch, + parsed.gens.renewable_dispatch, + ), + renewable_non_dispatch = _convert_dicts( + POAM.RenewableNonDispatch, + parsed.gens.renewable_non_dispatch, + ), + synchronous_condenser = _convert_dicts( + POAM.SynchronousCondenser, + parsed.gens.synchronous_condenser, + ), + ) + branches = (; + line = _convert_dicts(POAM.Line, parsed.branches.line), + two_winding_transformer = _convert_dicts( + POAM.TwoWindingTransformer, + parsed.branches.two_winding_transformer, + ), + discrete_controlled_ac_branch = _convert_dicts( + POAM.DiscreteControlledACBranch, + parsed.branches.discrete_controlled_ac_branch, + ), + ) + xfrm_3w = (; + three_winding_transformer = _convert_dicts( + POAM.ThreeWindingTransformer, + parsed.xfrm_3w.three_winding_transformer, + ), + ) + dclines = (; + two_terminal_lcc_line = _convert_dicts( + POAM.TwoTerminalLCCLine, + parsed.dclines.two_terminal_lcc_line, + ), + two_terminal_generic_hvdc_line = _convert_dicts( + POAM.TwoTerminalGenericHVDCLine, + parsed.dclines.two_terminal_generic_hvdc_line, + ), + ) + + return ParsedOpenAPIObjects( + _convert_dicts(POAM.ACBus, parsed.bus_dicts), + _convert_dicts(POAM.Area, parsed.area_dicts), + _convert_dicts(POAM.AreaInterchange, parsed.area_interchange_dicts), + _convert_dicts(POAM.LoadZone, parsed.loadzone_dicts), + loads, + _convert_dicts(POAM.SwitchedAdmittance, parsed.switched_shunt_dicts), + _convert_dicts(POAM.FixedAdmittance, parsed.shunt_dicts), + gens, + _convert_dicts(POAM.EnergyReservoirStorage, parsed.storage_dicts), + branches, + xfrm_3w, + _convert_dicts(POAM.TransformerCircuit, parsed.transformer_circuits), + _convert_dicts(POAM.DiscreteControlledACBranch, parsed.switch_dicts), + _convert_dicts(POAM.DiscreteControlledACBranch, parsed.breaker_dicts), + dclines, + _convert_dicts(POAM.TwoTerminalVSCLine, parsed.vscline_dicts), + _convert_dicts(POAM.FACTSControlDevice, parsed.facts_dicts), + # ICT dicts pass through untouched; `make_database` materializes them + # to `PowerOperationsOpenAPIModels.ImpedanceCorrectionData` at write + # time, right before the DB insert. + collect(values(parsed.ict_instances)), + _convert_dicts(POAM.Arc, parsed.arcs), + parsed.supplemental_attribute_associations, + parsed.ids, + ) +end + +""" +Parse a [`PowerModelsData`] into materialized OpenAPI objects in one call. +This is the public end-user entry point for the dict-to-objects pipeline; +the underlying [`ParsedOpenAPIDicts`] representation is kept as an +implementation detail. + +Internally: `_dicts_to_objects(parse_to_openapi_dicts(pm_data; kwargs...))`. +Calling this is strictly more work than `parse_to_openapi_dicts` — it adds +one `OpenAPI.from_json` call per output dict. The benefit is that +schema validation (required fields + enum membership) happens up front, +and the result is ready to hand to either [`make_database`] or a +downstream PSY-System constructor (e.g. SiennaOpenAPIModels' +`json_to_sienna/`, which lives in a separate package). # Arguments -- `pm_data::PowerModelsData`: PowerModels data object to correct +- `pm_data::PowerModelsData`: parsed PowerModels data. + +# Keyword Arguments + +Same as [`parse_to_openapi_dicts`]: all kwargs are forwarded to the +underlying readers. Each reader silently ignores kwargs it doesn't +recognize (`bus_name_formatter`, `gen_name_formatter`, etc.). + +# Throws + +`DataFormatError` if `pm_data` has no buses (from `parse_to_openapi_dicts`). +Whatever `OpenAPI.ValidationException` / `MethodError` `from_json` raises +if a dict fails schema validation. + +# Returns + +[`ParsedOpenAPIObjects`]. + +# Example + +```julia +pm_data = PowerModelsData("test/path/to/case.raw") +parsed = parse_to_openapi_objects(pm_data) + +parsed.buses # Vector{ACBus} +parsed.loads.power_load # Vector{PowerLoad} +parsed.gens.thermal_standard # Vector{ThermalStandard} +parsed.branches.line # Vector{Line} +parsed.arcs # Vector{Arc} +parsed.ict_instances # Vector{ImpedanceCorrectionData} + +# Pass downstream — e.g., to a PSY-System constructor (a separate package +# like SiennaOpenAPIModels' json_to_sienna/, not provided here). +``` +""" +function parse_to_openapi_objects(pm_data::PowerModelsData; kwargs...) + return _dicts_to_objects(parse_to_openapi_dicts(pm_data; kwargs...)) +end + +""" +Object-side counterpart of [`_extract_type_dicts`]: selects the right +sub-collection from a [`ParsedOpenAPIObjects`] for a given OpenAPI type +Symbol. Used by [`make_database`] when writing tables in dependency +order. Handles the `:DiscreteControlledACBranch` three-source merge +(zero-impedance + switches + breakers concatenated into one Vector). +""" +function _extract_type_objects(parsed::ParsedOpenAPIObjects, type_sym::Symbol) + if type_sym === :ACBus + return parsed.buses + elseif type_sym === :Area + return parsed.areas + elseif type_sym === :AreaInterchange + return parsed.area_interchanges + elseif type_sym === :LoadZone + return parsed.loadzones + elseif type_sym === :PowerLoad + return parsed.loads.power_load + elseif type_sym === :StandardLoad + return parsed.loads.standard_load + elseif type_sym === :InterruptibleStandardLoad + return parsed.loads.interruptible_standard_load + elseif type_sym === :SwitchedAdmittance + return parsed.switched_shunts + elseif type_sym === :FixedAdmittance + return parsed.shunts + elseif type_sym === :ThermalStandard + return parsed.gens.thermal_standard + elseif type_sym === :HydroDispatch + return parsed.gens.hydro_dispatch + elseif type_sym === :RenewableDispatch + return parsed.gens.renewable_dispatch + elseif type_sym === :RenewableNonDispatch + return parsed.gens.renewable_non_dispatch + elseif type_sym === :SynchronousCondenser + return parsed.gens.synchronous_condenser + elseif type_sym === :EnergyReservoirStorage + return parsed.storage + elseif type_sym === :Line + return parsed.branches.line + elseif type_sym === :TransformerCircuit + return parsed.transformer_circuits + elseif type_sym === :TwoWindingTransformer + return parsed.branches.two_winding_transformer + elseif type_sym === :ThreeWindingTransformer + return parsed.xfrm_3w.three_winding_transformer + elseif type_sym === :DiscreteControlledACBranch + # Three sources concatenated. Composite id keys (from the dict + # stage) prevented collisions, so the result is a flat Vector + # with unique ids. + return vcat( + parsed.branches.discrete_controlled_ac_branch, + parsed.switches, + parsed.breakers, + ) + elseif type_sym === :TwoTerminalLCCLine + return parsed.dclines.two_terminal_lcc_line + elseif type_sym === :TwoTerminalGenericHVDCLine + return parsed.dclines.two_terminal_generic_hvdc_line + elseif type_sym === :TwoTerminalVSCLine + return parsed.vsclines + elseif type_sym === :FACTSControlDevice + return parsed.facts + else + error("Unknown OpenAPI type symbol: $type_sym") + end +end + +# =================================================================================== + +""" +Corrects transformer status in PowerModelsData based on bus voltage differences. """ function correct_pm_transformer_status!(pm_data::PowerModelsData) for (k, branch) in pm_data.data["branch"] @@ -70,7 +758,9 @@ function correct_pm_transformer_status!(pm_data::PowerModelsData) branch["transformer"] = true branch["base_power"] = pm_data.data["baseMVA"] branch["ext"] = Dict{String, Any}() - @warn "Branch $(branch["f_bus"]) - $(branch["t_bus"]) has different voltage levels endpoints (from: $(f_bus_bvolt)kV, to: $(t_bus_bvolt)kV) which exceed the $(BRANCH_BUS_VOLTAGE_DIFFERENCE_TOL*100)% threshold; converting to transformer." + @warn "Branch $(branch["f_bus"]) - $(branch["t_bus"]) has different voltage levels + endpoints (from: $(f_bus_bvolt)kV, to: $(t_bus_bvolt)kV) which exceed the + $(BRANCH_BUS_VOLTAGE_DIFFERENCE_TOL*100)% threshold; converting to transformer." if !haskey(branch, "base_voltage_from") branch["base_voltage_from"] = f_bus_bvolt branch["base_voltage_to"] = t_bus_bvolt @@ -78,3 +768,2732 @@ function correct_pm_transformer_status!(pm_data::PowerModelsData) end end end + +""" +Names generic PowerModels device dict, falling back to source_id +or numeric index if no name is set. +""" +function _get_pm_dict_name(device_dict::Dict)::String + if haskey(device_dict, "shunt_bus") + # Shunt names must be qualified by bus number to avoid collisions + # between FixedAdmittance and SwitchedAdmittance attached to the same bus. + return join(strip.(string.((device_dict["shunt_bus"], device_dict["name"]))), "-") + elseif haskey(device_dict, "name") + return string(device_dict["name"]) + elseif haskey(device_dict, "source_id") + return strip(join(string.(device_dict["source_id"]), "-")) + else + return string(device_dict["index"]) + end +end + +""" +Resolve a bus name from a PowerModels bus dict. When `unique_names` is false, +the bus number is appended to disambiguate duplicates (a PSS/E quirk). +""" +function _get_pm_bus_name(device_dict::Dict, unique_names::Bool) + if haskey(device_dict, "name") + base = strip(device_dict["name"]) + return unique_names ? base : base * "_" * string(device_dict["bus_i"]) + else + return strip(join(string.(device_dict["source_id"]), "-")) + end +end + +""" +Translate one PowerModels bus row into an ACBus-shaped `Dict{String, Any}`. +The resulting dict is ready to hand to `OpenAPI.from_json(PowerOperationsOpenAPIModels.ACBus, d)`. +""" +function make_bus(bus_name::AbstractString, bus_number::Int, d::Dict, ids::IDGenerator) + return Dict{String, Any}( + "id" => getid!(ids, :ACBus, bus_number), + "number" => bus_number, + "name" => String(bus_name), + "available" => get(d, "bus_status", true), + "bustype" => _PM_BUS_TYPE_ENUM[d["bus_type"]], + "angle" => d["va"], + "magnitude" => d["vm"], + "voltage_limits" => + Dict{String, Any}("min" => d["vmin"], "max" => d["vmax"]), + "base_voltage" => d["base_kv"], + "area" => getid!(ids, :Area, d["area"]), + "load_zone" => getid!(ids, :LoadZone, get(d, "zone", nothing)), + ) +end + +""" +Walk every bus in `pm_data` and return a `Dict{Int, Dict{String, Any}}` mapping +each bus number to an ACBus-shaped dict ready for `OpenAPI.from_json`. +""" +function read_bus!(pm_data::PowerModelsData, ids::IDGenerator; kwargs...) + @info "Reading bus data" + data = pm_data.data + bus_number_to_bus = Dict{Int, Dict{String, Any}}() + + # PSSE doesn't enforce bus-name uniqueness. Detect duplicates up front so + # we can fall back to a number-suffixed naming scheme without forcing the + # caller to pass a custom formatter. + unique_bus_names = true + bus_data = SortedDict{Int, Any}() + bus_names = Set{String}() + for (_, b) in data["bus"] + if unique_bus_names && haskey(b, "name") + b["name"] ∈ bus_names && (unique_bus_names = false) + push!(bus_names, b["name"]) + end + bus_data[Int(b["bus_i"])] = b + end + isempty(bus_data) && @error "No bus data found" + + default_bus_naming = x -> _get_pm_bus_name(x, unique_bus_names) + _get_name = get(kwargs, :bus_name_formatter, default_bus_naming) + + for (_, d) in bus_data + bus_name = String(strip(_get_name(d))) + bus_number = Int(d["bus_i"]) + if !haskey(d, "bus_status") + d["bus_status"] = true + end + bus = make_bus(bus_name, bus_number, d, ids) + haskey(bus_number_to_bus, bus_number) && throw( + DataFormatError( + "Found duplicate bus number $bus_number for bus $bus_name", + ), + ) + bus_number_to_bus[bus_number] = bus + end + + return bus_number_to_bus +end + +""" +Build the dict equivalent of `LoadCost(variable = zero(CostCurve), fixed = 0.0)`. +""" +function _zero_loadcost_dict() + zero_io_curve = Dict{String, Any}( + "curve_type" => "INPUT_OUTPUT", + "function_data" => Dict{String, Any}( + "function_type" => "LINEAR", + "proportional_term" => 0.0, + "constant_term" => 0.0, + ), + ) + return Dict{String, Any}( + "cost_type" => "LOAD", + "fixed" => 0.0, + "variable" => Dict{String, Any}( + "power_units" => "NATURAL_UNITS", + "variable_cost_type" => "COST", + "value_curve" => zero_io_curve, + "vom_cost" => deepcopy(zero_io_curve), + ), + ) +end + +""" +Translate one PowerModels load row into an InterruptiblePowerLoad-shaped dict. +""" +function make_interruptible_powerload( + d::Dict, + sys_mbase::Float64, + ids::IDGenerator; + kwargs..., +) + _get_name = get(kwargs, :load_name_formatter, x -> strip(join(x["source_id"]))) + return Dict{String, Any}( + "id" => getid!(ids, :InterruptiblePowerLoad, d["index"]), + "name" => String(_get_name(d)), + "available" => d["status"], + "bus" => getid!(ids, :ACBus, d["load_bus"]), + "active_power" => d["pd"], + "reactive_power" => d["qd"], + "max_active_power" => d["pd"], + "max_reactive_power" => d["qd"], + "base_power" => sys_mbase, + "operation_cost" => _zero_loadcost_dict(), + ) +end + +""" +Translate one PowerModels load row into an InterruptibleStandardLoad-shaped dict. +""" +function make_interruptible_standardload( + d::Dict, + sys_mbase::Float64, + ids::IDGenerator; + kwargs..., +) + _get_name = get(kwargs, :load_name_formatter, x -> strip(join(x["source_id"]))) + return Dict{String, Any}( + "id" => getid!(ids, :InterruptibleStandardLoad, d["index"]), + "name" => String(_get_name(d)), + "available" => d["status"], + "bus" => getid!(ids, :ACBus, d["load_bus"]), + "base_power" => sys_mbase, + "operation_cost" => _zero_loadcost_dict(), + "conformity" => _loadconformity_string(d["conformity"]), + "constant_active_power" => d["pd"], + "constant_reactive_power" => d["qd"], + "current_active_power" => d["pi"], + "current_reactive_power" => d["qi"], + "impedance_active_power" => d["py"], + "impedance_reactive_power" => d["qy"], + "max_constant_active_power" => d["pd"], + "max_constant_reactive_power" => d["qd"], + "max_current_active_power" => d["pi"], + "max_current_reactive_power" => d["qi"], + "max_impedance_active_power" => d["py"], + "max_impedance_reactive_power" => d["qy"], + ) +end + +""" +Translate one PowerModels load row into a PowerLoad-shaped dict. +""" +function make_power_load(d::Dict, sys_mbase::Float64, ids::IDGenerator; kwargs...) + _get_name = get(kwargs, :load_name_formatter, x -> strip(join(x["source_id"]))) + return Dict{String, Any}( + "id" => getid!(ids, :PowerLoad, d["index"]), + "name" => String(_get_name(d)), + "available" => d["status"], + "bus" => getid!(ids, :ACBus, d["load_bus"]), + "active_power" => d["pd"], + "reactive_power" => d["qd"], + "max_active_power" => d["pd"], + "max_reactive_power" => d["qd"], + "base_power" => sys_mbase, + "conformity" => _loadconformity_string(d["conformity"]), + ) +end + +""" +Translate one PowerModels load row into a StandardLoad-shaped dict. +""" +function make_standard_load(d::Dict, sys_mbase::Float64, ids::IDGenerator; kwargs...) + _get_name = get(kwargs, :load_name_formatter, x -> strip(join(x["source_id"]))) + return Dict{String, Any}( + "id" => getid!(ids, :StandardLoad, d["index"]), + "name" => String(_get_name(d)), + "available" => d["status"], + "bus" => getid!(ids, :ACBus, d["load_bus"]), + "base_power" => sys_mbase, + "conformity" => _loadconformity_string(d["conformity"]), + "constant_active_power" => d["pd"], + "constant_reactive_power" => d["qd"], + "current_active_power" => d["pi"], + "current_reactive_power" => d["qi"], + "impedance_active_power" => d["py"], + "impedance_reactive_power" => d["qy"], + "max_constant_active_power" => d["pd"], + "max_constant_reactive_power" => d["qd"], + "max_current_active_power" => d["pi"], + "max_current_reactive_power" => d["qi"], + "max_impedance_active_power" => d["py"], + "max_impedance_reactive_power" => d["qy"], + ) +end + +""" +Walk every load in `pm_data` and return a NamedTuple of per-OpenAPI-type +sub-collections. + + PTI + has `interruptible` field + value != 1 → StandardLoad + PTI + has `interruptible` field + value == 1 → InterruptibleStandardLoad + otherwise (MATPOWER, or PTI without flag) → PowerLoad +""" +function read_loads!(pm_data::PowerModelsData, ids::IDGenerator; kwargs...) + @info "Reading load data" + data = pm_data.data + power_load = Dict{String, Dict{String, Any}}() + standard_load = Dict{String, Dict{String, Any}}() + interruptible_standard_load = Dict{String, Dict{String, Any}}() + + if !haskey(data, "load") + @error "There are no loads in this file" + return (; power_load, standard_load, interruptible_standard_load) + end + + sys_mbase = data["baseMVA"] + is_pti = data["source_type"] == "pti" + for d_key in keys(data["load"]) + d = data["load"][d_key] + is_interruptible = haskey(d, "interruptible") + if is_pti && is_interruptible && d["interruptible"] != 1 + load = make_standard_load(d, sys_mbase, ids; kwargs...) + bucket = standard_load + elseif is_pti && is_interruptible && d["interruptible"] == 1 + load = make_interruptible_standardload(d, sys_mbase, ids; kwargs...) + bucket = interruptible_standard_load + else + load = make_power_load(d, sys_mbase, ids; kwargs...) + bucket = power_load + end + load_name = load["name"] + haskey(bucket, load_name) && throw( + DataFormatError( + "Found duplicate load name $load_name; consider passing a `load_name_formatter` kwarg", + ), + ) + bucket[load_name] = load + end + + return (; power_load, standard_load, interruptible_standard_load) +end + +""" +Build a LoadZone-shaped `Dict{String, Any}` from precomputed aggregate values. +""" +function make_loadzone( + name::AbstractString, + zone_number::Int, + active_power::Float64, + reactive_power::Float64, + ids::IDGenerator; + kwargs..., +) + return Dict{String, Any}( + "id" => getid!(ids, :LoadZone, zone_number), + "name" => String(name), + "peak_active_power" => active_power, + "peak_reactive_power" => reactive_power, + ) +end + +""" +Walk every load zone referenced by buses and return a `Dict{Int, Dict{String, Any}}` +mapping each zone number to a LoadZone-shaped dict ready for `OpenAPI.from_json`. +""" +function read_loadzones!(pm_data::PowerModelsData, ids::IDGenerator; kwargs...) + @info "Reading load zone data" + data = pm_data.data + + zones = Set{Int}() + for (_, bus) in data["bus"] + push!(zones, bus["zone"]) + end + + load_zone_map = Dict{Int, Dict{String, Float64}}( + i => Dict("pd" => 0.0, "qd" => 0.0) for i in zones + ) + for (_, load) in data["load"] + zone = data["bus"][load["load_bus"]]["zone"] + load_zone_map[zone]["pd"] += load["pd"] + load_zone_map[zone]["qd"] += load["qd"] + # MATPOWER loads don't carry current/impedance components; PSS/E does. + load_zone_map[zone]["pd"] += get(load, "pi", 0.0) + load_zone_map[zone]["qd"] += get(load, "qi", 0.0) + load_zone_map[zone]["pd"] += get(load, "py", 0.0) + load_zone_map[zone]["qd"] += get(load, "qy", 0.0) + end + + _get_name = get(kwargs, :loadzone_name_formatter, string) + + @info "Reading Zone data" + if !haskey(data, "zone") + @info "There is no Zone data in this file" + else + for (_, v) in data["zone"] + zone_number = v["zone_number"] + if !(zone_number in zones) + @warn "Skipping empty LoadZone $(zone_number)-$(v["zone_name"])" + end + end + end + + load_zones = Dict{Int, Dict{String, Any}}() + for zone in zones + load_zones[zone] = make_loadzone( + _get_name(zone), + zone, + load_zone_map[zone]["pd"], + load_zone_map[zone]["qd"], + ids; + kwargs..., + ) + end + return load_zones +end + +""" +Translate one PowerModels switched-shunt row into a SwitchedAdmittance-shaped +dict. +""" +function make_switched_shunt(name::AbstractString, d::Dict, ids::IDGenerator) + out = Dict{String, Any}( + "id" => getid!(ids, :SwitchedAdmittance, d["index"]), + "name" => String(name), + "available" => Bool(d["status"]), + "bus" => getid!(ids, :ACBus, d["shunt_bus"]), + "Y" => _complex_to_dict(d["gs"] + d["bs"] * im), + "number_of_steps" => d["step_number"], + "Y_increase" => [_complex_to_dict(y) for y in d["y_increment"]], + "admittance_limits" => Dict{String, Any}( + "min" => d["admittance_limits"][1], + "max" => d["admittance_limits"][2], + ), + ) + if haskey(d, "initial_status") + out["initial_status"] = d["initial_status"] + end + return out +end + +""" +For each switched shunt, return a `Dict{String, Dict{String, Any}}` mapping +each shunt name to a SwitchedAdmittance-shaped dict ready for `OpenAPI.from_json`. + +Switched shunts are PSS/E-only; MATPOWER files have no `data["switched_shunt"]` +section and this function returns an empty dict for them. +""" +function read_switched_shunt!(pm_data::PowerModelsData, ids::IDGenerator; kwargs...) + @info "Reading switched shunt data" + shunts = Dict{String, Dict{String, Any}}() + data = pm_data.data + if !haskey(data, "switched_shunt") + @info "There is no switched shunt data in this file" + return shunts + end + + _get_name = get(kwargs, :switched_shunt_name_formatter, _get_pm_dict_name) + + for (d_key, d) in data["switched_shunt"] + d["name"] = get(d, "name", d_key) + name = String(_get_name(d)) + shunt = make_switched_shunt(name, d, ids) + haskey(shunts, name) && throw( + DataFormatError( + "Found duplicate switched shunt name $name; consider passing a `switched_shunt_name_formatter` kwarg", + ), + ) + shunts[name] = shunt + end + return shunts +end + +""" +Translate one PowerModels fixed shunt row into a FixedAdmittance-shaped dict. +""" +function make_shunt(name::AbstractString, d::Dict, ids::IDGenerator) + return Dict{String, Any}( + "id" => getid!(ids, :FixedAdmittance, d["index"]), + "name" => String(name), + "available" => Bool(d["status"]), + "bus" => getid!(ids, :ACBus, d["shunt_bus"]), + "Y" => _complex_to_dict(d["gs"] + d["bs"] * im), + ) +end + +""" +For each fixed shunt, return a `Dict{String, Dict{String, Any}}` mapping each +shunt name to a FixedAdmittance-shaped dict ready for `OpenAPI.from_json`. + +MATPOWER's `pm_io/matpower.jl:_split_loads_shunts!` synthesizes `data["shunt"]` +from bus rows; PSS/E exposes its `FIXED SHUNT DATA` section directly. +""" +function read_shunt!(pm_data::PowerModelsData, ids::IDGenerator; kwargs...) + @info "Reading shunt data" + shunts = Dict{String, Dict{String, Any}}() + data = pm_data.data + if !haskey(data, "shunt") + @info "There is no shunt data in this file" + return shunts + end + + _get_name = get(kwargs, :shunt_name_formatter, _get_pm_dict_name) + + for (d_key, d) in data["shunt"] + d["name"] = get(d, "name", d_key) + name = String(_get_name(d)) + shunt = make_shunt(name, d, ids) + haskey(shunts, name) && throw( + DataFormatError( + "Found duplicate shunt name $name; consider passing a `shunt_name_formatter` kwarg", + ), + ) + shunts[name] = shunt + end + return shunts +end + +# ============================================================================= +# Generators +# ============================================================================= + +""" +Build the dict equivalent of `HydroGenerationCost(zero(CostCurve), 0.0)`. +""" +function _zero_hydro_generation_cost_dict() + zero_io_curve = Dict{String, Any}( + "curve_type" => "INPUT_OUTPUT", + "function_data" => Dict{String, Any}( + "function_type" => "LINEAR", + "proportional_term" => 0.0, + "constant_term" => 0.0, + ), + ) + return Dict{String, Any}( + "cost_type" => "HYDRO_GEN", + "fixed" => 0.0, + "variable" => Dict{String, Any}( + "variable_cost_type" => "COST", + "power_units" => "NATURAL_UNITS", + "value_curve" => zero_io_curve, + "vom_cost" => deepcopy(zero_io_curve), + ), + ) +end + +""" +Build the dict equivalent of `RenewableGenerationCost(zero(CostCurve))`. +""" +function _zero_renewable_generation_cost_dict() + zero_io_curve = Dict{String, Any}( + "curve_type" => "INPUT_OUTPUT", + "function_data" => Dict{String, Any}( + "function_type" => "LINEAR", + "proportional_term" => 0.0, + "constant_term" => 0.0, + ), + ) + return Dict{String, Any}( + "cost_type" => "RENEWABLE", + "fixed" => 0.0, + "variable" => Dict{String, Any}( + "variable_cost_type" => "COST", + "power_units" => "NATURAL_UNITS", + "value_curve" => zero_io_curve, + "vom_cost" => deepcopy(zero_io_curve), + ), + "curtailment_cost" => Dict{String, Any}( + "variable_cost_type" => "COST", + "power_units" => "NATURAL_UNITS", + "value_curve" => deepcopy(zero_io_curve), + "vom_cost" => deepcopy(zero_io_curve), + ), + ) +end + +""" +Build an all-zero `ThermalGenerationCost` dict. +""" +function _zero_thermal_generation_cost_dict() + zero_io_curve = Dict{String, Any}( + "curve_type" => "INPUT_OUTPUT", + "function_data" => Dict{String, Any}( + "function_type" => "LINEAR", + "proportional_term" => 0.0, + "constant_term" => 0.0, + ), + ) + return Dict{String, Any}( + "cost_type" => "THERMAL", + "fixed" => 0.0, + "start_up" => 0.0, + "shut_down" => 0.0, + "variable" => Dict{String, Any}( + "variable_cost_type" => "COST", + # FLAG: NATURAL_UNITS here — see the matching FLAG in + # _thermal_variable_cost_and_fixed for why the real-data path + # uses DEVICE_BASE instead. The all-zero placeholder has no + # scaling to honor, so NATURAL_UNITS is harmless; keep this in + # sync if the DEVICE_BASE decision is ever revisited. + "power_units" => "NATURAL_UNITS", + "value_curve" => zero_io_curve, + "vom_cost" => deepcopy(zero_io_curve), + ), + ) +end + +""" +Translate one PowerModels generator row tagged for hydro-dispatch into a +HydroDispatch-shaped dict. +""" +function make_hydro_dispatch( + gen_name::AbstractString, + d::Dict, + sys_mbase::Float64, + ids::IDGenerator, +) + mbase = _resolve_mbase(d, sys_mbase, gen_name) + base_conversion = sys_mbase / mbase + return Dict{String, Any}( + "id" => getid!(ids, :HydroDispatch, d["index"]), + "name" => String(gen_name), + "available" => Bool(d["gen_status"]), + "bus" => getid!(ids, :ACBus, d["gen_bus"]), + "active_power" => d["pg"] * base_conversion, + "reactive_power" => d["qg"] * base_conversion, + "rating" => _calculate_gen_rating(d["pmax"], d["qmax"], base_conversion), + "prime_mover_type" => _normalize_prime_mover(d["type"]), + "active_power_limits" => + _min_max_dict(d["pmin"] * base_conversion, d["pmax"] * base_conversion), + "reactive_power_limits" => + _min_max_dict(d["qmin"] * base_conversion, d["qmax"] * base_conversion), + "ramp_limits" => _calculate_ramp_limit_dict(d, gen_name), + "time_limits" => nothing, + "operation_cost" => _zero_hydro_generation_cost_dict(), + "base_power" => mbase, + ) +end + +""" +Translate one PowerModels generator row tagged for hydro-turbine (YAML +target `HydroTurbine`) into a HydroDispatch-shaped dict. + +PowerModels has no way to define storage parameters for generators, +so even hydro-turbine entries can only be built as a HydroDispatch. +""" +function make_hydro_reservoir( + gen_name::AbstractString, + d::Dict, + sys_mbase::Float64, + ids::IDGenerator, +) + mbase = _resolve_mbase(d, sys_mbase, gen_name) + base_conversion = sys_mbase / mbase + return Dict{String, Any}( + "id" => getid!(ids, :HydroDispatch, d["index"]), + "name" => String(gen_name), + "available" => Bool(d["gen_status"]), + "bus" => getid!(ids, :ACBus, d["gen_bus"]), + "active_power" => d["pg"] * base_conversion, + "reactive_power" => d["qg"] * base_conversion, + "rating" => _calculate_gen_rating(d["pmax"], d["qmax"], base_conversion), + "prime_mover_type" => _normalize_prime_mover(d["type"]), + "active_power_limits" => + _min_max_dict(d["pmin"] * base_conversion, d["pmax"] * base_conversion), + "reactive_power_limits" => + _min_max_dict(d["qmin"] * base_conversion, d["qmax"] * base_conversion), + "ramp_limits" => _calculate_ramp_limit_dict(d, gen_name), + "time_limits" => nothing, + "operation_cost" => _zero_hydro_generation_cost_dict(), + "base_power" => mbase, + ) +end + +""" +Translate one PowerModels generator row tagged for renewable-dispatch into +a RenewableDispatch-shaped dict. Computed rating capped at `mbase`. +""" +function make_renewable_dispatch( + gen_name::AbstractString, + d::Dict, + sys_mbase::Float64, + ids::IDGenerator, +) + mbase = _resolve_mbase(d, sys_mbase, gen_name) + base_conversion = sys_mbase / mbase + rating = _calculate_gen_rating(d["pmax"], d["qmax"], base_conversion) + if rating > mbase + @warn "rating is larger than base power for $gen_name, setting to $mbase" + rating = mbase + end + return Dict{String, Any}( + "id" => getid!(ids, :RenewableDispatch, d["index"]), + "name" => String(gen_name), + "available" => Bool(d["gen_status"]), + "bus" => getid!(ids, :ACBus, d["gen_bus"]), + "active_power" => d["pg"] * base_conversion, + "reactive_power" => d["qg"] * base_conversion, + "rating" => rating * base_conversion, + "prime_mover_type" => _normalize_prime_mover(d["type"]), + "reactive_power_limits" => + _min_max_dict(d["qmin"] * base_conversion, d["qmax"] * base_conversion), + "power_factor" => 1.0, + "operation_cost" => _zero_renewable_generation_cost_dict(), + "base_power" => mbase, + ) +end + +""" +Translate one PowerModels generator row tagged for renewable-non-dispatch +into a RenewableNonDispatch-shaped dict. +""" +function make_renewable_non_dispatch( + gen_name::AbstractString, + d::Dict, + sys_mbase::Float64, + ids::IDGenerator, +) + mbase = _resolve_mbase(d, sys_mbase, gen_name) + base_conversion = sys_mbase / mbase + return Dict{String, Any}( + "id" => getid!(ids, :RenewableNonDispatch, d["index"]), + "name" => String(gen_name), + "available" => Bool(d["gen_status"]), + "bus" => getid!(ids, :ACBus, d["gen_bus"]), + "active_power" => d["pg"] * base_conversion, + "reactive_power" => d["qg"] * base_conversion, + "rating" => float(d["pmax"]) * base_conversion, + "prime_mover_type" => _normalize_prime_mover(d["type"]), + "power_factor" => 1.0, + "base_power" => mbase, + ) +end + +""" +Emit warnings for generator rows whose `pmin`/`pmax`/`pg` values look more +like a motor load than a generator. Mirrors PSY's depreciated helper +`_is_likely_motor_load`. + +This is purely diagnostic — the row still gets parsed as a ThermalStandard- +shaped dict with negative active-power limits. The warning text tells the +user they can convert the entry to a MotorLoad-shaped dict downstream if +more accurate motor modeling is desired. +""" +function _is_likely_motor_load(d::Dict, gen_name::AbstractString) + if d["pmin"] < 0 && d["pmax"] < 0 && d["pg"] < 0 + @warn "Generator $gen_name is likely a motor load with negative active power: $(d["pg"]) and negative power limits: (min = $(d["pmin"]), max = $(d["pmax"])) \ + this component will be parsed as a thermal generator with negative active power limits. You can convert the device to a MotorLoad for more accurate modeling." + end + if d["pmin"] == 0 && d["pmax"] == 0 && d["pg"] < 0 + @warn "Generator $gen_name is likely a motor load with negative active power: $(d["pg"]) and undefined active power limits \ + this component will be parsed as a thermal generator with negative active power injection. You can convert the device to a MotorLoad for more accurate modeling." + end + if d["pmin"] < 0 && d["pmax"] == 0 + @warn "Generator $gen_name is likely something that is not a ThermalGenerators with negative power limits: (min = $(d["pmin"]), max = $(d["pmax"])) \ + this component will be parsed as a thermal generator with negative active power limits. Check this entry for more accurate modeling." + end + return nothing +end + +""" +Translate the `mpc.gencost` attachment on a thermal generator row into a +`(variable_cost_dict, fixed_cost)` pair. + +Output dict shapes mirror `sienna_to_json/common.jl`: + - `get_variable_cost(::CostCurve)` for the outer `CostCurve` dict, + - `get_value_curve(::InputOutputCurve)` for the `value_curve` wrapper, + - `get_function_data(::PiecewiseLinearData|::QuadraticFunctionData)` for + the `function_data` payload. + +Two MATPOWER cost models are supported: + - `model == 1` → piecewise-linear cost. `d["cost"]` is the interleaved + `[p1, c1, p2, c2, ...]` MATPOWER layout. + - `model == 2` → polynomial cost with the highest-degree coefficient + first. +""" +function _thermal_variable_cost_and_fixed(d::Dict, sys_mbase::Float64) + model = Int(d["model"]) + if model == 1 + cost_component = d["cost"] + power_p = [v for (i, v) in enumerate(cost_component) if isodd(i)] + cost_p = [v for (i, v) in enumerate(cost_component) if iseven(i)] + points = collect(zip(float.(power_p), float.(cost_p))) + # FLAG: single-point piecewise will BoundsError on points[2] below. + first_x, first_y = points[1] + second_x, second_y = points[2] + first_slope = (second_y - first_y) / (second_x - first_x) + fixed = max(0.0, first_y - first_slope * first_x) + adjusted_points = + [Dict{String, Any}("x" => x, "y" => y - fixed) for (x, y) in points] + function_data = Dict{String, Any}( + "function_type" => "PIECEWISE_LINEAR", + "points" => adjusted_points, + ) + elseif model == 2 + coeffs = Dict{Int, Float64}() + for (i, c) in enumerate(reverse(float.(d["cost"][1:(end - 1)]))) + coeffs[i] = c / sys_mbase^i + end + if !(keys(coeffs) ⊆ Set((0, 1, 2))) + throw( + ArgumentError( + "Can only handle polynomials up to degree two; given coefficients $coeffs", + ), + ) + end + function_data = Dict{String, Any}( + "function_type" => "QUADRATIC", + "quadratic_term" => get(coeffs, 2, 0.0), + "proportional_term" => get(coeffs, 1, 0.0), + "constant_term" => get(coeffs, 0, 0.0), + ) + fixed = (get(d, "ncost", 0) >= 1) ? float(last(d["cost"])) : 0.0 + else + throw(ArgumentError("Unrecognized mpc.gencost model code: $model")) + end + + # FLAG: power_units = "DEVICE_BASE" intentionally diverges from the + # placeholder dicts, which use "NATURAL_UNITS". DEVICE_BASE matches PSY + # 997 (`CostCurve(InputOutputCurve(...), UnitSystem.DEVICE_BASE)`); the + # divisor (sys_mbase^i for polynomial, implicit p.u. axis for piecewise) + # is what makes DEVICE_BASE the right tag for real data. If the parser + # ever produces NATURAL_UNITS-scaled coefficients here, this string must + # change in lockstep. + variable_dict = Dict{String, Any}( + "variable_cost_type" => "COST", + "power_units" => "DEVICE_BASE", + "value_curve" => Dict{String, Any}( + "curve_type" => "INPUT_OUTPUT", + "function_data" => function_data, + ), + ) + return variable_dict, fixed +end + +""" +Translate one PowerModels generator row tagged for thermal into a +ThermalStandard-shaped dict. + +The PSY-side `ext` fields (`r`/`x`/`rt`/`xt` source impedances) have no +home in the OpenAPI `ThermalStandard` schema and are dropped. +""" +function make_thermal_gen( + gen_name::AbstractString, + d::Dict, + sys_mbase::Float64, + ids::IDGenerator, +) + mbase = _resolve_mbase(d, sys_mbase, gen_name) + base_conversion = sys_mbase / mbase + _is_likely_motor_load(d, gen_name) + + if haskey(d, "model") + variable_dict, fixed = _thermal_variable_cost_and_fixed(d, sys_mbase) + # FLAG: d["startup"]/d["shutdown"] are assumed to be scalar floats — + # true for MATPOWER's mpc.gencost (table B-4). PSS/E may not carry + # these on generator rows; if a PTI test case throws KeyError here, + # add a defensive `get(d, "startup", 0.0)` (and same for shutdown). + startup = float(d["startup"]) + shutdn = float(d["shutdown"]) + else + @warn "Generator cost data not included for Generator: $gen_name" + placeholder = _zero_thermal_generation_cost_dict() + variable_dict = placeholder["variable"] + fixed = placeholder["fixed"] + startup = placeholder["start_up"] + shutdn = placeholder["shut_down"] + end + + operation_cost = Dict{String, Any}( + "cost_type" => "THERMAL", + "variable" => variable_dict, + "fixed" => fixed, + "start_up" => startup, + "shut_down" => shutdn, + ) + + return Dict{String, Any}( + "id" => getid!(ids, :ThermalStandard, d["index"]), + "name" => String(gen_name), + "status" => Bool(d["gen_status"]), + "available" => Bool(d["gen_status"]), + "bus" => getid!(ids, :ACBus, d["gen_bus"]), + "active_power" => d["pg"] * base_conversion, + "reactive_power" => d["qg"] * base_conversion, + "rating" => _calculate_gen_rating(d["pmax"], d["qmax"], base_conversion), + "prime_mover_type" => _normalize_prime_mover(d["type"]), + "fuel_type" => _normalize_fuel(d["fuel"]), + "active_power_limits" => + _min_max_dict(d["pmin"] * base_conversion, d["pmax"] * base_conversion), + "reactive_power_limits" => + _min_max_dict(d["qmin"] * base_conversion, d["qmax"] * base_conversion), + "ramp_limits" => _calculate_ramp_limit_dict(d, gen_name), + "time_limits" => nothing, + "operation_cost" => operation_cost, + "base_power" => mbase, + ) +end + +""" +Translate one PowerModels generator row tagged for synchronous-condenser +into a SynchronousCondenser-shaped dict. + +The PSY-side `ext` (`r`/`x`/`rt`/`xt`) is dropped. +""" +function make_synchronous_condenser( + gen_name::AbstractString, + d::Dict, + sys_mbase::Float64, + ids::IDGenerator, +) + mbase = _resolve_mbase(d, sys_mbase, gen_name) + base_conversion = sys_mbase / mbase + return Dict{String, Any}( + "id" => getid!(ids, :SynchronousCondenser, d["index"]), + "name" => String(gen_name), + "available" => Bool(d["gen_status"]), + "bus" => getid!(ids, :ACBus, d["gen_bus"]), + "reactive_power" => d["qg"] * base_conversion, + "rating" => max(abs(d["qmax"]), abs(d["qmin"])) * base_conversion, + "reactive_power_limits" => + _min_max_dict(d["qmin"] * base_conversion, d["qmax"] * base_conversion), + "base_power" => mbase, + ) +end + +""" +For each generator, return a NamedTuple of per-OpenAPI-type sub-collections. + +Dispatch is driven by `generator_mapping_pm.yaml`, storage is handled by `read_storage!`. + +# Returns + +`(; thermal_standard, hydro_dispatch, renewable_dispatch, renewable_non_dispatch, +synchronous_condenser)` — each field is a homogeneous `Dict{String, +Dict{String, Any}}` keyed by generator name. +""" +function read_gen!(pm_data::PowerModelsData, ids::IDGenerator; kwargs...) + @info "Reading generator data" + data = pm_data.data + thermal_standard = Dict{String, Dict{String, Any}}() + hydro_dispatch = Dict{String, Dict{String, Any}}() + renewable_dispatch = Dict{String, Dict{String, Any}}() + renewable_non_dispatch = Dict{String, Dict{String, Any}}() + synchronous_condenser = Dict{String, Dict{String, Any}}() + if !haskey(data, "gen") + @error "There are no Generators in this file" + return (; + thermal_standard, + hydro_dispatch, + renewable_dispatch, + renewable_non_dispatch, + synchronous_condenser, + ) + end + + raw_mapping = get(kwargs, :generator_mapping, _GENERATOR_MAPPING_FILE) + mapping = if raw_mapping isa AbstractString + try + _get_generator_mapping(String(raw_mapping)) + catch e + @error "Error loading generator mapping $(raw_mapping)" + rethrow(e) + end + else + raw_mapping + end + + sys_mbase = float(data["baseMVA"]) + _get_name = get(kwargs, :gen_name_formatter, _get_pm_dict_name) + + for (_, pm_gen) in data["gen"] + gen_name = String(_get_name(pm_gen)) + pm_gen["fuel"] = get(pm_gen, "fuel", "OTHER") + pm_gen["type"] = get(pm_gen, "type", "OT") + + gen_type = _get_generator_type(pm_gen["fuel"], pm_gen["type"], mapping) + generator, bucket = if gen_type === :ThermalStandard + make_thermal_gen(gen_name, pm_gen, sys_mbase, ids), thermal_standard + elseif gen_type === :HydroDispatch + make_hydro_dispatch(gen_name, pm_gen, sys_mbase, ids), hydro_dispatch + elseif gen_type === :HydroTurbine + make_hydro_reservoir(gen_name, pm_gen, sys_mbase, ids), hydro_dispatch + elseif gen_type === :RenewableDispatch + make_renewable_dispatch(gen_name, pm_gen, sys_mbase, ids), renewable_dispatch + elseif gen_type === :RenewableNonDispatch + make_renewable_non_dispatch(gen_name, pm_gen, sys_mbase, ids), + renewable_non_dispatch + elseif gen_type === :SynchronousCondenser + make_synchronous_condenser(gen_name, pm_gen, sys_mbase, ids), + synchronous_condenser + elseif gen_type === :EnergyReservoirStorage + @warn "EnergyReservoirStorage should be defined as a PowerModels storage... Skipping" + continue + else + @error "Skipping unsupported generator" gen_type + continue + end + + haskey(bucket, gen_name) && throw( + DataFormatError( + "Found duplicate generator name $gen_name; consider passing a `gen_name_formatter` kwarg", + ), + ) + bucket[gen_name] = generator + end + return (; + thermal_standard, + hydro_dispatch, + renewable_dispatch, + renewable_non_dispatch, + synchronous_condenser, + ) +end + +# ============================================================================= +# Branches +# ============================================================================= + +""" +Set `d[group_key]` to the canonical OpenAPI `winding_group_number` +string for the phase-shift angle (in radians) stored at `d[angle_key]`. +""" +function _add_vector_control_group!(d::Dict, angle_key::AbstractString, group_key::AbstractString) + angle = d[angle_key] + for (deg, group) in _SHIFT_TO_GROUP_MAP + if isapprox(rad2deg(angle), deg) + d[group_key] = group + return + end + end + d[group_key] = "UNDEFINED" + return +end + +""" +Map PSS/E `COD1`/`COD2`/`COD3` control codes to a +`(is_tap_controllable, is_alpha_controllable)` pair. +""" +function _determine_control_modes(d::Dict, control_flag::AbstractString, tap_key::AbstractString) + control_code = get(d, control_flag, -99) + tap = d[tap_key] + + is_tap_controllable = false + is_alpha_controllable = false + + if control_code == 0 + # No control + elseif control_code ∈ (1, -1, 2, -2) + # Reactive Power Control / Voltage Control + is_tap_controllable = true + elseif control_code ∈ (3, -3, 4, -4, 5, -5) + # Active Power / DC-Line / Asymmetric-Active-Power Control + is_tap_controllable = true + is_alpha_controllable = true + elseif control_code == -99 + @warn "Can't determine control objective for the transformer from the $(control_flag) field for $d" + if d["shift"] != 0.0 + is_alpha_controllable = true + elseif (tap != 0.0) || (tap != 1.0) + is_tap_controllable = true + else + @warn "Can't determine control objective for the other fields. Will return a TwoWindingTransformer" + end + else + error(d) + end + return is_tap_controllable, is_alpha_controllable +end + +""" +Build the canonical PowerModels branch name. +""" +function _get_pm_branch_name(device_dict::Dict, bus_f_dict::Dict, bus_t_dict::Dict) + if haskey(device_dict, "name") + index = device_dict["name"] + elseif device_dict["source_id"][1] == "branch" && + length(device_dict["source_id"]) > 2 + index = strip(device_dict["source_id"][4]) + elseif ( + device_dict["source_id"][1] == "switch" || + device_dict["source_id"][1] == "breaker" + ) && length(device_dict["source_id"]) > 2 + index = string(device_dict["source_id"][4][2]) + elseif device_dict["source_id"][1] == "transformer" && + length(device_dict["source_id"]) > 3 + index = strip(device_dict["source_id"][5]) + else + index = device_dict["index"] + end + return "$(bus_f_dict["name"])-$(bus_t_dict["name"])-i_$index" +end + +""" +Return `true` if `device_dict` was synthesized from a PSS/E branch-shaped +section (branch/switch/breaker/transformer). +""" +function _is_psse_branch_source_id(device_dict::Dict) + if !haskey(device_dict, "source_id") || isempty(device_dict["source_id"]) + return false + end + return device_dict["source_id"][1] in ("branch", "switch", "breaker", "transformer") +end + +""" +PSS/E-specific branch naming that disambiguates parallel branches between +the same bus pair with a per-pair counter. +""" +function _get_pm_branch_name_with_counter!( + device_dict::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + branch_pair_counts::Dict{Tuple{String, String}, Int}, +) + if _is_psse_branch_source_id(device_dict) + pair_key = (String(bus_f_dict["name"]), String(bus_t_dict["name"])) + branch_pair_counts[pair_key] = get(branch_pair_counts, pair_key, 0) + 1 + index = branch_pair_counts[pair_key] + return "$(pair_key[1])-$(pair_key[2])-i_$index" + end + return _get_pm_branch_name(device_dict, bus_f_dict, bus_t_dict) +end + +""" +Resolve a branch rating field. +""" +function _get_rating( + branch_type::AbstractString, + name::AbstractString, + line_data::Dict, + key::AbstractString, +) + haskey(line_data, key) || return key == "rate_a" ? INFINITE_BOUND : nothing + if isapprox(line_data[key], 0.0) + @info "$branch_type $name rating value: $(line_data[key]). Unbounded value implied as per PSSe Manual" + return INFINITE_BOUND + end + return line_data[key] +end + +""" +MATPOWER branch-type dispatcher. +""" +function get_branch_type_matpower(d::Dict) + tap = d["tap"] + shift = d["shift"] + is_transformer = d["transformer"] + if !is_transformer + is_transformer = (tap != 0.0) && (tap != 1.0) || (shift != 0.0) + end + is_transformer || return :Line + + _add_vector_control_group!(d, "shift", "group_number") + if d["group_number"] == "UNDEFINED" + return :PhaseShiftingTransformer + elseif tap != 1.0 + return :TapTransformer + else + return :TwoWindingTransformer + end +end + +""" +PSS/E branch-type dispatcher. +""" +function get_branch_type_psse(d::Dict) + if d["br_r"] == 0.0 && d["br_x"] == 0.0 + return :DiscreteControlledACBranch + end + + is_transformer = d["transformer"] + tap = d["tap"] + if !is_transformer + if (tap != 0.0) && (tap != 1.0) + @warn "Transformer $d has tap ratio $tap, which is not 0.0 or 1.0; this is not a valid value for a Line. Parsing entry as a Transformer" + is_transformer = true + _add_vector_control_group!(d, "shift", "group_number") + else + return :Line + end + end + + _add_vector_control_group!(d, "shift", "group_number") + is_tap_controllable, is_alpha_controllable = _determine_control_modes(d, "COD1", "tap") + if d["group_number"] == "UNDEFINED" || is_alpha_controllable + return :PhaseShiftingTransformer + elseif (is_tap_controllable || (tap != 1.0)) && d["group_number"] != "UNDEFINED" + return :TapTransformer + elseif !is_tap_controllable && d["group_number"] != "UNDEFINED" + return :TwoWindingTransformer + else + error("Couldn't infer the branch type for branch $d") + end +end + +""" +Look up (or mint) the Arc id for an ordered bus pair, populating the +`arcs` accumulator on first sight. Parallel branches between the same +buses share an Arc id. +""" +function _get_or_mint_arc_id!( + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, + bus_f_id::Int, + bus_t_id::Int, +) + arc_id = getid!(ids, :Arc, (bus_f_id, bus_t_id)) + if !haskey(arcs, arc_id) + arcs[arc_id] = Dict{String, Any}( + "id" => arc_id, + "from_id" => bus_f_id, + "to_id" => bus_t_id, + ) + end + return arc_id +end + +""" +Resolve the `control_objective` field for a 2-winding transformer dict. +Supports formatters that return a plain integer COD code or a string +already in the OpenAPI enum vocabulary. Integer returns are normalized +via [`_normalize_control_objective`]. +""" +function _resolve_control_objective(d::Dict, name::AbstractString, formatter) + if formatter !== nothing + result = formatter(name) + if result !== nothing + return result isa AbstractString ? String(result) : + _normalize_control_objective(result) + end + end + return _normalize_control_objective(get(d, "COD1", -99)) +end + +""" +Translate one PowerModels branch row into a Line-shaped dict. +The PSY-side `ext` field is dropped: OpenAPI's Line schema has no `ext`. +""" +function make_line( + name::AbstractString, + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, +) + pf = get(d, "pf", 0.0) + qf = get(d, "qf", 0.0) + available_value = d["br_status"] == 1 + # PSY checks `get_bustype(bus) == ACBusTypes.ISOLATED` against the PSY + # enum; we compare against the canonical string emitted by `read_bus!`. + if bus_f_dict["bustype"] == "ISOLATED" || bus_t_dict["bustype"] == "ISOLATED" + available_value = false + end + + arc_id = + _get_or_mint_arc_id!(ids, arcs, Int(bus_f_dict["id"]), Int(bus_t_dict["id"])) + + return Dict{String, Any}( + "id" => getid!(ids, :Line, d["index"]), + "name" => String(name), + "available" => available_value, + "active_power_flow" => pf, + "reactive_power_flow" => qf, + "arc" => arc_id, + "r" => d["br_r"], + "x" => d["br_x"], + "b" => Dict{String, Any}("from" => d["b_fr"], "to" => d["b_to"]), + "rating" => _get_rating("Line", name, d, "rate_a"), + "rating_b" => _get_rating("Line", name, d, "rate_b"), + "rating_c" => _get_rating("Line", name, d, "rate_c"), + "angle_limits" => _min_max_dict(d["angmin"], d["angmax"]), + ) +end + +""" +Translate one PowerModels switch/breaker row into a DiscreteControlledACBranch- +shaped dict. + +Called by [`read_switch_breaker!`] when explicit switch/breaker sections +are present in PSS/E. The zero-impedance fallback path for plain branches +is handled by [`_make_switch_from_zero_impedance_line`] instead. +""" +function make_switch_breaker( + name::AbstractString, + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + section::AbstractString, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, +) + arc_id = + _get_or_mint_arc_id!(ids, arcs, Int(bus_f_dict["id"]), Int(bus_t_dict["id"])) + state = Int(d["state"]) + return Dict{String, Any}( + "id" => getid!(ids, :DiscreteControlledACBranch, (String(section), d["index"])), + "name" => String(name), + "available" => Bool(state), + "active_power_flow" => d["active_power_flow"], + "reactive_power_flow" => d["reactive_power_flow"], + "arc" => arc_id, + "r" => d["r"], + "x" => d["x"], + "rating" => d["rating"], + "discrete_branch_type" => + _DISCRETE_BRANCH_TYPE_MAP[Int(d["discrete_branch_type"])], + "branch_status" => _BRANCH_STATUS_MAP[state], + ) +end + +""" +Translate a zero-impedance branch row (PSS/E `br_r == 0 && br_x == 0`) into +a DiscreteControlledACBranch-shaped dict tagged as a SWITCH. +""" +function _make_switch_from_zero_impedance_line( + name::AbstractString, + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, +) + pf = get(d, "pf", 0.0) + qf = get(d, "qf", 0.0) + available_value = d["br_status"] == 1 + if bus_f_dict["bustype"] == "ISOLATED" || bus_t_dict["bustype"] == "ISOLATED" + available_value = false + end + status_value = available_value ? "CLOSED" : "OPEN" + + @warn "Branch $name has zero impedance and available = $available_value; converting + to a DiscreteControlledACBranch of type SWITCH with available = $available_value + and branch_status = $status_value" + + arc_id = + _get_or_mint_arc_id!(ids, arcs, Int(bus_f_dict["id"]), Int(bus_t_dict["id"])) + return Dict{String, Any}( + # Tag the section as "branch" so the id can't collide with switches + # or breakers parsed by `read_switch_breaker!`. + "id" => getid!(ids, :DiscreteControlledACBranch, ("branch", d["index"])), + "name" => String(name), + "available" => Bool(available_value), + "active_power_flow" => pf, + "reactive_power_flow" => qf, + "arc" => arc_id, + "r" => d["br_r"], + "x" => d["br_x"], + "rating" => _get_rating("Line", name, d, "rate_a"), + "discrete_branch_type" => "SWITCH", + "branch_status" => status_value, + ) +end + +""" +Build the TransformerCircuit dict shared by all three 2W transformer +variants. `tap`, `alpha`, and `control_objective` are optional; the caller +sets them via keyword args when the variant needs them. +""" +function _make_transformer_2w_circuit( + d::Dict, + arc_id::Int, + available_value::Bool, + circuit_id::Int; + tap::Union{Float64, Nothing} = nothing, + alpha::Union{Float64, Nothing} = nothing, + control_objective::Union{String, Nothing} = nothing, + rating_label::AbstractString = "TransformerCircuit", + name::AbstractString = "", +) + return Dict{String, Any}( + "id" => circuit_id, + "available" => available_value, + "arc" => arc_id, + "tap" => tap, + "alpha" => alpha, + "r" => d["br_r"], + "x" => d["br_x"], + "control_objective" => control_objective, + "rating" => _get_rating(rating_label, name, d, "rate_a"), + "rating_b" => _get_rating(rating_label, name, d, "rate_b"), + "rating_c" => _get_rating(rating_label, name, d, "rate_c"), + "active_power_flow" => get(d, "pf", 0.0), + "reactive_power_flow" => get(d, "qf", 0.0), + "base_power" => d["base_power"], + # PSS/E base voltages may differ from each bus's nominal base_kv — + # the upstream parser pulls these from the transformer winding + # records, not the bus rows. + "base_voltage_primary" => d["base_voltage_from"], + "base_voltage_secondary" => d["base_voltage_to"], + ) +end + +""" +Build the TwoWindingTransformer holder dict. `magnetizing_shunt` carries +the shunt originally attached to the "from" side of the branch (PSY's +`primary_shunt` field). `winding_group_number` still travels here for +downstream consumers that need it; POM's schema ignores unknown keys. +""" +function _make_two_winding_transformer_holder( + xfrm_id::Int, + name::AbstractString, + circuit_id::Int, + d::Dict, +) + return Dict{String, Any}( + "id" => xfrm_id, + "name" => String(name), + "circuit" => circuit_id, + "magnetizing_shunt" => _complex_to_dict(d["g_fr"] + d["b_fr"] * im), + "winding_group_number" => d["group_number"], + ) +end + +""" +Common preamble for every 2W emitter: resolve availability, mint the arc, +mint the circuit id, push the TransformerCircuit dict into the accumulator, +and mint the holder id. Returns `(xfrm_id, circuit_id, arc_id, available_value)`. +""" +function _prepare_2w_ids!( + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, +) + available_value = d["br_status"] == 1 + if bus_f_dict["bustype"] == "ISOLATED" || bus_t_dict["bustype"] == "ISOLATED" + available_value = false + end + arc_id = + _get_or_mint_arc_id!(ids, arcs, Int(bus_f_dict["id"]), Int(bus_t_dict["id"])) + xfrm_id = getid!(ids, :TwoWindingTransformer, d["index"]) + circuit_id = getid!(ids, :TransformerCircuit, (:TwoWinding, d["index"])) + return xfrm_id, circuit_id, arc_id, available_value +end + +""" +Translate one PowerModels branch row into a `(holder_dict, circuit_dict)` +pair — a TwoWindingTransformer + its TransformerCircuit. The circuit +carries r/x, ratings, and flows; the holder is the named entity that owns +the electricals via FK. +""" +function make_transformer_2w( + name::AbstractString, + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, + transformer_circuits::Dict{Int, Dict{String, Any}}; + kwargs..., +) + xfrm_id, circuit_id, arc_id, available_value = + _prepare_2w_ids!(d, bus_f_dict, bus_t_dict, ids, arcs) + + circuit = _make_transformer_2w_circuit( + d, + arc_id, + available_value, + circuit_id; + rating_label = "TwoWindingTransformer", + name = name, + ) + transformer_circuits[circuit_id] = circuit + + return _make_two_winding_transformer_holder(xfrm_id, name, circuit_id, d) +end + +""" +Translate one PowerModels branch row into a `(holder, circuit)` pair for +the tap-controlled variant — same shape as [`make_transformer_2w`] but +with `tap` and `control_objective` populated on the circuit. +""" +function make_tap_transformer( + name::AbstractString, + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, + transformer_circuits::Dict{Int, Dict{String, Any}}; + kwargs..., +) + xfrm_id, circuit_id, arc_id, available_value = + _prepare_2w_ids!(d, bus_f_dict, bus_t_dict, ids, arcs) + + control_objective = _resolve_control_objective( + d, + name, + get(kwargs, :transformer_control_objective_formatter, nothing), + ) + + circuit = _make_transformer_2w_circuit( + d, + arc_id, + available_value, + circuit_id; + tap = d["tap"], + control_objective = control_objective, + rating_label = "TapTransformer", + name = name, + ) + transformer_circuits[circuit_id] = circuit + + return _make_two_winding_transformer_holder(xfrm_id, name, circuit_id, d) +end + +""" +Translate one PowerModels branch row into a `(holder, circuit)` pair for +the phase-shifting variant — `tap`, `alpha` (from `d["shift"]`), and +`control_objective` all populated on the circuit. +""" +function make_phase_shifting_transformer( + name::AbstractString, + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, + transformer_circuits::Dict{Int, Dict{String, Any}}; + kwargs..., +) + xfrm_id, circuit_id, arc_id, available_value = + _prepare_2w_ids!(d, bus_f_dict, bus_t_dict, ids, arcs) + + control_objective = _resolve_control_objective( + d, + name, + get(kwargs, :transformer_control_objective_formatter, nothing), + ) + + circuit = _make_transformer_2w_circuit( + d, + arc_id, + available_value, + circuit_id; + tap = d["tap"], + alpha = d["shift"], + control_objective = control_objective, + rating_label = "PhaseShiftingTransformer", + name = name, + ) + transformer_circuits[circuit_id] = circuit + + return _make_two_winding_transformer_holder(xfrm_id, name, circuit_id, d) +end + +""" +Branch dict constructor. The `branch_type` Symbol is resolved upstream +(`read_branch!`) so the caller can use the same value to gate the ICT +attachment loop. + +The three 2W transformer variants (`:TwoWindingTransformer`, +`:TapTransformer`, `:PhaseShiftingTransformer`) all return a +`TwoWindingTransformer` holder dict and push their `TransformerCircuit` +into `transformer_circuits` — the variant determines which circuit fields +(tap, alpha, control_objective) get populated, not which OpenAPI type the +holder is. +""" +function make_branch( + name::AbstractString, + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + branch_type::Symbol, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, + transformer_circuits::Dict{Int, Dict{String, Any}}; + kwargs..., +) + if branch_type === :Line + return make_line(name, d, bus_f_dict, bus_t_dict, ids, arcs) + elseif branch_type === :TwoWindingTransformer + return make_transformer_2w( + name, d, bus_f_dict, bus_t_dict, ids, arcs, transformer_circuits; kwargs..., + ) + elseif branch_type === :TapTransformer + return make_tap_transformer( + name, d, bus_f_dict, bus_t_dict, ids, arcs, transformer_circuits; kwargs..., + ) + elseif branch_type === :PhaseShiftingTransformer + return make_phase_shifting_transformer( + name, d, bus_f_dict, bus_t_dict, ids, arcs, transformer_circuits; kwargs..., + ) + elseif branch_type === :DiscreteControlledACBranch + return _make_switch_from_zero_impedance_line( + name, + d, + bus_f_dict, + bus_t_dict, + ids, + arcs, + ) + end + + @error "Skipping branch $name: type $branch_type not yet implemented" + return nothing +end + +""" +For each branch return a NamedTuple of per-OpenAPI-type sub-collections. +Populates `arcs` with one Arc dict per unique ordered bus pair that +appears as a branch endpoint; parallel branches share the same Arc id. +Populates `transformer_circuits` with one TransformerCircuit dict per 2W +transformer holder (all three 2W variants — plain, tap, phase-shifting — +share this table). + +# Returns + +`(; line, two_winding_transformer, discrete_controlled_ac_branch)` — each +field is a homogeneous `Dict{String, Dict{String, Any}}` keyed by branch +name. All 2W transformer variants collapse into `two_winding_transformer`; +their electricals live on the separately-accumulated +`transformer_circuits`. `discrete_controlled_ac_branch` captures only the +zero-impedance switch path emitted from `read_branch!`; switches and +breakers parsed by [`read_switch_breaker!`] live in their own separate +collections. + +**ICT attachment**: when called with non-empty `ict_instances` (from +[`read_impedance_correction!`]) and a +`supplemental_attribute_associations` accumulator, each 2W transformer +gets one ICT association row appended. The accumulator is mutated in +place; pass the same Vector to [`read_3w_transformer!`] so all +transformer↔ICT links collect into one collection. +""" +function read_branch!( + pm_data::PowerModelsData, + ids::IDGenerator, + bus_dicts::Dict{Int, Dict{String, Any}}, + arcs::Dict{Int, Dict{String, Any}}, + transformer_circuits::Dict{Int, Dict{String, Any}}; + ict_instances::Dict{Tuple{Int, String}, Dict{String, Any}} = + Dict{Tuple{Int, String}, Dict{String, Any}}(), + supplemental_attribute_associations::Vector{Dict{String, Any}} = + Dict{String, Any}[], + kwargs..., +) + @info "Reading branch data" + data = pm_data.data + line = Dict{String, Dict{String, Any}}() + two_winding_transformer = Dict{String, Dict{String, Any}}() + discrete_controlled_ac_branch = Dict{String, Dict{String, Any}}() + if !haskey(data, "branch") + @info "There is no Branch data in this file" + return (; line, two_winding_transformer, discrete_controlled_ac_branch) + end + + _get_name = get(kwargs, :branch_name_formatter, nothing) + branch_pair_counts = Dict{Tuple{String, String}, Int}() + source_type = data["source_type"] + + # All three 2W variants share this bucket. TransformerCircuit fields + # differ by variant, but the holder dict shape is the same. + _is_2w_variant(t::Symbol) = + t === :TwoWindingTransformer || + t === :TapTransformer || + t === :PhaseShiftingTransformer + + for d in values(data["branch"]) + bus_f_dict = bus_dicts[d["f_bus"]] + bus_t_dict = bus_dicts[d["t_bus"]] + name = if isnothing(_get_name) + if source_type == "pti" + _get_pm_branch_name_with_counter!( + d, + bus_f_dict, + bus_t_dict, + branch_pair_counts, + ) + else + _get_pm_branch_name(d, bus_f_dict, bus_t_dict) + end + else + _get_name(d, bus_f_dict, bus_t_dict) + end + name = String(name) + + branch_type = if source_type == "matpower" + get_branch_type_matpower(d) + elseif source_type == "pti" + get_branch_type_psse(d) + else + error("Source Type $source_type not supported") + end + if d["transformer"] && branch_type === :Line + throw( + DataFormatError( + "Branch data mismatched, cannot build the branch correctly for $d", + ), + ) + end + + branch = make_branch( + name, + d, + bus_f_dict, + bus_t_dict, + branch_type, + ids, + arcs, + transformer_circuits; + kwargs..., + ) + isnothing(branch) && continue + + bucket = if branch_type === :Line + line + elseif _is_2w_variant(branch_type) + two_winding_transformer + elseif branch_type === :DiscreteControlledACBranch + discrete_controlled_ac_branch + else + error("Unexpected branch_type $branch_type produced a non-nothing dict") + end + + haskey(bucket, name) && throw( + DataFormatError( + "Found duplicate branch name $name; consider passing a `branch_name_formatter` kwarg", + ), + ) + bucket[name] = branch + + if _is_2w_variant(branch_type) + _attach_impedance_correction_tables!( + branch, + d, + ict_instances, + supplemental_attribute_associations; + is_3w = false, + ) + end + end + return (; line, two_winding_transformer, discrete_controlled_ac_branch) +end + +# ============================================================================= +# 3-Winding Transformers +# ============================================================================= + +""" +Build the canonical 3W transformer name +""" +function _get_pm_3w_name( + device_dict::Dict, + bus_primary_dict::Dict, + bus_secondary_dict::Dict, + bus_tertiary_dict::Dict, +) + ckt = device_dict["circuit"] + return "$(bus_primary_dict["name"])-$(bus_secondary_dict["name"])-$(bus_tertiary_dict["name"])-i_$ckt" +end + +""" +Dispatcher returns `:ThreeWindingTransformer` or `:PhaseShiftingTransformer3W`. +""" +function get_three_winding_transformer_type(d::Dict) + _add_vector_control_group!(d, "primary_phase_shift_angle", "primary_group_number") + _add_vector_control_group!(d, "secondary_phase_shift_angle", "secondary_group_number") + _add_vector_control_group!(d, "tertiary_phase_shift_angle", "tertiary_group_number") + _, primary_is_alpha_controllable = + _determine_control_modes(d, "COD1", "primary_turns_ratio") + _, secondary_is_alpha_controllable = + _determine_control_modes(d, "COD2", "secondary_turns_ratio") + _, tertiary_is_alpha_controllable = + _determine_control_modes(d, "COD3", "tertiary_turns_ratio") + if d["primary_group_number"] == "UNDEFINED" || + d["secondary_group_number"] == "UNDEFINED" || + d["tertiary_group_number"] == "UNDEFINED" || + primary_is_alpha_controllable || + secondary_is_alpha_controllable || + tertiary_is_alpha_controllable + return :PhaseShiftingTransformer3W + else + return :ThreeWindingTransformer + end +end + +""" +Mint the three primary-↔-star, secondary-↔-star, tertiary-↔-star Arc ids +for a 3W transformer and append the new arcs into the accumulator. +""" +function _mint_3w_star_arcs!( + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, + primary_id::Int, + secondary_id::Int, + tertiary_id::Int, + star_id::Int, +) + return ( + _get_or_mint_arc_id!(ids, arcs, primary_id, star_id), + _get_or_mint_arc_id!(ids, arcs, secondary_id, star_id), + _get_or_mint_arc_id!(ids, arcs, tertiary_id, star_id), + ) +end + +""" +Build a per-winding TransformerCircuit dict for one leg of a 3W +transformer. `alpha` populates only on the phase-shifting variant. +""" +function _make_3w_winding_circuit( + d::Dict, + name::AbstractString, + arc_id::Int, + circuit_id::Int, + winding::AbstractString, # "primary" | "secondary" | "tertiary" + rating_label::AbstractString; + alpha::Union{Float64, Nothing} = nothing, +) + cod_key = winding == "primary" ? "COD1" : winding == "secondary" ? "COD2" : "COD3" + return Dict{String, Any}( + "id" => circuit_id, + "available" => d["available_$winding"], + "arc" => arc_id, + "tap" => d["$(winding)_turns_ratio"], + "alpha" => alpha, + "r" => d["r_$winding"], + "x" => d["x_$winding"], + "control_objective" => _normalize_control_objective(get(d, cod_key, -99)), + "rating" => _get_rating(rating_label, name, d, "rating_$winding"), + "active_power_flow" => get(d, "pf", 0.0), + "reactive_power_flow" => get(d, "qf", 0.0), + "base_voltage_primary" => d["base_voltage_$winding"], + ) +end + +""" +Build the ThreeWindingTransformer holder dict — pairwise mutual impedances ++ base powers + star bus + three TransformerCircuit FKs. Field names track +POM's ThreeWindingTransformer struct (note: POM uses `r_31`/`x_31`, not +`r_13`/`x_13`). +""" +function _make_three_winding_transformer_holder( + xfrm_id::Int, + name::AbstractString, + primary_circuit_id::Int, + secondary_circuit_id::Int, + tertiary_circuit_id::Int, + star_id::Int, + d::Dict, +) + return Dict{String, Any}( + "id" => xfrm_id, + "name" => String(name), + "primary_circuit" => primary_circuit_id, + "secondary_circuit" => secondary_circuit_id, + "tertiary_circuit" => tertiary_circuit_id, + "star_bus" => star_id, + "r_12" => d["r_12"], + "x_12" => d["x_12"], + "r_23" => d["r_23"], + "x_23" => d["x_23"], + "r_31" => d["r_13"], + "x_31" => d["x_13"], + "base_power_12" => d["base_power_12"], + "base_power_23" => d["base_power_23"], + "base_power_31" => d["base_power_13"], + # Extra fields useful downstream; POM's from_json will ignore any + # that aren't in its struct. + "magnetizing_shunt" => _complex_to_dict(d["g"] + d["b"] * im), + "primary_group_number" => get(d, "primary_group_number", nothing), + "secondary_group_number" => get(d, "secondary_group_number", nothing), + "tertiary_group_number" => get(d, "tertiary_group_number", nothing), + ) +end + +""" +Common preamble for every 3W emitter: resolve star arcs and mint holder + +three circuit ids. Returns +`(xfrm_id, (primary_circuit_id, secondary_circuit_id, tertiary_circuit_id), +(primary_arc, secondary_arc, tertiary_arc), star_id)`. +""" +function _prepare_3w_ids!( + d::Dict, + bus_primary_dict::Dict, + bus_secondary_dict::Dict, + bus_tertiary_dict::Dict, + star_bus_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, +) + primary_id = Int(bus_primary_dict["id"]) + secondary_id = Int(bus_secondary_dict["id"]) + tertiary_id = Int(bus_tertiary_dict["id"]) + star_id = Int(star_bus_dict["id"]) + + star_arcs = + _mint_3w_star_arcs!(ids, arcs, primary_id, secondary_id, tertiary_id, star_id) + + xfrm_id = getid!(ids, :ThreeWindingTransformer, d["index"]) + circuit_ids = ( + getid!(ids, :TransformerCircuit, (:ThreeWinding, d["index"], :primary)), + getid!(ids, :TransformerCircuit, (:ThreeWinding, d["index"], :secondary)), + getid!(ids, :TransformerCircuit, (:ThreeWinding, d["index"], :tertiary)), + ) + return xfrm_id, circuit_ids, star_arcs, star_id +end + +""" +Translate one PowerModels 3W transformer row into a `(holder_dict, +[primary_circuit_dict, secondary_circuit_dict, tertiary_circuit_dict])` +pair. The four dicts are pushed into caller-owned collections; the +returned holder is the ThreeWindingTransformer entity keyed by name. +""" +function make_3w_transformer( + name::AbstractString, + d::Dict, + bus_primary_dict::Dict, + bus_secondary_dict::Dict, + bus_tertiary_dict::Dict, + star_bus_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, + transformer_circuits::Dict{Int, Dict{String, Any}}, +) + xfrm_id, (primary_cid, secondary_cid, tertiary_cid), + (primary_arc, secondary_arc, tertiary_arc), star_id = _prepare_3w_ids!( + d, + bus_primary_dict, + bus_secondary_dict, + bus_tertiary_dict, + star_bus_dict, + ids, + arcs, + ) + + transformer_circuits[primary_cid] = _make_3w_winding_circuit( + d, name, primary_arc, primary_cid, "primary", "ThreeWindingTransformer", + ) + transformer_circuits[secondary_cid] = _make_3w_winding_circuit( + d, name, secondary_arc, secondary_cid, "secondary", "ThreeWindingTransformer", + ) + transformer_circuits[tertiary_cid] = _make_3w_winding_circuit( + d, name, tertiary_arc, tertiary_cid, "tertiary", "ThreeWindingTransformer", + ) + + return _make_three_winding_transformer_holder( + xfrm_id, name, primary_cid, secondary_cid, tertiary_cid, star_id, d, + ) +end + +""" +Phase-shifting 3W variant. Same holder shape as [`make_3w_transformer`]; +each per-winding TransformerCircuit gets its `alpha` populated from +`d["_phase_shift_angle"]`. +""" +function make_3w_phase_shifting_transformer( + name::AbstractString, + d::Dict, + bus_primary_dict::Dict, + bus_secondary_dict::Dict, + bus_tertiary_dict::Dict, + star_bus_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, + transformer_circuits::Dict{Int, Dict{String, Any}}, +) + xfrm_id, (primary_cid, secondary_cid, tertiary_cid), + (primary_arc, secondary_arc, tertiary_arc), star_id = _prepare_3w_ids!( + d, + bus_primary_dict, + bus_secondary_dict, + bus_tertiary_dict, + star_bus_dict, + ids, + arcs, + ) + + transformer_circuits[primary_cid] = _make_3w_winding_circuit( + d, name, primary_arc, primary_cid, "primary", "PhaseShiftingTransformer3W"; + alpha = d["primary_phase_shift_angle"], + ) + transformer_circuits[secondary_cid] = _make_3w_winding_circuit( + d, name, secondary_arc, secondary_cid, "secondary", "PhaseShiftingTransformer3W"; + alpha = d["secondary_phase_shift_angle"], + ) + transformer_circuits[tertiary_cid] = _make_3w_winding_circuit( + d, name, tertiary_arc, tertiary_cid, "tertiary", "PhaseShiftingTransformer3W"; + alpha = d["tertiary_phase_shift_angle"], + ) + + return _make_three_winding_transformer_holder( + xfrm_id, name, primary_cid, secondary_cid, tertiary_cid, star_id, d, + ) +end + +""" +For each 3-winding transformer return a NamedTuple with one field. +Populates `transformer_circuits` with three TransformerCircuit dicts per +3W transformer (one per winding). Both plain and phase-shifting variants +share the ThreeWindingTransformer holder collection; the variant +determines whether the per-winding circuits carry an `alpha`. + +# Returns + +`(; three_winding_transformer,)` — a homogeneous +`Dict{String, Dict{String, Any}}` keyed by transformer name. + +3W transformers are PSS/E-only; MATPOWER files have no +`data["3w_transformer"]` section and this function returns an empty dict. + +**ICT attachment**: when called with non-empty `ict_instances` (from +[`read_impedance_correction!`]) and a +`supplemental_attribute_associations` accumulator, each 3W transformer +gets one ICT association row per winding (primary/secondary/tertiary) +that references a correction table. +""" +function read_3w_transformer!( + pm_data::PowerModelsData, + ids::IDGenerator, + bus_dicts::Dict{Int, Dict{String, Any}}, + arcs::Dict{Int, Dict{String, Any}}, + transformer_circuits::Dict{Int, Dict{String, Any}}; + ict_instances::Dict{Tuple{Int, String}, Dict{String, Any}} = + Dict{Tuple{Int, String}, Dict{String, Any}}(), + supplemental_attribute_associations::Vector{Dict{String, Any}} = + Dict{String, Any}[], + kwargs..., +) + @info "Reading 3W transformer data" + data = pm_data.data + three_winding_transformer = Dict{String, Dict{String, Any}}() + if !haskey(data, "3w_transformer") + @info "There is no 3W transformer data in this file" + return (; three_winding_transformer) + end + + _get_name = get(kwargs, :xfrm_3w_name_formatter, _get_pm_3w_name) + + for (_, d) in data["3w_transformer"] + bus_primary = bus_dicts[d["bus_primary"]] + bus_secondary = bus_dicts[d["bus_secondary"]] + bus_tertiary = bus_dicts[d["bus_tertiary"]] + star_bus = bus_dicts[d["star_bus"]] + + name = String(_get_name(d, bus_primary, bus_secondary, bus_tertiary)) + xfrm_type = get_three_winding_transformer_type(d) + + xfrm = if xfrm_type === :PhaseShiftingTransformer3W + make_3w_phase_shifting_transformer( + name, + d, + bus_primary, + bus_secondary, + bus_tertiary, + star_bus, + ids, + arcs, + transformer_circuits, + ) + elseif xfrm_type === :ThreeWindingTransformer + make_3w_transformer( + name, + d, + bus_primary, + bus_secondary, + bus_tertiary, + star_bus, + ids, + arcs, + transformer_circuits, + ) + else + error("Unsupported three winding transformer type $xfrm_type") + end + + haskey(three_winding_transformer, name) && throw( + DataFormatError( + "Found duplicate 3W transformer name $name; consider passing an `xfrm_3w_name_formatter` kwarg", + ), + ) + three_winding_transformer[name] = xfrm + + # Per PSY 1781: every 3W transformer (both plain and phase-shifting) + # gets ICT attachment per winding. + _attach_impedance_correction_tables!( + xfrm, + d, + ict_instances, + supplemental_attribute_associations; + is_3w = true, + ) + end + return (; three_winding_transformer) +end + +# ============================================================================= +# Switches/Breakers, DC Lines, VSC Lines, FACTS +# ============================================================================= + +""" +Build an `InputOutputCurve`-shaped dict carrying `LinearFunctionData`. +""" +function _linear_io_curve_dict(proportional, constant) + return Dict{String, Any}( + "curve_type" => "INPUT_OUTPUT", + "function_data" => Dict{String, Any}( + "function_type" => "LINEAR", + "proportional_term" => float(proportional), + "constant_term" => float(constant), + ), + ) +end + +""" +For each switch or breaker return a `Dict{String, Dict{String, Any}}` +mapping each branch name to a DiscreteControlledACBranch-shaped dict. + +The argument `section` is either "switch" or "breaker". +""" +function read_switch_breaker!( + pm_data::PowerModelsData, + ids::IDGenerator, + bus_dicts::Dict{Int, Dict{String, Any}}, + arcs::Dict{Int, Dict{String, Any}}, + section::AbstractString; + kwargs..., +) + @info "Reading $section data" + data = pm_data.data + branches = Dict{String, Dict{String, Any}}() + if !haskey(data, String(section)) + @info "There is no $section data in this file" + return branches + end + + _get_name = get(kwargs, :branch_name_formatter, _get_pm_branch_name) + + for (_, d) in data[String(section)] + bus_f_dict = bus_dicts[d["f_bus"]] + bus_t_dict = bus_dicts[d["t_bus"]] + name = String(_get_name(d, bus_f_dict, bus_t_dict)) + branch = make_switch_breaker(name, d, bus_f_dict, bus_t_dict, section, ids, arcs) + haskey(branches, name) && throw( + DataFormatError( + "Found duplicate $section name $name; consider passing a `branch_name_formatter` kwarg", + ), + ) + branches[name] = branch + end + return branches +end + +""" +Translate one PowerModels dcline row into a HVDC-line-shaped dict. + +Forks by `source_type`: +- `"pti"` → TwoTerminalLCCLine +- `"matpower"` → TwoTerminalGenericHVDCLine + +The `loss` field is a `TwoTerminalLoss` discriminated union over +`{IncrementalCurve, InputOutputCurve}`. PSY-side `ext` is dropped. +""" +function make_dcline( + name::AbstractString, + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + source_type::AbstractString, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, +) + arc_id = + _get_or_mint_arc_id!(ids, arcs, Int(bus_f_dict["id"]), Int(bus_t_dict["id"])) + pf = get(d, "pf", 0.0) + loss_dict = _linear_io_curve_dict(d["loss1"], d["loss0"]) + + if source_type == "pti" + return Dict{String, Any}( + "id" => getid!(ids, :TwoTerminalLCCLine, d["index"]), + "name" => String(name), + "available" => d["available"], + "arc" => arc_id, + "active_power_flow" => pf, + "r" => d["r"], + "transfer_setpoint" => d["transfer_setpoint"], + "scheduled_dc_voltage" => d["scheduled_dc_voltage"], + "rectifier_bridges" => d["rectifier_bridges"], + "rectifier_delay_angle_limits" => d["rectifier_delay_angle_limits"], + "rectifier_rc" => d["rectifier_rc"], + "rectifier_xc" => d["rectifier_xc"], + "rectifier_base_voltage" => d["rectifier_base_voltage"], + "inverter_bridges" => d["inverter_bridges"], + "inverter_extinction_angle_limits" => d["inverter_extinction_angle_limits"], + "inverter_rc" => d["inverter_rc"], + "inverter_xc" => d["inverter_xc"], + "inverter_base_voltage" => d["inverter_base_voltage"], + "power_mode" => d["power_mode"], + "switch_mode_voltage" => d["switch_mode_voltage"], + "compounding_resistance" => d["compounding_resistance"], + "min_compounding_voltage" => d["min_compounding_voltage"], + "rectifier_transformer_ratio" => d["rectifier_transformer_ratio"], + "rectifier_tap_setting" => d["rectifier_tap_setting"], + "rectifier_tap_limits" => d["rectifier_tap_limits"], + "rectifier_tap_step" => d["rectifier_tap_step"], + "rectifier_delay_angle" => d["rectifier_delay_angle"], + "rectifier_capacitor_reactance" => d["rectifier_capacitor_reactance"], + "inverter_transformer_ratio" => d["inverter_transformer_ratio"], + "inverter_tap_setting" => d["inverter_tap_setting"], + "inverter_tap_limits" => d["inverter_tap_limits"], + "inverter_tap_step" => d["inverter_tap_step"], + "inverter_extinction_angle" => d["inverter_extinction_angle"], + "inverter_capacitor_reactance" => d["inverter_capacitor_reactance"], + # Upstream psse.jl stashes these as NamedTuples; from_json expects + # a nested Dict for the MinMax sub-object, so re-wrap. + "active_power_limits_from" => _min_max_dict(d["pminf"], d["pmaxf"]), + "active_power_limits_to" => _min_max_dict(d["pmint"], d["pmaxt"]), + "reactive_power_limits_from" => _min_max_dict(d["qminf"], d["qmaxf"]), + "reactive_power_limits_to" => _min_max_dict(d["qmint"], d["qmaxt"]), + "loss" => loss_dict, + ) + elseif source_type == "matpower" + return Dict{String, Any}( + "id" => getid!(ids, :TwoTerminalGenericHVDCLine, d["index"]), + "name" => String(name), + "available" => d["br_status"] == 1, + "active_power_flow" => pf, + "arc" => arc_id, + "active_power_limits_from" => _min_max_dict(d["pminf"], d["pmaxf"]), + "active_power_limits_to" => _min_max_dict(d["pmint"], d["pmaxt"]), + "reactive_power_limits_from" => _min_max_dict(d["qminf"], d["qmaxf"]), + "reactive_power_limits_to" => _min_max_dict(d["qmint"], d["qmaxt"]), + "loss" => loss_dict, + ) + else + error("Not supported source type for DC lines: $source_type") + end +end + +""" +For every dcline return a NamedTuple of per-OpenAPI-type sub-collections. +""" +function read_dcline!( + pm_data::PowerModelsData, + ids::IDGenerator, + bus_dicts::Dict{Int, Dict{String, Any}}, + arcs::Dict{Int, Dict{String, Any}}; + kwargs..., +) + @info "Reading DC Line data" + data = pm_data.data + two_terminal_lcc_line = Dict{String, Dict{String, Any}}() + two_terminal_generic_hvdc_line = Dict{String, Dict{String, Any}}() + if !haskey(data, "dcline") + @info "There is no DClines data in this file" + return (; two_terminal_lcc_line, two_terminal_generic_hvdc_line) + end + + _get_name = get(kwargs, :dcline_name_formatter, _get_pm_branch_name) + source_type = data["source_type"] + bucket = if source_type == "pti" + two_terminal_lcc_line + elseif source_type == "matpower" + two_terminal_generic_hvdc_line + else + error("Not supported source type for DC lines: $source_type") + end + + for (d_key, d) in data["dcline"] + d["name"] = get(d, "name", d_key) + bus_f_dict = bus_dicts[d["f_bus"]] + bus_t_dict = bus_dicts[d["t_bus"]] + name = String(_get_name(d, bus_f_dict, bus_t_dict)) + dcline = make_dcline(name, d, bus_f_dict, bus_t_dict, source_type, ids, arcs) + haskey(bucket, name) && throw( + DataFormatError( + "Found duplicate dcline name $name; consider passing a `dcline_name_formatter` kwarg", + ), + ) + bucket[name] = dcline + end + return (; two_terminal_lcc_line, two_terminal_generic_hvdc_line) +end + +""" +Translate one PowerModels vscline row into a TwoTerminalVSCLine-shaped +dict. + +PSY-side `ext` is dropped. +""" +function make_vscline( + name::AbstractString, + d::Dict, + bus_f_dict::Dict, + bus_t_dict::Dict, + ids::IDGenerator, + arcs::Dict{Int, Dict{String, Any}}, +) + arc_id = + _get_or_mint_arc_id!(ids, arcs, Int(bus_f_dict["id"]), Int(bus_t_dict["id"])) + g_value = d["r"] == 0.0 ? 0.0 : 1.0 / d["r"] + return Dict{String, Any}( + "id" => getid!(ids, :TwoTerminalVSCLine, d["index"]), + "name" => String(name), + "available" => d["available"], + "arc" => arc_id, + "active_power_flow" => get(d, "pf", 0.0), + "rating" => d["rating"], + "active_power_limits_from" => _min_max_dict(d["pminf"], d["pmaxf"]), + "active_power_limits_to" => _min_max_dict(d["pmint"], d["pmaxt"]), + "g" => g_value, + "dc_current" => get(d, "if", 0.0), + "reactive_power_from" => get(d, "qf", 0.0), + "dc_voltage_control_from" => d["dc_voltage_control_from"], + "ac_voltage_control_from" => d["ac_voltage_control_from"], + "dc_setpoint_from" => d["dc_setpoint_from"], + "ac_setpoint_from" => d["ac_setpoint_from"], + "converter_loss_from" => _linear_curve_to_io_dict(d["converter_loss_from"]), + "max_dc_current_from" => d["max_dc_current_from"], + "rating_from" => d["rating_from"], + "reactive_power_limits_from" => _min_max_dict(d["qminf"], d["qmaxf"]), + "power_factor_weighting_fraction_from" => + d["power_factor_weighting_fraction_from"], + "reactive_power_to" => get(d, "qt", 0.0), + "dc_voltage_control_to" => d["dc_voltage_control_to"], + "ac_voltage_control_to" => d["ac_voltage_control_to"], + "dc_setpoint_to" => d["dc_setpoint_to"], + "ac_setpoint_to" => d["ac_setpoint_to"], + "converter_loss_to" => _linear_curve_to_io_dict(d["converter_loss_to"]), + "max_dc_current_to" => d["max_dc_current_to"], + "rating_to" => d["rating_to"], + "reactive_power_limits_to" => _min_max_dict(d["qmint"], d["qmaxt"]), + "power_factor_weighting_fraction_to" => d["power_factor_weighting_fraction_to"], + ) +end + +""" +For every vscline return a `Dict{String, Dict{String, Any}}` mapping each +line name to a TwoTerminalVSCLine-shaped dict. +""" +function read_vscline!( + pm_data::PowerModelsData, + ids::IDGenerator, + bus_dicts::Dict{Int, Dict{String, Any}}, + arcs::Dict{Int, Dict{String, Any}}; + kwargs..., +) + @info "Reading VSC Line data" + data = pm_data.data + vsclines = Dict{String, Dict{String, Any}}() + if !haskey(data, "vscline") + @info "There is no VSC lines data in this file" + return vsclines + end + + _get_name = get(kwargs, :vsc_line_name_formatter, _get_pm_branch_name) + + for (d_key, d) in data["vscline"] + d["name"] = get(d, "name", d_key) + bus_f_dict = bus_dicts[d["f_bus"]] + bus_t_dict = bus_dicts[d["t_bus"]] + name = String(_get_name(d, bus_f_dict, bus_t_dict)) + vscline = make_vscline(name, d, bus_f_dict, bus_t_dict, ids, arcs) + haskey(vsclines, name) && throw( + DataFormatError( + "Found duplicate vscline name $name; consider passing a `vsc_line_name_formatter` kwarg", + ), + ) + vsclines[name] = vscline + end + return vsclines +end + +""" +Translate one PowerModels FACTS row into a FACTSControlDevice-shaped dict. +Single-bus device — no Arc minted. +""" +function make_facts(name::AbstractString, d::Dict, bus_dict::Dict, ids::IDGenerator) + if d["tbus"] != 0 + @warn "Series FACTs not supported." + end + if d["control_mode"] > 3 + throw(DataFormatError("Operation mode not supported.")) + end + if d["reactive_power_required"] < 0 + throw(DataFormatError("% MVAr required must me positive.")) + end + + return Dict{String, Any}( + "id" => getid!(ids, :FACTSControlDevice, d["index"]), + "name" => String(name), + "available" => Bool(d["available"]), + "bus" => Int(bus_dict["id"]), + "control_mode" => _normalize_facts_control_mode(d["control_mode"]), + "voltage_setpoint" => d["voltage_setpoint"], + "max_shunt_current" => d["max_shunt_current"], + "reactive_power_required" => d["reactive_power_required"], + ) +end + +""" +For every FACTS device return a `Dict{String, Dict{String, Any}}` mapping each +device name to a FACTSControlDevice-shaped dict. +""" +function read_facts!( + pm_data::PowerModelsData, + ids::IDGenerator, + bus_dicts::Dict{Int, Dict{String, Any}}; + kwargs..., +) + @info "Reading FACTS data" + data = pm_data.data + facts = Dict{String, Dict{String, Any}}() + if !haskey(data, "facts") + @info "There is no facts data in this file" + return facts + end + + _get_name = get(kwargs, :bus_name_formatter, _get_pm_dict_name) + + for (d_key, d) in data["facts"] + d["name"] = get(d, "name", d_key) + name = String(_get_name(d)) + bus_dict = bus_dicts[d["bus"]] + full_name = "$(d["bus"])_$(name)" + facts_entry = make_facts(full_name, d, bus_dict, ids) + haskey(facts, full_name) && throw( + DataFormatError( + "Found duplicate FACTS name $full_name; consider passing a `bus_name_formatter` kwarg", + ), + ) + facts[full_name] = facts_entry + end + return facts +end + +# ============================================================================= +# Storage +# ============================================================================= + +""" +Build the dict equivalent of PSY's auto-defaulted `StorageCost(nothing)`. +""" +function _zero_storage_cost_dict() + zero_io_curve = Dict{String, Any}( + "curve_type" => "INPUT_OUTPUT", + "function_data" => Dict{String, Any}( + "function_type" => "LINEAR", + "proportional_term" => 0.0, + "constant_term" => 0.0, + ), + ) + zero_cost_curve = Dict{String, Any}( + "variable_cost_type" => "COST", + "power_units" => "NATURAL_UNITS", + "value_curve" => zero_io_curve, + "vom_cost" => deepcopy(zero_io_curve), + ) + return Dict{String, Any}( + "cost_type" => "STORAGE", + "charge_variable_cost" => zero_cost_curve, + "discharge_variable_cost" => deepcopy(zero_cost_curve), + "fixed" => 0.0, + "shut_down" => 0.0, + # StorageCostStartUp is a OneOf{Float64, StorageCostStartUpOneOf}; + # the plain-Float64 branch matches PSY's default scalar start_up. + "start_up" => 0.0, + "energy_shortage_cost" => 0.0, + "energy_surplus_cost" => 0.0, + ) +end + +""" +Translate one PowerModels storage row into an EnergyReservoirStorage-shaped +dict. +""" +function make_generic_battery( + storage_name::AbstractString, + d::Dict, + bus_dict::Dict, + ids::IDGenerator, +) + energy_rating = iszero(d["energy_rating"]) ? d["energy"] : d["energy_rating"] + return Dict{String, Any}( + "id" => getid!(ids, :EnergyReservoirStorage, d["index"]), + "name" => String(storage_name), + "available" => Bool(d["status"]), + "bus" => Int(bus_dict["id"]), + "prime_mover_type" => "BA", + "storage_technology_type" => "OTHER_CHEM", + "storage_capacity" => energy_rating, + "storage_level_limits" => _min_max_dict(0.0, energy_rating), + "initial_storage_capacity_level" => d["energy"] / energy_rating, + "rating" => d["thermal_rating"], + "active_power" => d["ps"], + "input_active_power_limits" => _min_max_dict(0.0, d["charge_rating"]), + "output_active_power_limits" => _min_max_dict(0.0, d["discharge_rating"]), + "efficiency" => _in_out_dict(d["charge_efficiency"], d["discharge_efficiency"]), + "reactive_power" => d["qs"], + "reactive_power_limits" => _min_max_dict(d["qmin"], d["qmax"]), + "base_power" => d["thermal_rating"], + "operation_cost" => _zero_storage_cost_dict(), + ) +end + +""" +For each storage device return a `Dict{String, Dict{String, Any}}` mapping each +device name to an EnergyReservoirStorage-shaped dict. + +Storage entries are skipped by `read_gen!`, they come in through this +separate `data["storage"]` pathway instead. MATPOWER files generally have +no `data["storage"]` section; PSS/E exposes it via its own data section. +""" +function read_storage!( + pm_data::PowerModelsData, + ids::IDGenerator, + bus_dicts::Dict{Int, Dict{String, Any}}; + kwargs..., +) + @info "Reading storage data" + data = pm_data.data + storage = Dict{String, Dict{String, Any}}() + if !haskey(data, "storage") + @info "There is no storage data in this file" + return storage + end + + _get_name = get(kwargs, :gen_name_formatter, _get_pm_dict_name) + + for (d_key, d) in data["storage"] + d["name"] = get(d, "name", d_key) + name = String(_get_name(d)) + bus_dict = bus_dicts[d["storage_bus"]] + battery = make_generic_battery(name, d, bus_dict, ids) + haskey(storage, name) && throw( + DataFormatError( + "Found duplicate storage name $name; consider passing a `gen_name_formatter` kwarg", + ), + ) + storage[name] = battery + end + return storage +end + +# ============================================================================= +# Areas + Inter-Area Interchange +# ============================================================================= + +""" +Build an Area-shaped `Dict{String, Any}`. PSY relies on the struct constructor's +field defaults; we emit them explicitly: +- `peak_active_power = 0.0` +- `peak_reactive_power = 0.0` +- `load_response = 0.0` +""" +function make_area(area_number::Int, name::AbstractString, ids::IDGenerator) + return Dict{String, Any}( + "id" => getid!(ids, :Area, area_number), + "name" => String(name), + "peak_active_power" => 0.0, + "peak_reactive_power" => 0.0, + "load_response" => 0.0, + ) +end + +""" +For each bus return a `Dict{Int, Dict{String, Any}}` mapping each unique area +number to an Area-shaped dict (areas are not duplicated). + +Areas are collected solely from bus rows (PSY-strict). PSS/E +`data["area_interchange"]` metadata (`ARNAME`/`I`/`ISW`/`PDES`/`PTOL`) +would land in PSY's `Area.ext` but the OpenAPI Area schema has no `ext`, +so that data is lost. +""" +function read_area!(pm_data::PowerModelsData, ids::IDGenerator; kwargs...) + @info "Reading Area data" + data = pm_data.data + areas = Dict{Int, Dict{String, Any}}() + if !haskey(data, "bus") + @info "There is no bus data — no areas to materialize" + return areas + end + + _get_name_area = get(kwargs, :area_name_formatter, string) + + seen = Set{Int}() + for (_, b) in data["bus"] + push!(seen, Int(b["area"])) + end + + for area_number in seen + name = String(_get_name_area(area_number)) + areas[area_number] = make_area(area_number, name, ids) + end + return areas +end + +""" +Build an AreaInterchange-shaped dict from a PSS/E `interarea_transfer` row. + +PSY hardcodes: `flow_limits = (from_to = -INFINITE_BOUND, to_from = +INFINITE_BOUND)`; `available` flag as `true`. +""" +function make_area_interchange( + name::AbstractString, + d::Dict, + from_area_id::Int, + to_area_id::Int, + ids::IDGenerator, +) + return Dict{String, Any}( + "id" => getid!(ids, :AreaInterchange, d["index"]), + "name" => String(name), + "available" => true, + "active_power_flow" => d["power_transfer"], + "from_area" => from_area_id, + "to_area" => to_area_id, + "flow_limits" => Dict{String, Any}( + "from_to" => -INFINITE_BOUND, + "to_from" => INFINITE_BOUND, + ), + ) +end + +""" +For each PSS/E `interarea_transfer` return a `Dict{String, Dict{String, Any}}` +mapping each transfer name to an AreaInterchange-shaped dict. + +PSS/E-only; MATPOWER files don't carry `interarea_transfer` and this +function returns an empty dict. +""" +function read_area_interchange!( + pm_data::PowerModelsData, + ids::IDGenerator, + area_dicts::Dict{Int, Dict{String, Any}}; + kwargs..., +) + @info "Reading area interchange data" + data = pm_data.data + interchanges = Dict{String, Dict{String, Any}}() + if data["source_type"] != "pti" || !haskey(data, "interarea_transfer") + @info "There is no interarea_transfer data in this file" + return interchanges + end + + _get_name_area = get(kwargs, :area_name_formatter, string) + + for (_, d) in data["interarea_transfer"] + area_from = Int(d["area_from"]) + area_to = Int(d["area_to"]) + + if !haskey(area_dicts, area_from) || !haskey(area_dicts, area_to) + @warn "Skipping interarea_transfer: from_area=$area_from to_area=$area_to references an area not present on any bus" + continue + end + + from_area_id = getid!(ids, :Area, area_from) + to_area_id = getid!(ids, :Area, area_to) + area_from_name = String(_get_name_area(area_from)) + area_to_name = String(_get_name_area(area_to)) + transfer_id = get(d, "transfer_id", "1") + name = "$(area_from_name)_$(area_to_name)_$transfer_id" + + haskey(interchanges, name) && throw( + DataFormatError( + "Found duplicate interarea_transfer name $name; consider passing an `area_name_formatter` kwarg", + ), + ) + interchanges[name] = + make_area_interchange(name, d, from_area_id, to_area_id, ids) + end + return interchanges +end + +# ============================================================================= +# Impedance Correction Tables (ICT) — supplemental attributes +# ============================================================================= + +""" +Read PSS/E impedance-correction-table data and return a +`Dict{Tuple{Int, String}, Dict{String, Any}}` mapping each +`(table_number, transformer_winding)` pair to an ImpedanceCorrectionData- +shaped dict ready for `OpenAPI.from_json`. + +PSS/E-only; MATPOWER files have no `data["impedance_correction"]` and this +function returns an empty dict. +""" +function read_impedance_correction!(pm_data::PowerModelsData, ids::IDGenerator) + @info "Reading Impedance Correction Table data" + ict_instances = Dict{Tuple{Int, String}, Dict{String, Any}}() + data = pm_data.data + if !haskey(data, "impedance_correction") + @info "There is no Impedance Correction Table data in this file" + return ict_instances + end + + for (_, table_data) in data["impedance_correction"] + table_number = Int(table_data["table_number"]) + x = table_data["tap_or_angle"] + y = table_data["scaling_factor"] + + if length(x) != length(y) + throw( + DataFormatError( + "Impedance correction mismatch at table $table_number: tap/angle and scaling count differs.", + ), + ) + end + if length(x) < 2 + @warn "Skipping impedance correction entry due to insufficient data points ($(length(x)) < 2): $x" + continue + end + + pwl_dict = Dict{String, Any}( + "function_type" => "PIECEWISE_LINEAR", + "points" => [ + Dict{String, Any}("x" => float(x[i]), "y" => float(y[i])) for + i in eachindex(x) + ], + ) + + table_type = + if PSSE_PARSER_TAP_RATIO_LBOUND <= x[1] <= PSSE_PARSER_TAP_RATIO_UBOUND + "TAP_RATIO" + else + "PHASE_SHIFT_ANGLE" + end + + for winding in _ICT_WINDING_CATEGORIES + ict_instances[(table_number, winding)] = Dict{String, Any}( + "id" => getid!( + ids, + :ImpedanceCorrectionData, + (table_number, winding), + ), + "table_number" => table_number, + "impedance_correction_curve" => deepcopy(pwl_dict), + "transformer_winding" => winding, + "transformer_control_mode" => table_type, + ) + end + end + return ict_instances +end + +""" +Look up the ICT for one `(table_number, transformer_winding)` pair and +append a `{"attribute_id", "entity_id"}` association entry linking the +transformer to the ICT. +""" +function _attach_single_ict!( + transformer_dict::Dict, + table_number::Int, + winding::AbstractString, + ict_instances::Dict{Tuple{Int, String}, Dict{String, Any}}, + supp_assoc::Vector{Dict{String, Any}}, +) + key = (table_number, String(winding)) + if !haskey(ict_instances, key) + @debug "No ICT associated with transformer $(transformer_dict["name"]) for winding $winding." + return + end + push!( + supp_assoc, + Dict{String, Any}( + "attribute_id" => ict_instances[key]["id"], + "entity_id" => transformer_dict["id"], + ), + ) + return +end + +""" +Attach the ICT(s) referenced by a transformer row to `supp_assoc`. +No-ops if `ict_instances` is empty. +""" +function _attach_impedance_correction_tables!( + transformer_dict::Dict, + d::Dict, + ict_instances::Dict{Tuple{Int, String}, Dict{String, Any}}, + supp_assoc::Vector{Dict{String, Any}}; + is_3w::Bool, +) + isempty(ict_instances) && return + + if is_3w + for (key_prefix, winding) in _ICT_3W_WINDING_KEYS + key = "$(key_prefix)_correction_table" + haskey(d, key) || continue + _attach_single_ict!( + transformer_dict, + Int(d[key]), + winding, + ict_instances, + supp_assoc, + ) + end + else + haskey(d, "correction_table") || return + _attach_single_ict!( + transformer_dict, + Int(d["correction_table"]), + "TR2W_WINDING", + ict_instances, + supp_assoc, + ) + end + return +end From 9551842ad7c1d5f0f6cede2fbe8bfc029a3385ab Mon Sep 17 00:00:00 2001 From: Chubin Date: Wed, 19 Aug 2026 11:55:49 -0600 Subject: [PATCH 2/2] Add coverage for parse_to_openapi_objects and make_database MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New test files exercise the POM/DB pipeline that landed in the rebase. Nothing tested it before this commit; only the shared PowerModelsData dict path had coverage. - test_parse_to_openapi_objects.jl (37 assertions): typed collection counts against case14.raw and case16_all_components.raw, transformer decomposition invariants (each 2W → 1 circuit, each 3W → 3 circuits), circuit-id references from holders, and cross-component id uniqueness. - test_make_database.jl (36 assertions): DB row counts by entity_type match parse_to_openapi_objects results, subtype-inheritance coverage (all 7 POM subtypes we route via shared parent tables appear as entity_type rows on case16), dedicated transformer table row counts, arcs and 3W FK integrity, and the on-disk path kwarg. - test/Project.toml: add DBInterface and SQLite as test-only deps so the DB tests can query the returned SQLite handle directly. case16_all_components.raw is the one PSB file that exercises every subtype simultaneously (TransformerCircuit, TwoWinding, ThreeWinding, DiscreteControlledACBranch, TwoTerminalLCCLine, TwoTerminalVSCLine, FACTSControlDevice, SwitchedAdmittance, InterruptibleStandardLoad). Full suite: 574 pass, 0 fail (73 new). --- test/Project.toml | 2 + test/test_make_database.jl | 132 ++++++++++++++++++++++++++ test/test_parse_to_openapi_objects.jl | 106 +++++++++++++++++++++ 3 files changed, 240 insertions(+) create mode 100644 test/test_make_database.jl create mode 100644 test/test_parse_to_openapi_objects.jl diff --git a/test/Project.toml b/test/Project.toml index e7e0d59..e5b60df 100644 --- a/test/Project.toml +++ b/test/Project.toml @@ -1,9 +1,11 @@ [deps] Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595" +DBInterface = "a10d1c49-ce27-4219-8d33-6db1a4562965" InfrastructureSystems = "2cd47ed4-ca9b-11e9-27f2-ab636a7671f1" Logging = "56ddb016-857b-54e1-b83d-db4d58db5568" PowerFlowFileParser = "bed98974-b02e-5e2f-9ee0-a103f5c450dd" PowerSystemCaseBuilder = "f00506e0-b84f-492a-93c2-c0a9afc4364e" +SQLite = "0aa819cd-b072-5ff4-a722-6bc24af294d9" Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" [compat] diff --git a/test/test_make_database.jl b/test/test_make_database.jl new file mode 100644 index 0000000..e7c5dc9 --- /dev/null +++ b/test/test_make_database.jl @@ -0,0 +1,132 @@ +import SQLite +import DBInterface + +# Return a Dict{String, Int} of entity_type => row count from the entities table. +function _entity_type_counts(db::SQLite.DB) + counts = Dict{String, Int}() + for row in DBInterface.execute(db, + "SELECT entity_type, COUNT(*) FROM entities GROUP BY entity_type", + ) + counts[row[1]] = row[2] + end + return counts +end + +_table_count(db::SQLite.DB, table::String) = + first(DBInterface.execute(db, "SELECT COUNT(*) FROM $table"))[1] + +@testset "make_database: returns a SQLite.DB" begin + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case14.raw")) + db = make_database(pm) + @test isa(db, SQLite.DB) + @test !isempty(SQLite.tables(db)) +end + +@testset "make_database: case14.raw entity_type counts match parse_to_openapi_objects" begin + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case14.raw")) + objs = parse_to_openapi_objects(pm) + db = make_database(pm) + counts = _entity_type_counts(db) + + @test counts["ACBus"] == length(objs.buses) + @test counts["Line"] == length(objs.branches.line) + @test counts["TwoWindingTransformer"] == + length(objs.branches.two_winding_transformer) + @test counts["TransformerCircuit"] == length(objs.transformer_circuits) + @test counts["ThermalStandard"] == length(objs.gens.thermal_standard) + @test counts["StandardLoad"] == length(objs.loads.standard_load) + @test counts["SwitchedAdmittance"] == length(objs.switched_shunts) + @test counts["Arc"] == length(objs.arcs) +end + +@testset "make_database: case16_all_components.raw covers every POM subtype" begin + # Each of the 7 subtypes we wired to inherit from a shared parent table + # must appear as an entity_type row on this file. + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case16_all_components.raw")) + db = make_database(pm) + counts = _entity_type_counts(db) + + for subtype in ( + "TransformerCircuit", + "TwoWindingTransformer", + "ThreeWindingTransformer", + "DiscreteControlledACBranch", + "TwoTerminalLCCLine", + "TwoTerminalVSCLine", + "FACTSControlDevice", + "SwitchedAdmittance", + "InterruptibleStandardLoad", + ) + @test get(counts, subtype, 0) > 0 + end +end + +@testset "make_database: dedicated transformer tables get the right row counts" begin + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case16_all_components.raw")) + objs = parse_to_openapi_objects(pm) + db = make_database(pm) + + @test _table_count(db, "transformer_circuits") == + length(objs.transformer_circuits) + @test _table_count(db, "two_winding_transformers") == + length(objs.branches.two_winding_transformer) + @test _table_count(db, "three_winding_transformers") == + length(objs.xfrm_3w.three_winding_transformer) +end + +@testset "make_database: arcs FK all point at existing entities" begin + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case16_all_components.raw")) + db = make_database(pm) + + orphan_from = first(DBInterface.execute(db, """ + SELECT COUNT(*) FROM arcs + WHERE from_id NOT IN (SELECT id FROM entities) + """))[1] + orphan_to = first(DBInterface.execute(db, """ + SELECT COUNT(*) FROM arcs + WHERE to_id NOT IN (SELECT id FROM entities) + """))[1] + + @test orphan_from == 0 + @test orphan_to == 0 +end + +@testset "make_database: 3W holders reference existing transformer_circuit rows" begin + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case16_all_components.raw")) + db = make_database(pm) + + orphan = first(DBInterface.execute(db, """ + SELECT COUNT(*) FROM three_winding_transformers t + WHERE t.primary_circuit_id NOT IN (SELECT id FROM transformer_circuits) + OR t.secondary_circuit_id NOT IN (SELECT id FROM transformer_circuits) + OR t.tertiary_circuit_id NOT IN (SELECT id FROM transformer_circuits) + """))[1] + @test orphan == 0 +end + +@testset "make_database: local modified_14bus_system.raw round-trips" begin + # Sanity check on the local fixture we've been using end-to-end. + pm = PowerModelsData(joinpath(@__DIR__, "modified_14bus_system.raw")) + db = make_database(pm) + counts = _entity_type_counts(db) + + @test counts["ACBus"] == 22 + @test counts["Line"] == 20 + @test counts["TwoWindingTransformer"] == 3 + @test counts["ThreeWindingTransformer"] == 2 + @test counts["TransformerCircuit"] == 9 + @test counts["FACTSControlDevice"] == 1 + @test counts["TwoTerminalLCCLine"] == 1 + @test counts["DiscreteControlledACBranch"] == 2 +end + +@testset "make_database: path kwarg writes to disk" begin + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case14.raw")) + path = tempname() * ".sqlite" + db = make_database(pm; path = path) + @test isfile(path) + @test filesize(path) > 0 + # The returned DB should still be usable. + @test _table_count(db, "entities") > 0 + rm(path; force = true) +end diff --git a/test/test_parse_to_openapi_objects.jl b/test/test_parse_to_openapi_objects.jl new file mode 100644 index 0000000..b939644 --- /dev/null +++ b/test/test_parse_to_openapi_objects.jl @@ -0,0 +1,106 @@ +@testset "parse_to_openapi_objects: basic invariants" begin + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case14.raw")) + objs = parse_to_openapi_objects(pm) + + @test isa(objs, ParsedOpenAPIObjects) + @test length(objs.buses) == length(pm.data["bus"]) + + # each 2W transformer contributes exactly one TransformerCircuit + @test length(objs.transformer_circuits) == + length(objs.branches.two_winding_transformer) + + 3 * length(objs.xfrm_3w.three_winding_transformer) + + # gens/loads NamedTuples cover all POM subtypes + total_gens = + length(objs.gens.thermal_standard) + + length(objs.gens.hydro_dispatch) + + length(objs.gens.renewable_dispatch) + + length(objs.gens.renewable_non_dispatch) + + length(objs.gens.synchronous_condenser) + @test total_gens == length(pm.data["gen"]) +end + +@testset "parse_to_openapi_objects: case14.raw expected counts" begin + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case14.raw")) + objs = parse_to_openapi_objects(pm) + @test length(objs.buses) == 14 + @test length(objs.branches.line) == 17 + @test length(objs.branches.two_winding_transformer) == 3 + @test length(objs.xfrm_3w.three_winding_transformer) == 0 + @test length(objs.transformer_circuits) == 3 + @test length(objs.gens.thermal_standard) == 5 + @test length(objs.loads.standard_load) == 11 + @test length(objs.arcs) == 20 +end + +@testset "parse_to_openapi_objects: case16_all_components.raw exercises all POM subtypes" begin + # This is the one PSB file that populates every subtype we care about + # (TwoTerminalLCCLine, TwoTerminalVSCLine, DiscreteControlledACBranch, + # FACTSControlDevice, SwitchedAdmittance, InterruptibleStandardLoad, + # plus TransformerCircuit / TwoWindingTransformer / ThreeWindingTransformer). + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case16_all_components.raw")) + objs = parse_to_openapi_objects(pm) + + @test length(objs.buses) == 18 + @test length(objs.branches.line) == 7 + @test length(objs.branches.two_winding_transformer) == 3 + @test length(objs.xfrm_3w.three_winding_transformer) == 2 + # 3 circuits per 3W (2 * 3 = 6) plus one per 2W (3) = 9 + @test length(objs.transformer_circuits) == 9 + + @test length(objs.dclines.two_terminal_lcc_line) == 2 + @test length(objs.vsclines) == 1 + @test length(objs.facts) == 1 + @test length(objs.switched_shunts) == 1 + @test length(objs.shunts) == 3 + @test length(objs.switches) == 1 + @test length(objs.breakers) == 1 + @test length(objs.loads.interruptible_standard_load) == 2 + @test length(objs.loads.standard_load) == 4 +end + +@testset "parse_to_openapi_objects: transformer decomposition circuits FK back to holders" begin + # Each ThreeWindingTransformer.primary_circuit_id / secondary / tertiary + # must refer to a TransformerCircuit id present in the accumulator. + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case16_all_components.raw")) + objs = parse_to_openapi_objects(pm) + + circuit_ids = Set(c.id for c in objs.transformer_circuits) + @test !isempty(circuit_ids) + + # Holders reference circuits by id (Int), not by nested object. + for tw in objs.branches.two_winding_transformer + @test tw.circuit in circuit_ids + end + for tri in objs.xfrm_3w.three_winding_transformer + @test tri.primary_circuit in circuit_ids + @test tri.secondary_circuit in circuit_ids + @test tri.tertiary_circuit in circuit_ids + end +end + +@testset "parse_to_openapi_objects: IDs are unique across all typed collections" begin + pm = PowerModelsData(joinpath(PSSE_RAW_DIR, "case16_all_components.raw")) + objs = parse_to_openapi_objects(pm) + + seen = Int[] + append!(seen, [x.id for x in objs.buses]) + append!(seen, [x.id for x in objs.areas]) + append!(seen, [x.id for x in objs.loadzones]) + append!(seen, [x.id for x in objs.branches.line]) + append!(seen, [x.id for x in objs.branches.two_winding_transformer]) + append!(seen, [x.id for x in objs.xfrm_3w.three_winding_transformer]) + append!(seen, [x.id for x in objs.transformer_circuits]) + append!(seen, [x.id for x in objs.gens.thermal_standard]) + append!(seen, [x.id for x in objs.loads.standard_load]) + append!(seen, [x.id for x in objs.loads.interruptible_standard_load]) + append!(seen, [x.id for x in objs.dclines.two_terminal_lcc_line]) + append!(seen, [x.id for x in objs.vsclines]) + append!(seen, [x.id for x in objs.facts]) + append!(seen, [x.id for x in objs.switched_shunts]) + append!(seen, [x.id for x in objs.shunts]) + append!(seen, [x.id for x in objs.switches]) + append!(seen, [x.id for x in objs.breakers]) + + @test length(seen) == length(unique(seen)) +end