diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 66fd8b42..265d9d4f 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -1,6 +1,6 @@ # PowerSystemCaseBuilder.jl (PSB) — psy6 branch -The Sienna **test-system fixture factory**: a registry of 200+ named `PSY.System` cases built from raw data (Matpower, PSSE raw/dyr, tabular CSV, PowerFlowData) with an on-disk serialized cache, so downstream test suites (PSY, PNM, PF, POM) get systems in seconds. Not an optimization package. Its serialization is PSY's own IS-based JSON+HDF5 — **not** the OpenAPI/GridDB pipeline. Platform conventions: the `sienna-psy6` skill; workspace architecture: the psy6 workspace root `CLAUDE.md`. +The Sienna **test-system fixture factory**: a registry of 200+ named `PSY.System` cases built from raw data (Matpower, PSSE raw/dyr, tabular CSV, PowerFlowData) with an on-disk serialized cache, so downstream test suites (PSY, PNM, PF, POM) get systems in seconds. Not an optimization package. Both building and caching go through the OpenAPI document: parsers emit one, `PSY.from_file` reads it, and the cache is PSY's `.sns` archive of it. Platform conventions: the `sienna-psy6` skill; workspace architecture: the psy6 workspace root `CLAUDE.md`. ## Why this package matters platform-wide @@ -16,7 +16,7 @@ build_system(PSITestSystems, "c_sys5"; force_build=true) # categories: PSY/PSI list_systems(...); show_systems(...); list_categories() ``` -- Flow (`src/build_system.jl` + `src/utils/utils.jl`): cache check → optional artifact download → registered `build_func(; raw_data, …)` → serialize (unless `skip_serialization`) → later calls deserialize. Cache dir: `data/serialized_system//.json` + `_metadata.json` + `_validation_descriptors.json` + `_time_series_storage.h5`. +- Flow (`src/build_system.jl` + `src/utils/utils.jl`): cache check → optional artifact download → registered `build_func(; raw_data, …)` → serialize (unless `skip_serialization`) → later calls deserialize. Cache entry: `data/serialized_system//.sns`, written by `PSY.to_file`. - `build_system` splits kwargs: keys in `PSY.SYSTEM_KWARGS` forward to `PSY.System`; the rest must match the descriptor's `supported_arguments` or error. Non-encodable `sys_args` skip caching entirely. - Catalog: `SYSTEM_CATALOG` in `src/system_descriptor_data.jl` (~228 `SystemDescriptor` entries; duplicate names error); builders in `src/library/` (8 catalog files); include order puts the catalog last, after all builders. - Raw data via lazy artifacts (`Artifacts.toml`): `CaseData` = PowerSystemsTestData tarball (currently a 5.0-dev tag), `rts` = RTS-GMLC. **The CaseData download can flake — retry once before digging**; there is no retry in the code. Re-pin the sha256 when PowerSystemsTestData re-tags. @@ -37,7 +37,7 @@ Why PSB feels this harder than other packages: **the cache stores serialized sys - Branch `psy6`; `[sources]` pins in **both** root and `test/Project.toml`: IS→`IS4`, PowerSystems→`psy6`, PowerFlowFileParser→`psy6`, PowerTableDataParser→`psy6`. ⚠️ Org URLs are inconsistent across the manifests, but note which way: **`NLR-Sienna/PowerTableDataParser.jl` is correct, not a typo** — it is the canonical location, and `NREL-Sienna/PowerTableDataParser.jl` only reaches it via a GitHub 301 (`Sienna-Platform/PowerTableDataParser.jl` is a 404). Every *other* Sienna repo has moved the opposite way: `NREL-Sienna/*` now 301-redirects to `Sienna-Platform/*`. So most `NREL-Sienna` URLs here are stale-but-working, while the one that looks misspelled is the accurate one. Verify with `curl -sI` before "correcting" any of them. - ⚠️ `test/Project.toml` may still pin PSY to `transformer-refactor` and PowerFlowFileParser to `mb/transformer-refactor`. Both are merged now — PSY `d19f3244f`, PFFP `adf5cb1` — so those revs are stale and should read `psy6`. The *root* `Project.toml` pin was always correct, which is why PSB works as a dependency even when its own test env does not resolve. - **`src/utils/psy6_compat.jl` is a sanctioned exception to the no-shims policy**: method overloads accepting old `Nothing`/Float64 signatures (`ReserveDemandCurve`/`MarketBidCost`) because the pinned PowerSystemsTestData artifact still uses pre-psy6 constructor calls. Scoped to external artifact data only; include-order sensitive (after `definitions.jl`, before `system_library.jl`). Remove it when the artifact is regenerated — never widen it. -- Parsing goes through PowerFlowFileParser/PowerTableDataParser (PSY has no parsers in this line). +- Parsing goes through PowerFlowFileParser/PowerTableDataParser (PSY has no parsers in this line). Every PSS/E, Matpower and tabular case builds via `system_from_openapi` (`src/parsers/openapi_pipeline.jl`): parser → OpenAPI document → `PSY.from_file`. `.dyr` dynamics are still attached afterwards by PSB's own `add_dyn_injectors!` (`src/parsers/psse_dynamic_data.jl`); PowerFlowFileParser does not read `.dyr`. - Reduction fixtures (for PNM/PF/POM work): `c_sys5`/`c_sys14` reduce **nothing**; `case11_network_reductions` has real series arcs but no forecasts; matpower RTS/case24 for larger cases. - Compat still reads PSY ^5.10 / IS ^3.2 — the `[sources]` revs, not compat, select the breaking line. No version bumps until release. diff --git a/Project.toml b/Project.toml index f97a4612..949785c8 100644 --- a/Project.toml +++ b/Project.toml @@ -22,7 +22,6 @@ PrettyTables = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d" Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" SHA = "ea8e919c-243c-51af-8825-aaa63cd721ce" TimeSeries = "9e3dc215-6440-5c97-bce1-76c03772f85e" -Unicode = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" YAML = "ddb6d928-2868-570f-bddf-ab3f9cf99eb6" # PowerSystems depends on these unregistered packages and [sources] are not inherited @@ -74,6 +73,5 @@ PrettyTables = "2.4, 3.1" Random = "1" SHA = "0.7, 1" TimeSeries = "^0.24, ^0.25" -Unicode = "1.11.0" YAML = "0.4.16" julia = "^1.10" diff --git a/src/PowerSystemCaseBuilder.jl b/src/PowerSystemCaseBuilder.jl index 562348ee..aed1c59e 100644 --- a/src/PowerSystemCaseBuilder.jl +++ b/src/PowerSystemCaseBuilder.jl @@ -39,9 +39,6 @@ import PrettyTables import PowerFlowFileParser import PowerTableDataParser -# imports for parsers/ to work -import Unicode: normalize - #TimeStamp Management Imports import TimeSeries import Dates @@ -147,10 +144,8 @@ Category for SiennaPRASInterface.jl examples. struct SPISystems <: SystemCategory end # Include Parsing files -include("parsers/common.jl") include("parsers/known_stale_fixture_data.jl") include("parsers/openapi_pipeline.jl") -include("parsers/power_models_data.jl") include("parsers/psse_dynamic_data.jl") include("parsers/psse_metadata_reimport.jl") diff --git a/src/library/matpowertest_library.jl b/src/library/matpowertest_library.jl index 33aa8fd9..4d3aa298 100644 --- a/src/library/matpowertest_library.jl +++ b/src/library/matpowertest_library.jl @@ -1,5 +1,6 @@ function build_matpower(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) - sys = make_system(PowerFlowFileParser.PowerModelsData(raw_data); sys_kwargs...) + pm_data = PowerFlowFileParser.PowerModelsData(raw_data) + sys = system_from_openapi(drop_known_unread_matpower_data!(pm_data); sys_kwargs...) return sys end diff --git a/src/library/psi_library.jl b/src/library/psi_library.jl index 0425da99..03529e16 100644 --- a/src/library/psi_library.jl +++ b/src/library/psi_library.jl @@ -246,7 +246,7 @@ end function build_5_bus_hydro_uc_sys(; add_forecasts, raw_data, sys_kwargs...) # `timeseries_metadata_file` is only known once `add_forecasts` is resolved, and - # `system_from_openapi` (unlike the retired `make_system`) has no per-call override + # `system_from_openapi` has no per-call override # for it — it always reads whatever is set on `rawsys` itself, so `rawsys` is built # per branch rather than shared. if add_forecasts diff --git a/src/library/psid_library.jl b/src/library/psid_library.jl index 6baa0e8c..3d7c4806 100644 --- a/src/library/psid_library.jl +++ b/src/library/psid_library.jl @@ -5,7 +5,7 @@ function build_psid_4bus_multigen(; raw_data, kwargs...) dyr_file = joinpath(raw_data, "FourBus_multigen.dyr") pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) for l in get_components(PSY.StandardLoad, sys) transform_load_to_constant_impedance(l) @@ -18,7 +18,7 @@ function build_psid_11bus_andes(; raw_data, kwargs...) raw_file = joinpath(raw_data, "11BUS_KUNDUR.raw") dyr_file = joinpath(raw_data, "11BUS_KUNDUR_TGOV.dyr") pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) for l in get_components(PSY.StandardLoad, sys) transform_load_to_constant_impedance(l) @@ -56,7 +56,7 @@ function build_psid_14bus_multigen(; raw_data, kwargs...) dyr_file = joinpath(raw_data, "dyn_data.dyr") pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) for l in get_components(PSY.StandardLoad, sys) transform_load_to_constant_impedance(l) @@ -68,14 +68,14 @@ function build_3bus_inverter(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) raw_file = joinpath(raw_data, "ThreeBusInverter.raw") pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) return sys end function build_psid_wecc_9_dynamic(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - sys = make_system(pm_data; runchecks = false, sys_kwargs...) + sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) # Manually change reactance of three branches to match Sauer & Pai (2007) Figure 7.4 set_x!(get_component(Branch, sys, "Bus 5-Bus 4-i_1"), 0.085 * IS.SU) @@ -166,7 +166,7 @@ end function build_psid_load_tutorial_omib(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - sys = make_system(pm_data; runchecks = false, sys_kwargs...) + sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) l = first(get_components(StandardLoad, sys)) exp_load = PSY.ExponentialLoad(; name = PSY.get_name(l), diff --git a/src/library/psidtest_library.jl b/src/library/psidtest_library.jl index 9a9e775b..914bdc03 100644 --- a/src/library/psidtest_library.jl +++ b/src/library/psidtest_library.jl @@ -79,7 +79,7 @@ function build_psid_psse_test_avr(; raw_data, kwargs...) ) end pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - avr_sys = make_system(pm_data; sys_kwargs...) + avr_sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(avr_sys, dyr_file) _set_zip_load_model!(avr_sys, :impedance) return avr_sys @@ -109,7 +109,7 @@ function build_psid_psse_test_tg(; raw_data, kwargs...) ) end pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - tg_sys = make_system(pm_data; sys_kwargs...) + tg_sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(tg_sys, dyr_file) for l in get_components(PSY.StandardLoad, tg_sys) @@ -158,7 +158,7 @@ function build_psid_psse_test_gen(; raw_data, kwargs...) ) end pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - gen_sys = make_system(pm_data; sys_kwargs...) + gen_sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(gen_sys, dyr_file) for l in get_components(PSY.StandardLoad, gen_sys) @@ -192,7 +192,7 @@ function build_psid_psse_test_pss(; raw_data, kwargs...) ) end pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - pss_sys = make_system(pm_data; sys_kwargs...) + pss_sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(pss_sys, dyr_file) for l in get_components(PSY.StandardLoad, pss_sys) @@ -205,7 +205,7 @@ end function build_psid_test_omib(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - omib_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + omib_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(omib_sys) function dyn_gen_classic(generator) @@ -234,7 +234,7 @@ end function build_psid_test_threebus_oneDoneQ(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(threebus_sys) function dyn_gen_oneDoneQ(generator) @@ -264,7 +264,7 @@ end function build_psid_test_threebus_simple_marconato(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(threebus_sys) function dyn_gen_simple_marconato(generator) @@ -294,7 +294,7 @@ end function build_psid_test_threebus_marconato(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(threebus_sys) function dyn_gen_marconato(generator) @@ -324,7 +324,7 @@ end function build_psid_test_threebus_simple_anderson(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(threebus_sys) function dyn_gen_simple_anderson(generator) @@ -354,7 +354,7 @@ end function build_psid_test_threebus_anderson(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(threebus_sys) function dyn_gen_anderson(generator) @@ -384,7 +384,7 @@ end function build_psid_test_threebus_5shaft(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(threebus_sys) #Reduce generator output @@ -436,7 +436,7 @@ end function build_psid_test_vsm_inverter(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - omib_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + omib_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(omib_sys) function inv_darco(static_device) @@ -467,7 +467,7 @@ end function build_psid_test_threebus_machine_vsm(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(threebus_sys) function dyn_gen_second_order(generator) @@ -520,7 +520,7 @@ end function build_psid_test_threebus_multimachine(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) function dyn_gen_multi(generator) return PSY.DynamicGenerator(; @@ -566,7 +566,7 @@ end function build_psid_test_threebus_psat_avrs(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(threebus_sys) function dyn_gen_avr_type2(generator) @@ -613,7 +613,7 @@ end function build_psid_test_threebus_vsm_reference(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - threebus_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + threebus_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) function inv_case78(static_device) return DynamicInverter(; @@ -660,7 +660,7 @@ end function build_psid_test_threebus_genrou_avr(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - sys = make_system(pm_data; runchecks = false, sys_kwargs...) + sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) #Replace Gen101 by Source remove_component!(ThermalStandard, sys, "generator-101-1") @@ -693,7 +693,7 @@ end function build_psid_test_droop_inverter(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - omib_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + omib_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(omib_sys) ############### Data Dynamic devices ######################## @@ -724,7 +724,7 @@ end function build_psid_test_gfoll_inverter(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - omib_sys = make_system(pm_data; runchecks = false, sys_kwargs...) + omib_sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(omib_sys) ############### Data Dynamic devices ######################## @@ -755,7 +755,7 @@ end function build_psid_test_threebus_multimachine_dynlines(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - sys = make_system(pm_data; runchecks = false, sys_kwargs...) + sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) ############### Data Dynamic devices ######################## function dyn_gen_marconato(generator) @@ -803,7 +803,7 @@ end function build_psid_test_pvs(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - sys = make_system(pm_data; runchecks = false, sys_kwargs...) + sys = system_from_openapi(pm_data; runchecks = false, sys_kwargs...) add_source_to_ref(sys) ############### Data Dynamic devices ######################## @@ -863,7 +863,7 @@ function build_psid_psse_test_constantP_load(; raw_data, kwargs...) raw_file = joinpath(raw_data, "ThreeBusMulti.raw") dyr_file = joinpath(raw_data, "ThreeBus_GENROU.dyr") pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) return sys end @@ -981,7 +981,7 @@ function build_psid_test_indmotor(; raw_data, kwargs...) raw_file = joinpath(raw_data, "TVC_System_motor.raw") dyr_file = joinpath(raw_data, "TVC_System_motor.dyr") pm_data = PowerFlowFileParser.PowerModelsData(raw_file) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) return sys end diff --git a/src/library/psitest_library.jl b/src/library/psitest_library.jl index fbd04f12..fa2c4451 100644 --- a/src/library/psitest_library.jl +++ b/src/library/psitest_library.jl @@ -5315,7 +5315,7 @@ function build_5_bus_matpower_DA(; raw_data, kwargs...) FORECASTS_DIR = joinpath(data_dir, "5-Bus", "5bus_ts", "7day") - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) reserves = [ OnlineReserve{ReserveUp}("REG1", true, 5.0, 0.1), OnlineReserve{ReserveUp}("REG2", true, 5.0, 0.06), @@ -5343,7 +5343,7 @@ function build_5_bus_matpower_RT(; raw_data, kwargs...) FORECASTS_DIR = joinpath(data_dir, "5-Bus", "5bus_ts", "7day") pm_data = PowerFlowFileParser.PowerModelsData(raw_data) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) PowerTableDataParser.add_time_series_from_pointers!( sys, @@ -5361,7 +5361,7 @@ function build_5_bus_matpower_AGC(; raw_data, kwargs...) FORECASTS_DIR = joinpath(data_dir, "5-Bus", "5bus_ts", "7day") - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) PowerTableDataParser.add_time_series_from_pointers!( sys, diff --git a/src/library/pssetest_library.jl b/src/library/pssetest_library.jl index e759b1c8..51632216 100644 --- a/src/library/pssetest_library.jl +++ b/src/library/pssetest_library.jl @@ -9,7 +9,7 @@ function build_psse_ACTIVSg2000_sys(; raw_data, kwargs...) sys_kwargs = filter_kwargs(; kwargs...) file_path = joinpath(raw_data, "ACTIVSg2000", "ACTIVSg2000.RAW") dyr_file = joinpath(raw_data, "psse_dyr", "ACTIVSg2000_dynamics.dyr") - sys = make_system(PowerFlowFileParser.PowerModelsData(file_path), sys_kwargs...) + sys = system_from_openapi(PowerFlowFileParser.PowerModelsData(file_path); sys_kwargs...) add_dyn_injectors!(sys, dyr_file) return sys end diff --git a/src/library/psytest_library.jl b/src/library/psytest_library.jl index 4058aafd..f4f68d12 100644 --- a/src/library/psytest_library.jl +++ b/src/library/psytest_library.jl @@ -92,7 +92,7 @@ function build_psse_Benchmark_4ger_33_2015_sys(; raw_data, kwargs...) file_path = joinpath(raw_data, "psse_raw", "Benchmark_4ger_33_2015.RAW") dyr_file = joinpath(raw_data, "psse_dyr", "Benchmark_4ger_33_2015.dyr") pm_data = PowerFlowFileParser.PowerModelsData(file_path) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) return sys end @@ -102,7 +102,7 @@ function build_psse_OMIB_sys(; raw_data, kwargs...) file_path = joinpath(raw_data, "psse_raw", "OMIB.raw") dyr_file = joinpath(raw_data, "psse_dyr", "OMIB.dyr") pm_data = PowerFlowFileParser.PowerModelsData(file_path) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) return sys end @@ -112,7 +112,7 @@ function build_psse_3bus_gen_cls_sys(; raw_data, kwargs...) file_path = joinpath(raw_data, "psse_raw", "ThreeBusNetwork.raw") dyr_file = joinpath(raw_data, "psse_dyr", "TestGENCLS.dyr") pm_data = PowerFlowFileParser.PowerModelsData(file_path) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) return sys end @@ -122,7 +122,7 @@ function psse_renewable_parsing_1(; raw_data, kwargs...) file_path = joinpath(raw_data, "psse_raw", "Benchmark_4ger_33_2015_RENA.RAW") dyr_file = joinpath(raw_data, "psse_dyr", "Benchmark_4ger_33_2015_RENA.dyr") pm_data = PowerFlowFileParser.PowerModelsData(file_path) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) return sys end @@ -132,7 +132,7 @@ function build_psse_3bus_sexs_sys(; raw_data, kwargs...) file_path = joinpath(raw_data, "psse_raw", "ThreeBusNetwork.raw") dyr_file = joinpath(raw_data, "psse_dyr", "test_SEXS.dyr") pm_data = PowerFlowFileParser.PowerModelsData(file_path) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) return sys end @@ -142,7 +142,7 @@ function build_psse_original_240_case(; raw_data, kwargs...) file_path = joinpath(raw_data, "psse_raw", "240busWECC_2018_PSS33.raw") dyr_file = joinpath(raw_data, "psse_dyr", "240busWECC_2018_PSS.dyr") pm_data = PowerFlowFileParser.PowerModelsData(file_path) - sys = make_system( + sys = system_from_openapi( pm_data; bus_name_formatter = x -> string(x["name"]) * "-" * string(x["index"]), sys_kwargs..., @@ -156,7 +156,7 @@ function build_psse_3bus_no_cls_sys(; raw_data, kwargs...) file_path = joinpath(raw_data, "psse_raw", "ThreeBusNetwork.raw") dyr_file = joinpath(raw_data, "psse_dyr", "Test-NoCLS.dyr") pm_data = PowerFlowFileParser.PowerModelsData(file_path) - sys = make_system(pm_data; sys_kwargs...) + sys = system_from_openapi(pm_data; sys_kwargs...) add_dyn_injectors!(sys, dyr_file) return sys end diff --git a/src/parsers/common.jl b/src/parsers/common.jl deleted file mode 100644 index 4f49b3fe..00000000 --- a/src/parsers/common.jl +++ /dev/null @@ -1,185 +0,0 @@ -const SKIP_PM_VALIDATION = false - -const GENERATOR_MAPPING_FILE_PM = - joinpath( - dirname(pathof(PowerSystemCaseBuilder)), - "parsers", - "generator_mapping_pm.yaml", - ) - -const PSSE_DYR_MAPPING_FILE = - joinpath( - dirname(pathof(PowerSystemCaseBuilder)), - "parsers", - "psse_dynamic_mapping.yaml", - ) - -const STRING2FUEL = - Dict( - (normalize(string(x); casefold = true) => x) for x in instances(ThermalFuels.Value) - ) -merge!( - STRING2FUEL, - Dict( - "ng" => ThermalFuels.NATURAL_GAS, - "nuc" => ThermalFuels.NUCLEAR, - "gas" => ThermalFuels.NATURAL_GAS, - "oil" => ThermalFuels.DISTILLATE_FUEL_OIL, - "dfo" => ThermalFuels.DISTILLATE_FUEL_OIL, - "sync_cond" => ThermalFuels.OTHER, - "geothermal" => ThermalFuels.GEOTHERMAL, - "ag_byproduct" => ThermalFuels.AG_BYPRODUCT, - ), -) - -const STRING2PRIMEMOVER = - Dict((normalize(string(x); casefold = true) => x) for x in instances(PrimeMovers.Value)) -merge!( - STRING2PRIMEMOVER, - Dict( - "w2" => PrimeMovers.WT, - "wind" => PrimeMovers.WT, - "pv" => PrimeMovers.PVe, - "solar" => PrimeMovers.PVe, - "rtpv" => PrimeMovers.PVe, - "nb" => PrimeMovers.ST, - "steam" => PrimeMovers.ST, - "hydro" => PrimeMovers.HY, - "ror" => PrimeMovers.HY, - "pump" => PrimeMovers.PS, - "pumped_hydro" => PrimeMovers.PS, - "nuclear" => PrimeMovers.ST, - "sync_cond" => PrimeMovers.OT, - "csp" => PrimeMovers.CP, - "un" => PrimeMovers.OT, - "storage" => PrimeMovers.BA, - "ice" => PrimeMovers.IC, - ), -) - -# Stays until `make_system(::PowerModelsData)` goes: that path resolves each mapping to a -# `DataType`, unlike PowerTableDataParser's, which resolves to a type name. -"""Return a dict where keys are a tuple of input parameters (fuel, unit_type) and values are -generator types.""" -function get_generator_mapping(filename::String) - genmap = open(filename) do file - YAML.load(file) - end - - mappings = Dict{NamedTuple, DataType}() - for (gen_type, vals) in genmap - if gen_type == "GenericBattery" - @warn "GenericBattery type is no longer supported. The new type is EnergyReservoirStorage" - gen = EnergyReservoirStorage - else - gen = getfield(PowerSystems, Symbol(gen_type)) - end - for val in vals - key = (fuel = val["fuel"], unit_type = val["type"]) - if haskey(mappings, key) - error("duplicate generator mappings: $gen $(key.fuel) $(key.unit_type)") - end - mappings[key] = gen - end - end - - return mappings -end - -# Union{...,String} predates the deleted power_system_table_data.jl; PowerTableDataParser.jl -# now owns that String-mapping path via its own get_generator_type. Only the -# power_models_data.jl (DataType) call site remains here. - -"""Return the PowerSystems generator type for this fuel and unit_type.""" -function get_generator_type( - fuel, - unit_type, - mappings::Union{Dict{NamedTuple, String}, Dict{NamedTuple, DataType}}, -) - fuel = isnothing(fuel) ? "" : uppercase(fuel) - unit_type = uppercase(unit_type) - generator = nothing - - # Try to match the unit_type if it's defined. If it's nothing then just match on fuel. - for ut in (unit_type, nothing), fu in (fuel, nothing) - key = (fuel = fu, unit_type = ut) - if haskey(mappings, key) - if typeof(mappings[key]) == DataType - generator = mappings[key] - elseif typeof(mappings[key]) == String - gen_type = mappings[key] - generator = getfield(PowerSystems, Symbol(gen_type)) - end - break - end - end - - if isnothing(generator) - @error "No mapping for generator fuel=$fuel unit_type=$unit_type" - end - - return generator -end - -function calculate_gen_rating( - active_power_limits::Union{PSY.MinMax, Nothing}, - reactive_power_limits::Union{PSY.MinMax, Nothing}, - base_conversion::Float64, -) - reactive_power_max = isnothing(reactive_power_limits) ? 0.0 : reactive_power_limits.max - return calculate_gen_rating( - active_power_limits.max, - reactive_power_max, - base_conversion, - ) -end - -function calculate_gen_rating( - active_power_max::Float64, - reactive_power_max::Float64, - base_conversion::Float64, -) - rating = sqrt(active_power_max^2 + reactive_power_max^2) - if rating == 0.0 - @warn "Rating calculation returned 0.0. Changing to 1.0 in the p.u. of the device." - return 1.0 - end - return rating * base_conversion -end - -function calculate_ramp_limit( - d::Dict{String, Any}, - gen_name::Union{SubString{String}, String}, -) - if haskey(d, "ramp_agc") - return (up = d["ramp_agc"], down = d["ramp_agc"]) - end - if haskey(d, "ramp_10") - return (up = d["ramp_10"], down = d["ramp_10"]) - end - if haskey(d, "ramp_30") - return (up = d["ramp_30"], down = d["ramp_30"]) - end - if abs(d["pmax"]) > 0.0 - @debug "No ramp limits found for generator $(gen_name). Using pmax as ramp limit." - return (up = abs(d["pmax"]), down = abs(d["pmax"])) - end - @warn "Not enough information to determine ramp limit for generator $(gen_name). Returning nothing" - return nothing -end - -function parse_enum_mapping(::Type{ThermalFuels.Value}, fuel::AbstractString) - return STRING2FUEL[normalize(fuel; casefold = true)] -end - -function parse_enum_mapping(::Type{ThermalFuels.Value}, fuel::Symbol) - return parse_enum_mapping(ThermalFuels.Value, string(fuel)) -end - -function parse_enum_mapping(::Type{PrimeMovers.Value}, prime_mover::AbstractString) - return STRING2PRIMEMOVER[normalize(prime_mover; casefold = true)] -end - -function parse_enum_mapping(::Type{PrimeMovers.Value}, prime_mover::Symbol) - return parse_enum_mapping(PrimeMovers.Value, string(prime_mover)) -end diff --git a/src/parsers/generator_mapping_pm.yaml b/src/parsers/generator_mapping_pm.yaml deleted file mode 100644 index 4ed6851d..00000000 --- a/src/parsers/generator_mapping_pm.yaml +++ /dev/null @@ -1,33 +0,0 @@ -# Parsing code ignores type=null. - -HydroTurbine: -- {fuel: HYDRO, type: null} -- {fuel: HYDRO, type: HYDRO} - -HydroDispatch: -- {fuel: HYDRO, type: ROR} - -RenewableDispatch: -- {fuel: SOLAR, type: PV} -- {fuel: SOLAR, type: UN} -- {fuel: WIND, type: WIND} -- {fuel: WIND, type: null} -- {fuel: SOLAR, type: CSP} # TODO: may need a new struct - -RenewableNonDispatch: -- {fuel: SOLAR, type: RTPV} - -ThermalStandard: -- {fuel: OIL, type: null} -- {fuel: COAL, type: null} -- {fuel: NG, type: null} -- {fuel: GAS, type: null} -- {fuel: NUCLEAR, type: null} -- {fuel: NUC, type: null} -- {fuel: OTHER, type: OT} - -SynchronousCondenser: -- {fuel: SYNC_COND, type: SYNC_COND} - -EnergyReservoirStorage: -- {fuel: STORAGE, type: null} diff --git a/src/parsers/known_stale_fixture_data.jl b/src/parsers/known_stale_fixture_data.jl index 89dadefb..7187efd8 100644 --- a/src/parsers/known_stale_fixture_data.jl +++ b/src/parsers/known_stale_fixture_data.jl @@ -120,3 +120,26 @@ function _with_timeseries_metadata_file( end return T(values...) end + +""" +Matpower fixture data that no Sienna type represents, and that PowerFlowFileParser's +OpenAPI builder therefore rejects rather than skips. Matched exactly so any new data of +either kind still errors. + +`ne_branch` (case3_tnep.m, case5_tnep.m): PowerModels' transmission-expansion candidate +branches. + +`313_STORAGE_1` (RTS_GMLC.m, gen row 158): a storage unit listed as a generator. Matpower +storage belongs in its own `storage` table, which this case does not have. +""" +const KNOWN_UNREAD_MATPOWER_SECTIONS = ("ne_branch",) +const KNOWN_STORAGE_GENERATOR_ROWS = Set(["313_STORAGE_1"]) + +function drop_known_unread_matpower_data!(pm_data::PowerFlowFileParser.PowerModelsData) + data = pm_data.data + for section in KNOWN_UNREAD_MATPOWER_SECTIONS + delete!(data, section) + end + filter!(p -> get(last(p), "name", nothing) ∉ KNOWN_STORAGE_GENERATOR_ROWS, data["gen"]) + return pm_data +end diff --git a/src/parsers/openapi_pipeline.jl b/src/parsers/openapi_pipeline.jl index 8258b186..f8873a0d 100644 --- a/src/parsers/openapi_pipeline.jl +++ b/src/parsers/openapi_pipeline.jl @@ -33,10 +33,9 @@ end """ Build a `System` from table data. `time_series_resolution` keeps only the series at that -resolution, matching the retired `make_system`'s kwarg of the same name; the filter is -applied to the parser's staged series, before the sidecar is written, because the sidecar's -catalog — not the document's association rows — is what PowerSystems reads the series back -through. +resolution; the filter is applied to the parser's staged series, before the sidecar is +written, because the sidecar's catalog — not the document's association rows — is what +PowerSystems reads the series back through. """ function system_from_openapi( rawsys::PowerTableDataParser.PowerSystemTableData; @@ -59,8 +58,19 @@ errors on an LCC line: `_hvdc_loss_curve` (PSY `import_handwritten.jl`) reads a field off the loss curve's function data, which only exists after a JSON round trip normalizes it — PSY's importer is written against the round-tripped shape. Take the shortcut once that is fixed, not before. + +The document carries no system base, so the case's `baseMVA` is handed to the importer +directly; left out, it would default to 100 MVA. """ function system_from_openapi(pm_data::PowerFlowFileParser.PowerModelsData; kwargs...) oapi = PowerFlowFileParser.build_openapi_system(pm_data; kwargs...) - return system_from_document(oapi; filter_kwargs(; kwargs...)...) + sys = system_from_document( + oapi; + base_power = Float64(pm_data.data["baseMVA"]), + filter_kwargs(; kwargs...)..., + ) + if get(kwargs, :runchecks, true) + check(sys) + end + return sys end diff --git a/src/parsers/power_models_data.jl b/src/parsers/power_models_data.jl deleted file mode 100644 index 106aaf7a..00000000 --- a/src/parsers/power_models_data.jl +++ /dev/null @@ -1,2135 +0,0 @@ -""" -Constructs a System from PowerFlowFileParser.PowerModelsData. - -# Arguments -- `pm_data::Union{PowerFlowFileParser.PowerModelsData, Union{String, IO}}`: PowerFlowFileParser.PowerModels -data object or supported load flow case (*.m, *.raw) - -# Keyword arguments -- `ext::Dict`: Contains user-defined parameters. Should only contain standard types. -- `runchecks::Bool`: Run available checks on input fields and when add_component! is called. - Throws InvalidValue if an error is found. -- `time_series_in_memory::Bool=false`: Store time series data in memory instead of HDF5. -- `config_path::String`: specify path to validation config file -- `pm_data_corrections::Bool=true` : Run the PowerModels data corrections (aka :validate in PowerModels) -- `import_all:Bool=false` : Import all fields from PTI files - -# Examples -```julia -sys = System( - pm_data, config_path = "ACTIVSg25k_validation.json", - bus_name_formatter = x->string(x["name"]*"-"*string(x["index"])), - load_name_formatter = x->strip(join(x["source_id"], "_")) -) -``` -""" -function make_system(pm_data::PowerFlowFileParser.PowerModelsData; kwargs...) - runchecks = get(kwargs, :runchecks, true) - data = pm_data.data - if length(data["bus"]) < 1 - throw(IS.DataFormatError("There are no buses in this file.")) - end - - @info "Constructing System from Power Models" data["name"] data["source_type"] - - sys_kwargs = filter_kwargs(; kwargs...) - sys = System(data["baseMVA"]; sys_kwargs...) - - source_type = data["source_type"] - - bus_number_to_bus = read_bus!(sys, data; kwargs...) - read_loads!(sys, data, bus_number_to_bus; kwargs...) - read_loadzones!(sys, data, bus_number_to_bus; kwargs...) - read_gen!(sys, data, bus_number_to_bus; kwargs...) - for component_type in ["switch", "breaker", "generic_connector"] - read_switch_breaker!(sys, data, bus_number_to_bus, component_type; kwargs...) - end - read_branch!(sys, data, bus_number_to_bus; kwargs...) - read_switched_shunt!(sys, data, bus_number_to_bus; kwargs...) - read_shunt!(sys, data, bus_number_to_bus; kwargs...) - read_dcline!(sys, data, bus_number_to_bus, source_type; kwargs...) - read_vscline!(sys, data, bus_number_to_bus; kwargs...) - read_facts!(sys, data, bus_number_to_bus; kwargs...) - read_storage!(sys, data, bus_number_to_bus; kwargs...) - read_3w_transformer!(sys, data, bus_number_to_bus; kwargs...) - if runchecks - check(sys) - end - - substation_data = get(data, "substation", Dict{String, Any}()) - add_geographic_info_to_buses!(sys, substation_data) - - return sys -end - -""" -Construct a System from a PSS/E raw file via the PowerModels dict pipeline. Thin shim -over [`make_system`](@ref); the metadata-reimport path threads its name-formatter kwargs -through it. `solved_case = true` declares the RAW written out after a converged power flow, -so every switched shunt keeps its BINIT as `solved_admittance`. -""" -function system_via_power_models( - file_path::AbstractString; - solved_case::Bool = false, - kwargs..., -) - pm_data = PowerFlowFileParser.PowerModelsData(file_path; solved_case = solved_case) - return make_system(pm_data; kwargs...) -end - -""" -Internal component name retrieval from pm2ps_dict -""" -function _get_pm_dict_name(device_dict::Dict)::String - if haskey(device_dict, "shunt_bus") - # With shunts, we have FixedAdmittance and SwitchedAdmittance types. - # To avoid potential name collision, we add the connected bus number to the name. - name = join(strip.(string.((device_dict["shunt_bus"], device_dict["name"]))), "-") - elseif haskey(device_dict, "name") - name = string(device_dict["name"]) - elseif haskey(device_dict, "source_id") - name = strip(join(string.(device_dict["source_id"]), "-")) - else - name = string(device_dict["index"]) - end - return name -end - -function _get_pm_bus_name(device_dict::Dict, unique_names::Bool) - if haskey(device_dict, "name") - if unique_names - name = strip(device_dict["name"]) - else - name = strip(device_dict["name"]) * "_" * string(device_dict["bus_i"]) - end - else - name = strip(join(string.(device_dict["source_id"]), "-")) - end - return name -end - -""" -Internal branch name retrieval from pm2ps_dict -""" -function _get_pm_branch_name(device_dict, bus_f::ACBus, bus_t::ACBus) - # Additional if-else are used to catch line id in PSSe parsing cases - if haskey(device_dict, "name") - index = device_dict["name"] - elseif device_dict["source_id"][1] == "branch" && length(device_dict["source_id"]) > 2 - index = strip(device_dict["source_id"][4]) - elseif ( - device_dict["source_id"][1] == "switch" || - device_dict["source_id"][1] == "breaker" || - device_dict["source_id"][1] == "generic_connector" - ) && length(device_dict["source_id"]) > 2 - # Legacy switches/breakers modeled as branches carry a marker-prefixed CKT - # (e.g. "@1", "*2"); strip the leading '@'/'*' marker to get the circuit id. - # Switching devices from a SWITCHING DEVICE record (including node-breaker - # substation devices) carry a plain circuit id with no marker. - ckt = strip(string(device_dict["source_id"][4])) - index = (!isempty(ckt) && first(ckt) in ('@', '*')) ? ckt[2:end] : ckt - elseif device_dict["source_id"][1] == "transformer" && - length(device_dict["source_id"]) > 3 - index = strip(device_dict["source_id"][5]) - else - index = device_dict["index"] - end - return "$(get_name(bus_f))-$(get_name(bus_t))-i_$index" -end - -""" -Internal 3WT name retrieval from pm2ps_dict -""" -function _get_pm_3w_name( - device_dict, - bus_primary::ACBus, - bus_secondary::ACBus, - bus_tertiary::ACBus, -) - ckt = device_dict["circuit"] - return "$(get_name(bus_primary))-$(get_name(bus_secondary))-$(get_name(bus_tertiary))-i_$ckt" -end - -"""Add geographic coordinates to all buses using pre-built lookup""" -function add_geographic_info_to_buses!(sys, substation_data) - if isempty(substation_data) - @warn "No substation data found" - return - end - - bus_coords_lookup = Dict{Int, GeographicInfo}() - substation_coords_lookup = Dict{Any, GeographicInfo}() - - for (_, substation) in substation_data - if haskey(substation, "nodes") && haskey(substation, "latitude") && - haskey(substation, "longitude") - lat, lon = substation["latitude"], substation["longitude"] - - geo_info = GeographicInfo(; - geo_json = Dict( - "type" => "Point", - "coordinates" => [lon, lat], - ), - ) - for node in substation["nodes"] - if haskey(node, "bus") - bus_coords_lookup[node["bus"]] = geo_info - end - end - if haskey(substation, "index") - substation_coords_lookup[substation["index"]] = geo_info - end - end - end - - begin_supplemental_attributes_update(sys) do - buses_with_coords = 0 - buses_without_coords = 0 - - for bus in get_components(ACBus, sys) - bus_number = get_number(bus) - - geo_info = get(bus_coords_lookup, bus_number, nothing) - if isnothing(geo_info) - # Injected node-breaker buses are keyed on a synthetic bus number, not - # the original PSS(R)E bus, so fall back to the substation they were - # split from (recorded in ext by node-breaker materialization). - nb_substation = get(get_ext(bus), "nb_substation", nothing) - if !isnothing(nb_substation) - geo_info = get(substation_coords_lookup, nb_substation, nothing) - end - end - - if !isnothing(geo_info) - add_supplemental_attribute!(sys, bus, geo_info) - buses_with_coords += 1 - else - buses_without_coords += 1 - end - end - - @info "Added coordinates to $(buses_with_coords) buses, $(buses_without_coords) buses without coordinates" - end -end - -""" -Parses ITC data from a dictionary and constructs a lookup table -of piecewise linear scaling functions. -""" -function _impedance_correction_table_lookup(data::Dict) - ict_instances = Dict{Tuple{Int64, WindingCategory.Value}, ImpedanceCorrectionData}() - - @info "Reading Impedance Correction Table data" - if !haskey(data, "impedance_correction") - @info "There is no Impedance Correction Table data in this file" - return ict_instances - end - - for (_, table_data) in data["impedance_correction"] - table_number = table_data["table_number"] - x = table_data["tap_or_angle"] - y = table_data["scaling_factor"] - - if length(x) == length(y) - if length(x) < 2 - @warn "Skipping impedance correction entry due to insufficient data points ($(length(x)) < 2): $(x)" - continue - end - pwl_data = PiecewiseLinearData([(x[i], y[i]) for i in eachindex(x)]) - table_type = - if ( - x[1] >= PSSE_PARSER_TAP_RATIO_LBOUND && - x[1] <= PSSE_PARSER_TAP_RATIO_UBOUND - ) - ImpedanceCorrectionTransformerControlMode.TAP_RATIO - else - ImpedanceCorrectionTransformerControlMode.PHASE_SHIFT_ANGLE - end - - for winding_index in instances(WindingCategory.Value) - ict_instances[(table_number, winding_index)] = ImpedanceCorrectionData(; - table_number = table_number, - impedance_correction_curve = pwl_data, - transformer_winding = winding_index, - transformer_control_mode = table_type, - ) - end - else - throw( - IS.DataFormatError( - "Impedance correction mismatch at table $table_number: tap/angle and scaling count differs.", - ), - ) - end - end - - return ict_instances -end - -""" -Function to attach ICTs to a single Transformer component. -""" -function _attach_single_ict!( - sys::System, - transformer::Union{TwoWindingTransformer, ThreeWindingTransformer}, - name::String, - d::Dict, - table_key::String, - winding_idx::WindingCategory.Value, - ict_instances::Dict{Tuple{Int64, WindingCategory.Value}, ImpedanceCorrectionData}, -) - if isempty(ict_instances) - return - end - if haskey(d, table_key) - table_number = d[table_key] - cache_key = (table_number, winding_idx) - if haskey(ict_instances, cache_key) - ict = ict_instances[cache_key] - add_supplemental_attribute!(sys, transformer, ict) - else - @debug "No correction table associated with transformer $name for winding $winding_idx." - end - end - return -end - -""" -Attaches the corresponding ICT data to a TwoWindingTransformer component. -""" -function _attach_impedance_correction_tables!( - sys::System, - transformer::TwoWindingTransformer, - name::String, - d::Dict, - ict_instances::Dict{Tuple{Int64, WindingCategory.Value}, ImpedanceCorrectionData}, -) - _attach_single_ict!( - sys, - transformer, - name, - d, - "correction_table", - WindingCategory.TR2W_WINDING, - ict_instances, - ) - return -end - -""" -Attaches the corresponding ICT data to a ThreeWindingTransformer component. -""" -function _attach_impedance_correction_tables!( - sys::System, - transformer::ThreeWindingTransformer, - name::String, - d::Dict, - ict_instances::Dict{Tuple{Int64, WindingCategory.Value}, ImpedanceCorrectionData}, -) - if isempty(ict_instances) - return - end - for winding_category in instances(WindingCategory.Value) - winding_category == WindingCategory.TR2W_WINDING && continue - key = "$(WINDING_NAMES[winding_category])_correction_table" - _attach_single_ict!(sys, transformer, name, d, key, winding_category, ict_instances) - end - return -end - -""" -Creates a PowerSystems.ACBus from a PowerSystems bus dictionary -""" -function make_bus(bus_dict::Dict{String, Any}) - bus = ACBus( - bus_dict["number"], - bus_dict["name"], - bus_dict["available"], - bus_dict["bustype"], - bus_dict["angle"], - bus_dict["voltage"], - bus_dict["voltage_limits"], - bus_dict["base_voltage"], - bus_dict["area"], - bus_dict["zone"], - ) - return bus -end - -function make_bus( - bus_name::Union{String, SubString{String}}, - bus_number::Int, - d, - bus_types, - area::Area, -) - bus = make_bus( - Dict{String, Any}( - "name" => bus_name, - "number" => bus_number, - "available" => d["bus_status"], - "bustype" => bus_types[d["bus_type"]], - "angle" => d["va"], - "voltage" => d["vm"], - "voltage_limits" => (min = d["vmin"], max = d["vmax"]), - "base_voltage" => d["base_kv"], - "area" => area, - "zone" => nothing, - ), - ) - return bus -end - -# Disabling this because not all matpower files define areas even when bus definitions -# contain area references. -#function read_area!(sys::System, data::Dict; kwargs...) -# if !haskey(data, "areas") -# @info "There are no Areas in this file" -# return -# end -# -# for (key, val) in data["areas"] -# area = Area(string(val["col_1"])) -# add_component!(sys, area; skip_validation = SKIP_PM_VALIDATION) -# end -#end - -function read_bus!(sys::System, data::Dict; kwargs...) - @info "Reading bus data" - - bus_number_to_bus = Dict{Int, ACBus}() - - bus_types = instances(ACBusTypes.Value) - unique_bus_names = true - bus_data = SortedDict{Int, Any}() - # Bus name uniqueness is not enforced by PSSE. This loop avoids forcing the users to have to - # pass the bus formatter always for larger datasets. - bus_names = Set{String}() - for (k, b) in data["bus"] - # If buses aren't unique stop searching and growing the set - if unique_bus_names && haskey(b, "name") - if b["name"] ∈ bus_names - unique_bus_names = false - end - push!(bus_names, b["name"]) - end - bus_data[k] = b - end - if isempty(bus_data) - @error "No bus data found" # TODO : need for a model without a bus - end - - default_bus_naming = x -> _get_pm_bus_name(x, unique_bus_names) - - _get_name = get(kwargs, :bus_name_formatter, default_bus_naming) - - default_area_naming = string - # The formatter for area_name should be a function that transform the Area Int to a String - _get_name_area = get(kwargs, :area_name_formatter, default_area_naming) - - for (i, (d_key, d)) in enumerate(bus_data) - # d id the data dict for each bus - # d_key is bus key - bus_name = strip(_get_name(d)) - bus_number = Int(d["bus_i"]) - - area_name = _get_name_area(d["area"]) - area = get_component(Area, sys, area_name) - if isnothing(area) - area = Area(area_name) - add_component!(sys, area; skip_validation = SKIP_PM_VALIDATION) - end - - # Store area data into ext dictionary - ext = Dict{String, Any}( - "ARNAME" => "", - "I" => "", - "ISW" => "", - "PDES" => "", - "PTOL" => "", - ) - if data["source_type"] == "pti" && haskey(data, "area_interchange") - for (_, area_data) in data["area_interchange"] - if haskey(area_data, "area_number") && - string(area_data["area_number"]) == area_name - ext["ARNAME"] = strip(get(area_data, "area_name", "")) - ext["I"] = string(get(area_data, "area_number", "")) - ext["ISW"] = string(get(area_data, "bus_number", "")) - ext["PDES"] = get(area_data, "net_interchange", "") - ext["PTOL"] = get(area_data, "tol_interchange", "") - break # Only one match is allowed - end - end - end - set_ext!(area, ext) - if !haskey(d, "bus_status") - d["bus_status"] = true - end - bus = make_bus(bus_name, bus_number, d, bus_types, area) - has_component(ACBus, sys, bus_name) && throw( - IS.DataFormatError( - "Found duplicate bus names for $(get_name(bus)), consider reviewing your `bus_name_formatter` function", - ), - ) - - bus_number_to_bus[bus.number] = bus - add_component!(sys, bus; skip_validation = SKIP_PM_VALIDATION) - - if get(d, "area_slack", false) == true - set_bustype!(bus, ACBusTypes.SLACK) - end - bus_ext = get(d, "ext", Dict{String, Any}()) - if !isempty(bus_ext) - merge!(get_ext(bus), bus_ext) - end - end - - if data["source_type"] == "pti" && haskey(data, "interarea_transfer") - # get Inter-area Transfers as AreaInterchange - for (k, d) in data["interarea_transfer"] - area_from_name = _get_name_area(d["area_from"]) - area_to_name = _get_name_area(d["area_to"]) - transfer_id = get(d, "transfer_id", "1") # 1 by default - - from_area = get_component(Area, sys, area_from_name) - to_area = get_component(Area, sys, area_to_name) - - name = "$(area_from_name)_$(area_to_name)_$(transfer_id)" - - if isnothing(from_area) || isnothing(to_area) - missing_areas = join( - filter( - !isnothing, - [ - isnothing(from_area) ? area_from_name : nothing, - isnothing(to_area) ? area_to_name : nothing, - ], - ), - ", ", - ) - @warn "Inter-area transfer record $k references undefined area(s) $missing_areas; skipping AreaInterchange $name" - continue - end - - available = true - active_power_flow = d["power_transfer"] - flow_limits = (from_to = -INFINITE_BOUND, to_from = INFINITE_BOUND) - - ext = Dict{String, Any}( - "index" => d["index"], - "source_id" => ["interarea_transfer", k], - ) - - interarea_inter = AreaInterchange(; - name = name, - available = available, - active_power_flow = active_power_flow, - from_area = from_area, - to_area = to_area, - flow_limits = flow_limits, - ext = ext, - input_basis = CU, - ) - - add_component!(sys, interarea_inter; skip_validation = SKIP_PM_VALIDATION) - end - end - - return bus_number_to_bus -end - -function make_interruptible_powerload(d::Dict, bus::ACBus, sys_mbase::Float64; kwargs...) - operation_cost = LoadCost(; - variable_operation_cost = zero(CostCurve), - fixed = 0.0, - ) - - _get_name = get(kwargs, :load_name_formatter, x -> strip(join(x["source_id"]))) - return InterruptiblePowerLoad(; - name = _get_name(d), - available = d["status"], - bus = bus, - active_power = d["pd"], - reactive_power = d["qd"], - max_active_power = d["pd"], - max_reactive_power = d["qd"], - base_power = sys_mbase, - operation_cost = operation_cost, - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function make_interruptible_standardload(d::Dict, bus::ACBus, sys_mbase::Float64; kwargs...) - operation_cost = LoadCost(; - variable_operation_cost = zero(CostCurve), - fixed = 0.0, - ) - - _get_name = get(kwargs, :load_name_formatter, x -> strip(join(x["source_id"]))) - return InterruptibleStandardLoad(; - name = _get_name(d), - available = d["status"], - bus = bus, - base_power = sys_mbase, - conformity = d["conformity"], - operation_cost = operation_cost, - constant_active_power = d["pd"], - constant_reactive_power = d["qd"], - current_active_power = d["pi"], - current_reactive_power = d["qi"], - impedance_active_power = d["py"], - impedance_reactive_power = d["qy"], - max_constant_active_power = d["pd"], - max_constant_reactive_power = d["qd"], - max_current_active_power = d["pi"], - max_current_reactive_power = d["qi"], - max_impedance_active_power = d["py"], - max_impedance_reactive_power = d["qy"], - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function make_power_load(d::Dict, bus::ACBus, sys_mbase::Float64; kwargs...) - _get_name = get(kwargs, :load_name_formatter, x -> strip(join(x["source_id"]))) - return PowerLoad(; - name = _get_name(d), - available = d["status"], - bus = bus, - active_power = d["pd"], - reactive_power = d["qd"], - max_active_power = d["pd"], - max_reactive_power = d["qd"], - base_power = sys_mbase, - conformity = d["conformity"], - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function make_standard_load(d::Dict, bus::ACBus, sys_mbase::Float64; kwargs...) - _get_name = get(kwargs, :load_name_formatter, x -> strip(join(x["source_id"]))) - return StandardLoad(; - name = _get_name(d), - available = d["status"], - bus = bus, - constant_active_power = d["pd"], - constant_reactive_power = d["qd"], - current_active_power = d["pi"], - current_reactive_power = d["qi"], - impedance_active_power = d["py"], - impedance_reactive_power = d["qy"], - max_constant_active_power = d["pd"], - max_constant_reactive_power = d["qd"], - max_current_active_power = d["pi"], - max_current_reactive_power = d["qi"], - max_impedance_active_power = d["py"], - max_impedance_reactive_power = d["qy"], - base_power = sys_mbase, - conformity = d["conformity"], - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function read_loads!(sys::System, data, bus_number_to_bus::Dict{Int, ACBus}; kwargs...) - @info "Reading Load data in PowerModels dict to populate System ..." - - if !haskey(data, "load") - @error "There are no loads in this file" - return - end - - sys_mbase = data["baseMVA"] - - dgen_lookup = Dict{Tuple{Int, String}, Dict}() - for dgen in values(get(data, "distributed_generation", Dict{String, Any}())) - dgen_lookup[(dgen["bus"], strip(string(dgen["source_id"][3])))] = dgen - end - unmatched_dgen_keys = Set(keys(dgen_lookup)) - - for d_key in keys(data["load"]) - d = data["load"][d_key] - bus = bus_number_to_bus[d["load_bus"]] - is_interruptible = haskey(d, "interruptible") - if data["source_type"] == "pti" && is_interruptible && d["interruptible"] != 1 - load = make_standard_load(d, bus, sys_mbase; kwargs...) - has_component(StandardLoad, sys, get_name(load)) && throw( - IS.DataFormatError( - "Found duplicate load names of $(summary(load)), consider formatting names with `load_name_formatter` kwarg", - ), - ) - elseif data["source_type"] == "pti" && is_interruptible && d["interruptible"] == 1 - load = make_interruptible_standardload(d, bus, sys_mbase; kwargs...) - has_component(InterruptibleStandardLoad, sys, get_name(load)) && throw( - IS.DataFormatError( - "Found duplicate interruptible load names of $(summary(load)), consider formatting names with `load_name_formatter` kwarg", - ), - ) - else - load = make_power_load(d, bus, sys_mbase; kwargs...) - has_component(PowerLoad, sys, get_name(load)) && throw( - IS.DataFormatError( - "Found duplicate load names of $(summary(load)), consider formatting names with `load_name_formatter` kwarg", - ), - ) - end - add_component!(sys, load; skip_validation = SKIP_PM_VALIDATION) - - load_source_id = get(d, "source_id", String[]) - if length(load_source_id) >= 3 && load_source_id[1] == "load" - dgen_key = (d["load_bus"], strip(string(load_source_id[3]))) - dgen = get(dgen_lookup, dgen_key, nothing) - if !isnothing(dgen) - delete!(unmatched_dgen_keys, dgen_key) - dgen_load = RenewableNonDispatch(; - name = string(get_name(load), "_dgen"), - available = Bool(dgen["status"]), - bus = bus, - active_power = dgen["pg"], - reactive_power = dgen["qg"], - rating = hypot(dgen["pg"], dgen["qg"]), - prime_mover_type = PrimeMovers.OT, - # The dgen injector mirrors the upstream distributed-generation - # contract, which reports net P/Q with a unity-power-factor placeholder. - power_factor = 1.0, - base_power = sys_mbase, - input_basis = CU, - ) - has_component(RenewableNonDispatch, sys, get_name(dgen_load)) && throw( - IS.DataFormatError( - "Found duplicate RenewableNonDispatch names of $(get_name(dgen_load)), consider formatting names with `load_name_formatter` kwarg", - ), - ) - add_component!(sys, dgen_load; skip_validation = SKIP_PM_VALIDATION) - end - end - end - - for (bus_number, id) in unmatched_dgen_keys - @warn "Distributed generation entry on bus $bus_number with id \"$id\" did not match any load; skipping" - end -end - -function make_loadzone( - name::String, - active_power::Float64, - reactive_power::Float64; - kwargs..., -) - return LoadZone(; - name = name, - peak_active_power = active_power, - peak_reactive_power = reactive_power, - input_basis = CU, - ) -end - -function read_loadzones!( - sys::System, - data::Dict{String, Any}, - bus_number_to_bus::Dict{Int, ACBus}; - kwargs..., -) - @info "Reading LoadZones data in PowerModels dict to populate System ..." - zones = Set{Int}() - zone_bus_map = Dict{Int, Vector}() - for (_, bus) in data["bus"] - push!(zones, bus["zone"]) - push!(get!(zone_bus_map, bus["zone"], Vector()), bus) - end - - load_zone_map = - Dict{Int, Dict{String, Float64}}(i => Dict("pd" => 0.0, "qd" => 0.0) for i in zones) - for (key, load) in data["load"] - zone = data["bus"][load["load_bus"]]["zone"] - load_zone_map[zone]["pd"] += load["pd"] - load_zone_map[zone]["qd"] += load["qd"] - # Use get with defaults because matpower data doesn't have other load representations - load_zone_map[zone]["pd"] += get(load, "pi", 0.0) - load_zone_map[zone]["qd"] += get(load, "qi", 0.0) - load_zone_map[zone]["pd"] += get(load, "py", 0.0) - load_zone_map[zone]["qd"] += get(load, "qy", 0.0) - end - - default_loadzone_naming = string - # The formatter for loadzone_name should be a function that transform the LoadZone Int to a String - _get_name = get(kwargs, :loadzone_name_formatter, default_loadzone_naming) - - @info "Reading Zone data" - if !haskey(data, "zone") - @info "There is no Zone data in this file" - else - for (_, v) in data["zone"] - zone_number = v["zone_number"] - if !(zone_number in zones) - @warn "Skipping empty LoadZone $(zone_number)-$(v["zone_name"])" - end - end - end - - for zone in zones - name = _get_name(zone) - load_zone = make_loadzone( - name, - load_zone_map[zone]["pd"], - load_zone_map[zone]["qd"]; - kwargs..., - ) - add_component!(sys, load_zone; skip_validation = SKIP_PM_VALIDATION) - for bus in zone_bus_map[zone] - set_load_zone!(bus_number_to_bus[bus["bus_i"]], load_zone) - end - end -end - -function make_hydro_dispatch( - gen_name::Union{SubString{String}, String}, - d::Dict, - bus::ACBus, - sys_mbase::Float64, -) - curtailcost = HydroGenerationCost(zero(CostCurve), 0.0) - - if d["mbase"] != 0.0 - mbase = d["mbase"] - else - @warn "Generator $gen_name has base power equal to zero: $(d["mbase"]). Changing it to system base: $sys_mbase" _group = - IS.LOG_GROUP_PARSING - mbase = sys_mbase - end - - base_conversion = sys_mbase / mbase - return HydroDispatch(; # No way to define storage parameters for gens in PM so can only make HydroDispatch - name = gen_name, - available = Bool(d["gen_status"]), - bus = bus, - active_power = d["pg"] * base_conversion, - reactive_power = d["qg"] * base_conversion, - rating = calculate_gen_rating(d["pmax"], d["qmax"], base_conversion), - prime_mover_type = parse_enum_mapping(PrimeMovers.Value, d["type"]), - active_power_limits = ( - min = d["pmin"] * base_conversion, - max = d["pmax"] * base_conversion, - ), - reactive_power_limits = ( - min = d["qmin"] * base_conversion, - max = d["qmax"] * base_conversion, - ), - ramp_limits = calculate_ramp_limit(d, gen_name), - time_limits = nothing, - operation_cost = curtailcost, - base_power = mbase, - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function make_hydro_reservoir( - gen_name::Union{SubString{String}, String}, - d::Dict, - bus::ACBus, - sys_mbase::Float64, -) - curtailcost = HydroGenerationCost(zero(CostCurve), 0.0) - - if d["mbase"] != 0.0 - mbase = d["mbase"] - else - @warn "Generator $gen_name has base power equal to zero: $(d["mbase"]). Changing it to system base: $sys_mbase" _group = - IS.LOG_GROUP_PARSING - mbase = sys_mbase - end - - base_conversion = sys_mbase / mbase - return HydroDispatch(; # No way to define storage parameters for gens in PM so can only make HydroDispatch - name = gen_name, - available = Bool(d["gen_status"]), - bus = bus, - active_power = d["pg"] * base_conversion, - reactive_power = d["qg"] * base_conversion, - rating = calculate_gen_rating(d["pmax"], d["qmax"], base_conversion), - prime_mover_type = parse_enum_mapping(PrimeMovers.Value, d["type"]), - active_power_limits = ( - min = d["pmin"] * base_conversion, - max = d["pmax"] * base_conversion, - ), - reactive_power_limits = ( - min = d["qmin"] * base_conversion, - max = d["qmax"] * base_conversion, - ), - ramp_limits = calculate_ramp_limit(d, gen_name), - time_limits = nothing, - operation_cost = curtailcost, - base_power = mbase, - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function make_renewable_dispatch( - gen_name::Union{SubString{String}, String}, - d::Dict, - bus::ACBus, - sys_mbase::Float64, -) - cost = RenewableGenerationCost(zero(CostCurve)) - - if d["mbase"] != 0.0 - mbase = d["mbase"] - else - @warn "Generator $gen_name has base power equal to zero: $(d["mbase"]). Changing it to system base: $sys_mbase" _group = - IS.LOG_GROUP_PARSING - mbase = sys_mbase - end - - base_conversion = sys_mbase / mbase - - rating = calculate_gen_rating(d["pmax"], d["qmax"], base_conversion) - if rating > mbase - @warn "rating is larger than base power for $gen_name, setting to $mbase" _group = - IS.LOG_GROUP_PARSING - rating = mbase - end - - generator = RenewableDispatch(; - name = gen_name, - available = Bool(d["gen_status"]), - bus = bus, - active_power = d["pg"] * base_conversion, - reactive_power = d["qg"] * base_conversion, - rating = rating * base_conversion, - prime_mover_type = parse_enum_mapping(PrimeMovers.Value, d["type"]), - reactive_power_limits = ( - min = d["qmin"] * base_conversion, - max = d["qmax"] * base_conversion, - ), - power_factor = 1.0, - operation_cost = cost, - base_power = mbase, - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) - - return generator -end - -function make_renewable_fix( - gen_name::Union{SubString{String}, String}, - d::Dict, - bus::ACBus, - sys_mbase::Float64, -) - if d["mbase"] != 0.0 - mbase = d["mbase"] - else - @warn "Generator $gen_name has base power equal to zero: $(d["mbase"]). Changing it to system base: $sys_mbase" _group = - IS.LOG_GROUP_PARSING - mbase = sys_mbase - end - - base_conversion = sys_mbase / mbase - generator = RenewableNonDispatch(; - name = gen_name, - available = Bool(d["gen_status"]), - bus = bus, - active_power = d["pg"] * base_conversion, - reactive_power = d["qg"] * base_conversion, - rating = float(d["pmax"]) * base_conversion, - prime_mover_type = parse_enum_mapping(PrimeMovers.Value, d["type"]), - power_factor = 1.0, - base_power = mbase, - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) - - return generator -end - -function make_generic_battery( - storage_name::Union{SubString{String}, String}, - d::Dict, - bus::ACBus, -) - energy_rating = iszero(d["energy_rating"]) ? d["energy"] : d["energy_rating"] - storage = EnergyReservoirStorage(; - name = storage_name, - available = Bool(d["status"]), - bus = bus, - prime_mover_type = PrimeMovers.BA, - storage_technology_type = StorageTech.OTHER_CHEM, - storage_capacity = energy_rating, - storage_level_limits = (min = 0.0, max = energy_rating), - initial_storage_capacity_level = d["energy"] / energy_rating, - rating = d["thermal_rating"], - active_power = d["ps"], - input_active_power_limits = (min = 0.0, max = d["charge_rating"]), - output_active_power_limits = (min = 0.0, max = d["discharge_rating"]), - efficiency = (in = d["charge_efficiency"], out = d["discharge_efficiency"]), - reactive_power = d["qs"], - reactive_power_limits = (min = d["qmin"], max = d["qmax"]), - base_power = d["thermal_rating"], - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) - return storage -end - -function _is_likely_motor_load(d::Dict, gen_name::Union{SubString{String}, String}) - # A motor load is likely if it has a negative active power and a non-zero reactive power. - # This is a heuristic and may not be accurate for all cases. - # likely_motor_load - if d["pmin"] < 0 && d["pmax"] < 0 && d["pg"] < 0 - @warn "Generator $gen_name is likely a motor load with negative active power: $(d["pg"]) and negative power limits: (min = $(d["pmin"]), max = $(d["pmax"])) \ - this component will be parsed as a thermal generator with negative active power limits. You can convert the device to a MotorLoad for more accurate modeling." - end - - if d["pmin"] == 0 && d["pmax"] == 0 && d["pg"] < 0 - @warn "Generator $gen_name is likely a motor load with negative active power: $(d["pg"]) and undefined active power limits \ - this component will be parsed as a thermal generator with negative active power injection. You can convert the device to a MotorLoad for more accurate modeling." - end - - if d["pmin"] < 0 && d["pmax"] == 0 - @warn "Generator $gen_name is likely something that is not a ThermalGenerators with negative power limits: (min = $(d["pmin"]), max = $(d["pmax"])) \ - this component will be parsed as a thermal generator with negative active power limits. Check this entry for more accurate modeling." - end - return -end - -function _thermal_status(gen_status) - if !iszero(gen_status) - return OperationalStates.ONLINE - else - return OperationalStates.OFFLINE - end -end - -# TODO test this more directly? -""" -The polynomial term follows the convention that for an n-degree polynomial, at least n + 1 components are needed. - c(p) = c_n*p^n+...+c_1p+c_0 - c_o is stored in the field in of the Econ Struct -""" -function make_thermal_gen( - gen_name::Union{SubString{String}, String}, - d::Dict, - bus::ACBus, - sys_mbase::Float64, -) - if haskey(d, "model") - model = PSY.GeneratorCostModels.Value(d["model"]) - # Input data layout: table B-4 of https://matpower.org/docs/MATPOWER-manual.pdf - if model == PSY.GeneratorCostModels.PIECEWISE_LINEAR - # For now, we make the fixed cost the y-intercept of the first segment of the - # piecewise curve and the variable cost a PiecewiseLinearData representing - # the data minus this fixed cost; in a future update, there will be no - # separation between the PiecewiseLinearData and the fixed cost. - cost_component = d["cost"] - power_p = [i for (ix, i) in enumerate(cost_component) if isodd(ix)] - cost_p = [i for (ix, i) in enumerate(cost_component) if iseven(ix)] - points = collect(zip(power_p, cost_p)) - (first_x, first_y) = first(points) - fixed = max(0.0, - first_y - first(get_slopes(PiecewiseLinearData(points))) * first_x, - ) - cost = PiecewiseLinearData([(x, y - fixed) for (x, y) in points]) - elseif model == PSY.GeneratorCostModels.POLYNOMIAL - # For now, we make the variable cost a QuadraticFunctionData with all but the - # constant term and make the fixed cost the constant term; in a future update, - # there will be no separation between the QuadraticFunctionData and the fixed - # cost. - # This transforms [3.0, 1.0, 4.0, 2.0] into [(1, 4.0), (2, 1.0), (3, 3.0)] - coeffs = enumerate(reverse(d["cost"][1:(end - 1)])) - coeffs = Dict((i, c / sys_mbase^i) for (i, c) in coeffs) - quadratic_degrees = [2, 1, 0] - (keys(coeffs) <= Set(quadratic_degrees)) || throw( - ArgumentError( - "Can only handle polynomials up to degree two; given coefficients $coeffs", - ), - ) - cost = QuadraticFunctionData(get.(Ref(coeffs), quadratic_degrees, 0)...) - fixed = (d["ncost"] >= 1) ? last(d["cost"]) : 0.0 - end - cost = CostCurve(InputOutputCurve((cost)), IS.CU) - startup = d["startup"] - shutdn = d["shutdown"] - else - @warn "Generator cost data not included for Generator: $gen_name" - tmpcost = ThermalGenerationCost(nothing) - cost = tmpcost.variable_operation_cost - fixed = tmpcost.fixed - startup = tmpcost.start_up - shutdn = tmpcost.shut_down - end - - operation_cost = ThermalGenerationCost(; - variable_operation_cost = cost, - fixed = fixed, - start_up = startup, - shut_down = shutdn, - ) - - if !haskey(d, "ext") - d["ext"] = Dict{String, Float64}() - end - - if haskey(d, "r_source") && haskey(d, "x_source") - d["ext"]["r"] = d["r_source"] - d["ext"]["x"] = d["x_source"] - end - - if haskey(d, "rt_source") && haskey(d, "xt_source") - d["ext"]["rt"] = d["rt_source"] - d["ext"]["xt"] = d["xt_source"] - end - - if d["mbase"] != 0.0 - mbase = d["mbase"] - else - @warn "Generator $gen_name has base power equal to zero: $(d["mbase"]). Changing it to system base: $sys_mbase" _group = - IS.LOG_GROUP_PARSING - mbase = sys_mbase - end - - base_conversion = sys_mbase / mbase - _is_likely_motor_load(d, gen_name) - thermal_gen = ThermalStandard(; - name = gen_name, - status = _thermal_status(d["gen_status"]), - available = Bool(d["gen_status"]), - bus = bus, - active_power = d["pg"] * base_conversion, - reactive_power = d["qg"] * base_conversion, - rating = calculate_gen_rating(d["pmax"], d["qmax"], base_conversion), - prime_mover_type = parse_enum_mapping(PrimeMovers.Value, d["type"]), - fuel = parse_enum_mapping(ThermalFuels.Value, d["fuel"]), - active_power_limits = ( - min = d["pmin"] * base_conversion, - max = d["pmax"] * base_conversion, - ), - reactive_power_limits = ( - min = d["qmin"] * base_conversion, - max = d["qmax"] * base_conversion, - ), - ramp_limits = calculate_ramp_limit(d, gen_name), - time_limits = nothing, - operation_cost = operation_cost, - base_power = mbase, - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) - - return thermal_gen -end - -function make_synchronous_condenser( - gen_name::Union{SubString{String}, String}, - d::Dict, - bus::ACBus, - sys_mbase::Float64, -) - ext = get(d, "ext", Dict{String, Any}()) - if haskey(d, "r_source") && haskey(d, "x_source") - ext["r"] = d["r_source"] - ext["x"] = d["x_source"] - end - - if haskey(d, "rt_source") && haskey(d, "xt_source") - ext["rt"] = d["rt_source"] - ext["xt"] = d["xt_source"] - end - - if d["mbase"] != 0.0 - mbase = d["mbase"] - else - @warn "Generator $gen_name has base power equal to zero: $(d["mbase"]). Changing it to system base: $sys_mbase" _group = - IS.LOG_GROUP_PARSING - mbase = sys_mbase - end - - # NOTE: qmax and qmin can be both negatives, so this approach is taken for the rating. - base_conversion = sys_mbase / mbase - synchronous_condenser = SynchronousCondenser(; - name = gen_name, - available = Bool(d["gen_status"]), - bus = bus, - reactive_power = d["qg"] * base_conversion, - rating = max(abs(d["qmax"]), abs(d["qmin"])) * base_conversion, - reactive_power_limits = ( - min = d["qmin"] * base_conversion, - max = d["qmax"] * base_conversion, - ), - base_power = mbase, - ext = ext, - input_basis = CU, - ) - - return synchronous_condenser -end - -""" -Transfer generators to ps_dict according to their classification -""" -function read_gen!(sys::System, data::Dict, bus_number_to_bus::Dict{Int, ACBus}; kwargs...) - @info "Reading generator data" - - if !haskey(data, "gen") - @error "There are no Generators in this file" - return nothing - end - - generator_mapping = get(kwargs, :generator_mapping, GENERATOR_MAPPING_FILE_PM) - try - generator_mapping = get_generator_mapping(generator_mapping) - catch e - @error "Error loading generator mapping $(generator_mapping)" - rethrow(e) - end - - sys_mbase = data["baseMVA"] - - _get_name = get(kwargs, :gen_name_formatter, _get_pm_dict_name) - for (name, pm_gen) in data["gen"] - gen_name = _get_name(pm_gen) - - bus = bus_number_to_bus[pm_gen["gen_bus"]] - pm_gen["fuel"] = get(pm_gen, "fuel", "OTHER") - pm_gen["type"] = get(pm_gen, "type", "OT") - @debug "Found generator" _group = IS.LOG_GROUP_PARSING gen_name bus pm_gen["fuel"] pm_gen["type"] - - gen_type = get_generator_type(pm_gen["fuel"], pm_gen["type"], generator_mapping) - if gen_type == ThermalStandard - generator = make_thermal_gen(gen_name, pm_gen, bus, sys_mbase) - elseif gen_type == HydroDispatch - generator = make_hydro_dispatch(gen_name, pm_gen, bus, sys_mbase) - elseif gen_type == HydroTurbine - # This method adds a - generator = make_hydro_reservoir(gen_name, pm_gen, bus, sys_mbase) - elseif gen_type == RenewableDispatch - generator = make_renewable_dispatch(gen_name, pm_gen, bus, sys_mbase) - elseif gen_type == RenewableNonDispatch - generator = make_renewable_fix(gen_name, pm_gen, bus, sys_mbase) - elseif gen_type == SynchronousCondenser - generator = make_synchronous_condenser(gen_name, pm_gen, bus, sys_mbase) - elseif gen_type == EnergyReservoirStorage - @warn "EnergyReservoirStorage should be defined as a PowerModels storage... Skipping" - continue - else - @error "Skipping unsupported generator" gen_type - continue - end - - has_component(typeof(generator), sys, get_name(generator)) && throw( - IS.DataFormatError( - "Found duplicate $(typeof(generator)) names of $(get_name(generator)), consider formatting names with `gen_name_formatter` kwarg", - ), - ) - add_component!(sys, generator; skip_validation = SKIP_PM_VALIDATION) - end -end - -# Tap changing and phase shifting are winding data (`tap`, `α`, `control`), not -# distinct component types, so these type-inference helpers only distinguish -# Line / switch / transformer. -function get_branch_type_matpower( - d::Dict, -) - tap = d["tap"] - shift = d["shift"] - is_transformer = d["transformer"] - if !is_transformer - is_transformer = (tap != 0.0) && (tap != 1.0) || (shift != 0.0) - end - - is_transformer || return Line - return TwoWindingTransformer -end - -function get_branch_type_psse( - d::Dict, -) - if d["br_r"] == 0.0 && d["br_x"] == 0.0 - return DiscreteControlledACBranch - end - - is_transformer = d["transformer"] - tap = d["tap"] - - if !is_transformer - if (tap != 0.0) && (tap != 1.0) - @warn "Transformer $d has tap ratio $tap, which is not 0.0 or 1.0; this is not a valid value for a Line. Parsing entry as a Transformer" - else - return Line - end - end - - return TwoWindingTransformer -end - -function make_branch( - name::String, - d::Dict, - bus_f::ACBus, - bus_t::ACBus, - source_type::String, -) - if source_type == "matpower" - branch_type = get_branch_type_matpower(d) - elseif source_type == "pti" - branch_type = get_branch_type_psse(d) - else - error("Source Type $source_type not supported") - end - - if d["transformer"] && branch_type == Line - throw( - IS.DataFormatError( - "Branch data mismatched, cannot build the branch correctly for $d", - ), - ) - elseif branch_type == DiscreteControlledACBranch - value = _make_switch_from_zero_impedance_line(name, d, bus_f, bus_t) - elseif branch_type == TwoWindingTransformer - value = make_transformer_2w(name, d, bus_f, bus_t) - elseif branch_type == Line - value = make_line(name, d, bus_f, bus_t) - else - error("Unsupported branch type $branch_type") - end - return value -end - -function _make_switch_from_zero_impedance_line( - name::String, - d::Dict, - bus_f::ACBus, - bus_t::ACBus, -) - pf = get(d, "pf", 0.0) - qf = get(d, "qf", 0.0) - available_value = d["br_status"] == 1 - if get_bustype(bus_f) == ACBusTypes.ISOLATED || - get_bustype(bus_t) == ACBusTypes.ISOLATED - available_value = false - end - if available_value == true - status_value = DiscreteControlledBranchStatus.CLOSED - else - status_value = DiscreteControlledBranchStatus.OPEN - end - @warn "Branch $name has zero impedance and available = $available_value; converting to a DiscreteControlledACBranch of type SWITCH with available = $available_value and branch_status = $status_value" - return DiscreteControlledACBranch(; - name = name, - available = Bool(available_value), - active_power_flow = pf, - reactive_power_flow = qf, - arc = Arc(bus_f, bus_t), - r = d["br_r"], - x = d["br_x"], - rating = _get_rating("Line", name, d, "rate_a"), - discrete_branch_type = DiscreteControlledBranchType.SWITCH, - branch_status = status_value, - input_basis = CU, - ) -end - -function _get_rating( - branch_type::String, - name::AbstractString, - line_data::Dict, - key::String, -) - haskey(line_data, key) || return key == "rate_a" ? INFINITE_BOUND : nothing - - if isapprox(line_data[key], 0.0) - @info( - "$branch_type $name rating value: $(line_data[key]). Unbounded value implied as per PSSe Manual" - ) - return INFINITE_BOUND - else - return line_data[key] - end -end - -function make_line(name::String, d::Dict, bus_f::ACBus, bus_t::ACBus) - pf = get(d, "pf", 0.0) - qf = get(d, "qf", 0.0) - available_value = d["br_status"] == 1 - if get_bustype(bus_f) == ACBusTypes.ISOLATED || - get_bustype(bus_t) == ACBusTypes.ISOLATED - available_value = false - end - - ext = haskey(d, "ext") ? d["ext"] : Dict{String, Any}() - - return Line(; - name = name, - available = available_value, - active_power_flow = pf, - reactive_power_flow = qf, - arc = Arc(bus_f, bus_t), - r = d["br_r"], - x = d["br_x"], - b = (from = d["b_fr"], to = d["b_to"]), - rating = _get_rating("Line", name, d, "rate_a"), - angle_limits = (min = d["angmin"], max = d["angmax"]), - rating_b = _get_rating("Line", name, d, "rate_b"), - rating_c = _get_rating("Line", name, d, "rate_c"), - ext = ext, - input_basis = CU, - ) -end - -function make_switch_breaker(name::String, d::Dict, bus_f::ACBus, bus_t::ACBus) - return DiscreteControlledACBranch(; - name = name, - available = Bool(d["state"]), - active_power_flow = d["active_power_flow"], - reactive_power_flow = d["reactive_power_flow"], - arc = Arc(bus_f, bus_t), - r = d["r"], - x = d["x"], - rating = d["rating"], - discrete_branch_type = d["discrete_branch_type"], - branch_status = d["state"], - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function read_switch_breaker!( - sys::System, - data::Dict, - bus_number_to_bus::Dict{Int, ACBus}, - device_type::String; - kwargs..., -) - @info "Reading $device_type data" - if !haskey(data, device_type) - @info "There is no $device_type data in this file" - return - end - - _get_name = get(kwargs, :branch_name_formatter, _get_pm_branch_name) - - for (_, d) in data[device_type] - bus_f = bus_number_to_bus[d["f_bus"]] - bus_t = bus_number_to_bus[d["t_bus"]] - name = _get_name(d, bus_f, bus_t) - value = make_switch_breaker(name, d, bus_f, bus_t) - - add_component!(sys, value; skip_validation = SKIP_PM_VALIDATION) - end -end - -# COD values whose control objective is a phase-shift (angle) control rather -# than a tap (voltage/reactive) control. -const _PSSE_PHASE_SHIFT_OBJECTIVES = ( - TransformerControlObjective.ACTIVE_POWER_FLOW, - TransformerControlObjective.ACTIVE_POWER_FLOW_DISABLED, - TransformerControlObjective.ASYMMETRIC_ACTIVE_POWER_FLOW, - TransformerControlObjective.ASYMMETRIC_ACTIVE_POWER_FLOW_DISABLED, -) - -""" -Resolve the flat per-winding control fields from a PowerModels transformer dict -`d` for winding `suffix` (1/2/3), mirroring the PSS/E per-winding control block -(COD/CONT/RMA/RMI/VMA/VMI/NTP). Returns a NamedTuple of the winding's flattened -control keyword arguments. - -`COD == -99` (or a missing `COD` key) yields `control_objective = UNDEFINED`, the -null state (no control block). Any other COD — including `0` (FIXED) and negative -(disabled) codes — is preserved as data. Matpower records carry no COD keys, so -`control_objective` stays `UNDEFINED` there (the `get` defaults handle it). -""" -function _transformer_control_fields(d::Dict, suffix::Int) - cod = get(d, "COD$suffix", -99) - objective = TransformerControlObjective.Value(cod) - phase_shifting = objective in _PSSE_PHASE_SHIFT_OBJECTIVES - # RMI/RMA and VMI/VMA are the lower/upper edges of a band. Some (typically - # synthetic) PSS/E data has them numerically inverted by rounding - # (e.g. frankenstein_70.raw has VMA1 = 0.984 < VMI1 = 0.985); warn and - # normalize the ordering so a valid band is produced rather than tripping - # the attach-time band-ordering check. The warning surfaces genuinely - # corrupt bands on actively-controlled windings instead of hiding them. - record = string(get(d, "name", get(d, "source_id", "unknown"))) - # RMA/RMI are PSS/E degrees for phase-shift CODs; -180/180 converts to the - # documented radian default (-π, π) below rather than the tap-band (0.9, 1.1). - if phase_shifting - rmi_default, rma_default = -180.0, 180.0 - else - rmi_default, rma_default = 0.9, 1.1 - end - rmi, rma = get(d, "RMI$suffix", rmi_default), get(d, "RMA$suffix", rma_default) - if rmi > rma - @warn "Transformer record $record winding $suffix has inverted control limits RMI$suffix = $rmi > RMA$suffix = $rma; normalizing to (min = $rma, max = $rmi)." - rmi, rma = rma, rmi - end - if phase_shifting - rmi, rma = deg2rad(rmi), deg2rad(rma) - end - vmi, vma = get(d, "VMI$suffix", 0.9), get(d, "VMA$suffix", 1.1) - if vmi > vma - @warn "Transformer record $record winding $suffix has inverted controlled-quantity limits VMI$suffix = $vmi > VMA$suffix = $vma; normalizing to (min = $vma, max = $vmi)." - vmi, vma = vma, vmi - end - return ( - control_objective = objective, - regulated_bus_number = get(d, "CONT$suffix", 0), - control_limits = (min = rmi, max = rma), - controlled_quantity_limits = (min = vmi, max = vma), - number_of_tap_positions = get(d, "NTP$suffix", 33), - ) -end - -""" -Build a [`TransformerCircuit`](@ref) from a PowerModels transformer dict `d`, -mapping the per-winding keys named by the keyword arguments. Shared by the 2W -maker (one circuit) and the 3W maker (three circuits); ratings are resolved by -the caller. -""" -function _make_transformer_circuit( - d::Dict, - arc::Arc; - tap_key::String, - angle_key::String, - control_suffix::Int, - available::Bool, - r, - x, - rating, - rating_b = nothing, - rating_c = nothing, - base_power, - base_voltage_primary, - base_voltage_secondary, - active_power_flow, - reactive_power_flow, -) - control = _transformer_control_fields(d, control_suffix) - return TransformerCircuit(; - arc = arc, - tap = get(d, tap_key, 1.0), - α = d[angle_key], - r = r, - x = x, - control_objective = control.control_objective, - regulated_bus_number = control.regulated_bus_number, - control_limits = control.control_limits, - controlled_quantity_limits = control.controlled_quantity_limits, - number_of_tap_positions = control.number_of_tap_positions, - available = available, - rating = rating, - rating_b = rating_b, - rating_c = rating_c, - active_power_flow = active_power_flow, - reactive_power_flow = reactive_power_flow, - base_power = base_power, - base_voltage_primary = base_voltage_primary, - base_voltage_secondary = base_voltage_secondary, - input_basis = CU, - ) -end - -# matpower files frequently leave bus base voltages unspecified (baseKV = 0); -# represent that as `nothing` (unknown) rather than a literal 0, which the -# transformer winding validation (base_voltage must be positive) rejects. -_base_voltage_or_nothing(v) = iszero(v) ? nothing : v - -function make_transformer_2w( - name::String, - d::Dict, - bus_f::ACBus, - bus_t::ACBus, -) - pf = get(d, "pf", 0.0) - qf = get(d, "qf", 0.0) - available_value = d["br_status"] == 1 - if get_bustype(bus_f) == ACBusTypes.ISOLATED || - get_bustype(bus_t) == ACBusTypes.ISOLATED - available_value = false - end - - # `d["base_power"]` is the winding (device) base, which is the transformer's - # device base. - base_power = d["base_power"] - - # BASE CONVENTION: `br_r`/`br_x` reach this maker already expressed in device - # base on `base_power` (PowerFlowFileParser converts PSSE data to device base - # in `psse.jl`, and `make_per_unit!` does not touch `br_r`/`br_x`; for - # matpower `base_power == system base` so device base == system base). - # The circuit's `r`/`x` store the series impedance in device base on - # `base_power`, so no rebasing is applied here. - circuit = _make_transformer_circuit( - d, - Arc(bus_f, bus_t); - tap_key = "tap", - angle_key = "shift", - control_suffix = 1, - available = available_value, - r = d["br_r"], - x = d["br_x"], - rating = _get_rating("TwoWindingTransformer", name, d, "rate_a"), - rating_b = _get_rating("TwoWindingTransformer", name, d, "rate_b"), - rating_c = _get_rating("TwoWindingTransformer", name, d, "rate_c"), - base_power = base_power, - # for psse inputs, this may differ from the buses' base voltages - base_voltage_primary = _base_voltage_or_nothing(d["base_voltage_from"]), - base_voltage_secondary = _base_voltage_or_nothing(d["base_voltage_to"]), - active_power_flow = pf, - reactive_power_flow = qf, - ) - - return TwoWindingTransformer(; - name = name, - circuit = circuit, - magnetizing_shunt = Complex(d["g_fr"], d["b_fr"]), - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function make_3w_transformer( - name::String, - d::Dict, - bus_primary::ACBus, - bus_secondary::ACBus, - bus_tertiary::ACBus, - star_bus::ACBus, -) - pf = get(d, "pf", 0.0) - qf = get(d, "qf", 0.0) - - # Each winding's device (winding) base power is the pairwise base of the pair - # whose first index is that winding: primary -> base_power_12, - # secondary -> base_power_23, tertiary -> base_power_31. The star-leg - # impedances (r_primary/x_primary, ...) reach this maker already on their - # respective winding device bases (PowerFlowFileParser performs the - # delta->star conversion), matching the circuit field semantics, so no - # rebasing is applied. The magnetizing shunt is transformer-level and lives - # on the parent (PRIMARY = winding-1 terminal side). The star side has no - # distinct base voltage, so each circuit's secondary base voltage defaults - # to its primary. - primary_circuit = _make_transformer_circuit( - d, - Arc(bus_primary, star_bus); - tap_key = "primary_turns_ratio", - angle_key = "primary_phase_shift_angle", - control_suffix = 1, - available = Bool(d["available_primary"]), - r = d["r_primary"], - x = d["x_primary"], - rating = _get_rating("ThreeWindingTransformer", name, d, "rating_primary"), - base_power = d["base_power_12"], - base_voltage_primary = d["base_voltage_primary"], - base_voltage_secondary = d["base_voltage_primary"], - active_power_flow = pf, - reactive_power_flow = qf, - ) - secondary_circuit = _make_transformer_circuit( - d, - Arc(bus_secondary, star_bus); - tap_key = "secondary_turns_ratio", - angle_key = "secondary_phase_shift_angle", - control_suffix = 2, - available = Bool(d["available_secondary"]), - r = d["r_secondary"], - x = d["x_secondary"], - rating = _get_rating("ThreeWindingTransformer", name, d, "rating_secondary"), - base_power = d["base_power_23"], - base_voltage_primary = d["base_voltage_secondary"], - base_voltage_secondary = d["base_voltage_secondary"], - active_power_flow = pf, - reactive_power_flow = qf, - ) - tertiary_circuit = _make_transformer_circuit( - d, - Arc(bus_tertiary, star_bus); - tap_key = "tertiary_turns_ratio", - angle_key = "tertiary_phase_shift_angle", - control_suffix = 3, - available = Bool(d["available_tertiary"]), - r = d["r_tertiary"], - x = d["x_tertiary"], - rating = _get_rating("ThreeWindingTransformer", name, d, "rating_tertiary"), - base_power = d["base_power_31"], - base_voltage_primary = d["base_voltage_tertiary"], - base_voltage_secondary = d["base_voltage_tertiary"], - active_power_flow = pf, - reactive_power_flow = qf, - ) - - return ThreeWindingTransformer(; - name = name, - primary_circuit = primary_circuit, - secondary_circuit = secondary_circuit, - tertiary_circuit = tertiary_circuit, - magnetizing_shunt = Complex(d["g"], d["b"]), - star_bus = star_bus, - r_12 = d["r_12"], - x_12 = d["x_12"], - r_23 = d["r_23"], - x_23 = d["x_23"], - r_31 = d["r_31"], - x_31 = d["x_31"], - base_power_12 = d["base_power_12"], - base_power_23 = d["base_power_23"], - base_power_31 = d["base_power_31"], - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function read_branch!( - sys::System, - data::Dict, - bus_number_to_bus::Dict{Int, ACBus}; kwargs..., -) - @info "Reading branch data" - if !haskey(data, "branch") - @info "There is no Branch data in this file" - return - end - - _get_name = get(kwargs, :branch_name_formatter, _get_pm_branch_name) - ict_instances = _impedance_correction_table_lookup(data) - - source_type = data["source_type"] - for d in values(data["branch"]) - bus_f = bus_number_to_bus[d["f_bus"]] - bus_t = bus_number_to_bus[d["t_bus"]] - name = _get_name(d, bus_f, bus_t) - value = make_branch(name, d, bus_f, bus_t, source_type) - - if !isnothing(value) - add_component!(sys, value; skip_validation = SKIP_PM_VALIDATION) - else - continue - end - - if isa(value, TwoWindingTransformer) - _attach_impedance_correction_tables!( - sys, - value, - name, - d, - ict_instances, - ) - end - end - return -end - -function read_3w_transformer!( - sys::System, - data::Dict, - bus_number_to_bus::Dict{Int, ACBus}; - kwargs..., -) - @info "Reading 3W transformer data" - if !haskey(data, "3w_transformer") - @info "There is no 3W transformer data in this file" - return - end - - _get_name = get(kwargs, :xfrm_3w_name_formatter, _get_pm_3w_name) - - ict_instances = _impedance_correction_table_lookup(data) - - for (_, d) in data["3w_transformer"] - bus_primary = bus_number_to_bus[d["bus_primary"]] - bus_secondary = bus_number_to_bus[d["bus_secondary"]] - bus_tertiary = bus_number_to_bus[d["bus_tertiary"]] - star_bus = bus_number_to_bus[d["star_bus"]] - - name = _get_name(d, bus_primary, bus_secondary, bus_tertiary) - value = make_3w_transformer( - name, - d, - bus_primary, - bus_secondary, - bus_tertiary, - star_bus, - ) - - add_component!(sys, value; skip_validation = SKIP_PM_VALIDATION) - - _attach_impedance_correction_tables!(sys, value, name, d, ict_instances) - end -end - -# PSY's loss fields are `LossCurve`s, which name the basis their curve is on. Input -# documents carry a bare curve, so one is supplied here: these parsers read MW-denominated -# loss coefficients, which is `NaturalUnit`. A value that already arrives as a `LossCurve` -# keeps the basis it came with. -_as_loss_curve(loss::AnyLossCurve) = loss -_as_loss_curve(curve::ValueCurve) = LossCurve(curve, NaturalUnit()) - -# PSY stores an LCC's power-mode `transfer_setpoint` per-unit on its base (always the system -# base); PSS/E's SETVL is MW. A current-mode setpoint is amperes and has no power base. -function _lcc_transfer_setpoint(d::Dict, base_power::Float64) - if d["power_mode"] - return d["transfer_setpoint"] / base_power - end - return d["transfer_setpoint"] -end - -function make_dcline( - name::String, - d::Dict, - bus_f::ACBus, - bus_t::ACBus, - source_type::String, - base_power::Float64, -) - if source_type == "pti" - return TwoTerminalLCCLine(; - name = name, - available = d["available"], - arc = Arc(bus_f, bus_t), - active_power_flow = get(d, "pf", 0.0), - r = d["r"], - transfer_setpoint = _lcc_transfer_setpoint(d, base_power), - scheduled_dc_voltage = d["scheduled_dc_voltage"], - rectifier_bridges = d["rectifier_bridges"], - rectifier_delay_angle_limits = d["rectifier_delay_angle_limits"], - rectifier_rc = d["rectifier_rc"], - rectifier_xc = d["rectifier_xc"], - rectifier_base_voltage = d["rectifier_base_voltage"], - inverter_bridges = d["inverter_bridges"], - inverter_extinction_angle_limits = d["inverter_extinction_angle_limits"], - inverter_rc = d["inverter_rc"], - inverter_xc = d["inverter_xc"], - inverter_base_voltage = d["inverter_base_voltage"], - power_mode = d["power_mode"], - switch_mode_voltage = d["switch_mode_voltage"], - compounding_resistance = d["compounding_resistance"], - min_compounding_voltage = d["min_compounding_voltage"], - rectifier_transformer_ratio = d["rectifier_transformer_ratio"], - rectifier_tap_setting = d["rectifier_tap_setting"], - rectifier_tap_limits = d["rectifier_tap_limits"], - rectifier_tap_step = d["rectifier_tap_step"], - rectifier_delay_angle = d["rectifier_delay_angle"], - rectifier_capacitor_reactance = d["rectifier_capacitor_reactance"], - inverter_transformer_ratio = d["inverter_transformer_ratio"], - inverter_tap_setting = d["inverter_tap_setting"], - inverter_tap_limits = d["inverter_tap_limits"], - inverter_tap_step = d["inverter_tap_step"], - inverter_extinction_angle = d["inverter_extinction_angle"], - inverter_capacitor_reactance = d["inverter_capacitor_reactance"], - active_power_limits_from = d["active_power_limits_from"], - active_power_limits_to = d["active_power_limits_to"], - reactive_power_limits_from = d["reactive_power_limits_from"], - reactive_power_limits_to = d["reactive_power_limits_to"], - loss = LossCurve(LinearCurve(d["loss1"], d["loss0"]), NaturalUnit()), - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) - elseif source_type == "matpower" - return TwoTerminalGenericHVDCLine(; - name = name, - available = d["br_status"] == 1, - active_power_flow = get(d, "pf", 0.0), - arc = Arc(bus_f, bus_t), - active_power_limits_from = (min = d["pminf"], max = d["pmaxf"]), - active_power_limits_to = (min = d["pmint"], max = d["pmaxt"]), - reactive_power_limits_from = (min = d["qminf"], max = d["qmaxf"]), - reactive_power_limits_to = (min = d["qmint"], max = d["qmaxt"]), - loss = LossCurve(LinearCurve(d["loss1"], d["loss0"]), NaturalUnit()), - input_basis = CU, - ) - else - error("Not supported source type for DC lines: $source_type") - end -end - -function read_dcline!( - sys::System, - data::Dict, - bus_number_to_bus::Dict{Int, ACBus}, - source_type::String; - kwargs..., -) - @info "Reading DC Line data" - if !haskey(data, "dcline") - @info "There is no DClines data in this file" - return - end - - _get_name = get(kwargs, :dcline_name_formatter, _get_pm_branch_name) - - for (d_key, d) in data["dcline"] - d["name"] = get(d, "name", d_key) - bus_f = bus_number_to_bus[d["f_bus"]] - bus_t = bus_number_to_bus[d["t_bus"]] - name = _get_name(d, bus_f, bus_t) - dcline = make_dcline(name, d, bus_f, bus_t, source_type, get_base_power(sys, IS.NU)) - add_component!(sys, dcline; skip_validation = SKIP_PM_VALIDATION) - end -end - -# PSS/E encodes "no remote regulated bus" as REMOT = 0, but PSY's -# `remote_bus_control_*` is `Union{Nothing, Int}` with a valid range of `>= 1`, and spells -# local terminal-bus regulation as `nothing`. Passing 0 straight through trips the -# descriptor's `error` validation action. -function _psse_remote_bus(d::Dict, key::String) - remote_bus = get(get(d, "ext", Dict()), key, 0) - if iszero(remote_bus) - return nothing - end - return remote_bus -end - -function make_vscline(name::String, d::Dict, bus_f::ACBus, bus_t::ACBus) - return TwoTerminalVSCLine(; - name = name, - available = d["available"], - arc = Arc(bus_f, bus_t), - active_power_flow = get(d, "pf", 0.0), - rating = d["rating"], - active_power_limits_from = (min = d["pminf"], max = d["pmaxf"]), - active_power_limits_to = (min = d["pmint"], max = d["pmaxt"]), - g = d["r"] == 0.0 ? 0.0 : 1.0 / d["r"], - rated_dc_voltage = d["rated_dc_voltage"], - dc_current = get(d, "if", 0.0), - reactive_power_from = get(d, "qf", 0.0), - dc_control_from = if d["dc_voltage_control_from"] - VSCDCControlModes.DC_VOLTAGE - else - VSCDCControlModes.DC_POWER - end, - ac_control_from = if d["ac_voltage_control_from"] - VSCACControlModes.AC_VOLTAGE - else - VSCACControlModes.AC_REACTIVE_POWER - end, - dc_setpoint_from = d["dc_setpoint_from"], - ac_setpoint_from = d["ac_setpoint_from"], - # PSS/E VSC records carry no rated AC voltage; PowerFlowFileParser already - # stamps the terminal bus's base_kv onto base_voltage_from, so fall back to - # the bus itself only if that key is somehow absent. - rated_ac_voltage_from = get(d, "base_voltage_from", get_base_voltage(bus_f)), - converter_loss_from = _as_loss_curve(d["converter_loss_from"]), - max_dc_current_from = d["max_dc_current_from"], - rating_from = d["rating_from"], - reactive_power_limits_from = (min = d["qminf"], max = d["qmaxf"]), - power_factor_weighting_fraction_from = d["power_factor_weighting_fraction_from"], - remote_bus_control_from = _psse_remote_bus(d, "REMOT_FROM"), - rmpct_from = get(get(d, "ext", Dict()), "RMPCT_FROM", 100.0), - reactive_power_to = get(d, "qt", 0.0), - dc_control_to = if d["dc_voltage_control_to"] - VSCDCControlModes.DC_VOLTAGE - else - VSCDCControlModes.DC_POWER - end, - ac_control_to = if d["ac_voltage_control_to"] - VSCACControlModes.AC_VOLTAGE - else - VSCACControlModes.AC_REACTIVE_POWER - end, - dc_setpoint_to = d["dc_setpoint_to"], - ac_setpoint_to = d["ac_setpoint_to"], - rated_ac_voltage_to = get(d, "base_voltage_to", get_base_voltage(bus_t)), - converter_loss_to = _as_loss_curve(d["converter_loss_to"]), - max_dc_current_to = d["max_dc_current_to"], - rating_to = d["rating_to"], - remote_bus_control_to = _psse_remote_bus(d, "REMOT_TO"), - rmpct_to = get(get(d, "ext", Dict()), "RMPCT_TO", 100.0), - reactive_power_limits_to = (min = d["qmint"], max = d["qmaxt"]), - power_factor_weighting_fraction_to = d["power_factor_weighting_fraction_to"], - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function read_vscline!( - sys::System, - data::Dict, - bus_number_to_bus::Dict{Int, ACBus}; - kwargs..., -) - @info "Reading VSC Line data" - if !haskey(data, "vscline") - @info "There is no VSC lines data in this file" - return - end - - _get_name = get(kwargs, :vsc_line_name_formatter, _get_pm_branch_name) - - for (d_key, d) in data["vscline"] - d["name"] = get(d, "name", d_key) - if !haskey(bus_number_to_bus, d["f_bus"]) || !haskey(bus_number_to_bus, d["t_bus"]) - @warn "VSC line $d_key references undefined bus(es) (from = $(d["f_bus"]), to = $(d["t_bus"])); skipping" - continue - end - bus_f = bus_number_to_bus[d["f_bus"]] - bus_t = bus_number_to_bus[d["t_bus"]] - name = _get_name(d, bus_f, bus_t) - vscline = make_vscline(name, d, bus_f, bus_t) - add_component!(sys, vscline; skip_validation = SKIP_PM_VALIDATION) - end -end - -function make_switched_shunt(name::String, d::Dict, bus::ACBus) - control_mode_value = d["control_mode"] - valid_control_modes = map(Integer, instances(SwitchedAdmittanceControlMode.Value)) - if !(control_mode_value in valid_control_modes) - throw( - IS.DataFormatError( - "Switched shunt $name: unsupported MODSW control mode $control_mode_value", - ), - ) - end - - params = Dict( - :name => name, - :available => Bool(d["status"]), - :bus => bus, - :number_of_steps => d["step_number"], - :Y_increase => d["y_increment"], - :admittance_limits => d["admittance_limits"], - :control_mode => SwitchedAdmittanceControlMode.Value(control_mode_value), - :regulated_bus_number => d["regulated_bus_number"], - :ext => d["ext"], - ) - - if haskey(d, "number_engaged") - params[:number_engaged] = d["number_engaged"] - end - # BINIT arrives in its own key only when PFFP deems it authoritative; otherwise leave the - # PSY default so this path agrees with PFFP's OpenAPI importer. - if haskey(d, "solved_admittance") - params[:solved_admittance] = d["solved_admittance"] - end - - return SwitchedAdmittance(; params...) -end - -function read_switched_shunt!( - sys::System, - data::Dict, - bus_number_to_bus::Dict{Int, ACBus}; - kwargs..., -) - @info "Reading switched shunt data" - if !haskey(data, "switched_shunt") - @info "There is no switched shunt data in this file" - return - end - - _get_name = get(kwargs, :switched_shunt_name_formatter, _get_pm_dict_name) - - for (d_key, d) in data["switched_shunt"] - d["name"] = get(d, "name", d_key) - name = _get_name(d) - bus = bus_number_to_bus[d["shunt_bus"]] - shunt = make_switched_shunt(name, d, bus) - - add_component!(sys, shunt; skip_validation = SKIP_PM_VALIDATION) - end -end - -function make_shunt(name::String, d::Dict, bus::ACBus) - return FixedAdmittance(; - name = name, - available = Bool(d["status"]), - bus = bus, - Y = (d["gs"] + d["bs"]im), - ) -end - -function make_facts(name::String, d::Dict, bus::ACBus) - if d["tbus"] != 0 - @warn "Series FACTs not supported." - end - - if d["control_mode"] > 3 - throw(IS.DataFormatError("Operation mode not supported.")) - end - - return FACTSControlDevice(; - name = name, - available = Bool(d["available"]), - bus = bus, - control_mode = d["control_mode"], - voltage_setpoint = d["voltage_setpoint"], - max_shunt_current = d["max_shunt_current"], - regulated_bus_number = d["regulated_bus_number"], - ext = get(d, "ext", Dict{String, Any}()), - input_basis = CU, - ) -end - -function read_facts!( - sys::System, - data::Dict, - bus_number_to_bus::Dict{Int, ACBus}; - kwargs..., -) - @info "Reading FACTS data" - if !haskey(data, "facts") - @info "There is no facts data in this file" - return - end - - _get_name = get(kwargs, :bus_name_formatter, _get_pm_dict_name) - - for (d_key, d) in data["facts"] - d["name"] = get(d, "name", d_key) - name = _get_name(d) - bus = bus_number_to_bus[d["bus"]] - full_name = "$(d["bus"])_$(name)" - facts = make_facts(full_name, d, bus) - - add_component!(sys, facts; skip_validation = SKIP_PM_VALIDATION) - end -end - -function read_shunt!( - sys::System, - data::Dict, - bus_number_to_bus::Dict{Int, ACBus}; - kwargs..., -) - @info "Reading shunt data" - if !haskey(data, "shunt") - @info "There is no shunt data in this file" - return - end - - _get_name = get(kwargs, :shunt_name_formatter, _get_pm_dict_name) - - for (d_key, d) in data["shunt"] - d["name"] = get(d, "name", d_key) - name = _get_name(d) - bus = bus_number_to_bus[d["shunt_bus"]] - shunt = make_shunt(name, d, bus) - - add_component!(sys, shunt; skip_validation = SKIP_PM_VALIDATION) - end -end - -function read_storage!( - sys::System, - data::Dict, - bus_number_to_bus::Dict{Int, ACBus}; - kwargs..., -) - @info "Reading storage data" - if !haskey(data, "storage") - @info "There is no storage data in this file" - return - end - - _get_name = get(kwargs, :gen_name_formatter, _get_pm_dict_name) - - for (d_key, d) in data["storage"] - d["name"] = get(d, "name", d_key) - name = _get_name(d) - bus = bus_number_to_bus[d["storage_bus"]] - storage = make_generic_battery(name, d, bus) - - add_component!(sys, storage; skip_validation = SKIP_PM_VALIDATION) - end -end diff --git a/src/parsers/psse_dynamic_data.jl b/src/parsers/psse_dynamic_data.jl index 2292d862..0d74a33f 100644 --- a/src/parsers/psse_dynamic_data.jl +++ b/src/parsers/psse_dynamic_data.jl @@ -1,3 +1,10 @@ +const PSSE_DYR_MAPPING_FILE = + joinpath( + dirname(pathof(PowerSystemCaseBuilder)), + "parsers", + "psse_dynamic_mapping.yaml", + ) + # Additional constants for repeated structs const TGOV1DU = SteamTurbineGov1 diff --git a/src/parsers/psse_metadata_reimport.jl b/src/parsers/psse_metadata_reimport.jl index 1096ffd1..74c2c9cc 100644 --- a/src/parsers/psse_metadata_reimport.jl +++ b/src/parsers/psse_metadata_reimport.jl @@ -122,8 +122,8 @@ function parse_export_metadata_dict(md::AbstractDict) function branch_name_formatter( device_dict::AbstractDict, - bus_f::ACBus, - bus_t::ACBus, + bus_f_name::AbstractString, + bus_t_name::AbstractString, )::String sid = device_dict["source_id"] @@ -133,8 +133,6 @@ function parse_export_metadata_dict(md::AbstractDict) if sid[1] in ["switch", "breaker"] p_name = replace(p_name, r"[@*]" => "_") end - bus_f_name = get_name(bus_f) - bus_t_name = get_name(bus_t) key = ((p_bus_1, p_bus_2), p_name) def_name = "$(bus_f_name)-$(bus_t_name)-i_$(p_name)" new_name = get(all_branch_name_map, key, def_name) @@ -143,13 +141,10 @@ function parse_export_metadata_dict(md::AbstractDict) function xfrm_3w_name_formatter( device_dict::AbstractDict, - p_bus::ACBus, - s_bus::ACBus, - t_bus::ACBus, + bus_primary::AbstractString, + bus_secondary::AbstractString, + bus_tertiary::AbstractString, )::String - bus_primary = get_name(p_bus) - bus_secondary = get_name(s_bus) - bus_tertiary = get_name(t_bus) ckt = device_dict["circuit"] return "$(bus_primary)-$(bus_secondary)-$(bus_tertiary)-i_$(ckt)" @@ -174,9 +169,11 @@ function parse_export_metadata_dict(md::AbstractDict) function make_hvdc_name_formatter(mapping) reversed_mapping = reverse_dict(mapping) - return function (device_dict, bus_f::ACBus, bus_t::ACBus) - bus_f_name = get_name(bus_f) - bus_t_name = get_name(bus_t) + return function ( + device_dict, + bus_f_name::AbstractString, + bus_t_name::AbstractString, + ) name = device_dict["name"] key = string(bus_f_name, "-", bus_t_name, "-i_", name) new_name = get(reversed_mapping, key, key) @@ -218,7 +215,7 @@ function parse_export_metadata_dict(md::AbstractDict) :xfrm_3w_name_formatter => xfrm_3w_name_formatter, :switched_shunt_name_formatter => switched_shunt_name_formatter, :dcline_name_formatter => dcline_name_formatter, - :vscline_name_formatter => vscline_name_formatter, + :vsc_line_name_formatter => vscline_name_formatter, ) bus_number_mapping = reverse_dict(md["bus_number_mapping"]) # PSS/E bus name -> Sienna bus name @@ -226,15 +223,27 @@ function parse_export_metadata_dict(md::AbstractDict) end "Construct a System from a `.raw` file and a dictionary corresponding to the `_export_metadata.json` file" -function PSY.System(file_path::AbstractString, md::AbstractDict; kwargs...) +function PSY.System( + file_path::AbstractString, + md::AbstractDict; + solved_case::Bool = false, + kwargs..., +) sys_kwargs, bus_number_mapping = parse_export_metadata_dict(md) - sys = system_via_power_models(file_path; merge(sys_kwargs, kwargs)...) + sys = system_from_openapi( + PowerFlowFileParser.PowerModelsData(file_path; solved_case = solved_case); + merge(sys_kwargs, kwargs)..., + ) # Remap bus numbers last because everything has been added to the system using PSS/E bus numbers remap_bus_numbers!(sys, bus_number_mapping) return sys end -function system_from_psse_reimport(file_path::AbstractString; kwargs...) +function system_from_psse_reimport( + file_path::AbstractString; + solved_case::Bool = false, + kwargs..., +) md_path = joinpath( dirname(file_path), splitext(basename(file_path))[1] * PSSE_EXPORT_METADATA_EXTENSION, @@ -242,9 +251,12 @@ function system_from_psse_reimport(file_path::AbstractString; kwargs...) if isfile(md_path) @info "Found a PowerFlows.jl PSS/E export metadata file at $md_path, will use it to perform remapping for round trip" md = JSON.parsefile(md_path) - return System(file_path, md; kwargs...) + return System(file_path, md; solved_case = solved_case, kwargs...) else @info "Did not find a PowerFlows.jl PSS/E export metadata file at $md_path, will not do any remapping" - return system_via_power_models(file_path; kwargs...) + return system_from_openapi( + PowerFlowFileParser.PowerModelsData(file_path; solved_case = solved_case); + kwargs..., + ) end end diff --git a/test/test_switched_shunt_parsing.jl b/test/test_switched_shunt_parsing.jl index 71f05a2e..5509f1f8 100644 --- a/test/test_switched_shunt_parsing.jl +++ b/test/test_switched_shunt_parsing.jl @@ -4,14 +4,16 @@ raw = joinpath(PSB.DATA_DIR, "psse_raw", "case25_v35_savnwb.raw") # Names are `-`: 101 '1' is "101-1", 104 '1' is "104-5". shunt_at(sys, name) = get_component(SwitchedAdmittance, sys, name) + build(; kwargs...) = + PSB.system_from_openapi(PSB.PowerFlowFileParser.PowerModelsData(raw; kwargs...)) # raw: 101 '1' MODSW=1, BINIT=-115.00 (discrete: BINIT is only a starting value); # 104 '1' MODSW=2, BINIT=245.63 (continuous: BINIT is always the admittance). - sys = PSB.system_via_power_models(raw) + sys = build() @test isnothing(get_solved_admittance(shunt_at(sys, "101-1"))) @test get_solved_admittance(shunt_at(sys, "104-5")) ≈ 2.4563 - solved = PSB.system_via_power_models(raw; solved_case = true) + solved = build(; solved_case = true) @test get_solved_admittance(shunt_at(solved, "101-1")) ≈ -1.15 @test get_solved_admittance(shunt_at(solved, "104-5")) ≈ 2.4563 end diff --git a/test/test_transformer_parsing.jl b/test/test_transformer_parsing.jl index d69c89c1..72050b2a 100644 --- a/test/test_transformer_parsing.jl +++ b/test/test_transformer_parsing.jl @@ -112,34 +112,34 @@ end end @testset "inverted control band warns and normalizes" begin - # Mirrors frankenstein_70.raw, where VMA1 = 0.984 < VMI1 = 0.985 (a benign - # rounding artifact). The maker must warn (naming the record) and normalize - # the band rather than silently swallowing potentially corrupt data. - d = Dict{String, Any}( - "name" => "synthetic-record", - "COD1" => 0, - "CONT1" => 0, - "RMA1" => 1.5, - "RMI1" => 0.5, - "VMA1" => 0.984, - "VMI1" => 0.985, - "NTP1" => 33, + # frankenstein_70.raw's T1 has VMA1 = 0.984 < VMI1 = 0.985 (a benign rounding + # artifact). The parser must warn (naming the record) and normalize the band rather + # than silently swallowing potentially corrupt data. + raw = joinpath(PSB.DATA_DIR, "psse_raw", "frankenstein_70.raw") + build(path) = PSB.system_from_openapi(PSB.PowerFlowFileParser.PowerModelsData(path)) + t1(sys) = get_circuit( + get_component(TwoWindingTransformer, sys, "FAV SPOT 01-FAV SPOT 04-i_1"), ) - ctrl = - @test_logs (:warn, r"synthetic-record.*inverted controlled-quantity limits") match_mode = - :any PSB._transformer_control_fields(d, 1) - @test ctrl.controlled_quantity_limits == (min = 0.984, max = 0.985) - @test ctrl.control_limits == (min = 0.5, max = 1.5) - - # Inverted RMI/RMA warns too. - d["RMI1"], d["RMA1"] = 1.5, 0.5 - d["VMI1"], d["VMA1"] = 0.9, 1.1 - ctrl2 = - @test_logs (:warn, r"synthetic-record.*inverted control limits") match_mode = :any PSB._transformer_control_fields( - d, - 1, + + sys = + @test_logs (:warn, r"inverted controlled-quantity limits") match_mode = :any build( + raw, ) - @test ctrl2.control_limits == (min = 0.5, max = 1.5) + @test get_controlled_quantity_limits(t1(sys)) == (min = 0.984, max = 0.985) + @test get_control_limits(t1(sys)) == (min = 0.5, max = 1.5) + + # Inverted RMI/RMA warns too: T1's RMA1/RMI1 swapped, VMA1/VMI1 set to a valid band. + inverted = joinpath(mktempdir(), "frankenstein_70.raw") + write( + inverted, + replace( + read(raw, String), + "1.500000,0.500000,0.984000,0.985000" => "0.500000,1.500000,1.100000,0.900000", + ), + ) + sys2 = @test_logs (:warn, r"inverted control limits") match_mode = :any build(inverted) + @test get_control_limits(t1(sys2)) == (min = 0.5, max = 1.5) + @test get_controlled_quantity_limits(t1(sys2)) == (min = 0.9, max = 1.1) end @testset "3W zero-impedance and mag fixtures build" begin