Parsing Data

PowerSystems.jl supports the creation of a System from a variety of common data formats:

• MATPOWER (code copied with permission from PowerModels.jl)
• PSS/e RAW Files (code copied with permission from PowerModels.jl)
• PSS/e DYR Files
• PowerSystems table data (CSV Files)

MATPOWER / PSS/e

The following code will create a System from a MATPOWER or PSS/e file:

julia> using PowerSystems
julia> file_dir = joinpath(pkgdir(PowerSystems), "docs", "src", "tutorials", "tutorials_data")"/home/runner/work/PowerSystems.jl/PowerSystems.jl/docs/src/tutorials/tutorials_data"
julia> sys = System(joinpath(file_dir, "case5.m"))[ Info: Correcting vm in bus 1 to 1.07762 to match generator set-point
[ Info: Correcting vm in bus 3 to 1.1 to match generator set-point
[ Info: Correcting vm in bus 4 to 1.06414 to match generator set-point
[ Info: Correcting vm in bus 10 to 1.06907 to match generator set-point
[ Info: extending matpower format with data: areas 1x3
[ Info: reversing the orientation of branch 6 (4, 3) to be consistent with other parallel branches
[ Info: removing 1 cost terms from generator 4: [4000.0, 0.0]
[ Info: removing 1 cost terms from generator 1: [1400.0, 0.0]
[ Info: removing 1 cost terms from generator 5: [1000.0, 0.0]
[ Info: removing 1 cost terms from generator 2: [1500.0, 0.0]
[ Info: removing 1 cost terms from generator 3: [3000.0, 0.0]
┌ Info: Constructing System from Power Models
│   data["name"] = "nesta_case5_pjm"
└   data["source_type"] = "matpower"
[ Info: Reading bus data
[ Info: Reading Load data in PowerModels dict to populate System ...
[ Info: Reading LoadZones data in PowerModels dict to populate System ...
[ Info: Reading generator data
[ Info: Reading branch data
[ Info: Reading branch data
[ Info: Reading DC Line data
[ Info: Reading storage data
System
┌───────────────────┬─────────────┐
│ Property          │ Value       │
├───────────────────┼─────────────┤
│ Name              │             │
│ Description       │             │
│ System Units Base │ SYSTEM_BASE │
│ Base Power        │ 100.0       │
│ Base Frequency    │ 60.0        │
│ Num Components    │ 28          │
└───────────────────┴─────────────┘

Static Components
┌──────────────────────────┬───────┬────────────────────────┬───────────────┐
│ Type                     │ Count │ Has Static Time Series │ Has Forecasts │
├──────────────────────────┼───────┼────────────────────────┼───────────────┤
│ ACBus                    │ 5     │ false                  │ false         │
│ Arc                      │ 6     │ false                  │ false         │
│ Area                     │ 1     │ false                  │ false         │
│ Line                     │ 5     │ false                  │ false         │
│ LoadZone                 │ 1     │ false                  │ false         │
│ PhaseShiftingTransformer │ 2     │ false                  │ false         │
│ PowerLoad                │ 3     │ false                  │ false         │
│ ThermalStandard          │ 5     │ false                  │ false         │
└──────────────────────────┴───────┴────────────────────────┴───────────────┘

PSS/e dynamic data parsing

PSS/e's dynamic model library is extensive, we currently support parsing a limited amount of models out of the box.

Creating a Dynamic System using .RAW and .DYR data

A PowerSystems.jl system can be created using a .RAW and a .DYR file. In this example we will create the following three bus system using the following RAW file:

0, 100, 33, 0, 0, 60  / 24-Apr-2020 19:28:39 - MATPOWER 7.0.1-dev


     101, 'BUS 1       ',       138, 3,    1,    1, 1,           1.02,        0,  1.1,  0.9,  1.1,  0.9
     102, 'BUS 2       ',       138, 2,    1,    1, 1,           1.0142,           0,  1.1,  0.9,  1.1,  0.9
     103, 'BUS 3       ',       138, 2,    1,    1, 1,           1.0059,           0,  1.1,  0.9,  1.1,  0.9
0 / END OF BUS DATA, BEGIN LOAD DATA
     101,  1, 1,    1,    1,       100,       20, 0, 0, 0, 0, 1, 1, 0
     102,  1, 1,    1,    1,       70,       10, 0, 0, 0, 0, 1, 1, 0
     103,  1, 1,    1,    1,       50,       10, 0, 0, 0, 0, 1, 1, 0
0 / END OF LOAD DATA, BEGIN FIXED SHUNT DATA
0 / END OF FIXED SHUNT DATA, BEGIN GENERATOR DATA
     101,  1,       20,         0,       100,      -100,    1.02, 0,     100, 0, 0, 0, 0, 1, 1, 100,       318,         0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1
     102,  1,       100,         0,       100,      -100,   1.0142, 0,     100, 0, 0.7, 0, 0, 1, 1, 100,       318,         0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1
     103,  1,       100,         0,       100,      -100,   1.0059, 0,     100, 0, 0.2, 0, 0, 1, 1, 100,       318,         0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1
0 / END OF GENERATOR DATA, BEGIN BRANCH DATA
     101,      103, 1,  0.01000,     0.12,      0.0,     250,     250,     250, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1
     101,      102, 1,  0.01000,     0.12,      0.0,     250,     250,     250, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1
     102,      103, 1,  0.01000,     0.12,      0.0,     250,     250,     250, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1
0 / END OF BRANCH DATA, BEGIN TRANSFORMER DATA
0 / END OF TRANSFORMER DATA, BEGIN AREA DATA
0 / END OF AREA DATA, BEGIN TWO-TERMINAL DC DATA
0 / END OF TWO-TERMINAL DC DATA, BEGIN VOLTAGE SOURCE CONVERTER DATA
0 / END OF VOLTAGE SOURCE CONVERTER DATA, BEGIN IMPEDANCE CORRECTION DATA
0 / END OF IMPEDANCE CORRECTION DATA, BEGIN MULTI-TERMINAL DC DATA
0 / END OF MULTI-TERMINAL DC DATA, BEGIN MULTI-SECTION LINE DATA
0 / END OF MULTI-SECTION LINE DATA, BEGIN ZONE DATA
0 / END OF ZONE DATA, BEGIN INTER-AREA TRANSFER DATA
0 / END OF INTER-AREA TRANSFER DATA, BEGIN OWNER DATA
0 / END OF OWNER DATA, BEGIN FACTS CONTROL DEVICE DATA
0 / END OF FACTS CONTROL DEVICE DATA, BEGIN SWITCHED SHUNT DATA
0 / END OF SWITCHED SHUNT DATA, BEGIN GNE DEVICE DATA
0 / END OF GNE DEVICE DATA, BEGIN INDUCTION MACHINE DATA
0 / END OF INDUCTION MACHINE DATA
Q

This system is a three bus system with three generators, three loads and three branches. The dynamic data for the generators is provided in the DYR file:

  101 'GENROE' 1   8.000000  0.030000  0.400000  0.050000  6.500000  0.000000  1.800000
  1.700000  0.300000  0.550000  0.250000  0.200000  0.039200  0.267200  /
  101 'ESST1A' 1   1  1  0.01  99  -99  1  10  1  1  200  0  4  -4  4  -4  0  0  1  0  3  /
  102 'GENCLS' 1   0.0   0.0 /
  103 'GENCLS' 1   3.1   2.0 /

That assigns a GENROU generator and a ESST1A voltage regulator at the generator located at bus 101, while classic machine models for the generators located at bus 102 and 103.

To create the system we can do it passing both files directories:

julia> RAW_dir = joinpath(file_dir, "ThreeBusNetwork.raw")"/home/runner/work/PowerSystems.jl/PowerSystems.jl/docs/src/tutorials/tutorials_data/ThreeBusNetwork.raw"
julia> DYR_dir = joinpath(file_dir, "TestGENCLS.dyr")"/home/runner/work/PowerSystems.jl/PowerSystems.jl/docs/src/tutorials/tutorials_data/TestGENCLS.dyr"
julia> dyn_system = System(RAW_dir, DYR_dir, runchecks = false)[ Info: The PSS(R)E parser currently supports buses, loads, shunts, generators, branches, transformers, and dc lines
[ Info: The PSS(R)E parser currently supports buses, loads, shunts, generators, branches, transformers, and dc lines
[ Info: Parsing PSS(R)E Bus data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Load data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Shunt data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Generator data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Branch data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Transformer data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Two-Terminal and VSC DC line data into a PowerModels Dict...
┌ Warning: This PSS(R)E parser currently doesn't support Storage data parsing...
└ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/pm_io/psse.jl:985
┌ Warning: This PSS(R)E parser currently doesn't support Switch data parsing...
└ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/pm_io/psse.jl:991
[ Info: angmin and angmax values are 0, widening these values on branch 1 to +/- 60.0 deg.
[ Info: angmin and angmax values are 0, widening these values on branch 2 to +/- 60.0 deg.
[ Info: angmin and angmax values are 0, widening these values on branch 3 to +/- 60.0 deg.
┌ Info: Constructing System from Power Models
│   data["name"] = "threebusnetwork"
└   data["source_type"] = "pti"
[ Info: Reading bus data
[ Info: Reading Load data in PowerModels dict to populate System ...
[ Info: Reading LoadZones data in PowerModels dict to populate System ...
[ Info: Reading generator data
[ Info: Reading branch data
[ Info: Reading branch data
[ Info: Reading DC Line data
[ Info: Reading storage data
[ Info: Generators provided in .dyr, without a generator in .raw file will be skipped.
[ Info: Machine at bus 102, id 1 has zero inertia. Modeling it as Voltage Source
System
┌───────────────────┬─────────────┐
│ Property          │ Value       │
├───────────────────┼─────────────┤
│ Name              │             │
│ Description       │             │
│ System Units Base │ SYSTEM_BASE │
│ Base Power        │ 100.0       │
│ Base Frequency    │ 60.0        │
│ Num Components    │ 19          │
└───────────────────┴─────────────┘

Static Components
┌─────────────────┬───────┬────────────────────────┬───────────────┐
│ Type            │ Count │ Has Static Time Series │ Has Forecasts │
├─────────────────┼───────┼────────────────────────┼───────────────┤
│ ACBus           │ 3     │ false                  │ false         │
│ Arc             │ 3     │ false                  │ false         │
│ Area            │ 1     │ false                  │ false         │
│ Line            │ 3     │ false                  │ false         │
│ LoadZone        │ 1     │ false                  │ false         │
│ Source          │ 1     │ false                  │ false         │
│ StandardLoad    │ 3     │ false                  │ false         │
│ ThermalStandard │ 2     │ false                  │ false         │
└─────────────────┴───────┴────────────────────────┴───────────────┘

Dynamic Components
┌──────────────────┬───────┐
│ Type             │ Count │
├──────────────────┼───────┤
│ DynamicGenerator │ 1     │
│ DynamicGenerator │ 1     │
└──────────────────┴───────┘

Common Issues

Please note that while PSS/e does not enforce unique bus names, PowerSystems.jl does. To reparse bus names to comply with this requirement the bus_name_formatter *kwarg can be used in System() as shown in the example below:

julia> dyn_system = System(RAW_dir, DYR_dir; bus_name_formatter = x -> strip(string(x["name"])) * "-" * string(x["index"]))[ Info: The PSS(R)E parser currently supports buses, loads, shunts, generators, branches, transformers, and dc lines
[ Info: The PSS(R)E parser currently supports buses, loads, shunts, generators, branches, transformers, and dc lines
[ Info: Parsing PSS(R)E Bus data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Load data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Shunt data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Generator data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Branch data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Transformer data into a PowerModels Dict...
[ Info: Parsing PSS(R)E Two-Terminal and VSC DC line data into a PowerModels Dict...
┌ Warning: This PSS(R)E parser currently doesn't support Storage data parsing...
└ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/pm_io/psse.jl:985
┌ Warning: This PSS(R)E parser currently doesn't support Switch data parsing...
└ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/pm_io/psse.jl:991
[ Info: angmin and angmax values are 0, widening these values on branch 1 to +/- 60.0 deg.
[ Info: angmin and angmax values are 0, widening these values on branch 2 to +/- 60.0 deg.
[ Info: angmin and angmax values are 0, widening these values on branch 3 to +/- 60.0 deg.
┌ Info: Constructing System from Power Models
│   data["name"] = "threebusnetwork"
└   data["source_type"] = "pti"
[ Info: Reading bus data
[ Info: Reading Load data in PowerModels dict to populate System ...
[ Info: Reading LoadZones data in PowerModels dict to populate System ...
[ Info: Reading generator data
[ Info: Reading branch data
[ Info: Reading branch data
[ Info: Reading DC Line data
[ Info: Reading storage data
[ Info: Generators provided in .dyr, without a generator in .raw file will be skipped.
[ Info: Machine at bus 102, id 1 has zero inertia. Modeling it as Voltage Source
┌ Warning: struct DynamicGenerator does not exist in validation configuration file, validation skipped
└ @ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/9btGb/src/validation.jl:51
┌ Warning: struct DynamicGenerator does not exist in validation configuration file, validation skipped
└ @ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/9btGb/src/validation.jl:51
System
┌───────────────────┬─────────────┐
│ Property          │ Value       │
├───────────────────┼─────────────┤
│ Name              │             │
│ Description       │             │
│ System Units Base │ SYSTEM_BASE │
│ Base Power        │ 100.0       │
│ Base Frequency    │ 60.0        │
│ Num Components    │ 19          │
└───────────────────┴─────────────┘

Static Components
┌─────────────────┬───────┬────────────────────────┬───────────────┐
│ Type            │ Count │ Has Static Time Series │ Has Forecasts │
├─────────────────┼───────┼────────────────────────┼───────────────┤
│ ACBus           │ 3     │ false                  │ false         │
│ Arc             │ 3     │ false                  │ false         │
│ Area            │ 1     │ false                  │ false         │
│ Line            │ 3     │ false                  │ false         │
│ LoadZone        │ 1     │ false                  │ false         │
│ Source          │ 1     │ false                  │ false         │
│ StandardLoad    │ 3     │ false                  │ false         │
│ ThermalStandard │ 2     │ false                  │ false         │
└─────────────────┴───────┴────────────────────────┴───────────────┘

Dynamic Components
┌──────────────────┬───────┐
│ Type             │ Count │
├──────────────────┼───────┤
│ DynamicGenerator │ 1     │
│ DynamicGenerator │ 1     │
└──────────────────┴───────┘

In this example the anonymous function x -> strip(string(x["name"])) * "-" * string(x["index"]) takes the bus name and index from PSSe and concatenates them to produce the name.

PowerSystems Table Data

This is a custom format that allows users to define power system component data by category and column with custom names, types, and units.

Categories

Components for each category must be defined in their own CSV file. The following categories are currently supported:

• branch.csv
• bus.csv (required)
  • columns specifying area and zone will create a corresponding set of Area and LoadZone objects.
  • columns specifying max_active_power or max_reactive_power will create PowerLoad objects when nonzero values are encountered and will contribute to the peak_active_power and peak_reactive_power values for the
  corresponding LoadZone object.
• dc_branch.csv
• gen.csv
• load.csv
• reserves.csv
• storage.csv

These must reside in the directory passed when constructing PowerSystemTableData.

Adding Time Series Data

PowerSystems requires a metadata file that maps components to their time series data in order to be able to automatically construct time_series from raw data files. The following fields are required for each time array:

• simulation: User description of simulation
• resolution: Resolution of time series in seconds
• module: Module that defines the abstract type of the component
• category: Type of component. Must map to abstract types defined by the "module" entry (Bus, ElectricLoad, Generator, LoadZone, Reserve)
• component_name: Name of component
• name: User-defined name for the time series data.
• normalization_factor: Controls normalization of the data. Use 1.0 for pre-normalized data. Use 'Max' to divide the time series by the max value in the column. Use any float for a custom scaling factor.
• scaling_factor_multiplier_module: Module that defines the accessor function for the

scaling factor

• scaling_factor_multiplier: Accessor function of the scaling factor
• data_file: Path to the time series data file

Notes:

• The "module", "category", and "component_name" entries must be valid arguments to retrieve

a component using get_component(${module}.${category}, sys, $name).

• The "scalingfactormultipliermodule" and the "scalingfactor_multiplier" entries must

be sufficient to return the scaling factor data using ${scaling_factor_multiplier_module}.${scaling_factor_multiplier}(component).

PowerSystems supports this metadata in either CSV or JSON formats. Refer to RTS_GMLC for an example.

Performance considerations

By default PowerSystems stores time series data in HDF5 files. It does not keep all of the data in memory. This means that every time you access a timeseries PowerSystems will have to read the data from storage, which will add latency. If you know ahead of time that all of your data will fit in memory then you can change this behavior by passing `timeseriesinmemory = true` when you create the System.

If the time series data is stored in HDF5 then PowerSystems will use the tmp filesystem by default. You can change this by passing time_series_directory = X when you create the System. This is required if the time series data is larger than the amount of tmp space available. You can also override the location by setting the environment variable SIIPTIMESERIES_DIRECTORY to another directory.

Customization

The tabular data parser in PowerSystems.jl can be customized to read a variety of datasets by configuring:

• which type of generator (<:Generator) to create based on the fuel and prime mover specifications
• property names, units, and per units conversions](@ref csvperunit) in *.csv files

Here is an example of how to construct a System with all customizations listed in this section:

data_dir = "/data/my-data-dir"
base_power = 100.0
descriptors = "./user_descriptors.yaml"
timeseries_metadata_file = "./timeseries_pointers.json"
generator_mapping_file = "./generator_mapping.yaml"
data = PowerSystemTableData(
    data_dir,
    base_power,
    descriptors;
    timeseries_metadata_file = timeseries_metadata_file,
    generator_mapping_file = generator_mapping_file,
)
sys = System(data, time_series_in_memory = true)

Examples configuration files can be found in the RTS-GMLC repo:

• user_descriptors.yaml
• generator_mapping.yaml

CSV Data Configurations

Custom construction of generators

PowerSystems supports custom construction of subtypes of the abstract type Generator based on fuel and type. The parsing code detects these fields in the raw data and then constructs the concrete type listed in the passed generator mapping file. The default file is src/parsers/generator_mapping.yaml. You can override this behavior by specifying your own file when constructing PowerSystemTableData.

Column names

PowerSystems provides am input mapping capability that allows you to keep your own column names.

For example, when parsing raw data for a generator the code expects a column called name. If the raw data instead defines that column as GEN UID then you can change the custom_name field under the generator category to GEN UID in your YAML file.

To enable the parsing of a custom set of csv files, you can generate a configuration file (such as user_descriptors.yaml) from the defaults, which are stored in src/descriptors/power_system_inputs.json.

python ./bin/generate_config_file.py ./user_descriptors.yaml

Next, edit this file with your customizations.

Note that the user-specific customizations are stored in YAML rather than JSON to allow for easier editing. The next few sections describe changes you can make to this YAML file. Do not edit the default JSON file.

Per-unit conversion

For more info on the per-unit conventions in PowerSystems.jl, refer to the per-unit section of the system documentation.

PowerSystems defines whether it expects a column value to be per-unit system base, per-unit device base, or natural units in power_system_inputs.json. If it expects a per-unit convention that differs from your values then you can set the unit_system in user_descriptors.yaml and PowerSystems will automatically convert the values. For example, if you have a max_active_power value stored in natural units (MW), but power_system_inputs.json specifies unit_system: device_base, you can enter unit_system: natural_units in user_descriptors.yaml and PowerSystems will divide the value by the value of the corresponding entry in the column identified by the base_reference field in power_system_inputs.json. You can also override the base_reference setting by adding base_reference: My Column to make device base per-unit conversion by dividing the value by the entry in My Column. System base per-unit conversions always divide the value by the system base_power value instantiated when constructing a System.

Unit conversion

PowerSystems provides a limited set of unit conversions. For example, if power_system_inputs.json indicates that a value's unit is degrees but your values are in radians then you can set unit: radian in your YAML file. Other valid unit entries include GW, GWh, MW, MWh, kW, and kWh.