Compose a spec from several files#

This guide shows how to build one specification (spec) out of several files. Use merge to join partial spec files (fragments), such as the files of a component library. Use override to lay files of changes (patches) over a whole spec (a base), such as the spec a framework ships. Use rename to give a fragment other names, so that one file states several components. You need mathspec installed, and Python to call the three functions.

Build a spec from a library of components#

Write the network as a spec. Its constraint Bus_balance sets Bus_injection to zero at each bus, and its objective is to minimise total_cost. The file lists both as names that another file declares (given:), and network.yaml declares no component.

network.yaml

given:
  expressions:
    Bus_injection:
      dims: [snapshot, bus]
      description: what the components put into a bus
    total_cost:
      dims: []
      description: what running the system costs
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
constraints:
  Bus_balance:
    dims: [snapshot, bus]
    expression: Bus_injection == 0
objective:
  sense: minimize
  expression: total_cost

Write each component as a spec against the network. Each component lists under given: each sum it adds to, and adds to each one a named expression with adds_to: (a term):

generator.yaml

given:
  expressions:
    Bus_injection: { dims: [snapshot, bus] }
    total_cost: { dims: [] }
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
  generator: { dtype: str }
relations:
  Generator_bus: { key: generator, values: bus }
parameters:
  Generator_p_nom: { dims: [generator] }
  Generator_marginal_cost: { dims: [generator] }
variables:
  Generator_p: { dims: [snapshot, generator], bounds: { lower: 0, upper: Generator_p_nom } }
expressions:
  Generator_injection:
    expression: sum(Generator_p, over=generator, by=Generator_bus[bus])
    adds_to: Bus_injection
  Generator_cost:
    expression: sum(Generator_p * Generator_marginal_cost)
    adds_to: total_cost

load.yaml

given:
  expressions:
    Bus_injection: { dims: [snapshot, bus] }
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
  load: { dtype: str }
relations:
  Load_bus: { key: load, values: bus }
parameters:
  Load_p_set: { dims: [snapshot, load] }
expressions:
  Load_injection:
    expression: -sum(Load_p_set, over=load, by=Load_bus[bus])
    adds_to: Bus_injection

Name each term after its component, such as Generator_injection, so that no two fragments declare the same name.

Check each file on its own. check prints a remark that is not a refusal (advice) for each name the file lists under given:, and does not refuse the file. On load.yaml, the command and the Python loop print one line:
Command linePython
python -m mathspec check load.yaml
import mathspec as ms for note in ms.advice('load.yaml'): print(note)
```
expression 'Bus_injection' is read here, and this file adds a term to it. merge() sums the term with the terms other files add under that name. Until then, the program reads it and does not build it.
```
mathspec advises that merge completes the sum. On network.yaml, mathspec advises instead that merge takes each name from the file that declares it, or builds it from terms.

Keep this Python session open, because later steps use ms.

Merge the files that you need, as a list:

spec = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml'])
print(spec.expressions['Bus_injection'].expression)
print(spec.expressions['total_cost'].expression)

Generator_injection + Load_injection
Generator_cost

merge builds each sum from its terms.

Add a component to a library#

Add a new file with a term of its own, and leave network.yaml as it is.

As in the steps above, write the component against the network:

store.yaml

given:
  expressions:
    Bus_injection: { dims: [snapshot, bus] }
dimensions:
  snapshot: { dtype: int, ordered: true }
  bus: { dtype: str }
  store: { dtype: str }
relations:
  Store_bus: { key: store, values: bus }
parameters:
  Store_e_nom: { dims: [store] }
variables:
  Store_p: { dims: [snapshot, store] }
  Store_e: { dims: [snapshot, store], bounds: { lower: 0, upper: Store_e_nom } }
constraints:
  Store_energy_balance:
    dims: [snapshot, store]
    expression: Store_e == shift(Store_e, along=snapshot, offset=1, edge='wrap') - Store_p
expressions:
  Store_injection:
    expression: sum(Store_p, over=store, by=Store_bus[bus])
    adds_to: Bus_injection

Add the file to the list:
```
spec = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml', 'store.yaml'])
```
Bus_injection is now Generator_injection + Load_injection + Store_injection.

To add a term to a spec that is already merged, pass that spec as one item of the list:

spec = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml'])
spec = ms.merge([spec, 'store.yaml'])

After this two-step merge, Bus_injection is the same sum as after the list of four files. Add a term with merge and not with a patch.

Use one fragment for several components#

To state two components with the same math, write the fragment once and give it other names with rename.

Write the fragment with names that stand for any component. In unit.yaml, the dimension is unit and the name of each entry starts with Unit_. mathspec loads the file on its own.

unit.yaml

given:
  expressions:
    Bus_injection: { dims: [snapshot, bus] }
    total_cost: { dims: [] }
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
  unit: { dtype: str }
relations:
  Unit_bus: { key: unit, values: bus }
parameters:
  Unit_p_nom: { dims: [unit] }
  Unit_marginal_cost: { dims: [unit] }
variables:
  Unit_p: { dims: [snapshot, unit], bounds: { lower: 0, upper: Unit_p_nom } }
expressions:
  Unit_injection:
    expression: sum(Unit_p, over=unit, by=Unit_bus[bus])
    adds_to: Bus_injection
  Unit_cost:
    expression: sum(Unit_p * Unit_marginal_cost)
    adds_to: total_cost

Write the rule for the new names as a function. rename calls the function once with each name that the fragment declares or lists under given:. Return the new name, or None to keep the name.

def named(prefix, dim):
    def new_name(name):
        if name == 'unit':
            return dim
        if name.startswith('Unit_'):
            return prefix + name.removeprefix('Unit')
        return None

    return new_name

Rename the fragment once for each component, and merge the results:

spec = ms.merge(
    [
        'network.yaml',
        ms.rename('unit.yaml', named('Gas', 'gas')),
        ms.rename('unit.yaml', named('Wind', 'wind')),
        'load.yaml',
    ]
)
print(spec.expressions['Bus_injection'].expression)
print(spec.expressions['total_cost'].expression)
print(sorted(spec.variables))

Gas_injection + Wind_injection + Load_injection
Gas_cost + Wind_cost
['Gas_p', 'Wind_p']

Each renamed fragment adds its own terms to the two sums.

To change only a few names, pass a map from each old name to its new name:

spec = ms.rename('generator.yaml', {'Generator_p_nom': 'Generator_capacity'})

Generator_capacity is then the name of the parameter and the upper bound of Generator_p. rename replaces a name everywhere it occurs in the spec, and keeps each description as written. What a rename changes lists what else rename keeps.

Write part of the sum in the network file#

To put part of the sum in the network file itself, define Bus_injection there, and leave it out of given:. The component files stay as they are.

Write the body of the sum in the network file:

network_slack.yaml

given:
  expressions:
    total_cost:
      dims: []
      description: what running the system costs
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
variables:
  Bus_slack: { dims: [snapshot, bus] }
expressions:
  Bus_injection:
    dims: [snapshot, bus]
    expression: Bus_slack
    description: what the components put into a bus
constraints:
  Bus_balance:
    dims: [snapshot, bus]
    expression: Bus_injection == 0
objective:
  sense: minimize
  expression: total_cost

Merge it with the components:
```
spec = ms.merge(['network_slack.yaml', 'generator.yaml', 'load.yaml'])
```
Bus_injection is Bus_slack + Generator_injection + Load_injection. The body in the network file comes first, and every term follows it.

Lay patches over a base#

Write the base for override as a least-cost dispatch spec that mathspec loads on its own.

base.yaml

dimensions:
  snapshot: { dtype: int }
  generator: { dtype: str }
parameters:
  capacity: { dims: [generator] }
  cost: { dims: [generator] }
  load: { dims: [snapshot] }
variables:
  dispatch: { dims: [snapshot, generator], bounds: { lower: 0, upper: capacity } }
constraints:
  power_balance:
    dims: [snapshot]
    expression: sum(dispatch, over=generator) == load
objective:
  sense: minimize
  expression: sum(dispatch * cost)

Write in each patch only the fields to change. To add an entry, write it whole under a new name:

operate.yaml

variables:
  dispatch: { where: "capacity > 0" }

carbon.yaml

parameters:
  emission_rate: { dims: [generator] }
constraints:
  emission_cap:
    dims: []
    expression: sum(dispatch * emission_rate) <= 1000

Lay the patches on the base, as a list:
```
spec = ms.override('base.yaml', ['carbon.yaml', 'operate.yaml'])
```
spec declares emission_cap beside power_balance. With data attached, dispatch exists only where capacity > 0 (a condition).

Remove an entry or a field with a patch#

Write null under the name of each entry to remove:
feasibility.yaml
```
constraints:
  emission_cap: null
objective: null
```
The spec that ms.override('base.yaml', ['carbon.yaml', 'feasibility.yaml']) returns has power_balance as its only constraint, and no objective.
Write null on a field to give the field its default, and keep the entry. With dispatch: { where: null }, the variable has no condition. With dispatch: { bounds: { upper: null } }, it has no upper bound.

To empty a section, write null on each of its entries, because mathspec refuses constraints: null.

Refine one patch with another#

Here project.yaml is a patch on carbon.yaml from the steps above, with a new expression for emission_cap only:

project.yaml

constraints:
  emission_cap:
    expression: sum(dispatch * emission_rate) <= 500

override replaces the fields of an earlier patch with those of a later one, so put the refining patch after the patch it refines:

spec = ms.override('base.yaml', ['carbon.yaml', 'project.yaml'])
print(spec.constraints['emission_cap'].expression)

sum(dispatch * emission_rate) <= 500

emission_cap has the dims: [] of carbon.yaml and the limit of project.yaml.

Remove a term from a merged sum#

To drop a term, write the body of the sum again without it, in a patch, and lay the patch after the last merge:

no_load.yaml

expressions:
  Bus_injection: Generator_injection

spec = ms.override(ms.merge(['network.yaml', 'generator.yaml', 'load.yaml']), ['no_load.yaml'])

Bus_injection is then Generator_injection alone, and Load_injection stays in the spec as a named expression that no sum includes.