Composing specs#

mathspec builds one specification of an optimisation model (a spec) from several files with two functions, merge and override. A third function, rename, gives one spec other names. This page describes what each function does with its files, what it carries into the result, and every refusal it can give.

merge joins whole specs as peers (fragments), such as the files of a component library. override lays files that state only a change (patches) over a spec that another party extends (a base). A base is often the spec that a framework ships.

import mathspec as ms

library = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml'])
project = ms.override('base.yaml', ['carbon.yaml'])

merge joins three fragments into library, and override lays the patch carbon.yaml over base.yaml to give project.

merge and override take their files as a list, and raise TypeError for one path in place of a list. An item of the list is a path to a YAML file, YAML text, a mapping or a loaded Spec. So the result of one call can be an item of the next, as in override(merge([…]), […]).

In a refusal, mathspec names a file by its path as it is written in the list. It names a mapping or a loaded spec by its place in the list, such as '#2'.

`merge`#

merge(fragments, description=None) loads each fragment with to_spec, joins their entries into one spec, and returns that spec loaded. merge refuses a name that two fragments declare, and renames nothing. The description= argument sets the description of the merged spec.

The three files below form a small library, and each name in it has a prefix of its own, such as Generator_ or Load_. network.yaml states the balance at each bus. It lists Bus_injection, what the components put into each bus, under the section for names another file declares (given:). Each component file adds its part as a named expression that adds to a sum (a term).

network.yaml

given:
  expressions:
    Bus_injection:
      dims: [snapshot, bus]
      description: what the components put into a bus
    total_cost:
      dims: []
      description: what running the system costs
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
constraints:
  Bus_balance:
    dims: [snapshot, bus]
    expression: Bus_injection == 0
objective:
  sense: minimize
  expression: total_cost

generator.yaml

given:
  expressions:
    Bus_injection: { dims: [snapshot, bus] }
    total_cost: { dims: [] }
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
  generator: { dtype: str }
relations:
  Generator_bus: { key: generator, values: bus }
parameters:
  Generator_p_nom: { dims: [generator] }
  Generator_marginal_cost: { dims: [generator] }
variables:
  Generator_p: { dims: [snapshot, generator], bounds: { lower: 0, upper: Generator_p_nom } }
expressions:
  Generator_injection:
    expression: sum(Generator_p, over=generator, by=Generator_bus[bus])
    adds_to: Bus_injection
  Generator_cost:
    expression: sum(Generator_p * Generator_marginal_cost)
    adds_to: total_cost

load.yaml

given:
  expressions:
    Bus_injection: { dims: [snapshot, bus] }
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
  load: { dtype: str }
relations:
  Load_bus: { key: load, values: bus }
parameters:
  Load_p_set: { dims: [snapshot, load] }
expressions:
  Load_injection:
    expression: -sum(Load_p_set, over=load, by=Load_bus[bus])
    adds_to: Bus_injection

Each file loads and prints on its own. generator.yaml sets no objective, so alone the file states a problem with nothing to minimise (a feasibility problem). merge joins the three:

spec = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml'])
print(spec.to_yaml())

version: 0
dimensions:
  bus: {dtype: str}
  generator: {dtype: str}
  load: {dtype: str}
  snapshot: {dtype: int}
relations:
  Generator_bus: {key: generator, values: bus}
  Load_bus: {key: load, values: bus}
parameters:
  Generator_marginal_cost: {dims: [generator]}
  Generator_p_nom: {dims: [generator]}
  Load_p_set: {dims: [load, snapshot]}
variables:
  Generator_p:
    dims: [generator, snapshot]
    bounds: {lower: 0.0, upper: Generator_p_nom}
expressions:
  Bus_injection:
    dims: [bus, snapshot]
    expression: Generator_injection + Load_injection
    description: what the components put into a bus
  Generator_cost: sum(Generator_p * Generator_marginal_cost)
  Generator_injection: sum(Generator_p, over=generator, by=Generator_bus[bus])
  Load_injection: -sum(Load_p_set, over=load, by=Load_bus[bus])
  total_cost:
    dims: []
    expression: Generator_cost
    description: what running the system costs
objective: {sense: minimize, expression: total_cost}
constraints:
  Bus_balance:
    dims: [bus, snapshot]
    expression: Bus_injection == 0

The merged spec has no given: section, because merge builds each sum that a fragment lists under given: from its terms. Bus_injection is the sum of the two terms, and each term stays a named expression without its adds_to:. The objective is the one that network.yaml sets.

Each kind of entry#

Two fragments may share a dimension, a relation, a given entry or a sum.

The entry	What `merge` does
a dimension or a relation	accepts it from every fragment that declares it, where every fragment declares it the same way
a `description` on a shared dimension or relation	keeps the first wording in the list
`ordered: true` on a shared dimension	makes the dimension ordered if one fragment sets it
`missing:` on a shared relation	requires the same value from every fragment that declares the relation, where `missing: refused` and no `missing:` are one value
an entry under `given:`	checks it against the fragment that declares the name, then folds it in (given entries)
an expression with `adds_to:`	adds it as a term to the sum in its `adds_to:` (terms and sums)
`objective`	takes it from the one fragment that sets it, and refuses a second
`version`	requires the same value from every fragment
`description` at the top of a fragment	drops it
any other entry	takes it from the one fragment that declares it, and refuses a second

Terms and sums#

merge builds each sum from the terms that fragments add to it. The merged body is the body that one fragment defines, if any, followed by every term by its name, in list order. The list order changes only the order in which the terms are written. So the layout that equal specs share (the canonical form) is the same for every order:

a = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml'])
b = ms.merge(['load.yaml', 'generator.yaml', 'network.yaml'])
a.expressions['Bus_injection'].expression
# 'Generator_injection + Load_injection'
b.expressions['Bus_injection'].expression
# 'Load_injection + Generator_injection'
a.to_yaml(canonical=True) == b.to_yaml(canonical=True)
# True

The two bodies list the terms in different orders, and the canonical forms are equal.

Two terms of one sum never collide, because each term is a named expression with a name of its own, such as Generator_injection and Load_injection.

A fragment may define the sum itself instead of listing it under given:. merge then keeps the dims: and the description of that fragment. Here network_slack.yaml defines Bus_injection as the variable Bus_slack:

network_slack.yaml

given:
  expressions:
    total_cost:
      dims: []
      description: what running the system costs
dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
variables:
  Bus_slack: { dims: [snapshot, bus] }
expressions:
  Bus_injection:
    dims: [snapshot, bus]
    expression: Bus_slack
    description: what the components put into a bus
constraints:
  Bus_balance:
    dims: [snapshot, bus]
    expression: Bus_injection == 0
objective:
  sense: minimize
  expression: total_cost

spec = ms.merge(['network_slack.yaml', 'generator.yaml', 'load.yaml'])
spec.expressions['Bus_injection'].expression
# 'Bus_slack + Generator_injection + Load_injection'

Bus_slack comes first, then the terms in list order.

A merged spec defines every sum it has terms for, so a later merge adds further terms to the body of that sum. Here store.yaml is the store fragment of compose a spec from several files:

two_steps = ms.merge([ms.merge(['network.yaml', 'generator.yaml', 'load.yaml']), 'store.yaml'])
one_step = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml', 'store.yaml'])
two_steps.expressions['Bus_injection'].expression
# 'Generator_injection + Load_injection + Store_injection'
one_step.expressions['Bus_injection'].expression
# 'Generator_injection + Load_injection + Store_injection'

merge gives Bus_injection the same three terms in the same order either way.

Given entries#

A given entry is a name that a file lists under given:. merge folds each given entry into the entry of the fragment that declares the name. The fragment that lists the name may state less than the file that declares it, such as the dimensions of a variable with no domain.

The given entry	What `merge` does
any kind, declared by a fragment	checks it against the declaring fragment, and gives the merged entry the description of the given entry, where the declaration has none
any kind, listed by two fragments	requires both to list it the same way, as for a dimension
a given expression	checks the body that the other fragment declares by the rules of given expressions
a given condition	checks the condition that the other fragment declares by the rules of given conditions
any kind, declared by no fragment	keeps it under `given:`

Here price.yaml lists co2_price under given:, and no other fragment declares it:

price.yaml

given:
  parameters:
    co2_price: { dims: [] }
  variables:
    Generator_p: { dims: [snapshot, generator] }
  expressions:
    total_cost: { dims: [] }
dimensions:
  snapshot: { dtype: int }
  generator: { dtype: str }
parameters:
  Generator_co2: { dims: [generator] }
expressions:
  co2_cost:
    expression: co2_price * sum(Generator_p * Generator_co2)
    adds_to: total_cost

spec = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml', 'price.yaml'])
print(spec.to_yaml())

version: 0
given:
  parameters:
    co2_price: {dims: []}
…

co2_price stays under given:, and merge builds total_cost from its terms Generator_cost and co2_cost:

spec.expressions['total_cost'].expression
# 'Generator_cost + co2_cost'

`override`#

override(base, patches) loads the base with to_spec and lays each patch over the result of the ones before it, in list order. override returns the patched spec loaded. A patch is not a spec, so override does not load a patch alone.

A patch lists only the fields it changes. An entry or a field that the patch leaves out stays as the base declares it. Here operate.yaml adds a where: condition to one variable, and carbon.yaml adds a parameter and a constraint:

base.yaml

dimensions:
  snapshot: { dtype: int }
  generator: { dtype: str }
parameters:
  capacity: { dims: [generator] }
  cost: { dims: [generator] }
  load: { dims: [snapshot] }
variables:
  dispatch: { dims: [snapshot, generator], bounds: { lower: 0, upper: capacity } }
constraints:
  power_balance:
    dims: [snapshot]
    expression: sum(dispatch, over=generator) == load
objective:
  sense: minimize
  expression: sum(dispatch * cost)

operate.yaml

variables:
  dispatch: { where: "capacity > 0" }

carbon.yaml

parameters:
  emission_rate: { dims: [generator] }
constraints:
  emission_cap:
    dims: []
    expression: sum(dispatch * emission_rate) <= 1000

spec = ms.override('base.yaml', ['carbon.yaml', 'operate.yaml'])
print(spec.to_yaml())

version: 0
dimensions:
  generator: {dtype: str}
  snapshot: {dtype: int}
parameters:
  capacity: {dims: [generator]}
  cost: {dims: [generator]}
  emission_rate: {dims: [generator]}
  load: {dims: [snapshot]}
variables:
  dispatch:
    dims: [generator, snapshot]
    where: capacity > 0
    bounds: {lower: 0.0, upper: capacity}
objective:
  sense: minimize
  expression: sum(dispatch * cost)
constraints:
  emission_cap:
    dims: []
    expression: sum(dispatch * emission_rate) <= 1000
  power_balance:
    dims: [snapshot]
    expression: sum(dispatch, over=generator) == load

override keeps the dimensions and bounds of dispatch, and adds the where: that operate.yaml writes. With capacity = {wind: 10, gas: 5, old: 0}, the patched dispatch has no row at old.

`null` in a patch#

On an entry, override removes the entry. For constraints: {emission_cap: null}, override removes that constraint.
On a field, override restores the default of that field and keeps the entry. For dispatch: {where: null}, override gives the variable no
On the section objective, override removes the objective.
On any other section, override refuses null.

What a patch writes#

The patch writes	What `override` does
some fields of an entry	changes those fields, and keeps the rest of the entry
a whole entry under a new name	adds it
a named expression as a bare string	replaces only its body, `expression:` or `cases:`, and keeps its `dims:`, description and `adds_to:`, so the new body may have no dimension outside those `dims:`
`null` on an entry, a field or a section	applies the rule in `null` in a patch
a dimension or a relation	adds it, or accepts it restated as the base declares it, field by field, where a field left out equals its default
`description:` on a dimension or a relation of the base	replaces the description of the base, alone or with other fields, and `null` removes it
`ordered:` on a restated dimension	makes the dimension ordered for `true`, and refuses `false` over an ordered dimension
`missing:` on a relation, alone or with other fields	changes whether every key combination needs a row, as on a parameter
an entry under one kind of `given:`	edits, adds or removes it like any entry, and keeps the other kinds
`version`, `description`	replaces the value of the base
a field that an earlier patch writes	replaces that value
a term on a base that lists the sum under `given:`	keeps the sum under `given:` and keeps the term with its `adds_to:`, and builds no body
a term on a base that defines the sum	refuses the patch (a term on a merged sum)
the body of a merged sum without one term	replaces the body, and keeps the left-out term as a named expression that no sum uses (remove a term from a merged sum)
`adds_to: null` on a term	keeps the entry as a named expression that adds to nothing

`rename`#

rename(source, names) loads a spec with to_spec, replaces the names that names changes, and returns the renamed spec loaded. Only names change, and every entry and expression has the same meaning as before. Because merge refuses two fragments that declare the same name, you use rename to state several components with one fragment in one merge.

source is one item of the kinds that merge takes: a path, YAML text, a mapping or a loaded Spec. Here the fragment states one kind of unit:

unit.yaml

dimensions:
  snapshot: { dtype: int }
  unit: { dtype: str }
parameters:
  Unit_p_nom: { dims: [unit], description: the most that Unit_p reaches }
  Unit_marginal_cost: { dims: [unit] }
variables:
  Unit_p: { dims: [snapshot, unit], bounds: { lower: 0, upper: Unit_p_nom } }
objective:
  sense: minimize
  expression: sum(Unit_p * Unit_marginal_cost, over=[snapshot, unit])

names = {
    'unit': 'generator',
    'Unit_p_nom': 'Generator_p_nom',
    'Unit_marginal_cost': 'Generator_marginal_cost',
    'Unit_p': 'Generator_p',
}
print(ms.rename('unit.yaml', names).to_yaml())

version: 0
dimensions:
  generator: {dtype: str}
  snapshot: {dtype: int}
parameters:
  Generator_marginal_cost: {dims: [generator]}
  Generator_p_nom:
    dims: [generator]
    description: the most that Unit_p reaches
variables:
  Generator_p:
    dims: [generator, snapshot]
    bounds: {lower: 0.0, upper: Generator_p_nom}
objective:
  sense: minimize
  expression: sum(Generator_p * Generator_marginal_cost, over=[snapshot, generator])

Every key, dims: list, bound and expression has the new names, and the description of Generator_p_nom still has the old name Unit_p.

The `names` argument#

In a map, each key is a name and its value is the new name. rename replaces all names at the same time, so two names can swap. An empty map returns source loaded.
A function sets the new names by a rule. rename calls it once with each name that source declares or lists under given:.

In both forms, a new name of None, or of the same name, keeps that name.

A second call with the inverse map returns a spec with the canonical form of source.

What a rename changes#

The spec has	What `rename` does
a name as the key of an entry, in `dims:`, on the sides of a relation, in `adds_to:` or in `given:`	replaces the name
a name in an expression, `where:`, bound, case or assumption	replaces the name, and keeps the spaces and line breaks around it
a relation column named after its dimension, such as `bus` in `Unit_bus[bus]`	replaces the column with the dimension
a relation column named after something else, such as `bus0` in `{bus0: bus}`	keeps the column, and refuses a map that changes a name to or from it
a longer name that contains a renamed name, such as `Unit_p_nom` for `Unit_p`	keeps the longer name
a keyword of an operator, such as `over=`	keeps the keyword
a quoted label, a label under `one_of:`, or a number such as `1e6`	keeps the label or the number
the name of a case or of a piecewise link	keeps the name, because it is a label of rows
an argument of a macro, and the template text that uses it	keeps the argument
a fixed value of a field, such as `domain: integer`	keeps the value
a `description`	keeps the text as written

Refusals#

In each message, mathspec names the entry and suggests the change that removes the refusal, and in most it also names the file.

A name two fragments declare#

Here gas.yaml and coal.yaml are two copies of a generator file, and both declare Generator_p_nom:

fragments 'gas.yaml' and 'coal.yaml' both declare the parameter 'Generator_p_nom'. If they are two rows of a dimension, merge the fragment once and put both rows in the data. Otherwise, rename one of them.

mathspec suggests two rewrites: one fragment with both generators as rows of the data, or a new name for one of the two parameters.

A given entry that disagrees with its declaration#

Here emissions.yaml lists Generator_p as binary under given:, and generator.yaml declares it as continuous:

emissions.yaml

given:
  variables:
    Generator_p: { dims: [snapshot, generator], domain: binary }
dimensions:
  snapshot: { dtype: int }
  generator: { dtype: str }
parameters:
  Generator_co2: { dims: [generator] }
  co2_cap: { dims: [] }
constraints:
  co2_limit:
    dims: []
    expression: sum(Generator_p * Generator_co2) <= co2_cap

merge refuses emissions.yaml because the two domains differ:

fragment 'emissions.yaml' reads the given variable 'Generator_p' as {'dims': ['generator', 'snapshot'], 'domain': 'binary'}, where 'generator.yaml' introduces it as {'dims': ['generator', 'snapshot'], 'bounds': {'lower': 0.0, 'upper': 'Generator_p_nom'}}. Restate the dims as 'generator.yaml' declares them, or leave the field out.

A fragment that does not load alone#

Where to_spec refuses a fragment, merge names the fragment, then quotes the message of to_spec. No other fragment in the list can make that fragment load. Here generator.yaml declares Generator_p and also lists it under given:, so the two entries collide:

fragment 'generator.yaml' does not load on its own. Declare what it builds, and read under 'given:' what another fragment builds.
Given variable 'Generator_p' collides with the variable of the same name. Rename one of them.

A partial entry under a new name#

A patch that lists only some fields of an entry must edit an entry that the base, or an earlier patch, declares. override refuses a misspelt name, and does not take it as a new entry. Here project.yaml writes power_balnce: {dims: [snapshot]}, and mathspec suggests the closest name:

patch 'project.yaml' edits the constraint 'power_balnce', which its base does not declare. Did you mean 'power_balance'? A patch creates an entry only by writing it whole, and this one is not: a constraint needs `expression`.

A dimension restated differently#

Over a dimension or a relation that its base declares, a patch may change only description, a dimension's ordered and a relation's missing. A patch that writes any other field restates the entry. Every other field must then be as the base declares it, and a field the patch leaves out equals its default.

Here relabelled.yaml writes snapshot: {dtype: str} over a base that declares snapshot: {dtype: int}. override refuses the patch because the two dtype values differ:

patch 'relabelled.yaml' declares the dimension 'snapshot' as {'dtype': 'str'}, where its base declares {'dtype': 'int'}. A patch may change only `description` and `ordered` of a dimension its base declares, because the math in the base is already written over it. Every other field must be as the base declares it, and a field the patch leaves out reads as its default. Restate the dimension as its base declares it, leave it out, or give the patch a dimension of its own under a name of its own.

`ordered: false` over an ordered dimension#

Here unordered.yaml writes snapshot: {dtype: int, ordered: false} over a base that declares snapshot ordered. override refuses the patch because it removes ordered:

patch 'unordered.yaml' says the dimension 'snapshot' is not ordered, where its base declares it ordered. A patch can add `ordered: true` but cannot remove it, because a construct in the base may step along it. Leave `ordered` out of the patch.

A section set to `null`#

override refuses null on a section other than objective. Here project.yaml writes constraints: null, and mathspec suggests two rewrites:

patch 'project.yaml' sets 'constraints' to null, which removes nothing. Remove the entries one at a time, each under its own name, or leave the section out of the patch.

`null` on an entry the base does not declare#

override refuses a null on an entry that the base does not declare. Here stale.yaml writes constraints: {power_balnce: null}, and mathspec suggests the closest name that the base declares:

patch 'stale.yaml' removes the constraint 'power_balnce', which its base does not declare. Delete the removal from the patch, or fix the name. Did you mean 'power_balance'?

A term on a merged sum#

Here a patch that declares a store and its term Store_injection is laid over the merged library. mathspec refuses the term because the base defines Bus_injection:

Named expression 'Store_injection': it adds to 'Bus_injection', which this file defines. Write the term into the body of 'Bus_injection', or read 'Bus_injection' under 'given: expressions:' and add its body as a term.

A rename of a name the spec lacks#

rename refuses a key of the map that source neither declares nor lists under given:. Here the map has Unit_pnom in place of Unit_p_nom:

rename cannot change 'Unit_pnom', which the spec neither declares nor lists under 'given:'. Did you mean 'Unit_p_nom'?

A new name that an entry already has#

rename refuses a new name that an entry outside the map already has, because the spec would then declare that name twice. Here the map is {'Unit_p': 'Unit_p_nom'}:

rename gives the variable 'Unit_p' the name 'Unit_p_nom', which the parameter 'Unit_p_nom' already has. Rename that one too, or choose another name.

One new name for two names#

rename refuses a map that gives two names the same new name. Here the map gives Unit_p and Unit_p_nom the new name Generator_p:

rename gives 'Unit_p' and 'Unit_p_nom' the one name 'Generator_p'. Give each a name of its own.

A new name that a macro uses as an argument#

rename refuses a new name that is an argument of a macro whose template uses the old name. The template would otherwise use the argument in place of the entry. Here the macro scaled has the argument x and the template x * Unit_marginal_cost, and the map is {'Unit_marginal_cost': 'x'}:

rename gives the parameter 'Unit_marginal_cost' the name 'x', which is an argument of the macro 'scaled', whose template uses 'Unit_marginal_cost'. Choose another name.

A name that is also a relation column#

rename refuses a map that changes a name to or from the name of a relation column that is not named after its dimension. An expression such as ends[bus0] uses the column by its name, so the rename would change the column too. Here the relation ends has the columns {bus0: bus, bus1: bus}, and the map is {'bus0': 'price'}:

rename changes 'bus0' to 'price', and 'bus0' is also a column of the relation 'ends'. Give the column another name in the file, or choose another name.

A renamed spec that does not load#

rename loads the renamed spec with to_spec, so a new name that is a built-in operator gets the refusal of to_spec. Here the map is {'Unit_p': 'sum'}:

Variable 'sum' collides with the built-in operator 'sum'. Rename the variable.