7  Loading models

7.1 Overview

The ems_model() function loads the model and its closure, conducts pre-deployment checks, determines temporal dynamics, and allows for the loading of aggregated coefficient data. The output of this function is a tibble with attributes containing the parsed model input and is a required input to the "model" argument of the ems_deploy() function.

ems_model(
  model_file,
  closure_file,
  backsolve = NULL,
  ignore_condense = FALSE,
  ...
)
Argument Type Description
model_file Character File name in working directory or path to a .tab file
closure_file Character File name in working directory or path to a .cls closure file
backsolve Character vector or NULL Endogenous variables to substitute out by backsolving
ignore_condense Logical Ignore the Substitute, Backsolve and Omit statements found in the model file
... Named arguments Coefficient value injections (numeric, data frame, or CSV path)

Both model_file and closure_file are required arguments.

7.2 The model file

Model input in teems is limited to Tablo (.tab) files. Vetted models and their closures are available at teems-models, and all but ORANI-G also through ems_example(); their documentation is listed under References:

Model Description Database
GTAPv6 Classic GTAP model, version 6.2 GTAP v6.2 format
GTAPv7 Standard GTAP model, version 7.1 GTAP v7 format
GTAP-RE Intertemporal GTAP with rational expectations: forward-looking investment, all periods solved at once GTAP v7 format
GTAP-AEZ GTAP with land use by agro-ecological zone: land enters production as a CES composite of the zones GTAP-AEZ release
GTAP-E GTAP with energy substitution and carbon taxes: a capital–energy composite inside value added GTAP-E release
GTAP-EP GTAP-E with electricity split into transmission and distribution and generation, and generation into base-load and peak-load technologies GTAP-Power release
ORANI-G Single-country model of the Centre of Policy Studies, 2013 edition Its own basedata.har

Users may load their own models by providing a path to a Tablo file and closure:

model <- ems_model(
  model_file = "path/to/custom_model.tab",
  closure_file = "path/to/custom_model.cls"
)

When preparing a custom model, the vetted models are a good reference, and often the easiest starting point for modifications.

7.3 Closure selection

Standard model-specific closures are available within the model repository: teems-models. The general format consists of an exogenous block listing all (\n delimited) exogenous variables, followed by ;, rest endogenous, and ; to mark the end:

exogenous
pop
aoreg("row")
qe(ENDWC,REG,INITIME)
txs(MARG,"row",REG,FWDTIME)
...
;
rest endogenous
;

Each entry is either a full variable name (pop), an entry with specific set elements (aoreg("row")), a mixed entry (txs(MARG,"chn",REG,FWDTIME)) or an entry with subsets (qe(ENDWC,REG,INITIME)).

7.4 Tablo compatibility

The teems-solver follows GEMPACK’s Tablo language, so most model files written for GEMPACK load as they are. This manual will not attempt to reproduce the very thorough GEMPACK documentation, rather lists what is supported and the few forms that are not. ems_model() checks the model file before anything is solved. Problems are reported by name, with the statement it occurs in, rather than surfacing later as a solver failure. If a model file does not load and you think it should, a supported statement behavior is not as expected, or an error propagates to the solver without warning, please get in touch.

7.4.1 Supported statements

File, Coefficient, Read, Update, Set, Subset, Mapping, Formula, Assertion, Variable, Equation, Write, Zerodivide, Complementarity, Loop with Break and Cycle, PostSim sections, and the condensation statements Substitute and Backsolve (see Condensation) are accepted. Omit statements hold the omitted variables exogenous and unshocked, as omission does in GEMPACK; they stay in the model and its outputs. Levels variables and equations, Formula & Equation, and Read (IfHeaderExists) are supported; levels equations may contain conditional sums, PROD and the functions SQRT, EXP, LOGE, LOG10 and IF. PROD, MAXS and MINS reduce over a set like SUM. Default statements (Variable (Default=Levels);, Equation (Default=Levels);, Formula (Default=Initial);, Coefficient (Default=Parameter);) apply to the statements that follow them. Read, Formula and Assertion statements take effect in the order they appear, including inside PostSim sections: a Read after a default Formula replaces the default, and a Formula after a Read replaces what was read. A levels variable X has a linear variable named p_X, or c_X for a (change) variable; Linear_Name= gives it another name and Linear_Var= makes an already declared linear variable serve as it. In closures, shocks and swaps either name may be used. p_X and c_X refer to nothing else: beside a linear variable X, names such as p_X or C_X are free for other coefficients and variables.

Display and Transfer are not supported and teems raises an error: every coefficient is in the coefficient dump, and teems-solver writes no Header Array output files.

7.4.2 Names

Names are case-insensitive. A set, coefficient, variable, file or equation may be written in any case after its declaration, and teems uses the declared spelling throughout, including in outputs. The same holds for header names (Read ... header "Nat" finds a header NAT), logical file names, the .tab and .har extensions, and the variable, set and element names in closures, shocks and swaps.

7.4.3 Variable types for homogeneity checks

VPQType marks a variable as a Value, Price or Quantity, or as None (tax rates, shifters, technology, foreign prices), for homogeneity tests (GEMPACK manual 57.2). All three GEMPACK forms are read, in file order:

Variable (begins p default VPQType Price);  ! later variables starting with p !
Variable (begins pf default VPQType None);  ! the longest matching prefix wins !
Variable (VPQType=Quantity) (all,i,IND) x1cap(i);
Variable (Name x6com VPQType Quantity);     ! anywhere in the file !

Variable (begins <prefix> VPQType default OFF); ends a default. Variables without a type stay Unspecified. ORIG_LEVEL= qualifiers are checked: the coefficient must be real and range over exactly the variable’s sets.

7.4.4 Formulas, updates and mappings

  • Formula quantifiers may carry general conditions, such as (all,c,COM: $POS(c) = $POS(g)), a coefficient on either side, or a mapping (MAP(z) = r). Update statements take quantifier conditions too; an element whose condition is false is left unchanged.
  • A qualifier list such as (initial, by_elements) is accepted. write updated value to file is accepted with a warning but writes no separate file; the value is in the coefficient outputs.
  • An index may be repeated within one reference (T(d,d), TRADE(c,"dom",r,r)).
  • A mapping may point into a subset of the set its argument position is declared over, and a mapped index may range over a subset of the mapping’s domain. A mapping may also be assigned an identity or a copy (MAP(c) = c, A(z) = B(z)), matched by element name when the sets are unrelated.
  • A mapping is read as element names (Read (by_elements)) or as element numbers (Read alone), over its whole domain or over a subset (Read (all,c,MARG) MAP(c) ...), with Formulas assigning the rest.
  • Update statements read the values at the start of each step, and several updates of one element apply in order, the later overriding. (change) and (explicit) updated data follow the solution path and are extrapolated like the variables; see single-pass runs for models where that is not wanted.

7.4.5 Sets

Sets may be read from file, listed explicitly, or built from other sets:

Set A123 = A1 + A2 + A3;
Set B = (C - D) INTERSECT E;
Set B2 # equal to A # = A;
Set UACT = UNITA x ACTS;
Set COALCOMM # coal # =
    (all,c,COMM: sum{cc,DCOMM: MCOM(cc)=c, UNITDCOAL(cc)} > 0);

Set expressions (UNION, INTERSECT, +, -, and the relative complement \), set equality, set products and conditional set builders are supported. A few points to note:

  • A set must not be named union or intersect.
  • A conditional set builder, like COALCOMM above, compares an expression with a constant. Its source set may be read, listed or itself built from other sets. Sets are built before the model is solved, so the condition is evaluated at deployment from the loaded data: the coefficients in it must be read from a file or assigned by Formula statements that precede the Set statement and use only what is known by then (arithmetic, IF, sums, the standard functions); RANDOM cannot appear. GTAP-E’s UNITDCOAL(c) = 0.0 + IF[c="coa", 1.0] is such a formula.
  • Some GTAP models build sets from the EFLG and SLUG headers, which teems excludes from the data. Declare such sets with explicit elements instead.
  • Intertemporal set equality (Set (Intertemporal) B = A;) is not supported.
  • A set may be empty, for example when a conditional set builder selects nothing under the data at hand. Statements over an empty set do nothing and sums over it are zero.

Set element names are lowercase throughout teems and teems-solver (see Set mappings); for example, the v6.2 CGDS sector becomes cgds.

Element order follows the data file wherever a set is not aggregated (see Loading data). It matters to anything that works by position, such as $POS and a mapping read from a header, whose entries line up with the elements of its domain in order.

7.4.6 IF conditions

IF[condition, expression] is supported wherever an expression may stand, in formulas, updates, assertions, PostSim statements and linear or levels equations:

Formula (all,c,COMM)(all,r,REG)
    VCB(c,r) = VDB(c,r) + sum{d,REG, VXSB(c,r,d)} + IF[c in MARG, VST(c,r)];
Equation E_qca (all,c,COMM)(all,a,ACTS)(all,r,REG)
    qca(c,a,r) = IF[MAKES(c,a,r) gt 0,
                    qo(a,r) - ETRAQ(a,r) * [ps(c,a,r) - po(a,r)]];
Formula (all,r,REG) SHR(r) = sum{s,REG, IF[s <> r, VDB("food",s)]} / 2;

A condition may test set membership (c in MARG), compare an index with an element or another index (d = "domestic", r <> s), or compare coefficient expressions (MAKES(c,a,r) gt 0). Comparisons may be joined with AND, OR and NOT ([VDB(c,r) > 0] and not [r = "usa"]), in IF terms as in the conditions of sums and quantifiers. IF terms may be nested, multiplied, divided, raised to powers, placed inside sums and function arguments, and used in formulas without quantifiers (Formula D = 0.0 + IF[NM > 1, 1];). Top-level terms of formulas and equations are rewritten at model load into equivalent unconditional statements, with helper coefficients and sets (named IFC*, IFX*, IFT*, IFV*, IFO*) that appear in outputs like any other coefficient or set; other placements are evaluated by the solver as written.

A condition reads coefficients and levels variables but not linear variables, and a set-membership test (c in MARG) cannot be combined with AND, OR or NOT; nest the IF instead.

7.5 Condensation

Condensation makes the system passed to the solver smaller without changing the model’s economics. teems condenses the model by backsolving variables, and teems-solver solves the condensed system.

The static vetted models come condensed. The GTAPv6, GTAPv7, GTAP-AEZ, GTAP-E and GTAP-EP model files carry the standard GTAP backsolves as Backsolve statements, and ORANI-G its own; ems_model() applies them on load. GTAP-RE carries none, because backsolving slows the parallel methods intertemporal models use (see below). To solve the full system, load a model with ignore_condense = TRUE.

7.5.1 Backsolving variables

A backsolved variable is substituted out of the system using its defining equation and recovered by the solver after each step, so it still appears in the outputs. By default the defining equation is E_<variable>; name it explicitly when it differs:

model <- ems_model(
  model_file = GTAPv7[["model_file"]],
  closure_file = GTAPv7[["closure_file"]],
  backsolve = c("pfd", "pfm", ps = "E_ps")
)

A backsolved variable must be endogenous, and its defining equation must be one GEMPACK could substitute with. If it is not, the error names the requirement that failed. #### Condensation coefficients (CSUB) {#csub}

Substituting a variable out multiplies the coefficients of its defining equation into every equation that used it. Rather than leave those products in the equation text, the engine stores each one, and each sum over an eliminated index, as a new coefficient with its own Formula, named CSUB<n> and appended to the deployed model file. Substituting pint out of GTAPv7 through E_pint, for example, produces

Coefficient (all,a,ACTS)(all,r,REG)(all,c,COMM)
    CSUB392(a,r,c) # backsolve product (E_pint) #;
Formula (all,a,ACTS)(all,r,REG)(all,c,COMM)
    CSUB392(a,r,c) = INTSHR(c,a,r)*[1 - FMSHR(c,a,r)];

and later substitutions build on earlier ones:

CSUB204(r,c) = sum{a,ACTS, CSUB53(r,c,a)*CSUB174(c,a,r)}

The comment names the kind: a backsolve product is such a product or sum; a backsolve pivot is the reciprocal of the coefficient on the backsolved variable when that coefficient is not a plain number.

They are derived from the model’s coefficients, not declared by it, and are re-evaluated with the other formulas at every step. The standard GTAPv7 condensation adds 650 of them to the 283 coefficients the model declares, so the solver’s write tally (verbosity = 2) reports the two groups separately. ems_compose() leaves them out of the default outputs and returns one only when it is named in which, and write_coefficients = TRUE writes no CSV for them.

7.5.2 When backsolving pays

Backsolving shrinks the system but makes it denser. That helps "LU" on static models: the standard GTAP condensation cut a 1.35 million equation Gragg solve by 41 % and a 3.6 million one by more than half. It hurts the parallel matrix methods ("SBBD", "DBBD", "NDBBD"), which rely on the sparsity it removes: on GTAP-RE the same condensation made "SBBD" runs 69 % slower at 200 thousand equations and 393 % slower at 4.4 million, and a condensed static system stops gaining from extra "DBBD" tasks where the full system keeps gaining. On a laptop, threaded "LU" on the condensed system is still the fastest measured below about 120 thousand condensed equations; above that, or on a machine with many cores, deploy without backsolve and use a parallel method. ems_probe() measures the block structure of a deployment and suggests which case it is likely in.

7.5.3 Condensation statements in the model file

Substitute and Backsolve statements in a model file are applied on load:

Substitute p1 using E_p1 ;
Backsolve p1lab using E_p1lab ;

Substitute is carried out as a backsolve, so the variable stays in the outputs. Statements in the file are applied first, then the backsolve argument. ignore_condense = TRUE skips the statements in the file but still applies the argument.

Omit statements add the listed variables to the closure as exogenous, unshocked entries, with a message. They stay in the model and its outputs, so a closure file need not repeat them.

Closure files may also carry xSet and xSubset lines as written in a GEMPACK command file; they are read as Set and Subset statements of the model, so closure entries can refer to them.

7.6 Coefficient value injection

The ... argument in ems_model() allows users to assign aggregation-specific values to model coefficients. This is used to inject modified values into existing coefficients as well as provide data to new coefficients. All data inputted here must be model-aggregation specific and will not be subject to aggregation. The left-hand side (name) must correspond to a coefficient (not header) declared within the model input. Data frame and CSV inputs are validated against the set aggregations from ems_data(). Dimensions must match the aggregated resolution, not the original database resolution. Three input types are supported, with model “Read” and “Formula” declarations modified accordingly:

7.6.1 Uniform numeric value

A single numeric value can be assigned to a coefficient. All set combinations for that coefficient will receive this uniform value.

GTAP_RE <- ems_example("GTAP-RE", tempdir())
model <- ems_model(
  model_file = GTAP_RE[["model_file"]],
  closure_file = GTAP_RE[["closure_file"]],
  KAPPA = 0.54321
)

Providing a numeric vector of length greater than 1 will raise an error. Instead here use a data frame with the appropriate sets to assign multiple heterogeneous values to a coefficient.

7.6.2 Data frame

A data frame (or data frame extension such as tibble or data.table) can be used to assign heterogeneous values across set combinations. The data frame must contain columns for each set index using the model-specific naming convention (set name + variable index, e.g., REGr, COMMc, ALLTIMEt) and a Value column.

# Heterogeneous values for SUBPAR coefficient (Read-type)
GTAP_RE <- ems_example("GTAP-RE", tempdir())
COMMc <- c("crops", "food", "livestock", "mnfcs", "svces")
REGr <- c("usa", "chn", "row")
ALLTIMEt <- seq(0, 10)

SUBPAR <- expand.grid(
  COMMc = COMMc,
  REGr = REGr,
  ALLTIMEt = ALLTIMEt
)

SUBPAR$Value <- runif(nrow(SUBPAR))

model <- ems_model(
  model_file = GTAP_RE[["model_file"]],
  closure_file = GTAP_RE[["closure_file"]],
  SUBPAR = SUBPAR
)

This works for both Read-type and Formula-type coefficients:

# Heterogeneous values for CPHI coefficient (Formula-type)
REGr <- c("usa", "chn", "row")
ALLTIMEt <- seq(0, 10)

CPHI <- expand.grid(
  REGr = REGr,
  ALLTIMEt = ALLTIMEt
)
CPHI$Value <- runif(nrow(CPHI))

model <- ems_model(
  model_file = GTAP_RE[["model_file"]],
  closure_file = GTAP_RE[["closure_file"]],
  CPHI = CPHI
)

7.6.3 CSV file

A path to a CSV file can be used as an alternative to a data frame. The CSV must follow the same structure: set index columns and a Value column.

# Write coefficient data to CSV
write.csv(SUBPAR, "SUBPAR.csv", row.names = FALSE)

model <- ems_model(
  model_file = GTAP_RE[["model_file"]],
  closure_file = GTAP_RE[["closure_file"]],
  SUBPAR = "SUBPAR.csv"
)

7.6.4 Combining injections

Multiple coefficient injections can be combined in a single call:

model <- ems_model(
  model_file = GTAP_RE[["model_file"]],
  closure_file = GTAP_RE[["closure_file"]],
  KAPPA = 0.03,
  SUBPAR = SUBPAR,
  CPHI = CPHI
)

If the coefficient name is not found in the model, an error is raised indicating that the aggregated data coefficient is not declared within the provided model input.

7.7 Intertemporal models

For intertemporal models, ems_model() determines temporal dynamics by detecting intertemporal sets in the Tablo file. By default, the expected header in the GTAP-RE model is "YEAR". If a model uses a different header for timestep data, set it via ems_option_set():

ems_option_set(timestep_header = "AYRS")

If the expected timestep header is not found in the model, an error will be raised with a suggestion to use ems_option_set() to configure a custom timestep header. The same concept applies to the number of timesteps:

ems_option_set(n_timestep_header = "NINTERVAL")

Default values for these settings (and others) may be obtained using:

ems_option_get()

7.7.1 Leads, lags and parallel solving

In intertemporal models, the parallel matrix methods (see Solving the model) solve each period as a block, with the variables that link periods held in a shared border. Only the elements a lead or lag actually references join that border: qo("capital",r,t+1) adds the capital slice of qo and nothing else, and the solver’s log reports the border it found (Border netcut). A lead or lag on a whole variable declared over large sets (for example qo(c,r,t+1) over all commodities and regions) does enlarge the border and can remove the benefit of solving in parallel.

Link periods through a small proxy variable in that case:

Variable (all,r,REG)(all,t,ALLTIME)
    kb(r,t) # beginning-of-period capital stock #;
Equation E_kb
(all,r,REG)(all,t,ALLTIME)
    kb(r,t) = qo("capital",r,t);

and place the lead on the proxy (kb(r,t+1)). The vetted intertemporal models already follow this pattern.