13  Solving the model

13.1 Overview

ems_solve() runs the teems solver on a deployed model and, by default, parses the results into structured R objects with ems_compose(). A teems Docker image must be available. After the run, the solve is checked for singularity, accuracy and errors, and the findings are written to model_diagnostics.txt in the run directory. On Linux the solver container runs as the calling user, so the files it writes to the run directory belong to you.

ems_solve() chooses nothing on your behalf: the run is exactly the arguments given, so it can be reproduced from its record. A bare ems_solve(cmf_path) solves with Gragg 2-4-6 (as GEMPACK) and LU on one task and one thread, which is accurate and correct for every model but slow for large ones. ems_probe() suggests what to use instead.

ems_solve(
  cmf_path,
  solution_method = c("Gragg", "Johansen", "Euler", "Midpoint", "RK2",
                      "Heun", "RK4", "BoSha32", "DoPri54"),
  matrix_method = c("LU", "DBBD", "SBBD", "NDBBD"),
  n_subintervals = 1L,
  steps = NULL,
  n_tasks = 1L,
  n_threads = 1L,
  precision = c("single", "double"),
  verbosity = 1L,
  suppress_outputs = FALSE,
  terminal_run = FALSE,
  accuracy_vars = NULL,
  ...
)

13.2 Basic usage

outputs <- ems_solve(
  cmf_path = cmf_path,
  solution_method = "Gragg",
  matrix_method = "LU"
)

13.3 Arguments

Argument Default Description
cmf_path Path to the CMF file generated by ems_deploy()
solution_method "Gragg" How the shock is applied (see Solution methods)
matrix_method "LU" How each linear system is solved (see Matrix methods)
n_subintervals 1L Splits the shock into this many parts, each solved in turn. Helps with large shocks
steps c(2L, 4L, 6L) Step counts. For "Gragg", "Midpoint" and "Euler", two or three strictly increasing counts are extrapolated, and a single count is one multi-step run without extrapolation (see Single-pass runs); a single count for the Runge-Kutta methods
n_tasks 1L Parallel tasks. Must be 1L with "LU"
n_threads 1L Threads per task
precision "single" "double" uses the double-precision solver: about twice the memory, for runs with many steps or a near-singular system
verbosity 1L Log detail: 0 errors, warnings and the accuracy summary; 1 adds progress and timings; 2 adds per-rank detail and a one-line tally of what was written (coefficients, sets, variables)
suppress_outputs FALSE When TRUE, outputs are not parsed; the CMF path is returned invisibly for a later ems_compose()
terminal_run FALSE When TRUE, prepares the run without starting it, so it can be launched from a terminal
... | | Named extras: the Runge-Kutta controls (adaptive, eps_tolerance, max_retries, retry_adjust; see Runge-Kutta controls), the MA48 workspace guesses (laA, laD, laDi) and expert solver flags (e.g. condest, inmemory, postsim). Unknown names are an error |

The severities of TAB Assertion failures and declared-range violations are package options (assertions, range_test_initial, range_test_updated).

13.4 Solution methods

Method Use it for
"Gragg" The default and the standard accurate method: several multi-step solutions combined by extrapolation (see the GEMPACK manual)
"Johansen" A quick one-step linear approximation. Good for checking that a model and closure run; not accurate for nonlinear results or large shocks
"Euler" Like "Gragg" but converges more slowly. With adaptive "DoPri54" or "BoSha32", the choice for severe shocks, such as a variable driven close to -100%, where "Gragg" and "Midpoint" can become unstable (GEMPACK manual 30.2; Schiffmann 2022)
"Midpoint" Gragg’s method without its final smoothing pass. It never carries the shocks past their end point, so it takes a shock of exactly -100%, which "Gragg" refuses
"RK2", "Heun", "RK4" Runge-Kutta methods with a single step count and no extrapolation. "RK4" with a few steps is an inexpensive way to get three to four correct digits on large shocks
"BoSha32", "DoPri54" Runge-Kutta methods that estimate their own error and can control their step size (adaptive = "yes")

"Gragg" suits most work; "Johansen" is a quicker way to check that a new setup runs, and adaptive "DoPri54" is an alternative for large shocks. A shock that is too large for the step schedule shows up as a singularity abort preceded by warnings that updated coefficients crossed their declared bounds; ems_solve() then names the remedies below rather than the closure. On large shocks, use more than one subinterval with "Gragg":

outputs <- ems_solve(
  cmf_path = cmf_path,
  solution_method = "Gragg",
  n_subintervals = 4,
  steps = c(2, 4, 8)
)

With matrix_method = "DBBD", a run split into subintervals can occasionally abort with a singularity partway through the shock although the closure is sound; the solver names the elements involved. Change n_subintervals (1 is the simplest) or use matrix_method = "LU" for that run.

Adaptive "DoPri54" reaches about the same accuracy with roughly half the work. It controls its own step size, so n_subintervals does not apply, and it returns a per-element error_estimate column. Treat the estimate as a pointer to the least settled results, not as a bound on the error.

outputs <- ems_solve(
  cmf_path = cmf_path,
  solution_method = "DoPri54",
  steps = 4,
  adaptive = "yes",
  eps_tolerance = 0.01
)
outputs$dat$qgdp$error_estimate

To check that a result has converged, repeat the run with a tighter eps_tolerance or more steps and compare. The finer Runge-Kutta controls are described below.

13.4.1 Step counts and single-pass runs

"Gragg", "Midpoint" and "Euler" normally solve three times with increasing step counts and extrapolate from the three results, which also gives the accuracy summary. The summary covers every endogenous result unless accuracy_vars names the variables it should cover, for example accuracy_vars = c("qgdp", "pm"). The "Gragg" and "Midpoint" counts must be all even or all odd (c(2, 4, 6) or c(3, 5, 7)); even counts are recommended. Two counts (c(2, 4)) extrapolate from two results, with no accuracy summary. The package option converge = TRUE applies GEMPACK’s treatment of results that do not converge cleanly: their extrapolation is replaced by the result with the most steps, or by the average when the three results are nearly equal.

A single count gives one multi-step run without extrapolation:

outputs <- ems_solve(
  cmf_path = cmf_path,
  solution_method = "Euler",
  steps = 16
)

Use it for models whose updated data must follow the steps themselves, such as a step counter or an iterative (RAS-style) balancing update, which extrapolation would distort. No accuracy summary is available for a single-pass run.

13.4.2 Runge-Kutta controls

ems_RK(
  cmf_path,
  solution_method = c("DoPri54", "BoSha32", "RK4", "Heun", "RK2"),
  steps = 4L,
  eps_tolerance = 0.01,
  adaptive = NULL,
  matrix_method = c("LU", "DBBD", "SBBD", "NDBBD"),
  n_tasks = 1L,
  n_threads = 1L,
  precision = c("single", "double"),
  n_subintervals = 1L,
  verbosity = 1L,
  suppress_outputs = FALSE,
  terminal_run = FALSE,
  accuracy_vars = NULL,
  max_retries = 3L,
  retry_adjust = 0.5,
  chart = c("log", "percent"),
  error_norm = c("max", "rms"),
  controller = c("std", "pi"),
  scope = c("pct", "all"),
  h_init = NULL,
  guard = NULL,
  ...
)

ems_RK() is ems_solve() for the Runge-Kutta methods. It defaults to adaptive "DoPri54" and adds the step-size controls; the remaining arguments are those of ems_solve(), with the same defaults. When most retries come from step checks (a variable approaching -100%, a range test) rather than from eps_tolerance, the cheaper steps of "BoSha32" can make it faster than "DoPri54" (Schiffmann 2022).

# adaptive DoPri54
outputs <- ems_RK(cmf_path)

# fixed-step RK4
outputs <- ems_RK(cmf_path, solution_method = "RK4", steps = 8L)
Argument Default Description
solution_method "DoPri54" "DoPri54", "BoSha32", "RK4", "Heun" or "RK2"
steps 4L Number of steps; under adaptive control, the starting number
adaptive "yes" for "DoPri54"/"BoSha32", otherwise "no" "yes" retries steps that miss eps_tolerance or fail a step check (a variable crossing -100%, a range test or an assertion, as in GEMPACK) and adjusts the step size; "accuracy-only" ignores the step checks
eps_tolerance 0.01 Accuracy target per step. GEMPACK suggests 0.1 for most simulations; on large shocks the default 0.01 reaches four correct digits on most elements, and 0.001 gains the last digits at about 1.6 times the cost
max_retries, retry_adjust 3L, 0.5 How often, and by how much, a step is shortened when it fails a step check
chart "log" "log" keeps every level positive; "percent" combines stages in percentage changes
error_norm "max" Whether the worst element ("max") or the root mean square ("rms") decides if a step is accepted
controller "std" "pi" rejects fewer steps on rough solution paths
scope "pct" Which variables steer the step size: percentage-change variables ("pct") or all variables ("all")
h_init, guard NULL First step length, at most 1/steps (a longer one is refused); and the level ratio at which a stage is rejected under chart = "log"

The defaults suit most runs.

13.5 Matrix methods

Method Models Description
"LU" all Sequential LU factorization. The most robust; one task only
"DBBD" static Splits the system into blocks (usually by region) solved in parallel, treating regions as separable economies (Kompas and Van Ha 2019)
"SBBD" intertemporal Splits the system by time period, solved in parallel (Van Ha and Kompas 2016; Van Ha et al. 2017)
"NDBBD" intertemporal A nested split for intertemporal models too large for "SBBD" in the available memory

13.6 Memory

Before a run starts, teems-R estimates its peak memory and compares it with the memory available to the container (on Docker Desktop, the limit under Settings > Resources). A run that clearly will not fit is refused before it starts, and the message names the remedies:

  • a coarser aggregation or a condensed model,
  • fewer tasks (each "DBBD" task holds its own copy of the data),
  • "NDBBD" for an intertemporal model,
  • more memory for Docker, or a larger machine.

A run close to the limit gets a warning instead. With "DBBD", each linear solve also gets one refinement step, which holds the system matrix through the solve; the memory estimate includes it (see the refine package option). During the solve, the log reports memory after each phase, and the peaks are recorded in model_diagnostics.txt.

The solver sizes its factorization workspace automatically and grows it when needed, so laA, laD and laDi are normally left unset. If the log shows the workspace growing, passing the size recorded as la_used in sol.stats.json skips that search on the next run. A single "LU" factorization cannot exceed the 32-bit workspace limit; a system that needs more must be condensed, aggregated more coarsely, or solved with a parallel method.

13.7 Checking solve quality

If a model solves but the results look implausible, the system may be numerically near-singular, typically because the closure depends on base-data flows close to zero. condest = TRUE measures how sensitive each linear solve is to the data:

outputs <- ems_solve(
  cmf_path = cmf_path,
  solution_method = "Gragg",
  matrix_method = "LU",
  condest = TRUE
)

Values of kappa_w2 around 1e8 are normal for CGE models; values approaching 1e15 trigger a warning that the solution is unreliable. Look for near-zero flows the closure depends on, or rerun with precision = "double". The check does not change results, costs a few percent of solve time, and applies to "LU" only.

jacdump = TRUE writes the linearized system at the base data, every equation element against every variable element, to sol.jac beside the solution, with the equation map in sol.jac.json. It is taken before the first step and does not change results.

13.8 Reproducibility

The same deployment solved with the same image gives byte-identical results with "LU", "SBBD" and "NDBBD", at any thread count and on any machine the image supports. Three cases agree only to rounding:

  • "DBBD" with more than one thread, where a multi-step solve of a poorly conditioned model can amplify the rounding on its least determined results; use n_threads = 1L when byte-for-byte results matter,
  • runs with more than one task, compared across machines,
  • different image variants (for example amd64 and arm64).

Each run records the solver version and numerical kernels it used in model_diagnostics.txt.

13.9 Solver version

Before every solve, teems-R checks that the image’s solver version matches the package. A missing, old or mismatched solver stops the solve and names the pull or rebuild step. If the image still does not accept a solver option or input file that teems sends, the solve stops and names it rather than running without it; use the image released with your version of teems.

13.10 Concurrent runs

Several runs can solve at the same time, one deploy directory each. The directory is set by the tempdir option, so give every run its own before deploying it:

library(parallel)
outputs <- mclapply(c("Johansen", "Gragg"), function(method) {
  ems_option_set(tempdir = file.path("~/runs", method))
  dat <- ems_data(dat_input, par_input, set_input,
                  REG = "big3", ACTS = "macro_sector")
  cmf <- ems_deploy(dat, model, shock)
  ems_solve(cmf, solution_method = method)
}, mc.cores = 2)

mclapply() forks the R session, so each run sets its own tempdir; forking is not available on Windows, where parLapply() serves instead.

A second solve in a deploy directory that is already solving is refused. If an interrupted run leaves its claim behind, the message names the .solve_lock directory to remove.

13.11 Terminal mode

For long-running models, terminal_run = TRUE prepares the run and exits, so you can close R and launch the solve from a terminal:

ems_solve(
  cmf_path = cmf_path,
  solution_method = "Gragg",
  terminal_run = TRUE
)

13.12 Solving in-situ

solve_in_situ() calls the teems-solver directly with user-supplied input files, bypassing the ems_data() / ems_model() / ems_deploy() pipeline. The model file is parsed, but input files must be provided in their final form: no checks or modifications are applied to them. This suits pre-prepared input files, or re-solving the files written to disk by ems_deploy().

solve_in_situ(
  ...,
  model_dir,
  model_file,
  closure_file,
  shock_file,
  ignore_condense = FALSE,
  solution_method = c("Gragg", "Johansen", "Euler", "Midpoint", "RK2",
                      "Heun", "RK4", "BoSha32", "DoPri54"),
  matrix_method = c("LU", "DBBD", "SBBD", "NDBBD"),
  n_subintervals = 1L,
  steps = NULL,
  n_tasks = 1L,
  n_threads = 1L,
  precision = c("single", "double"),
  verbosity = 1L,
  suppress_outputs = FALSE,
  terminal_run = FALSE,
  accuracy_vars = NULL,
  solver_args = NULL
)

13.12.1 Arguments

Argument Default Description
... Input files, named after the model’s File statements
model_dir Directory the run uses; input files not already there are copied in, and outputs are written there
model_file Path to the model Tablo (.tab) file
closure_file Path to the closure (.cls) file
shock_file Path to the shock (.shf) file
ignore_condense FALSE Set to TRUE when the closure and shock files come from a model loaded with ems_model(..., ignore_condense = TRUE)
suppress_outputs FALSE When TRUE, returns the CMF file path instead of parsed outputs
solver_args NULL Named list of everything ems_solve() takes through ..., the Runge-Kutta controls included, e.g. list(condest = TRUE) or list(adaptive = "yes")

The solve arguments (solution_method, matrix_method, n_subintervals, steps, n_tasks, n_threads, precision, accuracy_vars and the rest) work as in ems_solve().

13.12.2 Input files

Each name passed through ... must match a File statement in the model Tablo file. For GTAP-RE, the Tablo file declares:

File GTAPDATA # base data file;
File GTAPINT  # intertemporal data file;
File GTAPPARM # parameter file;
File GTAPSETS # set file;

These names are then used as the named arguments:

solve_in_situ(
  GTAPDATA = file.path(write_dir, "teems", "GTAPDATA.txt"),
  GTAPINT = file.path(write_dir, "teems", "GTAPINT.txt"),
  GTAPPARM = file.path(write_dir, "teems", "GTAPPARM.txt"),
  GTAPSETS = file.path(write_dir, "teems", "GTAPSETS.txt"),
  model_dir = "~/in_situ_run/",
  model_file = "GTAP-RE.tab",
  closure_file = "GTAP-RE.cls",
  shock_file = "a_shock_file.shf",
  solution_method = "Gragg",
  matrix_method = "SBBD",
  n_tasks = 4L
)

The model file is processed exactly as ems_model() processes it, so in-TAB Substitute and Backsolve statements apply.

By default, solve_in_situ() returns the model’s variables and coefficients; with suppress_outputs = TRUE it returns the CMF path, which can be passed to ems_compose().