Evaluate and Compare STICS Crop Model Versions • SticsREval

Overview

SticsREval is an R package for evaluating and comparing versions of the STICS crop model. It supports multiple types of evaluation, each implemented as an independent class sharing the same Configuration object. Tests can be run individually or combined into a custom pipeline.

The package currently provides two types of evaluation:

Statistical Evaluation — assesses whether a new version of STICS performs better, equally, or worse than a reference version, both against field observations and against reference simulation outputs
Balance Closure Test — checks the internal consistency of water and nitrogen balances in the simulations

Installation

You can install the development version of SticsREval from GitHub:

# install.packages("devtools")
devtools::install_github("SticsRPacks/SticsREval")

Or using the pak package:

# install.packages("pak")
pak::pak("SticsRPacks/SticsREval")

Reproducible environment with `renv`

SticsREval uses renv to ensure reproducible package dependencies. The renv.lock file records the exact versions of all dependencies.

To restore the project environment locally:

# install.packages("renv")
renv::restore()

This will install all required packages at the versions specified in renv.lock. It is recommended to run this after cloning the repository and before running any code.

Dependencies

SticsREval relies on the following SticsRPacks packages:

Package	Role
`SticsRFiles`	Reading simulated and observed data
`SticsOnR`	Running STICS simulations (optional)
`CroPlotR`	Computing statistical criteria and generating plots

Workflow

Configuration is the single entry point for all parameters. Once defined, it is passed to any combination of evaluation classes and export functions:

  Configuration$new(...)                         ← defines all parameters (paths, options, filters)
           │
           ├──► Evaluation$new(config)$run()                  ← statistical evaluation vs obs & reference
           │         │
           │         └──► export_stats_to_csv(config)         ← export statistics + deteriorated USMs to CSV
           │
           └──► BalanceClosureTest$new(config)$run()          ← water & nitrogen balance closure check

All evaluation results are stored in an EvalWorkspace, which manages Parquet datasets on disk.

Configuration

`Configuration`

Encapsulates and validates all configuration parameters for the package. The same object is passed to all workflow classes and functions. Fields are validated against a declarative schema at construction time — all errors are collected and reported together.

library(SticsREval)

config <- Configuration$new(
  stics_exe          = "/path/to/stics",
  usms_workspace     = "workspace/",
  metadata_file      = "metadata.csv",
  eval_workspace     = "eval_workspace/",
  output_dir         = "outputs/",
  run_simulations    = TRUE,
  verbose            = 1L,
  parallel           = FALSE,
  cores              = NA,
  reference_version  = NULL,
  percentage         = 5,
  species            = NULL,
  usms               = NULL,
  var2exclude        = NULL
)

Field	Description
`stics_exe`	Path to the STICS executable (required)
`usms_workspace`	Path to the USMs input data directory (required)
`metadata_file`	Path to the metadata CSV file describing simulations (required when `run_simulations = TRUE`)
`eval_workspace`	Path to the evaluation workspace — (required for evaluation)
`output_dir`	Output directory for CSV exports and plots (required for export and plots workflows)
`run_simulations`	Whether to run STICS simulations (default: `FALSE`)
`verbose`	Logging verbosity level: `0` = silent, `1` = info, `2` = debug (default: `1`)
`parallel`	Enable parallel execution (default: `FALSE`)
`cores`	Number of cores for parallel execution (`NA` = auto; required when `parallel = TRUE`)
`reference_version`	Version string present in `eval_workspace` to use as the reference for regression detection
`percentage`	Threshold (%) above which a variable is flagged as deteriorated vs. the reference (default: `5`)
`species`	Optional character vector of species to evaluate. `NULL` = all available.
`usms`	Optional character vector of USMs to evaluate. `NULL` = all available.
`var2exclude`	Optional character vector of variables to exclude from evaluation.

Configuration also exposes workflow-specific validation methods called internally by each function:

config$validate_eval() — checks requirements for the statistical evaluation workflow
config$validate_export() — checks that output_dir is set and writable

Tests and Evaluations

Statistical Evaluation

`Evaluation`

The core statistical evaluation class. It orchestrates the full evaluation workflow:

Optionally initializes the EvalWorkspace (loading simulations and observations from the USMs workspace)
Computes statistics per species (RMSE, nRMSE, bias, R², etc.) against field observations
When reference_version is set, compares statistics against the reference version to detect regressions
Flags variables and USMs where performance has deteriorated beyond the percentage threshold
Saves all results as Arrow/Parquet files in the eval_workspace directory
Displays a summary of comparison results

Evaluation$new(config)$run()

You can also inject custom dependencies for testing or advanced use:

Evaluation$new(
  config,
  workspace = EvalWorkspace$new("eval_workspace/"),
  backend   = ParallelBackend$new(FALSE, NA),
  logger    = default_logger
)$run()

`plot_comparison()`

Generates a scatter plot comparing the rRMSE of the new version vs. the reference version, one point per variable × USM combination. Points are colour-coded by regression status:

Colour	Status	Condition
🔴 Red	Critical	ratio ≥ `percentage` %
🟠 Orange	Warning	0 % < ratio < `percentage` %
🟢 Green	Improved	ratio ≤ 0 %

A diagonal line (slope = 1) marks perfect parity; a dashed line (slope = 1 + percentage/100) marks the deterioration threshold. Variable names are displayed as repelled labels.

ws <- EvalWorkspace$new("eval_workspace/")

ws$get_species_comparison(species = "wheat", percentage = 5)$
  plot_comparison("outputs/rmse_comparison_wheat.png")

The plot is saved as a PNG file at the path provided to output_path. get_species_comparison() returns NULL if no comparison data is available for the requested species (i.e. no reference version was set during evaluation).

`export_stats_to_csv()`

Exports the evaluation statistics to CSV files in output_dir:

File	Content
`species_stats.csv`	Statistical metrics per species
`global_stats.csv`	Global statistical criteria (RMSE, nRMSE, bias, R², etc.)
`RMSE_per_usm.csv`	RMSE broken down per USM
`Deteriorated_USM.csv`	USMs with deteriorated performance vs. the reference version

export_stats_to_csv(config)

`BalanceClosureTest`

Checks the water and nitrogen balance closure for each simulated USM. For each USM, the class compares the initial and final values of the following five balances:

Balance	Checked fields
Water	`init_H2O_balance` / `final_H2O_balance`
Plant nitrogen	`init_plant_N_balance` / `final_plant_N_balance`
Soil mineral nitrogen	`init_soil_mineral_N_balance` / `final_soil_mineral_N_balance`
Soil organic nitrogen	`init_soil_organic_N_balance` / `final_soil_organic_N_balance`
Soil organic carbon	`init_soil_organic_C_balance` / `final_soil_organic_C_balance`

A USM is flagged if its rounded initial and final values differ. USMs with missing fields or fully NA values are silently skipped.

config <- Configuration$new(
  stics_exe      = "/path/to/stics",
  metadata_file  = "metadata.csv",
  usms_workspace = "path/to/usms_workspace"
)

BalanceClosureTest$new(config)$run()

The run() method logs a summary of the test and lists any USMs with balance closure issues. It respects the usms, parallel, and cores filters defined in the Configuration.

`EvalWorkspace`

Manages reading and writing all evaluation data (simulations, observations, statistics, comparisons) stored as Parquet datasets on disk. Used internally by Evaluation and export_stats_to_csv but can also be used directly to inspect results.

ws <- EvalWorkspace$new("eval_workspace/")

# Read simulations and observations
sim <- ws$get_sim(species = "wheat")
obs <- ws$get_obs(species = "wheat")

# Read statistics
stats <- ws$get_stats(species = "wheat", collect = TRUE)

# List all evaluated versions
ws$get_all_versions()

# Access data for a specific version
ws_ref <- ws$with_version("v10.0")
ref_stats <- ws_ref$get_stats(species = "wheat")

Method	Description
`get_sim(species, usms, var2exclude)`	Return simulated data
`get_obs(species, usms, var2exclude)`	Return observed data
`get_stats(species)`	Return global evaluation statistics
`get_rmse_per_usm(species)`	Return RMSE broken down per USM
`get_deteriorated_usm(species, percentage)`	Return deteriorated USM comparison
`get_species_comparison(species, percentage)`	Return RMSE comparison vs reference
`get_all_versions()`	List all evaluated STICS versions in the workspace
`with_version(version)`	Return a new `EvalWorkspace` scoped to a specific version

Complete Example

library(SticsREval)

# 1. Configure the evaluation
config <- Configuration$new(
  stics_exe         = "/path/to/stics_candidate",
  usms_workspace    = "workspace/",
  metadata_file     = "metadata.csv",
  eval_workspace    = "eval_workspace/",
  output_dir        = "outputs/",
  run_simulations   = TRUE,
  reference_version = "v10.0",
  percentage        = 5
)

# 2. Run statistical evaluation vs observations and reference version
Evaluation$new(config)$run()

# 3. Export statistics and deteriorated USMs to CSV
export_stats_to_csv(config)

# 4. Check water and nitrogen balance closure
BalanceClosureTest$new(config)$run()

Docker

A pre-built Docker image is available on the GitHub Container Registry, so you can run SticsREval without installing R or any dependencies locally.

Pull the image

docker pull ghcr.io/sticsrpacks/stics-r-eval:latest

Run an interactive R session

docker run --rm -it \
  -v /path/to/your/workspace:/workspace \
  ghcr.io/sticsrpacks/stics-r-eval:latest \
  R

Then inside R:

library(SticsREval)

config <- Configuration$new(
  stics_exe       = "/path/to/stics",
  usms_workspace  = "/workspace/",
  metadata_file   = "/workspace/metadata.csv",
  eval_workspace  = "/workspace/eval_workspace/",
  output_dir      = "/workspace/outputs/",
  run_simulations = TRUE
)

Evaluation$new(config)$run()
BalanceClosureTest$new(config)$run()

Run a script non-interactively

docker run --rm \
  -v /path/to/your/workspace:/workspace \
  -v /path/to/your/script.R:/script.R \
  ghcr.io/sticsrpacks/stics-r-eval:latest \
  Rscript /script.R

Build the image locally

# Without a GitHub token
docker build -t stics-r-eval .

# With a GitHub token (needed to install private SticsRPacks dependencies)
docker build \
  --secret id=GITHUB_TOKEN,src=<(echo $GITHUB_PAT) \
  -t stics-r-eval .

Note: The image is based on rocker/r-ver:4 and uses renv to ensure reproducible package versions. Dependencies are restored from renv.lock at build time.

SticsRFiles — Read/write STICS input and output files
SticsOnR — Run STICS simulations from R
CroPlotR — Crop model evaluation statistics and plots

Citation

If you use SticsREval in your work, please cite it as follows:

citation("SticsREval")

Contributing

Contributions are welcome! Please open an issue or submit a pull request on GitHub.

Code of Conduct

Please note that this project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

SticsREval

Overview

Installation

Reproducible environment with `renv`

Dependencies

Workflow

Configuration

`Configuration`

Tests and Evaluations

Statistical Evaluation

`Evaluation`

`plot_comparison()`

`export_stats_to_csv()`

`BalanceClosureTest`

`EvalWorkspace`

Complete Example

Docker

Pull the image

Run an interactive R session

Run a script non-interactively

Build the image locally

Citation

Contributing

Code of Conduct

Links

License

Citation

Developers

Dev status

SticsREval

Overview

Installation

Reproducible environment with renv

Dependencies

Workflow

Configuration

Configuration

Tests and Evaluations

Statistical Evaluation

Evaluation

plot_comparison()

export_stats_to_csv()

BalanceClosureTest

EvalWorkspace

Complete Example

Docker

Pull the image

Run an interactive R session

Run a script non-interactively

Build the image locally

Related packages

Citation

Contributing

Code of Conduct

Links

License

Citation

Developers

Dev status

Reproducible environment with `renv`

`Configuration`

`Evaluation`

`plot_comparison()`

`export_stats_to_csv()`

`BalanceClosureTest`

`EvalWorkspace`