Evaluate and Compare STICS Crop Model Versions • SticsREval

Overview

SticsREval is an R package for evaluating and comparing versions of the STICS crop model. The core idea is to assess whether a new version of STICS performs better, equally, or worse than a reference version, both against field observations and against the reference simulation outputs.

The workflow:

Runs STICS simulations for the version under evaluation via SticsOnR
Compares the resulting outputs against field observations and against reference simulation statistics from a previous model version
Computes statistical evaluation criteria (RMSE, nRMSE, bias, R², etc.) using CroPlotR
Flags variables and USMs where performance has deteriorated relative to the reference, using a configurable threshold
Stores all results in Apache Arrow / Parquet format and exports them as CSV files and diagnostic plots

Installation

You can install the development version of SticsREval from GitHub:

# install.packages("devtools")
devtools::install_github("SticsRPacks/SticsREval")

Or using the pak package:

# install.packages("pak")
pak::pak("SticsRPacks/SticsREval")

Reproducible environment with `renv`

SticsREval uses renv to ensure reproducible package dependencies. The renv.lock file records the exact versions of all dependencies.

To restore the project environment locally:

# install.packages("renv")
renv::restore()

This will install all required packages at the versions specified in renv.lock. It is recommended to run this after cloning the repository and before running any code.

Dependencies

SticsREval relies on the following SticsRPacks packages:

Package	Role
`SticsRFiles`	Reading simulated and observed data
`SticsOnR`	Running STICS simulations (optional)
`CroPlotR`	Computing statistical criteria and generating plots

Workflow

SticsREval evaluates a candidate version of STICS by running it and comparing its outputs against two sources:

Field observations — to assess absolute model performance
Reference version statistics — to detect regressions relative to a known-good version

  STICS candidate version          Reference version stats
  (stics_exe + workspace)          (reference_workspace)
           │                                │
           ▼                                │
     make_config()   ◄──────────────────────┘
           │
           ▼
      evaluate()     ← runs simulations + compares vs obs & reference → saves .parquet files
           │
           ├──► export_stats_to_csv()   ← export statistics + deteriorated USMs to CSV
           └──► gen_plots()             ← comparison plots + scatter plots of regressions

Main Functions

`make_config()`

Creates and validates the configuration object that is passed to all other functions. All parameters for simulation, evaluation, and export are defined here.

library(SticsREval)

config <- make_config(
  stics_exe           = "/path/to/stics",
  workspace           = "workspace/",
  metadata_file       = "metadata.csv",
  run_simulations     = TRUE,      # set to FALSE to skip simulations
  verbose             = 1,
  parallel            = FALSE,
  cores               = NA,
  reference_workspace = NULL,      # path to reference version stats for regression detection
  percentage          = 5,         # threshold (%) to flag deteriorated variables
  eval_workspace      = "eval_workspace/",
  init_workspace      = TRUE,
  output_dir          = "outputs/",
  force               = TRUE,
  species             = NULL
  usms                = NULL,
  var2exclude         = NULL
)

Argument	Description
`stics_exe`	Path to the STICS executable
`workspace`	Path to the simulation input directory
`metadata_file`	Path to the metadata CSV file describing simulations
`run_simulations`	Whether to run STICS simulations (default: `TRUE`)
`verbose`	Logging verbosity level (default: `1`)
`parallel`	Enable parallel execution (default: `FALSE`)
`cores`	Number of cores for parallel execution (`NA` = auto)
`reference_workspace`	Path to the reference version simulation statistics, used to detect regressions
`percentage`	Threshold (%) above which a variable is flagged as deteriorated vs. the reference (default: `5`)
`eval_workspace`	Path to the evaluation workspace
`init_workspace`	Whether to initialize the workspace (default: `TRUE`)
`output_dir`	Output directory for exports
`force`	Whether to overwrite an existing non-empty evaluation workspace without prompting. (default: `FALSE`)
`species`	Optional list of species to evaluate. If NULL, all available species are evaluated. (default: `NULL`)
`usms`	Optional list of USMs to evaluate. If NULL, all available USMs are evaluated. (default: `NULL`)
`var2exclude`	Optional list of variables to exclude from evaluation. If NULL, all available variables are evaluated. (default: `NULL`)

`evaluate()`

The core function of the package. It orchestrates the full evaluation workflow:

Runs STICS simulations for the candidate version (via SticsOnR)
Compares simulation outputs against field observations (absolute performance)
Compares simulation statistics against reference version statistics to detect regressions (when reference_workspace is provided)
Flags variables and USMs where performance has deteriorated beyond the percentage threshold
Saves all results as Arrow/Parquet files in the eval_workspace directory
Displays a summary of comparison results

Errors during evaluation are caught and logged gracefully without stopping the full process.

evaluate(config)

After running, the eval_workspace directory will contain Parquet files with simulated data, observed data, and evaluation statistics per species.

`export_stats_to_csv()`

Exports the evaluation statistics per species to CSV files. For each species, three files are produced when available:

File	Content
`Criteres_stats.csv`	Global statistical criteria (RMSE, nRMSE, bias, R², etc.)
`RMSE_per_usm.csv`	RMSE broken down per USM (simulation unit)
`Deteriorated_USM.csv`	List of USMs with deteriorated performance

export_stats_to_csv(config)

`gen_plots()`

Generates diagnostic plots for each species:

Comparison plots — observed vs. simulated for all variables, for the candidate version
Scatter plots — comparing the candidate version against the reference version statistics, highlighting variables that have deteriorated beyond the percentage threshold (only generated when reference_workspace is set in make_config() and regressions are detected)

gen_plots(config)

Complete Example

library(SticsREval)

# 1. Configure the evaluation
#    - stics_exe points to the candidate version of STICS to evaluate
#    - reference_workspace points to the statistics of the reference version
config <- make_config(
  stics_exe          = "/path/to/stics_candidate",
  workspace          = "workspace/",
  metadata_file      = "metadata.csv",
  run_simulations    = TRUE,
  reference_workspace = "reference_version_stats/",  # stats from the reference version
  percentage         = 5,                            # flag variables with >5% deterioration
  eval_workspace     = "eval_workspace/",
  output_dir         = "outputs/"
)

# 2. Run simulations + evaluate vs observations and reference version stats
evaluate(config)

# 3. Export statistics and deteriorated USMs to CSV
export_stats_to_csv(config)

# 4. Generate comparison and regression scatter plots
gen_plots(config)

Docker

A pre-built Docker image is available on the GitHub Container Registry, so you can run SticsREval without installing R or any dependencies locally.

Pull the image

docker pull ghcr.io/sticsrpacks/stics-r-eval:latest

Run an interactive R session

docker run --rm -it \
  -v /path/to/your/workspace:/workspace \
  ghcr.io/sticsrpacks/stics-r-eval:latest \
  R

Then inside R:

library(SticsREval)

config <- make_config(
  stics_exe     = "/path/to/stics",
  workspace     = "/workspace/",
  metadata_file = "/workspace/metadata.csv",
  output_dir    = "/workspace/outputs/"
)

evaluate(config)

Run a script non-interactively

docker run --rm \
  -v /path/to/your/workspace:/workspace \
  -v /path/to/your/script.R:/script.R \
  ghcr.io/sticsrpacks/stics-r-eval:latest \
  Rscript /script.R

Build the image locally

# Without a GitHub token
docker build -t stics-r-eval .

# With a GitHub token (needed to install private SticsRPacks dependencies)
docker build \
  --secret id=GITHUB_TOKEN,src=<(echo $GITHUB_PAT) \
  -t stics-r-eval .

Note: The image is based on rocker/r-ver:4 and uses renv to ensure reproducible package versions. Dependencies are restored from renv.lock at build time.

SticsRFiles — Read/write STICS input and output files
SticsOnR — Run STICS simulations from R
CroPlotR — Crop model evaluation statistics and plots

Citation

If you use SticsREval in your work, please cite it as follows:

citation("SticsREval")

Contributing

Contributions are welcome! Please open an issue or submit a pull request on GitHub.

Code of Conduct

Please note that this project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

SticsREval

Overview

Installation

Reproducible environment with `renv`

Dependencies

Workflow

Main Functions

`make_config()`

`evaluate()`

`export_stats_to_csv()`

`gen_plots()`

Complete Example

Docker

Pull the image

Run an interactive R session

Run a script non-interactively

Build the image locally

Citation

Contributing

Code of Conduct

Links

License

Citation

Developers

Dev status

SticsREval

Overview

Installation

Reproducible environment with renv

Dependencies

Workflow

Main Functions

make_config()

evaluate()

export_stats_to_csv()

gen_plots()

Complete Example

Docker

Pull the image

Run an interactive R session

Run a script non-interactively

Build the image locally

Related packages

Citation

Contributing

Code of Conduct

Links

License

Citation

Developers

Dev status

Reproducible environment with `renv`

`make_config()`

`evaluate()`

`export_stats_to_csv()`

`gen_plots()`