VICENA GYM · HYDROGEN

A benchmark and learning environment built from laboratory records

Hydrogen is the first suite: tasks on materials for fuel cells and electrolysers, each graded against measurements. Models compute figures of merit from raw instrument files, audit records, forecast held-out experiments and run a virtual laboratory within a budget.

The benchmark shown here is built on synthetic placeholder records, which stand in for real laboratory records. Three real records are shown below, and the full real dataset is available under licence.

80
placeholder records
4
research lines
334
benchmark tasks
681
training tasks
116
benchmark episodes

LEADERBOARD

Baselines and model runs, scored on the same tasks

Every row is scored on the short set: 69 tasks and 8 laboratory episodes sampled from the benchmark. Scores run from 0 to 1. A tier's score is the mean of its task types, and the overall score is the mean of the four tiers, so the many single-skill tasks do not outweigh the few predictive and agentic ones.

EntryOverallMeasureJudgePredictActType
Reference analysis and best recipe The programmatic reference solutions and the best recipe of each laboratory. It shows that every task is solvable; forecasts have no reference solution.– 1.001.00–1.00Upper bound
Gaussian-process regression and Bayesian optimisation Standard statistical tools on the two tiers they can attempt. A model adds value there only where it beats them.– ––0.390.44Classical baseline
No-skill baseline Empty answers, “insufficient” for every claim, the campaign mean for every forecast and random search in the laboratory.0.12 0.000.050.070.36Baseline
Laboratory's standard recipe Submitting the standard recipe without running an experiment.– –––0.01Baseline

Model runs by task type

Each model ran once, each task in a fresh session that held only that task's files. An entry names the agent software the model ran in. A dash marks a task type an entry did not attempt.

EntryForecastCase
Reference analysis and best recipe –1.00
GPT-6.1 Sol OpenAI · Vicena agent0.500.66
Gaussian-process regression and Bayesian optimisation 0.39–
No-skill baseline 0.070.23

REAL RECORDS

Three real laboratory records

The benchmark's tasks are modelled on working laboratory records. These three are iron–nitrogen–carbon oxygen catalysts made by one route, each with its synthesis, its test conditions and its raw instrument files. The values below are computed from the raw files by the benchmark's reference analysis.

Diffraction pattern of FeNC1, oxygen-reduction curves of FeNC1 and FeNC2, and oxygen-evolution curves of FeNC3 on three substrates
From the raw files: (a) the unnamed impurity in FeNC1 is silica; (b) FeNC1 and FeNC2 are the same recipe and differ by 60 mV; (c) on nickel foam the substrate itself is active.
SampleLigand : zincOxygen-reduction half-wave potentialOxygen-evolution overpotential at 10 mA/cm²
FeNC14 : 10.767 Vnot measured
FeNC24 : 10.830 Vnot measured
FeNC38 : 10.807 V527 mV

What an audit of the three records found

An impurity nobody named

One sample is labelled “with impurity”. Its diffraction pattern shows quartz and cristobalite: silica from the furnace ware, not the residual zinc or iron a reader would assume.

File names that point to another sample

Every micrograph in one sample's folder carries another sample's name. Which sample a file belongs to has to be established from the record.

A result without its control

On nickel foam the catalyst passes four times the current it passes on glassy carbon, but the foam is a catalyst itself and no bare-foam control was run.

Only results are published here. The records' files are supplied under licence.

These records come from Vicena's experimental data, which supplies records like them for evaluation and training, from laboratory archives or new experiments. See the experimental data

TASKS

Four tiers, from single skills to a whole investigation

The lower tiers are the skills the higher ones use, and every grader gives partial credit per item, so a long task has a reward for each step.

Measure

Compute a figure of merit from a raw instrument export by a stated definition, or read a diffraction pattern, a micrograph or a spectrum. Graded by tolerance against the reference analysis of the same file.

  • Quantify
  • Characterise

156 benchmark · 313 training

Judge

List what makes a record misleading, decide whether a claim is supported, refuted or undecidable, and propose the controls that would settle a conclusion. Graded against planted defects, verdicts computed from the data and rubrics of decisive controls.

  • Audit
  • Claims
  • Controls

129 benchmark · 291 training

Predict

Forecast the held-out outcomes of a campaign, each with an 80% interval. Graded against the measured values, for ranking and for intervals that cover the outcome without being wide.

  • Forecast

24 benchmark · 22 training

Act

Analyse a record's whole folder in a container, or run a virtual laboratory within a budget and submit the best recipe. Graded per figure, phase, finding and verdict, or by how close the recipe is to the best one.

  • Case

25 benchmark · 55 training

EXAMPLE TASKS

What a task looks like

Four practice tasks from one synthetic record of a nanofibre oxygen electrode. A model receives the prompt and the record's files, and answers with one JSON object.

Task

From the four symmetric-cell impedance files (attached), report the electrode's ASR at 600 °C in Ω cm² and the activation energy of its polarisation in eV between 550 and 700 °C. The laboratory did not record the electrode area; use 0.5 cm² per electrode. Check that each file is what its name says before you use it.

Ohmic resistance: the real impedance where the imaginary part changes sign at high frequency (interpolate linearly between the two neighbouring points), or the real part at the highest frequency if it never changes sign. Total resistance: the real impedance at the lowest measured frequency. Polarisation resistance: total minus ohmic. Area-specific resistance (ASR): polarisation resistance times the electrode area, divided by two for a symmetric cell. Activation energy: minus the slope of ln(1/ASR) against 1/T (T in kelvin), times the Boltzmann constant 8.617 × 10⁻⁵ eV/K.

Task text is in English.

Files

  • GDC6@PBSCF4 sym EIS 700C.csv
  • GDC6@PBSCF4 sym EIS 650C.csv
  • GDC6@PBSCF4 sym EIS 600C.csv
  • GDC6@PBSCF4 sym EIS 550C.csv

Answer format

{
 "asr_600_ohm_cm2": "number",
 "ea_ev": "number"
}

The reference answer scores 1.00.

LEARNING ENVIRONMENT

A virtual laboratory with a budget

Each research line has a laboratory. An agent chooses what to make and what to measure, pays for each step from a budget, and submits the recipe it believes is best. Every episode draws its own laboratory, and the best recipe moves with it, so a recipe carried over from another episode is only a starting point: the best one has to be found from the episode's own measurements.

  1. Action
    {"action":"synthesise", "recipe":{"gdc_fraction":0.5, "sinter_c":900}}
    Observation
    {"batch":"B1", "recipe":{"gdc_fraction":0.5, "coaxial":true, "calcine_c":700, "sinter_c":900, "flow_ml_h":0.6, "voltage_kv":20}, "humidity_pct":33, "mat":"uniform fibres", "budget_left":13}
  2. Action
    {"action":"measure", "batch":"B1", "technique":"sym_eis"}
    Observation
    {"batch":"B1", "technique":"sym_eis", "asr_ohm_cm2":{"550":0.9638, "600":0.3818, "650":0.1621, "700":0.086}, "budget_left":12.5}
  3. Action
    {"action":"synthesise", "recipe":{"gdc_fraction":0.45, "sinter_c":900, "flow_ml_h":0.4, "voltage_kv":25}}
    Observation
    {"batch":"B2", "recipe":{"gdc_fraction":0.45, "coaxial":true, "calcine_c":700, "sinter_c":900, "flow_ml_h":0.4, "voltage_kv":25}, "humidity_pct":42, "mat":"uniform fibres", "budget_left":11.5}
  4. Action
    {"action":"measure", "batch":"B2", "technique":"sym_eis"}
    Observation
    {"batch":"B2", "technique":"sym_eis", "asr_ohm_cm2":{"550":0.7301, "600":0.2787, "650":0.1301, "700":0.068}, "budget_left":11}
  5. Action
    {"action":"submit", "recipe":{"gdc_fraction":0.45, "sinter_c":900, "flow_ml_h":0.4, "voltage_kv":25}}
    Observation
    {"result":"submitted", "recipe":{"gdc_fraction":0.45, "coaxial":true, "calcine_c":700, "sinter_c":900, "flow_ml_h":0.4, "voltage_kv":25}}
    Reward 0.59
A short episode in the nanofibre laboratory. The reward is 1 for the best recipe in the design space and 0 for the laboratory's standard recipe.

How far simple strategies get

Mean reward on the benchmark's episodes. Random search and Bayesian optimisation stay well below 1 at these budgets: doing better takes knowledge of the chemistry.

  • Standard recipe
  • Random search
  • Bayesian optimisation
M–N–C single-atom oxygen catalysts
0.00
0.33
0.32
Core–shell nanofibre oxygen electrodes
0.09
0.14
0.34
Catalytic layers for direct-ammonia fuel cells
0.00
0.38
0.27
Sintering of proton-conducting electrolytes
0.02
0.22
0.24

Run it on your own machine

The environment comes with the benchmark package and runs locally. One command starts the laboratories, the tasks and the grader as a service on your machine, with a Gym-style loop: start an episode, send actions, read observations and a final reward.

DATASET

Eighty placeholder records across four research lines

A record is one complete experiment on one material: why it was run, how the material was made, the test conditions, and the instrument files. Historical records are kept as a working group keeps them, with gaps and slips; commissioned records are a designed campaign with the synthesis as run, replicates and controls.

Research lineHistoricalCommissionedBasis
M–N–C single-atom oxygen catalysts Fe, Co, Fe–Co, Co–Mn and phosphorus-doped catalysts from pyrolysed zinc imidazolate frameworks, tested for oxygen reduction and evolution in alkaline electrolyte.1420Inventoried records
Core–shell nanofibre oxygen electrodes Electrospun perovskite fibres with a doped-ceria shell, tested in protonic ceramic cells from 700 to 550 °C in fuel-cell and electrolysis modes.414Inventoried records
Catalytic layers for direct-ammonia fuel cells Ni–Fe ammonia-decomposition layers on the anode of protonic ceramic fuel cells, compared in hydrogen and ammonia with outlet-gas conversion and 100 h holds.68Planned project
Sintering of proton-conducting electrolytes Barium zirconate-cerate electrolytes sintered with NiO, ZnO or CuO, characterised by density, diffraction, microstructure and conductivity from 700 to 400 °C.68Planned project

These records are synthetic placeholders

They stand in for real laboratory records, which are not public. Models of the materials generate them, calibrated to real records where real records exist, to show the structure and difficulty of the real dataset. They are not measurements, they are not what Vicena supplies, and they must not be cited as results.

Three real records are shown here

The results in the section on real records come from three unpublished laboratory records. The full real dataset consists of physical experiments with the scientists' notes and raw files. It is licensed for training and evaluation and is not published.

METHOD

What keeps the scores honest

Keys never travel with tasks

Ground truth is stored apart from everything a model sees, and a test checks that no record or task contains it.

Definitions are stated

Every prompt gives the definition of the quantity it asks for, so a score measures the analysis and not a guess at conventions.

File names cannot be trusted

Records contain mislabelled, duplicated, empty and mis-scaled files, as real ones do. A figure computed from a file taken at its name's word is wrong.

Clean records exist

Most commissioned records have no defects, so a model that reports problems everywhere loses points.

A canary marks the benchmark

Benchmark files carry a unique string so that training corpora can exclude them and leaks can be detected.

VICENA-GYM-HYDROGEN-CANARY 7c1e5a2e-3f0b-4d57-9c44-b1a6d2f8e913

ACCESS

The package, the real dataset and custom environments

Ask for the benchmark package to run the tasks and the environment on your own machine, for a licence to the real dataset, or for an environment built from your own laboratory's records.

CITATION

If you use the benchmark, please cite it as below.

@misc{vicena_gym_hydrogen_2026,
  title  = {Vicena Gym: Hydrogen. A benchmark and learning environment
            from laboratory records of fuel-cell and electrolyser materials},
  author = {Vicena},
  year   = {2026},
  note   = {Version v1, synthetic records},
  url    = {https://vicena.ai/rl-environments}
}