Physicality

Locality

Summary

Performance in respecting locality, by measuring deviations from a neglible interactions between acetone and distance atoms.

Metrics

1. Maximum difference in force due to “ghost atoms”

Forces on an isolated acetone molecule are calculated, and the forces on the same atoms are calculated after 20 Ne atoms are placed in a 60 Å cubic box, at least 40 Å from the acetone’s centre of mass. The magnitude of the maximum difference in force is reported.

2. Mean difference in force due to a distance hydrogen

Forces on an isolated acetone molecule are calculated, and the forces on the same atoms are calculated after a single hydrogen atom is placed between 20 and 50 Å from the acetone’s centre of mass. This is repeated for 30 different random placements of the hydrogen atom, the mean force difference on the acetone atoms is calculared.

3. Standard deviation in force due to a distance hydrogen

Same as (2), but the standard deviation of the force difference on the acetone atoms is calculated.

Benchmark speed

Fast: tests are likely to take less than a minute to run on CPU.

Data availability

None required.

Extensivity

Summary

Performance in respecting extensivity, by measuring differences in energy between isolated systems, and the same systems combined, but significantly separated.

Metrics

1. Absolute energy difference between isolated and combined slabs

The energy of two isolated slabs is calculated, and the energy of the combined system, with the two slabs separated by 100 Å is calculated. The absolute energy difference between the sum of the isolated slabs and that of the combined system is calculated.

Benchmark speed

Fast: tests are likely to take less than a minute to run on CPU.

Data availability

None required.

Diatomics

Summary

This benchmark probes the short- to medium-range behaviour of every homonuclear and heteronuclear diatomic pair in the periodic table. Each MLIP is evaluated on a 100-point linear distance grid spanning 0.18-6.0 Å and the resulting energies and projected forces are analysed for unphysical oscillations.

Metrics

1. Force flips

   Average number of times the projected bond force changes sign. Forces are projected onto the bond axis and values below \(10^{-2}\) eV/Å are rounded to zero to avoid counting noise-induced flips. A smooth curve should switch from attraction to repulsion only once at the minimum.

2. Energy minima

   Mean count of distinct minima in the energy-distance profile. Local minima are found from the second derivative, where a physical diatomic should show a single minimum.

3. Energy inflections

   Mean number of inflection points obtained from the second derivative of the energy curve. Inflections are flagged when the second derivative changes sign with a tolerance of 0.5 eV/Ų to avoid counting noise-induced inflections. A physical diatomic curve should show one inflection point.

4. \(\rho(E, \text{repulsion})\)

   Spearman correlation between atomic separation and energy on the repulsive side of the well (bond lengths ≥ the equilibrium spacing). A perfect diatomic curve should show a strong negative correlation, so a value of -1, indicating that as atoms get further apart, the energy decreases.

5. \(\rho(E, \text{attraction})\)

   Spearman correlation between distance and energy on the attractive side (bond lengths shorter than the equilibrium spacing). A perfect diatomic curve should show a strong positive correlation, so a value of +1, indicating that as atoms get closer together, the energy increases.

Benchmark speed

Slow: expected to take hours to run on GPU, or around one day for slower MLIPs.

Data availability

None required; diatomics are generated in ASE.

Oxidation States

Summary

Examines the model’s ability to capture different oxidation states of Fe in aqueous solution [1, 2]. Two systems containing Fe 2Cl (Fe+2 state) and Fe 3Cl (Fe+3 state) in water are simulated at 300K for 20ps with NVT MD. The solvation cell is expected to be tighter for iron in the Fe+3 state. This effect, if an MLIP can correctly capture the Fe oxidation state, appears as a split on the Fe-O RDF peaks. This test examines whether a split appears between the Fe-O RDF peaks of the two systems. Additionally, the benchmark examines whether the peaks fall into the expected experimental range [1].

[1] Kocer, Emir, et al. “Machine learning potentials for redox chemistry in solution.” arXiv preprint arXiv:2410.03299 (2024). [2] Batatia, Ilyes, et al. “MACE-POLAR-1: A Polarisable Electrostatic Foundation Model for Molecular Chemistry.” arXiv preprint arXiv:2602.19411 (2026).

Metrics

1. Fe-O RDF Peak Split

The similarity of the aqueous Fe 2Cl and Fe 3Cl system RDFs is examined. If a split is present the score is +1 and in the case there is no clear split the score is 0. This metric determines whether a model can capture the different oxidation states of Fe and is therefore weighted as 5x more important than the two following metrics.

2. Fe +2 Peak Experimental Ref Deviation

Deviation of the Fe 2Cl system’s RDF peak position from the experimental range.

3. Fe +3 Peak Experimental Ref Deviation

Deviation of the Fe 3Cl system’s RDF peak position from the experimental range.

Benchmark speed

Very slow: expected to take hours to run on GPU, or around one day for slower MLIPs.

Data availability

Starting configurations for the MD are available on S3 bucket. Experimental reference ranges for the RDF peaks were taken from [1].

[1] Kocer, Emir, et al. “Machine learning potentials for redox chemistry in solution.” arXiv preprint arXiv:2410.03299 (2024).

Compression

Summary

This benchmark is a many-body analogue to the diatomics benchmark. Single element crystal structures are generated using common prototypes, namely (sc, bcc, fcc, hcp and diamond). Further structures are randomly generated, subject to minimum distance constraints, using pyxtal. The final structures are then isotropically scaled such that min(d_ij/(r_i + r_j)) = 1.0, where d_ij is the distance between atoms i and j, and r_i and r_j are the covalent radii of the respective atoms. A linear grid of scale factors containing 100 points is generated from 0.25 to 3.0. The structures are then scaled (scale factor applied to all cell vectors) and the energy and pressure are calculated. The resulting energies and pressures are analysed for unphysical oscillations and “holes”. All structures were generated by running ml_peg/calcs/physicality/compression/calc_compression.py directly.

Metrics

1. Pressure sign flips and Big Pressure sign flips

   Analogous to force flips in the diatomics benchmark, this metric counts the number of times the isotropic pressure changes sign. Pressure sign flips uses a tolerance of 0.001 GPa whilst Big Pressure sign flips uses a tolerance of 1 GPa, to avoid counting noise-induced flips. Whilst it is plausible that some, very contrived, configurations should show multiple sign flips in the pressure curve almost all materials should show 1.

2. Energy minima and Deep Energy minima

   Mean count of distinct minima in the energy-scale factor profile. Local minima are found from the second derivative using tolerance of 1meV/atom and 100 meV/atom for Energy minima and Deep Energy minima respectively.

3. Holes

   A “hole” is defined as a point on the energy-scale factor curve where the energy is less than the energy at the largest scale factor (largest volume) by more than 100 eV/atom. This is a crude and “cautious” definition, with the idea of flagging catastrophic model failures only.

4. \(\rho(-E, V_{\text{small}})\)

   Spearman correlation between scale factor and minus energy for \(scale < scale(E_{\min})\) (i.e. the compressed side of the curve). A perfect curve should show a strong positive correlation, so a value of +1, indicating that as the structure is compressed, the energy increases. As above, this assumes a single minimum in the energy-scale factor curve, which is the case for almost all materials. This metric will be polluted in models containing holes.

5. \(\rho(E, V_{\text{large}})\)

   Spearman correlation between scale factor and energy for \(scale > scale(E_{\min})\) (i.e. the expanded side of the curve). A perfect curve should show a strong positive correlation, so a value of +1, indicating that as the structure is expanded, the energy increases. As above, this assumes a single minimum in the energy-scale factor curve, which is the case for almost all materials. This metric will be polluted in models containing holes.

6. Failed fraction

   Fraction of scale factors per structure at which the energy or pressure evaluation failed (returned NaN), averaged over all structures. Failures typically occur under strong compression, e.g. from neighbour-count caps or out-of-memory errors. Failed points are excluded from all other metrics, so this metric records the portion of each curve that was not assessed.

Benchmark speed

Very slow: expected to take hours to run on GPU, or around one day for slower MLIPs.

Data availability

None required; prototype structures are generated in ASE and random structures are generated using pyxtal.

Water Slab Dipoles

Summary

Distribution of dipole of water slab, checking for width of distribution and structures with dielectric breakdown.

Metrics

1. Standard Deviation of Dipole Distribution

For a number of samples from an MD simulation, the total dipole is calculated. Compare to a reference of a LR model trained on revPBE-D3.

2. Number of structures with dielectric breakdown

Estimate band gap based on dipole, count structures where band gap disappears.

Benchmark speed

Very slow: requires around 500 ps of MD of a 40 Å slab to obtain a converged distribution, taking around one day on one GPU.

Data availability

https://arxiv.org/html/2603.04228v1

Water-Cl2 cluster relaxation

Summary

This test is mainly for long-range models to probe the stability of a Cl2 molecule when two solvated Cl- ions are present far outside the receptive field of the Cl2 molecule, and the total charge of the system being -2.

Geometry relaxation may lead to the Cl2 molecule dissociating (incorrect behaviour) or staying stable (correct behaviour).

Metrics

1. Dissociation of the Cl-Cl bond based on the interatomic distance.

Benchmark speed

Fast: requires up to 1000 optimiser steps for a 400-atom system, taking several GPU-minutes to complete.

Data availability

The initial structures were generated for MACE-POLAR-1 https://arxiv.org/abs/2602.19411

Jacobian Symmetry

Summary

Performance in predicting conservative forces, by measuring the antisymmetric fraction of the finite-difference force Jacobian [1].

Metrics

1. Mean lambda

For ten diverse structures (eight molecules and two periodic carbon systems: diamond and a 2D sheet), the Jacobian of predicted forces with respect to atomic positions is built by central finite differences, perturbing each degree of freedom by \(\pm 10^{-3}\) Å. For a conservative force field, where forces are the gradient of an energy, this Jacobian must be symmetric. Lambda is the fraction of the Jacobian’s Frobenius norm that is antisymmetric, \(\lambda = \lVert J_{\text{anti}} \rVert / \lVert J \rVert\) where \(J_{\text{anti}} = (J - J^T)/2\). Lambda is 0 for perfectly conservative forces, and increases towards 1 for models predicting more decoupled, non-conservative forces. Lambda is calculated for each structure, and the mean across all structures is reported.

2. Max lambda

Same as (1), but the maximum (worst-case) lambda across all structures is reported.

Benchmark speed

Fast: tests are likely to take a few minutes to run on CPU.

Data availability

None required; structures are generated in ASE.

[1] Bigi, Filippo, Marcel Langer, and Michele Ceriotti. “The dark side of the forces: assessing non-conservative force models for atomistic machine learning.” arXiv preprint arXiv:2412.11569 (2024).

Rotational Symmetry

Summary

Performance in respecting rotational symmetry. Under a rigid rotation of a structure the energy is invariant and the forces are equivariant (they rotate with the structure), so any deviation reflects the model’s implementation rather than physics. Architectures that predict forces directly, rather than as the gradient of the energy, are not equivariant by construction, and this test measures how far they deviate.

Ten diverse structures are evaluated: eight molecules (H2O, CH4, NH3, C2H4, C2H2, SO2, CH3OH and C6H6) and two periodic systems (diamond and graphene), matching the translational symmetry test. Diamond is rattled with a fixed seed so that the atoms carry forces of a realistic magnitude: a perfect crystal’s true forces vanish by symmetry, so any violation that scales with the size of the forces is understated there. Each structure is evaluated before and after each of 114 rigid rotations about the origin, forming a single cumulative walk through orientation space: 14 steps increasing from 1 to 40°, followed by 100 uniformly random steps, all generated with a fixed seed so that every model sees the identical walk. For the periodic structures the cell is rotated together with the positions, preserving fractional coordinates: a rotation of the positions alone is not a symmetry of a periodic crystal. Random rotations sample orientation space rather than trusting one hand-picked axis and angle, and generically avoid the special rotations (90 or 180° about a coordinate axis) that permute or negate coordinates exactly in floating point and can cancel real violations out of the comparison. Two uniformly random orientations are almost never within 40° of each other, so the first 14 steps supply successive pairs at small and intermediate relative angles: a smooth model must fail gently at small relative angles, and an abrupt small-angle violation signals an implementation artifact rather than learned approximate symmetry.

Note

Forces are compared after rotating those of each orientation back into a common frame, where a perfect model gives identical forces for every orientation. The back-rotated forces are computed and stored during the calculation, so that the comparison is unaffected by the precision the output files are written at.

Metrics

Successive orientations (the unrotated structure, then each rotation in turn) are compared pairwise, giving 114 orientation pairs per structure; each metric is computed over all pairs of all ten structures. Comparing each orientation with the next, rather than every orientation with a fixed reference, means no single orientation is privileged. A model that fails to complete the calculation for any structure is not scored on the remainder: the metrics are left blank. Per-model plots resolve the individual pairs, showing each structure’s violation against the relative rotation angle within the pair.

1. Mean ΔE

Mean absolute change in energy per atom between successive orientations.

2. Max ΔE

Worst-case absolute change in energy per atom between successive orientations.

3. Mean ΔF

Mean of the per-pair force MAE: the mean absolute difference between the two orientations’ force predictions, compared component-wise in a common frame, adapted from the force rotational equivariance metric of MLIP Arena [1].

4. Max ΔF

Worst-case per-pair force MAE, as above.

[1] Chiang, Y., et al. “MLIP Arena: Advancing Fairness and Transparency in Machine Learning Interatomic Potentials via an Open, Accessible Benchmark Platform.” arXiv preprint arXiv:2509.20630 (2025).

Benchmark speed

Fast: single-point evaluations only (10 structures evaluated in 115 orientations each), likely to take seconds to minutes per model on GPU, or minutes to tens of minutes on CPU.

Data availability

None required; all structures are generated in ASE.

Translational Symmetry

Summary

Performance in respecting translational invariance. Energy and forces depend only on the relative positions of atoms, so rigidly translating every atom in a structure must leave both unchanged, and any deviation reflects the model’s implementation rather than physics.

Ten diverse structures are evaluated: eight molecules (H2O, CH4, NH3, C2H4, C2H2, SO2, CH3OH and C6H6) and two periodic systems (diamond and graphene), matching the rotational symmetry test. Diamond is rattled with a fixed seed so that the atoms carry forces of a realistic magnitude: a perfect crystal’s true forces vanish by symmetry, so any violation that scales with the size of the forces is understated there. Each structure is evaluated before and after a rigid translation along a fixed, non-axis-aligned direction, at magnitudes of 1, 40 and 1000 Å. The direction is deliberately not axis-aligned: for a shift along a single axis the rounding error can be identical for every atom, cancelling out of the relative geometry and hiding the effect entirely.

Testing more than one magnitude distinguishes a model that breaks under any shift at all from one that degrades as atoms move further from the origin, which is the signature of finite precision coordinates.

Note

For periodic structures the effective displacement is bounded by the cell, since a rigid translation is wrapped back into it. The larger translations therefore probe diamond (a 3.567 Å cell) no more strongly than the 1 Å translation does. Graphene is built with 20 Å of vacuum, giving a 40 Å cell, so that the 40 Å translation is not wrapped away to under 1 Å.

Metrics

Each metric is reported separately for each translation magnitude (1, 40 and 1000 Å). A model only receives scores for a magnitude if it evaluated all ten structures successfully at it; any failed evaluation leaves that magnitude’s metrics blank.

1. Mean ΔE

Mean absolute change in energy per atom, across the ten structures.

2. Max ΔE

Worst-case absolute change in energy per atom, across the ten structures.

3. Mean ΔF

Each structure’s force change is the largest absolute change in any force component, over every atom and Cartesian direction. Note that this differs from the identically named metric of the rotational symmetry test, which reports a force MAE. The mean of that per-structure worst case, across the ten structures, is reported.

4. Max ΔF

Same as (3), but the maximum (worst-case) across all structures is reported.

Benchmark speed

Fast: tests are likely to take less than a minute to run on CPU.

Data availability

None required; all structures are generated in ASE.