KSB

Article · Scientific Computing ·

When Can You Trust a Faster Scientific Model?

A practical question in scientific computing is when a faster approximation can be trusted. This post outlines a validation framework using atomic-resolution STEM surrogate models as an example.

Scientific computing often involves a trade-off between fidelity and computational cost.

A high-fidelity model may represent the underlying physics in considerable detail, but it can be expensive to evaluate repeatedly. A simpler surrogate can reduce that cost, but only if its limitations are understood.

This post uses atomic-resolution scanning transmission electron microscopy (STEM) as an example. The central question is not whether a surrogate can produce an image quickly. It is whether the surrogate preserves the information needed for a particular scientific task.

A surrogate does not need to reproduce every detail of a reference calculation. It needs to preserve the quantities required for the intended use, within stated limits.

Why use a surrogate?

In STEM simulation, a focused electron probe is scanned across a specimen and the scattered signal is calculated at many probe positions. High-fidelity methods such as multislice simulation can represent electron scattering in substantial detail, but repeated calculations across structures or microscope parameters can become computationally expensive. Background on multislice STEM simulation and its computational cost is available in Ishizuka (2004), Dwyer (2010), and Ophus (2017).

A simpler image model can be useful when the objective is to screen many candidate structures, perform sensitivity studies, or guide an optimization before using a higher-fidelity calculation.

One simple approximation represents projected atomic features as Gaussian peaks:

I(x, y) = Σᵢ Aᵢ exp(-((x-xᵢ)² + (y-yᵢ)²) / (2σᵢ²))

Here, xᵢ and yᵢ describe a projected position, Aᵢ controls intensity, and σᵢ controls width.

This representation does not reproduce the full electron-scattering physics. Its usefulness therefore depends on what it preserves relative to the reference calculation.

Similar-looking images are not enough

Visual comparison is useful for an initial check, but it is not a sufficient validation criterion.

Two images can have a small pixel-wise error while differing in a feature that matters for the scientific analysis. The reverse can also occur: a global image metric may report a noticeable difference even though the quantity used in the downstream analysis is preserved.

Validation should therefore be tied to the intended use of the model.

For atomic-resolution images, several quantities may be relevant:

  • Peak position: are atomic features located in the same place?
  • Peak intensity: are relative and absolute intensities reproduced adequately?
  • Peak width: is the local spatial profile consistent with the reference?
  • Local contrast: are neighboring features distinguished correctly?
  • Ranking: are candidate structures ordered in the same way?
  • Decision consistency: would the surrogate lead to the same scientific choice as the reference calculation?

Not every application requires all of these quantities. The appropriate set depends on the question being asked.

Validation is better treated as a profile than a single score

A single metric rarely captures every relevant failure mode.

Mean squared error, correlation, structural similarity, peak localization error, and intensity error describe different aspects of agreement. Combining them into one scalar can obscure which part of the approximation is responsible for a disagreement.

A more transparent approach is to report a small validation profile. For example:

  • image-level error for overall agreement
  • peak-position error for localization
  • peak-intensity error for signal strength
  • peak-width error for spatial shape
  • local-contrast error for neighboring features
  • ranking agreement when the model is used to compare alternatives

The aim is not to maximize the number of metrics. Each metric should correspond to a scientific quantity or decision that matters in the intended application.

Reference calculations also have numerical uncertainty

The high-fidelity calculation is itself numerical.

Its result can depend on choices such as grid spacing, slice thickness, sampling, convergence settings, and other discretization parameters. A discrepancy between a surrogate and a reference calculation should therefore not automatically be interpreted as surrogate error.

Numerical refinement helps separate these effects.

If a reference calculation changes when its numerical settings are refined, that change provides an estimate of numerical uncertainty. Surrogate error can then be interpreted relative to that scale.

This distinction is important when the surrogate-reference difference is similar in magnitude to the numerical variation within the reference method itself.

Development and validation should remain separate

Repeatedly inspecting validation cases while adjusting a model can weaken their value as independent tests.

A practical solution is to separate cases used during development from cases reserved for later validation. The model, metrics, and acceptance criteria can then be defined before the reserved cases are examined.

This is similar in spirit to train/test separation in machine learning, but it also applies to analytical and physics-based surrogate models.

A simple workflow is:

  • develop the model using designated development cases
  • define the evaluation metrics
  • specify acceptance criteria
  • freeze the evaluation procedure
  • test against reserved cases

This reduces the risk of adapting the model to the specific examples that are later used to support its performance.

Software quality is part of computational evidence

Reproducibility depends on more than the mathematical model.

A computational result should make it possible to determine which structure, parameters, numerical grid, software version, and evaluation procedure produced it.

Useful engineering practices include:

  • explicit configuration objects
  • typed domain models where appropriate
  • automated tests
  • version control
  • reproducible environments
  • recorded experiment provenance
  • validation of numerical inputs
  • continuous integration

These practices do not establish scientific validity on their own. They make it easier to reproduce, inspect, and audit the calculations that support a scientific claim.

Use the surrogate only within a defined domain

A surrogate should be associated with a stated domain of validity.

That domain may depend on atomic environments, elements, specimen conditions, microscope parameters, or other variables. A model validated in one region should not automatically be assumed to work outside it.

For cases outside the validated domain, the appropriate response may be to return to the high-fidelity calculation rather than produce a confident surrogate estimate.

This leads to a practical workflow:

surrogate → check applicability → use result or fall back to high-fidelity simulation

The surrogate and the reference method then serve different roles. The surrogate reduces the number of expensive calculations, while the high-fidelity method remains available for uncertain or sensitive cases.

What should a validation claim contain?

A useful validation statement should specify at least five things:

  • Domain: where is the surrogate intended to be used?
  • Observables: which quantities must it preserve?
  • Tolerance: what level of disagreement is acceptable?
  • Reference: which calculation or measurement is used for comparison?
  • Failure conditions: when should the surrogate not be used?

For example, a claim might state that a surrogate preserves selected image observables within defined tolerances over a specified range of structures and simulation conditions.

That is more informative than stating that the surrogate is simply “accurate” or “fast.”

Conclusion

The main question in surrogate modelling is not whether a simplified model resembles a high-fidelity calculation in general.

The more useful question is whether it preserves the quantities needed for a specified task, over a defined domain, with errors interpreted relative to both scientific requirements and numerical uncertainty.

For atomic-resolution STEM, this means examining image features such as position, intensity, width, local contrast, and decision consistency rather than relying on visual similarity or a single global metric.

The same principle applies more broadly to reduced-order models, emulators, analytical approximations, and machine-learning surrogates: validation should be tied to intended use, numerical uncertainty, and explicit limits.

References