PCMM (Predictive Capability Maturity Model)
Theory: six elements, four levels
Overview
My manager asked how far this simulation can be trusted, and I couldn't answer well.
The PCMM is a tool for exactly that question. Proposed by Sandia National Laboratories around 2007, it splits the evidence for trusting a simulation into six elements and rates each from 0 to 3. What matters is not the score itself but making clear where you fall short of the level the decision requires.
The six elements
| Element | What it covers |
|---|---|
| Representation and geometric fidelity | How closely geometry and omitted parts match reality |
| Physics and material model fidelity | Basis for physics models and properties |
| Code verification | Whether the software solves its equations correctly |
| Solution verification | Estimates of discretisation and iteration error |
| Model validation | Comparison with experiment |
| Uncertainty quantification and sensitivity | Effect of input variability on results |
Each element is rated 0 (individual judgement only), 1 (informal assessment, internal review), 2 (some formal assessment, independent review) or 3 (formal, comprehensive assessment, external review). Levels rise with rigour and with the independence of reviewers.
A question born of the nuclear test ban
PCMM's background is the US halt to nuclear testing in the 1990s. Weapon performance and safety that could no longer be tested had to be assured by simulation, and a systematic way to show how far the calculations could be trusted became necessary. Oberkampf and colleagues at Sandia organised verification and validation (V&V) concepts to answer this, and PCMM is one result. The same thinking later carried into NASA's model credibility standard and ASME V&V 40 for medical devices.
Worked example: electronics-enclosure thermal model
A thermal CFD model predicting the hottest component temperature in an electronics enclosure was assessed:
| Element | Assessed | Routine design | Safety decision |
|---|---|---|---|
| Geometric fidelity | 2 | 1 | 2 |
| Physics/material models | 1 | 1 | 2 (gap) |
| Code verification | 1 | 1 | 2 (gap) |
| Solution verification | 2 | 1 | 2 |
| Validation | 1 | 1 | 2 (gap) |
| Uncertainty/sensitivity | 0 | 1 (gap) | 1 (gap) |
| Mean (for reference) | 1.17 | 1.00 | 1.83 |
The mean is 1.17, above the 1.00 needed for routine design. So that's enough, right?
That's exactly the reading to avoid. Careful geometry doesn't make up for never examining uncertainty. Elements can't compensate for one another, so you don't average them. Element by element, even routine design falls short: uncertainty and sensitivity is 0 against a required 1. For a safety decision, four elements fall short. PCMM's value is that it shows concretely what to do next.
Example 2: building evidence for a level
As solution-verification evidence, the hottest component temperature on three grids refined by 2 was 75.8, 72.6 and 71.8 °C (coarse to fine). Observed order is 2.00, the extrapolated value 71.53 °C and the fine-grid GCI 0.46% (±0.33 K) — 0.7% of the rise above the 25 °C ambient.
For validation, the prediction was compared with a prototype measurement of 70.1 °C (measurement uncertainty ±1.5 K, 2σ). The difference $E$ is 1.7 K. Combining numerical 0.17 K, measurement 0.75 K and input (power and property scatter) 0.8 K gives a validation uncertainty of 1.1 K; the difference is 1.5 times that. It doesn't greatly exceed the uncertainty band, but suggests a model error of perhaps 1–2 K.
Assessment workflow
- Write down the decision the simulation informs and the consequences of getting it wrong.
- Set the required level for each of the six elements from those consequences (they need not all be equal).
- Assess the current state with evidence; assign no level without it.
- Compare each element with its requirement and plan work for the gaps. Don't average.
- Document the assessment and have it independently reviewed.
“I just put 2 for everything”
In one department introducing PCMM for the first time, the submitted assessment rated nearly every element at 2. Asked for evidence, there was no grid refinement study and no record of comparison with experiment; the assessor had reasoned that standard methods were used. PCMM levels depend not on how standard the methods are but on the assessment actually done and the independence of who reviewed it. Linking evidence (grid comparisons, test comparisons, sensitivity results) to each rating prevents this kind of self-assessment inflation.
Common mistakes
Mistakes and fixes
| Mistake | Effect | Fix |
|---|---|---|
| Averaging scores | Missing elements hidden | Compare each element with its requirement |
| Rating without evidence | Inflated assessment | Link evidence |
| Assessing without requirements | Adequacy undecidable | Set requirements from consequences first |
| Using PCMM as pass/fail | Confused with accuracy | Treat it as maturity assessment |
| Never updating | Model changes unreflected | Revisit with each change |
I'd like to try the calculations that build evidence.
Try grid convergence with the Richardson extrapolation tool, and uncertainty and sensitivity with the uncertainty propagation calculator and Sobol sensitivity index tool. For procedures see GCI, solution verification and validation metrics; for medical devices, ASME V&V 40.
Related Topics
Experience the theory firsthand with the interactive simulator for this field
All Simulators