ASME V&V 20 Validation Metrics

Good question. It's true that many people just look at "the percentage difference between the simulation and experimental values." But that alone is insufficient. Why? Because both experimental data and simulations have uncertainties.

For example, let's say the stress measured in an experiment is 100 MPa. The simulation result is 105 MPa. Is a 5% error unacceptable? But what if the experimental measurement accuracy is $\pm8$ MPa? What if the simulation's mesh dependency is $\pm3$ MPa? That 5 MPa difference might be within the range of uncertainty, meaning it could actually be considered "acceptable."

Exactly. ASME V&V 20 (Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer) is precisely the standardized framework for that judgment. Validation metrics are indicators that quantitatively evaluate the agreement between simulation results and experimental data, taking into account the uncertainties of both.

The starting point is the Comparison Error E. It's simple:

Since it's a signed difference, if the simulation is larger than the experiment, $E > 0$; if smaller, $E < 0$. If you want a "relative error," you can use $E/D$, but in the basic ASME V&V 20 framework, the absolute value-based $E$ is often used.

Yes. With just $E$, you can't tell "whether this difference is significant." That's why we next calculate the validation uncertainty $u_\text{val}$ and compare it with $E$.

ASME V&V 20 broadly categorizes uncertainties into three categories:

• $u_\text{num}$ (Numerical uncertainty): Uncertainty arising from discretization, such as mesh dependency, time step dependency, iterative convergence residuals. Often evaluated using GCI (Grid Convergence Index).
• $u_\text{input}$ (Input parameter uncertainty): Measurement accuracy or variability of material properties, boundary conditions, initial conditions. For example, if Young's modulus is $210 \pm 5$ GPa, that $\pm5$ GPa is the input uncertainty.
• $u_\text{exp}$ (Experimental uncertainty): Sensor accuracy, measurement location deviation, environmental fluctuations, etc. — uncertainty on the experimental data side.

If these can be assumed to be mutually independent, they are combined using RSS (Root Sum of Squares):

Exactly, it's the same concept as GUM (Guide to the Expression of Uncertainty in Measurement). However, there are two points to note. First, if the uncertainty sources are correlated, correlation terms need to be added. Second, the uncertainty here is generally treated as standard uncertainty ($k=1$, approximately 68% confidence interval), but sometimes it's treated with a 95% confidence interval ($k=2$), so you must always specify which one you are using in your report.

The judgment is simple. The basic idea of ASME V&V 20 is this:

• If $|E| \leq u_\text{val}$: The comparison error is within the range of uncertainty. Under this condition, it cannot be distinguished whether the difference between simulation and experiment is due to model error or uncertainty. Validation can be judged as successful.
• If $|E| > u_\text{val}$: There is a difference that cannot be explained by uncertainty. That means a model form error exists. Model improvement is needed.

Sharp observation! That's exactly right, and it's a key point in the V&V 20 philosophy. $|E| \leq u_\text{val}$ is not proof that "the simulation is correct"; it means "with the current level of knowledge (the magnitude of uncertainty), the existence of model error cannot be detected".

Therefore, the quality of validation is determined by the smallness of $u_\text{val}$. To reduce uncertainty, constant effort is required to use more precise experiments, finer meshes, and more accurate material properties.

Good point. The method used in such cases is the Area Validation Metric. Proposed by Ferson et al. (2008), it compares the cumulative distribution functions (CDFs) of the simulation results and experimental data.

$d_\text{area}$ takes a value greater than or equal to 0, and if the two CDFs match perfectly, $d_\text{area} = 0$. The advantages of this metric are:

• It reflects the shape of the distribution (not just the mean, but also spread and skewness).
• Multiple response quantities can be summarized into a single number.
• It's easy to set a threshold (e.g., pass if $d_\text{area} < 0.1$).

For example, when comparing acceleration time histories in a vehicle crash test, you can treat values at each time point as samples, create a CDF, and evaluate them collectively using the area metric.

The method to address that is the u-pooling method (Ferson & Oberkampf, 2009). The idea is this:

1. For each validation point $i$, construct the CDF $F_{D_i}$ of the experimental data $D_i$.
2. Determine where the simulation value $S_i$ lies on that CDF: $u_i = F_{D_i}(S_i)$.
3. If the model were perfect, $u_i$ should follow a uniform distribution on $[0, 1]$.
4. Compare the CDF of the collected $\{u_1, u_2, \ldots, u_n\}$ with the CDF of the standard uniform distribution $U[0,1]$.

If this $u_i$ deviates significantly from a uniform distribution, it's evidence of a systematic bias in the model. For example, if $u_i$ is consistently biased towards 0, it indicates "the simulation is systematically underestimating"; if biased towards 1, it indicates "systematic overestimation."

In practice, statistical tests like the Kolmogorov-Smirnov test (KS test) or Anderson-Darling test are often used to test the degree of deviation from the uniform distribution.

For $u_\text{num}$, the most widely used method is Richardson extrapolation + GCI. Let's outline the procedure:

1. Solve the same problem on three or more mesh densities (coarse, medium, fine). Let the representative mesh sizes be $h_1 > h_2 > h_3$.
2. Set the refinement ratio $r = h_1 / h_2$ (typically $r \approx 1.3\sim2.0$).
3. Determine the apparent order of convergence $p$:

Here, $f_1, f_2, f_3$ are the solution values on the coarse, medium, and fine meshes, respectively. Then calculate the GCI (Grid Convergence Index):

Exactly, the sensitivity coefficient method becomes computationally expensive with many parameters. That's where the Monte Carlo method comes in. The procedure is:

1. Set probability distributions (normal, uniform, etc.) for each input parameter.
2. Generate $N$ sample sets (e.g., $N = 100 \sim 10{,}000$) using methods like Latin Hypercube Sampling (LHS).
3. Run the simulation for each sample to obtain the distribution of the response.
4. The standard deviation of the response becomes $u_\text{input}$.

However, running $N=1000$ simulations is not realistic for CFD where a single simulation takes hours. Therefore, a practical approach is to insert a surrogate model (response surface method, Kriging, PCE: Polynomial Chaos Expansion) to reduce computational cost.

That's where uncertainty contribution decomposition (Uncertainty Budget) becomes important. Look at the ratio of each term in $u_\text{val}^2 = u_\text{num}^2 + u_\text{input}^2 + u_\text{exp}^2$.

For example, in an actual project where $u_\text{val} = 12.5$ MPa:

In this example, $u_\text{input}$ is dominant (77.4%), so refining the mesh would hardly reduce the overall uncertainty. You can see that improving the accuracy of material property values is the most cost-effective. Being able to make such judgments is a strength of the V&V 20 framework.

That's a common beginner mistake. First, you must create a validation plan. ASME V&V 20 recommends using a "PIRT (Phenomena Identification and Ranking Table)" and proceeding in the following order:

1. Decide on the SRQ (System Response Quantity) for validation: What will you compare? Maximum stress? Temperature distribution? Flow velocity profile? Vague "does the result match?" is not enough. Be specific about the response quantity.
2. Set the required accuracy criteria: Determine $\delta_\text{req}$ (allowable error). Without this, you cannot judge "pass."
3. Develop an experimental plan: Over what parameter range, how many measurements? Choose measurement methods that can sufficiently reduce uncertainty.
4. Complete code verification (Verification) first: Complete code verification and solution verification before validation. Validation is meaningless if there are numerical bugs.