How is the optimal step size h determined for central differences?

The total error of first-order central differences is the sum of truncation error O(h²) and round-off error O(ε/h). Minimizing gives h_opt = (3ε/|f'''|)^(1/3) ≈ ε^(1/3) ≈ 6×10⁻⁶ for float64 (ε ≈ 2.22e-16). The minimum achievable error is approximately ε^(2/3) ≈ 10⁻¹⁰–10⁻¹¹. On the log-log plot, this optimal point appears as the bottom of the V-shaped curve where truncation and round-off contributions are equal.

When should I use the 4-point O(h⁴) central difference scheme?

The 4-point central scheme [-f(x+2h)+8f(x+h)-8f(x-h)+f(x-2h)]/(12h) with O(h⁴) truncation is best for high-accuracy sensitivity analysis, Jacobian computation, or when function evaluations are expensive. However, its larger stencil coefficient makes it more susceptible to round-off, shifting the optimal h to around 10⁻⁴. For routine CAE Jacobian computation, central differences at h = 10⁻⁵ typically achieve sufficient accuracy.

How much more accurate is central differencing than forward differencing?

At the same step size h = 10⁻³, forward differences (O(h¹)) yield error ≈ 10⁻³, while central differences (O(h²)) yield error ≈ 10⁻⁶ — about 1,000 times more accurate. Central differences require two function evaluations vs. one for forward differences. In CFD iterative solvers where computational cost is critical, forward differences are common; for post-processing sensitivity analysis, central differences are preferred.

Why does making h smaller eventually increase the numerical error?

When h becomes very small (e.g., h < 10⁻¹²), f(x+h) and f(x) become nearly equal in floating-point representation. Subtracting nearly equal numbers causes catastrophic cancellation, losing significant digits. The resulting round-off error grows as O(ε/h). This is visible on the log-log plot as an upward-sloping line for small h. The fixed machine epsilon ε = 2.22e-16 for IEEE 754 double precision sets this lower limit.

Truncation Error & Order of Accuracy Visualization Tool

Function Settings

Function f(x)

Evaluation point x₀

-33

Derivative order

Difference Schemes

Forward difference O(h¹) Backward difference O(h¹) Central difference O(h²) 4-point central O(h⁴)

Formula

Forward: (f(x+h)-f(x))/h
Backward: (f(x)-f(x-h))/h
Central: (f(x+h)-f(x-h))/(2h)
4-point: (-f(x+2h)+8f(x+h)-8f(x-h)+f(x-2h))/(12h)

Round-off Error Constant

Machine ε ≈ 2.22e-16

float64 fixed value

|Error| vs Step Size h (log-log) — Truncation Error vs Rounding Error Balance

Results

—

Analytical derivative value

—

Optimal h (central difference)

—

Best achieved accuracy

2.22e-16

Rounding Error Floor ε

Stencil Visualization (f(x) neighborhood)

Error decomposition: Truncation vs Rounding (central difference)

What is Truncation Error?

🙋

What exactly is "truncation error" in numerical methods? I hear it's a big deal in simulations.

🎓

Basically, it's the error you make when you chop off an infinite mathematical series to make it computable. In practice, we can't sum an infinite number of terms, so we stop at a certain point, or "truncate" it. For instance, when approximating the derivative of a function like sin(x), the exact formula involves a limit as a step size goes to zero, but we have to use a finite step. Try changing the "Derivative order" slider above from 1 to 2; you'll see we use different finite formulas, each with its own truncation error.

🙋

Wait, really? So the error depends on the formula I pick. What does "order of accuracy" mean then?

🎓

Great question! The order tells you how fast the error shrinks as you make your calculation more precise. A higher order means the error drops much faster. In the simulator, if you set `f(x) = x^3` and look at the first derivative, the error for a 1st-order method scales with the step size $h$, but a 2nd-order method scales with $h^2$. Halve the step size, and a 2nd-order method cuts its error to a quarter! You can see this visually as the slope of the error line on the log-log plot.

🙋

That makes sense. So why wouldn't I always use the highest order method available? Is there a catch?

🎓

Exactly the right instinct! The catch is often complexity and computational cost. Higher-order formulas usually require more function evaluations or data points. For a real-world example, in a massive CAE fluid dynamics simulation, using a very high-order scheme everywhere might be overkill and slow things down tremendously. Engineers make a trade-off. Try the "Custom function" box with something like `Math.exp(x)*Math.sin(x)`. You'll see that while higher order is more accurate for large steps, all methods converge when the step is tiny enough—but computing at tiny steps is also expensive!

Physical Model & Key Equations

The core idea is approximating a derivative, which is fundamentally a limit, using a finite difference. The forward difference method is a common first-order approach.

$$f'(x) \approx \frac{f(x+h) - f(x)}{h}+ \mathcal{O}(h)$$

Here, $f'(x)$ is the true derivative, $h$ is the finite step size, and $\mathcal{O}(h)$ is the truncation error term, meaning the error is proportional to $h$ (first-order accurate).

A more accurate central difference method uses points on both sides. This "cancels out" more of the error, leading to second-order accuracy.

$$f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}+ \mathcal{O}(h^2)$$

The error term $\mathcal{O}(h^2)$ shows second-order accuracy. If you decrease $h$ by a factor of 10, the error decreases by a factor of 100. This is the power of higher-order methods.

Real-World Applications

Computational Fluid Dynamics (CFD): Simulating airflow over a wing or weather patterns requires solving partial differential equations (PDEs) for fluid motion. The choice of discretization scheme (e.g., 1st vs. 2nd order) directly impacts the accuracy of lift/drag predictions and the computational time. A lower-order method might be used for initial, rapid prototyping.

Finite Element Analysis (FEA): When calculating stress in a bridge or a car chassis, FEA software approximates derivatives of displacement fields. The order of the element shape functions determines the truncation error. Higher-order elements can capture complex stress gradients with fewer, larger elements, saving memory.

Financial Engineering: Pricing complex financial derivatives often involves solving the Black-Scholes PDE numerically. The order of accuracy in the numerical scheme affects the precision of the calculated option price, which can translate to significant monetary value in high-frequency trading.

Computer Graphics & Animation: Simulating realistic cloth, hair, or water surfaces uses numerical integration of physical laws. A game might use a fast, low-order method for real-time animation, while a movie studio uses a high-order method for offline rendering to achieve photorealistic, error-free fluid simulations.

Common Misconceptions and Points to Note

There are several important points to keep in mind when mastering this tool, especially with practical applications in view. First is the point that "the optimal h changes depending on the function and the point of differentiation." While the simulator's default function often shows an optimal h around $10^{-8}$, if the function's value is extremely large (e.g., $f(x)=10^{10}x$), the rounding error term becomes larger, so the optimal h becomes larger. Conversely, if the function's change is very gradual (close to $f(x)=constant$), the truncation error becomes smaller, allowing for a smaller h without issue. In short, blindly applying an optimal h found for one function to other cases is risky.

Next is the point that "higher-order difference schemes are not always the best." Four-point or six-point differences do indeed have a higher order of truncation error and are theoretically superior. However, these schemes require function values at more points, increasing computational cost. Furthermore, as the number of function evaluation points increases, the accompanying accumulation of rounding error cannot be ignored. Especially when the function itself contains noise (e.g., experimental data) or near discontinuities, oscillations in higher-order schemes can sometimes increase error. In practice, it's crucial to choose a scheme based on a comprehensive judgment of required accuracy, computational cost, and the nature of the function.

Finally, there is the fundamental question: "What do you do in real-world problems where the theoretical value is unknown?" In fields like CAE sensitivity analysis, the function you want to differentiate is often a black box (inside a solver). In such cases, the concept of the "error V-curve" learned with this tool becomes useful. You calculate the numerical derivative for several different h values (e.g., $10^{-4}, 10^{-6}, 10^{-8}$) and observe how the result changes as h becomes smaller. If the result doesn't change monotonically and starts oscillating at a certain point, it's a sign that rounding error influence has become dominant. Adopting the h value just before that as the "practical optimum" is an empirical approach that is vital in real-world settings.