OpenFOAM Parallel Computing
Theory: split, distribute and solve
Overview
Does OpenFOAM keep getting faster as I add cores?
Up to a point — beyond it, little speed-up and sometimes a slowdown. OpenFOAM splits the mesh into subdomains (decomposePar); each core solves its own, exchanging boundary-cell values with neighbours over MPI. More cores mean fewer cells each, but a larger share of boundary data, and the time spent waiting for all cores in global operations such as the linear solver's reductions (dot products) does not shrink. Below a certain number of cells per core, communication and waiting dominate computation. A guide is tens of thousands to around a hundred thousand cells per core, though it varies greatly with network (InfiniBand vs ordinary Ethernet) and solver (how the pressure equation is solved).
Amdahl's law and the boundary share
S is speed-up, f the serial fraction, N core count, E parallel efficiency and n cells per core. The boundary share is an upper estimate assuming all six faces of a cubic subdomain touch other subdomains.
Amdahl's 1967 observation
Amdahl's law on the limits of parallelism traces to an argument made by IBM engineer Gene Amdahl at a 1967 conference: however many processors you add, any non-parallel part caps the speed-up. In 1988 Gustafson countered that larger problems raise the parallel share, popularising the view that bigger problems put more cores to good use. Both hold for CFD: larger meshes use more cores efficiently.
Worked examples
Speed-up uses an Amdahl model with assumed 2% serial fraction, not an OpenFOAM benchmark. Boundary-cell share is an approximate upper scale. Measure optimal core count and cost on the actual system, including physical versus logical cores.
Example: a 4-million-cell case (2% serial fraction assumed)
| Cores | Cells per core | Speed-up (Amdahl) | Efficiency | Boundary share (estimate) |
|---|---|---|---|---|
| 8 | 500,000 | 7.0 | 88% | 8% |
| 32 | 125,000 | 19.8 | 62% | 12% |
| 128 | 31,250 | 36.2 | 28% | 19% |
| 512 | 7,812 | 45.6 | 9% | 30% |
Even a 2% serial fraction drives 512-core efficiency down to 9%, and communication time usually lowers it further. Balancing speed against core-hours (cost), 32–128 cores is the realistic range here.
My run on 64 cores was slower than on 32. What's happening?
Several suspects. First, did the run span nodes? If 32 cores fit in one machine and 64 needed two, inter-node communication slows everything, especially over ordinary Ethernet. Second, load balance: even with equal cell counts from scotch, very different boundary-face counts per subdomain make everyone wait for the slowest. decomposePar reports cells and boundary faces per subdomain — check them. Third, the pressure solver: GAMG (algebraic multigrid) adds global communication at coarse levels, so revisit settings such as nCellsInCoarsestLevel at high core counts. Finally, make sure you are not counting hyper-threading logical cores; running beyond physical cores does not speed things up.
Practical workflow
- Pick candidate core counts at tens of thousands to around a hundred thousand cells per core.
- Decompose with scotch in decomposeParDict and check cells and boundary faces per subdomain.
- Time a few dozen steps at several core counts to measure time per step.
- Choose the fastest count that keeps efficiency above roughly 50–70%.
- Join results with reconstructPar or open the decomposed case in ParaView.
“Post-process without reconstructing”
For large runs, reconstructing every time step with reconstructPar costs much time and disk. Reconstruct only the times you need with -time, or open the 'Decomposed Case' in ParaView and skip reconstruction. If function objects write forces and averages during the run, field data often never need reconstructing at all. Deciding up front which data are produced where, including post-processing, makes parallel work efficient.
Common mistakes
Mistakes and fixes
| Mistake | Effect | Fix |
|---|---|---|
| Too many cores | Communication slows the run | Choose core count by measurement |
| Spanning nodes | Sudden slowdown | Check efficiency within a node first |
| Unbalanced decomposition | All wait for the slowest | Use scotch; check boundary faces |
| Counting logical cores | No speed-up | Use physical cores |
| Reconstructing all times | Wastes time and disk | Only needed times or decomposed view |
I'd like to learn related topics.
Related pages include introduction to OpenFOAM, OpenFOAM mesh generation, OpenFOAM post-processing and OpenFOAM combustion analysis.
Frequently asked questions
How do I run OpenFOAM in parallel?
Set the number of subdomains (numberOfSubdomains) and method (e.g. scotch) in system/decomposeParDict, run decomposePar, then run mpirun -np N <solver> -parallel. Results are stored split across processor0, processor1 and so on; join them with reconstructPar or open the decomposed case directly in ParaView.
How many cores should I use?
Tens of thousands to around a hundred thousand cells per core is a guide. For a 4-million-cell case with 2% serial work, 32 cores (125,000 cells each) give 19.8×, 128 cores (31,000) about 36×, and 512 cores (7,800) only 45.6×, with efficiency down to 9%. Real speed depends on the network and solver, so measure upward from a small core count.
Which decomposition method should I choose?
scotch is usually the safe choice: it balances cell counts automatically while minimising subdomain boundary faces (communication). simple and hierarchical take a number of divisions per direction; efficient for simple shapes but prone to more boundaries on complex ones. The share of boundary faces grows as cells per core fall.
References and scope of verification
Numbers are educational reference calculations under the stated assumptions. Sources document theory and setup, not proof that these numerical values were reproduced in each package. Engineering decisions require validation for the actual conditions.
Related Topics
Experience the theory firsthand with the interactive simulator for this field
All Simulators