Speeding Up CAE with Transfer Learning — Troubleshooting Guide

Transfer Learning in CAE — the Anatomy and Where It Breaks

Transfer learning reuses a model trained on a data-rich "source" problem on a data-poor "target" problem. In the context of CAE surrogates three patterns are typical: ① a model from a past project with similar geometry and similar conditions carried over to a new product, ② pre-training on a large volume of coarse-mesh (low-fidelity) data followed by fine-tuning on a small high-fidelity set, and ③ a public dataset or a general-purpose pre-trained model adapted to an in-house problem. When it works, it cuts the expensive data you need (the number of FOM runs) by a factor of several to an order of magnitude. The danger of the technique is that it fails silently when the problem you assumed was similar turns out not to be, and this page works through those failures symptom by symptom.

Symptom 1: Transfer Does Worse Than Training From Scratch (Negative Transfer)

Symptom 2: Too Little Fine-Tuning Data, and the Model Overfits

A target set of only a few dozen cases is exactly where transfer learning earns its keep, but retraining every layer naively overfits immediately. The standard move is to match the degrees of freedom you update to the amount of data.

1. Retrain the output layer only (freeze the feature extractor) — the first choice at a few dozen samples
2. Progressive unfreezing — release one block at a time from the output end while watching the validation error. Stop when the improvement stops
3. Stronger regularization — instead of freezing, penalizing the distance from the source weights (L2-SP) also works well
4. Data augmentation — exploit symmetries of the problem (rotation, reflection) and parameter perturbations to increase the effective data volume

In every case, do not skip hold-out validation (cross-validation if the data is scarce). In a low-data regime the training error carries no information at all.

Symptom 3: The Transferred Model's Predictions Are Off by Orders of Magnitude

When the output is broken rather than merely inaccurate, the culprit is almost always a mismatch in the preprocessing pipeline. ① Normalization constants — are the target inputs normalized with the source's mean and variance? Recomputing them on the target is inconsistent with the trained weights in some designs and is exactly the right thing to do in others, so follow the design used at pre-training. ② The order of the input channels and the system of units. ③ Mesh resolution and interpolation method. ④ Coordinate system and the definition of the non-dimensionalization. The principle is to ship the whole preprocessing and postprocessing code as part of "the model", not just the weights; a transfer that neglects this always breaks here.

Symptom 4: Coarse-Mesh Pre-Training to Fine-Mesh Fine-Tuning Leaves a Bias

Pre-training on low-fidelity data is a sound strategy, but the model can learn the systematic errors peculiar to a coarse mesh (smeared peaks, an unresolved boundary layer) as "the right answer", and a small high-fidelity set may not be enough to correct it. The remedy is to change the design of the transfer: ① difference learning — train a small model on the difference between high and low fidelity (a multi-fidelity construction, the same idea as co-Kriging), ② feed the low-fidelity prediction in as an input feature, ③ concentrate the high-fidelity data where the error is large (near the peaks). Inheriting the weights of a single network is not the only form of transfer; in this area it is often cleaner to absorb the fidelity gap through the architecture.

Symptom 5: You Cannot Tell Whether to Transfer at All

A light diagnosis before you commit tells you a great deal about the odds.

• Zero-shot evaluation — apply the source model to the target data as it stands and measure the error. Clearly better than the initial error of a from-scratch model is evidence of shared structure
• Linear probing of the features — freeze the intermediate features of the source model and predict the target with linear regression alone. If that works, the feature-extraction layers are worth transferring
• Learning-curve comparison on a small budget — compare the validation error with and without transfer at 10, 20 and 40 samples. A healthy transfer shows the gap widening on the low-data side

All three are experiments that finish in a few hours, and they also supply the justification for the production fine-tuning strategy (how much to freeze).

Symptom 6: You Reported the Gain and Were Told It Was a Fluke

Reporting the effect of transfer learning has its own conventions. ① Compare at an equal data budget — "transfer + 20 high-fidelity cases" against "from scratch + 20 high-fidelity cases", stating the cost of the pre-training data separately. ② Report the mean and variance over several seeds — low-data training is highly sensitive to initialization, and a single comparison means nothing. ③ Present it as a learning curve — with data volume on the horizontal axis, so that the gain from transfer is visible as a shift toward the low-data side. ④ Behaviour out of distribution — check the degradation at the edges of the target distribution. With those four in place, the "fluke" objection is answered.

Checklist for a Transfer Learning Project

• Quantified how close the source and target are in distribution and in physical regime
• Ran the zero-shot and linear-probe diagnoses up front
• Shipped the whole preprocessing pipeline together with the model
• Matched the freezing strategy to the data volume, and started fine-tuning at a low learning rate
• Compared against a from-scratch baseline over several seeds
• Considered difference learning and a multi-fidelity construction for the fidelity gap, rather than only inheriting the network

Related: reduced order models (ROM), Kriging surrogates (multi-fidelity), Fourier neural operators.