Gradient Clipping Simulator

Training recurrent neural networks (RNN/LSTM): Gradient clipping first became widely used for training RNNs. Because an RNN multiplies the same weights repeatedly through time, backpropagating a long sequence makes the gradient grow exponentially and a steep cliff appears in the loss landscape. In early RNN/LSTM models for machine translation, speech recognition and text generation, leaving out norm clipping (typically tau = 1 to 5) routinely caused the loss to become NaN within a few epochs and training to stop. Clipping is, in effect, a mandatory stabilisation technique.

Training Transformers and large language models: Gradient clipping is still standard in modern, very large Transformers. Early in training, or occasionally on a "bad mini-batch", the gradient norm can spike, and without clipping a single spike can wreck the entire run. Training recipes for GPT-style and BERT-style models have long used "clip on the global norm, threshold 1.0" as an almost default value. It acts as insurance that lets a run of billions of parameters proceed for weeks without stopping.

Reinforcement learning and policy gradients: In reinforcement learning the reward scale is unstable, and policy-gradient estimates can become outliers of enormous magnitude. Implementations of algorithms such as PPO and A2C commonly apply norm clipping to the gradients of the policy and value networks. It is an unglamorous but effective safety device that keeps noisy signals from the environment from destroying training.

Divergence control in CAE and numerical optimisation: Beyond machine learning, gradient-based numerical optimisation of an objective function shares the problem of "the step is too large and the run diverges". Trust-region methods and line search are principled ways to control the step size, but gradient clipping can be seen as the simplest step limiter that just sets an upper bound. On an ill-conditioned objective where the design variables are not scaled consistently, it can serve as a quick remedy to keep the optimisation from blowing up.

The most common misconception is that gradient clipping also fixes vanishing gradients. Clipping addresses only "exploding gradients", where the gradient grows too large; it does nothing for "vanishing gradients", where the gradient becomes too small and training stalls. If anything, clipping is an operation that shrinks large gradients, so it has no bearing on vanishing. The cures for vanishing gradients are different: gated structures such as LSTM and GRU, residual (skip) connections, suitable weight initialisation and batch normalisation. Keep clipping and vanishing-gradient remedies clearly separated in your mind.

Next is the belief that the smaller the threshold tau, the safer. It is true that a smaller tau makes divergence less likely, but if tau is too small even the legitimate, normal-sized gradients are uniformly shrunk, the effective learning rate drops and convergence slows. In the extreme, setting tau near zero leaves every step with a tiny stride and you never reach the valley floor. Lower the threshold in this tool and you will see the number of clip events rise while the loss decreases more and more slowly. The key is to place tau at just the right value that stops only the explosion and lets the normal gradient pass straight through.

Finally, the misconception that with clipping in place, the learning rate can be as large as you like. Clipping caps the gradient size at tau, but the actual step is eta*tau. Raise the learning rate eta to an extreme and, even with a bounded clipped gradient, the eta*tau step becomes too large, overshooting the minimum and oscillating or diverging. In this tool too, raising the learning rate close to its upper limit with norm clipping enabled can still diverge. Clipping is only an "upper bound on the gradient side"; it does not remove the need to tune the learning rate.