Explore "dropout", the technique that curbs overfitting in neural networks. Change the keep probability p and the layer width to see the mask of neurons that are switched off, the binomial distribution of how many are kept, the 1/p scaling and the total number of sub-networks 2ⁿ that are implicitly trained.
Parameters
Neurons in layer n
Total number of neurons (units) in this layer
Keep probability p
Probability a neuron is kept. 1−p is the drop rate
Random seed
Seed of the PRNG that generates the mask. Deterministic
Mean activation a
Approximate size of each neuron's activation output
Results
—
Kept neurons
—
Dropped neurons
—
Scale factor 1/p
—
Expected active count
—
Total sub-networks
—
Output scale ratio
—
Dropout mask — sub-network animation
The layer's neurons are shown as a grid. Lit neurons are kept; greyed neurons with an X are dropped this draw. The mask is re-sampled about every 1.5 s, so a different sub-network appears each time.
Probability distribution of kept neurons (binomial)
During training, each neuron i is kept with probability p ($m_i=1$) and dropped with probability $1-p$ ($m_i=0$). The surviving activation $a_i$ is scaled by 1/p. This is inverted dropout, so at inference no mask and no scaling are needed and the network is used as is.
The probability that exactly k of n neurons are kept follows the binomial distribution, with expected kept count np. The expected output $\mathbb{E}[\sum m_i a_i / p]=\sum a_i$ equals the value before dropout.
Because each neuron has two states — kept or dropped — there are $2^{n}$ possible sub-networks. Dropout implicitly trains this exponentially large family of thinned networks at the same time.
What is the Dropout Regularization Simulator?
🙋
"Dropout" deliberately switches off neurons during training, right? Isn't it wasteful to throw away parts you worked hard to build?
🎓
I see the worry, but "temporarily resting them" is the right picture, not throwing them away. For each training mini-batch, each neuron is switched off at random with probability 1−p, as if flipped by a coin. On the next batch a different set of neurons rests. Set the keep probability p on the left to 0.5 and about half are resting every time. Watch the canvas above — the lit nodes and the grey X-marked nodes swap every 1.5 seconds. That is what "learning with a different sub-network each time" looks like.
🙋
You deliberately break it at random, and that improves performance? That feels counter-intuitive.
🎓
That is exactly what makes dropout interesting. If you train normally without removing neurons, the neurons start dividing labour: "if you watch for this feature, I will back you up". This is called co-adaptation — a team that gets along too well. It is strong on the training data, but if a member changes slightly on real data the whole thing collapses, and that is overfitting. Dropout creates a world where "you never know when your neighbour will rest", so each neuron has no choice but to learn features that are useful on their own, without leaning on others.
🙋
I see, you forbid leaning on each other. But if you remove half of them, won't the signal coming out of the layer get weaker?
🎓
Sharp point. With p=0.5 the sum roughly halves. That is where 1/p scaling comes in. The surviving neurons' outputs are amplified by 1/p — at p=0.5 that means doubled. Look at "Scale factor 1/p" and "Output scale ratio" on the left. The ratio wobbles around 1 each time you change the seed, but on average it is always 1. Keeping the expected layer output equal to the pre-dropout value means that at inference you can turn dropout and scaling fully off and use the network as is. This scheme is called inverted dropout.
🙋
The "Total sub-networks" of ≈10⁶ is a huge number. What does that actually mean?
🎓
With n=20 neurons each being kept or dropped, the number of masks is 2²⁰ ≈ one million. Push n up to 64 and it becomes an astronomical 1.8×10¹⁹. Each batch trains only one of those sub-networks, but the weights are shared by all of them. So dropout can be read as ensemble learning: it trains an exponentially large family of thinned networks together and takes an approximate average of them at inference. It is the same idea as a random forest averaging many decision trees, realized inside a single network.
🙋
So should I just make p small and drop as many as possible?
🎓
No, that is a trade-off. If p is too small almost everything is removed each time, the network's effective capacity is no longer enough, and you swing the other way into underfitting. Conversely p=1.0 turns dropout fully off and gives zero regularization. In practice p=0.5 for fully connected hidden layers, and a higher p of 0.8-0.9 near the input so you don't discard too much information, is the standard. In the "Expected active count vs p" chart below, move p and you will see the surviving count change linearly.
Frequently Asked Questions
On every training step dropout randomly switches off neurons with probability 1−p, so the network cannot adopt a strategy that relies entirely on any single neuron. This breaks co-adaptation, where one neuron divides labour by assuming another neuron is always present, and forces each neuron to learn features that are useful on their own. It is close to ensemble learning, which trains many thin sub-networks and averages them, so generalization improves and overfitting is suppressed.
When dropout removes some neurons, the sum of the layer output drops on average by a factor of p. In inverted dropout, the activations of the surviving neurons are scaled by 1/p during training, which cancels this reduction and keeps the expected layer output unchanged. As a result, no mask and no scaling are needed at inference time — the network is used as is. All major deep-learning frameworks use this inverted-dropout scheme.
In a layer with n neurons each neuron is either kept or dropped, so the number of possible masks is 2ⁿ. For n=20 that is about one million, and for n=64 it is roughly 1.8×10¹⁹ — an astronomical number. Each mini-batch trains only one of these sub-networks, but the weights are shared across all of them, so dropout can be seen as training an exponentially large family of networks at once and taking an approximate average of them at inference.
For fully connected hidden layers, p=0.5 (drop half) is the classic standard value, and it is the value used in the original paper by Hinton and colleagues. Near the input layer a higher p of 0.8-0.9 is used so that not too much information is discarded. Convolutional layers overfit less because of weight sharing and fewer parameters, so dropout is weakened or skipped there, or SpatialDropout, which drops whole spatial blocks, is used instead. p=1.0 is equivalent to turning dropout completely off.
Real-World Applications
Image-recognition CNN classifiers: In convolutional networks such as VGG and AlexNet, p=0.5 dropout in the final fully connected layers (the classification head) was a standard choice. Convolutional layers have few parameters and overfit little, while fully connected layers carry tens of millions of parameters and become a hotbed of overfitting. Concentrating dropout there preserves generalization even on large datasets like ImageNet. For convolutional layers, SpatialDropout — which drops whole spatially correlated blocks — is used rather than ordinary dropout.
Natural language processing and Transformers: Transformers in the BERT and GPT families apply a modest dropout of around p=0.1 to the output of each attention layer and feed-forward block. Attention dropout, applied to the attention weight matrix itself, is often used in addition. Large language models train on huge corpora, yet dropout still helps prevent over-fitting to specific token co-occurrence patterns.
Uncertainty estimation (MC Dropout): Dropout is normally turned off at inference, but if you deliberately keep it on and pass the same input dozens of times, you get predictions from a different sub-network each time. Reading that spread (variance) as the model's "lack of confidence" is the Monte Carlo Dropout method. In areas where prediction reliability matters — medical imaging, autonomous driving — it is used as a convenient approximation of Bayesian uncertainty estimation.
Training on small datasets: In medical, industrial-inspection and scientific-measurement domains where there are only hundreds to thousands of samples, overfitting in which the model memorizes the training set is a serious problem. Alongside data augmentation and transfer learning, dropout is one of the first regularizers to try because it costs almost nothing. When the validation loss starts to diverge sharply from the training loss, raising the dropout rate is a basic countermeasure.
Common Misconceptions and Pitfalls
The most common mistake is leaving dropout active at inference time. The 1/p scaling of inverted dropout is done during training, so at inference dropout must be turned fully off — all neurons used, no scaling. In frameworks this is handled by switching between "train mode" and "eval mode"; forgetting to switch to eval mode before evaluation makes predictions wobble randomly each time and reduces accuracy. The MC Dropout approach, which intentionally keeps dropout on, is a different thing entirely — do not confuse the two.
Next, carelessly stacking dropout and batch normalization (BatchNorm). Placing both in the same layer makes the random scale fluctuation that dropout creates interfere with the mini-batch statistics (mean and variance) that BatchNorm computes; the statistics differ between training and inference and performance drops. In practice you need to think about placement — skip dropout in layers that use BatchNorm, or place dropout after BatchNorm. The idea that adding both to every layer simply makes the model stronger is dangerous.
Finally, the belief that "the higher the dropout rate, the stronger and better the regularization". Making p small to remove a lot does suppress overfitting, but it reduces the capacity the network can use at once and tips you into underfitting, where the model cannot even learn the training data well. Training also becomes unstable and converges more slowly. The dropout rate is a hyperparameter and should be tuned while watching the validation loss. Rather than "it is overfitting, so just lower p", first look at the balance with model capacity, data volume and learning rate, and tune from a sensible range (around p=0.5 for hidden layers).
How to Use
Set the network layer width (nNum) between 32–512 neurons and keep probability (pNum) from 0.3–0.9 to simulate dropout masking.
Adjust the seed (seedNum) to generate different random dropout patterns; observe how kept vs. dropped neuron counts vary across initialization.
Configure activation function (aNum) as ReLU or sigmoid, then run the simulator to compute scaling factor 1/p and total possible sub-networks (2^n).
Worked Example
For a dense layer with 256 neurons and keep probability p=0.7: approximately 179 neurons remain active (256×0.7), 77 are dropped. The scale factor becomes 1/0.7≈1.43, so forward-pass outputs are multiplied by 1.43 during training to maintain expected activation magnitude. This generates 2^256 possible sub-networks, reducing co-adaptation. At inference with p=1.0, all 256 neurons fire without scaling, matching the rescaled training distribution.
Practical Notes
For convolutional feature maps (e.g., 64 filters, 28×28 spatial): apply spatial dropout (drop entire feature channels) rather than element-wise to preserve local structure.
In recurrent networks, use the same dropout mask across time steps; varying masks per step degrades temporal coherence.
Keep probability 0.5 is standard for hidden layers; input layers tolerate higher p (0.8–0.9) since raw features carry less redundancy than learned representations.
Combine dropout with batch normalization carefully—batch norm reduces internal covariate shift, potentially making aggressive dropout (p<0.5) unnecessary and increasing computational cost during inference due to mask sampling overhead.