For 2D data, run AdaBoost for T iterations using depth-1 decision stumps as weak classifiers. Visualize sample weights and the final strong classifier's decision boundary to learn why a combination of weak learners can reach high accuracy.
Parameters
Boosting iterations T
rounds
Query point x
—
Query point y
—
Data is generated by an LCG with seed 42 (class +1: near origin, class −1: donut). Not linearly separable.
Results
—
Training accuracy
—
Query prediction
—
Boosting rounds T
—
Sum of alpha weights
Sample weights and decision boundary
Blue = class +1 / Red = class −1 / point size = final weight / background shading = strong classifier boundary / black × = query point
Theory & Key Formulas
AdaBoost builds a strong classifier by weighted majority vote. At each iteration it concentrates weight on samples misclassified by the previous stage and trains weak learners sequentially.
Weighted error rate ε_t and weak-classifier weight α_t:
Depth-1 decision stumps (single-feature threshold tests) are enough as weak learners. As T grows, the training error decreases exponentially.
What is the AdaBoost Simulator?
🙋
I hear the term "weak classifier" all the time. It feels kind of magical that combining weak things ends up strong, doesn't it?
🎓
That's the whole point of AdaBoost. Roughly speaking, if you combine dozens of classifiers that are just slightly better than random — like depth-1 decision stumps — by a weighted majority vote, you can draw surprisingly complex decision boundaries. The simulator above is trying to separate the blue class near the origin from the donut-shaped red class. Each stump can only draw a single vertical or horizontal line like "x ≥ threshold" or "y ≥ threshold", but stack 20 of them and a curve-like boundary emerges.
🙋
Why are the dots different sizes? Some are big and some are small.
🎓
That's the heart of AdaBoost. The final sample weight is shown by the dot's size. At each iteration the weight of "samples the previous stage got wrong" is increased, so points near the decision boundary — the hard cases — get bigger. Easy points shrink. Through the weights, you're essentially telling the next weak learner "focus on the problems we've failed at so far."
🙋
There's a card called "Sum of alpha weights" — what is that?
🎓
The final strong classifier multiplies each stump's prediction by α_t and adds them up. α_t is a "confidence" that grows as the error rate shrinks: $\alpha_t = \frac{1}{2}\log\frac{1-\varepsilon_t}{\varepsilon_t}$, so ε=0.5 (random) gives α=0 and ε=0.1 gives α≈1.1. The sum reflects the "scale" of the strong-classifier score, and grows with T. Slide T from 1 to 50 and watch how the training accuracy and the alpha sum change together.
🙋
The query point (0,0) gave "+1" as expected — that's the center of the donut, so blue, right?
🎓
Right, the origin is the center of class +1, so that matches expectations. Try moving the query to around (3,0) — that's on the donut, so it should be −1. Push it further out to (5,5) and you're in a region with no training data, so the prediction depends on the particular combination of stumps. Boundary stability in extrapolation regions is generally weak in boosting too.
Frequently Asked Questions
Both are ensemble methods, but they train differently. Bagging (Random Forest is the canonical example) generates multiple training sets by bootstrap sampling, trains each learner independently in parallel, and aggregates by majority vote. AdaBoost trains learners sequentially while increasing the weights of samples misclassified by previous stages, then takes a weighted majority vote where each learner is weighted by α. Bagging primarily reduces variance, while boosting reduces bias.
AdaBoost can be interpreted as greedy minimization of the exponential loss, and is a special case of gradient boosting. Gradient boosting generalizes the loss function and fits weak learners to residual gradients for any differentiable loss such as log loss or squared error. XGBoost and LightGBM are large-scale implementations of gradient boosting using histogram approximation, regularization, and parallelization, and have become the go-to algorithm for tabular data on Kaggle and similar competitions. AdaBoost is the classical method at the root of this lineage.
The Viola-Jones face detector (2001) is the most famous practical application of AdaBoost. It uses AdaBoost to pick a small number of effective Haar-like features from a huge pool of weak-classifier candidates and arranges them in a cascade structure, enabling real-time face detection. OpenCV's haarcascade_frontalface and similar files implement this method, and the approach was widely deployed in everything from feature-phone cameras to digital camera face recognition.
In theory the training error decreases monotonically as you increase iterations T, but test error does not necessarily grow. This is explained by margin theory: even after the training error reaches zero, AdaBoost continues to widen the margin (distance from the decision boundary). However, on noisy-label data, weights concentrate on noisy samples and AdaBoost overfits. If noise is suspected, LogitBoost or gradient boosting are more robust.
Real-World Applications
Face and object detection: The Viola-Jones face detector is the most famous application of AdaBoost. It picks effective features from a huge pool of Haar-like feature candidates using AdaBoost, and uses a cascade structure to quickly reject "regions that are clearly not faces", enabling real-time face detection even on the modest hardware of the day. It was used for years in digital camera face recognition and smartphone autofocus, and is a workhorse of practical image processing.
Tabular data classification and regression: Gradient Boosted Decision Trees (GBDT), descendants of AdaBoost, are among the strongest methods for classification and regression on tabular data. Libraries like XGBoost, LightGBM, and CatBoost are widely used in Kaggle competitions and production credit scoring, demand forecasting, and anomaly detection, and shine on tabular data where deep learning struggles.
Text classification and spam filtering: Many early email spam filters were implemented with AdaBoost or similar boosting methods using word occurrences as weak classifiers. The presence or absence of any single word is a weak feature, but stacking thousands of them builds a highly accurate filter.
Medical diagnosis support: Boosting methods are also used for binary classification of medical images and biosignals, such as tumor detection in radiology images and arrhythmia detection in ECGs. Because the contribution (α weight) of each feature is interpretable, the approach fits the accountability requirements of clinical settings well.
Common Misconceptions and Caveats
The most common misconception is to simplistically think "more weak classifiers always means higher accuracy". It is true that the training error decreases monotonically with T, but the test error does not necessarily improve. Especially on data with many noisy labels, weights concentrate on those noisy samples and the later weak learners are spent memorizing noise, causing overfitting. If you set T=50 in the simulator and the boundary squirms exactly along the training points, that is a warning sign that the model will be weak on new data. In practice, choose T by cross-validation or introduce early stopping.
The next most common mistake is the assumption that "decision stumps are too weak for nonlinear problems". A single stump really can only draw one vertical or horizontal line, but a weighted majority vote in AdaBoost can separate even a complex donut-shaped class. Set T=20 in the simulator above and the training accuracy should be plenty high. The "weakness" of a weak learner is not the issue — as long as it does even slightly better than random, boosting reduces error exponentially.
Finally, note that sample weights are "loss weighting", not "resampling". Some implementations do resample training data in proportion to the weights, but the standard AdaBoost implementation only minimizes the weighted error rate during weak-learner training and does not duplicate data. This simulator likewise uses all 40 samples as-is and adjusts each sample's contribution through w_i. The weight update $w_i \leftarrow w_i\,\exp(-\alpha_t\,y_i\,h_t(\mathbf{x}_i))$ has a symmetric form that lowers the weight of correctly classified samples and raises the weight of misclassified ones.