Quick answer AdaBoost builds a strong classifier as the weighted vote H(x) = sign(Σ αt·ht(x)). Each stage weight follows from the weighted error εt via αt = (1/2)·ln((1 − εt)/εt), so εt = 0.3 gives αt ≈ 0.424. Sample weights then update as wi ← wi·exp(−αt·yi·ht(xi))/Zt.
For 2D data, run AdaBoost for T iterations using depth-1 decision stumps as weak classifiers. Visualize sample weights and the final strong classifier's decision boundary to learn why a combination of weak learners can reach high accuracy.
Parameters
Boosting iterations T
rounds
Query point x
—
Query point y
—
Data is generated by an LCG with seed 42 (class +1: near origin, class −1: donut). Not linearly separable.
Results
—
Training accuracy
—
Query prediction
—
Boosting rounds T
—
Sum of alpha weights
Sample weights and decision boundary
Blue = class +1 / Red = class −1 / point size = final weight / background shading = strong classifier boundary / black × = query point
Theory & Key Formulas
AdaBoost builds a strong classifier by weighted majority vote. At each iteration it concentrates weight on samples misclassified by the previous stage and trains weak learners sequentially.
Weighted error rate ε_t and weak-classifier weight α_t:
Depth-1 decision stumps (single-feature threshold tests) are enough as weak learners. As T grows, the training error decreases exponentially.
What is the AdaBoost Simulator?
🙋
I hear the term "weak classifier" all the time. It feels kind of magical that combining weak things ends up strong, doesn't it?
🎓
That's the whole point of AdaBoost. Roughly speaking, if you combine dozens of classifiers that are just slightly better than random — like depth-1 decision stumps — by a weighted majority vote, you can draw surprisingly complex decision boundaries. The simulator above is trying to separate the blue class near the origin from the donut-shaped red class. Each stump can only draw a single vertical or horizontal line like "x ≥ threshold" or "y ≥ threshold", but stack 20 of them and a curve-like boundary emerges.
🙋
Why are the dots different sizes? Some are big and some are small.
🎓
That's the heart of AdaBoost. The final sample weight is shown by the dot's size. At each iteration the weight of "samples the previous stage got wrong" is increased, so points near the decision boundary — the hard cases — get bigger. Easy points shrink. Through the weights, you're essentially telling the next weak learner "focus on the problems we've failed at so far."
🙋
There's a card called "Sum of alpha weights" — what is that?
🎓
The final strong classifier multiplies each stump's prediction by α_t and adds them up. α_t is a "confidence" that grows as the error rate shrinks: $\alpha_t = \frac{1}{2}\log\frac{1-\varepsilon_t}{\varepsilon_t}$, so ε=0.5 (random) gives α=0 and ε=0.1 gives α≈1.1. The sum reflects the "scale" of the strong-classifier score, and grows with T. Slide T from 1 to 50 and watch how the training accuracy and the alpha sum change together.
🙋
The query point (0,0) gave "+1" as expected — that's the center of the donut, so blue, right?
🎓
Right, the origin is the center of class +1, so that matches expectations. Try moving the query to around (3,0) — that's on the donut, so it should be −1. Push it further out to (5,5) and you're in a region with no training data, so the prediction depends on the particular combination of stumps. Boundary stability in extrapolation regions is generally weak in boosting too.
Frequently Asked Questions
Both are ensemble methods, but they train differently. Bagging (Random Forest is the canonical example) generates multiple training sets by bootstrap sampling, trains each learner independently in parallel, and aggregates by majority vote. AdaBoost trains learners sequentially while increasing the weights of samples misclassified by previous stages, then takes a weighted majority vote where each learner is weighted by α. Bagging primarily reduces variance, while boosting reduces bias.
AdaBoost can be interpreted as greedy minimization of the exponential loss, and is a special case of gradient boosting. Gradient boosting generalizes the loss function and fits weak learners to residual gradients for any differentiable loss such as log loss or squared error. XGBoost and LightGBM are large-scale implementations of gradient boosting using histogram approximation, regularization, and parallelization, and have become the go-to algorithm for tabular data on Kaggle and similar competitions. AdaBoost is the classical method at the root of this lineage.
The Viola-Jones face detector (2001) is the most famous practical application of AdaBoost. It uses AdaBoost to pick a small number of effective Haar-like features from a huge pool of weak-classifier candidates and arranges them in a cascade structure, enabling real-time face detection. OpenCV's haarcascade_frontalface and similar files implement this method, and the approach was widely deployed in everything from feature-phone cameras to digital camera face recognition.
In theory the training error decreases monotonically as you increase iterations T, but test error does not necessarily grow. This is explained by margin theory: even after the training error reaches zero, AdaBoost continues to widen the margin (distance from the decision boundary). However, on noisy-label data, weights concentrate on noisy samples and AdaBoost overfits. If noise is suspected, LogitBoost or gradient boosting are more robust.
Real-World Applications
Face and object detection: The Viola-Jones face detector is the most famous application of AdaBoost. It picks effective features from a huge pool of Haar-like feature candidates using AdaBoost, and uses a cascade structure to quickly reject "regions that are clearly not faces", enabling real-time face detection even on the modest hardware of the day. It was used for years in digital camera face recognition and smartphone autofocus, and is a workhorse of practical image processing.
Tabular data classification and regression: Gradient Boosted Decision Trees (GBDT), descendants of AdaBoost, are among the strongest methods for classification and regression on tabular data. Libraries like XGBoost, LightGBM, and CatBoost are widely used in Kaggle competitions and production credit scoring, demand forecasting, and anomaly detection, and shine on tabular data where deep learning struggles.
Text classification and spam filtering: Many early email spam filters were implemented with AdaBoost or similar boosting methods using word occurrences as weak classifiers. The presence or absence of any single word is a weak feature, but stacking thousands of them builds a highly accurate filter.
Medical diagnosis support: Boosting methods are also used for binary classification of medical images and biosignals, such as tumor detection in radiology images and arrhythmia detection in ECGs. Because the contribution (α weight) of each feature is interpretable, the approach fits the accountability requirements of clinical settings well.
Common Misconceptions and Caveats
The most common misconception is to simplistically think "more weak classifiers always means higher accuracy". It is true that the training error decreases monotonically with T, but the test error does not necessarily improve. Especially on data with many noisy labels, weights concentrate on those noisy samples and the later weak learners are spent memorizing noise, causing overfitting. If you set T=50 in the simulator and the boundary squirms exactly along the training points, that is a warning sign that the model will be weak on new data. In practice, choose T by cross-validation or introduce early stopping.
The next most common mistake is the assumption that "decision stumps are too weak for nonlinear problems". A single stump really can only draw one vertical or horizontal line, but a weighted majority vote in AdaBoost can separate even a complex donut-shaped class. Set T=20 in the simulator above and the training accuracy should be plenty high. The "weakness" of a weak learner is not the issue — as long as it does even slightly better than random, boosting reduces error exponentially.
Finally, note that sample weights are "loss weighting", not "resampling". Some implementations do resample training data in proportion to the weights, but the standard AdaBoost implementation only minimizes the weighted error rate during weak-learner training and does not duplicate data. This simulator likewise uses all 40 samples as-is and adjusts each sample's contribution through w_i. The weight update $w_i \leftarrow w_i\,\exp(-\alpha_t\,y_i\,h_t(\mathbf{x}_i))$ has a symmetric form that lowers the weight of correctly classified samples and raises the weight of misclassified ones.
Set the number of boosting rounds T with the "Boosting iterations T" slider (1–50, default 20). Each round adds one depth-1 decision stump; the stump itself is chosen automatically as the axis-aligned split with the lowest weighted error
Watch the point sizes: after T rounds each sample is drawn with its final weight w_i, so the points that the stumps keep misclassifying (near the ring boundary) grow and the easy ones shrink
Move the "Query point x" and "Query point y" sliders to place the black × anywhere in the plane and read "Query prediction" (+1 or −1) from the strong classifier's weighted vote
Read "Training accuracy" and "Sum of alpha weights"; α_t is large for stumps with a low weighted error, so the sum grows with T but by a shrinking amount per round as the remaining errors get harder
Worked Example
The tool classifies 40 fixed points: 20 of class +1 around the origin and 20 of class −1 on a ring of radius 3. The first stump is the horizontal split y ≈ 1.42 with weighted error ε_1 = 0.300, so α_1 = ½·ln(0.700/0.300) = 0.424 and the training accuracy after one round is 70.0 %. With T = 5 the training accuracy is 90.0 % and the sum of alpha weights is 0.424 + 0.473 + 0.415 + 0.382 + 0.274 ≈ 1.97. At the default T = 20 the ensemble reaches 100.0 % training accuracy with an alpha sum of about 6.36. The query point (1.5, 2.0) lies near the ring and is predicted as −1; the origin (0, 0) is predicted as +1 for every T.
Practical Notes
A single stump (T = 1) can only draw one vertical or horizontal line, so on the ring-shaped data it stays at 70.0 % training accuracy; compare T = 1, T = 5 (90.0 %) and T = 20 (100.0 %) in the simulator
Training accuracy is not test accuracy. On data with label noise the later stumps spend their capacity memorising the noisy points, so choose T by cross-validation or early stopping rather than by the training curve
Only axis-aligned stumps are available, so curved boundaries are approximated by a staircase of vertical and horizontal lines; expect the strong classifier to need many rounds (here about 20) before the ring is fully enclosed
On strongly imbalanced data (e.g. a 1:9 class ratio) the re-weighting concentrates on the minority class and can bias the vote; check per-class error rates, not just the overall accuracy