XGBoost: Watch the Machine Think

The Problem: We have 196 data points sampled from a wavy surface — imagine sensor readings from a terrain, where the height z depends on two coordinates (x, y). The true relationship is z = sin(x) + cos(y) + noise. Our goal: learn to predict z from (x, y) — a regression task.

The Method: XGBoost builds an ensemble of decision trees, one at a time. Each new tree focuses on correcting the mistakes of all previous trees. The secret sauce? It uses a second-order Taylor expansion of the loss function to figure out the optimal correction at each step — like using both the slope AND curvature of a hill to find the fastest path downhill.

What you'll see: Step through the training process and watch the prediction surface morph to fit the data. Click any data point to inspect the Taylor expansion math happening under the hood.

Cyan orbs = data points (true values)

Magenta surface = model's predictions

Blue beams = under-predicted (need to go up)

Red beams = over-predicted (need to go down)

Orange plane = tree split boundary

Green = improvement / optimal step

What Am I Looking At?

Cyan Orbs — Data Points

Each glowing sphere is one data sample. Its horizontal position is the input (x1, x2), and its height is the true target value z. This is the "ground truth" the model is trying to learn.

Magenta Surface — Predictions

The translucent surface shows what the model currently predicts at every (x, y) point. It starts flat (at the average value) and gradually warps to match the data as trees are added.

Blue Beams — Under-predictions

Vertical lines from the surface up to data points above it. The model is predicting too low here — these are positive residuals. Brighter = bigger error.

Red Beams — Over-predictions

Lines from data points up to the surface above them. The model is predicting too high — negative residuals. The next tree will try to push predictions down here.

Orange Plane — Tree Split

When a tree is being built, this plane shows where the algorithm is splitting the data. It's partitioning the feature space to group similar errors together, then assigning each group an optimal correction.

Loss Curve (HUD)

The sparkline in the bottom bar tracks Mean Squared Error over rounds. Watch it drop as the ensemble improves. Steep drops = the tree found a useful pattern. Plateaus = diminishing returns.

Taylor Lens (click a point)

Click any data point to see the math. The cyan curve is the actual loss function. The dashed orange parabola is the Taylor approximation — XGBoost uses this quadratic to cheaply find the optimal correction (green dot) without evaluating the full loss.

Controls

Step = add one tree. Learning Rate = how much each tree contributes (lower = more conservative). Max Depth = tree complexity (deeper = more specific corrections). Speed = animation pace.

XGBoost: Watch the Machine Think

What Am I Looking At?

Leaderboard

XGBoost

The Taylor Lens