Interpretability · interactive toy

Can a straight line read “is this a midpoint?”
from the model’s vector?

Each dot is one point-token, shown as a 2-number vector (in reality 3584 numbers). Filled = a true midpoint, open = not. Drag LAYER: low layers are early in the model (no signal yet); high layers are deep (the model has computed midpoint-ness). The dashed line is the probe — the direction it learned. Watch the held-out accuracy.

Held-out accuracy  (6 points the probe never trained on)
50%
…
midpoint (true)
not a midpoint
held-out test point (not used to train the probe)
the probe’s decision line
At layer 0 the dots are pure noise — no line can separate them, so the probe is ~a coin flip. That means the “midpoint” fact is NOT yet in the vector.