Interactive imagesPre-training, replayed
Pre-training, replayed
The recorded loss of two real pre-training runs, and labelled samples at recorded epochs of the smaller one beside the share whose neighbours share their label.
Scrub the loss of the reduced run, step through its snapshots, bring one label forward, and switch to the run that made the published encoder.
Pre-training with sits_ssl_lejepa()
The loss of the reduced run, and 1,532 labelled samples as it learns
Recorded run20,000 series · 30 epochs · CPU · 9 min20,000 series · 30 epochs · CPU
Loss · log scaletrainvalidation
This figure draws the snapshots from pretrain-snapshots.bin (74 kB), with JavaScript.
Epoch 30 · recorded snapshot
Same label among the 5 nearest neighbours in 64 dimensions85.6%
What to try
- Scrub the reduced run. With Reduced run pressed, drag along the loss curve, or focus it and use the arrow keys. Under the curve are labelled samples as the encoder placed them at the scrubbed epoch. Only some epochs were recorded; between them the figure says that the cloud is interpolated, and the number beside it is never interpolated.
- Step through the snapshots. Press the snapshot epochs one after the other and read the nearest-neighbour number beside the cloud. Two marks on its scale are fixed: the same number for the raw series, and for the published encoder. See where the small run stands: it passes the first at the snapshot of epoch 16, and it never reaches the second.
- Follow one label. Choose a label in the legend and scrub again: the samples of that label come forward. The encoder never saw the labels; they only colour the dots.
- Compare the two runs honestly. Press Published run. It has a curve and no cloud, because it kept no state per epoch. The runs differ in size, so compare the shapes of the curves, not their levels.