# Pre-training, replayed

The recorded loss of two real pre-training runs, and labelled samples at recorded epochs of the smaller one beside the share whose neighbours share their label.

[From step 4 of From series to embeddings](/embeddings.md).

In the sits book: [Building regional embeddings: a full example](https://e-sensing.github.io/sitsbook/emb_example.html) and [Embedding Methods in SITS](https://e-sensing.github.io/sitsbook/emb_methods.html).

Scrub the loss of the reduced run, step through its snapshots, bring one label forward, and switch to the run that made the published encoder.

## The figure

Recorded by sits: the loss after each epoch, on a log scale.

Figure: two recorded runs of `sits_ssl_lejepa()`, never drawn in one chart, behind a switch. "Reduced run" (Recorded run · 20,000 series · 30 epochs · CPU · 9 min): its training and validation loss per epoch, as a curve to scrub; under it the 1,532 labelled samples at the scrubbed epoch, coloured by their label (Annual crop, Cerradão, Cerrado, Mangrove, Natural non-vegetated, Open cerrado, Pasture, Perennial crop, Silviculture, Sugarcane and Water), with buttons for the snapshot epochs 1, 2, 4, 8, 16 and 30 and a legend that brings one class forward; between two snapshots the positions are interpolated and the figure says so. Beside it a scale of the share of samples whose label is the majority label of their 5 nearest neighbours in the 64 dimensions of the embedding: the reduced run at each snapshot, and marks for the raw series and the published encoder. "Published run" (Recorded run · 647,967 series · 54 of 150 epochs, stopped early · GPU · 7.1 h): its loss per epoch, and its mark on the scale; it stored no snapshots, so it has no cloud.

Loss per epoch of the reduced run.

| Epoch | Training loss | Validation loss |
| ----- | ------------- | --------------- |
| 1     | 1.952         | 1.070           |
| 2     | 0.853         | 0.660           |
| 3     | 0.617         | 0.535           |
| 4     | 0.508         | 0.486           |
| 5     | 0.466         | 0.417           |
| 6     | 0.436         | 0.402           |
| 7     | 0.404         | 0.373           |
| 8     | 0.391         | 0.354           |
| 9     | 0.373         | 0.351           |
| 10    | 0.357         | 0.340           |
| 11    | 0.351         | 0.343           |
| 12    | 0.343         | 0.323           |
| 13    | 0.337         | 0.310           |
| 14    | 0.336         | 0.303           |
| 15    | 0.322         | 0.302           |
| 16    | 0.311         | 0.290           |
| 17    | 0.313         | 0.283           |
| 18    | 0.306         | 0.272           |
| 19    | 0.298         | 0.277           |
| 20    | 0.299         | 0.276           |
| 21    | 0.292         | 0.267           |
| 22    | 0.293         | 0.272           |
| 23    | 0.291         | 0.258           |
| 24    | 0.284         | 0.257           |
| 25    | 0.279         | 0.247           |
| 26    | 0.280         | 0.245           |
| 27    | 0.277         | 0.256           |
| 28    | 0.274         | 0.237           |
| 29    | 0.269         | 0.238           |
| 30    | 0.269         | 0.243           |

Loss per epoch of the run that made the published encoder. The runs differ in size, so compare the shapes of their curves, not their levels.

| Epoch | Training loss | Validation loss |
| ----- | ------------- | --------------- |
| 1     | 0.379         | 0.225           |
| 2     | 0.203         | 0.175           |
| 3     | 0.179         | 0.154           |
| 4     | 0.166         | 0.149           |
| 5     | 0.157         | 0.138           |
| 6     | 0.151         | 0.130           |
| 7     | 0.146         | 0.128           |
| 8     | 0.142         | 0.122           |
| 9     | 0.138         | 0.120           |
| 10    | 0.135         | 0.117           |
| 11    | 0.132         | 0.119           |
| 12    | 0.130         | 0.115           |
| 13    | 0.128         | 0.116           |
| 14    | 0.126         | 0.112           |
| 15    | 0.125         | 0.109           |
| 16    | 0.123         | 0.109           |
| 17    | 0.122         | 0.109           |
| 18    | 0.121         | 0.108           |
| 19    | 0.120         | 0.107           |
| 20    | 0.119         | 0.106           |
| 21    | 0.118         | 0.104           |
| 22    | 0.117         | 0.103           |
| 23    | 0.117         | 0.105           |
| 24    | 0.116         | 0.103           |
| 25    | 0.115         | 0.102           |
| 26    | 0.115         | 0.102           |
| 27    | 0.114         | 0.101           |
| 28    | 0.114         | 0.103           |
| 29    | 0.113         | 0.100           |
| 30    | 0.113         | 0.101           |
| 31    | 0.112         | 0.101           |
| 32    | 0.112         | 0.099           |
| 33    | 0.111         | 0.099           |
| 34    | 0.111         | 0.098           |
| 35    | 0.111         | 0.100           |
| 36    | 0.110         | 0.099           |
| 37    | 0.110         | 0.097           |
| 38    | 0.110         | 0.098           |
| 39    | 0.110         | 0.098           |
| 40    | 0.109         | 0.097           |
| 41    | 0.109         | 0.097           |
| 42    | 0.109         | 0.096           |
| 43    | 0.109         | 0.096           |
| 44    | 0.109         | 0.097           |
| 45    | 0.108         | 0.096           |
| 46    | 0.108         | 0.097           |
| 47    | 0.108         | 0.097           |
| 48    | 0.108         | 0.096           |
| 49    | 0.108         | 0.096           |
| 50    | 0.108         | 0.096           |
| 51    | 0.108         | 0.096           |
| 52    | 0.107         | 0.095           |
| 53    | 0.108         | 0.096           |
| 54    | 0.107         | 0.096           |

### How it was made

1. Read the labelled samples: The pipeline reads the labelled samples: time series, each with its land cover class. It lists the classes and gives each the colour sits draws it in.
2. Encode the labelled samples: The pipeline draws the same number of labelled samples from each class of the period of the embeddings cube, and sits_encode() turns each into its 64 numbers with the published encoder.
3. Read how the encoder was trained: The published encoder keeps a record of its own training: the loss of every round on the training and the validation series, how long each round took, and the settings sits_ssl_lejepa() was called with. The pipeline reads that record.
4. Draw the series of a small run: The encoder also keeps the series it was trained on. The pipeline draws some of them under a fixed seed, for a smaller training run of its own.
5. Store the window's embeddings in one byte: The pipeline cuts the study window out of the embeddings cube: 64 numbers for each pixel. It stores each number in one byte, and checks that the nearest neighbours of the labelled samples give their class almost as often as before: the pipeline stops if the share they get right falls by more than 0.01.
6. Train a small encoder, and stop it early: sits_pre_train() trains an encoder of its own with sits_ssl_lejepa() on the drawn series. It trains again from the start for fewer rounds, under the same seed, and the pipeline checks that each shorter run is the start of the long one. sits_encode() embeds the fixed set after each run.
7. Make two views of one series: sits_ssl_lejepa() learns by comparing two views of the same series. The pipeline draws one of the series under a fixed seed and makes its two views as sits makes them in training.
8. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it.

### Reproducibility details

- classes.json and pretrain-snapshots.bin: A rerun on aarch64-apple-darwin23 gave the same files, byte for byte.
- pretrain.json: Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash.

Every step of the pipeline:

1. The published encoder of the Cerrado, as kept from Hugging Face. `hf_cache_path("e-sensing/encoders_cerrado", "ssl_lejepa_tcnn_model_2017_2024.rds", "model")`
2. The labelled Cerrado samples, as kept from Hugging Face. `hf_cache_path("e-sensing/samples_cerrado", "labelled_samples_cerrado.parquet", "dataset")`
3. The epochs of the snapshot runs, then of the full run. `c(1L, 2L, 4L, 8L, 16L, 30L)`
4. Reads the encoder once: the record of its training, and series drawn from those it was trained on. `encoder_contents(encoder_read(encoder_file), 20000L, 20261011L)`
5. Reads the samples into a table, one row per time series. `samples_read_cached(samples_file)`
6. Lists the land cover classes found in the samples. `sits::sits_labels(samples)`
7. The series of the small run, drawn under a fixed seed from those the encoder was trained on. `encoder_record[["series"]]`
8. The loss of every round of the published training, its seconds and its settings. `encoder_record[["published"]]`
9. The labelled samples of the period, the same number of each class, with their place in the period. `encoder_draw_per_class(encoder_period_samples(samples, "2017-01-01"), per_class = 500L, seed = 20261011L)`
10. Gives each class the colour sits uses to draw it. `samples_class_colors(labels)`
11. The drawn samples as the published encoder embeds them: 64 numbers each. `encoder_encode_samples(samples_subset, encoder_file, list(device = "default", memsize = 4L, multicores = 2L))`
12. Draws one series under a fixed seed and makes its two views as sits makes them in training. `pretrain_views(pretrain_series, seed = 20261011L)`
13. Draws a fixed set of the encoded samples, the same number of each class, under a fixed seed. `embeddings_fixed_set(samples_encoded, labels, 150L, 20261011L)`
14. Writes the list of classes, with their titles and colours, for the website. `site_write_json(samples_class_table(labels, colors, list(Cerradao = "Cerradão", Nat_NonVeg = "Natural non-vegetated")), "../web/src/data/classes.json")`
15. The labelled series of the fixed set, which every run encodes. `pretrain_fixed_samples(samples_subset, encoded = samples_encoded, fixed = fixed_set)`
16. Scores the classes of the fixed set by nearest neighbours on the raw series and on the published encoder's embeddings. `pretrain_knn_reference(pretrain_raw_series(fixed_samples, sits::sits_bands(pretrain_series)), published = embeddings_matrix(samples_encoded)[fixed_set, ], label = fixed_samples$label, k = 5L)`
17. Trains a LeJEPA encoder on the drawn series for each number of epochs, and encodes the fixed set with it. `pretrain_run(pretrain_series, fixed_samples, epochs = snapshot_epochs, settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"))`
18. Checks that each snapshot run has the losses of the first epochs of the full run, exactly. `pretrain_check_trajectory(pretrain_runs)`
19. Puts the fixed set of every run on the first two principal components of the full run, and scores its classes by nearest neighbours. `pretrain_snapshots(pretrain_runs, label = fixed_samples$label, k = 5L)`
20. Writes the record of the published run and of the small run, its snapshots, and the views of one series. `site_write_json(pretrain_record(list(published = published_run, runs = pretrain_runs, snapshots = snapshots, reference = knn_reference, views = views, trajectory = pretrain_trajectory, labels = labels, fixed_label = fixed_samples$label, n_series = nrow(pretrain_series), settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"), knn_k = 5L, encoder = list(repo = "e-sensing/encoders_cerrado", file = "ssl_lejepa_tcnn_model_2017_2024.rds"), file = "pretrain-snapshots.bin")), "../web/src/data/pretrain.json")`
21. Writes the fixed set of every run on one basis, for the website. `site_write_float32(pretrain_snapshot_values(snapshots, pretrain_trajectory), "../web/public/data/pretrain-snapshots.bin")`

The files:

- `pretrain.json`: SHA-256 `f14baa3757fb6ce5dba2a7bf97ddafd92457adfb4e3d8f26ea308cccf466d3cb`. Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash.
- `classes.json`: SHA-256 `e479d0f03cfc2258efe34563f02adc3999d25150b0f991643153d439209b2145`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `pretrain-snapshots.bin`: SHA-256 `ab07ffa8295ceae8ede8db5853fcbdc62463fa8ce14b5475aef7af230f8d4a94`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.

To make them again: `cd pipeline && Rscript make.R`.

## What to try

1. **Scrub the reduced run.** With *Reduced run* pressed, drag along the loss curve, or focus it and use the arrow keys. Under the curve are labelled samples as the encoder placed them at the scrubbed epoch. Only some epochs were recorded; between them the figure says that the cloud is interpolated, and the number beside it is never interpolated.
2. **Step through the snapshots.** Press the snapshot epochs one after the other and read the nearest-neighbour number beside the cloud. Two marks on its scale are fixed: the same number for the raw series, and for the published encoder. See where the small run stands: it passes the first at the snapshot of epoch 16, and it never reaches the second.
3. **Follow one label.** Choose a label in the legend and scrub again: the samples of that label come forward. The encoder never saw the labels; they only colour the dots.
4. **Compare the two runs honestly.** Press *Published run*. It has a curve and no cloud, because it kept no state per epoch. The runs differ in size, so compare the shapes of the curves, not their levels.