# One embedding, band by band

The values of one pixel's embedding as bars, each band as a map, and the window drawn from any three bands.

[From step 5 of From series to embeddings](/embeddings.md).

In the sits book: [Foundational Models and Embeddings](https://e-sensing.github.io/sitsbook/embeddings.html) and [Building regional embeddings: a full example](https://e-sensing.github.io/sitsbook/emb_example.html).

Point at any pixel to read its embedding as bars, point at a bar to see its band alone, or draw the window from any three bands.

## The figure

Recorded by sits: the embeddings cube that sits_encode() wrote for the window.

The figure (recorded: sits_encode() · encoder from sits_ssl_lejepa() · 256 × 256 px · 64 bands) draws the embeddings cube that `sits_encode()` wrote for the window, 256 × 256 pixels of 30 m (7.68 km across) in tile 009010, with 64 values per pixel stored as bands EMB01 to EMB64. The window is drawn with three bands as red, green and blue (EMB11, EMB12 and EMB13, the choice of the sits book), and the 64 values of the selected pixel are drawn below it as signed bars, on one scale for every pixel. Pointing at a bar shows that band alone as a grey map of the window; at the beat of one band, band arrows step through every band. At its last beat three menus choose the band behind each colour, a Surprise me button picks three at random and a Book’s three button brings back the book's; any pixel can be pointed at, clicked or reached with the arrow keys.

| Fact                           | Value                                          |
| ------------------------------ | ---------------------------------------------- |
| Tile                           | 009010                                         |
| Window                         | 256 × 256 pixels, 7.68 km across               |
| Pixel size                     | 30 m                                           |
| Values per pixel               | 64                                             |
| Default colour bands (R, G, B) | EMB11, EMB12, EMB13                            |
| Display stretch                | per band, 5% of the values clipped at each end |

### How it was made

1. Read the labelled samples: The pipeline reads the labelled samples: time series, each with its land cover class. It lists the classes and gives each the colour sits draws it in.
2. Cut out the study window: The pipeline finds the window in its tile of a regular cube and cuts it out of every band and month. It keeps the stored values and marks the months a pixel has none.
3. Fill the missing months: Where a pixel has no value in a month, sits fills one in on a straight line between the months around it. At either end of the series it repeats the nearest value.
4. Encode the labelled samples: The pipeline draws the same number of labelled samples from each class of the period of the embeddings cube, and sits_encode() turns each into its 64 numbers with the published encoder.
5. Read how the encoder was trained: The published encoder keeps a record of its own training: the loss of every round on the training and the validation series, how long each round took, and the settings sits_ssl_lejepa() was called with. The pipeline reads that record.
6. Draw the series of a small run: The encoder also keeps the series it was trained on. The pipeline draws some of them under a fixed seed, for a smaller training run of its own.
7. Store the window's embeddings in one byte: The pipeline cuts the study window out of the embeddings cube: 64 numbers for each pixel. It stores each number in one byte, and checks that the nearest neighbours of the labelled samples give their class almost as often as before: the pipeline stops if the share they get right falls by more than 0.01.
8. Encode the window again: sits_encode() runs the published encoder on the window's monthly images, the call that made the embeddings cube. The pipeline compares what comes out with the published cube, number by number.
9. Train a small encoder, and stop it early: sits_pre_train() trains an encoder of its own with sits_ssl_lejepa() on the drawn series. It trains again from the start for fewer rounds, under the same seed, and the pipeline checks that each shorter run is the start of the long one. sits_encode() embeds the fixed set after each run.
10. Make two views of one series: sits_ssl_lejepa() learns by comparing two views of the same series. The pipeline draws one of the series under a fixed seed and makes its two views as sits makes them in training.
11. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each.
12. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it.

### Reproducibility details

- embeddings.bin, embeddings.json and window.json: A rerun on aarch64-apple-darwin23 gave the same files, byte for byte.
- pretrain.json: Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash.

Every step of the pipeline:

1. Lists the band files of the regular Landsat cube on Hugging Face, at a fixed commit. `hf_cube_files("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85")`
2. Lists the band files of the embeddings cube on Hugging Face, at a fixed commit. `embeddings_cube_files("e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01")`
3. Reads how sits decodes the values of an embeddings cube. `embeddings_cube_values()`
4. The published encoder of the Cerrado, as kept from Hugging Face. `hf_cache_path("e-sensing/encoders_cerrado", "ssl_lejepa_tcnn_model_2017_2024.rds", "model")`
5. The labelled Cerrado samples, as kept from Hugging Face. `hf_cache_path("e-sensing/samples_cerrado", "labelled_samples_cerrado.parquet", "dataset")`
6. The epochs of the snapshot runs, then of the full run. `c(1L, 2L, 4L, 8L, 16L, 30L)`
7. The study window, chosen as pipeline/legacy/R/01_window.R describes. `list(tile = "009010", col_off = 2208L, row_off = 1536L, size = 256L, resolution = 30L, source = "BDC", collection = "LANDSAT-OLI-16D")`
8. Reads the encoder once: the record of its training, and series drawn from those it was trained on. `encoder_contents(encoder_read(encoder_file), 20000L, 20261011L)`
9. Reads the samples into a table, one row per time series. `samples_read_cached(samples_file)`
10. Reads the grid of the window's tile from a band file. `grid_read_tile(hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", cube_files$file[cube_files$tile == study_window$tile][[1L]]))`
11. Lists the land cover classes found in the samples. `sits::sits_labels(samples)`
12. The series of the small run, drawn under a fixed seed from those the encoder was trained on. `encoder_record[["series"]]`
13. The loss of every round of the published training, its seconds and its settings. `encoder_record[["published"]]`
14. The labelled samples of the period, the same number of each class, with their place in the period. `encoder_draw_per_class(encoder_period_samples(samples, "2017-01-01"), per_class = 500L, seed = 20261011L)`
15. Places the window in the tile's grid. `window_place(tile_grid, study_window)`
16. Crops the window out of the 64 bands of the embeddings cube by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/emb-cube", hf_resolve_url("e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01", embeddings_tile_files(embeddings_files, "009010")), window_grid, "e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01", 8L, crop = window_crop_http_files)`
17. The drawn samples as the published encoder embeds them: 64 numbers each. `encoder_encode_samples(samples_subset, encoder_file, list(device = "default", memsize = 4L, multicores = 2L))`
18. Draws one series under a fixed seed and makes its two views as sits makes them in training. `pretrain_views(pretrain_series, seed = 20261011L)`
19. Crops the window out of the cube's band files by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/reg", hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", hf_tile_files(cube_files, study_window$tile, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))), window_grid, "e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", 8L, crop = window_crop_http_files)`
20. Draws a fixed set of the encoded samples, the same number of each class, under a fixed seed. `embeddings_fixed_set(samples_encoded, labels, 150L, 20261011L)`
21. Reads the 64 numbers of every pixel of the window. `embeddings_read_window(embeddings_crops, embeddings_values, study_window$size)`
22. Names the bands and months of the crops. `window_crop_layers(window_crops, study_window$source, study_window$collection, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))`
23. Reads how sits scales the bands and marks missing values. `collection_band_values(study_window$source, study_window$collection, window_layers$bands)`
24. Picks the values of three bands drawn black and white, as sits plots a cube. `embeddings_rgb_stretch(window_embeddings, list(red = "EMB11", green = "EMB12", blue = "EMB13"))`
25. Stores the embeddings of the window and of the samples in one byte per value. `embeddings_to_int8(window_embeddings, embeddings_matrix(samples_encoded), c(0.005, 0.995))`
26. Runs the published encoder on the window's monthly images and compares its output with the published cube. `encoder_compare(encoder_encode_window(window_crops, encoder_file, list(device = "default", memsize = 4L, multicores = 2L, source = "BDC", collection = "LANDSAT-OLI-16D", output_dir = "cache/encode-check"), values = embeddings_values, size = study_window$size), window_embeddings)`
27. The labelled series of the fixed set, which every run encodes. `pretrain_fixed_samples(samples_subset, encoded = samples_encoded, fixed = fixed_set)`
28. Checks that the nearest neighbours of the fixed set give their class as often with the stored bytes. `embeddings_check_int8(embeddings_matrix(samples_encoded)[fixed_set, ], embeddings_restore(embeddings_int8$samples$q[fixed_set, ], embeddings_int8$params), samples_encoded$label[fixed_set], list(k = 5L, max_knn_drop = 0.01))`
29. Scores the classes of the fixed set by nearest neighbours on the raw series and on the published encoder's embeddings. `pretrain_knn_reference(pretrain_raw_series(fixed_samples, sits::sits_bands(pretrain_series)), published = embeddings_matrix(samples_encoded)[fixed_set, ], label = fixed_samples$label, k = 5L)`
30. Trains a LeJEPA encoder on the drawn series for each number of epochs, and encodes the fixed set with it. `pretrain_run(pretrain_series, fixed_samples, epochs = snapshot_epochs, settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"))`
31. Reads the stored values of the crops, missing ones as NA. `window_read_raw(window_crops, window_layers$bands, window_layers$dates, band_scale, study_window$size)`
32. Checks that each snapshot run has the losses of the first epochs of the full run, exactly. `pretrain_check_trajectory(pretrain_runs)`
33. Writes how the window's embeddings are stored, how closely the encoder repeats them, and the stretch of their image. `site_write_json(embeddings_record(list(int8 = embeddings_int8, samples = embeddings_matrix(samples_encoded), values = embeddings_values, check = encode_check, int8_check = int8_check, stretch = emb_rgb_stretch, file = "embeddings.bin", source_files = fs::path_file(embeddings_tile_files(embeddings_files, "009010")), repo = "e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", probs = c(0.005, 0.995), size = study_window$size)), "../web/src/data/embeddings.json")`
34. Writes the window's embeddings, one byte per value, for the website. `site_write_bin(embeddings_window_bytes(embeddings_int8, int8_check), "../web/public/data/embeddings.bin", "int8")`
35. Puts the fixed set of every run on the first two principal components of the full run, and scores its classes by nearest neighbours. `pretrain_snapshots(pretrain_runs, label = fixed_samples$label, k = 5L)`
36. Marks the months each pixel was observed in. `window_observed_mask(window_raw)`
37. Fills the missing months along time with sits::impute_linear(), as sits does before the encoder. `window_fill_series(window_raw, band_scale)`
38. Stores every other pixel's series in one byte per value, each band between two quantiles. `window_export_series(window_series, c(0.001, 0.999 ), 2L)`
39. Picks the reflectance drawn black and white in each colour, the same for every month. `window_export_sprite_stretch(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), c(0.02, 0.98))`
40. Writes the record of the published run and of the small run, its snapshots, and the views of one series. `site_write_json(pretrain_record(list(published = published_run, runs = pretrain_runs, snapshots = snapshots, reference = knn_reference, views = views, trajectory = pretrain_trajectory, labels = labels, fixed_label = fixed_samples$label, n_series = nrow(pretrain_series), settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"), knn_k = 5L, encoder = list(repo = "e-sensing/encoders_cerrado", file = "ssl_lejepa_tcnn_model_2017_2024.rds"), file = "pretrain-snapshots.bin")), "../web/src/data/pretrain.json")`
41. Writes where the window is, and how its series and monthly images are stored, for the website. `site_write_json(window_export_record(study_window, window_grid, window_layers, band_scale, window_mask, series_quantised, series_stretch, cube_files, list(repo = "e-sensing/cerrado_cube_reg_2018", revision = "refs/pr/3", commit = "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85"), list(png_scale = 2L, series_quantiles = c(0.001, 0.999), series_stride = 2L, series_units = "surface reflectance", series_rgb = list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch = c(0.02, 0.98), sprite_columns = 6L, sprite_levels = 32L, sprite_small_tile = 220L, tour_band = "NIR08"), list(series = "series.bin", mask = "series-mask.bin", sprite = "rgb-months.png", sprite_small = "rgb-months-small.png")), "../web/src/data/window.json")`

The files:

- `embeddings.bin`: SHA-256 `d7b1efae1bbcdf2ffeade9f57c94c662e53af94ac0e55a18c0e12af33c0df1e5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `embeddings.json`: SHA-256 `4e6b827b79e9fce5a8b8dca06de368b9ce8534c4ec268fae8f68084ce0822da1`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `window.json`: SHA-256 `60e731843a80b49590366d3313d69d24d23f2c23c914b42ecdff8ff53b06f9d5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `pretrain.json`: SHA-256 `f14baa3757fb6ce5dba2a7bf97ddafd92457adfb4e3d8f26ea308cccf466d3cb`. Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash.

To make them again: `cd pipeline && Rscript make.R`.

## What to try

1. **Cross a boundary.** Move the pointer over the image, from a field into the vegetation next to it. Inside a field the bars barely move; at its edge the whole profile changes shape.
2. **Look at one band alone.** Point at any bar, or focus the bars and walk along them with the arrow keys: the image shows that band alone, in grey. Which bands tell the fields from the vegetation around them, and which tell fields from each other? No band has a name, so looking is the only way to find out.
3. **Change the colours, not the picture.** Press *Surprise me* a few times. The colours change completely and the shapes do not: what separates one place from the next is in the vectors.
4. **Make a grey image from three menus.** Set R, G and B to the same band. The image is then that band alone, as a bar shows it.
5. **Read the bars that are cut.** The axis of the bars is fixed, so that two pixels can be compared. A bar that would pass it ends in a small point; point at it to read its true value.