# From a series to a vector

The raw series of one pixel as a table, with the months sits filled in, and the same pixel as the values its encoder writes.

[From step 2 of From series to embeddings](/embeddings.md).

In the sits book: [Foundational Models and Embeddings](https://e-sensing.github.io/sitsbook/embeddings.html) and [Building regional embeddings: a full example](https://e-sensing.github.io/sitsbook/emb_example.html).

Pick a pixel on the image, switch or slide between its table of reflectances and its embedding, and read any value before and after the encoder.

## The figure

Before: the values of one pixel, each row shaded from its lowest value to its highest. Round cells are months without a clear observation, filled by sits. After: the values sits_encode() wrote for the same pixel, warm above zero and cool below, the colour growing with the square root of the size. The funnel between them is a diagram, not a computation. Both ends are recorded: encoding the series of this window again gave the published embeddings back.

The figure (recorded: BDC LANDSAT-OLI-16D · tile 009010 · 24 months, 2017-01 to 2018-12) shows the selected pixel twice. First as a table of its raw series, 6 bands by 24 months, 144 values of surface reflectance, each row shaded from its lowest value to its highest; months with no clear view, which sits filled with impute_linear(), are round cells, and a legend counts them for the pixel. Then a funnel labelled as a diagram, the encoder, and the row of the 64 values sits_encode() wrote for the same pixel, warm above zero and cool below. At its last beat a small false-colour image of the window picks another pixel (click, or the arrow keys), a switch and a slider move between the table and the row, and pointing at a cell, or the arrow keys on the focused figure, reads out its value.

| Fact                                                                 | Value                                                |
| -------------------------------------------------------------------- | ---------------------------------------------------- |
| Values per pixel before                                              | 144 (6 bands × 24 months)                            |
| Values per pixel after                                               | 64                                                   |
| Bands                                                                | BLUE, GREEN, RED, NIR08, SWIR16, SWIR22              |
| Units of the series                                                  | surface reflectance                                  |
| Pixel-months of the window with no observation                       | 45%                                                  |
| Months with no observation anywhere in the window                    | 8 of 24                                              |
| Gaps filled by                                                       | impute_linear()                                      |
| Series kept for                                                      | every second pixel                                   |
| Encoding the window's series again, against the published embeddings | all 4,194,304 values identical, largest difference 0 |

### How it was made

1. Read the labelled samples: The pipeline reads the labelled samples: time series, each with its land cover class. It lists the classes and gives each the colour sits draws it in.
2. Cut out the study window: The pipeline finds the window in its tile of a regular cube and cuts it out of every band and month. It keeps the stored values and marks the months a pixel has none.
3. Fill the missing months: Where a pixel has no value in a month, sits fills one in on a straight line between the months around it. At either end of the series it repeats the nearest value.
4. Encode the labelled samples: The pipeline draws the same number of labelled samples from each class of the period of the embeddings cube, and sits_encode() turns each into its 64 numbers with the published encoder.
5. Read how the encoder was trained: The published encoder keeps a record of its own training: the loss of every round on the training and the validation series, how long each round took, and the settings sits_ssl_lejepa() was called with. The pipeline reads that record.
6. Draw the series of a small run: The encoder also keeps the series it was trained on. The pipeline draws some of them under a fixed seed, for a smaller training run of its own.
7. Store the window's embeddings in one byte: The pipeline cuts the study window out of the embeddings cube: 64 numbers for each pixel. It stores each number in one byte, and checks that the nearest neighbours of the labelled samples give their class almost as often as before: the pipeline stops if the share they get right falls by more than 0.01.
8. Encode the window again: sits_encode() runs the published encoder on the window's monthly images, the call that made the embeddings cube. The pipeline compares what comes out with the published cube, number by number.
9. Train a small encoder, and stop it early: sits_pre_train() trains an encoder of its own with sits_ssl_lejepa() on the drawn series. It trains again from the start for fewer rounds, under the same seed, and the pipeline checks that each shorter run is the start of the long one. sits_encode() embeds the fixed set after each run.
10. Make two views of one series: sits_ssl_lejepa() learns by comparing two views of the same series. The pipeline draws one of the series under a fixed seed and makes its two views as sits makes them in training.
11. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each.
12. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it.

### Reproducibility details

- rgb-months.png, series.bin, series-mask.bin, embeddings.bin, window.json and embeddings.json: A rerun on aarch64-apple-darwin23 gave the same files, byte for byte.
- pretrain.json: Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash.

Every step of the pipeline:

1. Lists the band files of the regular Landsat cube on Hugging Face, at a fixed commit. `hf_cube_files("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85")`
2. Lists the band files of the embeddings cube on Hugging Face, at a fixed commit. `embeddings_cube_files("e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01")`
3. Reads how sits decodes the values of an embeddings cube. `embeddings_cube_values()`
4. The published encoder of the Cerrado, as kept from Hugging Face. `hf_cache_path("e-sensing/encoders_cerrado", "ssl_lejepa_tcnn_model_2017_2024.rds", "model")`
5. The labelled Cerrado samples, as kept from Hugging Face. `hf_cache_path("e-sensing/samples_cerrado", "labelled_samples_cerrado.parquet", "dataset")`
6. The epochs of the snapshot runs, then of the full run. `c(1L, 2L, 4L, 8L, 16L, 30L)`
7. The study window, chosen as pipeline/legacy/R/01_window.R describes. `list(tile = "009010", col_off = 2208L, row_off = 1536L, size = 256L, resolution = 30L, source = "BDC", collection = "LANDSAT-OLI-16D")`
8. Reads the encoder once: the record of its training, and series drawn from those it was trained on. `encoder_contents(encoder_read(encoder_file), 20000L, 20261011L)`
9. Reads the samples into a table, one row per time series. `samples_read_cached(samples_file)`
10. Reads the grid of the window's tile from a band file. `grid_read_tile(hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", cube_files$file[cube_files$tile == study_window$tile][[1L]]))`
11. Lists the land cover classes found in the samples. `sits::sits_labels(samples)`
12. The series of the small run, drawn under a fixed seed from those the encoder was trained on. `encoder_record[["series"]]`
13. The loss of every round of the published training, its seconds and its settings. `encoder_record[["published"]]`
14. The labelled samples of the period, the same number of each class, with their place in the period. `encoder_draw_per_class(encoder_period_samples(samples, "2017-01-01"), per_class = 500L, seed = 20261011L)`
15. Places the window in the tile's grid. `window_place(tile_grid, study_window)`
16. Crops the window out of the 64 bands of the embeddings cube by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/emb-cube", hf_resolve_url("e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01", embeddings_tile_files(embeddings_files, "009010")), window_grid, "e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01", 8L, crop = window_crop_http_files)`
17. The drawn samples as the published encoder embeds them: 64 numbers each. `encoder_encode_samples(samples_subset, encoder_file, list(device = "default", memsize = 4L, multicores = 2L))`
18. Draws one series under a fixed seed and makes its two views as sits makes them in training. `pretrain_views(pretrain_series, seed = 20261011L)`
19. Crops the window out of the cube's band files by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/reg", hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", hf_tile_files(cube_files, study_window$tile, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))), window_grid, "e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", 8L, crop = window_crop_http_files)`
20. Draws a fixed set of the encoded samples, the same number of each class, under a fixed seed. `embeddings_fixed_set(samples_encoded, labels, 150L, 20261011L)`
21. Reads the 64 numbers of every pixel of the window. `embeddings_read_window(embeddings_crops, embeddings_values, study_window$size)`
22. Names the bands and months of the crops. `window_crop_layers(window_crops, study_window$source, study_window$collection, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))`
23. Reads how sits scales the bands and marks missing values. `collection_band_values(study_window$source, study_window$collection, window_layers$bands)`
24. Picks the values of three bands drawn black and white, as sits plots a cube. `embeddings_rgb_stretch(window_embeddings, list(red = "EMB11", green = "EMB12", blue = "EMB13"))`
25. Stores the embeddings of the window and of the samples in one byte per value. `embeddings_to_int8(window_embeddings, embeddings_matrix(samples_encoded), c(0.005, 0.995))`
26. Runs the published encoder on the window's monthly images and compares its output with the published cube. `encoder_compare(encoder_encode_window(window_crops, encoder_file, list(device = "default", memsize = 4L, multicores = 2L, source = "BDC", collection = "LANDSAT-OLI-16D", output_dir = "cache/encode-check"), values = embeddings_values, size = study_window$size), window_embeddings)`
27. The labelled series of the fixed set, which every run encodes. `pretrain_fixed_samples(samples_subset, encoded = samples_encoded, fixed = fixed_set)`
28. Checks that the nearest neighbours of the fixed set give their class as often with the stored bytes. `embeddings_check_int8(embeddings_matrix(samples_encoded)[fixed_set, ], embeddings_restore(embeddings_int8$samples$q[fixed_set, ], embeddings_int8$params), samples_encoded$label[fixed_set], list(k = 5L, max_knn_drop = 0.01))`
29. Scores the classes of the fixed set by nearest neighbours on the raw series and on the published encoder's embeddings. `pretrain_knn_reference(pretrain_raw_series(fixed_samples, sits::sits_bands(pretrain_series)), published = embeddings_matrix(samples_encoded)[fixed_set, ], label = fixed_samples$label, k = 5L)`
30. Trains a LeJEPA encoder on the drawn series for each number of epochs, and encodes the fixed set with it. `pretrain_run(pretrain_series, fixed_samples, epochs = snapshot_epochs, settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"))`
31. Reads the stored values of the crops, missing ones as NA. `window_read_raw(window_crops, window_layers$bands, window_layers$dates, band_scale, study_window$size)`
32. Checks that each snapshot run has the losses of the first epochs of the full run, exactly. `pretrain_check_trajectory(pretrain_runs)`
33. Writes how the window's embeddings are stored, how closely the encoder repeats them, and the stretch of their image. `site_write_json(embeddings_record(list(int8 = embeddings_int8, samples = embeddings_matrix(samples_encoded), values = embeddings_values, check = encode_check, int8_check = int8_check, stretch = emb_rgb_stretch, file = "embeddings.bin", source_files = fs::path_file(embeddings_tile_files(embeddings_files, "009010")), repo = "e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", probs = c(0.005, 0.995), size = study_window$size)), "../web/src/data/embeddings.json")`
34. Writes the window's embeddings, one byte per value, for the website. `site_write_bin(embeddings_window_bytes(embeddings_int8, int8_check), "../web/public/data/embeddings.bin", "int8")`
35. Puts the fixed set of every run on the first two principal components of the full run, and scores its classes by nearest neighbours. `pretrain_snapshots(pretrain_runs, label = fixed_samples$label, k = 5L)`
36. Marks the months each pixel was observed in. `window_observed_mask(window_raw)`
37. Fills the missing months along time with sits::impute_linear(), as sits does before the encoder. `window_fill_series(window_raw, band_scale)`
38. Stores every other pixel's series in one byte per value, each band between two quantiles. `window_export_series(window_series, c(0.001, 0.999 ), 2L)`
39. Picks the reflectance drawn black and white in each colour, the same for every month. `window_export_sprite_stretch(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), c(0.02, 0.98))`
40. Writes the record of the published run and of the small run, its snapshots, and the views of one series. `site_write_json(pretrain_record(list(published = published_run, runs = pretrain_runs, snapshots = snapshots, reference = knn_reference, views = views, trajectory = pretrain_trajectory, labels = labels, fixed_label = fixed_samples$label, n_series = nrow(pretrain_series), settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"), knn_k = 5L, encoder = list(repo = "e-sensing/encoders_cerrado", file = "ssl_lejepa_tcnn_model_2017_2024.rds"), file = "pretrain-snapshots.bin")), "../web/src/data/pretrain.json")`
41. Writes which months of the window were observed, for the website. `site_write_bin(window_export_row_major(window_export_mask(window_mask, 2L)), "../web/public/data/series-mask.bin", "uint8")`
42. Writes the monthly series of the window for the website. `site_write_bin(window_export_row_major(series_quantised$q), "../web/public/data/series.bin", "uint8")`
43. Draws the window in every month as false-colour images in one sheet, for the website. `site_write_png(window_export_sprite(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch, 32L, 6L), "../web/public/data/rgb-months.png")`
44. Writes where the window is, and how its series and monthly images are stored, for the website. `site_write_json(window_export_record(study_window, window_grid, window_layers, band_scale, window_mask, series_quantised, series_stretch, cube_files, list(repo = "e-sensing/cerrado_cube_reg_2018", revision = "refs/pr/3", commit = "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85"), list(png_scale = 2L, series_quantiles = c(0.001, 0.999), series_stride = 2L, series_units = "surface reflectance", series_rgb = list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch = c(0.02, 0.98), sprite_columns = 6L, sprite_levels = 32L, sprite_small_tile = 220L, tour_band = "NIR08"), list(series = "series.bin", mask = "series-mask.bin", sprite = "rgb-months.png", sprite_small = "rgb-months-small.png")), "../web/src/data/window.json")`

The files:

- `rgb-months.png`: SHA-256 `89424cbbdeeda1cd4d46f0e9b2b6c70708250c23cf3b4a2cb4064a46d8439d8e`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `series.bin`: SHA-256 `0a5dacbfea580c475943a977d3d2fc693788a2449acac1be59416a5b743de112`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `series-mask.bin`: SHA-256 `b6c4d861d4986323aa8937eb7efb59cb6cac8e4c8075f41162e82de1babdeecd`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `embeddings.bin`: SHA-256 `d7b1efae1bbcdf2ffeade9f57c94c662e53af94ac0e55a18c0e12af33c0df1e5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `window.json`: SHA-256 `60e731843a80b49590366d3313d69d24d23f2c23c914b42ecdff8ff53b06f9d5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `embeddings.json`: SHA-256 `4e6b827b79e9fce5a8b8dca06de368b9ce8534c4ec268fae8f68084ce0822da1`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `pretrain.json`: SHA-256 `f14baa3757fb6ce5dba2a7bf97ddafd92457adfb4e3d8f26ea308cccf466d3cb`. Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash.

To make them again: `cd pipeline && Rscript make.R`.

## What to try

1. **Slide through the encoder.** Press *Table*, then drag the slider beside it slowly towards *Row*. The passage is a drawing of the operation, labelled as one: no cell of the row comes from one cell of the table. The two ends are data.
2. **Follow another place.** Pick a pixel on the small image, on a bright patch and then on a dark one. Compare the two tables, then press *Row* and compare the two rows.
3. **Find the months nobody saw.** On the table, look for a column that is round in every row. Then pick other pixels: that column stays round, because in those months no pixel of the window was observed.
4. **Read a value.** Point at a cell, or focus the figure and use the arrow keys. Before the encoder a value is a reflectance with a band and a date. After it, a value has only a position.
5. **Read the check.** The step this figure comes from holds, in its details, a recorded comparison: how often the nearest neighbours of a labelled place share its label, with the places compared as raw series and as embeddings.