# From series to embeddings How does the time series of a place become a few numbers that say what it is? One window of the Brazilian Cerrado, month by month: first the time series of one pixel and why its raw values are awkward for a classifier. Then how an encoder learns from series nobody labelled, what two recorded training runs gave, and the numbers it writes for every pixel. In the sits book: [Foundational Models and Embeddings](https://e-sensing.github.io/sitsbook/embeddings.html), [Embedding Methods in SITS](https://e-sensing.github.io/sitsbook/emb_methods.html) and [Building regional embeddings: a full example](https://e-sensing.github.io/sitsbook/emb_example.html). Before this: [Building data cubes from image collections](/regular-cubes.md). ## 1. What does the satellite give us for one place? In the sits book: [Producing a two year data cube for the Cerrado](https://e-sensing.github.io/sitsbook/emb_example.html#producing-a-two-year-data-cube-for-the-cerrado). The book asks sits to turn the Landsat images of the Cerrado, from BDC, into one image a month, cut to the limits of the biome. The book's code: ``` cerrado_cube_reg_2018 <- sits_regularize( cube = cerrado_cube_local, period = "P1M", roi = cerrado_limits, multicores = 8, output_dir = tempdir_r_data, progress = TRUE ) ``` Where the sits book writes it: [emb_example.qmd, lines 234–241](https://github.com/e-sensing/sitsbook/blob/bc75e250e50bc6fb8222593973b8dfe3b692201f/emb_example.qmd#L234-L241). A window of the Brazilian Cerrado, 256 × 256 pixels of 30 m, 7.68 km across, as Landsat saw it in May 2017. The colours are false: SWIR16, NIR08 and RED as red, green and blue. Infrared says more about plants and soil than visible light, so growing vegetation comes out green and bare soil pink. The satellite comes back, and the cube keeps one image a month: 24 months, from January 2017 to December 2018. The book's cube is regular: one image a month, every month on the same grid. [Building data cubes from image collections](/regular-cubes.md) shows how sits makes one. Press Play, or drag along the strip. Now read one pixel, the one marked on the image, through the 24 months. The sensor measures 6 bands, BLUE, GREEN, RED, NIR08, SWIR16 and SWIR22, so the pixel has 6 lines, one per band. That set of lines is a time series: all the satellite knows about these 30 m of ground. How sits reads the series of a place: [How does one place become a time series?](/time-series.md) 45% of the pixel-months of this window were not observed: 8 of the 24 months have no observation at all, and only 8 are complete: they fall from May to August, the empty ones from September to March. The strip shows how much was observed each month, and the image fades the rest. sits fills the gaps with `impute_linear()`: a hollow dot is a filled value. Now point at the image, or click it. ### The figure The window month by month: the strip of months, one pixel's bands as lines, and the months nobody observed. The figure (recorded: BDC LANDSAT-OLI-16D, tile 009010, 24 months from January 2017 to December 2018) draws the 256 × 256 pixel window, 7.68 km across, in one month as a false-colour image (SWIR16, NIR08 and RED as red, green and blue), a strip of the 24 months under it with a Play button, and the 6 bands of one pixel (BLUE, GREEN, RED, NIR08, SWIR16 and SWIR22) as lines over the months, in surface reflectance. At its last beat the strip says how much of the window was observed each month, the image fades the pixels that were not, and a hollow dot on a dashed line is a value filled with impute_linear(); any pixel can then be pointed at, clicked or reached with the arrow keys. It opens on May 2017, the first month observed everywhere in the window. | Month | Share of the window observed | | -------------- | ---------------------------- | | January 2017 | none (filled) | | February 2017 | none (filled) | | March 2017 | none (filled) | | April 2017 | 61% | | May 2017 | 100% | | June 2017 | 100% | | July 2017 | 100% | | August 2017 | 100% | | September 2017 | none (filled) | | October 2017 | none (filled) | | November 2017 | 88% | | December 2017 | none (filled) | | January 2018 | 77% | | February 2018 | none (filled) | | March 2018 | 80% | | April 2018 | 50% | | May 2018 | 100% | | June 2018 | 100% | | July 2018 | 100% | | August 2018 | 100% | | September 2018 | 97% | | October 2018 | 62% | | November 2018 | none (filled) | | December 2018 | < 1% | #### How it was made 1. Cut out the study window: The pipeline finds the window in its tile of a regular cube and cuts it out of every band and month. It keeps the stored values and marks the months a pixel has none. 2. Fill the missing months: Where a pixel has no value in a month, sits fills one in on a straight line between the months around it. At either end of the series it repeats the nearest value. 3. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each. 4. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - A rerun on aarch64-apple-darwin23 gave the same files, byte for byte. Every step of the pipeline: 1. Lists the band files of the regular Landsat cube on Hugging Face, at a fixed commit. `hf_cube_files("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85")` 2. The study window, chosen as pipeline/legacy/R/01_window.R describes. `list(tile = "009010", col_off = 2208L, row_off = 1536L, size = 256L, resolution = 30L, source = "BDC", collection = "LANDSAT-OLI-16D")` 3. Reads the grid of the window's tile from a band file. `grid_read_tile(hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", cube_files$file[cube_files$tile == study_window$tile][[1L]]))` 4. Places the window in the tile's grid. `window_place(tile_grid, study_window)` 5. Crops the window out of the cube's band files by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/reg", hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", hf_tile_files(cube_files, study_window$tile, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))), window_grid, "e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", 8L, crop = window_crop_http_files)` 6. Names the bands and months of the crops. `window_crop_layers(window_crops, study_window$source, study_window$collection, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))` 7. Reads how sits scales the bands and marks missing values. `collection_band_values(study_window$source, study_window$collection, window_layers$bands)` 8. Reads the stored values of the crops, missing ones as NA. `window_read_raw(window_crops, window_layers$bands, window_layers$dates, band_scale, study_window$size)` 9. Marks the months each pixel was observed in. `window_observed_mask(window_raw)` 10. Fills the missing months along time with sits::impute_linear(), as sits does before the encoder. `window_fill_series(window_raw, band_scale)` 11. Stores every other pixel's series in one byte per value, each band between two quantiles. `window_export_series(window_series, c(0.001, 0.999 ), 2L)` 12. Picks the reflectance drawn black and white in each colour, the same for every month. `window_export_sprite_stretch(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), c(0.02, 0.98))` 13. Writes which months of the window were observed, for the website. `site_write_bin(window_export_row_major(window_export_mask(window_mask, 2L)), "../web/public/data/series-mask.bin", "uint8")` 14. Writes the monthly series of the window for the website. `site_write_bin(window_export_row_major(series_quantised$q), "../web/public/data/series.bin", "uint8")` 15. Draws the window in every month as false-colour images in one sheet, for the website. `site_write_png(window_export_sprite(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch, 32L, 6L), "../web/public/data/rgb-months.png")` 16. Writes where the window is, and how its series and monthly images are stored, for the website. `site_write_json(window_export_record(study_window, window_grid, window_layers, band_scale, window_mask, series_quantised, series_stretch, cube_files, list(repo = "e-sensing/cerrado_cube_reg_2018", revision = "refs/pr/3", commit = "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85"), list(png_scale = 2L, series_quantiles = c(0.001, 0.999), series_stride = 2L, series_units = "surface reflectance", series_rgb = list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch = c(0.02, 0.98), sprite_columns = 6L, sprite_levels = 32L, sprite_small_tile = 220L, tour_band = "NIR08"), list(series = "series.bin", mask = "series-mask.bin", sprite = "rgb-months.png", sprite_small = "rgb-months-small.png")), "../web/src/data/window.json")` The files: - `rgb-months.png`: SHA-256 `89424cbbdeeda1cd4d46f0e9b2b6c70708250c23cf3b4a2cb4064a46d8439d8e`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `series.bin`: SHA-256 `0a5dacbfea580c475943a977d3d2fc693788a2449acac1be59416a5b743de112`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `series-mask.bin`: SHA-256 `b6c4d861d4986323aa8937eb7efb59cb6cac8e4c8075f41162e82de1babdeecd`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `window.json`: SHA-256 `60e731843a80b49590366d3313d69d24d23f2c23c914b42ecdff8ff53b06f9d5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. To make them again: `cd pipeline && Rscript make.R`. Try it: [A pixel through time](/interactive-images/series.md) ### The cube these images come from The images are a 256 × 256 pixel crop of tile 009010 of the regular cube e-sensing/cerrado_cube_reg_2018 on Hugging Face: 12 tiles, 6 bands, 24 months. It is the cube the encoder of this page read. The sits book makes it from the BDC collection LANDSAT-OLI-16D: it copies the tiles of the Cerrado with `sits_cube_copy()`, makes them regular with one image a month with the call below, and reads the cube back with `sits_from_hf()`. ``` cerrado_cube_reg_2018 <- sits_regularize( cube = cerrado_cube_local, period = "P1M", roi = cerrado_limits, multicores = 8, output_dir = tempdir_r_data, progress = TRUE ) ``` Where the sits book writes it: [emb_example.qmd, lines 234–241](https://github.com/e-sensing/sitsbook/blob/bc75e250e50bc6fb8222593973b8dfe3b692201f/emb_example.qmd#L234-L241). Series are kept for every second pixel in each direction (128 × 128 of the 256 × 256 pixels); a pixel you pick shows the series of the kept pixel at or just before it. One stretch for every month: in each colour, the reflectance at the 2nd percentile of all months is drawn black and the one at the 98th white. | Colour | Band | Black | White | | ------ | ------ | ------ | ------ | | red | SWIR16 | 0.0968 | 0.4926 | | green | NIR08 | 0.1818 | 0.4379 | | blue | RED | 0.0184 | 0.1879 | ## 2. Why not classify the raw series? In the sits book: [What are embeddings?](https://e-sensing.github.io/sitsbook/embeddings.html#what-are-embeddings). The book asks sits to pass the series of every pixel of the Cerrado cube through the encoder trained with `sits_ssl_lejepa()`, which writes 64 values for each pixel. The book's code: ``` lejepa_embedding <- sits_encode( data = cerrado_cube_reg_2018, encoder = lejepa_encoder, memsize = 12, multicores = 6, gpu_memory = 12, batch_size = 4096, output_dir = tempdir_r_emb ) ``` Where the sits book writes it: [emb_example.qmd, lines 436–444](https://github.com/e-sensing/sitsbook/blob/bc75e250e50bc6fb8222593973b8dfe3b692201f/emb_example.qmd#L436-L444). The pixel of step 1 as a table: one row per band, one column per month, 6 × 24 = 144 values for one place. The stronger a cell, the higher its value within its row. Series compared as they come, value against value: [Do all places behave the same?](/time-series.md) Not all of them were measured. Round cells are months with no clear view, which sits filled with `impute_linear()`. Across the window, 45% of the pixel-months were filled. A classifier would read them as if they were seen. And so many numbers for one place are a lot: the book notes that patterns in so many dimensions are costly to find, and unreliable without many training samples. `sits_encode()` hands the 144 values to an encoder, a network that rewrites them. [Step 3](/embeddings.md) tells how it learned to. The funnel is a drawing: inside the encoder there is arithmetic on all the values at once, not movement. 64 values come back for the pixel, warm above zero and cool below. No cell of the row comes from one cell of the table. Both ends are recorded: encoding the window again with the published encoder gave the published values back, identical in all 4,194,304 values. Try another place on the small image, or on [step 1's image](/embeddings.md), which picks the same pixel at full size, and switch between the table and the row. Does the short list keep what matters? The details hold a recorded check. What a classifier makes of these numbers: [Classifying EO embeddings for land use and land cover applications](https://e-sensing.github.io/sitsbook/emb_example.html#classifying-eo-embeddings-for-land-use-and-land-cover-applications) ### The figure Before: the values of one pixel, each row shaded from its lowest value to its highest. Round cells are months without a clear observation, filled by sits. After: the values sits_encode() wrote for the same pixel, warm above zero and cool below, the colour growing with the square root of the size. The funnel between them is a diagram, not a computation. Both ends are recorded: encoding the series of this window again gave the published embeddings back. The figure (recorded: BDC LANDSAT-OLI-16D · tile 009010 · 24 months, 2017-01 to 2018-12) shows the selected pixel twice. First as a table of its raw series, 6 bands by 24 months, 144 values of surface reflectance, each row shaded from its lowest value to its highest; months with no clear view, which sits filled with impute_linear(), are round cells, and a legend counts them for the pixel. Then a funnel labelled as a diagram, the encoder, and the row of the 64 values sits_encode() wrote for the same pixel, warm above zero and cool below. At its last beat a small false-colour image of the window picks another pixel (click, or the arrow keys), a switch and a slider move between the table and the row, and pointing at a cell, or the arrow keys on the focused figure, reads out its value. | Fact | Value | | -------------------------------------------------------------------- | ---------------------------------------------------- | | Values per pixel before | 144 (6 bands × 24 months) | | Values per pixel after | 64 | | Bands | BLUE, GREEN, RED, NIR08, SWIR16, SWIR22 | | Units of the series | surface reflectance | | Pixel-months of the window with no observation | 45% | | Months with no observation anywhere in the window | 8 of 24 | | Gaps filled by | impute_linear() | | Series kept for | every second pixel | | Encoding the window's series again, against the published embeddings | all 4,194,304 values identical, largest difference 0 | #### A recorded check: nearest neighbours Fewer numbers would be no gain if they forgot what a place is. One recorded check: take 1,532 places whose land cover is known, and for each of them let the 5 most similar others vote on its label. | Places compared as | Values per place | Share whose neighbours vote for their label | | ------------------ | ---------------- | ------------------------------------------- | | raw series | 144 | 82.3% | | embeddings | 64 | 90.2% | Among 1,532 labelled samples, the 5 nearest neighbours of a sample vote for its own label for 82.3% of the samples when the samples are raw series, and for 90.2% when they are embeddings. +7.9 points, with 80 fewer numbers per place. So as embeddings a place's nearest neighbours vote for its own label more often. Not always: the vote still fails for 9.8%. How: 1,532 labelled samples · 5 nearest neighbours vote · the sample itself left out; at most 150 samples of each of 11 classes · Euclidean distance · encoder sits_ssl_lejepa(), 647,967 series. Recorded run. #### How it was made 1. Read the labelled samples: The pipeline reads the labelled samples: time series, each with its land cover class. It lists the classes and gives each the colour sits draws it in. 2. Cut out the study window: The pipeline finds the window in its tile of a regular cube and cuts it out of every band and month. It keeps the stored values and marks the months a pixel has none. 3. Fill the missing months: Where a pixel has no value in a month, sits fills one in on a straight line between the months around it. At either end of the series it repeats the nearest value. 4. Encode the labelled samples: The pipeline draws the same number of labelled samples from each class of the period of the embeddings cube, and sits_encode() turns each into its 64 numbers with the published encoder. 5. Read how the encoder was trained: The published encoder keeps a record of its own training: the loss of every round on the training and the validation series, how long each round took, and the settings sits_ssl_lejepa() was called with. The pipeline reads that record. 6. Draw the series of a small run: The encoder also keeps the series it was trained on. The pipeline draws some of them under a fixed seed, for a smaller training run of its own. 7. Store the window's embeddings in one byte: The pipeline cuts the study window out of the embeddings cube: 64 numbers for each pixel. It stores each number in one byte, and checks that the nearest neighbours of the labelled samples give their class almost as often as before: the pipeline stops if the share they get right falls by more than 0.01. 8. Encode the window again: sits_encode() runs the published encoder on the window's monthly images, the call that made the embeddings cube. The pipeline compares what comes out with the published cube, number by number. 9. Train a small encoder, and stop it early: sits_pre_train() trains an encoder of its own with sits_ssl_lejepa() on the drawn series. It trains again from the start for fewer rounds, under the same seed, and the pipeline checks that each shorter run is the start of the long one. sits_encode() embeds the fixed set after each run. 10. Make two views of one series: sits_ssl_lejepa() learns by comparing two views of the same series. The pipeline draws one of the series under a fixed seed and makes its two views as sits makes them in training. 11. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each. 12. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - rgb-months.png, series.bin, series-mask.bin, embeddings.bin, window.json and embeddings.json: A rerun on aarch64-apple-darwin23 gave the same files, byte for byte. - pretrain.json: Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash. Every step of the pipeline: 1. Lists the band files of the regular Landsat cube on Hugging Face, at a fixed commit. `hf_cube_files("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85")` 2. Lists the band files of the embeddings cube on Hugging Face, at a fixed commit. `embeddings_cube_files("e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01")` 3. Reads how sits decodes the values of an embeddings cube. `embeddings_cube_values()` 4. The published encoder of the Cerrado, as kept from Hugging Face. `hf_cache_path("e-sensing/encoders_cerrado", "ssl_lejepa_tcnn_model_2017_2024.rds", "model")` 5. The labelled Cerrado samples, as kept from Hugging Face. `hf_cache_path("e-sensing/samples_cerrado", "labelled_samples_cerrado.parquet", "dataset")` 6. The epochs of the snapshot runs, then of the full run. `c(1L, 2L, 4L, 8L, 16L, 30L)` 7. The study window, chosen as pipeline/legacy/R/01_window.R describes. `list(tile = "009010", col_off = 2208L, row_off = 1536L, size = 256L, resolution = 30L, source = "BDC", collection = "LANDSAT-OLI-16D")` 8. Reads the encoder once: the record of its training, and series drawn from those it was trained on. `encoder_contents(encoder_read(encoder_file), 20000L, 20261011L)` 9. Reads the samples into a table, one row per time series. `samples_read_cached(samples_file)` 10. Reads the grid of the window's tile from a band file. `grid_read_tile(hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", cube_files$file[cube_files$tile == study_window$tile][[1L]]))` 11. Lists the land cover classes found in the samples. `sits::sits_labels(samples)` 12. The series of the small run, drawn under a fixed seed from those the encoder was trained on. `encoder_record[["series"]]` 13. The loss of every round of the published training, its seconds and its settings. `encoder_record[["published"]]` 14. The labelled samples of the period, the same number of each class, with their place in the period. `encoder_draw_per_class(encoder_period_samples(samples, "2017-01-01"), per_class = 500L, seed = 20261011L)` 15. Places the window in the tile's grid. `window_place(tile_grid, study_window)` 16. Crops the window out of the 64 bands of the embeddings cube by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/emb-cube", hf_resolve_url("e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01", embeddings_tile_files(embeddings_files, "009010")), window_grid, "e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01", 8L, crop = window_crop_http_files)` 17. The drawn samples as the published encoder embeds them: 64 numbers each. `encoder_encode_samples(samples_subset, encoder_file, list(device = "default", memsize = 4L, multicores = 2L))` 18. Draws one series under a fixed seed and makes its two views as sits makes them in training. `pretrain_views(pretrain_series, seed = 20261011L)` 19. Crops the window out of the cube's band files by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/reg", hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", hf_tile_files(cube_files, study_window$tile, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))), window_grid, "e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", 8L, crop = window_crop_http_files)` 20. Draws a fixed set of the encoded samples, the same number of each class, under a fixed seed. `embeddings_fixed_set(samples_encoded, labels, 150L, 20261011L)` 21. Reads the 64 numbers of every pixel of the window. `embeddings_read_window(embeddings_crops, embeddings_values, study_window$size)` 22. Names the bands and months of the crops. `window_crop_layers(window_crops, study_window$source, study_window$collection, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))` 23. Reads how sits scales the bands and marks missing values. `collection_band_values(study_window$source, study_window$collection, window_layers$bands)` 24. Picks the values of three bands drawn black and white, as sits plots a cube. `embeddings_rgb_stretch(window_embeddings, list(red = "EMB11", green = "EMB12", blue = "EMB13"))` 25. Stores the embeddings of the window and of the samples in one byte per value. `embeddings_to_int8(window_embeddings, embeddings_matrix(samples_encoded), c(0.005, 0.995))` 26. Runs the published encoder on the window's monthly images and compares its output with the published cube. `encoder_compare(encoder_encode_window(window_crops, encoder_file, list(device = "default", memsize = 4L, multicores = 2L, source = "BDC", collection = "LANDSAT-OLI-16D", output_dir = "cache/encode-check"), values = embeddings_values, size = study_window$size), window_embeddings)` 27. The labelled series of the fixed set, which every run encodes. `pretrain_fixed_samples(samples_subset, encoded = samples_encoded, fixed = fixed_set)` 28. Checks that the nearest neighbours of the fixed set give their class as often with the stored bytes. `embeddings_check_int8(embeddings_matrix(samples_encoded)[fixed_set, ], embeddings_restore(embeddings_int8$samples$q[fixed_set, ], embeddings_int8$params), samples_encoded$label[fixed_set], list(k = 5L, max_knn_drop = 0.01))` 29. Scores the classes of the fixed set by nearest neighbours on the raw series and on the published encoder's embeddings. `pretrain_knn_reference(pretrain_raw_series(fixed_samples, sits::sits_bands(pretrain_series)), published = embeddings_matrix(samples_encoded)[fixed_set, ], label = fixed_samples$label, k = 5L)` 30. Trains a LeJEPA encoder on the drawn series for each number of epochs, and encodes the fixed set with it. `pretrain_run(pretrain_series, fixed_samples, epochs = snapshot_epochs, settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"))` 31. Reads the stored values of the crops, missing ones as NA. `window_read_raw(window_crops, window_layers$bands, window_layers$dates, band_scale, study_window$size)` 32. Checks that each snapshot run has the losses of the first epochs of the full run, exactly. `pretrain_check_trajectory(pretrain_runs)` 33. Writes how the window's embeddings are stored, how closely the encoder repeats them, and the stretch of their image. `site_write_json(embeddings_record(list(int8 = embeddings_int8, samples = embeddings_matrix(samples_encoded), values = embeddings_values, check = encode_check, int8_check = int8_check, stretch = emb_rgb_stretch, file = "embeddings.bin", source_files = fs::path_file(embeddings_tile_files(embeddings_files, "009010")), repo = "e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", probs = c(0.005, 0.995), size = study_window$size)), "../web/src/data/embeddings.json")` 34. Writes the window's embeddings, one byte per value, for the website. `site_write_bin(embeddings_window_bytes(embeddings_int8, int8_check), "../web/public/data/embeddings.bin", "int8")` 35. Puts the fixed set of every run on the first two principal components of the full run, and scores its classes by nearest neighbours. `pretrain_snapshots(pretrain_runs, label = fixed_samples$label, k = 5L)` 36. Marks the months each pixel was observed in. `window_observed_mask(window_raw)` 37. Fills the missing months along time with sits::impute_linear(), as sits does before the encoder. `window_fill_series(window_raw, band_scale)` 38. Stores every other pixel's series in one byte per value, each band between two quantiles. `window_export_series(window_series, c(0.001, 0.999 ), 2L)` 39. Picks the reflectance drawn black and white in each colour, the same for every month. `window_export_sprite_stretch(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), c(0.02, 0.98))` 40. Writes the record of the published run and of the small run, its snapshots, and the views of one series. `site_write_json(pretrain_record(list(published = published_run, runs = pretrain_runs, snapshots = snapshots, reference = knn_reference, views = views, trajectory = pretrain_trajectory, labels = labels, fixed_label = fixed_samples$label, n_series = nrow(pretrain_series), settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"), knn_k = 5L, encoder = list(repo = "e-sensing/encoders_cerrado", file = "ssl_lejepa_tcnn_model_2017_2024.rds"), file = "pretrain-snapshots.bin")), "../web/src/data/pretrain.json")` 41. Writes which months of the window were observed, for the website. `site_write_bin(window_export_row_major(window_export_mask(window_mask, 2L)), "../web/public/data/series-mask.bin", "uint8")` 42. Writes the monthly series of the window for the website. `site_write_bin(window_export_row_major(series_quantised$q), "../web/public/data/series.bin", "uint8")` 43. Draws the window in every month as false-colour images in one sheet, for the website. `site_write_png(window_export_sprite(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch, 32L, 6L), "../web/public/data/rgb-months.png")` 44. Writes where the window is, and how its series and monthly images are stored, for the website. `site_write_json(window_export_record(study_window, window_grid, window_layers, band_scale, window_mask, series_quantised, series_stretch, cube_files, list(repo = "e-sensing/cerrado_cube_reg_2018", revision = "refs/pr/3", commit = "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85"), list(png_scale = 2L, series_quantiles = c(0.001, 0.999), series_stride = 2L, series_units = "surface reflectance", series_rgb = list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch = c(0.02, 0.98), sprite_columns = 6L, sprite_levels = 32L, sprite_small_tile = 220L, tour_band = "NIR08"), list(series = "series.bin", mask = "series-mask.bin", sprite = "rgb-months.png", sprite_small = "rgb-months-small.png")), "../web/src/data/window.json")` The files: - `rgb-months.png`: SHA-256 `89424cbbdeeda1cd4d46f0e9b2b6c70708250c23cf3b4a2cb4064a46d8439d8e`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `series.bin`: SHA-256 `0a5dacbfea580c475943a977d3d2fc693788a2449acac1be59416a5b743de112`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `series-mask.bin`: SHA-256 `b6c4d861d4986323aa8937eb7efb59cb6cac8e4c8075f41162e82de1babdeecd`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `embeddings.bin`: SHA-256 `d7b1efae1bbcdf2ffeade9f57c94c662e53af94ac0e55a18c0e12af33c0df1e5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `window.json`: SHA-256 `60e731843a80b49590366d3313d69d24d23f2c23c914b42ecdff8ff53b06f9d5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `embeddings.json`: SHA-256 `4e6b827b79e9fce5a8b8dca06de368b9ce8534c4ec268fae8f68084ce0822da1`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `pretrain.json`: SHA-256 `f14baa3757fb6ce5dba2a7bf97ddafd92457adfb4e3d8f26ea308cccf466d3cb`. Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash. To make them again: `cd pipeline && Rscript make.R`. Try it: [From a series to a vector](/interactive-images/shape.md) ## 3. How does an encoder learn with no labels? In the sits book: [Lean Joint-Embedding Predictive Architecture (LeJEPA)](https://e-sensing.github.io/sitsbook/emb_methods.html#lean-joint-embedding-predictive-architecture-lejepa). The book asks sits to train an encoder on series of the Cerrado that carry no label, with the method `sits_ssl_lejepa()`, so that it writes 64 values for each series. The book's code: ``` lejepa_encoder <- sits_pre_train( samples = pretrain_samples, rl_method = sits_ssl_lejepa( embedding_dim = 64, epochs = 150, batch_size = 2048, encoder_model = sits_tempcnn(), verbose = TRUE ) ) ``` Where the sits book writes it: [emb_example.qmd, lines 322–331](https://github.com/e-sensing/sitsbook/blob/bc75e250e50bc6fb8222593973b8dfe3b692201f/emb_example.qmd#L322-L331). The encoder of this site was fitted on 518,374 time series (129,593 more held out to check it), and nobody told it what any of them shows. With no label to learn from, the method makes its own task out of the data. Here is one of those series: band SWIR22 over its 24 dates. What it shows on the ground is not said here; the encoder was never told either. While it trains, `sits_ssl_lejepa()` draws two views of it: it joins the values with straight lines, picks two sets of points along them that share none, and stretches each set back onto the 24 dates. Each view is the same curve read on other days. Nothing new is observed. Replay draws them again. Both views go through the same encoder: one network, the same weights. Each comes out as a vector of 64 numbers. A small second network, the projector, follows, with 128 outputs. The loss reads the projector's two outputs; the projector is dropped after training. Training pulls the two outputs together, and through them the encoder's vectors. The views differ in their days and agree in the shape of the seasons, so the encoder is pushed to keep what they share. This part of the loss is called invariance. In `sits_ssl_lejepa()` it carries 0.98 of the weight. There is a lazy way to win: give every series the same output. The views always agree, and the output says nothing. That is collapse. So the loss has a second part, SIGReg: over the outputs of many series at once, it keeps the cloud they form round. It carries the other 0.02. No label appears anywhere in the method. ### The figure A diagram, not a measurement. The series and its two views are real: sits drew the views from the series. The networks, their two outputs and the cloud of many series are drawn, not computed. The weights are the defaults of sits_ssl_lejepa(). The figure (diagram of the method · the series is real) draws, top to bottom: one real series of the pre-training (band SWIR22, 24 dates); the two views A and B sits draws from it while it trains; one encoder for both, with the same weights, and the projector, whose two outputs the loss reads; those two outputs, pulled together and through them the encoder's vectors (invariance, weight 0.98); and the cloud of the outputs of many series, kept round in its ring (SIGReg, weight 0.02) beside a collapsed point. Only the series and its two views are data; the cloud is a drawing. The weights are the defaults of `sits_ssl_lejepa()` in the pinned sits. #### How it was made 1. Read the labelled samples: The pipeline reads the labelled samples: time series, each with its land cover class. It lists the classes and gives each the colour sits draws it in. 2. Encode the labelled samples: The pipeline draws the same number of labelled samples from each class of the period of the embeddings cube, and sits_encode() turns each into its 64 numbers with the published encoder. 3. Read how the encoder was trained: The published encoder keeps a record of its own training: the loss of every round on the training and the validation series, how long each round took, and the settings sits_ssl_lejepa() was called with. The pipeline reads that record. 4. Draw the series of a small run: The encoder also keeps the series it was trained on. The pipeline draws some of them under a fixed seed, for a smaller training run of its own. 5. Store the window's embeddings in one byte: The pipeline cuts the study window out of the embeddings cube: 64 numbers for each pixel. It stores each number in one byte, and checks that the nearest neighbours of the labelled samples give their class almost as often as before: the pipeline stops if the share they get right falls by more than 0.01. 6. Train a small encoder, and stop it early: sits_pre_train() trains an encoder of its own with sits_ssl_lejepa() on the drawn series. It trains again from the start for fewer rounds, under the same seed, and the pipeline checks that each shorter run is the start of the long one. sits_encode() embeds the fixed set after each run. 7. Make two views of one series: sits_ssl_lejepa() learns by comparing two views of the same series. The pipeline draws one of the series under a fixed seed and makes its two views as sits makes them in training. 8. Read the settings of the methods: The pipeline reads every argument of sits_ssl_lejepa() and sits_tempcnn(), with its default, from the installed sits. 9. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - methods.json: A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - pretrain.json: Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash. Every step of the pipeline: 1. The published encoder of the Cerrado, as kept from Hugging Face. `hf_cache_path("e-sensing/encoders_cerrado", "ssl_lejepa_tcnn_model_2017_2024.rds", "model")` 2. Reads every argument of the methods, with its default, from the installed sits. `provenance_method_formals(c("sits_ssl_lejepa", "sits_tempcnn" ))` 3. The labelled Cerrado samples, as kept from Hugging Face. `hf_cache_path("e-sensing/samples_cerrado", "labelled_samples_cerrado.parquet", "dataset")` 4. The epochs of the snapshot runs, then of the full run. `c(1L, 2L, 4L, 8L, 16L, 30L)` 5. Reads the encoder once: the record of its training, and series drawn from those it was trained on. `encoder_contents(encoder_read(encoder_file), 20000L, 20261011L)` 6. Reads the samples into a table, one row per time series. `samples_read_cached(samples_file)` 7. Writes the arguments of the methods and their defaults, for the website. `site_write_json(methods_formals, "../web/src/data/methods.json")` 8. Lists the land cover classes found in the samples. `sits::sits_labels(samples)` 9. The series of the small run, drawn under a fixed seed from those the encoder was trained on. `encoder_record[["series"]]` 10. The loss of every round of the published training, its seconds and its settings. `encoder_record[["published"]]` 11. The labelled samples of the period, the same number of each class, with their place in the period. `encoder_draw_per_class(encoder_period_samples(samples, "2017-01-01"), per_class = 500L, seed = 20261011L)` 12. The drawn samples as the published encoder embeds them: 64 numbers each. `encoder_encode_samples(samples_subset, encoder_file, list(device = "default", memsize = 4L, multicores = 2L))` 13. Draws one series under a fixed seed and makes its two views as sits makes them in training. `pretrain_views(pretrain_series, seed = 20261011L)` 14. Draws a fixed set of the encoded samples, the same number of each class, under a fixed seed. `embeddings_fixed_set(samples_encoded, labels, 150L, 20261011L)` 15. The labelled series of the fixed set, which every run encodes. `pretrain_fixed_samples(samples_subset, encoded = samples_encoded, fixed = fixed_set)` 16. Scores the classes of the fixed set by nearest neighbours on the raw series and on the published encoder's embeddings. `pretrain_knn_reference(pretrain_raw_series(fixed_samples, sits::sits_bands(pretrain_series)), published = embeddings_matrix(samples_encoded)[fixed_set, ], label = fixed_samples$label, k = 5L)` 17. Trains a LeJEPA encoder on the drawn series for each number of epochs, and encodes the fixed set with it. `pretrain_run(pretrain_series, fixed_samples, epochs = snapshot_epochs, settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"))` 18. Checks that each snapshot run has the losses of the first epochs of the full run, exactly. `pretrain_check_trajectory(pretrain_runs)` 19. Puts the fixed set of every run on the first two principal components of the full run, and scores its classes by nearest neighbours. `pretrain_snapshots(pretrain_runs, label = fixed_samples$label, k = 5L)` 20. Writes the record of the published run and of the small run, its snapshots, and the views of one series. `site_write_json(pretrain_record(list(published = published_run, runs = pretrain_runs, snapshots = snapshots, reference = knn_reference, views = views, trajectory = pretrain_trajectory, labels = labels, fixed_label = fixed_samples$label, n_series = nrow(pretrain_series), settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"), knn_k = 5L, encoder = list(repo = "e-sensing/encoders_cerrado", file = "ssl_lejepa_tcnn_model_2017_2024.rds"), file = "pretrain-snapshots.bin")), "../web/src/data/pretrain.json")` The files: - `pretrain.json`: SHA-256 `f14baa3757fb6ce5dba2a7bf97ddafd92457adfb4e3d8f26ea308cccf466d3cb`. Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash. - `methods.json`: SHA-256 `fb24a43f16ed5b8d14f5b9efdeef0f36a5f0bc4b87c750214eaac5708297b583`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. To make them again: `cd pipeline && Rscript make.R`. ### How sits draws two views The series is one of the 647,967 the published encoder was pre-trained with (518,374 fitted, 129,593 held out for validation): 24 dates from 2017-01-01, in 6 bands (BLUE, GREEN, RED, NIR08, SWIR16 and SWIR22). While it trains, `sits_ssl_lejepa()` draws the views by resampling the series: it interpolates the 24 dates onto 48 steps, draws two sets of 12 steps with no step in common, taking steps from each quarter of the period, and stretches each set back onto the 24 dates. It draws new views each time training reads a series: these two are one draw. The figure draws band SWIR22, the one whose values span the widest range; the views were drawn on all 6 bands. This pair was drawn from the reflectance, for this page; in training sits draws them from the normalised values. In the sits book: [Time series augmentation for self-supervised learning](https://e-sensing.github.io/sitsbook/emb_methods.html#time-series-augmentation-for-self-supervised-learning). The book's figure of this augmentation is from [Saget et al. 2025](https://doi.org/10.48550/arXiv.2506.18587). ## 4. What did the training give? In the sits book: [Generating regional encoders using self-supervised learning](https://e-sensing.github.io/sitsbook/emb_example.html#generating-regional-encoders-using-self-supervised-learning). A reduced run, made for this site: sits pre-trained an encoder on a random subset of 20,000 series of the Cerrado, for 30 epochs on a CPU, in 9 min. The curve is the loss sits recorded after each epoch: from 1.952 after the first to 0.269 after the last. The dashed line is the loss on the 4,000 series it held out. The dots are 1,532 labelled samples, places whose land cover someone wrote down, where the encoder put them after epochs 1, 2, 4, 8, 16 and 30: flattened onto two axes, coloured by label. The encoder never saw these labels; they only colour the dots. Where a label comes from: [Data structure for image time series](https://e-sensing.github.io/sitsbook/ts_basics.html#data-structure-for-image-time-series). Press an epoch to see the cloud then. Do neighbours share a label? Take the 5 samples nearest to a sample in the 64 numbers the encoder gives it, not in the flat picture, and see whether most of them carry its label. That holds for 73.7% of the samples after epoch 1 and 85.6% after epoch 30. Compared as raw series, 82.3%: the reduced run passes it at the snapshot of epoch 16. The encoder of this site learned from 647,967 series and stopped early, after 54 of 150 epochs (7.1 h on a GPU). It kept only its loss per epoch: a curve, no cloud. Its neighbours share the label for 90.2% of the samples. Switch runs, drag along a curve, or press a label to bring it forward. Compare the shapes of the curves, not their levels. ### The figure Recorded by sits: the loss after each epoch, on a log scale. Figure: two recorded runs of `sits_ssl_lejepa()`, never drawn in one chart, behind a switch. "Reduced run" (Recorded run · 20,000 series · 30 epochs · CPU · 9 min): its training and validation loss per epoch, as a curve to scrub; under it the 1,532 labelled samples at the scrubbed epoch, coloured by their label (Annual crop, Cerradão, Cerrado, Mangrove, Natural non-vegetated, Open cerrado, Pasture, Perennial crop, Silviculture, Sugarcane and Water), with buttons for the snapshot epochs 1, 2, 4, 8, 16 and 30 and a legend that brings one class forward; between two snapshots the positions are interpolated and the figure says so. Beside it a scale of the share of samples whose label is the majority label of their 5 nearest neighbours in the 64 dimensions of the embedding: the reduced run at each snapshot, and marks for the raw series and the published encoder. "Published run" (Recorded run · 647,967 series · 54 of 150 epochs, stopped early · GPU · 7.1 h): its loss per epoch, and its mark on the scale; it stored no snapshots, so it has no cloud. Loss per epoch of the reduced run. | Epoch | Training loss | Validation loss | | ----- | ------------- | --------------- | | 1 | 1.952 | 1.070 | | 2 | 0.853 | 0.660 | | 3 | 0.617 | 0.535 | | 4 | 0.508 | 0.486 | | 5 | 0.466 | 0.417 | | 6 | 0.436 | 0.402 | | 7 | 0.404 | 0.373 | | 8 | 0.391 | 0.354 | | 9 | 0.373 | 0.351 | | 10 | 0.357 | 0.340 | | 11 | 0.351 | 0.343 | | 12 | 0.343 | 0.323 | | 13 | 0.337 | 0.310 | | 14 | 0.336 | 0.303 | | 15 | 0.322 | 0.302 | | 16 | 0.311 | 0.290 | | 17 | 0.313 | 0.283 | | 18 | 0.306 | 0.272 | | 19 | 0.298 | 0.277 | | 20 | 0.299 | 0.276 | | 21 | 0.292 | 0.267 | | 22 | 0.293 | 0.272 | | 23 | 0.291 | 0.258 | | 24 | 0.284 | 0.257 | | 25 | 0.279 | 0.247 | | 26 | 0.280 | 0.245 | | 27 | 0.277 | 0.256 | | 28 | 0.274 | 0.237 | | 29 | 0.269 | 0.238 | | 30 | 0.269 | 0.243 | Loss per epoch of the run that made the published encoder. The runs differ in size, so compare the shapes of their curves, not their levels. | Epoch | Training loss | Validation loss | | ----- | ------------- | --------------- | | 1 | 0.379 | 0.225 | | 2 | 0.203 | 0.175 | | 3 | 0.179 | 0.154 | | 4 | 0.166 | 0.149 | | 5 | 0.157 | 0.138 | | 6 | 0.151 | 0.130 | | 7 | 0.146 | 0.128 | | 8 | 0.142 | 0.122 | | 9 | 0.138 | 0.120 | | 10 | 0.135 | 0.117 | | 11 | 0.132 | 0.119 | | 12 | 0.130 | 0.115 | | 13 | 0.128 | 0.116 | | 14 | 0.126 | 0.112 | | 15 | 0.125 | 0.109 | | 16 | 0.123 | 0.109 | | 17 | 0.122 | 0.109 | | 18 | 0.121 | 0.108 | | 19 | 0.120 | 0.107 | | 20 | 0.119 | 0.106 | | 21 | 0.118 | 0.104 | | 22 | 0.117 | 0.103 | | 23 | 0.117 | 0.105 | | 24 | 0.116 | 0.103 | | 25 | 0.115 | 0.102 | | 26 | 0.115 | 0.102 | | 27 | 0.114 | 0.101 | | 28 | 0.114 | 0.103 | | 29 | 0.113 | 0.100 | | 30 | 0.113 | 0.101 | | 31 | 0.112 | 0.101 | | 32 | 0.112 | 0.099 | | 33 | 0.111 | 0.099 | | 34 | 0.111 | 0.098 | | 35 | 0.111 | 0.100 | | 36 | 0.110 | 0.099 | | 37 | 0.110 | 0.097 | | 38 | 0.110 | 0.098 | | 39 | 0.110 | 0.098 | | 40 | 0.109 | 0.097 | | 41 | 0.109 | 0.097 | | 42 | 0.109 | 0.096 | | 43 | 0.109 | 0.096 | | 44 | 0.109 | 0.097 | | 45 | 0.108 | 0.096 | | 46 | 0.108 | 0.097 | | 47 | 0.108 | 0.097 | | 48 | 0.108 | 0.096 | | 49 | 0.108 | 0.096 | | 50 | 0.108 | 0.096 | | 51 | 0.108 | 0.096 | | 52 | 0.107 | 0.095 | | 53 | 0.108 | 0.096 | | 54 | 0.107 | 0.096 | #### What differs between the two runs The reduced run, as it was run for this site: `sits_pre_train(samples, rl_method = sits_ssl_lejepa(epochs = 30, embedding_dim = 64, seed = 20261011))` Both runs are recorded by sits: the reduced one when it was made for this site, the published one as the encoder object stores it. The marked rows are the settings both runs recorded, where their values differ. | | Reduced run | Published encoder | Different | | --------- | ----------- | ------------------------------- | --------- | | Series | 20,000 | 647,967 | yes | | Fitted on | 16,000 | 518,374 | yes | | Held out | 4,000 | 129,593 | yes | | Epochs | 30 epochs | 54 of 150 epochs, stopped early | yes | | Batch | 512 series | 512 series | no | | Embedding | 64 values | 64 values | no | | Device | CPU | GPU | yes | | Duration | 9 min | 7.1 h | yes | | Seed | 20261011 | 10 | yes | Of the arguments of `sits_ssl_lejepa()` both runs recorded, 12 have the same value: embedding_dim, proj_dim, lambda, num_knots, num_slices, batch_size, validation_split, opt_hparams, lr_decay_epochs, lr_decay_rate, patience and min_delta. The book's call (the header of step 3) writes `batch_size = 2048`; the published encoder recorded batches of 512. Nothing here separates the effect of one difference from another. #### How the snapshots were made sits has no hook to save the encoder after each epoch, so each snapshot is a separate run made the same way, stopped at that epoch; the losses of those runs are identical to the full run's. The 1,532 samples are the same at every epoch. The two axes are the first two principal components of the final epoch, the same for every epoch, so a cloud that looks larger is larger. Between two snapshots the figure interpolates the positions and says so; the statistic is never interpolated. For each sample, its 5 nearest other samples (Euclidean distance, the sample itself left out) vote with their labels; on a tie the nearest neighbour among the tied labels decides. The distances are measured in all 64 dimensions of the embedding, not in the two of the picture; for the raw series, in their 144 reflectance values, which are not standardised. | Epoch | Same label among the nearest neighbours | Variance on the two axes | | ----- | --------------------------------------- | ------------------------ | | 1 | 73.7% | 31.8% + 17.5% | | 2 | 74.1% | 38.9% + 14.8% | | 4 | 78.8% | 29.0% + 19.1% | | 8 | 80.0% | 25.5% + 22.7% | | 16 | 82.5% | 24.6% + 21.5% | | 30 | 85.6% | 23.4% + 21.3% | | Reference | Same label among the nearest neighbours | | ----------------- | --------------------------------------- | | Raw series | 82.3% | | Published encoder | 90.2% | #### How it was made 1. Read the labelled samples: The pipeline reads the labelled samples: time series, each with its land cover class. It lists the classes and gives each the colour sits draws it in. 2. Encode the labelled samples: The pipeline draws the same number of labelled samples from each class of the period of the embeddings cube, and sits_encode() turns each into its 64 numbers with the published encoder. 3. Read how the encoder was trained: The published encoder keeps a record of its own training: the loss of every round on the training and the validation series, how long each round took, and the settings sits_ssl_lejepa() was called with. The pipeline reads that record. 4. Draw the series of a small run: The encoder also keeps the series it was trained on. The pipeline draws some of them under a fixed seed, for a smaller training run of its own. 5. Store the window's embeddings in one byte: The pipeline cuts the study window out of the embeddings cube: 64 numbers for each pixel. It stores each number in one byte, and checks that the nearest neighbours of the labelled samples give their class almost as often as before: the pipeline stops if the share they get right falls by more than 0.01. 6. Train a small encoder, and stop it early: sits_pre_train() trains an encoder of its own with sits_ssl_lejepa() on the drawn series. It trains again from the start for fewer rounds, under the same seed, and the pipeline checks that each shorter run is the start of the long one. sits_encode() embeds the fixed set after each run. 7. Make two views of one series: sits_ssl_lejepa() learns by comparing two views of the same series. The pipeline draws one of the series under a fixed seed and makes its two views as sits makes them in training. 8. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - classes.json and pretrain-snapshots.bin: A rerun on aarch64-apple-darwin23 gave the same files, byte for byte. - pretrain.json: Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash. Every step of the pipeline: 1. The published encoder of the Cerrado, as kept from Hugging Face. `hf_cache_path("e-sensing/encoders_cerrado", "ssl_lejepa_tcnn_model_2017_2024.rds", "model")` 2. The labelled Cerrado samples, as kept from Hugging Face. `hf_cache_path("e-sensing/samples_cerrado", "labelled_samples_cerrado.parquet", "dataset")` 3. The epochs of the snapshot runs, then of the full run. `c(1L, 2L, 4L, 8L, 16L, 30L)` 4. Reads the encoder once: the record of its training, and series drawn from those it was trained on. `encoder_contents(encoder_read(encoder_file), 20000L, 20261011L)` 5. Reads the samples into a table, one row per time series. `samples_read_cached(samples_file)` 6. Lists the land cover classes found in the samples. `sits::sits_labels(samples)` 7. The series of the small run, drawn under a fixed seed from those the encoder was trained on. `encoder_record[["series"]]` 8. The loss of every round of the published training, its seconds and its settings. `encoder_record[["published"]]` 9. The labelled samples of the period, the same number of each class, with their place in the period. `encoder_draw_per_class(encoder_period_samples(samples, "2017-01-01"), per_class = 500L, seed = 20261011L)` 10. Gives each class the colour sits uses to draw it. `samples_class_colors(labels)` 11. The drawn samples as the published encoder embeds them: 64 numbers each. `encoder_encode_samples(samples_subset, encoder_file, list(device = "default", memsize = 4L, multicores = 2L))` 12. Draws one series under a fixed seed and makes its two views as sits makes them in training. `pretrain_views(pretrain_series, seed = 20261011L)` 13. Draws a fixed set of the encoded samples, the same number of each class, under a fixed seed. `embeddings_fixed_set(samples_encoded, labels, 150L, 20261011L)` 14. Writes the list of classes, with their titles and colours, for the website. `site_write_json(samples_class_table(labels, colors, list(Cerradao = "Cerradão", Nat_NonVeg = "Natural non-vegetated")), "../web/src/data/classes.json")` 15. The labelled series of the fixed set, which every run encodes. `pretrain_fixed_samples(samples_subset, encoded = samples_encoded, fixed = fixed_set)` 16. Scores the classes of the fixed set by nearest neighbours on the raw series and on the published encoder's embeddings. `pretrain_knn_reference(pretrain_raw_series(fixed_samples, sits::sits_bands(pretrain_series)), published = embeddings_matrix(samples_encoded)[fixed_set, ], label = fixed_samples$label, k = 5L)` 17. Trains a LeJEPA encoder on the drawn series for each number of epochs, and encodes the fixed set with it. `pretrain_run(pretrain_series, fixed_samples, epochs = snapshot_epochs, settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"))` 18. Checks that each snapshot run has the losses of the first epochs of the full run, exactly. `pretrain_check_trajectory(pretrain_runs)` 19. Puts the fixed set of every run on the first two principal components of the full run, and scores its classes by nearest neighbours. `pretrain_snapshots(pretrain_runs, label = fixed_samples$label, k = 5L)` 20. Writes the record of the published run and of the small run, its snapshots, and the views of one series. `site_write_json(pretrain_record(list(published = published_run, runs = pretrain_runs, snapshots = snapshots, reference = knn_reference, views = views, trajectory = pretrain_trajectory, labels = labels, fixed_label = fixed_samples$label, n_series = nrow(pretrain_series), settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"), knn_k = 5L, encoder = list(repo = "e-sensing/encoders_cerrado", file = "ssl_lejepa_tcnn_model_2017_2024.rds"), file = "pretrain-snapshots.bin")), "../web/src/data/pretrain.json")` 21. Writes the fixed set of every run on one basis, for the website. `site_write_float32(pretrain_snapshot_values(snapshots, pretrain_trajectory), "../web/public/data/pretrain-snapshots.bin")` The files: - `pretrain.json`: SHA-256 `f14baa3757fb6ce5dba2a7bf97ddafd92457adfb4e3d8f26ea308cccf466d3cb`. Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash. - `classes.json`: SHA-256 `e479d0f03cfc2258efe34563f02adc3999d25150b0f991643153d439209b2145`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `pretrain-snapshots.bin`: SHA-256 `ab07ffa8295ceae8ede8db5853fcbdc62463fa8ce14b5475aef7af230f8d4a94`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. To make them again: `cd pipeline && Rscript make.R`. Try it: [Pre-training, replayed](/interactive-images/pretraining.md) ## 5. What are these 64 numbers? In the sits book: [Using encoders to produce Earth observation embeddings](https://e-sensing.github.io/sitsbook/emb_example.html#using-encoders-to-produce-earth-observation-embeddings). `plot()` draws an embeddings cube like any other cube: three of its bands as red, green and blue, for one tile. The book's code: ``` plot(lejepa_embedding, tile = "009010", red = "EMB11", green = "EMB12", blue = "EMB13") ``` Where the sits book writes it: [emb_example.qmd, lines 445–446](https://github.com/e-sensing/sitsbook/blob/bc75e250e50bc6fb8222593973b8dfe3b692201f/emb_example.qmd#L445-L446). For every pixel of the window, `sits_encode()` has turned its whole time series into 64 numbers. sits stores them as 64 bands, EMB01 to EMB64: an embeddings cube, again a stack of images. A screen has three colours to give, so the picture shows three bands at a time: EMB11, EMB12 and EMB13 as red, green and blue, as the sits book plots them. The colours mean nothing by themselves. Now one pixel, the one marked on the image. Its 64 numbers, in order, are the bars: that list is its embedding, a vector. A bar above the line is a positive number, a bar below it a negative one. The scale is the same for every pixel. Each number is also a map. Take one place in the list and look at its value in every pixel: a picture of the window in grey, dark where the number is low, bright where it is high. The band arrows step through all 64 bands, one at a time, and pointing at a bar shows its band. No band has a name like “red” or “moisture”: nobody chose what each one should measure; the encoder settled on it during training. Any three bands make a colour picture. Choose them with the R, G and B menus, or press Surprise me; Book’s three brings back the book's. Point at the image to read any pixel, and click to keep it. The colours change completely and the shapes do not: what tells one field from the next is in the vectors, not in a choice of colours. ### The figure Recorded by sits: the embeddings cube that sits_encode() wrote for the window. The figure (recorded: sits_encode() · encoder from sits_ssl_lejepa() · 256 × 256 px · 64 bands) draws the embeddings cube that `sits_encode()` wrote for the window, 256 × 256 pixels of 30 m (7.68 km across) in tile 009010, with 64 values per pixel stored as bands EMB01 to EMB64. The window is drawn with three bands as red, green and blue (EMB11, EMB12 and EMB13, the choice of the sits book), and the 64 values of the selected pixel are drawn below it as signed bars, on one scale for every pixel. Pointing at a bar shows that band alone as a grey map of the window; at the beat of one band, band arrows step through every band. At its last beat three menus choose the band behind each colour, a Surprise me button picks three at random and a Book’s three button brings back the book's; any pixel can be pointed at, clicked or reached with the arrow keys. | Fact | Value | | ------------------------------ | ---------------------------------------------- | | Tile | 009010 | | Window | 256 × 256 pixels, 7.68 km across | | Pixel size | 30 m | | Values per pixel | 64 | | Default colour bands (R, G, B) | EMB11, EMB12, EMB13 | | Display stretch | per band, 5% of the values clipped at each end | #### Values beyond the scale The bars of every pixel are drawn on one scale, so two pixels can be compared: the range that holds 99% of the 4,194,304 values of the window, rounded up to a multiple of 0.5. The bars are drawn from −5.5 to +5.5; 0.7% of the values of the window lie beyond that, and their bars end in a small point past the rule. Pointing at such a bar reads its true value. Separately, storing the values clips some of them. This page keeps each value in one byte: per band, 255 steps between the 0.5% and 99.5% quantiles of that band over the window and the labelled samples together, and a value beyond them kept as the nearest end. Of the values of this window, 0.7% were clipped that way (0.95% over window and samples together). Stored and restored, the values of the samples are off by 0.184 on average (root mean square). | Fact | Value | | ----------------------------------------------------------- | ------------ | | Range of the bars | −5.5 to +5.5 | | Values of the window beyond the bars | 0.7% | | Values per pixel | 64 | | Values in the window | 4,194,304 | | Values of the window clipped when stored (a separate count) | 0.7% | | Error after storing (root mean square) | 0.184 | #### How it was made 1. Read the labelled samples: The pipeline reads the labelled samples: time series, each with its land cover class. It lists the classes and gives each the colour sits draws it in. 2. Cut out the study window: The pipeline finds the window in its tile of a regular cube and cuts it out of every band and month. It keeps the stored values and marks the months a pixel has none. 3. Fill the missing months: Where a pixel has no value in a month, sits fills one in on a straight line between the months around it. At either end of the series it repeats the nearest value. 4. Encode the labelled samples: The pipeline draws the same number of labelled samples from each class of the period of the embeddings cube, and sits_encode() turns each into its 64 numbers with the published encoder. 5. Read how the encoder was trained: The published encoder keeps a record of its own training: the loss of every round on the training and the validation series, how long each round took, and the settings sits_ssl_lejepa() was called with. The pipeline reads that record. 6. Draw the series of a small run: The encoder also keeps the series it was trained on. The pipeline draws some of them under a fixed seed, for a smaller training run of its own. 7. Store the window's embeddings in one byte: The pipeline cuts the study window out of the embeddings cube: 64 numbers for each pixel. It stores each number in one byte, and checks that the nearest neighbours of the labelled samples give their class almost as often as before: the pipeline stops if the share they get right falls by more than 0.01. 8. Encode the window again: sits_encode() runs the published encoder on the window's monthly images, the call that made the embeddings cube. The pipeline compares what comes out with the published cube, number by number. 9. Train a small encoder, and stop it early: sits_pre_train() trains an encoder of its own with sits_ssl_lejepa() on the drawn series. It trains again from the start for fewer rounds, under the same seed, and the pipeline checks that each shorter run is the start of the long one. sits_encode() embeds the fixed set after each run. 10. Make two views of one series: sits_ssl_lejepa() learns by comparing two views of the same series. The pipeline draws one of the series under a fixed seed and makes its two views as sits makes them in training. 11. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each. 12. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - embeddings.bin, embeddings.json and window.json: A rerun on aarch64-apple-darwin23 gave the same files, byte for byte. - pretrain.json: Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash. Every step of the pipeline: 1. Lists the band files of the regular Landsat cube on Hugging Face, at a fixed commit. `hf_cube_files("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85")` 2. Lists the band files of the embeddings cube on Hugging Face, at a fixed commit. `embeddings_cube_files("e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01")` 3. Reads how sits decodes the values of an embeddings cube. `embeddings_cube_values()` 4. The published encoder of the Cerrado, as kept from Hugging Face. `hf_cache_path("e-sensing/encoders_cerrado", "ssl_lejepa_tcnn_model_2017_2024.rds", "model")` 5. The labelled Cerrado samples, as kept from Hugging Face. `hf_cache_path("e-sensing/samples_cerrado", "labelled_samples_cerrado.parquet", "dataset")` 6. The epochs of the snapshot runs, then of the full run. `c(1L, 2L, 4L, 8L, 16L, 30L)` 7. The study window, chosen as pipeline/legacy/R/01_window.R describes. `list(tile = "009010", col_off = 2208L, row_off = 1536L, size = 256L, resolution = 30L, source = "BDC", collection = "LANDSAT-OLI-16D")` 8. Reads the encoder once: the record of its training, and series drawn from those it was trained on. `encoder_contents(encoder_read(encoder_file), 20000L, 20261011L)` 9. Reads the samples into a table, one row per time series. `samples_read_cached(samples_file)` 10. Reads the grid of the window's tile from a band file. `grid_read_tile(hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", cube_files$file[cube_files$tile == study_window$tile][[1L]]))` 11. Lists the land cover classes found in the samples. `sits::sits_labels(samples)` 12. The series of the small run, drawn under a fixed seed from those the encoder was trained on. `encoder_record[["series"]]` 13. The loss of every round of the published training, its seconds and its settings. `encoder_record[["published"]]` 14. The labelled samples of the period, the same number of each class, with their place in the period. `encoder_draw_per_class(encoder_period_samples(samples, "2017-01-01"), per_class = 500L, seed = 20261011L)` 15. Places the window in the tile's grid. `window_place(tile_grid, study_window)` 16. Crops the window out of the 64 bands of the embeddings cube by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/emb-cube", hf_resolve_url("e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01", embeddings_tile_files(embeddings_files, "009010")), window_grid, "e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", "f3df925fbf90102a7f10d035d87ed8bb6ab0fb01", 8L, crop = window_crop_http_files)` 17. The drawn samples as the published encoder embeds them: 64 numbers each. `encoder_encode_samples(samples_subset, encoder_file, list(device = "default", memsize = 4L, multicores = 2L))` 18. Draws one series under a fixed seed and makes its two views as sits makes them in training. `pretrain_views(pretrain_series, seed = 20261011L)` 19. Crops the window out of the cube's band files by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/reg", hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", hf_tile_files(cube_files, study_window$tile, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))), window_grid, "e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", 8L, crop = window_crop_http_files)` 20. Draws a fixed set of the encoded samples, the same number of each class, under a fixed seed. `embeddings_fixed_set(samples_encoded, labels, 150L, 20261011L)` 21. Reads the 64 numbers of every pixel of the window. `embeddings_read_window(embeddings_crops, embeddings_values, study_window$size)` 22. Names the bands and months of the crops. `window_crop_layers(window_crops, study_window$source, study_window$collection, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))` 23. Reads how sits scales the bands and marks missing values. `collection_band_values(study_window$source, study_window$collection, window_layers$bands)` 24. Picks the values of three bands drawn black and white, as sits plots a cube. `embeddings_rgb_stretch(window_embeddings, list(red = "EMB11", green = "EMB12", blue = "EMB13"))` 25. Stores the embeddings of the window and of the samples in one byte per value. `embeddings_to_int8(window_embeddings, embeddings_matrix(samples_encoded), c(0.005, 0.995))` 26. Runs the published encoder on the window's monthly images and compares its output with the published cube. `encoder_compare(encoder_encode_window(window_crops, encoder_file, list(device = "default", memsize = 4L, multicores = 2L, source = "BDC", collection = "LANDSAT-OLI-16D", output_dir = "cache/encode-check"), values = embeddings_values, size = study_window$size), window_embeddings)` 27. The labelled series of the fixed set, which every run encodes. `pretrain_fixed_samples(samples_subset, encoded = samples_encoded, fixed = fixed_set)` 28. Checks that the nearest neighbours of the fixed set give their class as often with the stored bytes. `embeddings_check_int8(embeddings_matrix(samples_encoded)[fixed_set, ], embeddings_restore(embeddings_int8$samples$q[fixed_set, ], embeddings_int8$params), samples_encoded$label[fixed_set], list(k = 5L, max_knn_drop = 0.01))` 29. Scores the classes of the fixed set by nearest neighbours on the raw series and on the published encoder's embeddings. `pretrain_knn_reference(pretrain_raw_series(fixed_samples, sits::sits_bands(pretrain_series)), published = embeddings_matrix(samples_encoded)[fixed_set, ], label = fixed_samples$label, k = 5L)` 30. Trains a LeJEPA encoder on the drawn series for each number of epochs, and encodes the fixed set with it. `pretrain_run(pretrain_series, fixed_samples, epochs = snapshot_epochs, settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"))` 31. Reads the stored values of the crops, missing ones as NA. `window_read_raw(window_crops, window_layers$bands, window_layers$dates, band_scale, study_window$size)` 32. Checks that each snapshot run has the losses of the first epochs of the full run, exactly. `pretrain_check_trajectory(pretrain_runs)` 33. Writes how the window's embeddings are stored, how closely the encoder repeats them, and the stretch of their image. `site_write_json(embeddings_record(list(int8 = embeddings_int8, samples = embeddings_matrix(samples_encoded), values = embeddings_values, check = encode_check, int8_check = int8_check, stretch = emb_rgb_stretch, file = "embeddings.bin", source_files = fs::path_file(embeddings_tile_files(embeddings_files, "009010")), repo = "e-sensing/cerrado_emb_ssl_lejepa_tcnn_2018", probs = c(0.005, 0.995), size = study_window$size)), "../web/src/data/embeddings.json")` 34. Writes the window's embeddings, one byte per value, for the website. `site_write_bin(embeddings_window_bytes(embeddings_int8, int8_check), "../web/public/data/embeddings.bin", "int8")` 35. Puts the fixed set of every run on the first two principal components of the full run, and scores its classes by nearest neighbours. `pretrain_snapshots(pretrain_runs, label = fixed_samples$label, k = 5L)` 36. Marks the months each pixel was observed in. `window_observed_mask(window_raw)` 37. Fills the missing months along time with sits::impute_linear(), as sits does before the encoder. `window_fill_series(window_raw, band_scale)` 38. Stores every other pixel's series in one byte per value, each band between two quantiles. `window_export_series(window_series, c(0.001, 0.999 ), 2L)` 39. Picks the reflectance drawn black and white in each colour, the same for every month. `window_export_sprite_stretch(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), c(0.02, 0.98))` 40. Writes the record of the published run and of the small run, its snapshots, and the views of one series. `site_write_json(pretrain_record(list(published = published_run, runs = pretrain_runs, snapshots = snapshots, reference = knn_reference, views = views, trajectory = pretrain_trajectory, labels = labels, fixed_label = fixed_samples$label, n_series = nrow(pretrain_series), settings = list(epochs = 30L, embedding_dim = 64L, seed = 20261011L, device = "cpu"), knn_k = 5L, encoder = list(repo = "e-sensing/encoders_cerrado", file = "ssl_lejepa_tcnn_model_2017_2024.rds"), file = "pretrain-snapshots.bin")), "../web/src/data/pretrain.json")` 41. Writes where the window is, and how its series and monthly images are stored, for the website. `site_write_json(window_export_record(study_window, window_grid, window_layers, band_scale, window_mask, series_quantised, series_stretch, cube_files, list(repo = "e-sensing/cerrado_cube_reg_2018", revision = "refs/pr/3", commit = "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85"), list(png_scale = 2L, series_quantiles = c(0.001, 0.999), series_stride = 2L, series_units = "surface reflectance", series_rgb = list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch = c(0.02, 0.98), sprite_columns = 6L, sprite_levels = 32L, sprite_small_tile = 220L, tour_band = "NIR08"), list(series = "series.bin", mask = "series-mask.bin", sprite = "rgb-months.png", sprite_small = "rgb-months-small.png")), "../web/src/data/window.json")` The files: - `embeddings.bin`: SHA-256 `d7b1efae1bbcdf2ffeade9f57c94c662e53af94ac0e55a18c0e12af33c0df1e5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `embeddings.json`: SHA-256 `4e6b827b79e9fce5a8b8dca06de368b9ce8534c4ec268fae8f68084ce0822da1`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `window.json`: SHA-256 `60e731843a80b49590366d3313d69d24d23f2c23c914b42ecdff8ff53b06f9d5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `pretrain.json`: SHA-256 `f14baa3757fb6ce5dba2a7bf97ddafd92457adfb4e3d8f26ea308cccf466d3cb`. Not repeatable on this hardware: a rerun gives another file. This one is kept as the record, with its hash. To make them again: `cd pipeline && Rscript make.R`. Try it: [One embedding, band by band](/interactive-images/vector.md) ## Next On this site: [From a cube to labelled samples](/time-series.md). How does a place in a cube become a sample whose label we know? In the sits book: [Foundational Models and Embeddings](https://e-sensing.github.io/sitsbook/embeddings.html), [Embedding Methods in SITS](https://e-sensing.github.io/sitsbook/emb_methods.html) and [Building regional embeddings: a full example](https://e-sensing.github.io/sitsbook/emb_example.html).