# A pixel through time

The window month by month, and each band of one pixel as a time series, with the months sits filled in.

[From step 1 of From series to embeddings](/embeddings.md).

In the sits book: [Satellite image time series](https://e-sensing.github.io/sitsbook/timeseries.html) and [Foundational Models and Embeddings](https://e-sensing.github.io/sitsbook/embeddings.html).

Play the months or drag the strip, and click any pixel to draw its bands through the months.

## The figure

The window month by month: the strip of months, one pixel's bands as lines, and the months nobody observed.

The figure (recorded: BDC LANDSAT-OLI-16D, tile 009010, 24 months from January 2017 to December 2018) draws the 256 × 256 pixel window, 7.68 km across, in one month as a false-colour image (SWIR16, NIR08 and RED as red, green and blue), a strip of the 24 months under it with a Play button, and the 6 bands of one pixel (BLUE, GREEN, RED, NIR08, SWIR16 and SWIR22) as lines over the months, in surface reflectance. At its last beat the strip says how much of the window was observed each month, the image fades the pixels that were not, and a hollow dot on a dashed line is a value filled with impute_linear(); any pixel can then be pointed at, clicked or reached with the arrow keys. It opens on May 2017, the first month observed everywhere in the window.

| Month          | Share of the window observed |
| -------------- | ---------------------------- |
| January 2017   | none (filled)                |
| February 2017  | none (filled)                |
| March 2017     | none (filled)                |
| April 2017     | 61%                          |
| May 2017       | 100%                         |
| June 2017      | 100%                         |
| July 2017      | 100%                         |
| August 2017    | 100%                         |
| September 2017 | none (filled)                |
| October 2017   | none (filled)                |
| November 2017  | 88%                          |
| December 2017  | none (filled)                |
| January 2018   | 77%                          |
| February 2018  | none (filled)                |
| March 2018     | 80%                          |
| April 2018     | 50%                          |
| May 2018       | 100%                         |
| June 2018      | 100%                         |
| July 2018      | 100%                         |
| August 2018    | 100%                         |
| September 2018 | 97%                          |
| October 2018   | 62%                          |
| November 2018  | none (filled)                |
| December 2018  | < 1%                         |

### How it was made

1. Cut out the study window: The pipeline finds the window in its tile of a regular cube and cuts it out of every band and month. It keeps the stored values and marks the months a pixel has none.
2. Fill the missing months: Where a pixel has no value in a month, sits fills one in on a straight line between the months around it. At either end of the series it repeats the nearest value.
3. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each.
4. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it.

### Reproducibility details

- A rerun on aarch64-apple-darwin23 gave the same files, byte for byte.

Every step of the pipeline:

1. Lists the band files of the regular Landsat cube on Hugging Face, at a fixed commit. `hf_cube_files("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85")`
2. The study window, chosen as pipeline/legacy/R/01_window.R describes. `list(tile = "009010", col_off = 2208L, row_off = 1536L, size = 256L, resolution = 30L, source = "BDC", collection = "LANDSAT-OLI-16D")`
3. Reads the grid of the window's tile from a band file. `grid_read_tile(hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", cube_files$file[cube_files$tile == study_window$tile][[1L]]))`
4. Places the window in the tile's grid. `window_place(tile_grid, study_window)`
5. Crops the window out of the cube's band files by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/reg", hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", hf_tile_files(cube_files, study_window$tile, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))), window_grid, "e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", 8L, crop = window_crop_http_files)`
6. Names the bands and months of the crops. `window_crop_layers(window_crops, study_window$source, study_window$collection, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))`
7. Reads how sits scales the bands and marks missing values. `collection_band_values(study_window$source, study_window$collection, window_layers$bands)`
8. Reads the stored values of the crops, missing ones as NA. `window_read_raw(window_crops, window_layers$bands, window_layers$dates, band_scale, study_window$size)`
9. Marks the months each pixel was observed in. `window_observed_mask(window_raw)`
10. Fills the missing months along time with sits::impute_linear(), as sits does before the encoder. `window_fill_series(window_raw, band_scale)`
11. Stores every other pixel's series in one byte per value, each band between two quantiles. `window_export_series(window_series, c(0.001, 0.999 ), 2L)`
12. Picks the reflectance drawn black and white in each colour, the same for every month. `window_export_sprite_stretch(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), c(0.02, 0.98))`
13. Writes which months of the window were observed, for the website. `site_write_bin(window_export_row_major(window_export_mask(window_mask, 2L)), "../web/public/data/series-mask.bin", "uint8")`
14. Writes the monthly series of the window for the website. `site_write_bin(window_export_row_major(series_quantised$q), "../web/public/data/series.bin", "uint8")`
15. Draws the window in every month as false-colour images in one sheet, for the website. `site_write_png(window_export_sprite(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch, 32L, 6L), "../web/public/data/rgb-months.png")`
16. Writes where the window is, and how its series and monthly images are stored, for the website. `site_write_json(window_export_record(study_window, window_grid, window_layers, band_scale, window_mask, series_quantised, series_stretch, cube_files, list(repo = "e-sensing/cerrado_cube_reg_2018", revision = "refs/pr/3", commit = "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85"), list(png_scale = 2L, series_quantiles = c(0.001, 0.999), series_stride = 2L, series_units = "surface reflectance", series_rgb = list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch = c(0.02, 0.98), sprite_columns = 6L, sprite_levels = 32L, sprite_small_tile = 220L, tour_band = "NIR08"), list(series = "series.bin", mask = "series-mask.bin", sprite = "rgb-months.png", sprite_small = "rgb-months-small.png")), "../web/src/data/window.json")`

The files:

- `rgb-months.png`: SHA-256 `89424cbbdeeda1cd4d46f0e9b2b6c70708250c23cf3b4a2cb4064a46d8439d8e`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `series.bin`: SHA-256 `0a5dacbfea580c475943a977d3d2fc693788a2449acac1be59416a5b743de112`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `series-mask.bin`: SHA-256 `b6c4d861d4986323aa8937eb7efb59cb6cac8e4c8075f41162e82de1babdeecd`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `window.json`: SHA-256 `60e731843a80b49590366d3313d69d24d23f2c23c914b42ecdff8ff53b06f9d5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.

To make them again: `cd pipeline && Rscript make.R`.

## What to try

1. **Play the months.** Press *Play*, or drag along the strip under the image. The fields turn green, are harvested and lie bare, each on its own calendar, while the native vegetation around them hardly changes.
2. **Compare two places.** Click a field, then the dark vegetation beside it, and compare the two sets of lines: look at their shape through the year, not only at their level.
3. **Look for what was not seen.** Drag to the first months of the strip. The strip says how much of the window was observed in each month. Where it is empty, the image you see was not recorded: `sits` filled it in from the nearest months that were.
4. **Count the real measurements.** A solid dot on the chart is an observation and a hollow dot on a dashed line is a filled value. The panel beside the image counts them for the pixel. Try a few pixels: none has an observation in every month.
5. **Use the keyboard.** Focus the image and move the pixel with the arrow keys; focus the strip and step through the months.