# A map of temporal variability

The temporal standard deviation of every cell of the study window, computed with stars, beside the selected cell's monthly profile; a second view maps each cell's valid months, for each period.

[From step 4 of From time series to a map of temporal variability](/temporal-variability.md).

References: [stars, st_apply](https://r-spatial.github.io/stars/reference/st_apply.html), [Pebesma and Bivand, Spatial Data Science, chapter 6, Data Cubes](https://r-spatial.org/book/06-Cubes.html) and [Pinned cube snapshot, e-sensing/cerrado_cube_reg_2018](https://huggingface.co/datasets/e-sensing/cerrado_cube_reg_2018/tree/af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85).

Pick any cell of the map to see its monthly values and its score, switch to the count of valid months, and change the period.

## The figure

The temporal standard deviation of every cell of the study window, on one colour scale from 0 for every period, beside the selected cell's monthly cube values. A checker marks a cell with too few valid values for a score; a second view maps each cell's count of valid monthly values.

A map of the temporal standard deviation of the monthly cube values of NIR08, in dimensionless surface reflectance, for each of the 256 × 256 cells of the study window. Computed with stars in R, the same summary for every cell. Its colours run on the cividis ramp from 0 to 0.1849, the same scale for every period. A cell with fewer than 2 valid values has no score and is drawn in a pink checker, apart from a score of 0. A second view maps each cell's count of valid monthly values ("Valid monthly values"), from 0 to 15, in purples that lighten with the count, with 0 in orange. Beside the map, the selected cell's profile on a time axis of every month, with the months outside the chosen period faded and the mean of its valid values as a dashed line. The periods are 2017–2018, 2017 and 2018; changing the period keeps the cell.

Cell at row 148, column 199 (one-based), the cell followed in the earlier steps:

| Period    | Valid values | Mean of valid values | Temporal standard deviation |
| --------- | ------------ | -------------------- | --------------------------- |
| 2017–2018 | 13 of 24     | 0.2459               | 0.0476                      |
| 2017      | 5 of 12      | 0.2480               | 0.0473                      |
| 2018      | 8 of 12      | 0.2445               | 0.0509                      |

### How it was made

1. Cut out the study window: The pipeline finds the window in its tile of a regular cube and cuts it out of every band and month. It keeps the stored values and marks the months a pixel has none.
2. Fill the missing months: Where a pixel has no value in a month, sits fills one in on a straight line between the months around it. At either end of the series it repeats the nearest value.
3. Verify the calculation library: The pipeline checks stars against its pinned release source. sits prepares the shared inputs; stars computes these summaries.
4. Keep the measured values: The stored near-infrared values are calibrated on their native grid. The window's pictures, which only locate the cell, use the months sits fills; the stars cube keeps every missing monthly value missing, and no filled value enters a score.
5. Summarize the available months: For each period, stars keeps the spatial grid and reduces time to mean, sample standard deviation and valid-value count.
6. Follow three measured profiles: A fixed rule selects three cells with the same valid count. Their ordered observations explain the spread behind a score.
7. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each.
8. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it.

### Reproducibility details

- A rerun on aarch64-apple-darwin23 gave the same files, byte for byte.

Every step of the pipeline:

1. Lists the band files of the regular Landsat cube on Hugging Face, at a fixed commit. `hf_cube_files("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85")`
2. The study window, chosen as pipeline/legacy/R/01_window.R describes. `list(tile = "009010", col_off = 2208L, row_off = 1536L, size = 256L, resolution = 30L, source = "BDC", collection = "LANDSAT-OLI-16D")`
3. Identifies the exact dependency lock for stars. `"renv.lock"`
4. Reads the grid of the window's tile from a band file. `grid_read_tile(hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", cube_files$file[cube_files$tile == study_window$tile][[1L]]))`
5. Reads the pinned stars release source methods. `variability_source_files("cache/src/stars", variability_source_lock_version(variability_lock))`
6. Verifies stars release methods against the installed package and records their source links. `variability_source_package_record(variability_release_files)`
7. Places the window in the tile's grid. `window_place(tile_grid, study_window)`
8. Crops the window out of the cube's band files by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/reg", hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", hf_tile_files(cube_files, study_window$tile, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))), window_grid, "e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", 8L, crop = window_crop_http_files)`
9. Names the bands and months of the crops. `window_crop_layers(window_crops, study_window$source, study_window$collection, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))`
10. Reads how sits scales the bands and marks missing values. `collection_band_values(study_window$source, study_window$collection, window_layers$bands)`
11. Reads the stored values of the crops, missing ones as NA. `window_read_raw(window_crops, window_layers$bands, window_layers$dates, band_scale, study_window$size)`
12. Calibrates the masked monthly band without filling gaps. `variability_band(window_raw, window_layers, band_scale, "NIR08")`
13. Marks the months each pixel was observed in. `window_observed_mask(window_raw)`
14. Fills the missing months along time with sits::impute_linear(), as sits does before the encoder. `window_fill_series(window_raw, band_scale)`
15. Stores every other pixel's series in one byte per value, each band between two quantiles. `window_export_series(window_series, c(0.001, 0.999 ), 2L)`
16. Picks the reflectance drawn black and white in each colour, the same for every month. `window_export_sprite_stretch(window_series, window_layers$bands, list(red = "SWIR16", green = "NIR08", blue = "RED"), c(0.02, 0.98))`
17. Draws months of the window side by side in grey, empty where no valid value, for the stack of the home picture. `site_write_png(variability_export_picture_months(variability_selected_band, list(count = 8L, min_valid = 0.5, size = 128L, period = "full", ramp = c("#00224e", "#3b496c", "#7c7b78", "#bcaf6f", "#fee838"), checker = list(c(214L, 51L, 108L), c(253L, 224L, 239L)))), "../web/public/data/temporal-variability-picture-months.png")`
18. Writes lossless Int16 monthly series in date chunks. `variability_export_series(variability_selected_band, "../web/public/data", list(site_dir = "../web", stars_source = "cache/src/stars", variable = "NIR08", max_bytes = 3000000L, chunk_dates = 12L, picture = list(count = 8L, min_valid = 0.5, size = 128L, period = "full", ramp = c("#00224e", "#3b496c", "#7c7b78", "#bcaf6f", "#fee838"), checker = list(c(214L, 51L, 108L), c(253L, 224L, 239L)))))`
19. Constructs a stars cube on native cell edges. `variability_cube(variability_selected_band, window_grid, variability_package)`
20. Derives the full period and complete calendar years. `variability_periods(variability_selected_band[["dates"]])`
21. Writes where the window is, and how its series and monthly images are stored, for the website. `site_write_json(window_export_record(study_window, window_grid, window_layers, band_scale, window_mask, series_quantised, series_stretch, cube_files, list(repo = "e-sensing/cerrado_cube_reg_2018", revision = "refs/pr/3", commit = "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85"), list(png_scale = 2L, series_quantiles = c(0.001, 0.999), series_stride = 2L, series_units = "surface reflectance", series_rgb = list(red = "SWIR16", green = "NIR08", blue = "RED"), series_stretch = c(0.02, 0.98), sprite_columns = 6L, sprite_levels = 32L, sprite_small_tile = 220L, tour_band = "NIR08"), list(series = "series.bin", mask = "series-mask.bin", sprite = "rgb-months.png", sprite_small = "rgb-months-small.png")), "../web/src/data/window.json")`
22. Uses stars to reduce time to mean, sample SD and valid count independently at every cell in each period. `lapply(variability_windows, variability_reduce, cube = variability_stars_cube)`
23. Draws the full period's temporal SD on the site's ramp, cells with no score in its checker, for the home picture. `site_write_png(variability_export_picture_map(variability_statistics, variability_windows, list(count = 8L, min_valid = 0.5, size = 128L, period = "full", ramp = c("#00224e", "#3b496c", "#7c7b78", "#bcaf6f", "#fee838"), checker = list(c(214L, 51L, 108L), c(253L, 224L, 239L)))), "../web/public/data/temporal-variability-picture-map.png")`
24. Writes Float64 mean/SD maps and Uint16 valid counts. `variability_export_statistics(variability_statistics, variability_windows, "../web/public/data", 3000000L)`
25. Selects median, lower and higher SD profiles with equal valid counts by a deterministic percentile rule. `variability_examples(variability_selected_band, variability_statistics[[1L]], variability_windows)`
26. Collects calibration, periods, exact teaching profiles, fixed domains and measured binary manifests. `variability_export_record(list(band = variability_selected_band, grid = window_grid, periods = variability_windows, stats = variability_statistics, examples = variability_teaching_examples, package = variability_package, source = list(repo = "e-sensing/cerrado_cube_reg_2018", revision = "refs/pr/3", commit = "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", kind = "huggingface", upstream = list(source = "BDC", collection = "LANDSAT-OLI-16D"), window = list(tile = "009010", col_off = 2208L, row_off = 1536L, size_px = 256L, resolution_m = 30L), processingRecord = "https://huggingface.co/datasets/e-sensing/cerrado_cube_reg_2018/blob/af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85/README.md"), settings = list(site_dir = "../web", stars_source = "cache/src/stars", variable = "NIR08", max_bytes = 3000000L, chunk_dates = 12L, picture = list(count = 8L, min_valid = 0.5, size = 128L, period = "full", ramp = c("#00224e", "#3b496c", "#7c7b78", "#bcaf6f", "#fee838"), checker = list(c(214L, 51L, 108L), c(253L, 224L, 239L))))), c(site_variability_series, site_variability_statistics, site_variability_picture_months, site_variability_picture_map))`
27. Writes the temporal variability metadata contract. `variability_export_json(variability_metadata, "../web/src/data/temporal-variability.json")`

The files:

- `temporal-variability.json`: SHA-256 `837238265334bd4dfea204fa2e91a06ba34361dafa5f891c7a3b46591762dc14`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `window.json`: SHA-256 `60e731843a80b49590366d3313d69d24d23f2c23c914b42ecdff8ff53b06f9d5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `temporal-variability-stats-full.bin`: SHA-256 `fc7690eb504c9a568bf4e8b1f2d7b9ac4e8e62db1ebcc0fb20030aa685288259`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `temporal-variability-count-full.bin`: SHA-256 `27e546f9b47d6d5cb65bbb66879da607c51ea981669155b9727fb91ad6fe68d5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `temporal-variability-stats-2017.bin`: SHA-256 `fbb0b2585119143f60acc94231d11617e5b8b167453fe88999ad8fece637afe9`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `temporal-variability-count-2017.bin`: SHA-256 `205e75d1eaff1b08fe4c66349879f771d91f994c2ba76c9f78a6a1c4bdecce58`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `temporal-variability-stats-2018.bin`: SHA-256 `2bf202b5f1ba3a3011a0b7fc4a4d8544698f82e63899227220c3396f4b4e58e9`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `temporal-variability-count-2018.bin`: SHA-256 `b1c47a1f08cea9827d441edda46e513d8dadab056b11922acace8df6475fc96b`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `temporal-variability-series-000.bin`: SHA-256 `52268606dbacf18e5c915b92897e69422776ee0288f0a0dde3f444a75b9f157d`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.
- `temporal-variability-series-001.bin`: SHA-256 `30c069d9c8b1f066a5cc5f5dd050ada90e7de99dd8d1b5a5cf2bdc700a18d32e`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.

To make them again: `cd pipeline && Rscript make.R`.

## What to try

1. **Find the largest spread.** Look for the lightest cells and pick one, by click, tap or arrow keys. Does its profile swing between seasons, or does one month stand apart?
2. **Switch to valid months.** Choose *Valid months*: each cell's count of months with a valid value. Is a large spread resting on many values or on few?
3. **Change the year.** Choose a year under *Period*. The cell stays selected; compare its score and its count with the other year's. A difference between years is not land-cover change.