# Two cells, which spreads more?

Two cells of the study window with the same number of valid monthly values, their profiles on one axis. Guess which spreads more, then see each cell's temporal standard deviation.

[From step 3 of From time series to a map of temporal variability](/temporal-variability.md).

References: [R statistics, sd](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/sd.html) and [stars, st_apply](https://r-spatial.github.io/stars/reference/st_apply.html).

Look at two monthly profiles on the same axis, choose the one whose values spread more, and compare their temporal standard deviations.

## The figure

Two cells of the study window with the same number of valid monthly values, their profiles on one axis, each with a button to choose the one that spreads more. Their temporal standard deviations appear once you choose or show the answer; a line under them gives, for reference, the score of the cell the journey follows.

The departures from the mean of the monthly cube values of NIR08, in dimensionless surface reflectance, of the cell at row 148, column 199 (one-based), over 2017–2018. 13 of 24 months have a valid value. Each departure is squared and drawn as a bar on its own axis under its month; a month with no valid value reads "No valid value" and has neither.

| Month          | Departure from the mean | Squared departure |
| -------------- | ----------------------- | ----------------- |
| January 2017   | No valid value          | No valid value    |
| February 2017  | No valid value          | No valid value    |
| March 2017     | No valid value          | No valid value    |
| April 2017     | No valid value          | No valid value    |
| May 2017       | −0.0155                 | 0.000239          |
| June 2017      | −0.0232                 | 0.000536          |
| July 2017      | −0.0221                 | 0.000486          |
| August 2017    | −0.0150                 | 0.000224          |
| September 2017 | No valid value          | No valid value    |
| October 2017   | No valid value          | No valid value    |
| November 2017  | +0.0864                 | 0.007473          |
| December 2017  | No valid value          | No valid value    |
| January 2018   | +0.0496                 | 0.002465          |
| February 2018  | No valid value          | No valid value    |
| March 2018     | +0.0329                 | 0.001085          |
| April 2018     | No valid value          | No valid value    |
| May 2018       | −0.0019                 | 0.000003          |
| June 2018      | −0.0184                 | 0.000337          |
| July 2018      | −0.0192                 | 0.000367          |
| August 2018    | −0.0052                 | 0.000027          |
| September 2018 | −0.1042                 | 0.010848          |
| October 2018   | +0.0552                 | 0.003052          |
| November 2018  | No valid value          | No valid value    |
| December 2018  | No valid value          | No valid value    |

Added up, the 13 departures give 0.0000: those above the mean cancel those below. Squared, no departure can be negative: the 13 squares add up to 0.027142. 0.027142 ÷ 12 = 0.002262, the variance: the sum of squares divided by n − 1 = 12. √0.002262 = 0.0476, Temporal standard deviation, in dimensionless surface reflectance, from 13 valid values. This is the sample standard deviation.

Last, the figure shows two other cells on the same axis, with the same number of valid values, and asks which spreads more before it reveals their scores:

| Cell                       | Valid values | Temporal standard deviation |
| -------------------------- | ------------ | --------------------------- |
| Cell at row 40, column 3   | 13           | 0.0291                      |
| Cell at row 136, column 74 | 13           | 0.0849                      |

### How it was made

1. Cut out the study window: The pipeline finds the window in its tile of a regular cube and cuts it out of every band and month. It keeps the stored values and marks the months a pixel has none.
2. Verify the calculation library: The pipeline checks stars against its pinned release source. sits prepares the shared inputs; stars computes these summaries.
3. Keep the measured values: The stored near-infrared values are calibrated on their native grid. The window's pictures, which only locate the cell, use the months sits fills; the stars cube keeps every missing monthly value missing, and no filled value enters a score.
4. Summarize the available months: For each period, stars keeps the spatial grid and reduces time to mean, sample standard deviation and valid-value count.
5. Follow three measured profiles: A fixed rule selects three cells with the same valid count. Their ordered observations explain the spread behind a score.
6. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it.

### Reproducibility details

- A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.

Every step of the pipeline:

1. Lists the band files of the regular Landsat cube on Hugging Face, at a fixed commit. `hf_cube_files("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85")`
2. The study window, chosen as pipeline/legacy/R/01_window.R describes. `list(tile = "009010", col_off = 2208L, row_off = 1536L, size = 256L, resolution = 30L, source = "BDC", collection = "LANDSAT-OLI-16D")`
3. Identifies the exact dependency lock for stars. `"renv.lock"`
4. Reads the grid of the window's tile from a band file. `grid_read_tile(hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", cube_files$file[cube_files$tile == study_window$tile][[1L]]))`
5. Reads the pinned stars release source methods. `variability_source_files("cache/src/stars", variability_source_lock_version(variability_lock))`
6. Verifies stars release methods against the installed package and records their source links. `variability_source_package_record(variability_release_files)`
7. Places the window in the tile's grid. `window_place(tile_grid, study_window)`
8. Crops the window out of the cube's band files by georeference, keeping crops already made. `window_crop_files("cache/windows/window_009010_c2208_r1536/reg", hf_resolve_url("e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", hf_tile_files(cube_files, study_window$tile, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))), window_grid, "e-sensing/cerrado_cube_reg_2018", "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", 8L, crop = window_crop_http_files)`
9. Names the bands and months of the crops. `window_crop_layers(window_crops, study_window$source, study_window$collection, c("2017-01-01", "2017-02-01", "2017-03-01", "2017-04-01", "2017-05-01", "2017-06-01", "2017-07-01", "2017-08-01", "2017-09-01", "2017-10-01", "2017-11-01", "2017-12-01", "2018-01-01", "2018-02-01", "2018-03-01", "2018-04-01", "2018-05-01", "2018-06-01", "2018-07-01", "2018-08-01", "2018-09-01", "2018-10-01", "2018-11-01", "2018-12-01"))`
10. Reads how sits scales the bands and marks missing values. `collection_band_values(study_window$source, study_window$collection, window_layers$bands)`
11. Reads the stored values of the crops, missing ones as NA. `window_read_raw(window_crops, window_layers$bands, window_layers$dates, band_scale, study_window$size)`
12. Calibrates the masked monthly band without filling gaps. `variability_band(window_raw, window_layers, band_scale, "NIR08")`
13. Draws months of the window side by side in grey, empty where no valid value, for the stack of the home picture. `site_write_png(variability_export_picture_months(variability_selected_band, list(count = 8L, min_valid = 0.5, size = 128L, period = "full", ramp = c("#00224e", "#3b496c", "#7c7b78", "#bcaf6f", "#fee838"), checker = list(c(214L, 51L, 108L), c(253L, 224L, 239L)))), "../web/public/data/temporal-variability-picture-months.png")`
14. Writes lossless Int16 monthly series in date chunks. `variability_export_series(variability_selected_band, "../web/public/data", list(site_dir = "../web", stars_source = "cache/src/stars", variable = "NIR08", max_bytes = 3000000L, chunk_dates = 12L, picture = list(count = 8L, min_valid = 0.5, size = 128L, period = "full", ramp = c("#00224e", "#3b496c", "#7c7b78", "#bcaf6f", "#fee838"), checker = list(c(214L, 51L, 108L), c(253L, 224L, 239L)))))`
15. Constructs a stars cube on native cell edges. `variability_cube(variability_selected_band, window_grid, variability_package)`
16. Derives the full period and complete calendar years. `variability_periods(variability_selected_band[["dates"]])`
17. Uses stars to reduce time to mean, sample SD and valid count independently at every cell in each period. `lapply(variability_windows, variability_reduce, cube = variability_stars_cube)`
18. Draws the full period's temporal SD on the site's ramp, cells with no score in its checker, for the home picture. `site_write_png(variability_export_picture_map(variability_statistics, variability_windows, list(count = 8L, min_valid = 0.5, size = 128L, period = "full", ramp = c("#00224e", "#3b496c", "#7c7b78", "#bcaf6f", "#fee838"), checker = list(c(214L, 51L, 108L), c(253L, 224L, 239L)))), "../web/public/data/temporal-variability-picture-map.png")`
19. Writes Float64 mean/SD maps and Uint16 valid counts. `variability_export_statistics(variability_statistics, variability_windows, "../web/public/data", 3000000L)`
20. Selects median, lower and higher SD profiles with equal valid counts by a deterministic percentile rule. `variability_examples(variability_selected_band, variability_statistics[[1L]], variability_windows)`
21. Collects calibration, periods, exact teaching profiles, fixed domains and measured binary manifests. `variability_export_record(list(band = variability_selected_band, grid = window_grid, periods = variability_windows, stats = variability_statistics, examples = variability_teaching_examples, package = variability_package, source = list(repo = "e-sensing/cerrado_cube_reg_2018", revision = "refs/pr/3", commit = "af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85", kind = "huggingface", upstream = list(source = "BDC", collection = "LANDSAT-OLI-16D"), window = list(tile = "009010", col_off = 2208L, row_off = 1536L, size_px = 256L, resolution_m = 30L), processingRecord = "https://huggingface.co/datasets/e-sensing/cerrado_cube_reg_2018/blob/af1d4f67fee0fcaba35941f4ac5d7d9e87d4fa85/README.md"), settings = list(site_dir = "../web", stars_source = "cache/src/stars", variable = "NIR08", max_bytes = 3000000L, chunk_dates = 12L, picture = list(count = 8L, min_valid = 0.5, size = 128L, period = "full", ramp = c("#00224e", "#3b496c", "#7c7b78", "#bcaf6f", "#fee838"), checker = list(c(214L, 51L, 108L), c(253L, 224L, 239L))))), c(site_variability_series, site_variability_statistics, site_variability_picture_months, site_variability_picture_map))`
22. Writes the temporal variability metadata contract. `variability_export_json(variability_metadata, "../web/src/data/temporal-variability.json")`

The files:

- `temporal-variability.json`: SHA-256 `837238265334bd4dfea204fa2e91a06ba34361dafa5f891c7a3b46591762dc14`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte.

To make them again: `cd pipeline && Rscript make.R`.

## What to try

1. **Compare the profiles.** Both cells rest on the same number of valid values. Look for the values that lie far from the rest: which profile has more of them, or larger ones?
2. **Predict, then reveal.** Press *This one spreads more* under the cell you think spreads more, or *Show the answer*. Both scores appear: did the eye agree with the number?
3. **Open the equation.** [The equation, and why n − 1](/temporal-variability#summary/equation) opens on the journey's step: the sum of squares, the divisor and the root, for the followed cell.