# Building data cubes from image collections How do the images a provider hands over become a regular data cube? The sits book's own example, over one small region of Rondônia: first the images a provider hands over, cut into tiles and taken on uneven dates, then the grid a regular cube agrees on. Then sits_regularize() at work, from the images of each period to one value per cell, and what the cube it builds promises. Each step builds its picture as you scroll; the details open when you ask for them. In the sits book: [Analysis-ready image collections](https://e-sensing.github.io/sitsbook/dc_ardcollections.html) and [Building regular data cubes](https://e-sensing.github.io/sitsbook/dc_regularize.html). ## 1. What does a provider hand over? In the sits book: [Accessing ARD image collections in cloud providers](https://e-sensing.github.io/sitsbook/dc_ardcollections.html#accessing-ard-image-collections-in-cloud-providers). The book asks AWS for the Sentinel-2 images of 2 tiles from 30 Jun to 31 Aug 2018, in 3 bands plus the cloud band, CLOUD. The book's code: ``` s2_cube_rondonia <- sits_cube( source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LLP", "20LKP"), bands = c("B02", "B8A", "B11", "CLOUD"), start_date = as.Date("2018-06-30"), end_date = as.Date("2018-08-31") ) ``` Where the sits book writes it: [dc_regularize.qmd, lines 62–69](https://github.com/e-sensing/sitsbook/blob/bc75e250e50bc6fb8222593973b8dfe3b692201f/dc_regularize.qmd#L62-L69). Sentinel-2 files its images by the tiles of a fixed grid. Each tile is 109.8 km a side and has a name. The book's call asks for 2 tiles, 20LKP and 20LLP, each drawn here with one of its pictures. They overlap: a strip 9.8 km wide lies in both. The page follows one small region, 15.4 by 8 km, across that strip. Each tile covers part of it: 20LKP 97% and 20LLP 67% of its width. One tile: 20LLP has 45 items on 25 dates from 30 Jun to 31 Aug 2018. Each item is a picture of the whole tile on one date; the row shows its part over the region. 23 of them hold no data there. Under each, its date and the cloud cover of the whole scene. `sits_cube()` returns this collection as a table: one row per tile, with the files of its images listed. Switch the tile to see 20LKP's 24 items on 12 dates. ### The figure The tile grid around a region of Rondônia, Brazil, and one tile's images over the region, by date. The figure (recorded: 2 tiles, 69 items of the book's call) is a map of the Sentinel-2 grid around the region, with the 2 tiles of the call (109.8 km a side) in colour, the 9.8 km strip they share and the region across it. Under the map, a row of one tile's images over the region in date order, each with its date and scene cloud cover (20LKP: 24 items on 12 dates, 2 with no data over the region; 20LLP: 45 items on 25 dates, 23 with no data over the region), a toggle between the tiles, and the `sits_cube()` call that lists them. #### How it was made 1. Ask the catalogue: sits asks the provider's catalogue which images cover the book's tiles, bands and dates, and a whole year of one tile. The answers are kept, so every later run uses the same list. 2. Find the tiles: The pipeline finds the tiles of the satellite's grid around the region, from the grid that comes with sits. It reads a small picture of each tile of the cube from its least cloudy image. 3. Build the regular cube: sits makes a regular cube of the region from those images: one image per period, all on the same grid of pixels. For each pixel of a period it takes the first valid value, least cloudy image first. 4. Read the images as they come: The pipeline reads the images the catalogue listed, as they come. It marks each pixel as having no data, a clear value, or a value sits masks, such as cloud or shadow. 5. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each. 6. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - A rerun on aarch64-apple-darwin23 gave the same files, byte for byte. Every step of the pipeline: 1. Lists the bands read: the book's and those of the natural-colour pictures, in the order sits has them. `cube_read_bands("AWS", "SENTINEL-2-L2A", c("B02", "B8A", "B11"), c("B03", "B04"))` 2. The arguments of the book's call to sits_cube(). `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands)` 3. The same call for one tile over a whole year. `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands, "20LLP", "2018-01-01", "2018-12-31")` 4. Lists the images of the book's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_book_call, "cache/cubes", "cube_book")` 5. Lists the images of the year's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_year_call, "cache/cubes", "cube_year")` 6. Makes the regular cube of the book's call over the region with sits_regularize(), keeping the files already made. `cube_regularize(cube_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_book_B02-B03-B04-B8A-B11", 4L)` 7. Finds the Sentinel-2 tiles around the region. `cube_images_grid(list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), cube_listing, 100L)` 8. Reads a small picture of each tile from the overview of its clearest image, keeping the reads in cache/cubes/. `cube_images_read(cube_listing, list(red = "B04", green = "B03", blue = "B02"), 3L, "cache/cubes")` 9. Makes the regular cube of the year with sits_regularize(), keeping the files already made. `cube_regularize(cube_year_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_year_B02-B03-B04-B8A-B11", 4L)` 10. Reads every image of each tile over the tile's part of the region, as it comes, keeping the reads in cache/cubes/. `cube_tile_images(cube_listing, cube_book, cube_book_call, 40L, "EPSG:32720", "cache/cubes")` 11. Gathers the images, ranked in the order sits takes them, the regular values, the year and the map. `cube_export_gather(cube_listing, cube_book, cube_images, cube_year_listing, cube_year, cube_map_images, cube_map_grid, cube_book_call, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), 40L)` 12. Marks each pixel of each image as no data, clear or masked by sits. `purrr::map(cubes_record$tiles, cube_images_states, cubes_record$cloud_values)` 13. Picks the reflectance drawn black and white, one for red, green and blue. `cube_images_stretch(cubes_record, list(red = "B04", green = "B03", blue = "B02"), c(0.02, 0.98))` 14. Stores the year in one byte per value, each band between two percentiles. `cube_export_year_codes(cubes_record$year$values, c(0.001, 0.999))` 15. Draws every image of the tile as it comes, in natural colour, in one sheet, for the website. `site_write_png(cube_images_ard_sheet(cubes_record$tiles[["20LKP"]], cube_states[["20LKP"]], cubes_record$bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/cube-ard-20lkp.png")` 16. Draws every image of the tile as it comes, in natural colour, in one sheet, for the website. `site_write_png(cube_images_ard_sheet(cubes_record$tiles[["20LLP"]], cube_states[["20LLP"]], cubes_record$bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/cube-ard-20llp.png")` 17. Draws the whole tile on a web map, in natural colour, for the website. `site_write_png(cube_images_picture(cube_images_tile(cubes_record, "20LKP"), "EPSG:32720", cube_stretch, 32L), "../web/public/data/cube-map-20lkp.png")` 18. Draws the whole tile on a web map, in natural colour, for the website. `site_write_png(cube_images_picture(cube_images_tile(cubes_record, "20LLP"), "EPSG:32720", cube_stretch, 32L), "../web/public/data/cube-map-20llp.png")` 19. Writes the book's call, its images and regular cube, the map and the year, for the website. ``site_write_json(cube_export_record(cubes_record, cube_states, cube_stretch, cube_year_codes, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), list(tiles = list(`20LKP` = list(ard = "cube-ard-20lkp.png", mask = "cube-ard-20lkp-mask.png", regular = "cube-regular-20lkp.png", map = "cube-map-20lkp.png"), `20LLP` = list(ard = "cube-ard-20llp.png", mask = "cube-ard-20llp-mask.png", regular = "cube-regular-20llp.png", map = "cube-map-20llp.png")), year = "cube-year.bin")), "../web/src/data/cubes.json")`` The files: - `cubes.json`: SHA-256 `f29bb2e9135a6f824c0d512868ea1c3d67bd2532c285b6662f87926e071d5874`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-map-20lkp.png`: SHA-256 `f2867b1bcbe952b85b81f41a92fe4f085044ca958cf8a6f648a7671e2fc4a214`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-map-20llp.png`: SHA-256 `1f0fd138a76e56708fb3ab7dfea3fabce85e731ac1b585c4df1d90f6c2a2f3b7`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-ard-20lkp.png`: SHA-256 `54d23212cc41e26d91b44a38ca8b2031aca7243db4e6e1a1658138bd06119fef`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-ard-20llp.png`: SHA-256 `2c4d12fb2b588be680847130668ab1089acb717cad265cc4ab1005d854b17da5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. To make them again: `cd pipeline && Rscript make.R`. ## 2. Why can't the images be used as they come? In the sits book: [The need for regular EO data cubes](https://e-sensing.github.io/sitsbook/dc_regularize.html#the-need-for-regular-eo-data-cubes). The book asks sits for the dates of those images, tile by tile. The book's code: ``` sits_timeline(s2_cube_rondonia) ``` Where the sits book writes it: [dc_regularize.qmd, line 71](https://github.com/e-sensing/sitsbook/blob/bc75e250e50bc6fb8222593973b8dfe3b692201f/dc_regularize.qmd#L71). **Each tile has its own dates.** From 30 Jun to 31 Aug 2018, 20LKP has 12 dates and 20LLP 25. They share 12: every date of 20LKP is one of 20LLP's, and 20LLP has 13 more. The figure draws a recorded run of the book's call: 69 images of 2 tiles. **Some images cover only part of the tile.** Tile 20LLP on 30 Jun 2018: the image covers 52% of it. The hatched part has no data. **Bands come at different resolutions.** The same 480 m square of 20LLP: B04 at 10 m is 48 pixels a side, B11 at 20 m only 24. B02, B03 and B04 come at 10 m; B11, B8A and CLOUD at 20 m. **Clouds and their shadows hide the ground.** sits drops the pixels the CLOUD band marks: on 7 Aug 2018 that is 38% of the region in one of 20LKP's two images that day, most of it thin cirrus. **A cube needs one grid, one set of dates, and a value for each cell.** Images as they come promise none of the three. The next steps build a cube that does. ### The figure Four problems of the images of the book's call as they come, one per beat, then side by side. The figure (recorded: 69 images, 2 tiles) at its last beat: four small pictures in a row, one per problem. - **Dates** (12 and 25 dates): The dates of 20LKP and 20LLP, 30 Jun to 31 Aug 2018: one dot per date. - **Coverage** (52% of the tile): Tile 20LLP on 30 Jun 2018, the whole tile; hatched where the image has no data. - **Resolution** (10 m and 20 m): A 480 m square of 20LLP on 18 Jul 2018: B04 at 10 m, B11 at 20 m. - **Clouds** (38% masked): The region in 20LKP on 7 Aug 2018; drawn over it, the pixels the CLOUD band masks. #### What the CLOUD band masks on 7 Aug 2018, tile 20LKP sits masks a pixel when its CLOUD value is one of 0, 1, 2, 3, 8, 9, 10. On this image that is 38% of the pixels with data, and they hold: | Value | Class | Share | | ----- | ------------ | ----- | | 10 | thin cirrus | 32% | | 8 | cloud medium | 5% | | 9 | cloud high | 1% | #### How it was made 1. Ask the catalogue: sits asks the provider's catalogue which images cover the book's tiles, bands and dates, and a whole year of one tile. The answers are kept, so every later run uses the same list. 2. Find the tiles: The pipeline finds the tiles of the satellite's grid around the region, from the grid that comes with sits. It reads a small picture of each tile of the cube from its least cloudy image. 3. Build the regular cube: sits makes a regular cube of the region from those images: one image per period, all on the same grid of pixels. For each pixel of a period it takes the first valid value, least cloudy image first. 4. Read the images as they come: The pipeline reads the images the catalogue listed, as they come. It marks each pixel as having no data, a clear value, or a value sits masks, such as cloud or shadow. 5. Try other periods and pixel sizes: sits makes the regular cube again at other periods and pixel sizes. It also makes it once without the images' cloud cover, so sits takes the images in date order, not least cloudy first. 6. Read each image alone: Each image is read alone, on the grid sits builds for it. Stacked in the order sits takes them, the images show which one gave each pixel of the regular cube. 7. Check against sits: The pipeline compares its own reads with what sits made, and stops the run on the checks that must hold. The results are gathered only once those checks pass. 8. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each. 9. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - A rerun on aarch64-apple-darwin23 gave the same files, byte for byte. Every step of the pipeline: 1. Lists the bands read: the book's and those of the natural-colour pictures, in the order sits has them. `cube_read_bands("AWS", "SENTINEL-2-L2A", c("B02", "B8A", "B11"), c("B03", "B04"))` 2. Lists every period with every resolution of the runs. `regularize_choices(c("P8D", "P16D", "P1M"), c(20L, 40L, 80L))` 3. The arguments of the book's call to sits_cube(). `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands)` 4. The same call for one tile over a whole year. `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands, "20LLP", "2018-01-01", "2018-12-31")` 5. Lists the bands of the runs: the natural-colour ones, in the order sits has them. `regularize_run_bands(cube_bands, list(red = "B04", green = "B03", blue = "B02"))` 6. Lists the images of the book's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_book_call, "cache/cubes", "cube_book")` 7. Lists the images of the year's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_year_call, "cache/cubes", "cube_year")` 8. Reads how sits masks the cloud band, and the line of its source file that says so. `collection_cloud_source("cache/src/sits-78d26eb07370874192911854b48b60ebb1d668c8", "AWS", "SENTINEL-2-L2A", "CLOUD", regularize_bands)` 9. Makes the regular cube of the book's call over the region with sits_regularize(), keeping the files already made. `cube_regularize(cube_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_book_B02-B03-B04-B8A-B11", 4L)` 10. Finds the Sentinel-2 tiles around the region. `cube_images_grid(list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), cube_listing, 100L)` 11. Reads a small picture of each tile from the overview of its clearest image, keeping the reads in cache/cubes/. `cube_images_read(cube_listing, list(red = "B04", green = "B03", blue = "B02"), 3L, "cache/cubes")` 12. Makes the regular cube of the year with sits_regularize(), keeping the files already made. `cube_regularize(cube_year_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_year_B02-B03-B04-B8A-B11", 4L)` 13. Reads one tile whole on the date of the book's figure, from an overview, keeping the read in cache/regularize/. `regularize_images_coverage(cube_listing, list(tile = "20LLP", date = "2018-06-30"), list(red = "B04", green = "B03", blue = "B02"), 3L, "cache/regularize")` 14. Runs sits_regularize() on the book's images at one period and resolution, keeping the files in cache/regularize/. `regularize_run_cube(cube_listing, regularize_choice, regularize_bands, "CLOUD", list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "cache/regularize", 4L, 1L)` 15. Reads every image of each tile over the tile's part of the region, as it comes, keeping the reads in cache/cubes/. `cube_tile_images(cube_listing, cube_book, cube_book_call, 40L, "EPSG:32720", "cache/cubes")` 16. Compares the grid of each tile of the book's run with the view sits builds for it. `regularize_book_grids(regularize_find_run(regularize_run, "P16D", 40L), cube_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L))` 17. Reads the regular images each run wrote. `regularize_values(regularize_run, regularize_bands)` 18. Gathers the images, ranked in the order sits takes them, the regular values, the year and the map. `cube_export_gather(cube_listing, cube_book, cube_images, cube_year_listing, cube_year, cube_map_images, cube_map_grid, cube_book_call, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), 40L)` 19. Marks each pixel of each image as no data, clear or masked by sits. `purrr::map(cubes_record$tiles, cube_images_states, cubes_record$cloud_values)` 20. Picks the reflectance drawn black and white, one for red, green and blue. `cube_images_stretch(cubes_record, list(red = "B04", green = "B03", blue = "B02"), c(0.02, 0.98))` 21. Stores the year in one byte per value, each band between two percentiles. `cube_export_year_codes(cubes_record$year$values, c(0.001, 0.999))` 22. Runs sits_regularize() on each image of the year alone, with the date of its period, keeping the files in cache/regularize/. `regularize_year_alone(cube_year_listing, cubes_record, "B02", "CLOUD", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "cache/regularize/year_20llp_B02", 1L)` 23. Stops the run unless the book's run gives the regular cube of the book, value for value. `regularize_check_book(regularize_run_values, cubes_record, "P16D", 40L, regularize_bands)` 24. Reads a small square at the native resolution of two bands, keeping the read in cache/regularize/. `regularize_images_patch(cube_listing, cubes_record, list(tile = "20LLP", side = 480L, bands = c("B04", "B11")), "CLOUD", "cache/regularize")` 25. Finds which image gives each pixel of each period of the year, in the order sits takes them. `regularize_year_sources(regularize_alone, "20LLP", "B02")` 26. Draws every image of the tile as it comes, in natural colour, in one sheet, for the website. `site_write_png(cube_images_ard_sheet(cubes_record$tiles[["20LKP"]], cube_states[["20LKP"]], cubes_record$bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/cube-ard-20lkp.png")` 27. Draws where each image of the tile has data and where sits masks it, in one sheet, for the website. `site_write_png(cube_images_mask_sheet(cube_states[["20LKP"]])$sheet[, , 1L], "../web/public/data/cube-ard-20lkp-mask.png")` 28. Writes the book's call, its images and regular cube, the map and the year, for the website. ``site_write_json(cube_export_record(cubes_record, cube_states, cube_stretch, cube_year_codes, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), list(tiles = list(`20LKP` = list(ard = "cube-ard-20lkp.png", mask = "cube-ard-20lkp-mask.png", regular = "cube-regular-20lkp.png", map = "cube-map-20lkp.png"), `20LLP` = list(ard = "cube-ard-20llp.png", mask = "cube-ard-20llp-mask.png", regular = "cube-regular-20llp.png", map = "cube-map-20llp.png")), year = "cube-year.bin")), "../web/src/data/cubes.json")`` 29. Stops the run unless the images, stacked in sits' order, give the regular cube of the year. `regularize_check_year(regularize_sources, cubes_record)` 30. Gathers the runs, the images as they come and the sources of the year, once both self-checks pass. `regularize_export_gather(cube_listing, regularize_run_values, regularize_coverage, regularize_patch, regularize_sources, regularize_bands, list(regularize_book_check, regularize_year_check))` 31. Writes the runs, the images as they come, the sources of the year and how sits masks clouds, for the website. ``site_write_json(regularize_export_record(regularize_record, cubes_record, regularize_cloud, regularize_grids, list(coverage = "regularize-coverage.png", patch = list(B04 = "regularize-patch-b04.png", B11 = "regularize-patch-b11.png"), runs = list(run_p16d_40m = list(`20LKP` = "cube-regular-20lkp.png", `20LLP` = "cube-regular-20llp.png"), run_p16d_20m = list(`20LKP` = "regularize-run-p16d-20m-20lkp.png", `20LLP` = "regularize-run-p16d-20m-20llp.png"), run_p16d_80m = list(`20LKP` = "regularize-run-p16d-80m-20lkp.png", `20LLP` = "regularize-run-p16d-80m-20llp.png")), sheets = list(list(period = "P16D", res = 20L), list(period = "P16D", res = 80L)), sources = "regularize-sources.bin")), "../web/src/data/regularize.json")`` 32. Draws one tile whole on the date of the book's figure, in natural colour, for the website. `site_write_png(regularize_images_coverage_picture(regularize_record, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L), "../web/public/data/regularize-coverage.png")` 33. Draws a small square of one band at its native resolution, in grey, for the website. `site_write_png(regularize_images_patch_picture(regularize_record, "B04", c(0.02, 0.98)), "../web/public/data/regularize-patch-b04.png")` 34. Draws a small square of one band at its native resolution, in grey, for the website. `site_write_png(regularize_images_patch_picture(regularize_record, "B11", c(0.02, 0.98)), "../web/public/data/regularize-patch-b11.png")` The files: - `cubes.json`: SHA-256 `f29bb2e9135a6f824c0d512868ea1c3d67bd2532c285b6662f87926e071d5874`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize.json`: SHA-256 `880aaea58eae1cb95f1780d87cdb6fec7a34e99ab7eaaf1fe5aa8f5c5fe0ce6d`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize-coverage.png`: SHA-256 `9a5b823851a5ca7b88c2afb37a61ffafb8b789484db89585c133031335248ec8`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize-patch-b04.png`: SHA-256 `79e546db5db43cb2cf68a7d177572a2f176d05ee787146b27c2ced6dfacdcf46`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize-patch-b11.png`: SHA-256 `bdf7e7396ad6e6ac56b7ec7bca5e874df5c01726da2b02f7bf01d39f1e000af6`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-ard-20lkp.png`: SHA-256 `54d23212cc41e26d91b44a38ca8b2031aca7243db4e6e1a1658138bd06119fef`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-ard-20lkp-mask.png`: SHA-256 `38c90658f1803e743308bfe78f8624bdbc10bb4341cfedc569bc2416ba4ff243`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. To make them again: `cd pipeline && Rscript make.R`. ## 3. What must a regular cube agree on? In the sits book: [Regularizing Sentinel-2 images](https://e-sensing.github.io/sitsbook/dc_regularize.html#regularizing-sentinel-2-images). The book asks sits to make those images a regular cube: cells of 40 m, and one image for every 16 days. The book's code: ``` reg_cube_rondonia <- sits_regularize( cube = s2_cube_local, output_dir = tempdir_r_s2_reg, res = 40, period = "P16D", multicores = 6) ``` Where the sits book writes it: [dc_regularize.qmd, lines 204–209](https://github.com/e-sensing/sitsbook/blob/bc75e250e50bc6fb8222593973b8dfe3b692201f/dc_regularize.qmd#L204-L209). The cube keeps the projection of each tile: WGS 84 / UTM zone 20S, EPSG:32720, for every tile of the call. And it keeps the tile's extent. 20LLP runs from x 300,000 to 409,800 m and from y 8,790,220 to 8,900,020 m: 109.8 km a side. The figure draws 9 recorded runs of `sits_regularize()` on the 69 images of the book's call: 3 periods at 3 cell sizes. res = 40 sets the cell: a square of 40 m. Over the region, the cube of 20LLP is 256 by 200 cells; 20LKP has 372 by 200. Each tile keeps a grid of its own, on its own pixels. Every band is resampled to the cell. Under one 40 m cell lie 4 × 4 pixels of B02 (10 m) and 2 × 2 pixels of B8A and B11 (20 m). In 20LLP the cells sit exactly on them; in 20LKP their edges lie off the 20 m pixels by 10 m in x. period = "P16D" cuts time into 4 intervals of 16 days, each from its date t up to t + 16 days, that day left out. Each image belongs to the one it falls in. The timeline starts at the latest first date of the tiles, 3 Jul 2018, and stops at the earliest last date, 27 Aug 2018: the two images of 20LLP of 30 Jun 2018 fall outside. This is what sits asks gdalcubes to build, one view per tile, band and date. Change the period or the cell and see the grid and the dates change. ### The figure Tile bounds and cell grids as the pipeline's runs of sits_regularize() wrote them over the region; the pictures are their first regular images of the book's period. The cube view is written from sits' source, its extent shortened. The figure (recorded: 9 runs of sits_regularize(), 3 periods at 3 cell sizes) at its last beat: the tiles 20LKP and 20LLP in EPSG:32720, the region cut out of each as a grid of cells, one cell over the pixels of the bands, the images on a time axis with the intervals of the period, and the cube view sits asks gdalcubes for. Pills choose the period and the cell size. Cells of the region per tile, and where the cells lie on the pixels of each band size: | Cell | 20LKP cells | 20LLP cells | On the 10 m pixels (B02) | On the 20 m pixels (B8A and B11) | | ---- | ----------- | ----------- | ------------------------ | ----------------------------------- | | 20 m | 745 × 400 | 512 × 400 | 20LKP on and 20LLP on | 20LKP on and 20LLP on | | 40 m | 372 × 200 | 256 × 200 | 20LKP on and 20LLP on | 20LKP off by 10 m in x and 20LLP on | | 80 m | 186 × 100 | 128 × 100 | 20LKP on and 20LLP on | 20LKP off by 10 m in x and 20LLP on | The timeline and intervals of each period (an interval ends on the day written): | Period | Dates | Intervals | Images outside the timeline | | ------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- | | P8D | 7 | 2018-07-03 to 2018-07-10; 2018-07-11 to 2018-07-18; 2018-07-19 to 2018-07-26; 2018-07-27 to 2018-08-03; 2018-08-04 to 2018-08-11; 2018-08-12 to 2018-08-19; 2018-08-20 to 2018-08-27 | 20LLP: 2018-06-30, 2018-06-30, 2018-08-29 | | P16D | 4 | 2018-07-03 to 2018-07-18; 2018-07-19 to 2018-08-03; 2018-08-04 to 2018-08-19; 2018-08-20 to 2018-09-04 | 20LLP: 2018-06-30, 2018-06-30 | | P1M | 2 | 2018-07-01 to 2018-07-31; 2018-08-01 to 2018-08-31 | 20LLP: 2018-06-30, 2018-06-30 | The cube view of the book's call: ```r cube_view(extent = list(, t0 = date, t1 = date), srs = "EPSG:32720", dt = "P16D", dx = 40, dy = 40, aggregation = "first", resampling = "bilinear") ``` #### What is read at 40 m: B02 of 20LLP on 18 Jul 2018 Measured on one image, S2B_20LLP_20180718_1_L2A. Its B02 file is 10,980 pixels wide, 10 m each, and it holds smaller copies, overviews, of 5,490, 2,745, 1,373 and 687 pixels: 20 m, 40 m, 80 m and 159.8 m. Measured: the share of the pixels of a 40 m run of this image alone that gdalwarp gives back, by what it reads: | gdalwarp | Reads | Same as the run | | --------- | ----------------------------------- | --------------- | | -ovr AUTO | GDAL picks the overview | 100% | | -ovr 1 | the overview of 2,745 pixels (40 m) | 100% | | -ovr NONE | the full 10 m pixels | 7% | So at 40 m sits reads the file's overview, not its full pixels. A pixel counts as the same when sits' stored integer is the one nearest gdalwarp's value; 3 of them fall on a tie of one half. Measured: the grid each tile's cube was written on, over the region, at 40 m. | Tile | 40 m cells | On the 10 m pixels | On the 20 m pixels | | ----- | ------------------------ | ------------------ | ------------------ | | 20LKP | 372 × 200 from x 294,910 | on their edges | off by 10 m in x | | 20LLP | 256 × 200 from x 300,000 | on their edges | on their edges | #### How it was made 1. Ask the catalogue: sits asks the provider's catalogue which images cover the book's tiles, bands and dates, and a whole year of one tile. The answers are kept, so every later run uses the same list. 2. Find the tiles: The pipeline finds the tiles of the satellite's grid around the region, from the grid that comes with sits. It reads a small picture of each tile of the cube from its least cloudy image. 3. Build the regular cube: sits makes a regular cube of the region from those images: one image per period, all on the same grid of pixels. For each pixel of a period it takes the first valid value, least cloudy image first. 4. Read the images as they come: The pipeline reads the images the catalogue listed, as they come. It marks each pixel as having no data, a clear value, or a value sits masks, such as cloud or shadow. 5. Try other periods and pixel sizes: sits makes the regular cube again at other periods and pixel sizes. It also makes it once without the images' cloud cover, so sits takes the images in date order, not least cloudy first. 6. Read each image alone: Each image is read alone, on the grid sits builds for it. Stacked in the order sits takes them, the images show which one gave each pixel of the regular cube. 7. Check against sits: The pipeline compares its own reads with what sits made, and stops the run on the checks that must hold. The results are gathered only once those checks pass. 8. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each. 9. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - A rerun on aarch64-apple-darwin23 gave the same files, byte for byte. Every step of the pipeline: 1. Lists the bands read: the book's and those of the natural-colour pictures, in the order sits has them. `cube_read_bands("AWS", "SENTINEL-2-L2A", c("B02", "B8A", "B11"), c("B03", "B04"))` 2. Lists every period with every resolution of the runs. `regularize_choices(c("P8D", "P16D", "P1M"), c(20L, 40L, 80L))` 3. The arguments of the book's call to sits_cube(). `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands)` 4. The same call for one tile over a whole year. `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands, "20LLP", "2018-01-01", "2018-12-31")` 5. Lists the bands of the runs: the natural-colour ones, in the order sits has them. `regularize_run_bands(cube_bands, list(red = "B04", green = "B03", blue = "B02"))` 6. Lists the images of the book's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_book_call, "cache/cubes", "cube_book")` 7. Lists the images of the year's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_year_call, "cache/cubes", "cube_year")` 8. Reads how sits masks the cloud band, and the line of its source file that says so. `collection_cloud_source("cache/src/sits-78d26eb07370874192911854b48b60ebb1d668c8", "AWS", "SENTINEL-2-L2A", "CLOUD", regularize_bands)` 9. Makes the regular cube of the book's call over the region with sits_regularize(), keeping the files already made. `cube_regularize(cube_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_book_B02-B03-B04-B8A-B11", 4L)` 10. Finds the Sentinel-2 tiles around the region. `cube_images_grid(list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), cube_listing, 100L)` 11. Reads a small picture of each tile from the overview of its clearest image, keeping the reads in cache/cubes/. `cube_images_read(cube_listing, list(red = "B04", green = "B03", blue = "B02"), 3L, "cache/cubes")` 12. Makes the regular cube of the year with sits_regularize(), keeping the files already made. `cube_regularize(cube_year_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_year_B02-B03-B04-B8A-B11", 4L)` 13. Reads one tile whole on the date of the book's figure, from an overview, keeping the read in cache/regularize/. `regularize_images_coverage(cube_listing, list(tile = "20LLP", date = "2018-06-30"), list(red = "B04", green = "B03", blue = "B02"), 3L, "cache/regularize")` 14. Runs sits_regularize() on the book's images at one period and resolution, keeping the files in cache/regularize/. `regularize_run_cube(cube_listing, regularize_choice, regularize_bands, "CLOUD", list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "cache/regularize", 4L, 1L)` 15. Reads every image of each tile over the tile's part of the region, as it comes, keeping the reads in cache/cubes/. `cube_tile_images(cube_listing, cube_book, cube_book_call, 40L, "EPSG:32720", "cache/cubes")` 16. Compares the grid of each tile of the book's run with the view sits builds for it. `regularize_book_grids(regularize_find_run(regularize_run, "P16D", 40L), cube_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L))` 17. Reads the regular images each run wrote. `regularize_values(regularize_run, regularize_bands)` 18. Gathers the images, ranked in the order sits takes them, the regular values, the year and the map. `cube_export_gather(cube_listing, cube_book, cube_images, cube_year_listing, cube_year, cube_map_images, cube_map_grid, cube_book_call, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), 40L)` 19. Marks each pixel of each image as no data, clear or masked by sits. `purrr::map(cubes_record$tiles, cube_images_states, cubes_record$cloud_values)` 20. Picks the reflectance drawn black and white, one for red, green and blue. `cube_images_stretch(cubes_record, list(red = "B04", green = "B03", blue = "B02"), c(0.02, 0.98))` 21. Stores the year in one byte per value, each band between two percentiles. `cube_export_year_codes(cubes_record$year$values, c(0.001, 0.999))` 22. Reads each image of the book's periods alone, on the view sits builds for it with no mask, keeping the reads in cache/regular_cubes/. `cube_alone_book_reads(cube_listing, cubes_record, "B02", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "cache/regular_cubes/alone_book_B02", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp", 0.5)` 23. Runs sits_regularize() on the book's call with the scene cloud cover removed, keeping the result in cache/regular_cubes/. `cube_checks_copied_run(cube_listing, "B02", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP" ), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list( crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), cubes_record, "cache/regular_cubes", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp")` 24. Reads the grid of the region on each tile at each resolution from the runs of the book's period. `cube_checks_region_grids(cube_listing, regularize_run, cubes_record, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), c(20L, 40L, 80L))` 25. Runs sits_regularize() on one image alone and gdalwarp at three overview choices, keeping the result in cache/regular_cubes/. `cube_checks_overview_check_run(cube_listing, cubes_record, list(tile = "20LLP", band = "B02"), list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP" ), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98 ), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), "cache/regular_cubes", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp")` 26. Reads the same image alone in natural colour and runs sits_regularize() on it with those bands. `cube_checks_pictures_check_run(cube_listing, cubes_record, list(tile = "20LLP", band = "B02"), regularize_bands, list( source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98 ), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), "cache/regular_cubes", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp")` 27. Pairs the cells of the strip both tiles cover. `cube_checks_book_strip(cubes_record, c("20LKP", "20LLP" ), 40L)` 28. Runs sits_regularize() on each image of the year alone, with the date of its period, keeping the files in cache/regularize/. `regularize_year_alone(cube_year_listing, cubes_record, "B02", "CLOUD", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "cache/regularize/year_20llp_B02", 1L)` 29. Stops the run unless the book's run gives the regular cube of the book, value for value. `regularize_check_book(regularize_run_values, cubes_record, "P16D", 40L, regularize_bands)` 30. Reads a small square at the native resolution of two bands, keeping the read in cache/regularize/. `regularize_images_patch(cube_listing, cubes_record, list(tile = "20LLP", side = 480L, bands = c("B04", "B11")), "CLOUD", "cache/regularize")` 31. Stacks the images of each period in sits' order and finds the one each pixel came from. `cube_alone_book_stacks(regular_cubes_book_reads, cubes_record, "B02")` 32. Compares the run with no scene cloud cover with the book's regular cube, after checking it against the images read alone. `cube_checks_copied_entries(regular_cubes_copied_run, cubes_record, regular_cubes_book_reads, cube_listing, "B02")` 33. Compares the two tiles' regular images where they overlap. `cube_checks_overlap_periods(cubes_record, regular_cubes_strip, "B02")` 34. Stops the run unless gdalwarp -ovr AUTO and the images read alone give sits' values on every pixel. `cube_checks_check_overview(regular_cubes_overview, regular_cubes_book_reads, regular_cubes_pictures)` 35. Draws one tile's images as they come and its regular images, small, for the journey's picture. `cube_checks_export_picture(cubes_record, cube_states, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L, list(tile = "20LLP", card_tile = 64L, sheet_tile = 192L))` 36. Finds which image gives each pixel of each period of the year, in the order sits takes them. `regularize_year_sources(regularize_alone, "20LLP", "B02")` 37. Draws the regular images sits made of the tile, one per date, for the website. `site_write_png(cube_images_regular_sheet(cubes_record$regular[["20LKP"]], cubes_record$bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/cube-regular-20lkp.png")` 38. Draws the regular images sits made of the tile, one per date, for the website. `site_write_png(cube_images_regular_sheet(cubes_record$regular[["20LLP"]], cubes_record$bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/cube-regular-20llp.png")` 39. Writes the book's call, its images and regular cube, the map and the year, for the website. ``site_write_json(cube_export_record(cubes_record, cube_states, cube_stretch, cube_year_codes, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), list(tiles = list(`20LKP` = list(ard = "cube-ard-20lkp.png", mask = "cube-ard-20lkp-mask.png", regular = "cube-regular-20lkp.png", map = "cube-map-20lkp.png"), `20LLP` = list(ard = "cube-ard-20llp.png", mask = "cube-ard-20llp-mask.png", regular = "cube-regular-20llp.png", map = "cube-map-20llp.png")), year = "cube-year.bin")), "../web/src/data/cubes.json")`` 40. Stops the run unless the images read alone, stacked in sits' order, give the book's regular cube on every pixel. `cube_alone_check_book(regular_cubes_book, cubes_record, "B02", "cache/regular_cubes/alone_book_B02")` 41. Reads each image of the year alone in natural colour, with its cloud band, keeping the reads in cache/regular_cubes/. `cube_alone_year_reads(cube_year_listing, cubes_record, regularize_sources, regularize_bands, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP" ), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98 ), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), "cache/regular_cubes/alone_year_20llp", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp", 0.5)` 42. Stops the run unless the images, stacked in sits' order, give the regular cube of the year. `regularize_check_year(regularize_sources, cubes_record)` 43. Stops the run unless the cloud band read here masks each image of the year as sits did. `cube_alone_check_year_cloud(regular_cubes_year_reads, regularize_alone, cubes_record$cloud_values, "cache/regular_cubes/alone_year_20llp")` 44. Gathers the runs, the images as they come and the sources of the year, once both self-checks pass. `regularize_export_gather(cube_listing, regularize_run_values, regularize_coverage, regularize_patch, regularize_sources, regularize_bands, list(regularize_book_check, regularize_year_check))` 45. Gathers the grids, the sources of the book's pixels, the overview, the run with no cloud cover and the year, once the self-checks pass. `cube_checks_export_gather(regular_cubes_grids, regular_cubes_book, regular_cubes_strip, regular_cubes_overlap, regular_cubes_overview, regular_cubes_copied, regular_cubes_year_reads, list(regular_cubes_book_check, regular_cubes_overview_check, regular_cubes_cloud_check))` 46. Writes the runs, the images as they come, the sources of the year and how sits masks clouds, for the website. ``site_write_json(regularize_export_record(regularize_record, cubes_record, regularize_cloud, regularize_grids, list(coverage = "regularize-coverage.png", patch = list(B04 = "regularize-patch-b04.png", B11 = "regularize-patch-b11.png"), runs = list(run_p16d_40m = list(`20LKP` = "cube-regular-20lkp.png", `20LLP` = "cube-regular-20llp.png"), run_p16d_20m = list(`20LKP` = "regularize-run-p16d-20m-20lkp.png", `20LLP` = "regularize-run-p16d-20m-20llp.png"), run_p16d_80m = list(`20LKP` = "regularize-run-p16d-80m-20lkp.png", `20LLP` = "regularize-run-p16d-80m-20llp.png")), sheets = list(list(period = "P16D", res = 20L), list(period = "P16D", res = 80L)), sources = "regularize-sources.bin")), "../web/src/data/regularize.json")`` 47. Draws the regular images of one run of one tile, one per date, for the website. `site_write_png(regularize_export_run_sheet(regularize_find_run(regularize_record$runs, "P16D", 20L), "20LKP", regularize_bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/regularize-run-p16d-20m-20lkp.png")` 48. Draws the regular images of one run of one tile, one per date, for the website. `site_write_png(regularize_export_run_sheet(regularize_find_run(regularize_record$runs, "P16D", 20L), "20LLP", regularize_bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/regularize-run-p16d-20m-20llp.png")` 49. Draws the regular images of one run of one tile, one per date, for the website. `site_write_png(regularize_export_run_sheet(regularize_find_run(regularize_record$runs, "P16D", 80L), "20LKP", regularize_bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/regularize-run-p16d-80m-20lkp.png")` 50. Draws the regular images of one run of one tile, one per date, for the website. `site_write_png(regularize_export_run_sheet(regularize_find_run(regularize_record$runs, "P16D", 80L), "20LLP", regularize_bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/regularize-run-p16d-80m-20llp.png")` 51. Writes the grids, the sources of the book's pixels, the overview, the run with no cloud cover and the year, for the website. `site_write_json(cube_checks_export_record(regular_cubes_record, cubes_record, regularize_sources, regular_cubes_picture, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "B02", list(ranks = "regular-cubes-book-ranks.bin", cards = "regular-cubes-picture-cards-20llp.png", sheets = "regular-cubes-picture-sheets-20llp.png")), "../web/src/data/regular-cubes.json")` The files: - `regular-cubes.json`: SHA-256 `751974674c0e71d20c1054ce3e79129531244a10501768116130594eea997323`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize.json`: SHA-256 `880aaea58eae1cb95f1780d87cdb6fec7a34e99ab7eaaf1fe5aa8f5c5fe0ce6d`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cubes.json`: SHA-256 `f29bb2e9135a6f824c0d512868ea1c3d67bd2532c285b6662f87926e071d5874`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-regular-20lkp.png`: SHA-256 `04f081ce4387b725b64ceda711778ece73aac4016074f4caf05a57e80d3d5480`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-regular-20llp.png`: SHA-256 `2bedbff2a14c1f5a7b02f42a882e2b54f10f8fc9c97338076977bde0eb839820`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize-run-p16d-20m-20lkp.png`: SHA-256 `4bd4a015549fdb05e3fb41e87c361dc0772171ee41ace099645bbae90ba6cc2f`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize-run-p16d-20m-20llp.png`: SHA-256 `c495a2b592a81a5e21e5d885e72dc4eb1f9443592d53bf4441d4c95a0c32b8e2`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize-run-p16d-80m-20lkp.png`: SHA-256 `fc6eb4a93592b5f989ed390d468bfa595c2c963e8f822def634067ec9eff0dde`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize-run-p16d-80m-20llp.png`: SHA-256 `e0bbf6d11dfec56523a11029c309845ef39818776895522d05f97066047218fb`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. To make them again: `cd pipeline && Rscript make.R`. Try it: [Period and cell of a regular cube](/interactive-images/regular-cubes-period-resolution.md) ## 4. What does sits_regularize() do? In the sits book: [Regularizing Sentinel-2 images](https://e-sensing.github.io/sitsbook/dc_regularize.html#regularizing-sentinel-2-images). The book asks sits to give each 40 m cell one value per 16-day period, from the least cloudy image of the period that sees the ground there. The book's code: ``` reg_cube_rondonia <- sits_regularize( cube = s2_cube_local, output_dir = tempdir_r_s2_reg, res = 40, period = "P16D", multicores = 6) ``` Where the sits book writes it: [dc_regularize.qmd, lines 204–209](https://github.com/e-sensing/sitsbook/blob/bc75e250e50bc6fb8222593973b8dfe3b692201f/dc_regularize.qmd#L204-L209). The images of one tile, 20LLP, as they come: 45 items on 25 days, each a card at its date. 23 of them hold no data over the region, drawn empty: the satellite's swath passed beside it. The figure draws one recorded run of `sits_regularize()`: the book's call on 69 images of 2 tiles. period = "P16D" cuts the axis into four bins of 16 days, each from its date t up to t + 16 days, that day left out. Each image goes to the bin it falls in; the two items of 30 Jun 2018 fall before the first bin, and are left out. In each bin sits takes the images by scene cloud cover, least first: the cloud_cover the provider's catalogue gives for the whole scene. Equal covers keep the collection's order. The 21 items in the bins with no data here keep their place in the order, with nothing to give. For each cell, sits keeps the first value, in that order, that the image has there and the CLOUD band does not mask: aggregation "first". It writes one image per period at 40 m and crops each file to the region: on 20LKP the 373 × 200 cells of the view become 372 × 200; the files of 20LLP already lay inside the region and were copied as they are. In these two dry months the first image fills nearly every cell of every period, on both tiles: where it fills least, 74,393 of 74,400 (20LKP, 3 Jul). Open a period to see how its cells were chosen, image by image. Switch to 20LKP: each tile is regularized on its own grid, apart from its neighbour. In these two dry months the first image gives almost every cell. Over the year it does not: here are the 176 images of 20LLP in 2018, an extension of the book's example, in 23 periods. The first image gives half the cells or more in 11 of them, and nothing in 10. In 12 Apr to 27 Apr 2018, the first image has no data here and gives nothing; the 2nd and 3rd give 52% and 38%, and 10% of the cells stay empty. Open a period of the rainy season. ### The figure The images as sits_cube() lists them, each drawn over the region in natural colour, and the regular images sits_regularize() wrote. The figure (recorded: one run of sits_regularize() on 69 images of 2 tiles, 4 periods of P16D at 40 m) at the beat that opens the periods: the images of a tile as cards on a time axis, cut into P16D bins; in each bin the cards in sits' order; the sits_regularize() box; one regular image per period coming out of it. Each period opens its details; a toggle switches the tile. 20LLP: the two items of 30 Jun 2018 fall before the first bin. | Tile | Period | Images | With data in the region | First in sits' order (scene cloud cover) | Cells the first gave | Cells none could fill | | ----- | -------------------- | ------ | ----------------------- | ---------------------------------------- | -------------------- | --------------------- | | 20LKP | 3 Jul to 18 Jul 2018 | 6 | 6 | 3 Jul 2018 (< 0.1%) | 74,393 of 74,400 | 0 | | 20LKP | 19 Jul to 3 Aug 2018 | 6 | 6 | 23 Jul 2018 (< 0.1%) | 74,400 of 74,400 | 0 | | 20LKP | 4 Aug to 19 Aug 2018 | 8 | 6 | 12 Aug 2018 (< 0.1%) | 74,399 of 74,400 | 0 | | 20LKP | 20 Aug to 4 Sep 2018 | 4 | 4 | 22 Aug 2018 (< 0.1%) | 74,399 of 74,400 | 0 | | 20LLP | 3 Jul to 18 Jul 2018 | 11 | 6 | 18 Jul 2018 (< 0.1%) | 51,200 of 51,200 | 0 | | 20LLP | 19 Jul to 3 Aug 2018 | 9 | 6 | 28 Jul 2018 (< 0.1%) | 51,199 of 51,200 | 0 | | 20LLP | 4 Aug to 19 Aug 2018 | 16 | 6 | 12 Aug 2018 (< 0.1%) | 51,199 of 51,200 | 0 | | 20LLP | 20 Aug to 4 Sep 2018 | 7 | 4 | 22 Aug 2018 (< 0.1%) | 51,199 of 51,200 | 0 | At its last beat the line shows the year of one tile instead, each period's regular image shaded by where its cells came from; each period opens its details. **The year.** The 176 images of 20LLP in 2018, an extension of the book's example, in 23 periods; 52 images give cells to the regular cube. | Period | Images | With data in the region | Images that give cells | The first gives | Cells that stay empty | | ------------------------- | ------ | ----------------------- | ---------------------- | --------------- | --------------------- | | 6 Jan to 21 Jan 2018 | 5 | 3 | 2 | 0% | 39% | | 22 Jan to 6 Feb 2018 | 3 | 1 | 0 | 0% | 100% | | 7 Feb to 22 Feb 2018 | 9 | 4 | 3 | 43% | 54% | | 23 Feb to 10 Mar 2018 | 6 | 4 | 2 | 96% | 1% | | 11 Mar to 26 Mar 2018 | 6 | 4 | 3 | 83% | 17% | | 27 Mar to 11 Apr 2018 | 7 | 3 | 3 | 91% | 5% | | 12 Apr to 27 Apr 2018 | 4 | 2 | 2 | 0% | 10% | | 28 Apr to 13 May 2018 | 5 | 3 | 3 | 79% | 8% | | 14 May to 29 May 2018 | 7 | 4 | 3 | 0% | 19% | | 30 May to 14 Jun 2018 | 9 | 5 | 5 | 0% | 60% | | 15 Jun to 30 Jun 2018 | 13 | 5 | 1 | 0% | 0% | | 1 Jul to 16 Jul 2018 | 9 | 4 | 2 | > 99% | 0% | | 17 Jul to 1 Aug 2018 | 9 | 6 | 2 | > 99% | 0% | | 2 Aug to 17 Aug 2018 | 16 | 8 | 2 | > 99% | 0% | | 18 Aug to 2 Sep 2018 | 11 | 6 | 2 | > 99% | 0% | | 3 Sep to 18 Sep 2018 | 13 | 5 | 3 | 0% | 0% | | 19 Sep to 4 Oct 2018 | 10 | 6 | 3 | > 99% | 0% | | 5 Oct to 20 Oct 2018 | 9 | 3 | 1 | 0% | 56% | | 21 Oct to 5 Nov 2018 | 12 | 5 | 4 | 59% | 40% | | 6 Nov to 21 Nov 2018 | 1 | 1 | 1 | 82% | 18% | | 22 Nov to 7 Dec 2018 | 3 | 1 | 1 | 0% | 37% | | 8 Dec to 23 Dec 2018 | 6 | 3 | 3 | 2% | 21% | | 24 Dec 2018 to 8 Jan 2019 | 3 | 2 | 1 | 0% | 76% | #### The images of 20LKP, 3 Jul to 18 Jul 2018 6 items of 20LKP fall in this period; 6 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 74,393 of 74,400 cells; the later images 7. **Where each cell came from.** All but 7 cells came from the 1st image (> 99%). Cells no image could fill: 0 (0%). | Place | Date | Scene cloud cover | On the grid | Cells it gave | Share | | ----- | ----------- | ----------------- | ----------- | ------------- | ----- | | 1st | 3 Jul 2018 | < 0.1% | < 1% masked | 74,393 | > 99% | | 2nd | 8 Jul 2018 | < 0.1% | < 1% masked | 7 | < 1% | | 3rd | 3 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 4th | 18 Jul 2018 | 0.1% | < 1% masked | 0 | 0% | | 5th | 18 Jul 2018 | 0.1% | 0% masked | 0 | 0% | | 6th | 13 Jul 2018 | 4.8% | < 1% masked | 0 | 0% | **Tiles stay apart.** Both tiles cover a strip 245 cells wide. sits regularizes each tile on its own grid: here the cells of 20LKP lie 10 m off those of 20LLP, so the two cubes never hold the same cell. On the paired cells their B02 is equal on < 1% and differs by a median of 90. Copied first? The book copies the images with sits_cube_copy() before it regularizes; the copies carry no cloud_cover, so sits takes them by date. Run here as a stand-in: the same images regularized with cloud_cover removed. By date, the period starts on 3 Jul 2018 in both orders, and 94% of the cells of B02 change, by a median of 4 on a median of 171. #### The images of 20LKP, 19 Jul to 3 Aug 2018 6 items of 20LKP fall in this period; 6 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 74,400 of 74,400 cells; the later images none. **Where each cell came from.** Every cell came from the 1st image. Cells no image could fill: 0 (0%). | Place | Date | Scene cloud cover | On the grid | Cells it gave | Share | | ----- | ----------- | ----------------- | ----------- | ------------- | ----- | | 1st | 23 Jul 2018 | < 0.1% | 0% masked | 74,400 | 100% | | 2nd | 28 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 3rd | 28 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 4th | 23 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 5th | 2 Aug 2018 | 11.5% | < 1% masked | 0 | 0% | | 6th | 2 Aug 2018 | 13.6% | < 1% masked | 0 | 0% | **Tiles stay apart.** Both tiles cover a strip 245 cells wide. sits regularizes each tile on its own grid: here the cells of 20LKP lie 10 m off those of 20LLP, so the two cubes never hold the same cell. On the paired cells their B02 is equal on < 1% and differs by a median of 61. Copied first? The book copies the images with sits_cube_copy() before it regularizes; the copies carry no cloud_cover, so sits takes them by date. Run here as a stand-in: the same images regularized with cloud_cover removed. By date, the period starts on 23 Jul 2018 in both orders, and 97% of the cells of B02 change, by a median of 8 on a median of 202. #### The images of 20LKP, 4 Aug to 19 Aug 2018 8 items of 20LKP fall in this period; 6 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 74,399 of 74,400 cells; the later images 1. **Where each cell came from.** All but 1 cell came from the 1st image (> 99%). Cells no image could fill: 0 (0%). | Place | Date | Scene cloud cover | On the grid | Cells it gave | Share | | ----- | ----------- | ----------------- | ----------------- | ------------- | ----- | | 1st | 12 Aug 2018 | < 0.1% | < 1% masked | 74,399 | > 99% | | 2nd | 12 Aug 2018 | < 0.1% | < 1% masked | 1 | < 1% | | 3rd | 17 Aug 2018 | 2.1% | outside the swath | 0 | 0% | | 4th | 17 Aug 2018 | 2.2% | outside the swath | 0 | 0% | | 5th | 17 Aug 2018 | 10.0% | < 1% masked | 0 | 0% | | 6th | 17 Aug 2018 | 13.5% | < 1% masked | 0 | 0% | | 7th | 7 Aug 2018 | 74.1% | 24% masked | 0 | 0% | | 8th | 7 Aug 2018 | 78.3% | 38% masked | 0 | 0% | **Tiles stay apart.** Both tiles cover a strip 245 cells wide. sits regularizes each tile on its own grid: here the cells of 20LKP lie 10 m off those of 20LLP, so the two cubes never hold the same cell. On the paired cells their B02 is equal on < 1% and differs by a median of 73. Copied first? The book copies the images with sits_cube_copy() before it regularizes; the copies carry no cloud_cover, so sits takes them by date. Run here as a stand-in: the same images regularized with cloud_cover removed. By date, the period starts on 7 Aug 2018, not 12 Aug 2018, and > 99% of the cells of B02 change, by a median of 40 on a median of 228. #### The images of 20LKP, 20 Aug to 4 Sep 2018 4 items of 20LKP fall in this period; 4 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 74,399 of 74,400 cells; the later images 1. **Where each cell came from.** All but 1 cell came from the 1st image (> 99%). Cells no image could fill: 0 (0%). | Place | Date | Scene cloud cover | On the grid | Cells it gave | Share | | ----- | ----------- | ----------------- | ----------- | ------------- | ----- | | 1st | 22 Aug 2018 | < 0.1% | < 1% masked | 74,399 | > 99% | | 2nd | 22 Aug 2018 | < 0.1% | < 1% masked | 1 | < 1% | | 3rd | 27 Aug 2018 | 17.7% | 72% masked | 0 | 0% | | 4th | 27 Aug 2018 | 32.6% | 88% masked | 0 | 0% | **Tiles stay apart.** Both tiles cover a strip 245 cells wide. sits regularizes each tile on its own grid: here the cells of 20LKP lie 10 m off those of 20LLP, so the two cubes never hold the same cell. On the paired cells their B02 is equal on 4% and differs by a median of 8. Copied first? The book copies the images with sits_cube_copy() before it regularizes; the copies carry no cloud_cover, so sits takes them by date. Run here as a stand-in: the same images regularized with cloud_cover removed. By date, the period starts on 22 Aug 2018 in both orders, and 96% of the cells of B02 change, by a median of 6 on a median of 217. #### The images of 20LLP, 3 Jul to 18 Jul 2018 11 items of 20LLP fall in this period; 6 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 51,200 of 51,200 cells; the later images none. **Where each cell came from.** Every cell came from the 1st image. Cells no image could fill: 0 (0%). | Place | Date | Scene cloud cover | On the grid | Cells it gave | Share | | ----- | ----------- | ----------------- | ----------------- | ------------- | ----- | | 1st | 18 Jul 2018 | < 0.1% | < 1% masked | 51,200 | 100% | | 2nd | 8 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 3rd | 3 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 4th | 18 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 5th | 3 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 6th | 10 Jul 2018 | < 0.1% | outside the swath | 0 | 0% | | 7th | 13 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 8th | 5 Jul 2018 | 1.4% | outside the swath | 0 | 0% | | 9th | 5 Jul 2018 | 1.9% | outside the swath | 0 | 0% | | 10th | 15 Jul 2018 | 2.1% | outside the swath | 0 | 0% | | 11th | 15 Jul 2018 | 3.4% | outside the swath | 0 | 0% | **Tiles stay apart.** Both tiles cover a strip 245 cells wide. sits regularizes each tile on its own grid: here the cells of 20LKP lie 10 m off those of 20LLP, so the two cubes never hold the same cell. On the paired cells their B02 is equal on < 1% and differs by a median of 90. Copied first? The book copies the images with sits_cube_copy() before it regularizes; the copies carry no cloud_cover, so sits takes them by date. Run here as a stand-in: the same images regularized with cloud_cover removed. By date, the period starts on 3 Jul 2018, not 18 Jul 2018, and 100% of the cells of B02 change, by a median of 92 on a median of 322. #### The images of 20LLP, 19 Jul to 3 Aug 2018 9 items of 20LLP fall in this period; 6 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 51,199 of 51,200 cells; the later images 1. **Where each cell came from.** All but 1 cell came from the 1st image (> 99%). Cells no image could fill: 0 (0%). | Place | Date | Scene cloud cover | On the grid | Cells it gave | Share | | ----- | ----------- | ----------------- | ----------------- | ------------- | ----- | | 1st | 28 Jul 2018 | < 0.1% | < 1% masked | 51,199 | > 99% | | 2nd | 28 Jul 2018 | < 0.1% | < 1% masked | 1 | < 1% | | 3rd | 23 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 4th | 23 Jul 2018 | < 0.1% | < 1% masked | 0 | 0% | | 5th | 20 Jul 2018 | < 0.1% | outside the swath | 0 | 0% | | 6th | 25 Jul 2018 | 0.7% | outside the swath | 0 | 0% | | 7th | 2 Aug 2018 | 40.7% | < 1% masked | 0 | 0% | | 8th | 2 Aug 2018 | 47.3% | < 1% masked | 0 | 0% | | 9th | 30 Jul 2018 | 79.9% | outside the swath | 0 | 0% | **Tiles stay apart.** Both tiles cover a strip 245 cells wide. sits regularizes each tile on its own grid: here the cells of 20LKP lie 10 m off those of 20LLP, so the two cubes never hold the same cell. On the paired cells their B02 is equal on < 1% and differs by a median of 61. Copied first? The book copies the images with sits_cube_copy() before it regularizes; the copies carry no cloud_cover, so sits takes them by date. Run here as a stand-in: the same images regularized with cloud_cover removed. By date, the period starts on 20 Jul 2018, not 28 Jul 2018, and > 99% of the cells of B02 change, by a median of 54 on a median of 340. #### The images of 20LLP, 4 Aug to 19 Aug 2018 16 items of 20LLP fall in this period; 6 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 51,199 of 51,200 cells; the later images 1. **Where each cell came from.** All but 1 cell came from the 1st image (> 99%). Cells no image could fill: 0 (0%). | Place | Date | Scene cloud cover | On the grid | Cells it gave | Share | | ----- | ----------- | ----------------- | ----------------- | ------------- | ----- | | 1st | 12 Aug 2018 | < 0.1% | < 1% masked | 51,199 | > 99% | | 2nd | 12 Aug 2018 | < 0.1% | < 1% masked | 1 | < 1% | | 3rd | 19 Aug 2018 | 0.2% | outside the swath | 0 | 0% | | 4th | 19 Aug 2018 | 0.3% | outside the swath | 0 | 0% | | 5th | 4 Aug 2018 | 1.0% | outside the swath | 0 | 0% | | 6th | 4 Aug 2018 | 1.0% | outside the swath | 0 | 0% | | 7th | 17 Aug 2018 | 1.2% | outside the swath | 0 | 0% | | 8th | 17 Aug 2018 | 1.4% | outside the swath | 0 | 0% | | 9th | 14 Aug 2018 | 1.8% | outside the swath | 0 | 0% | | 10th | 14 Aug 2018 | 2.0% | outside the swath | 0 | 0% | | 11th | 9 Aug 2018 | 6.0% | outside the swath | 0 | 0% | | 12th | 9 Aug 2018 | 8.8% | outside the swath | 0 | 0% | | 13th | 7 Aug 2018 | 17.2% | 8% masked | 0 | 0% | | 14th | 7 Aug 2018 | 26.9% | 17% masked | 0 | 0% | | 15th | 17 Aug 2018 | 30.3% | < 1% masked | 0 | 0% | | 16th | 17 Aug 2018 | 40.3% | < 1% masked | 0 | 0% | **Tiles stay apart.** Both tiles cover a strip 245 cells wide. sits regularizes each tile on its own grid: here the cells of 20LKP lie 10 m off those of 20LLP, so the two cubes never hold the same cell. On the paired cells their B02 is equal on < 1% and differs by a median of 73. Copied first? The book copies the images with sits_cube_copy() before it regularizes; the copies carry no cloud_cover, so sits takes them by date. Run here as a stand-in: the same images regularized with cloud_cover removed. By date, the period starts on 4 Aug 2018, not 12 Aug 2018, and 99% of the cells of B02 change, by a median of 36 on a median of 367. #### The images of 20LLP, 20 Aug to 4 Sep 2018 7 items of 20LLP fall in this period; 4 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 51,199 of 51,200 cells; the later images 1. **Where each cell came from.** All but 1 cell came from the 1st image (> 99%). Cells no image could fill: 0 (0%). | Place | Date | Scene cloud cover | On the grid | Cells it gave | Share | | ----- | ----------- | ----------------- | ----------------- | ------------- | ----- | | 1st | 22 Aug 2018 | < 0.1% | < 1% masked | 51,199 | > 99% | | 2nd | 22 Aug 2018 | < 0.1% | < 1% masked | 1 | < 1% | | 3rd | 29 Aug 2018 | 0.1% | outside the swath | 0 | 0% | | 4th | 24 Aug 2018 | 0.1% | outside the swath | 0 | 0% | | 5th | 24 Aug 2018 | 0.2% | outside the swath | 0 | 0% | | 6th | 27 Aug 2018 | 42.4% | 89% masked | 0 | 0% | | 7th | 27 Aug 2018 | 45.5% | 95% masked | 0 | 0% | **Tiles stay apart.** Both tiles cover a strip 245 cells wide. sits regularizes each tile on its own grid: here the cells of 20LKP lie 10 m off those of 20LLP, so the two cubes never hold the same cell. On the paired cells their B02 is equal on 4% and differs by a median of 8. Copied first? The book copies the images with sits_cube_copy() before it regularizes; the copies carry no cloud_cover, so sits takes them by date. Run here as a stand-in: the same images regularized with cloud_cover removed. By date, the period starts on 22 Aug 2018 in both orders, and 97% of the cells of B02 change, by a median of 9 on a median of 298. #### The images of 20LLP, 6 Jan to 21 Jan 2018 5 items of 20LLP fall in this period; 3 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing; the 3rd and 4th give 61% and < 1%, and 39% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 21 Jan 2018 | 7.2% | 0% | 0% | 100% | 0 (0%) | | 2nd | 6 Jan 2018 | 53.5% | 0% | 0% | 100% | 0 (0%) | | 3rd | 19 Jan 2018 | 72.9% | 61% | 39% | 0% | 31,041 (61%) | | 4th | 19 Jan 2018 | 81.2% | 38% | 62% | 0% | 181 (< 1%) | | 5th | 9 Jan 2018 | 100% | 0% | 100% | 0% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images 31,222; 19,978 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 22 Jan to 6 Feb 2018 3 items of 20LLP fall in this period; 1 has data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing, and 100% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------ | | 1st | 5 Feb 2018 | 43.1% | 0% | 0% | 100% | 0 (0%) | | 2nd | 26 Jan 2018 | 51.5% | 0% | 0% | 100% | 0 (0%) | | 3rd | 29 Jan 2018 | 91.4% | 0% | 100% | 0% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images none; 51,200 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 7 Feb to 22 Feb 2018 9 items of 20LLP fall in this period; 4 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives 43%; the 2nd and 7th give 1% and 2%, and 54% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 18 Feb 2018 | 41.8% | 43% | 57% | 0% | 21,797 (43%) | | 2nd | 18 Feb 2018 | 52.6% | 34% | 66% | 0% | 712 (1%) | | 3rd | 15 Feb 2018 | 77.7% | 0% | 0% | 100% | 0 (0%) | | 4th | 15 Feb 2018 | 77.7% | 0% | 0% | 100% | 0 (0%) | | 5th | 15 Feb 2018 | 81.8% | 0% | 0% | 100% | 0 (0%) | | 6th | 8 Feb 2018 | 94.7% | 0% | 100% | 0% | 0 (0%) | | 7th | 13 Feb 2018 | 96.7% | 6% | 94% | 0% | 1,112 (2%) | | 8th | 20 Feb 2018 | 98.5% | 0% | 0% | 100% | 0 (0%) | | 9th | 10 Feb 2018 | > 99.9% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 21,797 of 51,200 cells; the later images 1,824; 27,579 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 23 Feb to 10 Mar 2018 6 items of 20LLP fall in this period; 4 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives 96%; the 2nd gives 2%, and 1% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 5 Mar 2018 | 12.5% | 96% | 4% | 0% | 49,383 (96%) | | 2nd | 28 Feb 2018 | 20.4% | 61% | 39% | 0% | 1,122 (2%) | | 3rd | 23 Feb 2018 | 99.9% | 0% | 100% | 0% | 0 (0%) | | 4th | 2 Mar 2018 | 100% | 0% | 0% | 100% | 0 (0%) | | 5th | 7 Mar 2018 | 100% | 0% | 0% | 100% | 0 (0%) | | 6th | 10 Mar 2018 | 100% | 0% | 100% | 0% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 49,383 of 51,200 cells; the later images 1,122; 695 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 11 Mar to 26 Mar 2018 6 items of 20LLP fall in this period; 4 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives 83%; the 2nd and 3rd give < 1% and < 1%, and 17% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 15 Mar 2018 | 5.5% | 83% | 17% | 0% | 42,573 (83%) | | 2nd | 15 Mar 2018 | 9.6% | 59% | 41% | 0% | 47 (< 1%) | | 3rd | 20 Mar 2018 | 93.1% | < 1% | > 99% | 0% | 2 (< 1%) | | 4th | 12 Mar 2018 | 97.8% | 0% | 0% | 100% | 0 (0%) | | 5th | 25 Mar 2018 | > 99.9% | 0% | 100% | 0% | 0 (0%) | | 6th | 17 Mar 2018 | 100% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 42,573 of 51,200 cells; the later images 49; 8,578 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 27 Mar to 11 Apr 2018 7 items of 20LLP fall in this period; 3 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives 91%; the 2nd and 5th give < 1% and 4%, and 5% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 30 Mar 2018 | 32.2% | 91% | 9% | 0% | 46,709 (91%) | | 2nd | 30 Mar 2018 | 41.4% | 67% | 33% | 0% | 77 (< 1%) | | 3rd | 11 Apr 2018 | 73.1% | 0% | 0% | 100% | 0 (0%) | | 4th | 6 Apr 2018 | 73.4% | 0% | 0% | 100% | 0 (0%) | | 5th | 9 Apr 2018 | 89.9% | 24% | 76% | 0% | 1,945 (4%) | | 6th | 1 Apr 2018 | 99.3% | 0% | 0% | 100% | 0 (0%) | | 7th | 27 Mar 2018 | 99.7% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 46,709 of 51,200 cells; the later images 2,022; 2,469 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 12 Apr to 27 Apr 2018 4 items of 20LLP fall in this period; 2 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing; the 2nd and 3rd give 52% and 38%, and 10% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 26 Apr 2018 | 9.4% | 0% | 0% | 100% | 0 (0%) | | 2nd | 24 Apr 2018 | 46.9% | 52% | 48% | 0% | 26,450 (52%) | | 3rd | 19 Apr 2018 | 51.7% | 55% | 45% | 0% | 19,504 (38%) | | 4th | 21 Apr 2018 | 69.3% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images 45,954; 5,246 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 28 Apr to 13 May 2018 5 items of 20LLP fall in this period; 3 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives 79%; the 2nd and 5th give 12% and 1%, and 8% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 9 May 2018 | 12.5% | 79% | 21% | 0% | 40,670 (79%) | | 2nd | 4 May 2018 | 19.5% | 60% | 40% | 0% | 6,075 (12%) | | 3rd | 11 May 2018 | 28.3% | 0% | 0% | 100% | 0 (0%) | | 4th | 1 May 2018 | 74.5% | 0% | 0% | 100% | 0 (0%) | | 5th | 29 Apr 2018 | 93.1% | 20% | 80% | 0% | 615 (1%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 40,670 of 51,200 cells; the later images 6,690; 3,840 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 14 May to 29 May 2018 7 items of 20LLP fall in this period; 4 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing; the 3rd, 5th and 7th give 75%, 5% and 1%, and 19% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 26 May 2018 | 1.9% | 0% | 0% | 100% | 0 (0%) | | 2nd | 26 May 2018 | 2.0% | 0% | 0% | 100% | 0 (0%) | | 3rd | 24 May 2018 | 23.0% | 75% | 25% | 0% | 38,416 (75%) | | 4th | 21 May 2018 | 28.6% | 0% | 0% | 100% | 0 (0%) | | 5th | 29 May 2018 | 39.3% | 72% | 28% | 0% | 2,519 (5%) | | 6th | 29 May 2018 | 47.4% | 55% | 45% | 0% | 0 (0%) | | 7th | 14 May 2018 | 62.4% | 1% | 99% | 0% | 510 (1%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images 41,445; 9,755 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 30 May to 14 Jun 2018 9 items of 20LLP fall in this period; 5 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing; the 2nd, 4th, 7th, 8th and 9th give 38%, < 1%, 1%, < 1% and 1%, and 60% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 31 May 2018 | 10.2% | 0% | 0% | 100% | 0 (0%) | | 2nd | 8 Jun 2018 | 19.8% | 38% | 62% | 0% | 19,560 (38%) | | 3rd | 10 Jun 2018 | 24.9% | 0% | 0% | 100% | 0 (0%) | | 4th | 8 Jun 2018 | 38.8% | 13% | 87% | 0% | 8 (< 1%) | | 5th | 5 Jun 2018 | 53.9% | 0% | 0% | 100% | 0 (0%) | | 6th | 5 Jun 2018 | 57.1% | 0% | 0% | 100% | 0 (0%) | | 7th | 13 Jun 2018 | 84.7% | 1% | 99% | 0% | 485 (1%) | | 8th | 13 Jun 2018 | 88.6% | 1% | 99% | 0% | 7 (< 1%) | | 9th | 3 Jun 2018 | 94.0% | 1% | 99% | 0% | 305 (1%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images 20,365; 30,835 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 15 Jun to 30 Jun 2018 13 items of 20LLP fall in this period; 5 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing; the 2nd gives 100%. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------- | | 1st | 25 Jun 2018 | < 0.1% | 0% | 0% | 100% | 0 (0%) | | 2nd | 18 Jun 2018 | < 0.1% | 100% | 0% | 0% | 51,200 (100%) | | 3rd | 25 Jun 2018 | < 0.1% | 0% | 0% | 100% | 0 (0%) | | 4th | 18 Jun 2018 | < 0.1% | > 99% | < 1% | 0% | 0 (0%) | | 5th | 30 Jun 2018 | < 0.1% | 0% | 0% | 100% | 0 (0%) | | 6th | 30 Jun 2018 | < 0.1% | 0% | 0% | 100% | 0 (0%) | | 7th | 28 Jun 2018 | < 0.1% | > 99% | < 1% | 0% | 0 (0%) | | 8th | 20 Jun 2018 | 1.0% | 0% | 0% | 100% | 0 (0%) | | 9th | 20 Jun 2018 | 1.1% | 0% | 0% | 100% | 0 (0%) | | 10th | 23 Jun 2018 | 16.1% | 97% | 3% | 0% | 0 (0%) | | 11th | 23 Jun 2018 | 19.9% | 95% | 5% | 0% | 0 (0%) | | 12th | 15 Jun 2018 | 72.4% | 0% | 0% | 100% | 0 (0%) | | 13th | 15 Jun 2018 | 80.8% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images 51,200. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 1 Jul to 16 Jul 2018 9 items of 20LLP fall in this period; 4 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives > 99%; the 2nd gives < 1%. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | -------------- | | 1st | 8 Jul 2018 | < 0.1% | > 99% | < 1% | 0% | 51,194 (> 99%) | | 2nd | 3 Jul 2018 | < 0.1% | 100% | 0% | 0% | 6 (< 1%) | | 3rd | 3 Jul 2018 | < 0.1% | > 99% | < 1% | 0% | 0 (0%) | | 4th | 10 Jul 2018 | < 0.1% | 0% | 0% | 100% | 0 (0%) | | 5th | 13 Jul 2018 | < 0.1% | > 99% | < 1% | 0% | 0 (0%) | | 6th | 5 Jul 2018 | 1.4% | 0% | 0% | 100% | 0 (0%) | | 7th | 5 Jul 2018 | 1.9% | 0% | 0% | 100% | 0 (0%) | | 8th | 15 Jul 2018 | 2.1% | 0% | 0% | 100% | 0 (0%) | | 9th | 15 Jul 2018 | 3.4% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 51,194 of 51,200 cells; the later images 6. **Where each cell came from.** All but 6 cells came from the 1st image (> 99%). The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 17 Jul to 1 Aug 2018 9 items of 20LLP fall in this period; 6 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives > 99%; the 2nd gives < 1%. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | -------------- | | 1st | 28 Jul 2018 | < 0.1% | > 99% | < 1% | 0% | 51,199 (> 99%) | | 2nd | 28 Jul 2018 | < 0.1% | > 99% | < 1% | 0% | 1 (< 1%) | | 3rd | 23 Jul 2018 | < 0.1% | 100% | 0% | 0% | 0 (0%) | | 4th | 18 Jul 2018 | < 0.1% | 100% | 0% | 0% | 0 (0%) | | 5th | 23 Jul 2018 | < 0.1% | > 99% | < 1% | 0% | 0 (0%) | | 6th | 18 Jul 2018 | < 0.1% | > 99% | < 1% | 0% | 0 (0%) | | 7th | 20 Jul 2018 | < 0.1% | 0% | 0% | 100% | 0 (0%) | | 8th | 25 Jul 2018 | 0.7% | 0% | 0% | 100% | 0 (0%) | | 9th | 30 Jul 2018 | 79.9% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 51,199 of 51,200 cells; the later images 1. **Where each cell came from.** All but 1 cell came from the 1st image (> 99%). The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 2 Aug to 17 Aug 2018 16 items of 20LLP fall in this period; 8 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives > 99%; the 2nd gives < 1%. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | -------------- | | 1st | 12 Aug 2018 | < 0.1% | > 99% | < 1% | 0% | 51,199 (> 99%) | | 2nd | 12 Aug 2018 | < 0.1% | > 99% | < 1% | 0% | 1 (< 1%) | | 3rd | 4 Aug 2018 | 1.0% | 0% | 0% | 100% | 0 (0%) | | 4th | 4 Aug 2018 | 1.0% | 0% | 0% | 100% | 0 (0%) | | 5th | 17 Aug 2018 | 1.2% | 0% | 0% | 100% | 0 (0%) | | 6th | 17 Aug 2018 | 1.4% | 0% | 0% | 100% | 0 (0%) | | 7th | 14 Aug 2018 | 1.8% | 0% | 0% | 100% | 0 (0%) | | 8th | 14 Aug 2018 | 2.0% | 0% | 0% | 100% | 0 (0%) | | 9th | 9 Aug 2018 | 6.0% | 0% | 0% | 100% | 0 (0%) | | 10th | 9 Aug 2018 | 8.8% | 0% | 0% | 100% | 0 (0%) | | 11th | 7 Aug 2018 | 17.2% | 92% | 8% | 0% | 0 (0%) | | 12th | 7 Aug 2018 | 26.9% | 83% | 17% | 0% | 0 (0%) | | 13th | 17 Aug 2018 | 30.3% | > 99% | < 1% | 0% | 0 (0%) | | 14th | 17 Aug 2018 | 40.3% | > 99% | < 1% | 0% | 0 (0%) | | 15th | 2 Aug 2018 | 40.7% | > 99% | < 1% | 0% | 0 (0%) | | 16th | 2 Aug 2018 | 47.3% | > 99% | < 1% | 0% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 51,199 of 51,200 cells; the later images 1. **Where each cell came from.** All but 1 cell came from the 1st image (> 99%). The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 18 Aug to 2 Sep 2018 11 items of 20LLP fall in this period; 6 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives > 99%; the 2nd gives < 1%. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | -------------- | | 1st | 22 Aug 2018 | < 0.1% | > 99% | < 1% | 0% | 51,199 (> 99%) | | 2nd | 22 Aug 2018 | < 0.1% | > 99% | < 1% | 0% | 1 (< 1%) | | 3rd | 29 Aug 2018 | 0.1% | 0% | 0% | 100% | 0 (0%) | | 4th | 24 Aug 2018 | 0.1% | 0% | 0% | 100% | 0 (0%) | | 5th | 19 Aug 2018 | 0.2% | 0% | 0% | 100% | 0 (0%) | | 6th | 24 Aug 2018 | 0.2% | 0% | 0% | 100% | 0 (0%) | | 7th | 19 Aug 2018 | 0.3% | 0% | 0% | 100% | 0 (0%) | | 8th | 1 Sep 2018 | 1.0% | 99% | 1% | 0% | 0 (0%) | | 9th | 1 Sep 2018 | 1.8% | 98% | 2% | 0% | 0 (0%) | | 10th | 27 Aug 2018 | 42.4% | 11% | 89% | 0% | 0 (0%) | | 11th | 27 Aug 2018 | 45.5% | 5% | 95% | 0% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 51,199 of 51,200 cells; the later images 1. **Where each cell came from.** All but 1 cell came from the 1st image (> 99%). The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 3 Sep to 18 Sep 2018 13 items of 20LLP fall in this period; 5 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing; the 3rd, 4th and 5th give > 99%, < 1% and < 1%. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | -------------- | | 1st | 8 Sep 2018 | < 0.1% | 0% | 0% | 100% | 0 (0%) | | 2nd | 8 Sep 2018 | < 0.1% | 0% | 0% | 100% | 0 (0%) | | 3rd | 6 Sep 2018 | 0.1% | > 99% | < 1% | 0% | 51,188 (> 99%) | | 4th | 11 Sep 2018 | 0.2% | > 99% | < 1% | 0% | 10 (< 1%) | | 5th | 11 Sep 2018 | 0.3% | > 99% | < 1% | 0% | 2 (< 1%) | | 6th | 3 Sep 2018 | 10.9% | 0% | 0% | 100% | 0 (0%) | | 7th | 3 Sep 2018 | 20.1% | 0% | 0% | 100% | 0 (0%) | | 8th | 18 Sep 2018 | 47.8% | 0% | 0% | 100% | 0 (0%) | | 9th | 18 Sep 2018 | 57.6% | 0% | 0% | 100% | 0 (0%) | | 10th | 13 Sep 2018 | > 99.9% | 0% | 0% | 100% | 0 (0%) | | 11th | 13 Sep 2018 | > 99.9% | 0% | 0% | 100% | 0 (0%) | | 12th | 16 Sep 2018 | 100% | 0% | 100% | 0% | 0 (0%) | | 13th | 16 Sep 2018 | 100% | 0% | 100% | 0% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images 51,200. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 19 Sep to 4 Oct 2018 10 items of 20LLP fall in this period; 6 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives > 99%; the 2nd and 3rd give < 1% and < 1%. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | -------------- | | 1st | 1 Oct 2018 | 0.7% | > 99% | < 1% | 0% | 51,194 (> 99%) | | 2nd | 26 Sep 2018 | 10.0% | > 99% | < 1% | 0% | 5 (< 1%) | | 3rd | 1 Oct 2018 | 11.2% | > 99% | < 1% | 0% | 1 (< 1%) | | 4th | 21 Sep 2018 | 11.5% | 98% | 2% | 0% | 0 (0%) | | 5th | 26 Sep 2018 | 11.8% | 100% | 0% | 0% | 0 (0%) | | 6th | 21 Sep 2018 | 24.8% | 95% | 5% | 0% | 0 (0%) | | 7th | 23 Sep 2018 | 45.0% | 0% | 0% | 100% | 0 (0%) | | 8th | 23 Sep 2018 | 51.7% | 0% | 0% | 100% | 0 (0%) | | 9th | 28 Sep 2018 | 80.2% | 0% | 0% | 100% | 0 (0%) | | 10th | 3 Oct 2018 | 99.2% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 51,194 of 51,200 cells; the later images 6. **Where each cell came from.** All but 6 cells came from the 1st image (> 99%). The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 5 Oct to 20 Oct 2018 9 items of 20LLP fall in this period; 3 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing; the 2nd gives 44%, and 56% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 8 Oct 2018 | 0.3% | 0% | 0% | 100% | 0 (0%) | | 2nd | 16 Oct 2018 | 31.2% | 44% | 56% | 0% | 22,506 (44%) | | 3rd | 13 Oct 2018 | 51.0% | 0% | 0% | 100% | 0 (0%) | | 4th | 13 Oct 2018 | 63.2% | 0% | 0% | 100% | 0 (0%) | | 5th | 6 Oct 2018 | 85.0% | 0% | 100% | 0% | 0 (0%) | | 6th | 6 Oct 2018 | 96.1% | 0% | 0% | 100% | 0 (0%) | | 7th | 6 Oct 2018 | 97.1% | 0% | 0% | 100% | 0 (0%) | | 8th | 18 Oct 2018 | > 99.9% | 0% | 0% | 100% | 0 (0%) | | 9th | 11 Oct 2018 | 100% | 0% | 100% | 0% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images 22,506; 28,694 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 21 Oct to 5 Nov 2018 12 items of 20LLP fall in this period; 5 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives 59%; the 2nd, 5th and 9th give < 1%, 1% and < 1%, and 40% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 26 Oct 2018 | 24.0% | 59% | 41% | 0% | 30,299 (59%) | | 2nd | 26 Oct 2018 | 38.2% | 32% | 68% | 0% | 141 (< 1%) | | 3rd | 28 Oct 2018 | 42.7% | 0% | 0% | 100% | 0 (0%) | | 4th | 28 Oct 2018 | 51.4% | 0% | 0% | 100% | 0 (0%) | | 5th | 21 Oct 2018 | 77.9% | 3% | 97% | 0% | 380 (1%) | | 6th | 31 Oct 2018 | 89.9% | 0% | 100% | 0% | 0 (0%) | | 7th | 2 Nov 2018 | 90.5% | 0% | 0% | 100% | 0 (0%) | | 8th | 2 Nov 2018 | 90.8% | 0% | 0% | 100% | 0 (0%) | | 9th | 5 Nov 2018 | 91.5% | < 1% | > 99% | 0% | 2 (< 1%) | | 10th | 23 Oct 2018 | 98.3% | 0% | 0% | 100% | 0 (0%) | | 11th | 23 Oct 2018 | 99.8% | 0% | 0% | 100% | 0 (0%) | | 12th | 23 Oct 2018 | 99.9% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 30,299 of 51,200 cells; the later images 523; 20,378 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 6 Nov to 21 Nov 2018 1 item of 20LLP fall in this period; 1 has data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives 82%, and 18% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 10 Nov 2018 | 23.5% | 82% | 18% | 0% | 42,065 (82%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 42,065 of 51,200 cells; the later images none; 9,135 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 22 Nov to 7 Dec 2018 3 items of 20LLP fall in this period; 1 has data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing; the 2nd gives 63%, and 37% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 22 Nov 2018 | 48.6% | 0% | 0% | 100% | 0 (0%) | | 2nd | 5 Dec 2018 | 49.2% | 63% | 37% | 0% | 32,318 (63%) | | 3rd | 27 Nov 2018 | 51.0% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images 32,318; 18,882 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 8 Dec to 23 Dec 2018 6 items of 20LLP fall in this period; 3 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image gives 2%; the 3rd and 5th give 28% and 49%, and 21% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 10 Dec 2018 | 27.2% | 2% | 98% | 0% | 1,088 (2%) | | 2nd | 17 Dec 2018 | 40.9% | 0% | 0% | 100% | 0 (0%) | | 3rd | 20 Dec 2018 | 57.2% | 29% | 71% | 0% | 14,101 (28%) | | 4th | 12 Dec 2018 | 63.0% | 0% | 0% | 100% | 0 (0%) | | 5th | 15 Dec 2018 | 69.4% | 71% | 29% | 0% | 25,166 (49%) | | 6th | 22 Dec 2018 | 77.3% | 0% | 0% | 100% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 1,088 of 51,200 cells; the later images 39,267; 10,845 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### The images of 20LLP, 24 Dec 2018 to 8 Jan 2019 3 items of 20LLP fall in this period; 2 have data over the region. sits takes them in the order below and keeps, for each cell, the first value that is not masked. The first image has no data here and gives nothing; the 2nd gives 24%, and 76% of the cells stay empty. | Place | Date | Scene cloud cover | Value | Masked | No data | Gave | | ----- | ----------- | ----------------- | ----- | ------ | ------- | ------------ | | 1st | 27 Dec 2018 | 76.2% | 0% | 0% | 100% | 0 (0%) | | 2nd | 25 Dec 2018 | 77.5% | 24% | 76% | 0% | 12,074 (24%) | | 3rd | 30 Dec 2018 | > 99.9% | 0% | 100% | 0% | 0 (0%) | **Ranked.** In sits' order: by scene cloud cover, least first. The covers are shown rounded; equal covers keep the collection's order. **On the grid, masked.** Each image on the 40 m grid of the cube. Hatched: the cells where the CLOUD band holds a value sits masks, 0, 1, 2, 3, 8, 9 and 10 (missing data, defective pixel, shadows, cloud shadows, cloud medium, cloud high and thin cirrus); they give nothing. sits keeps 4, 5, 6, 7 and 11 (vegetation, non-vegetated, water, unclassified and snow or ice). Dotted: no data, the image has no value there. **First fills.** An empty grid. Image by image, in sits' order, the cells it fills light up; a cell already filled is not touched again. The first image fills 0 of 51,200 cells; the later images 12,074; 39,126 stay empty. The year 2018 of 20LLP: an extension of the book's two months, regularized by the pipeline with the book's call (16-day periods, B02 for the source of each cell). #### How it was made 1. Ask the catalogue: sits asks the provider's catalogue which images cover the book's tiles, bands and dates, and a whole year of one tile. The answers are kept, so every later run uses the same list. 2. Find the tiles: The pipeline finds the tiles of the satellite's grid around the region, from the grid that comes with sits. It reads a small picture of each tile of the cube from its least cloudy image. 3. Build the regular cube: sits makes a regular cube of the region from those images: one image per period, all on the same grid of pixels. For each pixel of a period it takes the first valid value, least cloudy image first. 4. Read the images as they come: The pipeline reads the images the catalogue listed, as they come. It marks each pixel as having no data, a clear value, or a value sits masks, such as cloud or shadow. 5. Try other periods and pixel sizes: sits makes the regular cube again at other periods and pixel sizes. It also makes it once without the images' cloud cover, so sits takes the images in date order, not least cloudy first. 6. Read each image alone: Each image is read alone, on the grid sits builds for it. Stacked in the order sits takes them, the images show which one gave each pixel of the regular cube. 7. Check against sits: The pipeline compares its own reads with what sits made, and stops the run on the checks that must hold. The results are gathered only once those checks pass. 8. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each. 9. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - A rerun on aarch64-apple-darwin23 gave the same files, byte for byte. Every step of the pipeline: 1. Lists the bands read: the book's and those of the natural-colour pictures, in the order sits has them. `cube_read_bands("AWS", "SENTINEL-2-L2A", c("B02", "B8A", "B11"), c("B03", "B04"))` 2. Lists every period with every resolution of the runs. `regularize_choices(c("P8D", "P16D", "P1M"), c(20L, 40L, 80L))` 3. The arguments of the book's call to sits_cube(). `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands)` 4. The same call for one tile over a whole year. `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands, "20LLP", "2018-01-01", "2018-12-31")` 5. Lists the bands of the runs: the natural-colour ones, in the order sits has them. `regularize_run_bands(cube_bands, list(red = "B04", green = "B03", blue = "B02"))` 6. Lists the images of the book's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_book_call, "cache/cubes", "cube_book")` 7. Lists the images of the year's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_year_call, "cache/cubes", "cube_year")` 8. Reads how sits masks the cloud band, and the line of its source file that says so. `collection_cloud_source("cache/src/sits-78d26eb07370874192911854b48b60ebb1d668c8", "AWS", "SENTINEL-2-L2A", "CLOUD", regularize_bands)` 9. Makes the regular cube of the book's call over the region with sits_regularize(), keeping the files already made. `cube_regularize(cube_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_book_B02-B03-B04-B8A-B11", 4L)` 10. Finds the Sentinel-2 tiles around the region. `cube_images_grid(list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), cube_listing, 100L)` 11. Reads a small picture of each tile from the overview of its clearest image, keeping the reads in cache/cubes/. `cube_images_read(cube_listing, list(red = "B04", green = "B03", blue = "B02"), 3L, "cache/cubes")` 12. Makes the regular cube of the year with sits_regularize(), keeping the files already made. `cube_regularize(cube_year_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_year_B02-B03-B04-B8A-B11", 4L)` 13. Reads one tile whole on the date of the book's figure, from an overview, keeping the read in cache/regularize/. `regularize_images_coverage(cube_listing, list(tile = "20LLP", date = "2018-06-30"), list(red = "B04", green = "B03", blue = "B02"), 3L, "cache/regularize")` 14. Runs sits_regularize() on the book's images at one period and resolution, keeping the files in cache/regularize/. `regularize_run_cube(cube_listing, regularize_choice, regularize_bands, "CLOUD", list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "cache/regularize", 4L, 1L)` 15. Reads every image of each tile over the tile's part of the region, as it comes, keeping the reads in cache/cubes/. `cube_tile_images(cube_listing, cube_book, cube_book_call, 40L, "EPSG:32720", "cache/cubes")` 16. Compares the grid of each tile of the book's run with the view sits builds for it. `regularize_book_grids(regularize_find_run(regularize_run, "P16D", 40L), cube_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L))` 17. Reads the regular images each run wrote. `regularize_values(regularize_run, regularize_bands)` 18. Gathers the images, ranked in the order sits takes them, the regular values, the year and the map. `cube_export_gather(cube_listing, cube_book, cube_images, cube_year_listing, cube_year, cube_map_images, cube_map_grid, cube_book_call, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), 40L)` 19. Marks each pixel of each image as no data, clear or masked by sits. `purrr::map(cubes_record$tiles, cube_images_states, cubes_record$cloud_values)` 20. Picks the reflectance drawn black and white, one for red, green and blue. `cube_images_stretch(cubes_record, list(red = "B04", green = "B03", blue = "B02"), c(0.02, 0.98))` 21. Stores the year in one byte per value, each band between two percentiles. `cube_export_year_codes(cubes_record$year$values, c(0.001, 0.999))` 22. Reads each image of the book's periods alone, on the view sits builds for it with no mask, keeping the reads in cache/regular_cubes/. `cube_alone_book_reads(cube_listing, cubes_record, "B02", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "cache/regular_cubes/alone_book_B02", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp", 0.5)` 23. Runs sits_regularize() on the book's call with the scene cloud cover removed, keeping the result in cache/regular_cubes/. `cube_checks_copied_run(cube_listing, "B02", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP" ), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list( crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), cubes_record, "cache/regular_cubes", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp")` 24. Reads the grid of the region on each tile at each resolution from the runs of the book's period. `cube_checks_region_grids(cube_listing, regularize_run, cubes_record, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), c(20L, 40L, 80L))` 25. Runs sits_regularize() on one image alone and gdalwarp at three overview choices, keeping the result in cache/regular_cubes/. `cube_checks_overview_check_run(cube_listing, cubes_record, list(tile = "20LLP", band = "B02"), list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP" ), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98 ), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), "cache/regular_cubes", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp")` 26. Reads the same image alone in natural colour and runs sits_regularize() on it with those bands. `cube_checks_pictures_check_run(cube_listing, cubes_record, list(tile = "20LLP", band = "B02"), regularize_bands, list( source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98 ), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), "cache/regular_cubes", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp")` 27. Pairs the cells of the strip both tiles cover. `cube_checks_book_strip(cubes_record, c("20LKP", "20LLP" ), 40L)` 28. Runs sits_regularize() on each image of the year alone, with the date of its period, keeping the files in cache/regularize/. `regularize_year_alone(cube_year_listing, cubes_record, "B02", "CLOUD", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "cache/regularize/year_20llp_B02", 1L)` 29. Stops the run unless the book's run gives the regular cube of the book, value for value. `regularize_check_book(regularize_run_values, cubes_record, "P16D", 40L, regularize_bands)` 30. Reads a small square at the native resolution of two bands, keeping the read in cache/regularize/. `regularize_images_patch(cube_listing, cubes_record, list(tile = "20LLP", side = 480L, bands = c("B04", "B11")), "CLOUD", "cache/regularize")` 31. Stacks the images of each period in sits' order and finds the one each pixel came from. `cube_alone_book_stacks(regular_cubes_book_reads, cubes_record, "B02")` 32. Compares the run with no scene cloud cover with the book's regular cube, after checking it against the images read alone. `cube_checks_copied_entries(regular_cubes_copied_run, cubes_record, regular_cubes_book_reads, cube_listing, "B02")` 33. Compares the two tiles' regular images where they overlap. `cube_checks_overlap_periods(cubes_record, regular_cubes_strip, "B02")` 34. Stops the run unless gdalwarp -ovr AUTO and the images read alone give sits' values on every pixel. `cube_checks_check_overview(regular_cubes_overview, regular_cubes_book_reads, regular_cubes_pictures)` 35. Draws one tile's images as they come and its regular images, small, for the journey's picture. `cube_checks_export_picture(cubes_record, cube_states, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L, list(tile = "20LLP", card_tile = 64L, sheet_tile = 192L))` 36. Finds which image gives each pixel of each period of the year, in the order sits takes them. `regularize_year_sources(regularize_alone, "20LLP", "B02")` 37. Draws every image of the tile as it comes, in natural colour, in one sheet, for the website. `site_write_png(cube_images_ard_sheet(cubes_record$tiles[["20LKP"]], cube_states[["20LKP"]], cubes_record$bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/cube-ard-20lkp.png")` 38. Draws every image of the tile as it comes, in natural colour, in one sheet, for the website. `site_write_png(cube_images_ard_sheet(cubes_record$tiles[["20LLP"]], cube_states[["20LLP"]], cubes_record$bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/cube-ard-20llp.png")` 39. Draws where each image of the tile has data and where sits masks it, in one sheet, for the website. `site_write_png(cube_images_mask_sheet(cube_states[["20LKP"]])$sheet[, , 1L], "../web/public/data/cube-ard-20lkp-mask.png")` 40. Draws where each image of the tile has data and where sits masks it, in one sheet, for the website. `site_write_png(cube_images_mask_sheet(cube_states[["20LLP"]])$sheet[, , 1L], "../web/public/data/cube-ard-20llp-mask.png")` 41. Draws the regular images sits made of the tile, one per date, for the website. `site_write_png(cube_images_regular_sheet(cubes_record$regular[["20LKP"]], cubes_record$bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/cube-regular-20lkp.png")` 42. Draws the regular images sits made of the tile, one per date, for the website. `site_write_png(cube_images_regular_sheet(cubes_record$regular[["20LLP"]], cubes_record$bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L)$sheet, "../web/public/data/cube-regular-20llp.png")` 43. Writes the book's call, its images and regular cube, the map and the year, for the website. ``site_write_json(cube_export_record(cubes_record, cube_states, cube_stretch, cube_year_codes, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), list(tiles = list(`20LKP` = list(ard = "cube-ard-20lkp.png", mask = "cube-ard-20lkp-mask.png", regular = "cube-regular-20lkp.png", map = "cube-map-20lkp.png"), `20LLP` = list(ard = "cube-ard-20llp.png", mask = "cube-ard-20llp-mask.png", regular = "cube-regular-20llp.png", map = "cube-map-20llp.png")), year = "cube-year.bin")), "../web/src/data/cubes.json")`` 44. Stops the run unless the images read alone, stacked in sits' order, give the book's regular cube on every pixel. `cube_alone_check_book(regular_cubes_book, cubes_record, "B02", "cache/regular_cubes/alone_book_B02")` 45. Reads each image of the year alone in natural colour, with its cloud band, keeping the reads in cache/regular_cubes/. `cube_alone_year_reads(cube_year_listing, cubes_record, regularize_sources, regularize_bands, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP" ), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98 ), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), "cache/regular_cubes/alone_year_20llp", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp", 0.5)` 46. Stops the run unless the images, stacked in sits' order, give the regular cube of the year. `regularize_check_year(regularize_sources, cubes_record)` 47. Stops the run unless the cloud band read here masks each image of the year as sits did. `cube_alone_check_year_cloud(regular_cubes_year_reads, regularize_alone, cubes_record$cloud_values, "cache/regular_cubes/alone_year_20llp")` 48. Gathers the runs, the images as they come and the sources of the year, once both self-checks pass. `regularize_export_gather(cube_listing, regularize_run_values, regularize_coverage, regularize_patch, regularize_sources, regularize_bands, list(regularize_book_check, regularize_year_check))` 49. Gathers the grids, the sources of the book's pixels, the overview, the run with no cloud cover and the year, once the self-checks pass. `cube_checks_export_gather(regular_cubes_grids, regular_cubes_book, regular_cubes_strip, regular_cubes_overlap, regular_cubes_overview, regular_cubes_copied, regular_cubes_year_reads, list(regular_cubes_book_check, regular_cubes_overview_check, regular_cubes_cloud_check))` 50. Writes the runs, the images as they come, the sources of the year and how sits masks clouds, for the website. ``site_write_json(regularize_export_record(regularize_record, cubes_record, regularize_cloud, regularize_grids, list(coverage = "regularize-coverage.png", patch = list(B04 = "regularize-patch-b04.png", B11 = "regularize-patch-b11.png"), runs = list(run_p16d_40m = list(`20LKP` = "cube-regular-20lkp.png", `20LLP` = "cube-regular-20llp.png"), run_p16d_20m = list(`20LKP` = "regularize-run-p16d-20m-20lkp.png", `20LLP` = "regularize-run-p16d-20m-20llp.png"), run_p16d_80m = list(`20LKP` = "regularize-run-p16d-80m-20lkp.png", `20LLP` = "regularize-run-p16d-80m-20llp.png")), sheets = list(list(period = "P16D", res = 20L), list(period = "P16D", res = 80L)), sources = "regularize-sources.bin")), "../web/src/data/regularize.json")`` 51. Writes which image gives each pixel of each period of the year, one byte per pixel, for the website. `site_write_bin(regularize_source_bytes(regularize_record$sources$source), "../web/public/data/regularize-sources.bin", "uint8")` 52. Writes which image gave each pixel of the book's regular cube, one byte per pixel, for the website. `site_write_bin(cube_checks_export_rank_bytes(cube_checks_export_ranks(purrr::map(unname(regular_cubes_record$book), "ranks"))$ranks), "../web/public/data/regular-cubes-book-ranks.bin", "uint8")` 53. Writes the grids, the sources of the book's pixels, the overview, the run with no cloud cover and the year, for the website. `site_write_json(cube_checks_export_record(regular_cubes_record, cubes_record, regularize_sources, regular_cubes_picture, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "B02", list(ranks = "regular-cubes-book-ranks.bin", cards = "regular-cubes-picture-cards-20llp.png", sheets = "regular-cubes-picture-sheets-20llp.png")), "../web/src/data/regular-cubes.json")` 54. Draws each image of each period of the year read alone, in natural colour, one file per period, for the website. `cube_checks_export_year_sheets(regular_cubes_record, "image", regularize_bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L, cubes_record$cloud_values, structure("../web/public/data", class = c("fs_path", "character")))` 55. Draws where each image of each period of the year has a value, is masked or has none, one file per period, for the website. `cube_checks_export_year_sheets(regular_cubes_record, "state", regularize_bands, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L, cubes_record$cloud_values, structure("../web/public/data", class = c("fs_path", "character")))` The files: - `cubes.json`: SHA-256 `f29bb2e9135a6f824c0d512868ea1c3d67bd2532c285b6662f87926e071d5874`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regular-cubes.json`: SHA-256 `751974674c0e71d20c1054ce3e79129531244a10501768116130594eea997323`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize.json`: SHA-256 `880aaea58eae1cb95f1780d87cdb6fec7a34e99ab7eaaf1fe5aa8f5c5fe0ce6d`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-ard-20llp.png`: SHA-256 `2c4d12fb2b588be680847130668ab1089acb717cad265cc4ab1005d854b17da5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-ard-20lkp.png`: SHA-256 `54d23212cc41e26d91b44a38ca8b2031aca7243db4e6e1a1658138bd06119fef`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-regular-20llp.png`: SHA-256 `2bedbff2a14c1f5a7b02f42a882e2b54f10f8fc9c97338076977bde0eb839820`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-regular-20lkp.png`: SHA-256 `04f081ce4387b725b64ceda711778ece73aac4016074f4caf05a57e80d3d5480`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-ard-20llp-mask.png`: SHA-256 `36d74706605e270514f3dde36e436638ae7d18c03cacbec0c12dd0a721aa128c`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-ard-20lkp-mask.png`: SHA-256 `38c90658f1803e743308bfe78f8624bdbc10bb4341cfedc569bc2416ba4ff243`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regular-cubes-book-ranks.bin`: SHA-256 `58990db759d676317fdcb5a3dd444da5048e6f92a94d68631f8911fd20aa38a5`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regularize-sources.bin`: SHA-256 `8bbbf5b19899603d13872fe3cbde3b72b7640c952d8c1c8319341b6d7be62f95`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regular-cubes-year-2018-04-12.png`: SHA-256 `d2ed994c855695922f1d8915cba9a5be16e78b4fce5617f14bf03f350ffe4001`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regular-cubes-year-2018-04-12-state.png`: SHA-256 `d2906b649e160aa3c09a96b7517886654ffbc2f600435c7b1903aca77e0f4bd3`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. To make them again: `cd pipeline && Rscript make.R`. Try it: [Where each cell of a regular image came from](/interactive-images/regular-cubes-provenance.md), or the tool [Regularization settings](/interactive-tools/regularization-settings.md) ## 5. What does a regular cube guarantee? In the sits book: [Regular Earth observation data cubes](https://e-sensing.github.io/sitsbook/datacubes.html#regular-earth-observation-data-cubes). One image per period, every period the same grid. In the recorded run the figure draws, sits_regularize() made the 176 images of the tile into 23, one every 16 days, each 256 × 200 cells of 40 m. This is the whole of 2018 on tile 20LLP: an extension of the book's example, whose call covers 2 months (30 Jun 2018 to 31 Aug 2018). A cell is the same ground on every date: its values are a time series. The outlined cell goes through all 23 sheets; the chart draws its B02 reflectance, date by date. And in every band, all at 40 m: B02, B03, B04, B8A and B11. B03 and B04 were read in addition to the book's bands, to draw the pictures. The sits book lists 5 properties of a regular cube. Beside each, what this cube has. Regular is not complete. sits_regularize() mosaics the images of each period; where none had a clear view of a cell, the cell stays empty. On 16 of the 23 dates part of the region has no value; on 22 Jan 2018 none of it has. The bars give each date's share. Read the cube by date or by place. Choose a date on the strip, a cell on the open sheet, or one band. ### The figure The year of one tile made regular by sits, as a stack of one image per period; each sheet shows a small window of cells around the chosen cell, and the chart its series in every band. The figure (recorded: 176 images made regular by sits, 23 dates) draws the cube as a stack of one sheet per period, time going up, each sheet a window of 12 × 12 of the 256 × 200 cells around the chosen cell; the open sheet carries the picture in natural colour (B04, B03, B02). The chosen cell is outlined on every sheet, dashed where it has no value. Beside it, the properties of a regular cube; under it, the series of the cell in every band, open where there is no value, and a strip with one bar per date for the share of the region with a value. The reader chooses a date on the strip, a cell on the open sheet, and all the bands or one. | Property of a regular cube (in the sits book's words) | This cube | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------- | | A regular data cube is a set of three-dimensional arrays with two spatial dimensions and one temporal dimension. Each array contains cells which have the same spatial resolution, the same temporal duration, and the same set of attributes (spectral bands and/or indices). | 256 × 200 cells of 40 m × 23 intervals of 16 days | | The number of dimensions is fixed (2D + time), while the number of attributes is not limited. | 5 bands: B02, B03, B04, B8A, B11 | | The spatial dimensions are associated with a coordinate projection system, such as the MGRS grid (Military Grid Reference System). A tile of the grid corresponds to a unique zone of the coordinate system. A data cube may span various tiles and projection zones. | EPSG:32720 (UTM zone 20S), tile 20LLP | | The temporal dimension is a set of continuous and equally spaced intervals. | 23 intervals of 16 days from 6 Jan 2018 | | For each position in space, the cube should provide a multi-attribute time series. For each time interval, the cube should provide a multi-attribute 2D image. | 51,200 series of 23 dates; 23 images | Of the 23 dates, 7 are complete and 16 have cells with no value. The longest run of complete dates is 7, from 15 Jun 2018 to 19 Sep 2018. | Date | Share of the region with no value | | ----------- | --------------------------------- | | 6 Jan 2018 | 39% | | 22 Jan 2018 | 100% | | 7 Feb 2018 | 54% | | 23 Feb 2018 | 1% | | 11 Mar 2018 | 17% | | 27 Mar 2018 | 5% | | 12 Apr 2018 | 10% | | 28 Apr 2018 | 8% | | 14 May 2018 | 19% | | 30 May 2018 | 60% | | 15 Jun 2018 | none | | 1 Jul 2018 | none | | 17 Jul 2018 | none | | 2 Aug 2018 | none | | 18 Aug 2018 | none | | 3 Sep 2018 | none | | 19 Sep 2018 | none | | 5 Oct 2018 | 56% | | 21 Oct 2018 | 40% | | 6 Nov 2018 | 18% | | 22 Nov 2018 | 37% | | 8 Dec 2018 | 21% | | 24 Dec 2018 | 76% | The figure opens on cell col 126 · row 136 and on 15 Jun 2018. Its series, in surface reflectance: | Date | B02 | B03 | B04 | B8A | B11 | | ----------- | -------- | -------- | -------- | -------- | -------- | | 6 Jan 2018 | no value | no value | no value | no value | no value | | 22 Jan 2018 | no value | no value | no value | no value | no value | | 7 Feb 2018 | no value | no value | no value | no value | no value | | 23 Feb 2018 | 0.02 | 0.04 | 0.02 | 0.29 | 0.14 | | 11 Mar 2018 | 0.03 | 0.04 | 0.02 | 0.35 | 0.17 | | 27 Mar 2018 | 0.03 | 0.05 | 0.03 | 0.35 | 0.17 | | 12 Apr 2018 | 0.04 | 0.06 | 0.04 | 0.33 | 0.16 | | 28 Apr 2018 | 0.05 | 0.07 | 0.04 | 0.36 | 0.17 | | 14 May 2018 | 0.02 | 0.03 | 0.02 | 0.28 | 0.13 | | 30 May 2018 | 0.21 | 0.23 | 0.21 | 0.49 | 0.35 | | 15 Jun 2018 | 0.02 | 0.03 | 0.02 | 0.28 | 0.13 | | 1 Jul 2018 | 0.02 | 0.03 | 0.02 | 0.26 | 0.13 | | 17 Jul 2018 | 0.02 | 0.04 | 0.02 | 0.27 | 0.14 | | 2 Aug 2018 | 0.03 | 0.04 | 0.03 | 0.29 | 0.15 | | 18 Aug 2018 | 0.02 | 0.04 | 0.02 | 0.29 | 0.15 | | 3 Sep 2018 | 0.03 | 0.04 | 0.03 | 0.30 | 0.14 | | 19 Sep 2018 | 0.03 | 0.05 | 0.03 | 0.33 | 0.17 | | 5 Oct 2018 | no value | no value | no value | no value | no value | | 21 Oct 2018 | 0.06 | 0.06 | 0.04 | 0.17 | 0.07 | | 6 Nov 2018 | 0.03 | 0.05 | 0.03 | 0.40 | 0.17 | | 22 Nov 2018 | 0.03 | 0.04 | 0.02 | 0.18 | 0.07 | | 8 Dec 2018 | 0.09 | 0.09 | 0.08 | 0.23 | 0.14 | | 24 Dec 2018 | no value | no value | no value | no value | no value | #### How it was made 1. Ask the catalogue: sits asks the provider's catalogue which images cover the book's tiles, bands and dates, and a whole year of one tile. The answers are kept, so every later run uses the same list. 2. Find the tiles: The pipeline finds the tiles of the satellite's grid around the region, from the grid that comes with sits. It reads a small picture of each tile of the cube from its least cloudy image. 3. Build the regular cube: sits makes a regular cube of the region from those images: one image per period, all on the same grid of pixels. For each pixel of a period it takes the first valid value, least cloudy image first. 4. Read the images as they come: The pipeline reads the images the catalogue listed, as they come. It marks each pixel as having no data, a clear value, or a value sits masks, such as cloud or shadow. 5. Try other periods and pixel sizes: sits makes the regular cube again at other periods and pixel sizes. It also makes it once without the images' cloud cover, so sits takes the images in date order, not least cloudy first. 6. Read each image alone: Each image is read alone, on the grid sits builds for it. Stacked in the order sits takes them, the images show which one gave each pixel of the regular cube. 7. Check against sits: The pipeline compares its own reads with what sits made, and stops the run on the checks that must hold. The results are gathered only once those checks pass. 8. Prepare the pictures: The pipeline picks the brightness drawn black and white, the same for every date, so the dates can be compared. Values are packed into one byte each. 9. Write the files: The pipeline writes the files the website reads, each with a record of its sources and of how closely a rerun repeats it. #### Reproducibility details - A rerun on aarch64-apple-darwin23 gave the same files, byte for byte. Every step of the pipeline: 1. Lists the bands read: the book's and those of the natural-colour pictures, in the order sits has them. `cube_read_bands("AWS", "SENTINEL-2-L2A", c("B02", "B8A", "B11"), c("B03", "B04"))` 2. Lists every period with every resolution of the runs. `regularize_choices(c("P8D", "P16D", "P1M"), c(20L, 40L, 80L))` 3. The arguments of the book's call to sits_cube(). `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands)` 4. The same call for one tile over a whole year. `cube_call(list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11" ), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list( tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), cube_bands, "20LLP", "2018-01-01", "2018-12-31")` 5. Lists the bands of the runs: the natural-colour ones, in the order sits has them. `regularize_run_bands(cube_bands, list(red = "B04", green = "B03", blue = "B02"))` 6. Lists the images of the book's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_book_call, "cache/cubes", "cube_book")` 7. Lists the images of the year's call with sits_cube(): a copy of the live AWS catalogue, kept in cache/cubes/ and read from there. `cube_kept_listing(cube_year_call, "cache/cubes", "cube_year")` 8. Makes the regular cube of the book's call over the region with sits_regularize(), keeping the files already made. `cube_regularize(cube_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_book_B02-B03-B04-B8A-B11", 4L)` 9. Finds the Sentinel-2 tiles around the region. `cube_images_grid(list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), cube_listing, 100L)` 10. Reads a small picture of each tile from the overview of its clearest image, keeping the reads in cache/cubes/. `cube_images_read(cube_listing, list(red = "B04", green = "B03", blue = "B02"), 3L, "cache/cubes")` 11. Makes the regular cube of the year with sits_regularize(), keeping the files already made. `cube_regularize(cube_year_listing, list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "P16D", 40L, "cache/cubes/reg_year_B02-B03-B04-B8A-B11", 4L)` 12. Runs sits_regularize() on the book's images at one period and resolution, keeping the files in cache/regularize/. `regularize_run_cube(cube_listing, regularize_choice, regularize_bands, "CLOUD", list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), "cache/regularize", 4L, 1L)` 13. Reads every image of each tile over the tile's part of the region, as it comes, keeping the reads in cache/cubes/. `cube_tile_images(cube_listing, cube_book, cube_book_call, 40L, "EPSG:32720", "cache/cubes")` 14. Gathers the images, ranked in the order sits takes them, the regular values, the year and the map. `cube_export_gather(cube_listing, cube_book, cube_images, cube_year_listing, cube_year, cube_map_images, cube_map_grid, cube_book_call, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), 40L)` 15. Marks each pixel of each image as no data, clear or masked by sits. `purrr::map(cubes_record$tiles, cube_images_states, cubes_record$cloud_values)` 16. Picks the reflectance drawn black and white, one for red, green and blue. `cube_images_stretch(cubes_record, list(red = "B04", green = "B03", blue = "B02"), c(0.02, 0.98))` 17. Stores the year in one byte per value, each band between two percentiles. `cube_export_year_codes(cubes_record$year$values, c(0.001, 0.999))` 18. Reads each image of the book's periods alone, on the view sits builds for it with no mask, keeping the reads in cache/regular_cubes/. `cube_alone_book_reads(cube_listing, cubes_record, "B02", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "cache/regular_cubes/alone_book_B02", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp", 0.5)` 19. Runs sits_regularize() on the book's call with the scene cloud cover removed, keeping the result in cache/regular_cubes/. `cube_checks_copied_run(cube_listing, "B02", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP" ), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list( crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), cubes_record, "cache/regular_cubes", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp")` 20. Reads the grid of the region on each tile at each resolution from the runs of the book's period. `cube_checks_region_grids(cube_listing, regularize_run, cubes_record, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), c(20L, 40L, 80L))` 21. Runs sits_regularize() on one image alone and gdalwarp at three overview choices, keeping the result in cache/regular_cubes/. `cube_checks_overview_check_run(cube_listing, cubes_record, list(tile = "20LLP", band = "B02"), list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP" ), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98 ), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), "cache/regular_cubes", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp")` 22. Reads the same image alone in natural colour and runs sits_regularize() on it with those bands. `cube_checks_pictures_check_run(cube_listing, cubes_record, list(tile = "20LLP", band = "B02"), regularize_bands, list( source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98 ), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), "cache/regular_cubes", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp")` 23. Pairs the cells of the strip both tiles cover. `cube_checks_book_strip(cubes_record, c("20LKP", "20LLP" ), 40L)` 24. Runs sits_regularize() on each image of the year alone, with the date of its period, keeping the files in cache/regularize/. `regularize_year_alone(cube_year_listing, cubes_record, "B02", "CLOUD", list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "cache/regularize/year_20llp_B02", 1L)` 25. Stacks the images of each period in sits' order and finds the one each pixel came from. `cube_alone_book_stacks(regular_cubes_book_reads, cubes_record, "B02")` 26. Compares the run with no scene cloud cover with the book's regular cube, after checking it against the images read alone. `cube_checks_copied_entries(regular_cubes_copied_run, cubes_record, regular_cubes_book_reads, cube_listing, "B02")` 27. Compares the two tiles' regular images where they overlap. `cube_checks_overlap_periods(cubes_record, regular_cubes_strip, "B02")` 28. Stops the run unless gdalwarp -ovr AUTO and the images read alone give sits' values on every pixel. `cube_checks_check_overview(regular_cubes_overview, regular_cubes_book_reads, regular_cubes_pictures)` 29. Draws one tile's images as they come and its regular images, small, for the journey's picture. `cube_checks_export_picture(cubes_record, cube_states, list(red = "B04", green = "B03", blue = "B02"), cube_stretch, 32L, list(tile = "20LLP", card_tile = 64L, sheet_tile = 192L))` 30. Finds which image gives each pixel of each period of the year, in the order sits takes them. `regularize_year_sources(regularize_alone, "20LLP", "B02")` 31. Writes one tile's part of the region over the year, one byte per value, for the website. `site_write_bin(cube_export_year_bytes(cube_year_codes$codes), "../web/public/data/cube-year.bin", "uint8")` 32. Writes the book's call, its images and regular cube, the map and the year, for the website. ``site_write_json(cube_export_record(cubes_record, cube_states, cube_stretch, cube_year_codes, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), list(tiles = list(`20LKP` = list(ard = "cube-ard-20lkp.png", mask = "cube-ard-20lkp-mask.png", regular = "cube-regular-20lkp.png", map = "cube-map-20lkp.png"), `20LLP` = list(ard = "cube-ard-20llp.png", mask = "cube-ard-20llp-mask.png", regular = "cube-regular-20llp.png", map = "cube-map-20llp.png")), year = "cube-year.bin")), "../web/src/data/cubes.json")`` 33. Stops the run unless the images read alone, stacked in sits' order, give the book's regular cube on every pixel. `cube_alone_check_book(regular_cubes_book, cubes_record, "B02", "cache/regular_cubes/alone_book_B02")` 34. Reads each image of the year alone in natural colour, with its cloud band, keeping the reads in cache/regular_cubes/. `cube_alone_year_reads(cube_year_listing, cubes_record, regularize_sources, regularize_bands, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP" ), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98 ), levels = 32L, quantiles = c(0.001, 0.999), map = list( around_km = 100L, overview = 3L)), "cache/regular_cubes/alone_year_20llp", "/Users/felipecarlos/Documents/sits/sits-embeddings/pipeline/cache/tmp", 0.5)` 35. Stops the run unless the cloud band read here masks each image of the year as sits did. `cube_alone_check_year_cloud(regular_cubes_year_reads, regularize_alone, cubes_record$cloud_values, "cache/regular_cubes/alone_year_20llp")` 36. Gathers the grids, the sources of the book's pixels, the overview, the run with no cloud cover and the year, once the self-checks pass. `cube_checks_export_gather(regular_cubes_grids, regular_cubes_book, regular_cubes_strip, regular_cubes_overlap, regular_cubes_overview, regular_cubes_copied, regular_cubes_year_reads, list(regular_cubes_book_check, regular_cubes_overview_check, regular_cubes_cloud_check))` 37. Writes the grids, the sources of the book's pixels, the overview, the run with no cloud cover and the year, for the website. `site_write_json(cube_checks_export_record(regular_cubes_record, cubes_record, regularize_sources, regular_cubes_picture, list(source = "AWS", collection = "SENTINEL-2-L2A", tiles = c("20LKP", "20LLP"), bands = c("B02", "B8A", "B11"), picture_bands = c("B03", "B04"), cloud_band = "CLOUD", start_date = "2018-06-30", end_date = "2018-08-31", period = "P16D", res = 40L, region = list(crs = "EPSG:32720", xmin = 294880L, xmax = 310240L, ymin = 8840000L, ymax = 8848000L), year = list(tile = "20LLP", start_date = "2018-01-01", end_date = "2018-12-31", window = 12L), multicores = 4L, rgb = list(red = "B04", green = "B03", blue = "B02"), stretch = c(0.02, 0.98), levels = 32L, quantiles = c(0.001, 0.999), map = list(around_km = 100L, overview = 3L)), "B02", list(ranks = "regular-cubes-book-ranks.bin", cards = "regular-cubes-picture-cards-20llp.png", sheets = "regular-cubes-picture-sheets-20llp.png")), "../web/src/data/regular-cubes.json")` The files: - `cubes.json`: SHA-256 `f29bb2e9135a6f824c0d512868ea1c3d67bd2532c285b6662f87926e071d5874`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `regular-cubes.json`: SHA-256 `751974674c0e71d20c1054ce3e79129531244a10501768116130594eea997323`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. - `cube-year.bin`: SHA-256 `f8300b9b4fad10dbeb3582248d101b35e84c401e847c3cffba2fd167632e2dd2`. A rerun on aarch64-apple-darwin23 gave the same file, byte for byte. To make them again: `cd pipeline && Rscript make.R`. Try it: [A regular cube, by date and by place](/interactive-images/regular-cubes-cube.md) ## Next On this site: [From series to embeddings](/embeddings.md). How does the time series of a place become a few numbers that say what it is? In the sits book: [Analysis-ready image collections](https://e-sensing.github.io/sitsbook/dc_ardcollections.html) and [Building regular data cubes](https://e-sensing.github.io/sitsbook/dc_regularize.html).