From series to embeddings

A window of the Brazilian Cerrado, 256 × 256 pixels of 30 m, 7.68 km across, as Landsat saw it in May 2017.

The colours are false: SWIR16, NIR08 and RED as red, green and blue. Infrared says more about plants and soil than visible light, so growing vegetation comes out green and bare soil pink.

The satellite comes back, and the cube keeps one image a month: 24 months, from January 2017 to December 2018.

The book’s cube is regular: one image a month, every month on the same grid. Building data cubes from image collections shows how sits makes one. Press Play, or drag along the strip.

Now read one pixel, the one marked on the image, through the 24 months. The sensor measures 6 bands, BLUE, GREEN, RED, NIR08, SWIR16 and SWIR22, so the pixel has 6 lines, one per band.

That set of lines is a time series: all the satellite knows about these 30 m of ground. How sits reads the series of a place: How does one place become a time series?

45% of the pixel-months of this window were not observed: 8 of the 24 months have no observation at all, and only 8 are complete: they fall from May to August, the empty ones from September to March.

The strip shows how much was observed each month, and the image fades the rest. sits fills the gaps with impute_linear(): a hollow dot is a filled value. Now point at the image, or click it.

The pixel of step 1 as a table: one row per band, one column per month, 6 × 24 = 144 values for one place. The stronger a cell, the higher its value within its row.

Series compared as they come, value against value: Do all places behave the same?

Not all of them were measured. Round cells are months with no clear view, which sits filled with impute_linear(). Across the window, 45% of the pixel-months were filled.

A classifier would read them as if they were seen. And so many numbers for one place are a lot: the book notes that patterns in so many dimensions are costly to find, and unreliable without many training samples.

sits_encode() hands the 144 values to an encoder, a network that rewrites them. Step 3 tells how it learned to.

The funnel is a drawing: inside the encoder there is arithmetic on all the values at once, not movement.

64 values come back for the pixel, warm above zero and cool below. No cell of the row comes from one cell of the table.

Both ends are recorded: encoding the window again with the published encoder gave the published values back, identical in all 4,194,304 values.

Try another place on the small image, or on step 1’s image, which picks the same pixel at full size, and switch between the table and the row. Does the short list keep what matters? The details hold a recorded check.

What a classifier makes of these numbers: Classifying EO embeddings for land use and land cover applications

The encoder of this site was fitted on 518,374 time series (129,593 more held out to check it), and nobody told it what any of them shows. With no label to learn from, the method makes its own task out of the data.

Here is one of those series: band SWIR22 over its 24 dates. What it shows on the ground is not said here; the encoder was never told either.

While it trains, sits_ssl_lejepa() draws two views of it: it joins the values with straight lines, picks two sets of points along them that share none, and stretches each set back onto the 24 dates.

Each view is the same curve read on other days. Nothing new is observed. Replay draws them again.

Both views go through the same encoder: one network, the same weights. Each comes out as a vector of 64 numbers.

A small second network, the projector, follows, with 128 outputs. The loss reads the projector’s two outputs; the projector is dropped after training.

Training pulls the two outputs together, and through them the encoder’s vectors. The views differ in their days and agree in the shape of the seasons, so the encoder is pushed to keep what they share.

This part of the loss is called invariance. In sits_ssl_lejepa() it carries 0.98 of the weight.

There is a lazy way to win: give every series the same output. The views always agree, and the output says nothing. That is collapse.

So the loss has a second part, SIGReg: over the outputs of many series at once, it keeps the cloud they form round. It carries the other 0.02. No label appears anywhere in the method.

A reduced run, made for this site: sits pre-trained an encoder on a random subset of 20,000 series of the Cerrado, for 30 epochs on a CPU, in 9 min.

The curve is the loss sits recorded after each epoch: from 1.952 after the first to 0.269 after the last. The dashed line is the loss on the 4,000 series it held out.

The dots are 1,532 labelled samples, places whose land cover someone wrote down, where the encoder put them after epochs 1, 2, 4, 8, 16 and 30: flattened onto two axes, coloured by label.

The encoder never saw these labels; they only colour the dots. Where a label comes from: Data structure for image time series. Press an epoch to see the cloud then.

Do neighbours share a label? Take the 5 samples nearest to a sample in the 64 numbers the encoder gives it, not in the flat picture, and see whether most of them carry its label.

That holds for 73.7% of the samples after epoch 1 and 85.6% after epoch 30. Compared as raw series, 82.3%: the reduced run passes it at the snapshot of epoch 16.

The encoder of this site learned from 647,967 series and stopped early, after 54 of 150 epochs (7.1 h on a GPU). It kept only its loss per epoch: a curve, no cloud. Its neighbours share the label for 90.2% of the samples.

Switch runs, drag along a curve, or press a label to bring it forward. Compare the shapes of the curves, not their levels.

For every pixel of the window, sits_encode() has turned its whole time series into 64 numbers. sits stores them as 64 bands, EMB01 to EMB64: an embeddings cube, again a stack of images.

A screen has three colours to give, so the picture shows three bands at a time: EMB11, EMB12 and EMB13 as red, green and blue, as the sits book plots them. The colours mean nothing by themselves.

Now one pixel, the one marked on the image. Its 64 numbers, in order, are the bars: that list is its embedding, a vector.

A bar above the line is a positive number, a bar below it a negative one. The scale is the same for every pixel.

Each number is also a map. Take one place in the list and look at its value in every pixel: a picture of the window in grey, dark where the number is low, bright where it is high.

The band arrows step through all 64 bands, one at a time, and pointing at a bar shows its band. No band has a name like “red” or “moisture”: nobody chose what each one should measure; the encoder settled on it during training.

Any three bands make a colour picture. Choose them with the R, G and B menus, or press Surprise me; Book’s three brings back the book’s. Point at the image to read any pixel, and click to keep it.

The colours change completely and the shapes do not: what tells one field from the next is in the vectors, not in a choice of colours.