Module 8: Geospatial Foundation Models

Phase: III, Deep learning
Level: Advanced
Estimated time: ~12 hours (about 4 h concepts, 7 h lab, 1 h write-up)
Prerequisites: Module 7 (your U-Net baseline), Module 4 (features and spatial CV), Phase II.
Portfolio thread: builds Portfolio Project C and Public write-up #2, a head-to-head benchmark of a U-Net, geospatial foundation models, and satellite embeddings.

Why this module

Job descriptions in 2026 increasingly ask for "experience fine-tuning geospatial foundation models" and "embedding-based workflows." The real skill is not naming a model; it is knowing when a foundation model beats a plain U-Net and when it just costs more, and being able to prove it with a fair benchmark. Foundation models earn their keep most clearly in the low-label regime, which is exactly the situation most real EO projects are in. This module makes you fluent in the current landscape and, more importantly, in evaluating it honestly.

Learning outcomes

After this module you can:

1. Explain masked-autoencoder pretraining and how geospatial foundation models produce reusable representations.
2. Name the current models and their differences (Prithvi-EO-2.0, Clay, TerraMind, DOFA) and when each fits.
3. Use embedding-based workflows (AlphaEarth satellite embeddings) and compare them to feature engineering.
4. Fine-tune a foundation model with TerraTorch, including frozen-backbone linear probing.
5. Benchmark a U-Net against foundation models and embeddings on full data and a low-label regime, fairly.
6. Read foundation-model benchmarks (PANGAEA, GEO-Bench) critically and produce a cost table.

Concept lessons (about 4 hours)

Lesson 1: What a geospatial foundation model is

A foundation model is pretrained with self-supervision on enormous unlabeled archives, then fine-tuned on your small labeled task. The dominant pretraining recipe is the masked autoencoder: hide random patches of an image and train the network to reconstruct them, which forces it to learn general structure without labels. The payoff is a backbone whose representations transfer, so you need far fewer labels to reach good accuracy on a new task. That low-label advantage is the whole point, because labels are the scarce resource in EO.

Lesson 2: The current model landscape (2026)

Learn these as a toolkit with different strengths:

• Prithvi-EO-2.0 (NASA and IBM): pretrained on HLS, handles multispectral and temporal input, and is the model the burn-scar demonstration was built around. A solid, well-supported default.
• Clay: an open, global self-supervised model with an active community, good for embedding and fine-tuning workflows.
• TerraMind (IBM, ESA, and Julich, 2025): the first any-to-any multimodal generative EO foundation model, which learns across modalities and introduced "Thinking in Modalities" fine-tuning. In an ESA evaluation on the PANGAEA benchmark it outperformed a dozen other models by a meaningful margin, making it the current front-runner for multimodal tasks.
• DOFA: a wavelength-conditioned design that adapts to arbitrary sensors and band configurations, useful when your input does not match a fixed sensor.

The point is not to memorize a leaderboard; it is to match a model to your data (single-sensor versus multimodal, temporal versus single-date, standard versus unusual bands).

Lesson 3: Embedding-based workflows

Google's AlphaEarth satellite embeddings distill a year of multi-sensor data into a 64-band per-pixel vector at 10 m, available annually from 2017 onward. Instead of hand-engineering the features you built in Module 4, you can pull these embeddings and train a tiny classifier or regressor on top, often matching or beating engineered features with a fraction of the code and compute. This "embedding then small model" pattern is increasingly how production systems classify and detect, and it is worth contrasting directly against your Module 4 pipeline: same task, far less feature plumbing.

Lesson 4: Fine-tuning with TerraTorch

TerraTorch is the config-driven toolkit for this work: a backbone registry (Prithvi, TerraMind, and others), a factory that pairs a backbone with a decoder head for your task, and PyTorch Lightning underneath. Two modes to know: full fine-tuning (update the whole model, best accuracy, most compute) and frozen-backbone linear probing (train only a small head on frozen features, fast and a strong low-label baseline). Always try the linear probe; it tells you how good the pretrained representation already is.

Lesson 5: Honest benchmarking and cost

Reuse the exact splits and metrics from Module 7 so the comparison is fair. Run every method on both the full training set and a low-label regime (for example 10 percent of labels), because that is where foundation models tend to separate from a U-Net trained from scratch. Read community benchmarks (PANGAEA, GEO-Bench) critically: a model that tops a leaderboard on one task set may not win on yours, because results are strongly task- and domain-dependent. Finally, add the dimension most learners forget: cost. Record GPU-hours to fine-tune and inference throughput (square kilometers per minute), because a small accuracy gain that triples inference cost may not be worth deploying, and hiring managers love a candidate who reasons about that trade-off.

Guided lab (about 7 hours): a fair four-way benchmark

Step 1: Fine-tune a foundation model

Fine-tune Prithvi-EO-2.0 (use a smaller variant if GPU-limited) on the Module 7 burn-scar dataset with TerraTorch, using the same geographic splits and IoU/F1 metrics as Module 7. Also run a frozen-backbone linear probe.

Step 2: Add a second model and the embeddings route

Add Clay or TerraMind as a second foundation model, and separately build the embedding route: in Earth Engine, pull AlphaEarth 64-band embeddings over your AOI and train a small model for forest/non-forest (as in Module 4) and canopy-height regression (as in Module 5), comparing against your engineered-feature results.

Step 3: Benchmark helpers with tests

"""benchmark.py: fair-comparison helpers (pure, testable)."""
from __future__ import annotations
import numpy as np

def subset_indices(n, fraction, seed=0):
    """Deterministic label subset for a low-label regime (e.g. 10 percent)."""
    rng = np.random.default_rng(seed)
    k = max(1, int(round(n * fraction)))
    return np.sort(rng.choice(n, size=k, replace=False))

def macro_f1(cm):
    """Macro-averaged F1 from a confusion matrix (rows = true, cols = pred)."""
    cm = np.asarray(cm, float)
    f1s = []
    for i in range(cm.shape[0]):
        tp = cm[i, i]
        fp = cm[:, i].sum() - tp
        fn = cm[i, :].sum() - tp
        p = tp / (tp + fp + 1e-12)
        r = tp / (tp + fn + 1e-12)
        f1s.append(2 * p * r / (p + r + 1e-12))
    return float(np.mean(f1s))

# tests/test_benchmark.py
from eo_portfolio.benchmark import subset_indices, macro_f1

def test_subset_deterministic_and_sized():
    a = subset_indices(100, 0.1, seed=1)
    b = subset_indices(100, 0.1, seed=1)
    assert len(a) == 10 and len(set(a.tolist())) == 10
    assert list(a) == list(b) and a.min() >= 0 and a.max() < 100

def test_macro_f1():
    assert abs(macro_f1([[10, 0], [0, 10]]) - 1.0) < 1e-6
    assert abs(macro_f1([[8, 2], [0, 10]]) - 0.8990) < 1e-3

Use subset_indices to build the low-label training set identically for every method, so the comparison is fair.

Step 4: The benchmark table and cost

Produce one table: each method (U-Net, Prithvi, Clay or TerraMind, embeddings-plus-small-model) by IoU and macro-F1, on full data and on 10 percent of labels, plus GPU-hours to train and inference throughput. This table is the heart of the write-up.

Step 5: Public write-up #2 and commit

Write "U-Net versus foundation models versus embeddings for burn-scar mapping: accuracy, labels, and cost," stating plainly where the foundation model helped, where the U-Net was enough, and where embeddings won on effort. Commit benchmark.py, tests, the notebook, and the write-up.

git add src/eo_portfolio/benchmark.py tests/test_benchmark.py notebooks/10_gfm_benchmark.ipynb writeups/02_gfm_vs_unet.md
git commit -m "Module 8: GeoFM vs U-Net vs embeddings benchmark and write-up 2"
git push

Checkpoint

Self-check (answers below).

1. What does masked-autoencoder pretraining learn, and why does it help a small labeled task?
2. In what regime do foundation models most clearly beat a U-Net trained from scratch?
3. What is an embedding-based workflow, and how does it compare to feature engineering?
4. Why must a benchmark control the label subset and the splits across methods?

Interview-style questions (practice out loud).

• A foundation-model fine-tune underperforms your U-Net. Give the five most likely reasons.
• When would you choose an embedding-based workflow over fine-tuning a foundation model?
• How do you make a fair accuracy comparison across a U-Net, a GeoFM, and embeddings?
• A GeoFM wins IoU by two points but triples inference cost. How do you decide whether to deploy it?

Answers. (1) It learns general structure by reconstructing masked patches without labels, giving a transferable representation so fewer labels are needed to fine-tune a new task.
(2) The low-label regime, where a from-scratch network lacks data but a pretrained backbone already encodes useful structure.
(3) You use pretrained per-pixel embeddings (for example AlphaEarth's 64 bands) as input to a small model, replacing hand-engineered features with a learned, general representation and much less code.
(4) If methods see different labels or splits, differences reflect the data split rather than the model, so the comparison is meaningless; fix the subset and the geography for all.

Deliverable

A tested benchmark.py, a notebooks/10_gfm_benchmark.ipynb fine-tuning a foundation model with TerraTorch and comparing U-Net, foundation models, and AlphaEarth embeddings on full and 10-percent-label regimes with a cost table, and Public write-up #2. This is Portfolio Project C.

What a hiring manager sees

Anyone can fine-tune a model from a tutorial. Showing a fair, reproducible benchmark, with a low-label regime and a cost column, and a plain-spoken conclusion about when the foundation model is and is not worth it, demonstrates exactly the judgment teams need before adopting expensive models. Naming the current landscape correctly (Prithvi, Clay, TerraMind, DOFA, AlphaEarth embeddings) and knowing why you would pick each shows you track the field, which is a strong signal for a 2026 EO-ML role.

Currency note

Verified August 2026. TerraMind 1.0 (IBM, ESA, Julich) was released in 2025, is on Hugging Face (ibm-esa-geospatial), is integrated in TerraTorch, and led the PANGAEA benchmark in ESA's evaluation. Prithvi-EO-2.0 (NASA and IBM) and Clay are current open backbones; DOFA is the wavelength-conditioned option. Google's AlphaEarth satellite embeddings launched in July 2025, providing 64-band annual embeddings at 10 m from 2017 onward in the Earth Engine catalog and Google Cloud Storage, with annual updates. TerraTorch is under the same governance as TorchGeo. Model versions, checkpoints, and licenses move quickly, so confirm the current release and terms of any model before use, and pin versions in your Module 0.1 environment file. GPU is required (a free-tier notebook GPU handles the smaller variants).