Blog

What is label leakage, and why it wrecks a small training set

A metal fence is almost fully swallowed by dense green shrubs and vines at a wooded field edge.

Label leakage is when information that should only exist in your validation set sneaks into training, so the classifier looks sharp on paper but hasn't learned the feature. In imagery work it's less obvious than in tabular data, because the leak usually isn't a duplicated row. It's spatial.

Here's the version that catches most people: you digitize fifteen polygons around a feature, you need to split off some for validation, so you grab a few polygons from the far corner of the same tile. Looks like a clean split. Except pixels forty meters apart on the same image share lighting conditions, the same seasonal growth stage, the same sensor noise. The model is really just being retested on a slightly cropped version of what it already saw. Your validation accuracy climbs, your confidence climbs, and then the classifier falls apart the moment you run it on a neighboring parcel with different shadow angles.

Training examples vs validation examples aren't about count, they're about independence

The instinct is to think of training and validation as two buckets you divide your labelled examples into, eighty-twenty or whatever ratio feels safe. That's the right shape for a spreadsheet and the wrong mental model for a raster.

What actually matters is whether the validation examples are independent of training in every way the classifier could exploit. Different acquisition date if you can manage it. Different part of the AOI, ideally far enough apart that they're not sharing atmospheric conditions or the same mosaic seam. If you've only got one tile to work with, hold out a geographically separate block of it rather than scattering your validation polygons throughout the training area. A held-out field on the other side of a ridge tells you something. A held-out corner of the same field tells you almost nothing.

Overfitting a small labelled set happens faster than you'd expect

With a handful of labelled instances, a model has very little to generalize from, so it latches onto whatever's easiest. If three of your five examples happen to sit next to a dirt access road, don't be surprised when the classifier starts flagging dirt roads instead of your target feature. This isn't a flaw in the method so much as the arithmetic of small samples. Five examples give you five chances for a coincidental pattern to look like a real one.

More variety in what you label fixes this, not more math. Pull your training instances from different lighting conditions, different parts of the terrain, different adjacent land cover, so the only thing they consistently share is the feature itself. If every example you can get your hands on looks the same, say so when you hand off the labels, because the person building the classifier needs to know the training set is thin on variety, not just thin on count.

Quality checks worth running before you trust the output

A few checks catch most leakage and overfitting problems before they cost you a field trip:

  • Pull up the held-out examples and confirm none of them are spatially adjacent to a training polygon.
  • Spot-check a handful of flagged instances against imagery the model never saw during labelling or review, not just the original AOI.
  • Look at what the false positives have in common. Clustering around one terrain type or one acquisition date usually points to a labelling gap you can go back and fix.
  • Ask whether the training examples represent the full range of conditions across your actual project area, or just the easiest corner of it.

None of this requires a benchmark score to be useful. A reviewer looking at ten flagged instances and asking "would I have drawn a polygon here" catches more leakage than a confusion matrix does on a five-example training set.

If you're labelling a custom feature class for an upcoming project and want someone else doing the quality pass alongside you, Few-Shot Mapping runs a foundation-model workflow against your labelled examples and your imagery, built for exactly this kind of small, specific class. Get in touch if you've got a feature that doesn't exist in any off-the-shelf classifier and a few good examples to start from.

Request early access

← Back to the blog