Last lesson’s fix (splitting by time, not randomly) closes one specific leak. Leakage in general is the umbrella term for any way information about the outcome sneaks into training that wouldn’t be available at real prediction time — and it comes in a few recognizable shapes worth being able to name on sight, because “the model works suspiciously well” is often the only symptom before it ships.
- Target leakage: a feature that is itself downstream of the outcome — it only exists, or only takes a meaningful value, because the thing being predicted already happened. Using it isn’t predicting the future; it’s reading an answer key smuggled into the questions.
- Temporal leakage: training data effectively includes information from after the point in time the model would actually need to make its prediction (last lesson’s random-shuffle CV failure was one specific mechanism producing exactly this).
- Group leakage: rows that are correlated because they share a real-world entity (the same patient across multiple hospital visits, the same user across sessions) get split so that some of that entity’s rows land in training and others in test — letting the model partly recognize the entity, not learn the general pattern, and get credit for it at test time.
A loan-default model includes account_closed_date — a field that only gets populated once an
account has already been closed due to default. Trained with it, the model scores near-perfectly on
both train and test.
Which kind of leakage is this?