What does point-in-time correctness mean, and how does a naive join break it?

The rule and why it is not obvious

Point-in-time correctness says that every row of a training set must contain only information that existed at the moment the row's event occurred. If a payment was scored at 09:22 on the fourteenth, then every feature attached to that payment must be computable from data available at 09:22 on the fourteenth, and from nothing later.

The reason this is easy to break is that feature tables are usually keyed by entity, not by entity and time. customer_features has one row per customer holding their current values, updated nightly. Joining your labelled events to that table on customer_id is the obvious thing to write, it runs, and it produces a dataset with no nulls and excellent metrics. It is also worthless, because the feature values in it were computed after most of the events they are attached to.

The leaky join and the correct one

Suppose you are predicting whether a customer will default, and one feature is their count of failed payments.

-- LEAKY: joins on entity only. The feature row is whatever the nightly
-- job last wrote, which for an event from March reflects data through July.
SELECT e.event_id, e.event_ts, e.label, f.failed_payments_30d
FROM   labelled_events e
JOIN   customer_features f
  ON   f.customer_id = e.customer_id;

A customer who defaulted in March has, by July, a large failed-payment count precisely because they defaulted. The model learns that high failed payments predict default, which is true but circular: at scoring time in March the count was low. The feature is a downstream consequence of the label.

The corrected form joins on entity and time, taking the most recent feature version strictly before the event.

-- CORRECT: as-of join. For each event, the latest feature version
-- whose validity began strictly before the event timestamp.
SELECT e.event_id, e.event_ts, e.label, f.failed_payments_30d
FROM   labelled_events e
LEFT JOIN LATERAL (
  SELECT failed_payments_30d
  FROM   customer_features_history h
  WHERE  h.customer_id = e.customer_id
    AND  h.valid_from < e.event_ts          -- strict: equal timestamps leak
  ORDER BY h.valid_from DESC
  LIMIT 1
) f ON true;

Two details in that query carry most of the weight. The join is LEFT, deliberately: if no feature version existed before the event, the correct value is null and the model must learn to cope, because that is exactly the situation at serving time for a new customer. Replacing the left join with an inner join silently drops every cold-start row from training, which is a different leak — the model is evaluated only on the easy population.

The second is that the inequality is strict. If valid_from records the timestamp of the batch that computed the feature, and an event happens in the same second, a non-strict comparison can admit a value computed from data that includes the event. In practice you want the comparison against the moment the value became available to the serving path, which may be later than the moment it was computed — a nightly job finishing at 02:00 with data through midnight means the value is knowable from 02:00, not from midnight. Getting this offset right is fiddly and is where careful teams still find leakage.

The symptom, and why it fools people

Leakage announces itself in a very specific way: offline metrics that are markedly better than anything the business has seen, and no production lift whatsoever. The model's AUC is 0.94 in the notebook and its live performance is barely distinguishable from the old rules engine. Nothing errors. No monitoring fires, because the input distributions are fine — the features being served are real, correctly computed, current features. They simply do not carry the predictive signal that the training features carried, because that signal was the future.

The second recognisable symptom is a feature importance chart where one feature dominates implausibly. If a single feature is doing most of the work and it is not one a domain expert would have nominated, treat that as a leakage hypothesis before treating it as a discovery.

The failure that separates a strong answer here is conflating two different problems. A feature that is unavailable at serving time is a plumbing problem: you notice it when you try to deploy, because there is nothing to call. A feature that is available but temporally contaminated in training is a silent problem: deployment works perfectly, the pipeline is healthy, and only the business metric knows. Candidates who describe leakage purely as "using a feature you won't have in production" have described the harmless version.

Two ways out, and one is much easier

The rigorous way is the as-of join, which requires your feature tables to keep history — an append-only log of feature versions with validity timestamps, rather than a table of current values overwritten in place. This is the single largest reason to run a feature store: not the online cache, but the fact that a well-built one stores feature history and implements the as-of join for you, so nobody writes that lateral join by hand and gets the inequality wrong.

The much easier way, where you can afford the wait, is to stop constructing training rows from history at all. Have the serving path log the exact feature vector it computed for every request, then attach the label when it arrives and train on those logged vectors. Temporal correctness is guaranteed by construction, because the vector is the one that was served. The cost is that you cannot train until you have accumulated logs, so it does not help with the first model, and it does not help you evaluate a feature you have never served. Most mature setups use both: logged vectors as the default, as-of joins when a new feature has to be backfilled over history.

Excellent offline metrics with no production lift is the signature of leakage, not of a good model. Any join that keys on entity without also keying on time should be treated as guilty until the timestamps are shown.