Cross-validation that respects structure, ROC versus precision-recall, calibration, threshold selection, and tying a metric to a business decision.
3 questions in Machine Learning.
ROC-AUC hides false positives under imbalance because the false-positive rate divides by a huge negative count, so thousands of wrong alerts barely move it. Precision-recall exposes that honestly but is anchored to the base rate, so its numbers cannot be compared once prevalence moves.
A model is calibrated when among all cases it scores 0.30, close to 30% are positive. Most classifiers emit a score that ranks well and is not a probability, so any decision built from expected value is wrong by an amount nobody measures. Fit isotonic or Platt scaling on a held-out set to repair it.
Build the split to imitate the gap between training and deployment. Group all of an entity's rows onto one side when rows repeat per entity, split forward in time for anything temporal with an embargo covering the label window, and treat a plain random shuffle as the default that silently inflates almost every score.