Loading...
Cross-validation that respects structure, ROC versus precision-recall, calibration, threshold selection, and tying a metric to a business decision.
3 questions in Machine Learning.
ROC-AUC vs precision-recall is a metric choice for imbalanced classification. ROC-AUC measures ranking across all negatives, while PR curves expose alert precision and recall at the positive-class prevalence.
A model is calibrated when among all cases it scores 0.30, close to 30% are positive. Most classifiers emit a score that ranks well and is not a probability, so any decision built from expected value is wrong by an amount nobody measures. Fit isotonic or Platt scaling on a held-out set to repair it.
Build the split to imitate the gap between training and deployment. Group all of an entity's rows onto one side when rows repeat per entity, split forward in time for anything temporal with an embargo covering the label window, and treat a plain random shuffle as the default that silently inflates almost every score.