Evaluate Machine Learning Beyond Accuracy

A 95% accuracy trap

Suppose an illustrative dataset contains 950 negative cases and 50 positive cases. A rule predicting every case as negative is 95% accurate but has zero recall for the positive class. Always relate a metric to the decision the model will support.

Build a valid comparison

1. Define the task, prediction time and available features.
2. Separate train, validation and test data by the unit that matters, such as person or time.
3. Fit preprocessing and resampling only within training folds.
4. Include a majority-class rule and an appropriate simple model as baselines.

Match metrics to costs

Inspect the confusion matrix, precision and recall. Precision asks how many predicted positives are truly positive; recall asks how many true positives were found. Precision–recall summaries can help with rare positive classes, but must be interpreted alongside prevalence, calibration and operational costs. Choose thresholds using validation data rather than the final test set.

Explain limitations

1. Report the data split, prevalence, uncertainty and baseline results.
2. Inspect errors and relevant subgroup performance.
3. Keep the final test set separate from model selection.
4. Avoid deployment claims without evidence from the intended setting.