Your model is 97% accurate and completely useless

A heart disease classifier I was asked to review reported 97% accuracy. It also caught almost none of the patients who actually had the disease. Both facts were true at once, and only one of them mattered.

Three percent of the dataset were positive cases. A model that predicts "healthy" for every single patient, with no learning whatsoever, scores 97%. That is the number that had been reported to stakeholders as evidence the model worked.

What accuracy is actually measuring

On an imbalanced dataset, accuracy is mostly a measurement of the imbalance. It answers "how often is the model right" when the question you care about is "of the cases that matter, how many did it find, and how much noise did it create getting there."

• Recall — of the real positive cases, what fraction did we catch? This is the one clinical screening lives on.
• Precision — of the cases we flagged, what fraction were real? This is what determines whether people start ignoring the alerts.
• PR-AUC — precision and recall traded off across every threshold. Far more informative than ROC-AUC when positives are rare, because ROC-AUC is flattered by the enormous true-negative pile.
• The confusion matrix itself — four numbers, no summarising, no hiding.

Look at the report, not the score

from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    average_precision_score,
)

# Stratify the split, or a rare class can end up absent from one side entirely.
probabilities = model.predict_proba(X_test)[:, 1]
predictions = (probabilities >= 0.5).astype(int)

print(confusion_matrix(y_test, predictions))
print(classification_report(y_test, predictions, digits=3))

# PR-AUC is threshold-free and does not flatter the majority class
# the way ROC-AUC does when positives are rare.
print("PR-AUC:", average_precision_score(y_test, probabilities))

That 0.5 threshold is a default, not a decision. It is the single cheapest lever you have: moving it costs no retraining and directly trades precision for recall. Tune it against the metric your problem actually cares about, on validation data, before you reach for anything more elaborate.

On SMOTE

SMOTE synthesises new minority examples by interpolating between existing ones. It genuinely helps sometimes. It is also the single most misapplied technique I encounter, for one specific reason.

from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

# imblearn's Pipeline resamples the training fold only. sklearn's Pipeline
# would apply SMOTE to the validation fold too, which is the leak.
pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("balance", SMOTE(random_state=42)),
    ("model", RandomForestClassifier(random_state=42)),
])

scores = cross_val_score(
    pipeline,
    X,
    y,
    cv=StratifiedKFold(5, shuffle=True, random_state=42),
    scoring="average_precision",
)
print(f"PR-AUC: {scores.mean():.3f} (+/- {scores.std():.3f})")

Before reaching for SMOTE at all, try class_weight="balanced". It is one argument, it invents no data, and on tabular problems it is frequently as good or better. Synthetic minority points in a region where the two classes overlap are, after all, just confidently mislabelled training data.

The order that works

1. Agree with the domain owner on which error is worse. Everything downstream follows from this.
2. Establish the trivial baseline — always predict the majority. Every later number is judged against it.
3. Use a stratified split, and keep a test set you touch exactly once.
4. Report the confusion matrix and PR-AUC. Never report accuracy alone on imbalanced data.
5. Tune the decision threshold before touching the data distribution.
6. Try class weights. Only then try resampling, and only inside the pipeline.

The uncomfortable part is that this often makes your headline number look worse. A model that reports 82% recall and 61% precision sounds less impressive than 97% accuracy, and it is enormously more useful, because it is describing something real.