Conformal prediction is popular right now because it offers something rare in machine learning: a real, distribution-free statistical guarantee, not just a calibration score that looks reassuring. Give it a trained model and a calibration set, and it promises that its prediction sets will contain the true answer at whatever rate you ask for: 90%, 95%, whatever the application demands.
Building one of these systems against a real dataset (multimodal ICU admission data, predicting a clinical outcome) surfaced two separate failure modes. Neither shows up if you only look at the headline coverage number. Both are checkable in an afternoon once you know to look for them, and both generalize well past healthcare data.
Lesson one: a data-splitting bug that quietly voids the guarantee
The conformal guarantee has exactly one load-bearing assumption: the calibration data and the test data have to be exchangeable: drawn from the same distribution in a way that doesn't depend on their order. Violate that assumption, even slightly and even by accident, and the guarantee stops holding formally, whether or not the numbers still look fine.
The dataset used here allows multiple ICU stays within a single hospital admission, so a patient can appear more than once. A naive split into calibration and test sets, done at the level of the individual stay rather than the individual patient, will happily place two stays from the same person on opposite sides of that split. That's an exchangeability violation, and it wasn't hypothetical: checking for it directly, roughly 3% of the calibration and test sets combined shared a patient under a naive split. The coverage numbers computed on that split weren't obviously wrong: they still looked like a normal, plausible 90%-ish result. The formal guarantee behind them was already broken; it just wasn't visible from the output.
Most teams building on structured healthcare or financial data reach for a standard train/test split utility, run it, and move on. The split "worked" in the sense that it ran without an error. Nothing in that workflow prompts anyone to ask whether the unit being split (a visit, a transaction, a session) is actually independent of other units in the dataset, or whether it's secretly grouped by something else (a patient, a customer, an account) that can leak across the split. The fix is straightforward once you're looking for it: group by the entity that should never repeat across your split, and verify, don't assume, that there's zero overlap.
This isn't specific to healthcare data. Any dataset with repeated measurements per entity (multiple sessions per user, multiple orders per customer, multiple claims per policyholder) carries the same latent risk.
Lesson two: marginal accuracy is a vanity metric
Most teams evaluating a prediction model ask one question: how accurate is it, on average? That number is comforting, and it's also close to useless for deciding whether to trust the model on any specific case in front of you.
This system's headline result looked clean: prediction sets hit 90.6% empirical coverage against a 90% target. Textbook. The problem showed up once the analysis moved past the average and into subgroups. Patients with high lab-result missingness (a proxy for how closely a patient was being monitored) got prediction sets that were correct roughly 94% of the time. Patients with low missingness, generally the sicker, more closely watched patients with longer stays, were correct only around 85–87% of the time. Same model, same 90% marginal guarantee, and a 7–10 percentage point gap in reliability depending on which patient you were.
That gap isn't a bug: exact, distribution-free conditional coverage is provably impossible to guarantee in general, without further assumptions about the data. A marginal guarantee is the thing you can actually promise. The catch is that "on average" can quietly hide a large gap for exactly the subgroup where a wrong answer costs the most. In a lending model, that subgroup might be first-time borrowers. In a fraud model, it might be a transaction pattern rare enough to vanish in an aggregate accuracy number but common enough to matter operationally.
Group-conditional calibration, giving each subgroup its own threshold instead of one pooled threshold, closed most of the gap here. It also introduced a failure mode pooled calibration didn't have: a small percentage of patients in the tightly-calibrated group ended up with an empty prediction set, the model effectively refusing to commit to either outcome. That's a real cost, not a rounding error, and it's the kind of trade-off that's easy to leave off a results slide if you're only reporting the headline number. It got reported anyway: a calibration fix that trades a known, disclosed failure mode for a smaller reliability gap is a legitimate engineering decision; hiding the new failure mode to make the "after" chart look cleaner is not.
What this means if you're evaluating a model
Neither failure mode announces itself. A leaked split still produces a plausible-looking coverage number. A conditional coverage gap still produces a clean marginal average. The only way to catch either is to check for them directly: verify the split respects the actual grouping structure in your data, and ask for coverage broken out by the subgroups that matter to your business, not just the marginal number. If nobody's checked, that's not proof the model is fine. It's just proof nobody's looked yet.