Six weeks before the deadline I figured out how I was going to lose.
The public leaderboard had a floor that a lot of us were clustered against, and I became convinced that the scores down there were partly an illusion. Not fraud, nothing like that. Just a technique going around that squeezed extra performance out of the specific test rows everyone could see, in a way that had no reason to survive contact with the rows nobody could see. A good chunk of the visible standings was measuring how well people had fitted the visible data.
That diagnosis was correct. When the hidden half was finally scored, that whole family of models degraded by a bit over two feet, which is a lot in a competition decided by fractions of one.
I saw it coming. I wrote it down. I changed my entire strategy around it, stopped chasing public rank, and built a hedge.
The hedge did nothing.
How the hedge was supposed to work
You pick two final submissions. Your better private score is what counts, so the two finals are a genuine option: if one of them falls apart on the hidden data, the other one catches you.
That only works if they fall apart for different reasons. Two models that fail the same way are not a hedge, they are one model submitted twice. So the thing to optimize is how correlated their errors are. Low correlation means different failure modes means one of them plausibly survives whatever the private set throws at you.
I did this properly, by the book. I took my primary submission and my intended hedge, computed the correlation between their errors, and got 0.643. Meaningfully below one. Different enough to be a real second bet.
I checked the box and moved on to other things.
The question I actually needed to ask
I had measured the correlation between my primary and my hedge.
The thing I was afraid of was neither of them.
The risk was a specific failure mode: an entire lineage of public models that had over-fitted the visible rows and would deflate when scored on hidden ones. That was the danger. That was the thing my whole strategy had been reorganized around six weeks earlier.
And my primary submission was built on that lineage.
I had forked one of those public models early on, because forking it was worth an enormous jump in score and refusing it on principle would have been vanity. Then I bolted my own work onto it, retrained pieces of it, differentiated it, and over two months came to think of it as mine. It was mine, in every way that mattered for the work. It was also, structurally, a member of the family I was hedging against, and when the family shifted, it shifted with them. It took the full hit.
So the shape of my error was this. I asked:
How correlated is my hedge with my primary?
When the question that mattered was:
How correlated is my primary with the risk I am hedging against?
The first question assumes your primary is outside the danger. Mine was inside it. And once your primary is inside the failure mode, the hedge is not protecting the position, it is the only thing standing outside a fire you thought you were watching from across the street.
There is a well-known version of this in finance. You hold a portfolio, you buy protection against a downturn, and then the downturn arrives and your protection turns out to have been issued by a bank that also owns the thing collapsing. Everybody involved measured a correlation. Nobody measured the one that mattered.
Why the correct diagnosis made it worse
Here is the part I find most uncomfortable, and it is why I wanted to write this entry rather than quietly fold it into a summary.
Being right about the diagnosis made me less likely to catch the flaw in the remedy.
The moment I worked out that the public floor was an illusion, I got the satisfying feeling of having seen something most people in the standings had not. I had a thesis. I reorganized around it. Everything after that point was executing on a correct insight, and executing on a correct insight feels like the safest possible position to be in.
It is not. The insight and the response to it are two separate pieces of work, and being right about the first one buys you nothing on the second. If anything it costs you, because the confidence from the diagnosis flows straight into the remedy without stopping to be checked. I never re-examined whether my primary belonged to the family I had just identified as dangerous, and the reason I never re-examined it is that I was busy being right about something else.
The habit I want out of this: after any decision that follows from a diagnosis, go back and check the diagnosis against the thing you already own, not just against the thing you are about to buy. Ask what your existing position shares with the failure mode. Lineage, training data, preprocessing, the feature table, the assumptions baked in three layers down. If it shares any of those, it is exposed, no matter what your correlation number says.
The uncomfortable counterfactual
There is a second half to this and it complicates the moral.
After the competition ended I went back and scored every choice I could have made. Not just what I picked, but what the alternatives would have produced. Three of them mattered: what I actually selected, what Kaggle would have picked for me automatically if I had done nothing at all, and the best possible pair chosen with perfect hindsight.
My careful, deliberate, strategically-motivated selection came in slightly behind what the platform would have done on its own if I had never touched the setting.
By a small margin, in score terms. About 370 places, in rank terms. All that thinking, and the null action beat it.
And then the number that actually settles it: the best pair available to me, chosen with perfect knowledge of the future, was better than my choice by roughly the same small margin again. Perfect foresight across every submission I owned was worth a few hundred places, and not one of those options reached a medal. Not the one I picked, not the one the platform would have picked, not the one an oracle would have picked.
So the honest scoring of the endgame is: I was slightly wrong, it cost slightly, and the entire decision barely mattered from the beginning.
That is not the same as saying the work was wasted, and I want to be careful here because the tempting lesson is the wrong one. The tempting lesson is "you overthought it, you should have done nothing." But I only know the range of outcomes because I measured it afterward. Going in, I could not see that every option landed within a few hundred places of each other. And the process caught a genuine trap along the way: if you select only one of your two finals, the platform quietly fills the empty slot with your best-scoring unselected submission, which in my case would have silently inserted exactly the kind of model I was trying to avoid. The interface still reads "2 of 2" while it does this. Verifying the selection programmatically, rather than trusting what the screen said, was correct and I would do it again.
The lesson is narrower and more useful than "don’t overthink." It is this: being careful is not the same as it counting for anything. Before spending real effort on a decision, bound the range. What is the best outcome available here, what is the worst, and how far apart are they? If every path lands within a few hundred places of the others and the goal is thousands of places away, then rigor applied to that decision is rigor pointed at the wrong variable. It will still be rigorous. It will still not matter.
I spent the last stretch of that competition being extremely careful about a choice whose entire outcome range was smaller than the gap I needed to close. The care was real. The stakes were never there, and I never checked whether it was.
Frequently asked questions
How do you choose two final submissions in a Kaggle competition?
The standard advice is to pick two models whose errors are weakly correlated, so they fail in different circumstances and at least one survives a shift between the visible and hidden test data. That advice is right as far as it goes. The trap is measuring correlation between your two finals while forgetting to check what your primary shares with the specific failure mode you are worried about. If your primary is built on the lineage you fear, the pair can look nicely decorrelated and still leave you fully exposed, because the hedge is the only asset outside the risk.
What does it mean for a hedge to be correlated with the risk?
A hedge only pays when it behaves differently from your main position under the exact conditions you fear. If it shares the ingredients that cause the failure, whether that is training data, a forked codebase, a preprocessing step or an underlying assumption, it moves the same direction at the same moment and provides no protection. The check is not "are these two things different from each other" but "does my main position share anything with the thing that would hurt me." Audit lineage and shared components, not just an error correlation between the two candidates.
Should you trust automatic final submission selection?
Understand what it does before you decide. On Kaggle, any final slot you leave empty gets filled with your best-scoring unselected submission, and this happens per slot, so selecting one of two silently inserts a model you may have deliberately rejected. The interface may still display a complete selection while this is true. Verify the actual state through the API rather than reading the screen. In my case the automatic default happened to score slightly better than my manual choice, but that was luck rather than a reason to trust it.
How do you know if a decision is worth spending time on?
Bound the outcome range before you invest the effort. Estimate the best and worst results available across every option, and compare that spread against the distance to your actual goal. If every path lands close together and your target is much further away, the decision barely matters no matter how important it feels, and the effort belongs somewhere else. Being careful about a decision and that decision mattering are two different things, and they feel identical while you are inside them.
More in this series
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full ROGII Wellbore Geology Prediction thread.
- How I killed sixteen ideas without wasting a submission · the pre-registered gates that made the endgame decisions cheap.
The competition is over. This is the whole picture, including the parts that did not work.
