This is where the final score lands, and what it says about four months of work. I finished four tenths of a foot from a medal, and everything I built in the second half was worth less than that gap.
Six weeks before the deadline I worked out exactly how my models would fail, and built a hedge against it. The hedge was drawn from the same family as the thing it was hedging. Here is the question I should have asked instead.
For two months I optimized a score that had stopped ranking my models. It was still a valid measure of quality. It had just gone blind at the resolution I was working at, and three lines of code would have told me.
The competition closed and I pulled my final rank. The number was internally consistent, completely plausible, and wrong by nine hundred places. Here is the two-word API difference that caused it and how a competitor's blog post caught it.
Five lessons from a machine-learning competition where you predict the depth of rock you cannot see. Almost none of them are about models. A summary of the arc so far, and the through-line underneath it.
A passing test suite is the most reassuring lie in software. Three times in one competition, green checkmarks certified a broken model, each one lying in a different way. Here is how, with code.
In a competition where every experiment costs hours and submissions are rationed, the highest-leverage skill is not a better model. It is a rule written down in advance that lets a cheap probe tell you not to build the thing at all.
For a month my local validation and the competition leaderboard disagreed about whether I was improving at all. They were computing two different statistics. Here is how I found out, and the one-submission experiment that fixed it.