I wrapped search around a working agent and every time I let the search overrule it, it played worse. The experiment was built so a negative result cost zero rating, which is why I could afford to find out.
I named a package agents. So had the platform. Its loader appends to the path instead of prepending, so its module wins, mine never loads, and every match is a loss. Local tests could not see it, for two separate reasons.
For three weeks I thought my agent was rated around 709. It was 784.9. The number had been unavailable the entire time and my own code had quietly replaced it with a guess I stopped questioning.
I built a clean general search agent and it beat every opponent I had. Then I pointed it at a 400-line hand-written bot and lost three games in four. The opponents I had been measuring against were the problem.
The Pokemon Company put the Trading Card Game up as an AI problem, and I entered. Here is what the competition actually is, why this game is harder for a machine than chess, and what you are going to see over the next few weeks.
This is where the final score lands, and what it says about four months of work. I finished four tenths of a foot from a medal, and everything I built in the second half was worth less than that gap.
Six weeks before the deadline I worked out exactly how my models would fail, and built a hedge against it. The hedge was drawn from the same family as the thing it was hedging. Here is the question I should have asked instead.
For two months I optimized a score that had stopped ranking my models. It was still a valid measure of quality. It had just gone blind at the resolution I was working at, and three lines of code would have told me.
The competition closed and I pulled my final rank. The number was internally consistent, completely plausible, and wrong by nine hundred places. Here is the two-word API difference that caused it and how a competitor's blog post caught it.
Five lessons from a machine-learning competition where you predict the depth of rock you cannot see. Almost none of them are about models. A summary of the arc so far, and the through-line underneath it.
A passing test suite is the most reassuring lie in software. Three times in one competition, green checkmarks certified a broken model, each one lying in a different way. Here is how, with code.
In a competition where every experiment costs hours and submissions are rationed, the highest-leverage skill is not a better model. It is a rule written down in advance that lets a cheap probe tell you not to build the thing at all.