For three weeks I believed my agent was rated around 709.
It was 784.9. I was ranked 1,267th out of 4,666, and the cutoff for the prize positions was 1,106, which meant the gap I was actually trying to close was different from the gap I thought I was closing, in a direction that changed what I should have been working on.
The number had been unavailable the whole time. My own tooling had been handing me nothing, and somewhere in my notes a plausible figure had filled the space and then hardened into a fact.
This is the Pokémon TCG AI Battle Challenge, where you submit an agent that plays the card game continuously against everyone else’s on a live ladder. The rating is the entire feedback signal. It is the only thing that tells you whether the last three weeks were worth anything.
The typo
I had written a small command to check my standing. It asked Kaggle for my submission and read the score off it, roughly like this:
score = getattr(submission, "publicScore", None)
The field is called public_score. Snake case, not camel case.
There is no error in that line. getattr with a default is doing exactly what it was told: look for a thing, and if it is not there, hand back None. So my status command printed score=None, every time, for three weeks.
And I looked at score=None a number of times and thought the ladder must not have updated yet.
Why I did not chase it
I want to be honest about this rather than presenting it as an obvious slip, because the reason I ignored it is the interesting part.
None did not look like a bug. It looked like an absence, and absences have innocent explanations. The submission was still being evaluated. The ladder updates on a delay. Not enough games had been played yet. Every one of those was plausible, none of them required action, and all of them let me get back to the work I actually wanted to do.
A crash would have taken ninety seconds to fix. An exception has a stack trace pointing at the line, and it stops you, which is the entire value of it. What I got instead was a quiet, well-formed, syntactically valid nothing, and quiet nothing is survivable. You can build on top of it for weeks.
Then, because I still needed a number to think with, I used the last one I had seen from somewhere else and carried it forward. That number was 709. I have no memory of deciding to trust it. It was in my notes, so it was true.
What the real numbers changed
When I fixed the accessor and looked properly, three things came apart at once.
My rating was 784.9, not 709. Better than I thought, which is a strange thing to be annoyed about, but it moved the target. The distance to where I wanted to be was smaller and the field around me was different.
My earlier agents had scored worse than I believed. My own general-purpose search agents had settled at 299.2 and 350.6. My notes said roughly 420. So when I had abandoned my own work in favour of the organizers’ sample agent, that decision had been worth about 434 rating points, not the ~290 I had recorded. I had been undercounting the single most consequential thing I had done, which meant I had also been undercounting the lesson attached to it.
And a number I had never questioned turned out to be off by three orders of magnitude. I had assumed running a full game was expensive, so my local testing used about forty games per comparison, which produces error bars around fifteen percent. Fifteen percent error bars cannot detect anything a competition like this is decided by. That is why an entire earlier session of tuning had "found nothing." It was not that nothing was there, it was that I could not see it.
A full game actually costs around fifty-two milliseconds. Two thousand games take under ten seconds and give error bars of about two percent.
That single fact rearranged the whole project. Testing went from a thing you did sparingly, on a hunch, to a thing you did before forming the hunch. Almost every good decision I made afterward depended on it, and I had been sitting on it for three weeks because I had never measured what I assumed was expensive.
The shape of this failure
The thing I keep coming back to is that this was not a hard bug. It was one word. What made it expensive was the class of failure it belonged to.
A metric that breaks tells you it broke. A metric that goes quiet asks you to explain its silence, and you will, because you are the one holding the explanation. Every story I told myself about None was reasonable. Reasonable is exactly the problem: it costs nothing to believe, so nothing forces the check.
So the rule I took out of it, which I have used repeatedly since:
When a measurement goes quiet, suspect the thermometer before the patient.
And structurally, the fix is to stop writing code that can go quiet:
# Don't do this. A typo becomes a plausible value.
score = getattr(submission, "publicScore", None)
# Do this. A typo becomes an exception.
score = submission.public_score
getattr with a default is for fields that are truly optional. Using it on a field you require converts a spelling mistake into data. Same for dict.get(), same for try/except around a parse, same for any fallback that returns something harmless-looking when it should be shouting.
There is a version of this rule I picked up later in the same project and it sharpens it: fall back to another source, never to emptiness. If the value can come from two places, try both. If it can only come from one and it is not there, raise. An empty default is only ever appropriate when you would actually be happy with empty, and if you would be happy with empty you probably did not need the field.
The part that still bothers me
I did not lose three weeks. The work I did in those three weeks was mostly fine, and some of it is still in the project.
What I lost was three weeks of steering. Every decision in that window was made against a number that was not real, so the ones that happened to be right were right by luck. I cannot tell you which of them were which, and that is the actual cost.
Since then I check the instrument first. Not because I am more careful, but because I now know what it feels like when the instrument is wrong, and it feels exactly like everything being fine.
Frequently asked questions
Why does the Kaggle API return None for my submission score?
Most likely you are reading the wrong attribute name. The Python SDK exposes the field as public_score in snake case, and asking for publicScore returns nothing. Because most code reads it with getattr(obj, name, None) or a dictionary .get(), a misspelled field name does not raise, it quietly produces None, which is easy to mistake for a score that has not been computed yet. Read the attribute directly so a typo becomes an exception instead of a plausible value.
Why is a metric that returns None worse than one that crashes?
Because a crash stops you and points at the line, while a silent absence invites you to explain it. Absences have innocent explanations, and you will supply one: the job has not finished, the update is delayed, not enough data yet. All of those are reasonable, none requires action, so nothing forces the check. In my case I looked at an empty score repeatedly over three weeks and reasoned it away every time, then filled the gap with a number from my own notes that I stopped questioning.
How many games do you need to compare two game-playing agents?
Enough that your error bars are smaller than the effect you are trying to detect, which is usually far more than people assume. I was running about forty games per comparison, which gives roughly fifteen percent error bars and cannot resolve anything a close competition is decided by. When I actually measured the cost of a game it was about fifty-two milliseconds, so two thousand games took under ten seconds and cut the error to around two percent. Measure how expensive your evaluation really is before deciding you cannot afford a bigger sample.
What should you use instead of a default value when reading a required field?
Raise. A default is appropriate for truly optional data, and a fallback to a second source is fine when the value can legitimately arrive from more than one place. What you want to avoid is falling back to emptiness on a field you require, because that converts a mistake into data that flows downstream looking normal. The question to ask when writing any default is simple: if this fires unexpectedly, will anything notice? If the answer is no, do not write it.
More in this series
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full PTCG AI Battle Challenge thread.
Written from the notes I kept while the work was happening, in the order it happened. You are seeing where I was, not where I am.
Pokémon and the Pokémon Trading Card Game are trademarks of Nintendo, Creatures Inc. and GAME FREAK inc. This is an independent write-up of a public competition and is not affiliated with or endorsed by The Pokémon Company or Kaggle.
