The first agent I built for this competition beat every opponent I had.
It beat a bot that picks randomly. It beat a bot that always takes the first legal option. It beat my own earlier, simpler version eighty percent of the time. Every number on my screen said the thing was working.
Then I pointed it at the sample agent the competition organizers had published, a four-hundred-line hand-written thing for one specific deck, and lost three games out of four.
Nothing about my agent had changed between those two measurements. The only thing that changed was who it was playing.
What the competition is
This is the second entry in my log of the Pokémon TCG AI Battle Challenge. Kaggle is running a Pokémon Trading Card Game simulator. You submit a Python function that receives the game state and returns a list of integers, the indices of the options you want. It plays continuously against other people’s submissions on a skill ladder, and your rating moves with the results.
That framing matters more than it sounds. There is no fixed test set. Your score is a function of a population of opponents you cannot see, most of whom are also changing. The thing you are trying to be good at is defined entirely by other people.
I did not think about that at all for the first three weeks.
The thing I built
I built it properly, and I want to be specific about that, because "I built it badly" would be a more comfortable explanation than the real one.
There is a safety floor that filters the engine’s legal moves, so a crash or an illegal return can never cost a match. There is a card database loader over the real 1,267-card export, which caught three genuine schema surprises on the way in. There is a self-play harness so nothing gets submitted without being measured first. There is an evaluator that scores a position by prize race, and a one-ply lookahead that uses it, and then a turn-aware search on top of that which handles the fact that a Pokémon TCG turn is a sequence of decisions rather than one move.
That last piece was a real improvement and I could prove it. The turn-aware search beat the one-ply version eighty percent of the time, because one-ply is blind to combinations: attach an energy then attack, and the second decision is only good because the first one happened.
So: layered, tested, each layer beating the one below it on measured evidence. This is what good engineering looks like from inside.
The measurement that was actually happening
Here is the opponent pool I was scoring against, in full:
- a bot that picks a legal option at random
- a bot that always picks option zero
- previous versions of my own agent
Every one of those is either trivial or me.
The random bot has no strategy to defeat, so beating it measures whether I have any strategy. Option-zero is worse than it sounds and in one specific way that took me weeks to appreciate: on my deck it ties random. It is not a floor, it is a coin.
And my own previous versions share every assumption I have. When my search beats my greedy agent eighty percent of the time, I have learned that the search is better at the game as I have modeled it. If my model of what matters is wrong, both agents are wrong in the same direction, and the comparison cannot see it. A mirror match is not an experiment. It is a reflection.
What I had built was a very good measurement of one thing: whether my newest code beat my previous code. What I believed I had built was a measurement of whether my agent was good.
The four-hundred-line bot
The organizers publish sample agents. One of them plays Dragapult ex, and it is not clever in any way I would have recognized as clever. It is a long list of hand-written rules about one deck. It knows that this specific card should be attached before that specific attack. It counts prizes. It has a hardcoded number that says a position is worth minus twelve hundred when the opponent is two prizes from winning, and plus twelve hundred when they are zero.
There is no search in it. No lookahead, no model, no evaluation function in the sense I would have used the term. It is somebody who knows the deck writing down what they know.
It beat my search three games in four.
My first instinct was that I had a bug, which is the reasonable first instinct and was wrong. The agent worked. It searched correctly, evaluated correctly, and returned legal moves. It was simply playing a worse game than a list of rules written by someone who understood the deck.
That distinction is worth holding onto. General machinery, applied to a domain you have not modeled well, loses to specific knowledge. My search was optimizing hard toward an objective I had written in an afternoon: prizes weighted ten, cards in play one, energy a tenth, hand a twentieth. Those weights were a guess. Search does not fix a wrong objective, it pursues it more efficiently.
The submission that never ran
Before any of that, I had to actually get a submission working, and the first one failed in a way I want to record because it is the same shape as almost everything else in this project.
My agent errored on every episode. Locally it was fine. Every test passed.
Kaggle does not import your agent as a module. It reads the source and runs it with empty globals, which means __file__ does not exist. My code used __file__ to locate its card data. Locally, importing it as a module, __file__ is defined and everything works. On the ladder it raises before the first decision and every match is a loss.
The environment I was testing in was not the environment the code would run in. My tests were not wrong; they were answering a question I had not realized was different from the one that mattered. The fix took a minute. Finding it took a submission and a day.
The check that catches this is running your agent the way the platform actually runs it:
from kaggle_environments import make
env = make('cabt', debug=True)
env.run([path_to_agent_file, path_to_agent_file]) # want DONE/DONE, not ERROR
That is a file path, in a fresh process, not an import. It is the only version of the test that was ever going to find the bug.
What the replays said
The other thing I did that week, and the only one that produced real information, was download the replays of games my agent had actually played on the ladder and read them.
Two things fell out immediately, and neither had shown up in any local metric.
My agent was decking itself out. Running out of cards in your own deck is a loss condition, and mine was walking into it. And it was taking almost no prizes, meaning it was not pursuing the actual win condition, it was just surviving.
Neither of those is visible in a mirror match, because when both players share a flaw, the flaw cancels. Both agents deck out at the same rate. Both take no prizes. The win rate comes out near fifty percent and tells you nothing is wrong.
Real opponents do not share your blind spots. That is the entire reason they are worth measuring against.
What I actually did
I fielded the sample agent.
Not my search. Not a hybrid. I took the organizers’ four-hundred-line Dragapult bot, verified it byte-for-byte, and submitted it as my entry. My rating went from six hundred to roughly seven hundred and nine, and it kept climbing, which meant it was beating the field.
That was the correct call and it felt terrible, and I think the discomfort is worth naming rather than skipping past. Three weeks of building, and the thing that moved the number was recognising that somebody else’s work was better than mine and standing on it. The engineering was not wasted, the harness and the safety floor and the card loader are all still load-bearing today, but the agent itself was not the asset I thought it was.
There is a version of that decision I got wrong, though, and it took six weeks to surface. I wrote down the wrong lesson. What I recorded was roughly only adopting someone else’s agent has ever raised our rating, and I let that harden into strategy. It was true for six weeks, and then a pilot I built myself beat it, and the honest version turned out to be narrower: the builds that lost were the ones driven by intuition, and the one that won was driven by a measured brief. But that is a later entry.
The lesson I keep
An evaluation is only as informative as the strongest opponent in it.
Not the average. The strongest. Because what you are trying to learn is whether your agent is good, and every opponent weaker than the thing you actually face contributes nothing to that question no matter how many games you play against it. I had thousands of games of evidence and a pool with a ceiling somewhere far below my real competition.
The practical form is a question to ask before trusting any comparison: what is the best thing in my test set, and is it better than me? If the answer is no, the number you are looking at measures the test set, not the model.
I ran into a version of this again in a different competition, where my local validation and the official scoreboard disagreed for a month because they were computing different statistics. Same disease, different symptom.
I have since spent a lot of this competition building better opponents, and every one of them has been worth more than an equivalent amount of time spent on the agent.
Frequently asked questions
Why isn’t self-play enough to evaluate a game AI?
Self-play measures your agent against its own assumptions. If your model of the game is wrong, both sides of the mirror are wrong in the same direction and the error cancels rather than showing up. In my case both players were decking themselves out and taking almost no prizes, so the win rate sat near fifty percent and looked healthy while two serious flaws went undetected. Self-play tells you reliably whether version two beats version one. It cannot tell you whether either is any good.
What makes a good baseline opponent?
One that is strong enough to lose to, and that does not share your blind spots. A random policy and an always-pick-the-first-option policy sound like sensible floors, but they measure only whether you have any strategy at all, and their strength is domain-dependent in ways that surprise you: on my deck, always picking the first option tied random rather than losing to it. The most valuable opponent I had was a hand-written expert published by the organizers, because it was written by someone who understood the domain differently than I did.
Why did a rule-based agent beat a search agent?
Search does not fix a wrong objective, it pursues it more efficiently. My search evaluated positions with weights I had guessed in an afternoon, and it optimized hard toward that guess. The rule-based agent encoded actual knowledge of one specific deck, including which cards combo and what a position is worth near the end of a prize race. When a general method is applied to a domain you have modeled badly, specific knowledge wins, and adding more search depth makes it worse rather than better.
Why does an agent pass local tests and fail on the platform?
Because the platform probably loads it differently than your tests do. Kaggle reads the agent source and executes it with empty globals, so __file__ is undefined, and any code that uses it to find data files raises before the first decision. Importing the same file as a module locally defines __file__ and hides the problem completely. The fix is to test through the real path: build the submission, then run it from a file path in a fresh process using the platform’s own environment rather than importing it.
More in this series
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full PTCG AI Battle Challenge thread.
Written from the notes I kept while the work was happening, in the order it happened. You are reading where I was, not where I am.
Pokémon and the Pokémon Trading Card Game are trademarks of Nintendo, Creatures Inc. and GAME FREAK inc. This is an independent write-up of a public competition and is not affiliated with or endorsed by The Pokémon Company or Kaggle.
