I was seven when Pokémon started.
I have owned everything since. The first games, the ones after those, the ones after those, all the way through to what is on shelves now. Cards, consoles, the cartoons on a Saturday morning. I am thirty-seven, which means I have been part of Pokémon’s life for about as long as it has been part of mine. That is not a small thing to say about a piece of media, and I do not think I am unusual for saying it. A lot of us are in the same position.
Then, recently, everything changed, in the way everything has recently changed. Artificial intelligence arrived in the middle of all of it.
So I want to introduce you to something: The Pokémon Company’s Trading Card Game AI Battle Challenge. I entered it. Over the next few weeks I am going to write about what happened.
What the competition actually is
The Pokémon Company put the Trading Card Game up as an artificial intelligence problem, in partnership with Kaggle, with HEROZ and the Matsuo Institute, and with support from Google, Google Cloud and NVIDIA.
There are two tracks and they are connected.
The first is a simulation ladder. You submit a program that plays the card game. It receives the state of the board and returns the moves it wants to make. Then it plays continuously, around the clock, against every other entrant’s program, and a skill rating moves up or down with the results. There is no fixed test to pass. Your score is a number produced by thousands of games against opponents you never see and cannot choose. Nearly seven thousand teams entered that one.
The second is a strategy track, judged on a written report rather than a rating: what you built, why, and whether the reasoning holds up. That is where the money is, around two hundred and forty thousand dollars of it. Three hundred and thirty-six teams entered that one, which is a detail worth noticing on its own.
The card pool is roughly two thousand cards from the current Standard format. Not a toy subset. The real game.
Why this is harder than chess
This is the part I would want explained to me, so let me do it properly.
When a computer beat the world chess champion in 1997, and when one beat the best Go players two decades later, both of those games shared something that made them tractable. Everything is on the board. Both players see the entire position at all times. There is no luck. If you and I sit down at the same chess position, we are looking at exactly the same problem, and a machine that can search far enough ahead will eventually see further than you can.
The Pokémon Trading Card Game breaks all three of those.
You cannot see your opponent’s hand. You know how many cards are in it. You do not know which ones. Everything you decide is a decision made against a range of possibilities rather than a fact.
The draws are random. Two identical decks played by two identical players produce different games. A move that was correct is not guaranteed to work, and a move that was wrong sometimes wins anyway. That is an awkward property for a machine that learns by association, because the feedback it gets is noisy about the very thing it is trying to learn.
And the board keeps changing shape. Pokémon evolve. Cards attach to other cards. Abilities alter what the rules even are for a turn. The set of legal moves is not fixed the way it is in chess, where a knight moves like a knight forever. Here, what you are allowed to do depends on a board that rewrites itself as you play.
Games with hidden information and randomness are where the field is now. Poker was the famous one. Card games with a two-thousand-card pool and evolving rules are a considerably wider version of the same problem, and there is no reason to think the technique that solved one solves the other.
What it defines, and what it limits
I want to be careful here, because the temptation with a competition like this is to make it sound like more than it is, and I would rather it sound like exactly what it is.
What it defines is a public, measurable benchmark for a class of problem that matters outside games. Deciding under incomplete information, where the world is partly random and the rules shift underneath you, is not a card game problem. It is most real problems. Having a shared arena with a scoreboard is how a field finds out which ideas actually work, as opposed to which ideas sound good in a paper.
The limitations are real too, and I ran into them.
The judging environment gives you two processor cores and about twelve gigabytes of memory. That sounds generous until you remember the thing has to decide in real time inside a ten-minute match. A lot of what works in a research paper assumes a machine that is not that.
There is no way to replay a game. The simulator has no seed you can set, which means you cannot run the same game twice and see what a different choice would have done. Every comparison you make has to be statistical, over hundreds or thousands of games, because you can never isolate a single one. That single constraint shaped more of my work than any other.
And the ladder is slow relative to the thing you want to measure. Real improvements are small. Distinguishing a truly better agent from a slightly luckier one takes more games than the clock allows. I will come back to that, because getting it wrong cost me weeks.
Why I think it matters
We have all grown up with the cartoons. Most of us have played the video games. Plenty of us have shuffled a real deck.
This is the first time I have been able to watch an artificial intelligence sit down and play the game in front of me.
Not a scripted opponent from a video game, which is a set of rules somebody wrote to be beatable. Something that was given the rules and the cards and had to work out the rest, playing against thousands of other attempts at the same thing, with a number at the end saying which of them understood it better.
I have done this before with other games, most recently teaching a machine to play something it had never seen. This one is different, because I already know the game.
That is a strange thing to see happen to something you grew up with. Not bad. Strange. And worth writing down while it is happening rather than afterward, when it has become obvious.
What you are going to see
Over the next few weeks I am going to publish what actually happened, in the order it happened.
You will see how I started, and what I thought this problem was before I understood it. You will see the agent I built carefully and correctly, which then lost to something far simpler than itself. You will see a measurement I trusted for three weeks that was quietly wrong the entire time, and a bug that made a month of tests pass while proving nothing at all.
You will also see the things that worked, because some of them did, and the reasoning that got there.
I am writing these from the notes I kept while the work was going on, so they are not a tidied-up version assembled after the fact. Some of them record me being confidently wrong, and then record the day I found out. I have left that in, because the moment where a belief breaks is the only part of this that actually teaches you anything, and it is the part most write-ups quietly delete.
You are seeing where I was, not where I am. More than anything, you are seeing the problems as I hit them, and the solutions I came up with at the same time.
Let me show you what happened.
Frequently asked questions
What is the Pokémon TCG AI Battle Challenge?
It is an artificial intelligence competition run by The Pokémon Company with Kaggle, in partnership with HEROZ and the Matsuo Institute and with support from Google, Google Cloud and NVIDIA. Entrants build programs that play the Pokémon Trading Card Game using roughly two thousand cards from the Standard format. There are two connected tracks: a simulation ladder where agents play continuously against each other for a skill rating, and a strategy track judged on a written report explaining the approach, which carries around $240,000 in prizes.
Is this the same as the ARC-AGI 3 challenge?
No, and the difference is the interesting part. They are separate competitions with different organizers, and they test close to opposite things. ARC-AGI 3 puts an agent into games it has never seen before and measures whether it can work out the rules from scratch, so the whole subject is generalizing to novelty. The Pokémon TCG challenge does the reverse: everybody knows the game, the roughly two thousand cards are public, and nobody is hiding the rules from you. The difficulty is that you cannot see your opponent’s hand, the draws are random, and you are being scored against thousands of other agents that are also improving. One asks whether a machine can learn an unfamiliar game. The other asks whether it can play a familiar one well when the information is incomplete and the opponent is real. I have competed in both, and my ARC-AGI 3 entries are here if you want the contrast.
Why is the Pokémon Trading Card Game hard for AI compared to chess or Go?
Chess and Go are perfect-information games with no randomness, so both players always see the entire position and a sufficiently deep search will eventually outplay a human. The Pokémon TCG breaks that in three ways at once: you cannot see your opponent’s hand, card draws are random so identical decisions produce different outcomes, and the board itself changes shape as Pokémon evolve and abilities alter what moves are even legal. That combination of hidden information, stochasticity and a shifting action space is much closer to real-world decision-making than a chessboard is.
What are the practical constraints on building an agent for it?
The judging environment provides about two processor cores and twelve gigabytes of memory, with a ten-minute cap per match, so approaches that assume large amounts of compute are not available. More restrictive still, the simulator has no seed hook, meaning you cannot replay a game deterministically or isolate the effect of a single decision. Every comparison has to be made statistically across hundreds or thousands of games, which sets a floor on how small an improvement you can reliably detect.
Can you learn anything from a competition you did not win?
The measurable failures are usually the transferable part. A competition supplies something ordinary projects rarely do: an objective external scoreboard that will contradict you. Across this one I found a broken measurement that had misreported my standing for three weeks, a test suite that passed for a month while validating nothing, and several confident conclusions that collapsed when I finally built the right control. None of those depend on the final ranking to be useful, and all of them are the kind of mistake that repeats outside games.
More in this series
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full PTCG AI Battle Challenge thread.
Written from the notes I kept while the work was happening, in the order it happened. You are seeing where I was, not where I am.
Pokémon and the Pokémon Trading Card Game are trademarks of Nintendo, Creatures Inc. and GAME FREAK inc. This is an independent write-up of a public competition and is not affiliated with or endorsed by The Pokémon Company or Kaggle.
