I spent a week building a search layer for my agent, measured it properly, and found out it makes the agent play worse the more I let it decide.
That is a failure. It is also, I think, the best-designed thing I did in this competition, because it cost me nothing to find out. No wasted submission, no rating lost, and a clean answer about exactly which part was broken.
The setup
At this point my live entry was the organizers’ own sample agent, a hand-written expert for one deck that had beaten my own search agent badly. It was the thing to beat, and it was also the thing I was standing on.
So the obvious idea: keep the expert, and add search on top. Let the expert propose a move, look ahead a few steps, and if search finds something clearly better, take that instead.
The obvious version of that design is search decides, expert is the fallback. I did not build that one, and not building it is the entire reason this entry has a happy ending.
The design that made being wrong free
Instead I put a threshold in front of it. Call it delta. The rule was:
Take the expert’s move, unless search believes some other move is better by at least delta.
At delta of infinity, search can never clear the bar, so the agent is the expert. Exactly the expert. Move for move.
That single property does three things, and they are worth separating because I only understood two of them at the time.
It is a provable floor. Whatever else happens, the worst version of my agent is the agent I already had. I verified it rather than assuming: across forty decisions, the search version disagreed with the expert zero times at delta infinity, and over six hundred benchmark games it scored 51.8% with a confidence interval bracketing fifty percent, which is what an identical agent must do.
It localizes the fault. If I sweep delta down from infinity toward zero, the only thing changing is how often I act on the search’s opinion. Not the code, not the determinizer, not the deadline handling. One variable.
And it made the experiment safe to lose. This is the part I underrated. Because delta infinity is the champion, an unfavourable result is not a setback, it is a measurement. I never had to risk the live entry to learn the answer.
The result
I ran the ladder of delta values, four hundred games each:
| delta | how often search overrules | win rate | verdict |
|---|---|---|---|
| infinity | 0% | 51.8% | tie, by construction |
| 200 | 1.9% | 49.5% | tie |
| 5 | 7.7% | 46.0% | tie |
| 1 | 26.9% | 49.5% | tie |
| 0 | 50.5% | 42.2% | significantly worse |
At delta zero, where search decides half of all decisions, the agent loses. Not marginally. That is a z-score of −3.10 against the champion, and it is monotone in override rate: the more I let search decide, the worse it plays.
There is no ambiguity to argue with here, and that is the design paying off. If I had built "search decides, expert is fallback," I would have had one number, no floor, and no way to tell a broken search from a correct search pointed at the wrong target.
What was actually wrong
The search worked. It determinized the hidden information correctly, it looked ahead correctly, it returned legal moves within the time budget. What it was searching toward was the problem.
A search needs something to evaluate a position with, and mine was a scoring function I had written in an afternoon: a prize is worth ten, a Pokémon in play one, an energy a tenth, a card in hand a twentieth. Those numbers were a guess dressed as a model.
And here is the uncomfortable property of search: it does not care that they were a guess. It optimizes hard toward whatever you give it. A weak agent with a wrong objective wanders near the right answer by accident. A strong search with a wrong objective marches confidently away from it. Adding search to a bad evaluation makes it worse, not better, and the better the search, the worse the effect.
I later found two other teams had hit the same wall in the same competition, one of them measuring a beam search that gained on a weak policy and lost on a strong one. Three teams, three forward-search implementations, none shipped. That is not a coincidence, it is the shape of the problem.
The mistake inside the experiment
I want to record one thing I got wrong, because it wasted real time and it is the sort of thing that is invisible in a tidy write-up.
I chose the rungs of the delta ladder before checking what the numbers meant. My scoring function ran on a scale where a single prize was worth ten and a whole game swing about sixty. So rungs of 200, 75 and 25 were all far above anything the evaluator could produce short of a terminal win.
All three rungs overrode the same four decisions out of 208. I ran three copies of one experiment, twenty minutes each, and read them as three data points.
Five minutes of printing the actual distribution of the quantity I was thresholding would have saved an hour of full-throttle compute. That is now a standing habit: before you threshold something, look at its distribution. A threshold is a claim about scale, and if you have not looked, it is a guess about a guess.
I also, twice, saturated every core on the machine without warning anyone, which was reported to me as "VS Code is using too much RAM." It was neither VS Code nor RAM. It was twenty of my worker processes. Both tools now carry a warning in their help text.
What I would keep
The result was negative and the method was the thing worth keeping.
Make the new system contain the old one as a limiting case. One parameter that reproduces the previous behaviour exactly gives you a floor you can prove, a fault you can localize, and permission to run the experiment at all. It is the same instinct as a feature flag, or a strangler-fig migration, or any change that can be turned off. Here it happened to be a number rather than a switch, but the shape is identical.
And it turns "we do not know" into "we measured, and no." A negative result you can trust is worth more than a positive result you cannot. I came out of that week without a better agent and with a fact worth having: my scoring function, not my search, was the thing standing between me and a better agent. Every piece of work after that pointed somewhere else, which is exactly what a good experiment is for.
Frequently asked questions
Why would adding search make a game AI play worse?
Because search optimizes toward whatever objective you hand it, and if that objective is wrong, searching harder pursues the wrong thing more efficiently. My evaluation function used weights I had guessed in an afternoon, and the deeper the search committed to them, the further it drifted from good play. A weak policy with a bad objective wanders near the right answer by accident; a strong search with a bad objective marches away from it. The fix is never more depth, it is a better objective.
How do you test a risky change without risking your live system?
Build the new behaviour so one parameter setting reproduces the old behaviour exactly, then verify that it does rather than assuming. I added a threshold where an infinite value meant the search could never override the existing agent, confirmed it produced identical moves across forty decisions and a benchmark result bracketing fifty percent, and only then swept the threshold down. That gives you a provable floor, isolates the single variable you are changing, and means an unfavourable result is a measurement rather than a loss.
What does it mean if a result is monotone across a sweep?
It is strong evidence the effect is real rather than noise. In my sweep the win rate fell steadily as the search was allowed to override more decisions, ending significantly below baseline at a z-score of −3.10. A single bad data point could be chance; a consistent trend across rungs where the only variable is how often you act on the thing under test is much harder to explain away. It also tells you the direction of causation, which a single comparison cannot.
How should you choose thresholds for a parameter sweep?
Look at the distribution of the quantity you are thresholding before you pick values, or derive them from something you know about the mechanism. I picked rungs by intuition and three of them turned out to sit far above anything my scoring function could produce, so they all overrode the same four decisions out of 208 and I ran three copies of one experiment. Printing the actual spread would have taken five minutes and saved an hour of compute. A threshold is a claim about scale, and if you have not measured the scale, it is a guess about a guess.
More in this series
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full PTCG AI Battle Challenge thread.
Written from the notes I kept while the work was happening, in the order it happened. You are seeing where I was, not where I am.
Pokémon and the Pokémon Trading Card Game are trademarks of Nintendo, Creatures Inc. and GAME FREAK inc. This is an independent write-up of a public competition and is not affiliated with or endorsed by The Pokémon Company or Kaggle.
