I called a folder agents.
That is the whole bug. Everything else follows from it, and if I had submitted without checking, my agent would have failed every single match on the Pokémon TCG ladder without ever making a decision. Not lost. Errored. A returned nothing, an invalid episode, a rating that decays while the code sits there being correct.
I caught it one step before submitting, and only because I had started testing the submission the way the platform loads it rather than the way my editor does.
The collision
The competition runs on kaggle_environments, the library that hosts Kaggle’s simulation competitions. That library ships a lot of environments, and one of them, for a completely unrelated game, contains a module called agents.
So there are now two things called agents: theirs, and mine.
Which one wins depends on the order of sys.path, and here is the line that decides it. The loader does the equivalent of:
sys.path.append(agent_directory) # append, not insert
Append. My directory goes on the end of the search path, behind everything already there. So when my code says from agents.hybrid import ..., Python walks the path in order, finds their agents first, and imports that. Theirs then fails on its own internal relative import, because it was never meant to be loaded from outside its own package.
The result is an exception before my agent does anything at all. Every episode. Every match.
One word in the standard library’s search order, append rather than insert, is the entire difference between my code running and my code never existing.
Why my tests could not see it
This is the part worth more than the bug, because the bug is specific to one library and the blindness is not.
I had a local harness that plays thousands of games. I had a test suite. Both were green. Neither could have caught this, and for two different reasons, which is what makes it worth writing down.
The harness never loaded the agent. It called the decision function directly, because that is faster and it was built to measure playing strength. Loading is a separate concern from playing, so a harness built to measure the second cannot see a failure in the first. It was not a weak test. It was a test of a different thing.
The test suite was poisoned by its own success. By the time any test that touched my package ran, agents was already sitting in sys.modules from an earlier import in the same process. Python does not re-resolve a name that is already cached. So my package won, every time, by accident of having been imported first in a process that the platform would never create.
Both of those are ordinary, sensible testing setups. Together they produced complete confidence about a submission that could not start.
The only test that would have found it
Nothing short of the real path works. Extract the actual submission archive, run it from a file path, in a fresh process, from an unrelated working directory:
from kaggle_environments import make
env = make('cabt', debug=True)
env.run([extracted_main_path, extracted_main_path]) # want DONE/DONE
A file path, not an import. A new process, so sys.modules is empty. A different directory, so nothing is on the path by accident.
I went further afterward and made the test hostile on purpose. It now spawns a subprocess that deliberately plants decoy modules named agents, utils and common earlier on the path, and then loads the submission. If any generically-named package of mine ever loses its name again, that test fails immediately rather than on the ladder.
One caution I learned the hard way when writing it: probe before you decoy. My first version injected a fake top-level agents module and "failed" a bundle that was actually fine, because I had manufactured a collision that did not exist in the real environment. A test that invents its own bug will happily fail good code. Check which names are really taken first, then decoy that.
The second time
What actually made me build this check was that it was the second bug of exactly this shape.
The first one had already cost me a submission. Kaggle does not import your agent as a module; it reads the source and executes it with empty globals, which means __file__ is undefined. My code used __file__ to find its card data. Locally, imported as a module, __file__ exists and everything works. On the ladder it raised before the first decision.
Different symptom, identical cause: the environment I was testing in was not the environment the code would run in. Once is bad luck. Twice is a category, and a category deserves a permanent test rather than another fix.
The fix for the collision itself was trivial. I renamed the package to something nobody else would ever use, and the risk went away permanently:
agents/ -> ptcg_agent/
Generic names are the vulnerability. agents, utils, common, models, config, helpers. Every one of those is a name somebody else has also used, and in any environment that puts your code on a shared path you are gambling that you got there first. A distinctive name costs nothing and removes the entire class.
What I actually took from it
The lesson is not "test in production" and it is not "be careful with names," though both are fine as far as they go.
It is that a test is only evidence about the conditions it runs under. My suite proved my code worked when imported into a warm process from my own directory. That is a true statement, and it is not the statement I needed. The platform loads from a file path, into a cold process, from somewhere else, with its own packages already present. None of those differences are exotic, and every one of them was load-bearing.
So before trusting a green suite on anything that will run somewhere else, the useful question is not do my tests pass. It is: what is different about the place this actually runs, and does anything I have run under those conditions?
For me the answer was no, twice, and the second time I built the test rather than the fix.
Frequently asked questions
Why does my Kaggle agent error on every episode but work locally?
Almost always because the platform loads your code differently than your tests do. Kaggle reads the agent source and executes it with empty globals, so __file__ is undefined, and it appends your directory to sys.path rather than prepending it, so a generically named package of yours can lose its name to one already shipped by the environment. Neither problem appears when you import the same file locally, because importing defines __file__ and your module gets cached first. Test by running the extracted submission from a file path in a fresh process.
Why did sys.path.append cause an import collision?
Because appending puts your directory at the end of the search order, behind everything already installed. If any package on that path shares a name with yours, Python finds theirs first and imports it instead. In my case kaggle_environments ships a module called agents for an unrelated environment, and my package was also called agents, so theirs won and then failed on its own internal relative import. Prepending would have made mine win; I do not control that line, so the fix was a distinctive package name.
Why don’t local tests catch agent loading bugs?
Because most local test setups never exercise loading at all. A self-play harness typically calls the decision function directly, which measures playing strength and says nothing about whether the module can be imported. And a test suite running in one long process will have cached your package in sys.modules from an earlier import, so your name wins by accident in a way it never would in the fresh process the platform creates. Both are reasonable setups that are structurally blind to this failure.
How do you test that a submission bundle loads correctly?
Reproduce the real conditions rather than approximating them. Extract the actual archive you are going to upload, run it from a file path instead of importing it, do it in a fresh subprocess so no modules are cached, and start from an unrelated working directory so nothing is found by accident. Going further, plant decoy modules with generic names earlier on the path so a future collision fails your test rather than the platform. Probe which names are really taken before decoying, or you will manufacture a collision that does not exist and fail code that is fine.
More in this series
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full PTCG AI Battle Challenge thread.
Written from the notes I kept while the work was happening, in the order it happened. You are seeing where I was, not where I am.
Pokémon and the Pokémon Trading Card Game are trademarks of Nintendo, Creatures Inc. and GAME FREAK inc. This is an independent write-up of a public competition and is not affiliated with or endorsed by The Pokémon Company or Kaggle.
