Builder Journal · Hyperspectral Object Tracking Challenge 2026
The win that was a coin flip
There is a kind of gambler who wins once, decides he has a system, and spends the rest of the night giving it all back. A parameter sweep tried very hard to turn me into that gambler this week. It almost worked.
This is the fifth entry in a log from inside the Hyperspectral Object Tracking Challenge 2026. The ground rule of how I work here: I measure every change on a small home test set before I spend one of my rationed leaderboard submissions. This is the story of a home result I was right to throw away.
The tempting number
I swept a single setting across a range of values, hunting for a better spot than the one currently on the board. One value lit up. On my dev scenes it posted the best score I had seen, a clear jump over what I was running. Every instinct said send it.
I made myself look closer before I did, and three things were wrong with the win.
Three red flags
The curve was the wrong shape. Turn a real dial and the response moves smoothly: a little better, then best, then a little worse. Mine was flat, then a sudden spike at one value, then flat again. Smooth knobs do not produce lone spikes. A spike is usually noise wearing a knob’s clothing.
One scene was the entire story. A single tiny target, about a hundred pixels across, was bistable. Depending on the setting it either locked on cleanly or lost the plot completely, and its score swung wildly between those two fates. The "win" was that one scene happening to land on the lucky side of its own knife-edge at that one value. Luck, written down as a result.
The math was damning. That one scene’s swing was larger than the entire gap between the winning value and the one I was already using. Strip that scene out and, across the other nine, the new value was collectively worse. The win was not merely fragile. It was negative everywhere except the slot machine.
The boring, correct move
So I did the dull thing. I kept the value I already had and did not spend the submission. A scarce measurement is far too expensive to burn confirming a coin that came up heads once.
What it actually means
The danger of a small test set is not that it is wrong. It is that it is occasionally, seductively, right for the wrong reason. The number is real. The conclusion is a lie. And the gap between those two is where competitions are quietly lost.
The skill is not running the sweep. Anyone can run the sweep. The skill is refusing the answer it hands you when that answer is a single lucky scene in a trench coat. Most of competitive machine learning, it turns out, is the unglamorous discipline of declining to fool yourself.
Where I’m standing right now
The same number on the board as last time, and that is the point. Sitting just under the podium line, with one fewer way to trip over my own feet.
Next entry: the contribution I was certain would be the centerpiece, and what killing it taught me.
More in this series
- ← Start here: I entered a competition to track objects in light you can’t see — what the challenge is, and why it’s hard.
- My own test set lied to me — the sibling lesson on a small eval that’s seductively right for the wrong reason.
- The full Hyperspectral Object Tracking Challenge thread — every entry from this competition.
- The Builder Journal — the live log across all my competitions.
This is part of an ongoing builder’s log written from inside live competitions. You’re reading where I was, not where I am.
Frequently asked questions
How do you tell a real improvement from noise?
Look at the shape of the response rather than just the best number in it. Turn a real dial and the result moves smoothly: a little better, then best, then a little worse. Mine was flat, then a sudden spike at one value, then flat again, and smooth knobs do not produce lone spikes. A spike is usually noise wearing a knob’s clothing.
What are the warning signs that a parameter sweep found noise?
Three showed up together for me. The curve was the wrong shape, a lone spike rather than a smooth response. One scene carried the entire story, a small bistable target that either locked on cleanly or lost the plot depending on the setting. And the arithmetic was damning: that single scene’s swing was larger than the whole gap between the winning value and the one I was already using.
Should you ship a new best score you do not trust?
No, and declining to is most of competitive machine learning. Strip that one lucky scene out of my result and, across the other nine, the new value was collectively worse. The win was not merely fragile. It was negative everywhere except the slot machine.
Why is a small evaluation set dangerous?
Because a single unstable case can dominate the aggregate. With few enough scenes, one target that swings between two fates carries more weight than any genuine effect you are trying to measure, and a sweep will cheerfully find the setting where it lands lucky and hand it to you as a result.
