Every improvement I made in the back half of this competition was real, in the sense that the number went down. I moved from 7.623 to 7.618 to 7.604 to 7.547. Four submissions, four small gains, each one earned by a change I could explain, each one confirming that the work was going somewhere.
None of it meant anything.
Not because the metric was wrong. The metric was fine. Root mean squared error in feet, computed exactly the way the competition computed it, on data I trusted. If you had handed me two models a full ten feet apart in quality, that score would have told you which was better with near perfect reliability.
The problem was that my models were not ten feet apart. They were four tenths of a foot apart. And somewhere between those two scales, the metric quietly stopped being able to tell them apart at all, while continuing to produce numbers with three decimal places and an air of complete authority.
This is the most expensive thing I learned all year, and I want to be careful about how I say it, because it is easy to mistake for a point about noise and it is not.
The setup
The competition scores you on half the test data during the run, and on the other half at the end. The half you see is the public leaderboard. The half you do not see is the private one, and it is the only one that decides anything.
So all through the competition, you are using the public number as a proxy. You make a change, you submit, the public score moves, and you decide whether to keep the change. Every decision I made for two months ran through that loop.
The question everybody knows to ask is whether the gain is bigger than the noise. I asked it constantly. I had measured the rerun spread on my own model family, meaning what happens when you resubmit the exact same code and let the platform’s own randomness do its thing: somewhere between three and eleven hundredths of a foot. So a change worth seven or eight hundredths was, by that standard, real. Bigger than the noise band. Keep it.
That reasoning is correct and it is also not enough, and it took the competition ending for me to see why.
The question I was not asking
Beating the noise band tells you that A and B are different. It does not tell you that the metric knows which direction they differ in.
Those are separate properties, and I had never separated them. I had been treating "this measurement is precise enough to distinguish my candidates" as equivalent to "sorting my candidates by this measurement gives me the right order." The first was true. The second was not, and nothing about the first implies the second.
When the private board finally landed, I had nine submissions from the same model family with both scores attached. Public and private, side by side, for all nine. So I did the thing I should have done in June and computed the rank correlation between them.
from scipy.stats import spearmanr
rho, p = spearmanr(public_scores, private_scores)
print(rho, p)
Spearman’s rho on those nine came back at 0.117, with a p-value of 0.77.
Zero would mean the public ordering told you nothing whatsoever about the private ordering. One would mean it told you everything. I got 0.117 on a sample where the honest reading of a p-value of 0.77 is that this is indistinguishable from zero.
Two months of decisions, every one of them gated on a signal that was, at the resolution I cared about, a coin.
Nine points is not a lot
That was my first objection to my own finding, and it is the right objection. Nine submissions is a small sample. A rank correlation on nine points is a fragile thing, and if I had stopped there I would have been telling you a story built on a statistical accident.
So I went and got more points. The final leaderboard gave me the public and private score for every team in the competition, which meant I could compute this at field scale instead of on my own handful.
Three numbers came out of that, and together they are the actual finding.
Across the whole field, all six thousand-plus teams, rank correlation between public and private was 0.874. Very strong. The metric works. If you spread teams across the full range of quality, from the people who barely finished to the winners, the public board sorts them correctly almost every time.
Inside the public top hundred, it fell to 0.453. Still positive, meaningfully weaker.
And inside the narrow band where I had spent two months, a stretch about a tenth of a foot wide containing 278 teams, it was negative 0.135.
Not weak. Negative. Inside that band, across 278 independent teams, a better public score was very slightly associated with a worse private one. My nine points had not been a fluke or a small-sample artifact. They had been an accurate reading of a real, field-wide phenomenon that every single team in that band was subject to, and almost none of us knew.
What is actually going on
The metric never broke. It is measuring exactly what it claims to measure the whole way down.
What changes is the ratio between two things: how far apart your candidates really are, and how much the score wobbles for reasons that have nothing to do with quality. At ten feet of separation, the real difference dwarfs the wobble and the ordering survives. At four tenths of a foot, the wobble is the same size as the difference, and the ordering is the wobble.
There is a version of this you already know from photography. A lens has a resolving limit, and past it, two objects sitting closer together than that limit do not appear as two blurry objects. They appear as one perfectly sharp object that happens to be in the wrong place. The image does not look degraded. It looks fine. Nothing in the picture tells you that you have crossed the line where the picture stopped corresponding to what is out there.
My leaderboard score was that photograph. Sharp to three decimals, and past its resolving limit.
The tell I kept explaining away was that my local validation and the leaderboard sometimes disagreed about direction. A change would make things better at home and worse on the board. I wrote about that during the competition and concluded, reasonably, that my local setup was the thing at fault. It was, partly. But the other half of the answer is that once you are inside a band where the board itself has stopped ordering correctly, the board cannot referee that disagreement either. There was no arbiter. I had two noisy instruments arguing, and I kept assuming one of them was the judge.
The three lines that would have caught it
The check costs nothing. You need two scores for each of your own candidates that are supposed to measure the same underlying quality, and you need at least a handful of candidates. Then you ask whether sorting by one gives you the right order in the other.
from scipy.stats import spearmanr
# your own candidates, two measurements each:
# local_cv - what you optimize against day to day
# lb_score - the number that actually decides
rho, p = spearmanr(local_cv, lb_score)
print(f"rank correlation {rho:.3f} (p={p:.2f}) over {len(local_cv)} candidates")
if rho < 0.5:
print("your day-to-day metric is not ordering your candidates")
Run it in the range you actually work in, not across everything you have ever built. That distinction is the whole point. If you compute it over every model you have made since day one, including the bad early ones, you will get a reassuringly high number, exactly like the 0.874 I got field-wide, and it will tell you nothing about whether you can trust the decision you are making today between two models that are nearly identical.
You want the correlation among the candidates you are currently choosing between. That number grades your optimization loop.
And run it early. This is the part that stings. I had the ingredients for the nine-point version by roughly the halfway mark. I had multiple submissions with scores, I had scipy installed, and I had every reason in the world to want to know. I never ran it, because I did not know it was a question. The whole cost of finding out was three lines and a minute, and I paid two months instead.
What you do when the answer is bad
Suppose you run it and the number comes back low. Your metric has gone blind at your working resolution. Now what?
The unhelpful answer is "get a better metric," because usually you cannot. The competition decides the metric. Your production system decides the metric. You are not always in a position to change what you are scored on.
What you can change is what you make decisions with. Once I understood the shape of this, the useful move was to stop asking the score to rank things it cannot rank, and to start gating on properties that are structural rather than fine-grained.
Correlation between two models’ errors is one. Whether two approaches make different mistakes is a much coarser question than which of them is 0.05 feet better, and coarse questions are exactly the ones a blunt instrument can still answer. Whether a blend of two models wants a non-zero weight on the second one is another. Both of those held up for me all competition, while the fine-grained ranking was falling apart, because both are asking about structure instead of asking for a sort.
The general form: when your metric can no longer order candidates, stop trying to pick the best one and start trying to pick a different one. Diversity is measurable when quality is not.
That reframing came too late to change my result. It is the first thing I will set up next time.
Frequently asked questions
What is rank correlation and why use it for model selection?
Rank correlation, usually Spearman’s rho, measures whether sorting by one variable gives you the same order as sorting by another. It ignores the actual sizes and looks only at ordering, which is exactly what you want when the question is "if I pick the model that looks best by this measurement, am I picking the model that is best?" A value near 1 means your proxy orders candidates correctly, near 0 means it orders them randomly, and negative means it actively misleads you. It takes three lines with scipy.stats.spearmanr and it grades the entire loop you are using to make decisions.
Can a metric be valid overall but useless for choosing between similar models?
Yes, and this is the failure that cost me the most. In my competition, rank correlation between the visible score and the deciding score was 0.874 across the whole field, meaning the metric was strongly valid. Inside the narrow band where 278 teams including mine were actually competing, it was negative 0.135. The metric had not broken. The candidates had simply gotten closer together than the metric could resolve, so within that range the ordering was driven by noise rather than by quality. Global validity says nothing about local resolution.
How do you know if your improvement is real or just noise?
Checking that the gain exceeds the measurement’s noise band is necessary but not sufficient. It establishes that two results are different; it does not establish that the metric knows which one is better. Test both: measure the spread when you rerun an identical setup to get the noise floor, and compute the rank correlation among the candidates you are choosing between to confirm the metric still orders them. I passed the first test repeatedly and never ran the second, and the second was the one that mattered.
What should you optimize when the metric stops ranking your models?
Switch from fine-grained quality comparisons to structural properties, which stay measurable when precise ordering does not. Error correlation between two models tells you whether they fail in different ways. Whether an ensemble assigns meaningful weight to a second model tells you whether it contributes anything independent. Both are coarse questions, and a blunt instrument can still answer coarse questions reliably. The practical shift is to stop trying to identify the single best candidate and start trying to assemble candidates that fail differently from each other.
More in this series
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full ROGII Wellbore Geology Prediction thread.
- The validation score that lied to me for a month · the same disease one level down, from inside the competition.
The competition is over. This is the whole picture, including the parts that did not work.
