Here is the number I have been holding back for three entries.
Final error, 9.598 feet. Rank roughly 2,692 out of 6,125 teams, which is the top 44 percent. The bronze medal cutoff was 9.164.
I missed a medal by 0.434 feet.
Four months. Sixteen banked negative results, five model families, a fork of the public state of the art, a foundation model evaluated and rejected, a validation harness rebuilt twice. Four tenths of a foot.
I have been dreading writing this entry and looking forward to it in roughly equal measure, because the honest accounting is worse than "I came up short" and also more useful.
What the second half of the competition was worth
On the tenth of June I made the single best decision of the project. The public leaderboard had moved a long way past me, and rather than defend my own architecture out of pride, I forked the leading public approach and shipped it. It was worth about five feet in one submission. Everything before that moment is a different competition than everything after it.
So take everything after it. Every feature I bolted on, every retrain, every recombination, every careful evaluation, the entire back half of the project.
All of it moved my final score by 0.345 feet.
Set that against the gap to a medal, 0.434 feet.
The sum of two months of work is smaller than the distance I needed to close. Not smaller by a rounding error. Comparable in size, and on the wrong side.
I want to resist the clean version of that, because the clean version is wrong in a specific way. The tempting read is "nothing in the second half could have mattered, it was all noise." It is tempting because the previous entry in this series established that my metric had stopped ranking models at that resolution, and it is emotionally convenient, because if nothing could have mattered then nothing was wasted.
But 0.345 and 0.434 are the same order of magnitude. One real improvement, one genuine half-foot gain, would have put me on the board. The work was not too small to matter. The work was the wrong shape, and I did the same wrong-shaped thing about fifteen times.
Every single thing I tried in that second half was a recombination of a fixed set of features. New model on the same features. Different blend of the same estimators. Same table, rearranged. I was searching hard inside a box and never once checked whether the box contained the answer.
The afternoon I never spent
There is a technique that would have told me. It is not clever, it does not involve a model, and it cannot overfit, which is exactly why it is easy to skip in favour of something more interesting.
Decompose your target into its component parts. Then, one at a time, replace a component with its true value from the labeled data, leave everything else as your model has it, and score the result.
That is it. You are asking: if this one piece were perfect, how much better would I be?
Do it for each component and you get a short list of numbers. Those numbers are an upper bound on what improving each piece can ever be worth. Call it an oracle ladder, because you are handing your model an oracle for one part at a time and watching what it does with it.
# For each component, substitute the ground truth and rescore.
for name, component in components.items():
patched = prediction.copy()
patched[name] = truth[name] # this part is now perfect
print(f"{name:20s} perfect -> {score(patched, truth):.3f}")
print(f"{'nothing perfect':20s} -> {score(prediction, truth):.3f}")
An afternoon. Maybe less. No training, no tuning, no submissions burned.
I never ran it. And the reason I never ran it is not that I was lazy or in a hurry. It is that it does not feel like progress. It produces no model, improves no score, and ships nothing. It only tells you where to point, and pointing feels like something you do when you are stuck rather than something you do while things are going fine.
Things were going fine. I was shipping. The score was moving. And I was pouring two months into a component that a perfect solution would barely have improved, while the actual bottleneck sat there unexamined the whole time.
An ablation, which I did run, tells you what a part contributes right now. An oracle tells you what a perfect part would contribute. Only the second one bounds your effort, and I only ever ran the first.
The exit was in a sentence I wrote myself
There is a moment in this project I keep returning to.
Partway through, I closed off an entire family of approaches. Five different implementations of the same underlying idea, all of them tried, all of them worse than what I already had. I wrote it up cleanly: this family is closed, here is the evidence, move on. It felt like good practice. It was good practice, in form.
A team finished twenty-sixth in this competition using that family. Just that family, essentially on its own, roughly eight feet better than my best attempt at it.
So the family was not closed. My five implementations were closed, and I had written a sentence about the space when I had only earned a sentence about my search.
The difference between those two things is where the whole competition was hiding. Because when I went back and asked what all five of my failed attempts had in common, the answer was immediate and it was the same in every one: they all tried to estimate a path, a value at every single step along the well. The teams that made it work estimated a single number for each well, one offset, fitted robustly.
One number, pooled across all the data in a well. Not a decision at every step, re-litigated hundreds of times, with hundreds of opportunities to drift.
The measurements I needed to see this were in my own notes, written four separate times in different words, never assembled. And the components to build the scalar version already existed in my codebase, tested, since the first week of June. It was not a research problem. It was a four-line objective function over parts I already owned, and I never wrote it because I had already declared the neighbourhood empty.
When you bank a family-level negative, write down what every attempt shared. That shared assumption is not a footnote. It is the location of the exit.
What I would keep
Four entries of things that went wrong earns a paragraph on what did not.
Pre-registering the decision rule before running an experiment worked, consistently. Deciding in advance what result would kill an idea, then running it and honouring the answer, killed about sixteen ideas for roughly an hour of compute each and zero submissions. Nearly all of those were correct kills.
Keeping the infrastructure when banking a negative was right. Every failed idea left behind a tested module, and the pieces I needed for the approach I never took were sitting in that pile, ready.
Verifying state through the API instead of trusting the interface caught something real, and I would do it every time.
And choosing final submissions by how differently they fail, rather than by which scored best, was the right principle even though I applied it to the wrong pair. The silver medalist’s own write-up lists picking finals by public ranking as a mistake they made. I got the principle right and the execution wrong, which is a better place to be starting from than the reverse.
The actual lesson
If I ran this again, the order would be inverted.
I spent week one building. The build was competent, the code was tested, the harness was real. What I did not spend was an afternoon finding out which part of the problem a perfect solution would actually fix.
Everything downstream of that omission was well-executed work aimed at the wrong target, and no amount of rigor further down corrects for it. The metric analysis, the hedging strategy, the careful endgame verification, all of it was real work and all of it was arranged around a bottleneck I had assumed rather than measured.
Four tenths of a foot. Two months of effort worth three tenths. One afternoon, never spent, that would have told me where the other half foot was.
Before you build the interesting model, measure which part of the error a perfect solution to each sub-problem would remove.
That is the entire lesson, it fits in a sentence, and it cost me a medal to learn.
Frequently asked questions
What is an oracle ladder in machine learning?
It is a diagnostic that bounds where your headroom is before you commit effort. Break your prediction into components, then one at a time substitute a component’s true value from labeled data while leaving everything else as your model produced it, and rescore. Each result tells you the maximum possible gain from perfecting that component. It requires no training and cannot overfit, because you are not fitting anything, and it usually takes an afternoon. The output is a short ranked list of where your remaining error actually lives.
What is the difference between an ablation study and an oracle test?
An ablation removes or degrades a component and measures what your current implementation of it contributes today. An oracle replaces the component with its perfect value and measures what the best conceivable version could ever contribute. They answer different questions, and only the oracle bounds your future effort. A component can contribute little in an ablation because your implementation is weak, while a perfect version would be worth a great deal, and that combination is exactly the one worth working on.
How should you write up a failed family of approaches?
Record what every attempt had in common, not just that they all failed. A family-level negative is a claim about the space, but what you actually tested is your search, and those are different. I closed a family after five implementations and later found a team placing in the top thirty using it, because all five of mine shared a structural assumption I never wrote down. Documenting the shared assumption converts a dead end into a signpost, since the exit is usually on the other side of whatever every attempt took for granted.
Is finishing in the top half of a Kaggle competition a good result?
The raw percentile hides how tightly the field is packed. I finished in the top 44 percent, four tenths of a foot below the bronze cutoff, with 710 teams sitting within five hundredths of a foot of my score. At that density a very small score difference translates to hundreds of rank places, so a mid-table finish can mean a fundamentally wrong approach or a nearly-medaling one. The way to tell is to measure how much your own work moved the final score. Mine moved it 0.345 feet after the halfway point against a 0.434 foot gap, which says the approach was close and the search was too narrow.
More in this series
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full ROGII Wellbore Geology Prediction thread.
- What drilling blind taught me about machine learning · the mid-competition summary, written before any of this was known.
The competition is over. This is the whole picture, including the parts that did not work.
