Builder Journal · ROGII Wellbore Geology Prediction

What Drilling Blind Taught Me About Machine Learning

This is a Builder Journal summary, a step back from inside a competition I am still competing in, with prize money on the line and the deadline getting close. I have written a few dispatches from the trenches of this one already. This is the map of the whole thing in one place, and the handful of lessons that turned out to matter, most of which surprised me. You are reading where I was, not where I am, with the parts that are still my edge kept dark.

Start with the strangeness of the task, because it frames everything else. The competition is ROGII Wellbore Geology Prediction. An oil well drills straight down for a while, then turns and runs sideways for a mile or more through solid rock. Nobody can see down there. The drillers are trying to keep the bit inside one specific layer of rock, a layer that rises and dips as it goes, and to do that they need to know how deep that layer sits at every single foot of the horizontal run. The only instrument I get is a gamma-ray log, a readout of how naturally radioactive the rock is as it grinds past the bit, because different layers glow at different levels. I also get a reference well nearby that was drilled straight down and measured properly. From that faint radioactive whisper and that reference, I have to call the depth, foot by foot, for a mile of rock I will never see. It is scored in feet of error, on wells I never get the answers to, and there is fifty thousand dollars for the team that gets closest.

I came into this expecting the hard part to be the model. Some clever architecture, some sharp feature. I was wrong in a way that has changed how I work, and it comes down to five lessons. Almost none of them are about models.

One: the scoreboard has to be honest before anything else counts

For my first month, the scoreboard I built at home to grade myself disagreed with the real competition leaderboard about whether I was improving at all. Not about the size of my progress. About its existence. It turned out I was computing a different statistic than the one they score you on, on a handful of wells that did not represent the hidden ones, so my home number could go down while my real number sat still. I was optimizing a lie with total confidence. Nothing else I did mattered until I fixed the ruler, because every decision I made rested on it. I told that whole story in The Validation Score That Lied to Me for a Month. The short version is that a model you cannot measure honestly will wander for weeks while you congratulate yourself the entire way.

Two: most of the job is saying no, cheaply

Once I could measure honestly, I discovered that most of my ideas were bad, which is normal and not the interesting part. The interesting part is how expensive it is to find that out the naive way, by building each idea in full and testing it. So I got strict about killing bad ideas cheaply instead, with quick probes and a kill-rule written down before I looked at the result, so my own hope could not talk me into keeping something dead. I have buried sixteen ideas that way, several of them for about an hour of compute each and not one of them for a submission I could not spare. The full account is in How I Killed Sixteen Ideas Without Wasting a Submission. The leaderboard shows you everyone’s one idea that worked and hides the graveyard, so it is easy to believe the winners simply had better ideas. Mostly they just had a faster, more honest way to find out which of their ideas were bad.

Three: test on reality, not on a fixture that flatters you

Three separate times, a screen full of passing tests certified a model that was badly broken. Each time, the test was easier than the world in some specific way. A synthetic signal cleaner than real rock. A fixture that quietly assumed away the exact ambiguity that makes the problem hard. A test that never ran the messy code path the real data actually travels. Green checkmarks, broken model, every time. I wrote it up in The Green Test Suite That Hid a Broken Model. The lesson I carry now is to verify my machinery on synthetic data but verify my signal on real data, and to distrust any test built on a fixture I designed to be easy.

Four: the ceiling you accept is only this week’s ceiling

At one point I hit what felt like a hard wall, a score I decided was the ceiling of my whole approach, and I stopped pushing on it for a few days. When I looked back at the leaderboard, the entire field had blown past that ceiling while I was not watching. A better public method had become available and everyone had picked it up, and my "ceiling" had quietly decayed into the middle of the pack. In a live competition the ground moves under you. The frontier is not a fact about your method, it is a fact about what everyone else figured out this week, and if you are not watching them you are already behind. Humility plus a habit of watching the field turned out to beat a clever idea I had stopped questioning.

Five: optimize the score that pays, not the score that is shown

This is the one I can say the least about, because it is the closest to the live edge, so I will keep it to the principle. The score on the public leaderboard and the score that actually decides the prize are two different things, and a chunk of the visible leaderboard is built on tricks that look brilliant in public and quietly fall apart on the hidden final test. It is very easy to spend your last weeks climbing a number that will not pay, because climbing it feels like progress and it is the number you can see. The discipline is to keep optimizing the score that actually decides the outcome, even when it is invisible and the shown one is right there, tempting you. How I am actually doing that is the part that stays dark until the competition closes.

The through-line

Line those up and the pattern is hard to miss. An honest ruler. Killing bad ideas cheaply. Testing against reality. Watching the field. Optimizing the score that pays. Not one of them is a modeling trick. Every single one is about not fooling yourself, in a domain built to fool you, where the answers are hidden and the feedback is slow and expensive and every instrument you own will lie to you if you let it.

That is the thing this competition has actually taught me, and I think it generalizes far past wellbores. In any messy real-world problem, the model is rarely the bottleneck. The bottleneck is measurement you can trust, the discipline to abandon what is not working, and a working suspicion of your own certainty. The clever part, the model, is the last five percent, and it only pays off once the boring ninety-five is honest. I spent my first month wanting the competition to be about the five percent. It was always about the ninety-five.

Where I stand right now: mid-pack in a field of thousands and still climbing, with a home scoreboard I trust, a graveyard of ideas I am glad I buried, and the one truly interesting piece, the part that is still my edge, kept deliberately out of frame until the deadline passes. There is a long way to go. But I am no longer flying blind about how to fly blind, and in a competition like this one, that is most of the job.

More in this series