Builder Journal · ROGII Wellbore Geology Prediction
The Green Test Suite That Hid a Broken Model
This is a Builder Journal entry from inside a machine-learning competition I am still competing in, real money on the line and a deadline bearing down. It is not a clean postmortem written once the dust had settled. It runs long, longer than usual, and it shows code, because it is about the most reassuring lie in all of software: a screen full of passing tests. You are reading where I was, not where I am, with the parts that are still my edge kept dark.
There is a specific feeling every engineer chases, and it is the color green. You run your tests, the whole column lights up, and something in your chest unclenches. The code works. You can move on. It is one of the quiet comforts of the job, and this competition taught me, three separate times, that the comfort is sometimes a complete fabrication. A test can pass with total confidence and be telling you nothing at all, because the thing it checks is easier than the thing you actually face. Green does not mean correct. Green means the test agreed with the code, and if the test was asking the wrong question, the two of them can agree with each other all the way to disaster.
I want to walk through the three times a passing test lied to me in this competition, because each one lied in a different way, and together they rewired how I think about testing anything that touches real data.
A quick orientation if you are new here. The competition is ROGII Wellbore Geology Prediction. A well drills straight down, turns, and runs sideways for a mile through rock nobody can see, and my job is to predict how deep a particular rock layer sits at every foot along the way, using a gamma-ray log, the faint radioactive signature of the rock scraping past the bit. I get a reference well, a typewell, that someone already drilled and measured properly, and the core of the task is matching my blind well’s signal against that reference to work out where I am. Scored in feet of error, on hidden wells, fifty thousand dollars at the end. That matching step is where the first lie lived.
Lie one: the fixture was cleaner than the world
My first real idea was a matched filter. The concept is simple and it felt obviously right: take a window of my well’s gamma-ray signal, slide it along the reference typewell, and find the depth where the two shapes line up best. That depth is my guess. Here is the heart of it.
import numpy as np
def best_depth(window, typewell):
"""Slide the well's GR window along the reference typewell and return
the depth whose signature matches best. Normalized, so it scores the
shape of the signal, not its absolute level."""
w = (window - window.mean()) / (window.std() + 1e-9)
best, best_score = 0, -np.inf
for d in range(len(typewell) - len(w)):
seg = typewell[d:d + len(w)]
seg = (seg - seg.mean()) / (seg.std() + 1e-9)
score = float(np.mean(w * seg)) # 1.0 means identical shape
if score > best_score:
best, best_score = d, score
return best
I tested it the sensible way. I built a synthetic typewell, planted a window at a depth I chose, and checked that the matcher found it. It did, dead on. I added a little noise and checked again. Still dead on, within a foot. Green. The matcher worked. I was proud of it.
Then I ran it on real wells and it missed by a median of two hundred and twenty-nine feet.
Not a little off. Two hundred and twenty-nine feet, and on some wells the error marched steadily worse as the well went on, drifting more than three hundred and fifty feet from the truth by the end, so reliably wrong that its predictions correlated negatively with reality. The matcher was not broken in a way my test could see. It was broken in a way my test had quietly removed. My synthetic typewell was clean, a signal with one obvious answer. A real gamma-ray log is noisy, and worse, it repeats. The same rough pattern of radioactivity shows up at many different depths, because rock layers themselves recur. So when my matcher went hunting for the single best place its window fit, the real world handed it a dozen places that fit almost as well, and it confidently picked a wrong one. My test had given it a puzzle with a unique solution. Reality gave it a puzzle with a hundred near-solutions and no signpost pointing at the right one.
And the errors did not stay put and average out. They fed on each other. My matcher placed each new window relative to where it believed the last one had landed, a greedy forward march down the well, so a single confident wrong turn near the start became the anchor for every guess that followed. One bad match did not cost me one bad foot. It bent the entire rest of the well around itself. That is why the error grew as the well went on instead of washing out, and why on the worst wells the predictions ended up marching in the opposite direction from the truth, so wrong they were anti-correlated with reality. A test on a clean fixture can never surface this, because on a clean fixture there is never a wrong turn early enough to compound. The failure mode did not exist in the world I had built to check my work. It only existed in the one I was being scored in.
I spent real effort taming the drift after that, adding constraints so the match could not wander so freely, and I got the median error from two hundred and twenty-nine feet down to about thirteen. A massive improvement, and still worse than the dumbest baseline in the competition, which is to hold the last known depth flat and predict no movement at all. That baseline scores around eight feet. My clever, carefully repaired matched filter scored thirteen. The whole idea went in the ground. But the lesson from the test outlived the idea, and it was this: my fixture had been easier than the world in the one specific way that decided everything, and a test easier than the world is not a safety net. It is a blindfold with the word "green" written across it.
Lie two: the fixture assumed away the hard part
The second lie was a more sophisticated version of the first, and I fell for it anyway, which is the humbling part. Much later, wiser, or so I thought, I built a smarter aligner, one that could handle the way these wells cross geological faults, the places where rock layers suddenly jump. It was a real piece of engineering and I tested it carefully. Every test passed.
Here is what I did not notice. To write clean, readable tests, I had built a synthetic typewell that climbed steadily, every depth a little more radioactive than the one before it. It made the tests easy to reason about. It also, without my ever deciding it, made every depth unique. In my fixture no two depths looked alike, so there was always exactly one correct answer and my aligner always found it. Green, green, green.
But "every depth looks different" is exactly the thing that is false in reality, and it is the entire difficulty of the competition. Let me show the trap directly. Here is a matcher trying to locate the same noisy window against two typewells: one that climbs, like my fixture, and one that wanders and revisits its own levels, like real rock.
import numpy as np
def match_scores(window, typewell):
"""Score how well `window` fits at every depth of the typewell."""
w = (window - window.mean()) / (window.std() + 1e-9)
out = []
for d in range(len(typewell) - len(w)):
seg = typewell[d:d + len(w)]
seg = (seg - seg.mean()) / (seg.std() + 1e-9)
out.append(np.mean(w * seg))
return np.array(out)
rng = np.random.default_rng(1)
depth, win = 180, 40
# My fixture: a signal that climbs. Every depth is unique by construction.
climbing = np.linspace(0, 50, 500)
# Real rock: a signal that wanders and keeps revisiting the same levels.
wandering = np.cumsum(rng.normal(0, 1, 500))
for name, typewell in [("climbing (my fixture)", climbing), ("wandering (real)", wandering)]:
window = typewell[depth:depth + win] + rng.normal(0, 0.5, win) # same noisy window
scores = match_scores(window, typewell)
guess = int(np.argmax(scores))
runner_up = np.sort(scores)[-2]
print(f"{name:22s} guess={guess:3d} truth={depth} "
f"best={scores.max():.2f} runner_up={runner_up:.2f}")
On the climbing fixture, the matcher lands on the true depth and its best score towers over everything else. There is one answer and it is obvious. On the wandering signal, the best score barely edges out a whole crowd of runners-up, so the matcher can lock onto one of them and sit hundreds of feet from the truth while reporting a match score that looks nearly as strong as the real one. On the actual competition data, this exact failure put my aligner anywhere from forty-seven to nearly three hundred feet off, because a real typewell spans hundreds of feet of candidate depths while the slice of it I am trying to place covers only a few dozen. My test had assumed away the ambiguity, and the ambiguity was the whole problem. I had written a fixture that deleted the reason the competition is hard, then let it certify my code as correct.
Lie three: the test never ran the code that broke
The third lie is the quietest, and in ordinary day-to-day work it is the most common of the three. My tests passed because they were exercising code my actual model never runs.
A lot of my core math is written in plain numerical arrays, clean and fast, and my tests hammered that math with synthetic arrays and confirmed it was correct. It was correct. But in the real pipeline the data does not show up as tidy arrays. It shows up as dataframes, with missing values and mismatched types and all the grit of real tabular data, and there is a layer of code that wrangles that mess into shape before the clean math ever sees it. My tests never touched that layer. They fed the math exactly the tidy input it wanted and never once walked the path the real data walks.
That is where the bugs were living. Not in the math I had tested into the ground, but in the unglamorous wrangling I had not tested at all. It took a careful code review, a human tracing the actual route the data travels, to find a spot where a missing-value fix would crash on the real dataframe, and another where a safety check had been quietly weakened until it no longer caught the thing it was built to catch. A green test suite sat serenely on top of both, because it had never once driven down the road where the potholes were.
What I build differently now
Three lies in one competition is enough to change how a person works, and it did. My testing looks different now, in four concrete ways.
The first is that no method gets to be trusted on synthetic green alone. Every new idea has to clear a real-well smoke test before I believe any amount of passing tests, a quick run against actual wells with actual answers, scored the way the competition scores. Synthetic tests still gate whether my code does what I told it. They no longer get to certify that the idea works. Those are different questions and I stopped letting the first one answer the second.
The second is that I keep a small, permanent cast of the nastiest real wells I have found, the ones that have broken things before, and nothing is allowed to be called done until it has survived them. They are my adversaries. A method that cannot handle my worst known cases is not ready no matter how green the rest of the suite is.
The third is that I stopped writing fixtures to be easy to reason about and started writing them to be honest about the difficulty. If the real problem is that the signal repeats, my synthetic typewell now deliberately repeats itself, so that a test can actually fail for the right reason. A fixture should contain the hard thing, not a cleaned-up cartoon of it. If I catch myself building a test case because it is tidy, that is now a warning sign, not a convenience.
The fourth is that I read the data path as carefully as I read the algorithm. The clean math was never where my bugs lived. They lived in the unglamorous layer that wrangles messy real input into the shape the math expects, the exact layer my array-based tests skipped. So that layer gets reviewed line by line now, with real, ugly data in mind, because that is the road the real data actually drives down.
None of this is exotic. It is just the difference between checking that I built the thing right and checking that I built the right thing, and I had been quietly doing only the first while believing I had done both.
The signal so far
Three lies, three shapes. A fixture cleaner than the world. A fixture that assumed away the hard part. A test that never ran the real code path. In all three the checkmark was green and the model was broken, and the distance between those two facts is exactly where a competition is quietly won or lost.
The rule I took from it is simple to say and hard to live: verify your machinery on synthetic data, but verify your signal on real data. Synthetic tests are wonderful for one job, proving your code does what you told it to do. They are worthless, worse than worthless, for proving that what you told it to do is the right thing on inputs you did not get to design. A test built on a fixture easier than reality does not shrink your uncertainty. It launders it. It takes "I do not know if this works" and hands you back "all tests passing," which feels like knowledge and is really just your own assumptions read back to you in a calming shade of green.
The part that stays with me is that none of these were careless mistakes. Each test was reasonable. Each fixture was built for a sensible reason. The failure was subtler than sloppiness: I had let the easy version of the problem, the one I could cleanly write a test for, stand in for the hard version, the one I was actually there to solve. That substitution is invisible while you are making it, because everything in front of you is green. The only cure I have found is to distrust my own fixtures on purpose, to keep asking what the easy version quietly deleted, and to save my real belief for the moment the code finally meets data I did not build.
Where I stand right now: three good ideas deep in the graveyard, each one buried by a test that told me the truth only after I stopped trusting the color of it. Still climbing, and a great deal more suspicious of my own certainty than I was when I started. That suspicion, it turns out, is one of the most useful instruments I own.
More in this series
- ← Start here: The Validation Score That Lied to Me for a Month · the metric that was measuring the wrong thing.
- How I Killed Sixteen Ideas Without Wasting a Submission · the cheap gates that kill a bad idea before it costs you.
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full ROGII Wellbore Geology Prediction thread.
This is part of an ongoing builder’s log written from inside live competitions. You’re reading where I was, not where I am.
