A Stranger’s Blog Post Told Me I Was 900 Places Wrong

The competition closed on August 5th. The next morning I pulled my final standing, wrote it into my project documentation, rewrote the overview around it, and moved the whole build plan to closed. Four months of work, and there was the number that graded it.

It was wrong by nine hundred places.

Not rounded wrong, not off by a rank or two. Nine hundred, and the distance it reported between me and a medal was overstated by a factor of seven.

Nothing errored. No exception, no warning, no empty response. The data I got back was real data, pulled live from Kaggle, internally consistent from top to bottom. I had simply asked the wrong question and gotten a beautiful answer to it.

What the competition was

ROGII Wellbore Geology Prediction, a fifty thousand dollar Kaggle code competition. You are given a horizontal well being drilled and a nearby vertical reference well, the typewell. The typewell has been logged from top to bottom, so you know what the rock looks like at every depth. The horizontal well only has a gamma ray reading as the bit moves forward, and past a certain point, the Prediction Start, nobody tells you where you are in the rock column anymore.

Your job is to answer that question at every single foot after the cut. Not where the bit is in space, which surveying already tells you, but where it sits stratigraphically, which layer it is actually cutting through. You do it by matching the gamma ray signature coming off the bit against the typewell’s known profile.

The score is pooled root mean squared error in feet across every row of every hidden well. Lower is better. Six thousand one hundred and twenty five teams entered.

Half the test wells are scored live on a public leaderboard you can see all through the competition. The other half are hidden until the deadline, and that private half is the only one that decides anything. You pick two final submissions before the close. Your better private score is your result.

The two doors

Here is the entire bug.

api.competition_leaderboard_download(competition, path)   # returns the PUBLIC board
api.competition_leaderboard_view(competition)             # returns the PRIVATE board

They read like synonyms. One downloads the leaderboard, one views the leaderboard. In every mental model I had, those were two interfaces onto the same object, and the only difference was where the bytes ended up.

They are two different boards. And after the deadline passes, when the private leaderboard becomes the only one that means anything, download still hands you the public one. It does not warn you. It does not append a column telling you which board you are holding. It returns a clean, well-formed, six-thousand-row leaderboard, because that is exactly what it is.

There is a scene in Indiana Jones and the Last Crusade where the knight guarding the grail says you must choose, and choosing wrong is fatal, and the false cup on the shelf is the most magnificent object in the room. That is the failure mode here. The wrong answer is not broken or ugly or obviously missing something. It is gorgeous. It has a full distribution, a sensible spread, a plausible winner at the top and a long tail at the bottom, and every sanity check I could run against it passed, because all of my sanity checks were internal and the data was internally perfect.

I took my private score, which was correct, and I ranked it against the public distribution, which was also correct, and the combination was nonsense.

For what it is worth, I am not the first person to walk into this. The behaviour has been raised more than once on the Kaggle API issue tracker, including the observation that the show and download paths disagree about which board they hand back depending on whether a private one exists yet. It is a known rough edge. It is just not a rough edge that announces itself at the moment you step on it.

Why I did not catch it myself

I want to be precise about this part, because it is the part I would have skipped if I were reading someone else’s write-up.

The wrong number made me look worse.

It said I had finished in the bottom half, at 57.9 percent of the field, and missed a medal by 3.2 feet. That is a comfortable, unremarkable, slightly disappointing result. It is exactly the shape a four-month project ending without a medal is supposed to have. It even came with a tidy narrative: the field moved faster than I did, my approach was mid-pack, better luck next time.

If the error had gone the other way, if the bug had told me I finished in the top ten percent, I would have torn it apart within about ninety seconds. I know what my model was worth. A number that flattered me would have set off every alarm I have.

But nobody audits bad news. You accept it, you write your honest post-mortem about how you came up short, and you close the file. The error was in the one direction I had no instinct to question, and it wore the exact costume of a result I was already prepared to believe.

That asymmetry is worth sitting with, because it is not really about an API. Any measurement that confirms your lower expectations gets less scrutiny than one that exceeds them, and it gets that pass from the same person who would swear they treat all their data equally.

A stranger caught it

I was reading around after the close, the way you do, looking for other people’s retrospectives to see what the teams above me had done differently. I found one from a competitor named pilkwangkim, writing up their own run at the same competition.

Their post named the winner. Ruby, at 5.639.

On my board, Ruby was not the winner. My board said the top score was 4.608, from a team called shu01.

That is the whole moment. Two documents, both describing the same finished competition, disagreeing about who won it. One of us had the wrong file, and the person with six thousand rows of leaderboard sitting in front of him was not automatically the one who was right.

Their post also mentioned that shu01 had been leading the public board and had dropped hard on private, finishing 28th. Which meant that the number sitting at the top of "my" leaderboard was not the winning score at all. It was the top of the public board, the one that stopped mattering at 23:59 UTC on the fifth.

Everything unravelled from there, and it unravelled in the good direction.

Verifying the correction

Finding out you are wrong is not the same as finding out what is right. I had just learned that a plausible leaderboard can be entirely the wrong leaderboard, so pulling a second one and trusting it because it was newer would have been the same mistake with fresh paint on it.

I needed the new pull to agree with something outside my own machine.

The retrospective gave me three checkable facts, stated by someone with no idea I existed and no stake in my numbers being right. So I checked the corrected board against all three:

• Ruby, first place, 5.639. Matched.
• shu01, twenty-eighth, 6.653. Matched.
• pilkwangkim themselves, 8.197, which they described as rank 210. My pull put them at 209. Off by one, which is what you get when someone counts themselves on a paginated board, and close enough to be a confirmation rather than a discrepancy.

Then one more, because three anchors from a single source is really one source. The retrospective quoted the median gap between public and private scores across the whole field, a measure of how much the average team’s score degraded when the hidden half was scored. They had 1.929 feet. I computed 1.926 feet from my own corrected pull, independently, over 6,125 matched teams.

Three named anchors and one aggregate statistic, from a document written by someone else for their own reasons. That is what turned the second pull from another plausible file into a number I was willing to publish.

Here is what I had published, before any of that. Every number in this table is wrong:

Every one of those came out of the same call, and every one of them was measured against a distribution that stopped meaning anything the moment the deadline passed.

I am going to hold the corrected figures for the last entry in this series, because what they actually say about the four months takes some unpacking and it deserves its own piece. What matters here is the direction. The wrong numbers said the medal had never been within reach and the last two months of work could not have changed that. The real ones say something considerably less comfortable.

The rule I took out of it

Internal consistency proves nothing when the failure is correct data from the wrong source.

Every check I would normally run is a check within the dataset. Does it parse, are there nulls, does the distribution look sane, do the ranks increase monotonically, does the count match what I expect. My wrong leaderboard passed all of them, because it was a real leaderboard. It was just not mine.

So the question I now ask before any externally sourced number becomes a claim I make in public is a different one. Not "is this data clean," but:

What fact about this data can I verify somewhere outside the system that produced it?

A published result from another team. A number quoted in someone’s blog post. A total that should match a figure from a completely separate report. Anything at all that was not generated by the same call that generated the thing I am trying to check. One external anchor would have caught this in a minute. I had none, so it took a stranger’s retrospective and a coincidence of reading habits.

The competition is over and this is the part I will still be using in a year. Not the model, not the feature engineering, not the ranking. The habit of asking where a number came from before deciding what it means.

Frequently asked questions

What is the difference between the public and private leaderboard on Kaggle?

The public leaderboard is scored on a visible slice of the test set and updates every time you submit, all through the competition. The private leaderboard is scored on the rest of the test set, stays hidden until the deadline, and is the only one that determines your final placing. The gap between them measures how much of your public score was real signal and how much was fitted to the particular rows you could see. In this competition the median team’s error grew by 1.926 feet when the hidden half was scored, and roughly seven in ten teams got worse.

Why did my Kaggle rank come back wrong from the API?

Most likely you used competition_leaderboard_download, which returns the public board even after the deadline has passed. The private board comes from competition_leaderboard_view, which is paginated at about two hundred rows per page, and on some versions the continuation token is printed to standard output rather than returned on the response object, so you have to capture stdout to page through it. Neither call errors and both return well-formed data, so the only symptom is a rank that is quietly measured against the wrong distribution.

How do you verify a leaderboard pull is correct?

Check it against facts published outside the system that gave it to you. Competitors’ retrospectives are ideal, because they name specific teams, scores and ranks, and they were written without any knowledge of your numbers. I validated my corrected pull against three named placings from another team’s write-up and one aggregate statistic, the field-wide median gap between public and private scores, which I recomputed myself and matched to within three thousandths of a foot. Internal checks like row counts and monotonic ranks cannot catch this class of error, because the wrong data passes all of them.

Why don’t normal data quality checks catch this kind of error?

Because every standard check runs inside the dataset, and this dataset was not corrupt. Row counts, null scans, monotonic ordering, distribution shape, expected column types: my wrong leaderboard passed all of them, because it was a genuine leaderboard that had been fetched successfully. Validation confirms that data is well formed. It cannot confirm that it is the data you meant to ask for. Whenever a wrong result and a right result would both look structurally perfect, the only test with any power is one that compares against a fact from outside the system that produced it.

More in this series

• The Builder Journal · the live log across every competition I’m in.
• Every entry from this competition · the full ROGII Wellbore Geology Prediction thread.
• The validation score that lied to me for a month · the same disease one level down, from inside the competition.

The competition is over. This is the whole picture, including the parts that did not work.