Builder Journal · Mars Environmental Dynamics Analyzer (MEDA) Virtual Sensor Recovery
Everyone Knows It Scores Half. Nobody Checks Which Half.
Every Kaggle competition page carries the same line of small print under the standings. This leaderboard is calculated with approximately 50% of the test data. The final results will be based on the other 50%. Everybody reads it. Everybody nods. Almost nobody asks which 50%.
I asked, and then I measured it, and it is not a random sample. It is the first 1,974,995 rows in spacecraft clock order, which is to say in time order, exactly half of the file, and the boundary falls inside sol 253.
That one fact rewrote six weeks of this project.
Where the competition stood
Quick refresher for anyone landing here first. Perseverance carries an environmental station called MEDA that reads pressure, temperature, wind and dust from the floor of Jezero Crater. Some of the pressure readings are missing. The competition is to reconstruct sols 201 through 300 while training only on sols 1 through 100, with a hundred-sol hole between the two, scored on mean squared error.
The first submission was a physics decomposition with no machine learning in it at all and it scored 61.04. Since then the model has been a three-part split: a seasonal baseline B driven by orbital position, a diurnal term D for the daily thermal tides, and a residual R that is the only part a learned model touches.
Every layer after that was tuned against the public leaderboard.
v1 physics baseline, no ML .......... 61.04
v2 climatology + anomaly diurnal .... 34.28
v3 gated residual LightGBM .......... 29.15
v4 SHRINK 0.5 -> 1.0 ................ 25.98
v5 + TCN blend, alpha = 0.7 ......... 24.69
v6 -2.33 Pa constant offset ......... 19.24
v7 diurnal gamma = 0.90 ............. 18.99
Seven versions, a factor of three, and a very satisfying downward staircase. Look at that list and you can see the shape of the work: pick a lever, probe it against the board, keep the value that scores best, move on. It felt like a tight feedback loop. It was a tight feedback loop. The question I had never bothered to ask is what, exactly, was on the other end of it.
Why "which half" is not a pedantic question
Here is the thing about a 50/50 test split. If it is drawn at random, public and private are two samples of the same distribution, and a constant you fit on one transfers to the other almost by definition. The public score is a noisy but unbiased estimate of the private score. Tuning against it is legitimate, and the only real risk is overfitting to sampling noise.
If it is drawn chronologically, none of that holds.
A chronological split means public is the early part of the test season and private is the late part. Which means the public score is not a sample of my private score, it is a measurement of a different regime. And on this problem, where the seasonal baseline is falling more than a hundred pascals across the test window, "a different regime" is not a technicality. Sol 210 and sol 290 are not two draws from one bag. They are two different places on a curve.
Every correction I had fit was shaped as a function of sol, or of something that moves with sol. A constant offset. A diurnal gamma. A blend weight. Under a random split, fitting those on the visible rows and applying them to the hidden rows is interpolation, and interpolation is safe. Under a time split, it is extrapolation across a boundary I cannot measure past, which is the exact failure mode this entire competition is built out of.
Same numbers. Same submissions. Completely different meaning depending on which way the file was cut.
So I stopped tuning and went to find out.
The leaderboard is not a black box
The trick that makes this measurable is that mean squared error is not opaque. It is a quadratic, and you can solve it.
Take your current prediction vector v and any perturbation direction e. Submit v + c*e for some scalar c. Then:
MSE(v + c*e) = MSE(v) + 2*c*<err, e>/N + c**2 * <e, e>/N
That is a parabola in c, and every piece of it is knowable except one. The c**2 coefficient is <e,e>/N, which you compute yourself from your own files, because e is your own vector. If you normalise e to unit RMS, that coefficient is exactly 1. The constant term is your current public score, which the board already told you.
Which leaves a single unknown. One submission at c = 1 and you have it.
# e normalised to unit RMS, so the quadratic coefficient is exactly 1
# MSE(v + c*e) = MSE(v) + 2*c*g + c**2
g = (mse_probe - mse_base - 1.0) / 2.0 # <err, e>, one exact number
c_star = -g # the optimal coefficient
gain = g * g # MSE the correction buys back
g is the projection of your unknown error onto a direction you chose. Read that again, because it is the whole tool. It is a measurement of the hidden labels, one number per submission, and you never see a single label. Pick your directions to be mutually orthogonal and the corrections do not interact, so the optimal combined fix is just the sum of the individual ones. No joint fitting, no re-running anything.
This is not exotic. Leaderboard probing is old and well documented, and hosts defend against it by truncating the decimals they display on the score. It is the sort of thing you read about and file under interesting. I had been doing it for weeks without thinking of it as measurement.
The bases were already sitting on disk
The part that surprised me is how cheap the probes were.
Every past probe submission was, by construction, best + c * basis. Which means the difference between two submission CSVs recovers the basis vector exactly. Not approximately. Exactly, because it is arithmetic on files I already wrote.
# no model re-runs, no re-fitting, no compute at all
s = trend_sp1.PRESSURE - v6.PRESSURE # the sol coordinate
D = (v6.PRESSURE - v7.PRESSURE) * 10.0 # v7 was v6 - 0.1 * D
Two independent basis vectors, one for the seasonal coordinate and one for the diurnal component, both reconstructed out of CSVs from previous weeks. The entire day’s analysis ran off files already on disk. There is a small lesson buried in there about how much information is sitting in your own artifacts once you start treating them as data instead of as output.
Twelve significant figures
With two independent bases in hand, the test is simple. Compute the inner products two ways. Once implied by the leaderboard scores, and once directly off the raw CSVs under a hypothesis about which rows are being scored.
If the hypothesis is right, the two numbers agree to the precision of the arithmetic. If it is wrong, they do not.
leaderboard-implied direct from the CSVs
<s, s> 0.335887303844 0.335887303844
<D, D> 25.52751 25.527508
Two independent directions, agreement to roughly twelve significant figures, under exactly one hypothesis: the public score covers the first 1,974,995 rows in spacecraft clock order. Exactly 50.00% of the file. Not a sol boundary, not a random draw, a straight chronological cut that lands in the middle of sol 253.
Public is sols 201 up to a point partway through sol 253. Private is the remainder of sol 253 and everything after it, out to sol 300. There is no clean boundary to point at, which is part of why I got this wrong the first time.
The banner told me half. It did not tell me which half, and I had spent six weeks assuming the friendly answer.
What that costs
Go back to the staircase.
SHRINK = 1.0. alpha = 0.7. offset = -2.33. gamma = 0.90. Every one of those constants was chosen because it improved the public score. Every one of them was therefore fit on the first half of the test season and then applied, unmeasured, to the second half. Every one of them is an extrapolation wearing the costume of a validated result.
I am not saying they are wrong. Some of them are probably close to right. I am saying I had no evidence about the half that decides the competition, while believing I had six weeks of it.
There is a version of this in nearly every competition I have written up. On a wellbore-geology problem, a validation score lied to me for a month because I was pooling an error metric the wrong way, and the number was answering a narrower question than the one I was reading it as. Different mechanism, identical shape. The trusted number was measuring less than I thought it was measuring.
The difference this time is that the trusted number was the leaderboard itself.
The anomaly that had been sitting there since June
Then something clicked that had been bothering me for a month.
Back in June I probed a linear trend in sol. If the seasonal baseline had a slow drift error across the test window, that probe should have caught it. It came back at +0.06. Negligible. Flat. I wrote the lever off as exhausted and went to work on the diurnal term instead.
The probe was correct and my reading of it was not. The public window sits near the peak of the seasonal baseline’s error bump, and near a peak the slope really is flat. A linear probe run entirely inside that window is measuring the flattest part of the curve and faithfully reporting that it is flat.
All the decay lives in the half the board cannot see.
That is the detail that turned this from an interesting fact into a work item. A negative result I had accepted, filed and moved on from was not a negative result. It was a measurement of the wrong interval, delivered in a form that looked exactly like a measurement of the right one.
The trap I almost shipped
Armed with the mask, the obvious next move is to fit the correction properly. Restrict least squares to the rows the board actually scores, fit a flexible shape in sol, and let the data tell you the bias curve.
I did that. The six-term polynomial fit promised 0.80 MSE of public gain, which at a score of 18.99 is not nothing. Then I plotted what it does outside the fitted window.
| sol | region | fitted correction |
|---|---|---|
| 230 | public, fitted | +0.65 Pa |
| 254 | public, fitted | -1.07 Pa |
| 265 | private, extrapolated | -27 Pa |
| 300 | private, extrapolated | -1397 Pa |
Minus fourteen hundred pascals. On a planet whose entire surface pressure is around seven hundred.
High-order terms are near zero inside the fit interval, which is why the fit looks excellent there, and they dominate everywhere outside it, which is why it detonates. The higher the degree, the more violent. Runge’s phenomenon, in a competition submission, with real slots on the line.
Look at what that actually is. A flexible model, fit where the data lives, applied where the data does not, quietly assuming the shape continues. That is the founding problem of this entire competition, the reason gradient boosting could not touch the seasonal term in the first place. It came back wearing a different hat, and it came back in the shape of a genuine 0.80 improvement on the only number I could see.
Shape-fitting on the public mask went in the bin.
Provenance as a prior
What replaced it is the part I like, because it is not fit to anything hidden at all. It comes from reading my own source file.
The climatology module interpolates 23 PCHIP knots. I had been treating those knots as if they were equally trustworthy. They are not, and the code comments say so. Three of them come from the paper’s text, at Ls 13, Ls 55 and Ls 147. The other twenty were digitized off a figure, by eye, at something like plus or minus 2 to 3 Pa.
So the error structure is predictable without any leaderboard at all. Small at the text anchors, where a published number pins the curve. Largest in between, where the curve is floating on figure-reading precision. That is a bump, and it peaks near sol 202.
# error is pinned near the paper-text knots and free between them
def bias_shape(sol):
return (sol - 105.0) * (300.0 - sol) / NORM # peaks near sol 202
Which retro-explains the June anomaly precisely. The bump peaks at sol 202, the public window starts at sol 201, and a linear probe across a peak reads flat.
And it says something uncomfortable about the offset. The -2.33 Pa constant was calibrated where the bump peaks. It is sized for the worst part of the curve, and then applied uniformly to private sols where the bump has decayed. Which means it over-corrects the half that counts.
The fix is a give-back. Add some of that offset back on the private rows, sized by how far each sol sits from a text anchor. It is not a tuned constant. It is a structural claim about where my own pipeline is weakest, derived from reading the provenance of my own inputs.
That is the whole difference. One correction is fit to a signal I can see. The other is derived from knowing why the model is wrong.
Five slots, one number
Then the endgame problem, which turned out to be a different kind of thinking entirely.
The rules allow five final submissions and rank is the best of them. Sit with that for a second, because it changes the objective. Extra slots are free. You are not being asked to pick the most likely answer. You are being asked to cover an interval.
I had ten candidate files and a comfortable feeling that I was hedging across two different physical hypotheses. Then I correlated the two supposed axes on the private rows.
0.999995.
They were not two hypotheses. One was the other rescaled by a factor of 1.15. Ten files, two named theories, one actual degree of freedom: a single scalar delta, the mean pascals added back on the private half, somewhere in a range of roughly 0 to 2.7.
Collapsing candidates to their real degrees of freedom before placing bets is the step I would have skipped a year ago. It is deeply unglamorous and it changed the entire plan, because once you know the uncertainty is one-dimensional, placement becomes a solved problem. Space the rungs to minimise the worst-case distance to the truth.
The corollary is the part I did not expect. With two slots, {delta = 0.61, delta = 1.82} beats {delta = 0, delta = 1.21}. The second pair looks more principled, because each rung is a named position: do nothing, or apply the physical estimate. The first pair is better because it covers the interval more evenly. You are not defending hypotheses. You are minimising regret across a range, and named positions are a worse basis for that than well-spaced ones.
So here is what actually went in. Five slots, five rungs, delta running 0.00, 0.85, 1.40, 2.10 and 2.67 Pascals. The bottom rung is the untouched model that applies no give-back at all, because zero is a live possibility and covering an interval means covering its ends.
| final slot | delta (Pa) | public score |
|---|---|---|
| untouched, no give-back | 0.00 | 18.9919 |
| anchor hedge | 0.85 | 18.9205 |
| anchor hedge | 1.40 | 18.8775 |
| anchor hedge | 2.10 | 18.8507 |
| anchor hedge | 2.67 | 18.8579 |
Look at the right-hand column and you can watch the parabola do its work. The public score falls as delta climbs, bottoms out around 2.10, and starts back up by 2.67, exactly the shape the algebra says it has to have. It would be very easy to read that column as a ranking and conclude the 2.10 rung is the answer.
It is not a ranking. Every number in it was computed on the first half of the test season, which is the half that does not decide anything. The vertex of that little parabola is the best give-back for rows I can see, and the rungs exist precisely because I cannot see the rows that count. The column is a by-product of the hedge, not a verdict on it.
One file did not go in. The six-term shape fit scored 18.7822, the best public score I have posted on this competition, better than any of the five I selected. It is also the model that reaches minus 1397 Pa by sol 300. I left it out.
That is the whole entry compressed into one decision. The best number on the board came from the model I trusted least, and the only reason I was able to walk past it is that I had spent the day working out what that number was measuring. A week earlier I would have selected it without hesitating, because it was winning.
This is the same instinct as the cheap pre-registered gates I ran on a wellbore-geology problem, pointed at a different part of the process. Decide the decision rule before the outcome, then let arithmetic place the bets rather than intuition.
Doing nothing is a decision
One last piece of the endgame, and it is the one most likely to bite someone reading this.
If you do not manually select your final submissions, Kaggle picks them for you, and it picks the ones with the best public score.
On a normal competition that default is sensible. On this one it selects for the exact trap I just spent a day measuring. The best public scores are the ones most aggressively tuned to the visible half, which is the half that does not decide anything. The auto-selection leaves a hole precisely where the physically motivated estimate sits.
Work through what that means here. Left alone, the default would have reached past all five of my rungs and handed a slot to the shape fit, on the strength of the 18.7822 that made it the best thing I had ever submitted. The model that goes to minus 1397 Pa would have been entered on my behalf.
There is no API method for changing it. You do it in the browser, by hand, before the deadline.
Not choosing is choosing. The default is not neutral just because it happens without you.
What I would take from this
Three things, and none of them are about Mars.
Know what your metric covers before you spend weeks improving it. Every extra decimal of public score I earned in the back half of this project came from terms that may well be hurting the invisible half. The improvement was real. The thing it improved was not the thing being graded.
A negative result is a measurement of an interval, not a verdict on a lever. The June trend probe came back flat and honest, and I read it as "no trend exists" when it meant "no trend exists here." That distinction cost a month of not looking at the seasonal baseline.
Extrapolation does not stop being extrapolation because the fit looks good. The six-term polynomial had a legitimate 0.80 MSE gain on every row I could measure and reached minus fourteen hundred pascals on rows I could not. If your model is flexible and your evaluation is local, the evaluation cannot warn you.
The through line, again, is that a number I trusted was answering a question I had not asked. I have now written that sentence in four competitions. It does not get easier to spot. What changes is how quickly you go looking for the question once the answer starts feeling too clean.
There is a coda to this one. Having built the mask, the hedge and the ladder, I put the whole endgame through an adversarial review, and it came back with errors in my reasoning that I had not caught. Some of what you just read survived that review. Some of it did not.
That is the next entry.
Frequently asked questions
Is the Kaggle public leaderboard a random sample of the test set?
Not necessarily, and the competition page usually does not say. The banner states what fraction is scored, not how the rows were chosen. In this competition the public half turned out to be the first 1,974,995 rows in chronological order, exactly 50.00%, rather than a random draw. Assuming random when it is chronological changes tuning from interpolation into extrapolation without any visible warning.
Why does it matter whether the public split is random or chronological?
Under a random split, public and private are samples from the same distribution, so a constant fit on the public half transfers to the private half and the public score is an unbiased estimate of the final one. Under a chronological split, the two halves are different regimes, and any correction shaped as a function of time is being extrapolated across a boundary you cannot measure past. On a problem with a strong seasonal trend, that difference is the whole result.
How can you tell which rows the public leaderboard scores?
By probing. If the metric is mean squared error, then submitting your prediction plus a known perturbation makes the score a parabola whose only unknown is the projection of your error onto that perturbation direction. Normalise the perturbation to unit RMS and one submission pins one direction exactly. Compute the resulting inner products directly from your own CSVs under a candidate mask, and the correct mask is the one where both sets of numbers agree to full arithmetic precision.
Why do high-degree polynomial corrections blow up outside the fitted range?
Because the high-order terms are near zero inside the fitted interval and dominate outside it, so the fit looks excellent exactly where you can check it. A six-term fit in this project gained 0.80 MSE on the fitted half and reached minus 1397 Pa fifty sols beyond it. This is Runge’s phenomenon, and it is why splines and shape-preserving interpolation exist. If your evaluation is local and your model is flexible, the evaluation cannot see the failure.
How should you use five final submission slots when rank is the best of them?
Stop trying to pick the most likely answer and start covering the uncertainty interval, because extra slots cost nothing. First collapse your candidates to their real degrees of freedom, since files that look like different hypotheses are often one parameter rescaled. Then space the rungs across the plausible range to minimise worst-case distance to the truth. Covering the interval evenly beats bracketing named positions.
Standing: rank 2 of 15, best public score 18.8441, behind a perfect zero posted by someone submitting published ground truth. The only run that ever scored better, 18.7822, is the shape fit I left out of my finals on purpose. The public score is no longer the number I am trying to move.
More in this series
- I Took First Place Without Training a Model · how the physics decomposition took the top of the board with no ML.
- The Bug Was in My Beliefs · the ARC-AGI version, where the failure was in my assumptions rather than anywhere I could set a breakpoint.
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full MEDA Virtual Sensor Recovery thread.
This is part of an ongoing builder’s log written from inside live competitions. You’re reading where I was, not where I am.
