Builder Journal · Mars Environmental Dynamics Analyzer (MEDA) Virtual Sensor Recovery

Every Knob My Validation Picked Was Too Timid

I spent a full session building a dilated temporal convolutional network. Windowing, NaN masks, a center-tap head, deterministic CUDA, six tasks of test-driven development, an independent review on each one. Good engineering. It moved the public leaderboard by about 1.3.

Then I shifted every prediction down by 2.33 Pascals and the leaderboard moved 5.4.

That is the entry. Everything else is me working out why, and the answer turns out to be the same shape as the answer to three other questions I had already gotten wrong in this competition.

The setup, in one paragraph

Perseverance carries an environmental station that reads pressure, temperature, humidity, wind and irradiance from the floor of Jezero Crater. Some pressure readings are missing and the job is to reconstruct them. Scored on mean squared error. The split is the whole problem: training data covers sols 1 through 100, test data covers sols 201 through 300, and nothing exists in between. Training sits entirely on the rising limb of the seasonal pressure cycle. Test sits entirely on the falling limb.

I covered how I got a first submission out of that in the entry on taking first place without training a model. The short version is a decomposition, PRESSURE = B(Ls) + D(time_of_day) + R, where the seasonal term and the daily cycle are physics that can extrapolate and only the leftover residual gets a learned model.

Here is the consequence that governs everything below. Cross-validation can only hold out sols I have labels for, and I have labels for exactly one hundred sols on the rising limb. So my validation was measuring in-distribution fit quality on a season I was not being scored on, while the leaderboard measured extrapolation error on a season I could not see. Those two numbers were never going to agree, and I already knew that.

What I did not know is that the disagreement has a direction.

The lever that was an artifact

I came into the residual-model session with a next action already written down. It was the highest-value thing on the list, in my own notes, from my own previous diagnostics: the diurnal amplitude appeared to grow with the season, measured at plus 0.027 Pa per sol. If the daily pressure swing gets bigger as the season advances, then modeling that growth and carrying it into the test window should be worth real MSE.

It took two commands to kill it.

# peak-to-peak pressure per sol, against how many hours that sol actually samples
per_sol = df.groupby("sol").agg(
    ptp=("PRESSURE", lambda p: p.max() - p.min()),
    hours=("LTST", lambda t: t.round().nunique()),
)
print(per_sol.head())
# the first sols cover only one or two hours out of twenty-four
#   -> their peak-to-peak is tiny for reasons that are not atmospheric
# reliable full-day coverage does not start until around sol 30

The earliest sols do not sample a full Martian day. They sample an hour or two of it. If you only observe a slice of the daily cycle, the largest value minus the smallest value inside that slice is small, and it is small for a reason that has nothing to do with the atmosphere. The rover had just landed and the instrument was not yet running a full cadence.

So my measured amplitude growth was mostly the first few sols climbing out of a coverage hole. Fit a trend through that and extrapolate it a hundred sols forward and you get a number with confidence attached and no physics behind it. It is the extrapolation trap again, wearing a new costume, and I had written it into my own plan as the top priority.

The fix was to filter to full-coverage sols before fitting anything diurnal. The lesson is cheaper than that. On a wellbore-geology competition I built a set of pre-registered gates whose entire purpose was killing ideas before they cost a submission, and this is the same trade. A two-command diagnostic protected a week of work from being spent on an instrument artifact.

Ask what your measurement is made of before you extrapolate it.

Thirty-one percent of the gain was a constant

With the diurnal term fixed, I put a LightGBM on the residual. Not on pressure, on R, the few Pascals of wiggle the physics did not explain. Small target, small blast radius, nothing to overturn.

Forward cross-validation went from 10.2 to 3.3. Down sixty-eight percent. That is the kind of number that makes you want to ship immediately.

Then the review asked which features were doing the work, and Ls was number one by importance, responsible for roughly thirty-one percent of the gain.

train["Ls"].max()                     # about 54
test["Ls"].min(), test["Ls"].max()    # 99, 147

Every single test row sits past the largest threshold the tree ever learned. A decision tree does not extrapolate. It falls into its outermost leaf and returns that leaf’s value, so Ls was contributing the identical constant to all one hundred sols of test predictions. Not a weak signal. A number that does not vary.

And cross-validation loved it, because on cross-validation Ls was in range and carrying real signal. The feature was carrying real weight on the rising limb and precisely zero information on the falling one. High importance, near-zero test value, and a validation score inflated by exactly the component that was about to go dead on deployment.

This is the part I want to state as flatly as I can. Feature importance is an in-training statistic. It measures how much a feature contributed to splits the model actually made, on data the model actually saw. It says nothing whatsoever about whether that feature will be usable on rows you have not seen yet.

The check is one line and I now run it on every model I ship into a distribution shift: for each feature, does the test range overlap the training range at all? Ls and ROVER_VELOCITY both failed it. Both came out. The seasonal level was already owned by the climatology term anyway, so the tree had been re-learning, badly, something the physics already knew.

The features that survived were still shifted

Dropping the dead ones is the easy half. The harder half is that the surviving features were not clean either.

Test humidity has a median of 0.25. Train humidity has a median of 0.62. About thirty percent of test humidity readings sit below the training minimum. So every split the model learned on humidity was learned on wetter, rising-limb air, and it was going to be applied to drier air the model had barely seen.

That is covariate shift, and it is a quieter failure than the Ls one. The relationship is not dead, it is weakened by an unknown amount. You cannot measure the amount, because measuring it would require labels in the shifted regime, which is the thing you do not have. This is where I have watched myself and other people go wrong in the same way: the temptation is to declare the model either fine or broken, when the honest position is that the signal is real and the magnitude is uncertain.

So I did not try to resolve the uncertainty. I priced it.

Shipping scared, on purpose

I was in first place at the time. A careless residual model could take that away for no gain, so the correction went out wrapped in four independent layers of protection.

A gate, meaning the machine learning correction only ships if it beats the physics-only baseline on cross-validation. A shrink factor of 0.5, multiplying the whole correction by a half. A clip bounding the final output to a physically sensible range. And a fallback, the safe physics-only prediction sitting one config flag away.

The shrink is the one worth understanding, because the arithmetic is nicer than it looks. If your correction is wrong, the damage you take is the square of what you added. Halving the correction quarters the worst case. The review put numbers on it: at full strength the worst case was about plus 8 Pa², enough to lose the lead. At half strength, about plus 2 Pa². Half the upside, a quarter of the downside, and when your validation cannot see the regime you are being scored on, that is a trade worth making.

It shipped at 34.28 to 29.15, down fifteen percent, and comfortably ahead of the review’s cautious estimate of about 33.5.

Read the session back and it is a clean win for caution. The hedging worked. The gate held. The two disciplined kills stopped two plausible-looking regressions. This is the version of the story where caution is the hero.

Then the leaderboard told me I was scared for nothing

The next session started with the cheapest possible experiment. Turn the shrink from 0.5 up to 1.0 and see what happens.

29.15 to 25.98. Down eleven percent, from removing a safety measure.

So I built the temporal model, a dilated one-dimensional convolutional network reading a window of covariates around each prediction point. It cannot be autoregressive, incidentally, because the test pressure column is one hundred percent missing, so there is no past pressure to feed it. It maps trajectories of humidity and irradiance to residuals, which is a different kind of error surface than a tree splitting on instantaneous values, which is exactly why blending the two should help.

Cross-validation picked a blend weight of 0.1. Ten percent neural network, ninety percent tree. It shipped at 25.61.

Then I swept the blend weight against the leaderboard and the optimum was 0.7, scoring 24.69.

Seven times the weight my validation wanted. The temporal model generalized about four times better than the rising-limb cross-validation had shown, and if I had trusted the number my own pipeline produced I would have thrown away most of the value of a build I had just spent a session on.

Three knobs, one pattern:

Knob	Cross-validation said	Leaderboard said
Residual shrink	0.5	1.0
Blend weight	0.1	0.7
Seasonal offset	invisible, zero	2.33 Pa, worth 5.4 MSE

Every one of them too timid. Not randomly wrong in both directions, which is what noise looks like. Systematically conservative, which is what bias looks like.

Once you see it stated that way the mechanism is obvious. My validation only ever saw the rising limb, so anything whose value shows up mainly on the falling limb registers as marginal, and anything defensive registers as prudent. The bias was baked into the geometry of the split. I had been reading a number that was structurally incapable of rewarding the thing I was trying to build.

On a disjoint-extrapolation problem, cross-validation is for go or no-go decisions and for catching artifacts. It is not for setting magnitudes. For any scalar that faces the extrapolation gap, the leaderboard is the selector. That is uncomfortable to write down, and the last section of this entry is about the price of taking it too far.

It is not the first time a number I trusted turned out to be answering a different question. On a wellbore-geology competition, a validation score lied to me for a month because I had pooled the error metric the wrong way across wells. Different mechanism entirely. Same experience of finding out the instrument was pointed somewhere else.

One constant, and a parabola with nothing to hide

Here is the trick that beat all of it, and it is close to free.

If you shift every prediction by a constant δ, the new mean squared error is not approximately anything. It is exactly:

MSE(δ) = mean((pred + δ - y)²)
       = MSE(0) + 2·δ·b + δ²        where b = mean(pred - y)

A parabola in δ, with the leading coefficient known to be exactly 1 before you run a single experiment. So you do not have to guess. Submit your predictions shifted by a handful of offsets, read the scores off the board, and fit the parabola.

deltas = np.array([-4.0, -2.0, 0.0, 2.0, 4.0])
scores = np.array([...])              # five leaderboard MSEs

c2, c1, c0 = np.polyfit(deltas, scores, 2)
# c2 came back 1.0000
b = c1 / 2.0                          # mean signed bias of my predictions
delta_star = -b                       # -2.33

That c2 value is the part worth stopping on. The theory says the quadratic coefficient must be exactly 1 for a pure constant shift. It came back at 1.0000. That is not a nice coincidence, it is a self-check: if my output clipping had been biting on a meaningful number of rows, or if the scorer were doing something other than plain MSE, the coefficient would have drifted off 1 and the whole inference would be untrustworthy. The fit validating its own assumption is worth more than the answer it produced.

The vertex sat at minus 2.33 Pascals. My seasonal baseline was reading 2.33 Pa too high across the entire test season. Not per sol, not per hour. A flat, constant, whole-season bias sitting underneath everything.

Applying it took the leaderboard from 24.69 to 19.24. Down twenty-two percent, the single biggest jump of the entire project, from one number.

The diagnostics had been saying this for weeks. Roughly 21 of my 25 MSE was seasonal extrapolation error that cross-validation is physically blind to. I had that written down. I built a neural network anyway, because the residual is where the modeling is interesting and the seasonal term felt finished.

The same session also handed me a free lunch I want to flag, because I nearly paid for it twice. A blend is linear in its weight, so any point on the sweep can be reconstructed from two scored submissions with no retraining at all.

# two anchors span the entire alpha sweep
a000 = pd.read_csv("v4_alpha000.csv")["PRESSURE"].to_numpy()
a010 = pd.read_csv("v4_alpha010.csv")["PRESSURE"].to_numpy()
step = (a010 - a000) / 0.10

def blend(alpha):
    return a000 + alpha * step        # any alpha, zero GPU time

Every offset probe and every blend weight in this entry cost one thing: a submission slot. No training runs. When your feedback channel is a daily submission cap, the cheapest experiments are algebra on files you already have, and it is worth an hour to work out which of your knobs are linear before you queue up a job.

I burned all twenty submissions that day, and the twenty-first came back a 400 mid-script.

The gains I did not take

After the offset landed I had a working method and a leaderboard that answered questions. So I asked two more.

A linear sol-trend on top of the constant recovered plus 0.06. A diurnal amplitude scale, with the optimum at 0.90, recovered plus 0.25.

Both real. Both small. Both firmly in the region where I could no longer tell improvement from noise in the public split.

I stopped.

That decision is worth more than either number. Kaggle scores you on a public subset during the competition and a hidden private subset at the end, and every parameter you fit to the public score is a parameter that might be fitting public-specific noise. A single constant offset is defensible, because a whole-season mean bias in the seasonal baseline is a broad physical effect that will be present in both halves. The correction generalizes because the cause generalizes.

Fifty per-sol offsets would not be. Neither, at some point, is the fourth or fifth global scalar tuned against the same public feedback. There is a line where leaderboard calibration stops being measurement and starts being memorization of a specific split, and the honest signal that you are approaching it is exactly this: the gains get small, and your justification for each new parameter gets thinner.

The rule I settled on is that a leaderboard-fit parameter has to correspond to something I can name physically. The offset corrects a baseline that reads high. That is a sentence about Mars. "A sol-trend coefficient of 0.06" is a sentence about the public leaderboard.

I have made the opposite call before and paid for it. On ARC-AGI I added capability to a solver and watched the more capable version score worse, because the extra machinery was fitting the examples in front of it rather than the rule underneath them. Same failure, different competition.

What I would take from this

Feature importance measures training, not usefulness. Before you trust any feature in a model that will face shifted or out-of-range inputs, check whether its test values overlap its training values at all. If they do not, a tree contributes a constant from that feature and your validation score has been quietly inflated by a column that will be dead on arrival.

A validation scheme that cannot see the deployment regime is not just noisy, it is biased in a direction. Mine could only reward things that helped on the rising limb, so it recommended caution three times in a row and was wrong three times in a row. Once you can name which slice of reality your validation covers, you can predict which way it will be wrong before you run it, and that prediction is more useful than the score.

Find the bias before you build the model. A constant offset is one parameter, costs five submissions, requires no training, and self-validates through the leading coefficient of its own parabola. It beat a week of neural network engineering by a factor of four. Spend the cheap probes first, not because sophistication is bad, but because you cannot tell how much room a sophisticated model has to work in until you have removed the parts that a single number can explain.

The thing that keeps happening in this log is not that I write bad code. It is that I keep asking a measurement for an answer it was never in a position to give, and then acting on the reply with complete confidence. In the last entry it was a polynomial degree chosen by a metric that could not see the falling limb. Here it was three hyperparameters in a row, chosen by the same blind metric, each one talking me into being smaller than I needed to be.

The instrument was fine. I was reading it as though it pointed at something it does not.

Frequently asked questions

What does high feature importance mean if the feature is out of range on test data?
It means the feature was useful during training and will be useless during inference. A gradient boosted tree splits on thresholds it observed, so any test value beyond the training range falls into the outermost leaf and returns a constant. The feature then contributes the same number to every prediction, while your cross-validation score, computed where the feature was in range, still credits it fully.

Should you tune hyperparameters on the leaderboard instead of cross-validation?
Only when your cross-validation is structurally unable to see the regime being scored, which happens when training and test data do not overlap in time or distribution. In that situation cross-validation systematically over-rewards caution, so scalars like shrinkage and blend weights should be selected against the leaderboard. Keep cross-validation for go or no-go gating and for catching artifacts, and keep the number of leaderboard-fitted parameters small.

How do you find a constant bias in your predictions using the leaderboard?
Shift every prediction by a constant and submit. Mean squared error under a shift of δ is exactly MSE(0) plus 2δb plus δ squared, where b is the mean signed bias, so five probes at different offsets pin the parabola and its vertex gives the optimal correction. A quadratic coefficient that comes back at 1.0000 confirms the metric is plain mean squared error and that clipping is not distorting the result.

What is covariate shift and how do you tell whether you have it?
Covariate shift is when the input distribution changes between training and deployment even though the underlying relationship is stable. Compare per-feature medians and ranges across the two sets: in this competition test humidity had a median of 0.25 against 0.62 in training, with about thirty percent of test values below the training minimum. The learned relationships are weakened by an amount you cannot measure without labels, which is an argument for shrinking corrections rather than for discarding them.

Why can a constant offset beat a neural network?
Because they are correcting different kinds of error. A neural network refines local structure that the baseline already gets approximately right, while a constant offset removes a whole-population bias that sits underneath every prediction equally. If your dominant error is a systematic level shift, no amount of local refinement touches it, and one subtraction removes all of it at once.

Standing: 19.24 on the public leaderboard, the top legitimate model by roughly 4.6x, with a single 0.0 above me that is published ground truth rather than a model. The next entry is about the gains I decided not to bank.

More in this series

• I Took First Place Without Training a Model · how the physics decomposition took the top of the board with no ML.
• My Own Test Set Lied to Me · the same bias, built by hand, on a hyperspectral tracking problem.
• The Builder Journal · the live log across every competition I’m in.
• Every entry from this competition · the full MEDA Virtual Sensor Recovery thread.

This is part of an ongoing builder’s log written from inside live competitions. You’re reading where I was, not where I am.