The Model Was Never the Variable

View on GitHub

Over two weeks I measured four real improvements to this detector, and not one came from the model. They came from how timestamps were bucketed before windowing, how forecast errors were collapsed into a score, how two arms were tuned against each other, and which metric the parameter search was told to maximize. The LSTM itself never changed.

Three of those four ship to production — a grid, a collapse rule and a threshold are what decide whether a reading gets called an anomaly, and they are as much the detector as the network is. The fourth exists only at tuning time. What unites them is that each time, the number moved for a different reason than the one I was testing.

Two of the four never left the building; two got out. A result can look real, survive your own review, and still be an artifact of the instrument. What decides whether it reaches a reader is whether re-measuring is cheap enough that you actually do it.

This is a follow-up to the ESA platform build and taking it live on the ISS feed.

Finding 01 — The 23% that wasn’t there

First read: Multivariate forecasting: +23% F0.5
Re-measured: +0.000002

The benchmark I was chasing forecasts six channels jointly where I forecast one at a time. That was the last architectural gap between us, so I closed it. Held-out F0.5 went from roughly 0.41 to 0.50 and I started writing it up.

Then I tuned both arms exhaustively over the same threshold grid. They land at 0.50724 and 0.50724 — a difference of 2 × 10⁻⁶, detecting the same 7 of 25 events. The per-channel arm’s sweep had landed on a worse operating point inside a search space that contained a better one, and 204 trials missed it. I had compared a well-tuned model against a badly-tuned one and called the difference architecture.

Finding 02 — Eighty percent of the gain was a scoring statistic

First read: 10-step horizon: 0.507 → 0.753
Re-measured: Horizon +0.048 · Error averaging +0.198

The benchmark also forecasts ten steps ahead where I forecast one — a (C, H) output head, vector targets, and a rule for collapsing ten error series into the one-per-timestep series the scorer consumes. Built it: 0.507 → 0.753.

But that collapse rule is a scoring-time knob, so one trained model can be scored several ways. first uses only the one-step-ahead error, identical to how an H=1 model is scored, so only the training signal differs. mean averages absolute error across all ten lead times.

Ten-step supervision is worth +0.048. The remaining +0.198 is averaging error across the horizon — a variance-reduction trick on the error series, not a better forecaster. Either number alone supports the wrong conclusion; scoring the same model twice is what separated them.

Finding 03 — The result rested on a coincidence

First read: One joint model per subsystem
Re-measured: Those 6 channels happen to share a clock

Multivariate worked on channels 41–46 and nowhere else. Joint windowing takes a strict timestamp intersection across a group, and ESA channels are never resampled, so each carries its own sampling phase.

channel_70   n=2,945,639   first=2011-03-15 04:48:28.935   median_dt=30.000s
channel_71   n=2,945,368   first=2011-03-15 04:48:12.819   median_dt=30.000s
                                    ^^ 16.1s phase offset

channel_70 ∩ channel_71 = 0 rows

Two channels in the same subsystem and the same group, both on 30-second cadences, phase-offset by 16.1 seconds — sharing not one timestamp. Channels 41–46 all sit on 04:48:09.825 + 30s, a 100% intersection — and they are the one place in the mission where a subsystem and a channel group are the same set of channels. The benchmark authors happened to pick six channels that share a clock, and nothing in the repository recorded that my result depended on it.

Alignment was not the binding constraint, though — fragmentation was. One 29-channel family already had a 981,782-row intersection, shattered across 947,222 joint segments because the segmenter cuts a boundary wherever any member’s segment changes. Resampling onto a common 30-second grid collapsed it to one segment and took joint windows from 31,464 to 981,523.

That grid became the single largest detection gain in the project: 0.507 → 0.797, almost entirely through precision. A data-representation change — no model was touched. It is also not deployed: the serving path cannot load a grouped model, so the live demo still runs one forecaster per channel.

Finding 04 — Wrong diagnosis, right build

First read: The sampler can’t find the ridge
Re-measured: The objective was the wrong question

That grid arm, tuned by the standard 50-sample Ray Tune sweep, scored 0.212. Tuned by a hand-run exhaustive grid, 0.798. The sweep had chosen threshold_z = 1.393, below the entire range the grid sweeps, and a second independently trained arm did the same. Read as architecture rather than tuning, that says the common time grid halved detection quality.

My diagnosis: HyperOpt cannot find this ridge. So I built the grid into the pipeline as a deterministic second stage. The build was right. The diagnosis was wrong.

Most of the gap — 0.212 → 0.756 — was objective mis-specification. The search selected on per-channel segment F0.5 while the report published mission-level corrected event-wise F0.5; across 350 archived trials those correlate at only ρ ≈ 0.65. The optimizer was answering the question I asked it. Fix the double-smoothing on top and Ray Tune reaches 0.798 on its own, exactly the hand-run grid’s ceiling.

So why keep the grid? Not to rescue the sampler — it is worth +0.015, not the +0.6 the original diagnosis implied. It stays because a 50-sample Bayesian sweep explores a different set of points on every run, and a grid returns the same answer every time. Two arms tuned by different draws are not being compared, which is exactly how Finding 01 happened.

What I built so it stops happening

All four share a root cause: the operating point was chosen by a human. Run a grid, read the “optimum sits on a GRID EDGE” warning, widen the ranges by eye, re-run. That loop ran four times while measuring the horizon result alone.

It is a driver now, running inside the tuning job. It sweeps, detects which axis sits on an edge, extends that axis, and refuses to return a configuration whose optimum is still a lower bound. The acceptance test was the manual loop it replaces: re-running the hand-widened sweep through it produced best_score bit-for-bit equal to the human’s 0.6249993599795226, at threshold_z = 14.0 where I had typed 15.0. Every config it emits now records how the choice was made — because the benchmark report had been asserting “per-subsystem Ray Tune HPO” on rows whose configs came from a hand-run grid, false for every horizon row I had published.

The threshold nobody had to pick

After four corrections, here is the one that survives. ESA-ADB benchmarked around forty algorithms and found Telemanom-ESA-Pruned the best for this mission at 0.786. That result rests on an error floor of 0.007 the authors call “arbitrarily selected” and “probably not optimal,” with the caveat that the approach “is highly parametrised and the selected thresholds may not be optimal for other missions.” That is the paper setting the bar, telling you plainly that a human typed the number that got it there.

That is the step a tuning layer exists to remove. Mine selects (threshold_z, min_error_value) per subsystem by exhaustive grid on the first 60% of the test half, and reports on the untouched final 40%. Nothing is typed in, and the procedure that produced this row is the one that would run on a mission it has never seen.

The claim is the procedure, not the margin. A hand-picked floor scores 0.786 and a searched one scores 0.797, with the same off-the-shelf model on both sides — the operating point that used to require an engineer’s judgment about this particular mission now falls out of a search whose ranges and slices are disclosed, and the selection never touches the reported slice. Strip the floor out and that same detector scores 0.001. The threshold and the floor do nearly all the work, which is this article’s thesis from the other direction.

Why this is platform work

The detector is the same off-the-shelf Telemanom LSTM it was on day one. What two weeks produced is a measurement instrument I trust, and it moved the published number further than every architecture change combined.

A detector is not a model. It is a resampler, a forecaster, an error-collapse rule and a threshold — plus, at tuning time, an objective and a metric. Every one of those sits between a change and the number you read off, and any of them can move that number further than the model does. Compare two models tuned differently and you have measured your tuner. Stack two wins without checking and you have assumed a linearity nobody verified. A platform’s job is to hold all of it fixed, so the only thing still varying is the thing you meant to change.

Code, full evaluation protocol, and every number above: github.com/loganrudd/spacecraft-telemetry-anomaly-detection