Updated August 15th 2026. The evaluation numbers originally published here were superseded — partly by re-tuning, partly by finding a bug in my own ground truth. They’ve been corrected below, and the full accounting is in the project README’s evaluation section. The engineering notes are unchanged.
Architecture at a glance
ESA telemetry (Parquet, ~225 channels)
→ Ray fan-out preprocessing
→ per-channel Telemanom LSTM (Ray parallel train · Ray Tune HyperOpt Bayesian search)
→ MLflow registry (@champion)
→ FastAPI + SSE stream replay
→ React / Recharts dashboard
Infra: GKE Autopilot + KubeRay · Cloud Run (scale-to-zero) · Terraform
· GitHub Actions + Workload Identity Federation
The problem
Real spacecraft telemetry is a fan-out problem before it’s a modeling problem. The ESA benchmark used here is roughly 225 channels across three missions and 31 GB of sensor data, and the honest baseline — Telemanom, a small per-channel LSTM forecaster — means training and serving hundreds of independent models. The interesting question isn’t “what’s the best model,” it’s “how do you keep hundreds of per-channel models trainable, tunable, and servable without the infrastructure cost exploding.” I used an off-the-shelf model deliberately, so the platform — not the model — was the work.
The architecture
Preprocessing and training fan out across channels with Ray Core on KubeRay. Each channel gets its own Telemanom LSTM (2-layer, hidden_dim 80), trained only on nominal data as a one-step-ahead forecaster, with Ray Tune (HyperOpt Bayesian search) doing per-subsystem hyperparameter search. Models land in an MLflow registry under a telemanom-{mission}-{channel} scheme with a @champion alias for promotion. Serving is a FastAPI app that replays streams over server-sent events, scoring each tick against EWMA-smoothed residuals and a dynamic threshold, with a React/Recharts dashboard drawing ground-truth vs. predicted anomaly bands. It all runs on GCP: GKE Autopilot + KubeRay for batch compute, Cloud Run (scale-to-zero) for the API and MLflow, Terraform for IaC, and GitHub Actions with Workload Identity Federation for CI/CD.
What was hard, and what I decided
Memory under heterogeneous fan-out. Channels range 50–240 MB compressed, and naive fan-out OOM-killed 4 GB Ray workers. Rather than throw bigger machines at it — the cost-explosion trap — I profiled peak RSS per task and bin-packed by actual footprint: pd.Categorical encoding, building arrays with Categorical.from_codes instead of Python lists (per-task peak 2.35 GB → 0.5 GB), dropping defensive copies, and Ray memory= hints with max_calls=1 to force reclamation between tasks. Measure peak RSS before tuning concurrency.
The wrong tool on the hot path. I started with Evidently for drift detection, but its batch reports (~58 ms each) saturated the thread pool at high replay speeds. I swapped to a direct scipy Wasserstein computation (under 1 ms) offloaded with asyncio.to_thread, keeping Evidently for the offline path. A great batch tool can be the wrong choice on a latency budget.
Scale-to-zero makes startup a first-class problem. Cold starts on a 2 GiB Cloud Run instance OOM’d because every champion model loaded via an unbounded asyncio.gather. Fix: deferred background loading behind a bounded semaphore, per-channel bounded queues so one slow channel can’t stall the rest, and a single merged SSE connection. /health returns early with status="loading" while the app already accepts requests.
Credentials that outlive the job. GCP ID tokens expire after an hour, so Ray workers on training runs longer than that hit 403s mid-job. I added an in-flight refresh_mlflow_auth() called once per epoch — a cheap dict lookup when there’s more than 10 minutes left, a refetch only near expiry. Any long-running distributed job that outlives a credential needs refresh, not just auth at startup.
On results, honestly
Because the model is a deliberate baseline, the detection numbers are modest — segment-overlap F0.5 ≈ 0.41 fleet-mean on the held-out final 40%, with 22 of the 31 labeled channels registering any detection at all and five channels above 0.7. I report segment-overlap F0.5 rather than the point-adjust convention common in the SMAP/MSL literature because point-adjust inflates scores by crediting an entire anomaly segment for a single detected point. The project is about the platform and the engineering judgment behind it, and honest evaluation is part of that.
That 0.41 replaces the 0.19 I first published, and the path between them is worth spelling out, because the two changes pull in opposite directions. Re-tuning on a widened threshold search space took 0.19 → 0.43; essentially all of the gain is there. Then correcting the ground truth moved it back down to 0.41. label_timesteps had been silently dropping zero-duration “point” anomalies — a half-open [t, t) interval matches nothing — so those events never made it into is_anomaly. Restoring them added five true segments on channel 41 alone, which means every earlier number, including the original 0.19, was scored against an incomplete and easier label set than the dataset actually poses. A score that goes down because you fixed your own labels is worth more than one that goes up.
