# Can we make fast neural weather models handle 100-year storms? — an honest synthesis *NSF NCAR / RAL · research assessment · draft July 2026* This is the integrated verdict from two deliberately opposed analyses — a senior ML-for-weather scientist proposing a research agenda ([`research-ood-extremes-ml-expert.md`](research-ood-extremes-ml-expert.md)) and a skeptical devil's-advocate tasked with dismantling it ([`research-ood-extremes-devils-advocate.md`](research-ood-extremes-devils-advocate.md)). Read those for the full argument and citations; this is the reconciliation. --- ## Bottom line up front **The honest answer is uncomfortable but clarifying: for the actual fear — "don't get blindsided by an unprecedented storm" — retraining the fast emulator is *not* the primary lever.** The two analyses, from opposite starting postures, converge on it: 1. **Physics still wins on record-breaking events, and the gap widens with severity.** A 2026 *Science Advances* study (Zhang, Fischer, Zscheischler & Engelke) tested GraphCast, Pangu, and FuXi against ECMWF's physics-based HRES on record-breaking heat, cold, and wind, and found physics "consistently outperforms" the AI models, which underestimate both the frequency and intensity of records — with errors *growing* as the record is exceeded further. That is the empirical confirmation of a first-principles ceiling: MSE-trained models provably regress toward the conditional mean in skewed tails, and that bias compounds under autoregressive rollout. 2. **The load-bearing distinction is magnitude-OOD vs. regime-OOD.** A "stronger version of a storm type the model knows" (magnitude-OOD) has real but *narrow* headroom above baseline. An event needing dynamics the network was never shown (regime-OOD — e.g., the "gray swan" tropical-cyclone study found ~zero transfer from extratropical training to TC intensity, with gradient-wind balance violated) has **essentially zero** headroom for *any* method that only touches the 40-year ERA5 record. No loss function, coordinate transform, or generative augmentation trained on that record can manufacture information that isn't in it. 3. **So the operationally responsible build order leads with engineering, not architecture.** Both analyses — the skeptic emphatically — land on: a **physics-model fallback triggered by novelty/skill-degradation detection**, plus **mature EVT/ensemble post-processing** of existing models, retires the named risk soonest. The seven-approach research menu is legitimate 2–5-year *science* with real (if narrower-than-advertised) value in a few items — but it should run *in parallel to* the fallback engineering and be communicated honestly, never sold as the fix for the operational risk. **The one genuine, under-appreciated edge fast emulators do have** is not accuracy but **ensemble size**: ACE2-class "Huge Ensembles" (a 10,560-year emulator ensemble) surface location-specific maxima beyond the training record as sample count grows. Read precisely, that is a *sampling* advantage (more draws from the model's own learned distribution → better return-period statistics for in-regime events), **not** a *generalization* advantage — it does nothing for regime-OOD and doesn't contradict the physics-wins-on-real-events finding. Valuable, but for statistics, not for forecasting the unprecedented. For NCAR/RAL this is not a discouraging result — it's a **differentiating** one, and it ties directly back to the trusted-referee thesis (project #1): the credible, fundable, public-institution move is to be the institution that *honestly* couples fast ML, physics fallback, and rigorous extreme-event verification — not the one overselling autonomous 100-year-storm skill. --- ## The reconciled scorecard Expert's promise score → devil's-advocate's adjusted score, with the integrated verdict. (5 = strong; scores are judgment, not measurement.) | Approach | Expert | Adjusted | Integrated verdict | |---|:--:|:--:|---| | **Evaluation harness** (leave-severe-out + physical-consistency) | first move | first move | **Build first — but trap-resistant** (see below). Infrastructure, not a hypothesis. Both agree; the skeptic wants it hardened. | | **Physics-model handoff pipeline** (novelty/skill trigger → NWP/human) | Sec-5 bullet | **#1 priority** | The skeptic promotes this to *the* top-line deliverable. It's mostly engineering, uses RAL's existing WRF/MPAS + forecasters, and directly answers the real question. | | **Q2.1 — EVT-informed loss + tail curriculum** | 4 | 4 | **Keep — cheapest first experiment.** Attacks the one *provable* mechanism (MSE tail bias). Caveat that must travel with it: fixes under-representation of the *observed* tail; does **not** generalize beyond the record. And "beats other AI models" ≠ "closes the AI–physics gap." | | **Physical self-consistency diagnostics** (on CREDIT conservation work) | (eval axis) | keep, jointly | Keep — but **only reported jointly with a tail-magnitude metric**: a blurred, mean-reverting forecast can look *more* self-consistent exactly as it fails. | | **Q2.4(i) — LENS large-ensemble sampling** | 4 (bundled) | ~4 | **Do it.** Cheapest new-data-per-effort (CESM2-LENS is free); needs honest GCM→ERA5 bias handling. | | **Q1.3 — climate-invariant / regime-normalized inputs** | 4 | 3 | **Time-boxed diagnostic pilot (weeks).** The skeptic finds a real bait-and-switch: its motivating evidence is the *cross-regime* failure, but the mechanism only plausibly fixes the *within-regime* half that already worked. Value is mapping the magnitude/regime boundary, not a performance win. PV-inversion cost is understated. | | **Q2.2 — TEAMS/GPA rare-event distillation** | 4 | 3 | **Gate behind a 4–8 wk feasibility check.** Physically the most principled tail-data idea, but a research bet on a research bet: the 2026 sampler literature admits it still "replicates the same moderate extreme" for transient events, and importance-weighted → i.i.d.-loss distillation is untested anywhere. Rare-event samplers can *amplify* the driving model's worst bias. | | **Q2.4(ii) — km-scale super-resolution transfer** | 4 (bundled) | 2.5 | **Its own multi-year bet, gated separately.** Real mechanism (ERA5 representativeness ceiling) but it *substitutes* one hard open problem (cross-resolution downscaling) for another. Don't hide it under a 4/5. | | **Q1.1 — latent-manifold novelty flag** | 3 | 2.5 | **Cheap pilot, low expectations.** The failure it must catch is the model collapsing to a *plausible, moderate* state — which a novelty detector scores as *normal*. Near-OOD detection is a documented hard subproblem; far-OOD success rates don't transfer. | | **Q1.2a — EFI/SOT-style ensemble-spread index** | 3 (bundled) | ~4 | **Do it** — a mature, operationally-verified concept (ECMWF EFI/SOT), cheaply reimplemented on a generative ensemble. | | **Q1.2b — critical-slowing-down / early-warning signals** | 3 (bundled) | 1.5–2 | **Bounded 2-week pilot, expect null.** CSD assumes slow approach to a bifurcation; synoptic rapid-intensification is fast instability growth. CSD has documented *systematic false positives even in its home domain*. | | **Q2.3a — storyline / pseudo-global-warming perturbation** | 3 | 3 | **Boutique only** — a handful of flagship case studies. Physically credible but too expensive to scale, and pushes the attribution methodology past its validated forcing range. | | **Q2.3b — generative synthetic-extreme generation** | 2 | 2 | **Kill** as a route to teaching the prognostic emulator new physics. Trained on the same rare record it's meant to exceed → an imbalance-corrector wearing a generalization costume. Fine only for auxiliary classification. | --- ## Evaluating any of this honestly — the five traps There is no ground truth for the genuinely unprecedented, so evaluation is the crux, and it's where good-looking results mislead. A convincing result must clear all five: 1. **Leave-severe-out must exclude at the storm-lifecycle level, not a peak-intensity threshold** — otherwise a storm peaking just under the cutoff leaks near-identical precursor dynamics into the "held-out" test, and "generalization" is memorization of a continuous trajectory. 2. **Perfect-model OSSEs must use a truth model deliberately *different* in physics/numerics** from whatever shaped ERA5 and the emulator's priors — otherwise you only prove the emulator can extrapolate one model family's idiosyncrasies (most NCAR-accessible GCMs are CESM-family, so this trap is live). 3. **Physical self-consistency must be reported jointly with a tail-magnitude metric, never as a standalone pass** — it can improve precisely as the model blurs past the real extreme. 4. **Use full-distribution proper scoring (weighted/threshold-decomposed CRPS), not just conditional-on-extreme POD/FAR/CSI** — the "forecaster's dilemma" (arXiv:2606.21170): a crying-wolf model that over-forecasts severity is never penalized for its ordinary-condition false alarms by threshold-conditional metrics. 5. **The baseline must be a physics model (HRES-class / regional high-res NWP), not just another AI model, and the result must replicate on a second, independently curated hazard type.** Given the *Science Advances* finding, "beat GraphCast" and "beat physics" are different claims; only the second is operationally meaningful. One public leave-severe-out benchmark exists today, so single-benchmark gains risk being overfitting. --- ## What a small team should actually do (reordered) The expert's sequencing was "harness → cheap wins → gate the expensive." The skeptic's amendment, which I adopt, changes the lead: **Phase 0 — the two things that are correct regardless of how the science turns out:** - **(A) The physics-handoff pipeline.** Define a trigger from *existing* tools (EFI/SOT departure, ensemble-spread growth, "did skill degrade on the last N analogous cases"), an SLA for handoff to regional NWP/HRES-class physics or a human forecaster, and validate the *trigger's own false-alarm rate* against a real cost function. This directly answers "don't get blindsided," uses infrastructure RAL already has, and is engineering, not open research. - **(B) The trap-resistant evaluation harness** (the five traps above), built on CREDIT's conservation diagnostics. **Phase 1 — cheapest, most falsifiable science, run against (B):** - **Q2.1** (EVT loss + curriculum) — days-per-experiment, attacks the provable mechanism. - **Q1.2a** (EFI/SOT-style spread index) and **Q1.1** (novelty flag) as cheap pilots feeding trigger (A). - **Q1.3** as a strictly time-boxed *diagnostic* to map the magnitude/regime boundary per hazard. **Phase 2 — gated by Phase-1 evidence and a narrow feasibility check, not funded upfront:** - **Q2.4(i)** LENS sampling (cheapest data lever), then **Q2.2** (only after the transient-event sampler and i.i.d.-distillation questions pass a 4–8 wk check), then **Q2.4(ii)** km-scale transfer as its own multi-year program. **Do not fund:** Q2.3b for generalization; the CSD/EWS half of Q1.2; Q2.3a beyond one flagship case. **Separately, and cheaply: exploit the one real edge — massive emulator ensembles for return-period *statistics*** (ACE2-style), clearly labeled as a sampling tool for in-regime magnitude extremes, never conflated with generalization. --- ## What this means for NCAR / RAL The most important non-technical conclusion is a *positioning* one, and it reinforces the rest of this package: - **The honest framing is the competitive advantage.** Big Tech's incentive is to market autonomous AI skill; the field has just shown, at scale, that these models are *structurally worst exactly where the stakes are highest*. A public institution that says so plainly — and builds the tested physics-handoff and the extreme-event verification to prove it — occupies ground a vendor can't credibly take. This is the same neutrality-as-product thesis as **project #1 (the verification "referee")**, and the OOD-extremes work is a natural flagship application of it. - **RAL is unusually well-positioned to build the *right* thing.** The responsible architecture — fast ML in the bulk, novelty-triggered handoff to physics, EVT post-processing, rigorous held-out-severity verification — needs exactly RAL's combination: MILES-CREDIT (the emulator + CREDIT conservation diagnostics), MPAS/WRF (the physics fallback), MET/METplus + BEACON (the verification), and forecasters in the loop. Almost no other single organization has all four. - **Fundability.** Framed as "trustworthy AI for high-consequence weather — with an honest, tested fallback and a public extreme-event benchmark," this fits NOAA (R2O, operational trust), DOE ASCR (the hard ML/UQ), DoD/FAA (high-consequence operational AI), and NSF — and it's the kind of program that survives peer review *because* it doesn't overclaim. **The sentence to carry out of this:** the goal isn't a neural model that predicts the unprecedented — that's likely unreachable from a single 40-year record — it's a *system* that knows when it's out of its depth and hands off, and that can be *proven* to do so. Building and verifying that is the honest, fundable, and genuinely NCAR-shaped contribution. --- ### Sources (see the two source documents for the full lists) Key anchors: Zhang et al., *Physics-based models outperform AI weather forecasts of record-breaking extremes*, Science Advances 2026 · the "gray swan" tropical-cyclone leave-out study (arXiv:2410.14932) · ExtremeCast (arXiv:2402.01295) · Beucler et al. climate-invariant ML (Science Advances) · Finkel & O'Gorman TEAMS / rare-event sampling for moving targets (JAMES 2024/2026) · ACE2 "Huge Ensembles" (arXiv:2510.08893) · the weighted-potential-CRPS "forecaster's dilemma" (arXiv:2606.21170) · Fort, Ren & Lakshminarayanan near/far-OOD (NeurIPS 2021) · NCAR CREDIT conservation work.