# OOD Extremes in Fast Neural Weather Models: A Candidate Research Agenda **Scope**: Detection/generalization to unseen-severity precursor patterns (Q1) and training strategies across severity levels (Q2), for fast ML weather emulators (GraphCast/GenCast, Pangu, FourCastNet, AIFS, Aurora, NCAR MILES-CREDIT) trained on ~40 years of ERA5. **Posture**: mechanistic, current (2024–2026 literature), and deliberately unsparing — including toward my own proposals. --- ## 1. The problem, restated, and why it resists easy fixes Fast neural weather models are trained to minimize an expected loss (almost always some flavor of MSE/latitude-weighted L2, sometimes with a spectral or perceptual term) over empirical samples drawn from one ~40-year realization of Earth's weather (ERA5). Three facts compound into a genuinely hard problem, not just an engineering gap: 1. **Rare events are rare by definition, and reanalysis is a single historical draw.** A "100-year event" has an expected count of ~0.4 occurrences in a 40-year record, at a *specific location*. Aggregated globally there are more tail samples, but they are heterogeneous (a Category 5 Pacific typhoon and a European derecho do not share a loss landscape), so the *effective* sample size for any specific hazard-type tail is tiny. This is a fundamental statistical fact, not a data-engineering oversight — you cannot fix it by "collecting more ERA5." 2. **Squared-error training provably regresses toward the conditional mean under a multimodal or skewed conditional distribution.** When the true conditional distribution of the next state given the current state is a mixture (say, "storm intensifies further" vs "storm weakens," or "ridge holds" vs "ridge breaks catastrophically"), an MSE-optimal deterministic predictor outputs something close to the probability-weighted mean of the modes — which for rare, high-variance tails is neither mode, and is systematically biased low relative to the true extreme. This is not a training-data-volume problem; it is baked into the loss function's optimum, and it gets *worse* with longer autoregressive rollout because the smoothed state is fed back in as the next input (compounding blur), which is exactly the failure mode operationally reported for GraphCast/Pangu-class models and documented mechanistically in EVT-based analyses (e.g., ExtremeCast, arXiv:2402.01295, proves this bias formally for symmetric losses). 3. **There is no ground truth for the genuinely unprecedented.** For anything worse than what has been observed, "correct" is not empirically knowable — we only have physical plausibility arguments (conservation, balance relations, thermodynamic scaling) and statistical extrapolation (EVT) as substitutes for held-out labels. This makes Q1/Q2 unusual among ML problems: you cannot simply hold out a test set that contains the answer you actually care about, because if you had it, it wouldn't be unprecedented. The three facts interact adversarially: the model most needs to extrapolate exactly where its training signal (MSE against a thin, single-realization tail) is least informative, and we are least able to check whether it succeeded exactly where it matters most. Any credible research agenda has to (a) attack the training-signal problem (Q2), (b) attack the extrapolation/flagging problem (Q1), and (c) — the part that is usually glossed over — build honest evaluation machinery that doesn't quietly assume away fact #3. Section 3 treats that as a first-class deliverable, not an afterthought. A useful sub-distinction that the recent "gray swan" tropical-cyclone literature makes concrete (a FourCastNet study that removed all Category 3–5 TCs from ERA5 training and tested on 2018–2023 Category 5s; PMC12130898): there appear to be (at least) two qualitatively different kinds of "OOD extreme": - **Magnitude-OOD within a known regime** — a stronger instance of a dynamical regime the model has seen plenty of (e.g., a bigger version of a mid-latitude cyclone). The cited study found *some* transfer here: a model trained without strong TCs in one basin still showed skill on strong TCs in that *same* basin. - **Regime-OOD** — the event requires dynamics/balances the model has not learned at all (the same study found essentially zero transfer from extratropical-cyclone training examples to tropical-cyclone intensity, and found gradient-wind balance violated in the outputs). No amount of loss reweighting fixes this; the network has never been shown the relevant physics. This distinction should organize expectations for everything below: the approaches in this document can plausibly move the needle on magnitude-OOD; none of them credibly solve regime-OOD, and I say so explicitly per-approach and again in Section 5. --- ## 2. Candidate approaches (7 total) ### Q1 — Detection / generalization (3 approaches) #### Q1.1 — Latent-manifold novelty scoring as an operational "unprecedented" flag **Mechanism.** Fit a density/novelty model (Mahalanobis distance in a PCA-reduced space, a normalizing flow, or a deep-SVDD-style one-class model) on the emulator's *own* internal representation — encoder embeddings, or the processor's hidden state at each autoregressive step — using only training-distribution rollouts. At inference, score every initialization *and every step of the rollout* (extremes emerge mid-trajectory, not just at t=0). When the score crosses a calibrated threshold, emit an explicit "extrapolation regime" flag rather than silently returning a point forecast, and trigger a fallback (widen ensemble spread, hand off to a physical-model ensemble, inflate downstream uncertainty). **Why it might work.** This reframes the ask from "predict the unseen magnitude correctly" (hard) to "recognize you're off-manifold" (comparatively easy — novelty/anomaly detection is a mature ML subfield with strong results in far higher-dimensional, messier domains like autonomous-driving perception). It doesn't require the backbone to be accurate in the tail, only self-aware that it's there. **Data/compute/eval.** Cheap: reuses existing trained-model forward passes; one-time indexing of ~40 years of latent activations; a flow/GMM trains in GPU-hours, not days. Eval: check the score is monotonic in actual outlier-ness using moderate held-out extremes, and check it correlates with genuine skill degradation (not just superficial novelty) — e.g., against the leave-severe-out benchmark in Q1.3. **Novel vs. established.** OOD/novelty detection (isolation forests, Mahalanobis, deep SVDD, flow-based density, k-NN-in-embedding) is 2015–2023-vintage, well-established ML. I did not find published work applying it systematically to autoregressive weather-emulator rollouts as an operational "unprecedented event developing" alarm — that packaging is the novel part. **Caveats.** (1) The emulator's latent space was optimized for MSE regression, not novelty preservation — it may have already smoothed away exactly the structure that would make an incipient extreme look "weird" (the detector inherits the backbone's blind spots). (2) 40 years of ERA5 has a real warming trend; "distance from the training manifold" partially reflects secular climate drift, not event rarity, and needs detrending or it will cry wolf on ordinary climate change. (3) False-alarm/missed-detection tradeoff needs real calibration against a cost function; an alarm that fires constantly is useless operationally. **Promise: 3/5.** Cheap, directly answers the literal ask ("flag, don't silently regress"), but produces a warning, not a better forecast, and the backbone-inherited-blind-spot risk is real and untested. --- #### Q1.2 — Generative-ensemble spread + critical-slowing-down diagnostics as a precursor signal **Mechanism.** Run a generative ensemble (GenCast-class diffusion, or a deep ensemble of independently trained/perturbed MILES-CREDIT checkpoints) and track, per lead time: (i) spread/variance of key scalar diagnostics (central pressure minimum, IVT, max 10 m wind, CAPE); (ii) the *rate* of spread growth (fast growth = flag); (iii) classic dynamical-systems early-warning-signal (EWS) statistics — rising variance, rising lag-1 autocorrelation, rising skewness — computed across ensemble members or along the rollout, borrowed from the critical-slowing-down (CSD) literature used for climate tipping points (PNAS 2021 deep-learning-EWS work; ESD 2024 tipping-point review). **Why it might work.** This is not starting from zero: ECMWF's operational Extreme Forecast Index / Shift-of-Tails (EFI/SOT) already formalizes "how far is this ensemble from model climatology" as an operational extreme-risk index, and generative ensembles are reported (DeepMind's own GenCast material) to have better-calibrated spread/tail behavior than deterministic models — so an ML-native, cheaply-scalable EFI/SOT analog is a low-risk extension of an already-validated operational concept. The EWS layer on top is the speculative addition. **Data/compute/eval.** Requires a generative-ensemble backbone as a prerequisite (not built from scratch). Compute for large member counts (GenCast reports ~8 min/15-day-ensemble on TPU; comparable order on Derecho GPUs) is real but tractable. Eval: back-test against catalogued rapid-intensification/bomb-cyclogenesis events with a matched case-control design (developing vs. non-developing near-analogs) to see if EWS statistics actually rise before onset above noise. **Novel vs. established.** EFI/SOT: established (2000s, operational). CSD/EWS for slow bifurcating systems: established (10+ years). Transplanting EWS specifically to fast, chaotic, few-day-lead synoptic extremes (TC rapid intensification, bombogenesis) via a neural generative ensemble: to my knowledge unexplored — a real but speculative cross-pollination. **Caveats.** CSD theory assumes a slowly-forced approach to a bifurcation with timescale separation from noise; TC rapid intensification and bombogenesis are arguably fast instability-growth phenomena (WISHE, baroclinic growth) rather than slow tipping points, so the EWS statistics may simply not apply — this is the shakiest link in the chain (flagged again in Section 5). Also, ensemble-spread signals from generative models can be *structurally* miscalibrated (matching marginals while getting cross-variable/cross-time correlations wrong), so a "spread is growing" signal could be a generative-process artifact rather than genuine dynamical divergence. **Promise: 3/5.** The EFI/SOT-grounded half is solid; the CSD/EWS-for-fast-weather half is a genuine, cheap-to-pilot experiment but should not be assumed to work — treat as a focused feasibility study, not a program. --- #### Q1.3 — Physically-invariant, regime-normalized inputs (extending "climate-invariant" transforms to severity generalization) **Mechanism.** Beucler et al.'s climate-invariant framework (Science Advances 2024; arXiv:2112.08440) rescales/nondimensionalizes ML inputs (e.g., relative humidity instead of specific humidity, to strip out most of the exponential T-dependence of saturation vapor pressure) so a parameterization generalizes across climates it wasn't trained on. Apply the same idea along a *different* generalization axis — severity, not external forcing: use potential-vorticity/potential-temperature coordinates instead of raw pressure-level fields for extratropical systems; use nondimensional TC descriptors (normalize by potential intensity, use gradient-wind-balance-consistent wind/pressure relationships à la parametric Holland wind models) so a stronger storm looks like a *scaled* version of a weaker one along an axis the network already has to learn, instead of requiring it to invent new nonlinear behavior in raw physical units. Test directly against the gray-swan leave-out protocol (remove Cat 3–5 TCs from training, evaluate on held-out Cat 5s) with and without the invariant transform, and explicitly map out which "directions" of extremeness are magnitude-OOD (plausibly learnable) vs. regime-OOD (see Section 1) before investing further. **Why it might work.** The gray-swan study's own diagnosis was gradient-wind-balance violation and clean *within-regime* (same-basin) transfer but failed *cross-regime* transfer — exactly the signature climate-invariant-style feature engineering is designed to fix: give the network a coordinate system in which "more extreme" is a scaling operation it has already had to learn from the bulk of the distribution, rather than an unconstrained extrapolation in raw units. **Data/compute/eval.** Cheap-to-moderate: reuses existing ERA5 pipeline plus preprocessing; needs several retraining/fine-tuning runs per phenomenon class (order 10–50 GPU-days each on Derecho, small relative to full pretraining). Eval: leave-severe-out per hazard type (TC intensity, bomb-cyclone deepening rate, AR-by-IVT-percentile) plus physical-consistency diagnostics (gradient-wind and thermal-wind balance residuals) as an accuracy-independent sanity check. **Novel vs. established.** The climate-invariant framework itself is established, but for a different generalization axis (unseen climate) and a different model class (offline single-column parameterizations). Applying it to severity-generalization in a fully prognostic, autoregressive global emulator is a genuine, testable transfer of the idea. **Caveats.** Phenomenon-specific — TCs, bomb cyclones, ARs, derechos each need their own invariant features; this is a family of bespoke transforms, not a universal fix. Unproven in fully prognostic rollout (vs. offline parameterization): errors in mapping back from invariant space to raw prognostic state could accumulate across autoregressive steps in ways the original climate-invariant papers never had to contend with. **Promise: 4/5.** Mechanistically well-motivated, cheap to prototype, directly falsifiable against an already-published benchmark protocol, and it's the kind of "boring" domain-informed feature engineering that has a real track record elsewhere in physics-ML. Highest-confidence idea in the Q1 set. --- ### Q2 — Training across severity levels (4 approaches) #### Q2.1 — EVT-informed asymmetric loss + tail up-weighting + severity curriculum **Mechanism.** Replace/augment MSE with an asymmetric, EVT-motivated loss that penalizes under-prediction of high-magnitude targets more than over-prediction (ExtremeCast's "Exloss," arXiv:2402.01295, proves MSE is provably biased-low for skewed conditional tails and is not just a heuristic fix). Add importance weighting/resampling proportional to inverse empirical frequency of the target's climatological percentile (focal-loss-style tail up-weighting). Layer a curriculum: train/fine-tune first on the full climatological distribution, then on progressively tail-enriched subsamples, rather than asking the optimizer to fit bulk and tail simultaneously from scratch. Optionally add ExtremeCast's training-free "ExBooster" (ensemble-sampling-based hit-rate boosting) as a cheap inference-time layer on top. **Why it might work.** This is the most direct fix to the *literal, provable* mechanism named in Section 1.2 (MSE regresses to the mean under a skewed conditional). It is not speculative — ExtremeCast reports measurable, published, code-available gains from exactly this mechanism on comparable architectures. **Data/compute/eval.** Cheap: same ERA5 data, modify loss/sampler only, no new data collection; retraining cost comparable to a standard MILES-CREDIT run (days on Derecho, not weeks). Eval: full verification suite, not just tail metrics — POD/FAR/CSI at multiple thresholds, CRPS decomposition and reliability diagrams stratified by severity percentile, *and* an explicit check that bulk skill (ACC/RMSE at climatological-normal percentiles) hasn't regressed (tail-reweighting can trade bulk accuracy for tail accuracy; that trade needs to be visible, not hidden). **Novel vs. established.** Low-to-moderate novelty: this is largely "adopt and transplant an already-published technique (ExtremeCast) into a different backbone (MILES-CREDIT)," plus the added curriculum-ordering piece. **Caveats — be honest about scope.** This reweights *within the observed ERA5 distribution*. It makes the under-sampled-but-observed tail better represented; it does **not**, by itself, manufacture information about severities beyond what ERA5 has ever recorded. It is squarely a fix for "under-representation of the rare tail that IS in the data," not for "generalization beyond the data" — don't oversell it as solving Q1's harder half. **Promise: 4/5.** Cheap, established, immediately actionable with the stated resources (small team, Derecho), directly targets the named mechanism. Honest caveat above should travel with any claim of success. --- #### Q2.2 — Physical rare-event sampling (TEAMS/genealogical-particle-analysis) as a tail-data generator, distilled into the fast emulator **Mechanism.** Adopt a dynamical-systems rare-event algorithm — TEAMS (Finkel & O'Gorman, JAMES 2024/2025) or genealogical particle analysis (Ragone & Bouchet 2018/2021) — run against an actual physical model (a moderate-resolution GCM, or MPAS/CAM at a resolution NCAR can afford to run many times), using resampling/cloning biased toward a chosen extreme-event score function (regional 3-day precipitation total, a TC-intensity index, an AR IVT threshold). This produces many more *physically simulated, dynamically self-consistent* rare realizations than direct simulation would, at a fraction of the compute, while — critically — preserving statistically correct (reweighted) event frequencies rather than injecting arbitrary bias. Use the (properly importance-weighted) output as additional fine-tuning data for the fast emulator. This last step needs care: TEAMS/GPA output is a weighted, family-tree-structured ensemble, not i.i.d. samples, so naive supervised training on it (which implicitly assumes i.i.d. draws) will bias the fine-tuned model unless the weights are handled explicitly (importance-weighted loss, or resampling-with-replacement per the algorithm's own weights to approximate an unweighted tail-enriched set). **Why it might work.** Unlike a generative augmentation model, this method's tail samples are dynamically consistent by construction (they are literally outputs of a real dynamical core), and the reweighting scheme is mathematically justified by large-deviation/rare-event theory rather than being a statistical guess. A 2025 "AI-boosted rare event sampling" line of work (arXiv:2510.27066) even shows ML surrogates can accelerate the sampler itself — a virtuous loop: cheap ML surrogate guides which ensemble members to clone/kill, generating tail data faster, which then fine-tunes the operational fast emulator. **Data/compute/eval.** Needs a dynamical core (MPAS/CAM, available at NCAR) and meaningful *CPU* allocation on Derecho for the resampling ensemble (order of 10²–10³ member-equivalents depending on target rarity — much less than brute force, still a real HPC campaign), then modest GPU compute to fine-tune the emulator. Eval: verify reweighted tail statistics against independent long control runs or known return-period estimates where available; check the fine-tuned emulator's skill on genuinely held-out events; where feasible, cross-check against a second independent physical model to bound "physical-model-specific" tail bias (see next caveat). **Novel vs. established.** TEAMS/GPA are established in the climate-dynamics/large-deviation-theory community for *estimating extreme-event statistics*. Using their output as *training data for a fast ML emulator* is, to my knowledge, not yet published — a genuine and fairly clean cross-pollination opportunity. **Caveats.** The method is bottlenecked by the driving physical model's own ceiling: if the GCM/MPAS configuration used cannot produce a truly unprecedented storm (resolution, physics, forcing), the rare-event sampler will efficiently find *that model's own tail*, not necessarily the real atmosphere's — it amplifies existing physical-model extremes, it does not conjure genuinely implausible-for-that-model events. Implementation is nontrivial research in itself (defining a good score/reaction-coordinate function per hazard type, correct weight bookkeeping) — expect a multi-quarter methods effort before any training payoff. **Promise: 4/5.** The most physically principled of the training-side ideas — real dynamics, real reweighting theory, not statistical invention. Not a 5 because of genuine implementation complexity, the physical-model-ceiling caveat, and because it's a bigger organizational/compute lift than Q2.1. --- #### Q2.3 — Physically-conditioned amplification: (a) storyline/pseudo-global-warming perturbation vs. (b) generative driver-conditioned synthesis Two mechanistically different flavors worth keeping separate because their credibility differs sharply. **(a) Storyline / "what-if" perturbed physical simulation.** Rerun historical high-impact-but-not-record events through a convection-permitting regional model (WRF/MPAS-A) with physically motivated perturbations to boundary/initial conditions (warmer SSTs, higher moisture per Clausius-Clapeyron scaling, shifted steering flow) — the established storyline/pseudo-global-warming methodology already used in event-attribution science (e.g., amplification studies of Harvey/Florence-class events). Use the resulting amplified-but-dynamically-real fields as additional supervised fine-tuning targets. - *Why it might work*: genuinely dynamically consistent (comes out of real governing equations with perturbed forcing, not a statistical model), and the perturbation itself is physically defensible when grounded in a known scaling relationship (thermodynamic moisture scaling is defensible; arbitrary multiplication is not). - *Cost*: the most expensive item in this whole document — each storyline is a real convection-permitting regional simulation (order 10s–100s of GPU/CPU-days depending on domain/duration); realistically a curated handful of high-value case studies, not a bulk-data strategy. - *Promise: 3/5* — physically the most credible route to genuinely novel severities, but too expensive to scale beyond boutique case studies, so its contribution to a global emulator's tail will be a patch, not a systematic fix. **(b) Generative, driver-conditioned synthesis.** Train a conditional diffusion model on physical drivers (wind shear, ocean heat content, storm development stage) to synthesize additional extreme fields/imagery — as in a 2026 preprint using a context-conditioned diffusion model to correct ~400× class imbalance in extreme-vs-normal storm imagery (arXiv:2603.06782). - *Why it might seem to work, and why I'm skeptical*: the generator is trained on the same rare historical distribution it's meant to augment beyond — it is fundamentally an interpolator over the observed record, useful for correcting *sample-imbalance* (which is a real, demonstrated use case in the cited paper) but not demonstrated, and not obviously capable of, manufacturing genuinely unprecedented severity. Synthetic fields that look statistically plausible (match learned marginals) are not guaranteed to be dynamically consistent when unrolled prognostically; without independent physical-consistency checks (conservation, balance relations) before use, you risk distilling the generative model's own artifacts into the operational emulator (garbage-in-garbage-out one level removed). - *Promise: 2/5* — reasonable as an imbalance-correction tool for auxiliary tasks (e.g., a satellite-image extreme-storm classifier); I would flag using it to teach a prognostic global emulator "new" physics as closer to a dead end for the stated goal (see Section 5). --- #### Q2.4 — Multi-fidelity transfer from large ensembles and storm-resolving simulations (attacking the ERA5 representativeness ceiling directly) **Mechanism.** ERA5 is a *single* 40-year realization of internal variability at ~31 km resolution. Two distinct deficiencies follow, and this approach targets both: (i) many physically plausible tail realizations simply never occurred in this one historical draw (a sampling problem), and (ii) ERA5's resolution structurally cannot represent true sub-grid intensity for convective/mesoscale extremes (a representativeness ceiling, not a sampling problem — no amount of reweighting the labels you have fixes labels that are systematically too weak). Address (i) by incorporating a large ensemble (e.g., CESM2-LENS, ~100 members × decades, publicly available, same GCM family as MPAS/CAM) as auxiliary pretraining/fine-tuning data — many more internal-variability tail phasings of a physically similar climate system, at the cost of needing bias-correction/domain-adaptation to keep GCM mean-state biases from leaking into an ERA5-anchored operational product. Address (ii) by mining or commissioning km-scale convection-permitting MPAS/CAM output as auxiliary high-resolution "ground truth" specifically for the intensity tail, via a downscaling/super-resolution-style transfer architecture (related to existing NCAR work like CAMulator and the generative-downscaling literature). **Why it might work.** This is the least glamorous but arguably highest-leverage lever: it enlarges effective tail sample size using data NCAR already has (LENS) or can generate more cheaply than field campaigns (MPAS is operational at NCAR), rather than relying on a loss-function or architectural trick to conjure information from a fixed, thin record. It also directly names a mechanism (the representativeness ceiling) that the other six approaches mostly cannot touch. **Data/compute/eval.** LENS acquisition is essentially free (public, already exists) but incorporating it at scale is a substantial additional training-compute lift (potentially comparable in order to a meaningful fraction of the original ERA5 pretraining cost) — the largest compute ask in this document if pursued at full scale, though scopeable down to a targeted hazard subset. Km-scale MPAS auxiliary data requires resolution-mismatch reconciliation (31 km vs. ~3 km) — a nontrivial architecture problem, though NCAR likely has archived storm-resolving output to mine before commissioning new runs. Eval: does fine-tuned-on-LENS-then-ERA5 skill on held-out real extremes beat ERA5-only training at matched compute; does it introduce GCM-mean-state artifacts detectable via bias diagnostics against ERA5 climatology. **Novel vs. established.** "Use more/better data" is the least novel idea in ML generally; using multi-model/multi-fidelity geophysical simulation ensembles specifically to close the extreme-event gap in a *deployed fast emulator* (as opposed to using them for climate-statistics estimation, which is standard) is not something I found conclusively demonstrated in the literature as of this writing — moderate-to-high novelty in the targeted application, low novelty in the underlying idea. **Caveats.** Domain adaptation across simulators (GCM-native statistics → ERA5-anchored operational statistics) is itself an unresolved, nontrivial ML research problem, arguably not much easier than the original OOD challenge — this approach partially trades "extrapolate in severity" for "adapt across simulator domains," which is a different but comparably hard problem, not a free lunch. **Promise: 4/5.** Cheapest new-data-per-unit-effort (LENS already exists), targets a real and under-discussed mechanism (representativeness ceiling, single-realization sampling), but real risk of GCM-bias leakage if domain adaptation isn't done carefully, and a large compute lift if pursued at full scale. --- ## 3. Evaluating any of this honestly with no ground truth for the truly unprecedented This is the crux, and it deserves a direct answer rather than a gesture. Propose a **three-tier evaluation ladder**, ordered from cheapest/least-realistic to most expensive/most-realistic, plus a ground-truth-*free* axis that runs orthogonal to all three tiers. **Tier 1 — Idealized dynamical-systems testbeds with computable extreme statistics.** Quasi-geostrophic models, Lorenz-96, intermediate-complexity GCMs (SPEEDY/PUMA). Cheap enough to brute-force simulate the true tail directly, or in some idealized cases derive analytic/GEV limiting distributions. Use as a fast "unit test": does a candidate Q1/Q2 method demonstrably close a known, computable extrapolation gap in a toy system before spending real compute on it in a full model. Low realism, fast iteration, genuinely known ground truth. **Tier 2 — Perfect-model OSSE (Observing System Simulation Experiment).** Treat a real physical model (MPAS/CAM/CESM) as "truth." Generate a long synthetic run (or use a rare-event sampler, Q2.2, to generate a tail-enriched synthetic archive). Train the fast emulator only on a "training-length" segment (e.g., a synthetic 40-year window) and test on synthetic extremes that are more severe than anything in that window (either occurring later in a longer control run, or algorithmically sampled to exceed the training window's max severity). This sidesteps the "no ground truth for the real 100-year storm" problem by substituting a "we do have ground truth in a synthetic universe, by construction" experiment — it is the single most rigorous internal validation harness available, and should be the mandatory gate before any of the Q1/Q2 methods above are claimed to work, let alone deployed. Caveat to state loudly: this only validates that a method closes the gap *in a world where the physical model is the truth and it's stationary/known* — it does not guarantee the same technique transfers to the real, imperfectly known atmosphere. It is necessary, not sufficient. **Tier 3 — Real-world leave-severe-out protocols.** Systematize the gray-swan-TC-style design (remove all Cat 3–5 TCs, or all top-percentile ARs, or all bomb-cyclogenesis events, from ERA5 training; evaluate on held-out real severe events) across multiple hazard types as a standard, community-shared benchmark suite (a natural companion to WeatherBench-style leaderboards, but curated specifically for held-out-severity extrapolation). Be explicit about what this does and doesn't test: it tests extrapolation-within-record (magnitude-OOD), *not* generalization to a genuinely novel regime with no historical analog at all (regime-OOD) — conflating the two overclaims what the benchmark shows. **Ground-truth-free axis, orthogonal to all three tiers — physical self-consistency.** Conservation of mass/moisture/energy, gradient-wind and thermal-wind balance residuals, and known physical bounds (precipitation cannot exceed what column water vapor can plausibly supply in one timestep; wind fields implying superbalanced/unbalanced flow beyond physically observed ranges) are checkable *without knowing the true answer*. A forecast that violates these is objectively wrong regardless of whether the "true" outcome is known — this is the one evaluation axis genuinely suited to the unprecedented-event problem, since self-consistency is always checkable even when correctness against an observation is not. NCAR's own CREDIT conservation-scheme work (Sha, Schreck, Chapman, Gagne et al., 2025; github.com/NCAR/CREDIT-physics-run) is directly relevant infrastructure here and should be the backbone for this axis, not a bolt-on. **Two supporting, non-substitute tools.** (i) EVT/GEV-POT statistical backtesting: fit a GEV/GPD tail model to withheld, less-extreme folds and check whether the ML model's implied event probabilities are statistically consistent with the EVT extrapolation — a principled, if assumption-laden, comparison target that doesn't require having observed the exact event. (ii) Prospective, forecaster-in-the-loop elicitation: going forward operationally, have RAL/NWS forecasters record real-time subjective "is this unprecedented" judgments and confidence for actual events as they occur, building a slowly-accumulating, small-but-genuinely-true labeled record — the only source of true *future* ground truth, explicitly a multi-year effort, but the honest long-run validation for any "flag as unprecedented" claim (Q1.1/Q1.2). --- ## 4. Ranking and what I'd actually pursue first (small team, Derecho GPU time) Promise scores from above: Q1.3 (4), Q2.1 (4), Q2.2 (4), Q2.4 (4), Q1.1 (3), Q1.2 (3), Q2.3a (3), Q2.3b (2). Four approaches tie at "4," so sequencing logic matters more than the raw score. My actual first move with a small team and Derecho *GPU* time (note: Q2.2 and Q2.4 lean CPU/data-ingestion-heavy, and Q2.3a needs a regional-model campaign — all three are bigger organizational lifts before any payoff can even be measured): **First: build the Tier-3 evaluation harness (Section 3) — leave-severe-out benchmark + physical-consistency diagnostics — as shared infrastructure.** This isn't optional preamble; every other claim in this document is unverifiable without it, and it's cheap (reuses ERA5, existing CREDIT conservation-diagnostic code, standard verification metrics). **Second, in parallel, run the two cheapest, fastest, most falsifiable methods against that harness:** 1. **Q2.1 (EVT-informed asymmetric loss + tail up-weighting + curriculum).** Cheapest possible experiment: no new data, no new simulation infrastructure, code (ExtremeCast) already public to adapt into MILES-CREDIT. Fast iteration cycle (days, not months, per experiment on Derecho). Directly attacks the one *provable* mechanism named in the problem statement. Even a negative or mixed result (tail improves, bulk regresses) is immediately actionable and cheaply obtained — ideal first experiment for a small team building momentum and tooling. 2. **Q1.3 (climate-invariant/regime-normalized inputs + leave-severe-out benchmark).** Similarly cheap (preprocessing + several fine-tuning reruns), and it directly reproduces-and-extends an already-published, falsifiable protocol (the gray-swan TC study), which gives a credible external baseline to beat or fail against rather than inventing a new, unvalidated metric from scratch. It also has a built-in "know when to stop" diagnostic: mapping out magnitude-OOD vs. regime-OOD directions tells the team early which future investments (Q2.2, Q2.4, Q2.3a) are worth the bigger lift. **Why not start with Q2.2, Q2.4, or Q2.3a first, despite equal promise scores:** all three require either a CPU-heavy dynamical-core rare-event campaign (Q2.2), a large new data-ingestion and domain-adaptation effort (Q2.4), or a bespoke high-resolution regional simulation campaign (Q2.3a) — real infrastructure investments that should be justified by evidence from the cheap Phase-1 experiments and validated internally via the Tier-2 perfect-model OSSE (Section 3) before committing a small team's limited HPC allocation to them. Sequence: harness → cheap wins (Q2.1, Q1.3) → OSSE-validate the expensive ones (Q2.2, Q2.4) → commit real compute to whichever survives. --- ## 5. Likely dead ends / overhyped — including my own ideas Being direct, in decreasing order of confidence that the idea is weak: - **Q2.3b, generative driver-conditioned synthetic-extreme generation, as a tail-*generalization* tool (not as an imbalance-correction tool).** The clearest candidate dead end in this document for the stated goal. It is trained on the same rare historical record it's meant to extrapolate beyond, so structurally it can smooth out sample imbalance but has no principled route to inventing genuinely unprecedented dynamics; unrolled prognostically, statistically-plausible-looking synthetic fields are not guaranteed dynamically consistent, and distilling them into the operational emulator risks teaching it the generator's artifacts rather than physics. Keep it, if at all, scoped narrowly to auxiliary classification/imbalance tasks — not as a route to teaching the prognostic emulator new physics. - **Q1.2's critical-slowing-down/EWS layer specifically** (as distinct from the EFI/SOT-grounded ensemble-spread half, which is solid). CSD theory assumes a slow, externally forced approach to a bifurcation with timescale separation from noise; TC rapid intensification and bombogenesis look more like fast instability growth than slow tipping-point approach, so there's a real chance the EWS statistics (rising variance/autocorrelation) simply don't clear the noise floor at 6-hourly synoptic timescales. Treat as a bounded, quick pilot — if the case-control back-test doesn't show a clean signal in the first pass, drop the EWS-specific framing and keep only the established EFI/SOT-style spread monitoring. - **"Just scale the model" as an implicit assumption.** Not one of the 7 proposals above, and deliberately excluded, but worth naming because it's a fashionable 2024–2026 foundation-model-for-Earth-system framing (Aurora, and general large-model-for-weather discourse): bigger models trained with the same MSE loss on the same ERA5 distribution have no strong mechanistic reason to fix a tail-representation problem that is a training-*signal* problem (loss function shape, data distribution), not a capacity problem. Expect scale to lower bulk RMSE while leaving the same qualitative regression-to-the-mean behavior in the tail. Low promise (2/5) as a standalone fix; worth stating explicitly since it's the default assumption many stakeholders bring. - **Q2.3a, storyline/pseudo-global-warming perturbation, as a *scalable systematic* fix** (though it remains a real 3/5 as a boutique tool). The physical credibility is genuinely high, but the cost structure means it can only ever cover a curated handful of case studies — treating it as a general solution to the tail-training problem (rather than a valuable patch for a few flagship hazards) would be overreach. - **A general meta-caution about the entire Q1 slate.** None of Q1.1–Q1.3 make the model *correctly forecast* an unprecedented event; at best they (a) raise a flag, or (b) narrow the extrapolation gap for magnitude-scaling within an already-known regime. The gray-swan study's own finding — clean within-regime transfer, ~zero cross-regime transfer — suggests a real ceiling exists for genuinely novel-regime events that no loss/architecture/input trick will cross, because the network has never been shown the relevant dynamics at all. The single most important non-technical conclusion of this document: fast neural weather models should be deployed with an explicit, *tested* handoff protocol to physical models (ensemble NWP, on-demand storm-resolving simulation) for flagged out-of-regime cases, rather than marketed as handling arbitrary unprecedented severity autonomously. Any research agenda here should budget for building and validating that handoff, not just for shaving the ML model's tail error. --- ## Sources consulted - [ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast](https://arxiv.org/abs/2402.01295) (arXiv:2402.01295) - [Can AI weather models predict out-of-distribution gray swan tropical cyclones?](https://pmc.ncbi.nlm.nih.gov/articles/PMC12130898/) (PMC) - [Forecasting the Unseen: AI Weather Models and Gray Swan Extreme Events](https://climate.uchicago.edu/insights/forecasting-the-unseen-ai-weather-models-and-gray-swan-extreme-events/) - [Bringing Statistics to Storylines: Rare Event Sampling for Sudden, Transient Extreme Events (TEAMS)](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2024MS004264) - [Rare Event Sampling for Moving Targets (JAMES 2026)](https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2025MS005456) - [AI-boosted rare event sampling to characterize extreme weather](https://arxiv.org/html/2510.27066v1) - [Climate-Invariant Machine Learning](https://arxiv.org/abs/2112.08440) (Beucler et al., Science Advances 2024) - [GenCast predicts weather and the risks of extreme conditions — Google DeepMind](https://deepmind.google/blog/gencast-predicts-weather-and-the-risks-of-extreme-conditions-with-sota-accuracy/) - [Power Ensemble Aggregation for Improved Extreme Event AI Prediction](https://arxiv.org/pdf/2511.11170) - [NCAR/CREDIT-physics-run — mass and energy conservation schemes](https://github.com/NCAR/CREDIT-physics-run) - [NCAR/miles-credit](https://github.com/NCAR/miles-credit) - [Physics-Informed Diffusion Model for Generating Synthetic Extreme Rare Weather Events Data](https://arxiv.org/abs/2603.06782) - [Deep learning for early warning signals of tipping points (PNAS 2021)](https://www.pnas.org/doi/10.1073/pnas.2106140118) - [Tipping point detection and early warnings in climate, ecological, and human systems (ESD 2024)](https://esd.copernicus.org/articles/15/1117/2024/)