# Concept Note — Trustworthy AI Weather Prediction for High-Consequence Extremes

*A one-page funding concept · NSF NCAR / Research Applications Laboratory · draft July 2026*
*Target sponsors: NOAA (EPIC / R2O / weather program) · DOE (ASCR — ML & UQ for Earth systems)*

---

## The opportunity, and the problem underneath it

AI weather models (GraphCast, GenCast, ECMWF's AIFS, NCAR's MILES-CREDIT) are now operational, open-weight, and fast — but a 2026 *Science Advances* study (Zhang et al.) shows they are **systematically worst exactly where the stakes are highest**: physics-based models still beat them on record-breaking heat, cold, and wind, and the AI–physics gap *widens* as an event grows more extreme. This is not a tuning problem; it follows from training a squared-error model on a single 40-year record in which 100-year events are, by definition, almost absent. The result is an operational trust gap: a forecaster, an emergency manager, or an acquisitions officer has **no reliable, neutral way to know when to trust a fast AI forecast of a dangerous, unprecedented event — and when not to.**

## The approach — build the trustworthy *system*, not a better guess

Our honest finding (from a paired expert/red-team analysis) is that trying to make the emulator itself predict the unprecedented is largely unreachable from the available data. The fundable, achievable goal is different and better:

> **A fast-AI weather system that knows when it is out of its depth, hands off to physics, and can be *proven* to do so.**

Three concrete deliverables, each independently useful:

1. **An extreme-event verification benchmark and service** — a trap-resistant, held-out-severity evaluation (leave-severe-out at the storm-lifecycle level, a physics-model baseline, full-distribution proper scoring, physical-consistency diagnostics), built on **MET/METplus** and the **BEACON AI testbed**. The neutral, public reference for "how good is any model in the tail."
2. **A tested physics-handoff protocol** — an operational trigger (novelty / ensemble-spread / skill-degradation detection) that flags out-of-distribution events and hands off to a physics ensemble (**MPAS / WRF**) or a human forecaster, with the *trigger's own false-alarm rate* validated against a real cost function.
3. **A focused research arm** — the two cheapest, most falsifiable ML experiments (EVT-informed tail loss; regime-normalized inputs as a diagnostic) run against deliverable #1, plus cheap massive-ensemble return-period statistics — advancing the science without overclaiming.

## Why NSF NCAR / RAL, uniquely

This system requires four capabilities that almost no other single organization holds together: the **emulator** (MILES-CREDIT + CREDIT conservation diagnostics), the **physics fallback** (MPAS/WRF), the **verification standard** (MET/METplus + BEACON, already NOAA's UFS verification capability), and **operational forecasters** in the loop. RAL has all four, plus institutional neutrality — the essential ingredient for a *trusted referee* that a commercial vendor cannot credibly supply.

## Alignment with sponsor missions

- **NOAA (EPIC / R2O):** a defensible research-to-operations trust gate for AI in the Unified Forecast System, and a shared extreme-event benchmark for the community. Directly serves NOAA's move toward operational AI and (separately) MPAS-based unified modeling.
- **DOE (ASCR):** the hard ML/UQ core — rare-event evaluation, uncertainty quantification, and physics-ML coupling at leadership-computing scale (Derecho).
- Complements, rather than competes with, vendor AI models — it verifies and safely deploys *everyone's*.

## Ask and milestones (illustrative)

A **phased, ~24-month effort, 3–5 FTE**, structured so early phases de-risk later ones:

- **Phase 0 (months 0–6):** stand up the trap-resistant verification harness (#1) and a first physics-handoff trigger from existing tools (#2). *Both are correct regardless of how the research turns out.*
- **Phase 1 (months 6–15):** run the cheap ML experiments against the harness; pilot the handoff with one operational partner; release the first public extreme-event benchmark.
- **Phase 2 (months 15–24):** gated expansion (rare-event data augmentation, multi-fidelity ensembles) only where Phase-1 evidence justifies; certification pilot with a NOAA and/or DoD/FAA partner.

## Why this survives review

It doesn't overclaim. It targets the *operational* risk (getting blindsided) with mature ingredients, treats the speculative science as a clearly-bounded parallel research arm, and delivers a public good — a trusted extreme-event benchmark — that outlives any single model. **The honest framing is the competitive advantage.**

---

*Basis: `research-ood-extremes-synthesis.md` (with `-ml-expert` and `-devils-advocate` companions) and `deep-dive-01-verification-service.md`. Key anchor: Zhang et al., "Physics-based models outperform AI weather forecasts of record-breaking extremes," Science Advances 2026.*