# Deep Dive #1 — The Trusted Referee: an open verification & benchmarking service for AI weather models *NSF NCAR / RAL · strategy working document · draft July 2026* **One line:** Turn MET/METplus + BEACON into the neutral, physics-grounded authority that verifies and certifies the flood of open-weight AI weather models — the one role a corporation structurally cannot hold, and one RAL is ~70% of the way toward already. --- ## Why this is *your* project This maps directly onto your existing work. Your `~/NCAR/devx` workspace already contains MET, and MET/METplus is the substrate this whole initiative extends. You bring exactly the three things it needs: - **Verification domain knowledge** — you already work inside the MET toolchain. - **Software/DevX engineering** — the deliverable is a service + pipelines + a leaderboard, not a science paper. - **Visualization** — your `wrf-viewer` / `webgl_viewer` experience is directly reusable for the public leaderboard and diagnostic dashboards, which are a large part of what makes a verification service *credible and used* rather than a static report. You would not be learning a new domain. You would be pointing tools you already know at the highest-leverage target in the field. --- ## The problem, precisely AI weather prediction crossed from research into operations between 2023 and 2026, and the flagship models are largely **open-weight**: GraphCast, GenCast (Google DeepMind), Pangu-Weather (Huawei), FourCastNet (NVIDIA Earth-2), ECMWF's operational **AIFS**, and Microsoft's **Aurora** (open-sourced Nov 2025). When anyone can run the model, "who has the best model" stops being the interesting question. The blocking question becomes **"whose numbers do we trust, and how do we compare fairly?"** — and the field's tooling for that is visibly behind: - **Deterministic metrics unfairly favor physics models over AI ensembles.** Gneiting et al. (2025) show standard accuracy scores systematically disadvantage probabilistic AI output and propose a "Potential CRPS" to compare fairly. The community has no standardized, tooled implementation. - **Reanalysis is a poor proxy for reality.** WeatherReal (2024) shows ERA5 — used to both *train and verify* most AI models — diverges materially from in-situ observations for near-surface temperature, wind, precipitation, and cloud. Verifying against the thing you trained on flatters the model. - **Global-average scores hide who gets bad forecasts.** SAFE (2025) shows headline metrics mask large regional skill disparities, disadvantaging under-observed regions. - **There is no standard benchmark for ML data assimilation.** DAMBench (2025) exists precisely because one didn't. - **MET/METplus has no publicly documented AI-model verification extension** — the community-standard verification suite, used across NOAA's Unified Forecast System, was built for gridded/point NWP. Net: **operational adoption of AI weather models is blocked on *trust*, not on accuracy.** That's a verification problem, and verification is RAL's home turf. --- ## Why a public institution wins here Verification is a **public good with a trust prerequisite**. A vendor cannot credibly grade its own model; a startup's leaderboard is a marketing surface. The value of a referee is exactly its neutrality and its permanence — two things a for-profit cannot manufacture. NCAR/RAL has: - **MET/METplus** — the community verification standard, jointly developed via the DTC with NOAA and the US Air Force, already the designated verification capability for the UFS. - **The BEACON AI Testbed** — RAL's new (2025–26) initiative under the WISP program, already benchmarking AI vs. physics-based models across 30+ projects spanning government, industry, and international partners. This is the seed of the referee role. - **Institutional standing** — decades of verification science and R2O relationships that a three-year-old company cannot replicate. This is the rare initiative where the public-good framing and the competitive-moat framing are the *same* framing. --- ## What exactly to build — "MET-AI" Four components, sequenced so each is independently useful: **A. AI-native verification metrics (extend MET/METplus).** Add first-class support for the ways data-driven models actually need to be judged: - Proper scoring rules for ensembles (CRPS, Potential/weighted-CRPS per Gneiting) so AI ensembles and NWP ensembles compare fairly. - Spatial / neighborhood / object-based methods aligned to how forecasters actually use output (not just point-to-point RMSE, which can rank AIWP vs. NWP differently than spatial methods do). - Extremes- and impact-weighted metrics (the tail is where AI models are most doubted). - **In-situ truthing**, not reanalysis-only — a QC'd observation layer as ground truth (the WeatherReal insight, productized). - Regional / equity-aware reporting so skill disparities are visible, not averaged away. **B. A public, versioned benchmark & leaderboard.** Anyone can submit a model or a forecast set; results are reproducible and permanently versioned (think WeatherBench 2's role, but operationally oriented and institution-backed). This is where your visualization skills carry: an interactive, trustworthy diagnostics dashboard is what makes a leaderboard *authoritative* rather than ignored. **C. A "referee" certification path.** A defined, documented evaluation an operational adopter (NOAA R2O, DOD, FAA) can point to when deciding whether an AI model is fit for a given use. This is the piece with direct *revenue* potential. **D. Open datasets + reproducible pipelines.** The evaluation datasets, the truthing observations, and the containerized pipelines, released openly — which both builds trust and doubles as a citable public good (and a data-stewardship funding hook). ### Architecture sketch ``` submissions (model weights / forecast sets) │ ┌──────────────▼───────────────┐ │ reproducible eval pipeline │ ← containerized (Apptainer), runs on Derecho/Casper or cloud │ METplus core + MET-AI ext. │ └──────┬────────────────┬───────┘ in-situ obs │ │ metrics DB (versioned) (truth layer) │ ┌────────▼─────────┐ │ public leaderboard│ ← your viz layer; static-hostable (Cloudflare/S3) │ + diagnostics │ └────────┬─────────┘ │ certification reports (PDF/API) ← the revenue surface ``` ### Phased build - **MVP (≈2 quarters):** MET-AI metric extensions (CRPS/Potential-CRPS + one spatial method) + a static leaderboard for 3–4 public models (GraphCast, GenCast, AIFS, one NWP baseline) against ERA5, with the in-situ-truth gap documented as the v1 differentiator. - **v1 (≈4 quarters):** in-situ truthing layer; submission portal; regional/equity reporting; reproducible containerized pipeline released. - **v2:** certification path piloted with one NOAA and one commercial partner; API for programmatic submission. --- ## Technical risks & unknowns - **Compute for at-scale re-runs** is nontrivial; mitigate by verifying *forecast sets* (submitted outputs) before requiring on-platform *model* runs. - **In-situ truthing is genuinely hard** (QC, representativeness) — but that difficulty is the moat; it's why a startup won't do it well. - **Neutrality governance** — the referee must be seen as unbiased, including toward NCAR's own MILES-CREDIT model. Needs an explicit governance/conflict-of-interest posture from day one. - **BEACON scope overlap** — confirm internally how "MET-AI" sits relative to BEACON (extend it, don't duplicate it). BEACON is the natural home. ## What success looks like A NOAA R2O decision, a DOD/FAA procurement, or a commercial press release that *cites the NCAR benchmark* as the basis for trusting an AI model. At that point NCAR owns the reference standard for a decade. --- # Positioning & funding — #1 ## The narrative "AI weather models are now everywhere and mostly open. The bottleneck to using them for anything that matters — aviation, defense, disaster response, energy, insurance — is no longer building a model; it's *trusting* one. NCAR/RAL is the only institution with the verification tools (MET/METplus), the testbed (BEACON), and the neutrality to be the referee the whole field accepts. Fund us to build it before the standard is set by someone with a product to sell." ## Target sponsors & why each buys | Sponsor | The pitch to them | Vehicle | |---|---|---| | **NOAA EPIC** | AI needs an R2O gate for the UFS; MET/METplus is already your verification capability | EPIC community funding; UIFCW26 (July 2026) is a live venue to pitch | | **NSF CSSI** | Sustained cyberinfrastructure: an open benchmark + pipelines as national software infrastructure | CSSI solicitation | | **DOD / FAA** | You will not field an AI forecast model without independent, trusted validation | Mission/contract funding; RAL's national-security & aviation lines already reach here | | **Commercial weather firms** | Independent NCAR validation is market credibility you can't self-certify | BEACON already has 30+ partners → service/certification revenue | | **International met services** | A shared, neutral standard everyone can adopt | Partnership/contract (RAL's international precedent: Kuwait, Korea, Saudi Arabia) | ## The differentiated value prop, by audience - **To government:** de-risking. A defensible basis for operational go/no-go on AI. - **To industry:** credibility they can't buy or fake — "verified against the NCAR benchmark." - **To the field:** fairness — finally comparing AI and physics models on equal, physically-meaningful terms. ## Partnerships to line up first NOAA (via DTC — the relationship already exists), one marquee commercial partner from the BEACON roster, and JCSDA if the DA-benchmark angle (DAMBench gap) is pursued. ## The ask (illustrative) Seed **3–4 FTE for ~18 months** to (1) extend MET/METplus with AI-native verification, (2) stand up the public leaderboard, (3) pilot certification with one NOAA + one commercial partner. Modest, catalytic, and — critically — **earns external revenue at exactly the moment NSF core funding faces a proposed ~40% cut.** ## Risks to the funding case - Perceived overlap/competition with WeatherBench 2 → position as *operational + institution-backed + in-situ-truthed*, complementary not duplicative. - Neutrality questions if NCAR both builds models (MILES-CREDIT) and referees them → pre-empt with governance. - BEACON already being funded → frame as scaling a proven seed, not a net-new bet (this is a *strength* in a pitch).