ML | ECMWF | Anemoi

Anemoi

Anemoi is a collaborative and open-source framework for developing machine learning weather forecasting models. Named after the Greek gods of the winds, the goal of Anemoi is to provide the key building blocks to train state‑of-the‑art data-driven weather forecasting models and run them in an operational context. As a framework it seeks to handle many of the complexities that meteorological organisations will share, allowing them to easily train models from existing recipes but with their own data.

AIFS

AIFS is based on a graph neural network (GNN) encoder and decoder, and a sliding window transformer processor, and is trained on ECMWF’s ERA5 re-analysis and ECMWF’s operational numerical weather prediction (NWP) analyses.

Webinar: Anemoi - a framework for building the AIFS and other data-driven weather forecasting models | Virtual | April 30, 2024

  • https://events.ecmwf.int/event/409/
  • Description
    • Last year ECMWF introduced the AIFS, building on a series of innovative papers which demonstrated the potential of machine learning in creating data-driven weather forecasting models.
    • Machine learning models typically build on open-source frameworks, such as Tensorflow, PyTorch and JAX which allow communities to work together and build on one another's work. Weather forecasting brings its own challenges for machine learning, with large datasets and spatial-temporal problems.
    • In this webinar, we will introduce Anemoi, the toolbox from which the AIFS is trained.
    • We have been developing this toolbox with a view to many users both inside and outside of ECMWF to use the toolbox to train both global and local weather forecasting models. The toolbox aims to allow users to bring their own datasets and customise their ML models.
    • We will introduce the toolbox, talk through the steps needed to train a data-driven model and explain how Anemoi helps with these tasks. Finally we will outline a roadmap for future Anemoi development.
  • Additional utility tools exist in supporting repositories beyond these core modules. Anemoi builds on top of ECMWF’s Artificial Intelligence Forecasting System (AIFS), expanding that codebase to enable wider functionality for a greater range of users. Future AIFS models will be trained using Anemoi.

Discover Anemoi: Webinar Series | Online | 13-30 January 2025

Discover Anemoi will feature six webinars, demonstrating the use of key components of the Anemoi ecosystem. The webinars will cover:

  • An introduction to Anemoi
  • Creating datasets
  • Building graphs
  • Training models
  • Inference
  • Contributing as a developer

High resolution inside the Nordic

The AIFS is a global system, but work has also been going on in Anemoi to create regional (limited-area) ML systems. For example, MET Norway has created a regional model for Scandinavia (Nipen et al., 2024, arXiv:2409.02891).

The plot shows a 7-day forecast of 10 m wind speed (shading) and sea-level pressure (contours). The model has learned to forecast at high resolution (here ~10 km) inside the Nordic region, and at low resolution (here ~100 km) outside of this domain. The model successfully creates a higher-resolution structure over the Nordics.

The DDM was trained on compute nodes equipped with four AMD Instinct MI250X GPUs, each with 128 GB memory. To speed up training, we used data parallelism across 32 model instances for stages A–C and 16 for stage D. For stages B–D, we used model parallelism to run one instance of the model on each node. This splits the nodes and edges of the graphs across multiple GPUs and is necessary for fitting the 31 km global and 2.5 km regional stretched-grid model within the available GPU memory

Build Anemoi

Anemoi is a framework for developing machine learning weather forecasting models. It comprises of components or packages for

  • preparing training datasets,
  • conducting ML model training and
  • a registry for datasets and trained models.

Anemoi provides tools for operational inference, including interfacing to verification software. As a framework it seeks to handle many of the complexities that meteorological organisations will share, allowing them to easily train models from existing recipes but with their own data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
name: anemoi
channels:
  - conda-forge
dependencies:
  - python==3.10.15
  - cuda==12.4.1
  - kerchunk
  - h5py
  - s3fs
  - pip:
    - anemoi-utils[all]
    - anemoi-transform[all]
    - anemoi-datasets[create]
    - anemoi-models
    - anemoi-graphs
    - anemoi-training
    - anemoi-inference
    - anemoi-registry[all]

Dependencies

  • Arrows indicate dependencies between packages.
  • Dashed lines indicate optional dependencies.

Flow of data in inference

The schema above is a high-level overview of the data flow in the inference process (click on the image for a larger version).

Several key concepts are introduced:

Input:

  • Prognostic variables:
  • Diagnostic variables:

Output:

  • Constant forcings:
  • Dynamic forcings:
  • Computed constants:
  • Computed forcings:

Hugging Face | AIFS-single via anemoi

save input_state

These functions are all that you need for saving and loading almost any object:

1
2
3
4
5
6
7
import pickle 

with open('saved_dictionary.pkl', 'wb') as f:
    pickle.dump(dictionary, f)
        
with open('saved_dictionary.pkl', 'rb') as f:
    loaded_dict = pickle.load(f)

Discover Anemoi: Webinar Series

References

  1. AIFS: a new ECMWF forecasting system | ECMWF | January 2024
  2. Bringing together users, researchers and coders in ECMWF’s machine learning efforts | 15 May 2024
  3. Machine learning to play growing role in weather forecasting, says DG | ECMWF | 9 September 2024
    1. Machine learning is well suited to weather forecasting due to the vast amounts of data that are used to determine the best possible initial conditions and to produce global forecasts.
  4. Machine learning ensembles | ECMWF | October 2024
  5. 欧洲中期天气预报中心联合多国共推人工智能天气预报计划 | CMA
    1. 目前，Anemoi吸引了西班牙国家气象局、丹麦气象研究所、德国气象局、芬兰气象研究所、意大利空军气象局、荷兰皇家气象研究所、挪威气象局、法国气象局、瑞士气象局和比利时皇家气象研究所的参与。一些国家基于Anemoi在机器学习模型的创建上取得了进展，如挪威气象局为斯堪的纳维亚创建了区域模型，德国气象局正利用其全球数值预报模式(ICON)数据开发一个名为AICON的数据驱动天气预报模型。
ML
#NWP #ECMWF #AI #ML #Anemoi
ML | ECMWF | Anemoi
https://waipangsze.github.io/2025/01/25/ML-ECMWF-Anemoi/
Author
wpsze
Posted on
January 25, 2025
Updated on
February 28, 2025
Licensed under