lesson home

Machine Learning for Physical Sciences

Data Sources

next episode

Overview

Questions

  • Understanding data sources

  • How to get data from online sources

  • How to retrieve dataset with the Toolbox?


Objectives

  • Brief overview of various data souces

  • Discuss the benefits and disadvantages of each.

  • Learn to combine Climate data with your own research topic

  • Learn how to manipulate netCDF data within the CDS Toolbox

Climate and Weather Data Sources

1) CHIRPS

CHIRPS (Climate Hazards Group Infrared Precipitation with Stations) offers a comprehensive global precipitation dataset, combining satellite observations with ground-based station data to deliver accurate rainfall estimates.

CHIRPS stands out for its high spatial and temporal resolution of rainfall estimates, accommodating regional variations in dataset relationships. Despite its strengths, the dataset has limitations, including its focus solely on rainfall, limited coverage to land areas between 50S-50N, and its primary application in drought and environmental monitoring.

Main Features

With spatial resolution at approximately 5 km, CHIRPS is instrumental in various applications. It aids in disaster management by monitoring droughts and floods, supports climate research for studying climate change impacts, facilitates agricultural planning, and assists in water resource management.

Data Source

Spearheaded by the Climate Hazards Group and USGS scientists, CHIRPS receives support from USAID, NASA, and NOAA.

Description

The dataset features high-resolution gridded precipitation data, providing daily, monthly, and seasonal estimates. Covering temporal data from 1981 onwards, CHIRPS facilitates trend analysis and is regularly updated in near real-time, ensuring timely information availability.

Applications

Key Strengths

Key Limitations

Downloading CHIRPS Data

2) CPC Global Unified Gauge-Based Analysis of Daily Precipitation

Key Strengths:

Key Limitations:

Downloading CPC Data

3) International Research Institute for Climate and Society (IRI), Data Library (DL)

Example1:

Example2:

Hands-on:

4) Copernicus Climate Data Store (CDS)

Climate Data Store (CDS) Registration

To be able to use CDS services, you need to register. Registration to the Climate Data Store (CDS) is free as well as access to climate data. Before starting, and once registred, login to the Climate Data Store (CDS).

Retrieve Climate data with CDS API

Using CDS web interface is very useful when you need to retrieve small amount of data and you do not need to customize your request. However, it is often very useful to retrieve climate data directly on the computer where you need to run your postprocessing workflow.

In that case, you can use the CDS API (Application Programming Interface) to retrieve Climate data directly in Python from the Climate Data Store.

We will be using cdsapi python package.

Get your API key

url: https://cds.climate.copernicus.eu/api/v2
key: UID:KEY

Where UID is your uid and KEY your API key. See documentation to get your API and related information.

Install the CDS API client

pip3 install cdsapi

Use CDS API

Once the CDS API client is installed, it can be used to request data from the datasets listed in the CDS catalogue. It is necessary to agree to the Terms of Use of every datasets that you intend to download.

Attached to each dataset download form, the button Show API Request displays the python code to be used. The request can be formatted using the interactive form. The api call must follow the syntax:

import cdsapi
c = cdsapi.Client()

c.retrieve("dataset-short-name", 
           {... sub-selection request ...}, 
           "target-file")

For instance to retrieve the same ERA5 dataset e.g. near surface air temperature for June 2003:

Let’s try it:

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'reanalysis-era5-single-levels-monthly-means',
    {
        'product_type':'monthly_averaged_reanalysis',
        'variable':'2m_temperature',
        'year':'2003',
        'month':'06',
        'time':'00:00',
        'format':'netcdf'
    },
    'download.nc')

Geographical subset

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'reanalysis-era5-single-levels-monthly-means',
    {      
        'area'          : [60, -10, 50, 2], # North, West, South, East. Default: global
        'product_type':'monthly_averaged_reanalysis',
        'variable':'2m_temperature',
        'year':'2003',
        'month':'06',
        'time':'00:00',
        'format':'netcdf'
    },
    'download_small_area.nc')

Change horizontal resolution

For instance to get a coarser resolution:

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'reanalysis-era5-single-levels-monthly-means',
    {      
        'area'          : [60, -10, 50, 2], # North, West, South, East. Default: global
        'grid'          : [1.0, 1.0], # Latitude/longitude grid: east-west (longitude) and north-south resolution (latitude). Default: 0.25 x 0.25
        'product_type':'monthly_averaged_reanalysis',
        'variable':'2m_temperature',
        'year':'2003',
        'month':'06',
        'time':'00:00',
        'format':'netcdf'
    },
    'download_small.nc')

More information can be found here.

To download CMIP 5 Climate data via CDS API

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'projections-cmip5-monthly-single-levels',
    {
        'variable':'2m_temperature',
        'model':'noresm1_m',
        'experiment':'historical',
        'ensemble_member':'r1i1p1',
        'period':'185001-200512'
    },
    'download_CMIP5.nc')

Exercise: Download CMIP5 from Climate Data Store with cdsapi

Get near surface air temperature (2m temperature) and precipitation (mean precipitation flux) in one single request and save the result in a file cmip5_sfc_monthly_1850-200512.zip. What do you get when you unzip this file?

Solution

  • Download the file
  • Uncompress it
  • If you select one variable, one experiment, one model, etc., then you get one file only, and it is a netCDF file (even if it says otherwise!). As soon as you select more than one variable, or more than one experiment, etc., then you get a zip or tgz (depending on the format you chose).
import cdsapi
import os
import zipfile
c = cdsapi.Client()
c.retrieve(
    'projections-cmip5-monthly-single-levels', 
    { 
       'variable': ['2m_temperature',
      'mean_precipitation_flux'],
       'model': 'noresm1_m',
        'experiment': 'historical',
        'ensemble_member': 'r1i1p1',
        'period': '185001-200512',
        'format': 'tgz'
    },
    'cmip5_sfc_monthly_1850-200512.zip'
)
os.mkdir("./cmip5")
with zipfile.ZipFile('cmip5_sfc_monthly_1850-200512.zip', 'r') as zip_ref:
    zip_ref.extractall('./cmip5')

5) Meteostat

Installation

The Meteostat Python package is available through PyPI

$ pip install meteostat

Hands-on Meteostat

6) CliMetLab

TO install CliMetLab, just run the following command:

pip install climetlab

Hands-on climetlab

Climate Data Processsing with Xarray

Hands-on xarray

Key Points

lesson home

next episode