Create Atmospheric Cubes CDS
This notebook demonstrates how to access Climate Data Store (CDS) data via the dedicated xcube store, which provides dynamic data cube views into each gridded data set. Furthermore, an overview will be given on how to write data cubes into a team storage.
Please, also refer to the DeepESDL documentation and visit the platform's website for further information!
Brockmann Consult, 2025
This notebook runs with the python environment users-deepesdl-xcube-1.9.1
, please checkout the documentation for help on changing the environment.
Please note:
To access data from the Climate Data Store, you need a CDS API key.
Obtain a CDS Personal Access Token¶
You can obtain a CDS Personal Access Token as follows:
- Create a user account on the CDS Website.
- Log in to the website with your username and password.
- Navigate to your user page, where you can find your Personal Access Token.
Configure CDS API access¶
Your CDS Personal Access Token must be made available to the CDS API library.
You can do this by creating a file named .cdsapirc
in your home directory,
with the following format:
url: https://cds.climate.copernicus.eu/api
key: <PERSONAL-ACCESS-TOKEN>
Replace <PERSONAL-ACCESS-TOKEN>
with your Personal Access Token.
Then export the CDSAPI_URL
and CDSAPI_KEY
environment variables:
export CDSAPI_URL=https://cds.climate.copernicus.eu/api
export CDSAPI_KEY=[PERSONAL-ACCESS-TOKEN]
Agree to the terms of use for the datasets you require¶
The datasets available through CDS have associated terms of use. Before accessing a dataset via the API, you must agree to its terms of use, which can only be done via the CDS website, as follows:
- Log in to the CDS website, and go to 'Datasets' to find the dataset you require.
- On the dataset's web page, select the ‘Download’ tab.
- Scroll to the bottom of the page, and you will see a section titled ‘Terms of use’, which will contain either an ‘Accept terms’ button to allow you to accept the terms, or a confirmation that you have already accepted the terms.
Once you have accepted the terms on the website, the dataset will also be made available to you through the API.
import os
os.environ['CDSAPI_URL'] = 'https://cds.climate.copernicus.eu/api'
os.environ['CDSAPI_KEY'] = '[PERSONAL-ACCESS-TOKEN]'
# mandatory imports
from xcube.core.store import find_data_store_extensions
from xcube.core.store import get_data_store_params_schema
from xcube.core.store import new_data_store
# Utilities for notebook visualization
import shapely.geometry
import IPython.display
from IPython.display import JSON
import matplotlib as mpl
import matplotlib.pyplot as plt
Configure matplotlib to display graphs inline directly in the notebook and set a sensible default figure size.
%matplotlib inline
plt.rcParams["figure.figsize"] = 16,12
Connect to cds store¶
Check whether the cds
store is among the available stores, if not please follow the installation information from the top of this notebook.
JSON({e.name: e.metadata for e in find_data_store_extensions()})
<IPython.core.display.JSON object>
Usually we need more information to get the actual data store object. Which data store parameters are available for cds
?
get_data_store_params_schema('cds')
<xcube.util.jsonschema.JsonObjectSchema at 0x7fa8baff35b0>
Provide mandatory parameters to instantiate the store class:
store = new_data_store('cds')
store
<xcube_cds.store.CDSDataStore at 0x7fa8bafddbe0>
Which datasets are provided? (the list may contain both gridded and vector datasets):
#list(store.get_data_ids())
Get more info about a specific dataset. This includes a description of the possible open formats:
Retrieve and open requested dataset¶
There are 4 required parameters, so we need to provide them to open a dataset:
store.describe_data('reanalysis-era5-single-levels-monthly-means:monthly_averaged_reanalysis')
<xcube.core.store.descriptor.DatasetDescriptor at 0x7fa8baeb57f0>
Let's set a bbox¶
bbox=[-5, 45, 35, 65]
IPython.display.GeoJSON(shapely.geometry.box(*bbox).__geo_interface__)
<IPython.display.GeoJSON object>
Now set the other parameters for opening the dataset from the store:¶
dataset = store.open_data('reanalysis-era5-single-levels-monthly-means:monthly_averaged_reanalysis',
variable_names=['2m_temperature'],
bbox=bbox,
spatial_res=0.25,
time_range=['2006-01-01', '2010-12-31'])
dataset
xcube-cds version 1.0.0 2025-05-22 10:22:36,340 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics. 2025-05-22 10:22:36,341 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the "Show API request code" tool on the dataset Download Form to check you are using the correct syntax for your API request. 2025-05-22 10:22:36,758 INFO Request ID is c8e73bc1-a9f8-40ab-b81c-0817a48eb6d7 2025-05-22 10:22:36,830 INFO status has been updated to accepted 2025-05-22 10:22:51,280 INFO status has been updated to running 2025-05-22 10:22:58,958 WARNING Structural differences in grib fields detected when opening in xarray. Opening the grib file safely, however this may result in files with non-intuitive filenames. 2025-05-22 10:23:10,465 INFO status has been updated to successful
e19485b22e26b2d3794f643c54bfdf0d.nc: 0%| | 0.00/1.31M [00:00<?, ?B/s]
<xarray.Dataset> Size: 3MB Dimensions: (time: 60, lat: 80, lon: 160) Coordinates: number int64 8B ... * time (time) datetime64[ns] 480B 2006-01-01 2006-02-01 ... 2010-12-01 * lat (lat) float64 640B 64.88 64.62 64.38 64.12 ... 45.62 45.38 45.12 * lon (lon) float64 1kB -4.875 -4.625 -4.375 -4.125 ... 34.38 34.62 34.88 expver (time) <U4 960B ... Data variables: t2m (time, lat, lon) float32 3MB ... Attributes: GRIB_centre: ecmf GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts GRIB_subCentre: 0 Conventions: CF-1.7 institution: European Centre for Medium-Range Weather Forecasts
We can explore this dataset by plotting a temperature map for selected time points. First, we select January 2001. Land areas – and mountain ranges in particular – show up on the map as colder regions.
t2m_2001_jan = dataset.t2m.sel(time='2010-01-01 00:00:00', method='nearest')
t2m_2001_jan.plot.imshow(vmin=260, vmax=285, figsize=(14, 8), cmap='plasma')
<matplotlib.image.AxesImage at 0x7fa8babec830>
Write cube in team storage¶
To store the cube in your teams user space, please first retrieve the details from your environment variables as the following:
S3_USER_STORAGE_KEY = os.environ["S3_USER_STORAGE_KEY"]
S3_USER_STORAGE_SECRET = os.environ["S3_USER_STORAGE_SECRET"]
S3_USER_STORAGE_BUCKET = os.environ["S3_USER_STORAGE_BUCKET"]
You need to instantiate a s3 datastore pointing to the team bucket:
from xcube.core.store import new_data_store
team_store = new_data_store("s3",
root=S3_USER_STORAGE_BUCKET,
storage_options=dict(anon=False,
key=S3_USER_STORAGE_KEY,
secret=S3_USER_STORAGE_SECRET))
If you have stored no data to your user space, the returned list will be empty:
list(team_store.get_data_ids())
['ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-1981-2016-v02.0-fv01.0.rechunked.zarr', 'LC-1x720x1440-0.25deg-2.0.0-v1.zarr', 'LC-1x720x1440-0.25deg-2.0.0-v2.zarr', 'SST.levels', 'SeasFireCube-8D-0.25deg-1x720x1440-3.0.0.zarr', 'amazonas_v8.zarr', 'amazonas_v9.zarr', 'analysed_sst.zarr', 'analysed_sst_2.zarr', 'analysed_sst_3.zarr', 'analysed_sst_4.zarr', 'esa-cci-permafrost-1x1151x1641-0.1.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.4.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.5.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.6.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.7.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.8.0.zarr', 'esa-cci-permafrost-1x1151x1641-1.0.0.zarr', 'esa_gda-health_pakistan_ERA5_precipitation_and_temperature_testdata.zarr', 'noise_trajectory.zarr']
team_store.write_data(dataset,'reanalysis-era5-single-levels-monthly-means-subset-2006-2010_TMH.zarr', replace=True)
'reanalysis-era5-single-levels-monthly-means-subset-2006-2010_TMH.zarr'
list(team_store.get_data_ids())
['ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-1981-2016-v02.0-fv01.0.rechunked.zarr', 'LC-1x720x1440-0.25deg-2.0.0-v1.zarr', 'LC-1x720x1440-0.25deg-2.0.0-v2.zarr', 'SST.levels', 'SeasFireCube-8D-0.25deg-1x720x1440-3.0.0.zarr', 'amazonas_v8.zarr', 'amazonas_v9.zarr', 'analysed_sst.zarr', 'analysed_sst_2.zarr', 'analysed_sst_3.zarr', 'analysed_sst_4.zarr', 'esa-cci-permafrost-1x1151x1641-0.1.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.4.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.5.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.6.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.7.0.zarr', 'esa-cci-permafrost-1x1151x1641-0.8.0.zarr', 'esa-cci-permafrost-1x1151x1641-1.0.0.zarr', 'esa_gda-health_pakistan_ERA5_precipitation_and_temperature_testdata.zarr', 'noise_trajectory.zarr', 'reanalysis-era5-single-levels-monthly-means-subset-2006-2010_TMH.zarr']
Open data from team storage¶
cds_subset = team_store.open_data('reanalysis-era5-single-levels-monthly-means-subset-2006-2010_TMH.zarr')
cds_subset
<xarray.Dataset> Size: 3MB Dimensions: (time: 60, lat: 80, lon: 160) Coordinates: expver (time) <U4 960B dask.array<chunksize=(60,), meta=np.ndarray> * lat (lat) float64 640B 64.88 64.62 64.38 64.12 ... 45.62 45.38 45.12 * lon (lon) float64 1kB -4.875 -4.625 -4.375 -4.125 ... 34.38 34.62 34.88 number int64 8B ... * time (time) datetime64[ns] 480B 2006-01-01 2006-02-01 ... 2010-12-01 Data variables: t2m (time, lat, lon) float32 3MB dask.array<chunksize=(30, 40, 80), meta=np.ndarray> Attributes: Conventions: CF-1.7 GRIB_centre: ecmf GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts GRIB_subCentre: 0 institution: European Centre for Medium-Range Weather Forecasts
Visualize the dataset using xcube viewer¶
from xcube.webapi.viewer import Viewer
Let's set some attributes of the variable, so the colormapping does not fall back on the default
cds_subset.t2m.attrs["color_value_min"] = 270
cds_subset.t2m.attrs["color_value_max"] = 310
cds_subset.t2m.attrs["color_bar_name"] = "plasma"
cds_subset.attrs["title"] = "ERA5 2m Temperature"
viewer = Viewer()
404 GET /viewer/assets/index-CHdlbXuI.js (127.0.0.1) 1.06ms
viewer.add_dataset(cds_subset)
'0514f29e-6944-44f4-9eea-5756dfad83a8'
You can click on the viewer link to open xcube Viewer in a new browser tab:
viewer.info()
Server: https://deep.earthsystemdatalab.net/user/alicebalfanz/proxy/8000 Viewer: https://deep.earthsystemdatalab.net/user/alicebalfanz/proxy/8000/viewer/?serverUrl=https://deep.earthsystemdatalab.net/user/alicebalfanz/proxy/8000
You can also open xcube Viewer inlined here:
viewer.show()
viewer.stop_server()
In case you wish to delete data:
team_store.delete_data('reanalysis-era5-single-levels-monthly-means-subset-2006-2010_TMH.zarr')