Copernicus EU “…ERA Explorer is a little wonder of data processing technology. When you select a location on the map, the ERA Explorer calculates daily, monthly and annual statistics on-the-fly from over 80 years of hourly ERA5 data – all within a couple of seconds. This innovative approach is partly made possible by storing the data using Zarr, a community project for optimised cloud storage of large, multidimensional datasets. ERA5 is a multidimensional dataset including many dimensions like time, space, altitude for each parameter and point in the map from nearly the present day to 1940, and therefore is heavy to handle with traditional processing, that needs to find the data requested among the “data cubes” involved in the request. The Zarr archive is fast because it stores multidimensional data in independently accessible “chunks”, making it much more efficient reading and writing the specific subsets requested. For ERA Explorer we have two versions of the Zarr data, one chunked in time and the other chunked in space. This means that users can quickly pull the data they need, whether it’s a long timeseries for a single location or small area or a large area for a short time span, without scanning through massive files (for ERA Explorer we focus on long time series). Since Zarr is designed for cloud storage systems like S3, used in the C3S Climate Data Store (CDS), it enables parallel reads avoiding the input and output bottlenecks of traditional formats like NetCDF, which makes analysis incredibly fast and scalable. The new dataset behind the ERA Explorer app is also available for everyone through the CDS. This is the first time we use a Zarr-archived version of some ERA5 variables in the C3S, as part of the efforts to modernise the CDS. The system has been utilised previously in the WEkEO viewer since early 2024. The C3S team is currently archiving more CDS, Atmosphere Data Store (ADS) and Early Warning Data Sytems (EWDS) sets of variables, which will allow the creation of more agile and intuitive apps based on them. ..”
Sorry, comments are closed for this post.