historical_spatial_etl¶
Processes gridded climate data from global/regional sources (ERA5, CHIRPS, AgERA5) into spatial indicators and aggregated data.
Connectors¶
The connectors/ module downloads and transforms data from external sources:
| Connector | Source | Variables | Frequency |
|---|---|---|---|
| ERA5 Connector | ECMWF ERA5 reanalysis | Temperature, precipitation, solar radiation, humidity, wind | Monthly downloads |
| CHIRPS Connector | UCSB CHIRPS | Precipitation only | Monthly downloads |
| Local Data Connector | Local/regional gridded datasets | Configurable | Configurable |
Climate Processing¶
The climate_processing/ module handles data transformation:
| Component | Description |
|---|---|
| aggregate_daily_data.py | Aggregates daily climate data to the required temporal resolution |
| aggregate_monthly_data.py | Computes monthly aggregates from daily data |
| indicators_processor.py | Orchestrates calculation of configured indicators per country |
Tools¶
| Tool | Description |
|---|---|
| raster_clipper | Clips raster data to country/region boundaries |
| raster_resampler | Resamples rasters to match target resolution |
| raster_upload | Uploads processed rasters to GeoServer for spatial services |
| download_pipeline | Orchestrates download, processing, and upload workflow |
| config_manager | Manages configuration files for data sources and parameters |
| file_namer | Standardizes file naming according to conventions |
| validation_utils | Validates data integrity and completeness |
| logging_manager | Centralized logging with component tracking |
Configuration¶
The config/ directory contains JSON configuration files:
chirps_config.json- CHIRPS download and processing parametersclipping_config.json- Region/polygon clipping settingscopernicus_config.json- Copernicus/ERA5 API credentials and parametersgeoserver_config.json- GeoServer connection settingslocal_data_config.json- Local data source configurationnaming_config.json- File naming conventions
Spatial Indicators¶
Calculated using a plugin-based architecture via CalculatorLoader. Currently supports 8 indicators:
| Code | Indicator | Category | Description |
|---|---|---|---|
| CDD | Consecutive Dry Days | Drought | Maximum number of consecutive days with precipitation < 1mm |
| R95pTOT | Very Wet Days | Precipitation | Total precipitation from days > 95th percentile |
| Rx1day | Max 1-day Precipitation | Precipitation | Highest 1-day precipitation amount |
| SDII | Simple Daily Intensity Index | Precipitation | Average precipitation on wet days (>=1mm) |
| TR20 | Tropical Nights | Temperature | Days where minimum temperature > 20°C |
| TX10p | Cool Days | Temperature | Days where maximum temperature < 10th percentile |
| TX90p | Warm Days | Temperature | Days where maximum temperature > 90th percentile |
| TXX | Maximum Temperature | Temperature | Monthly/seasonal maximum of daily maximum temperature |
Each indicator uses a BaseCalculator class that reads climate data, applies the calculation formula, and stores results. The percentile_calculator.py computes the required percentiles for percentile-based indicators (R95pTOT, TX10p, TX90p).