Skip to content

05. ETL Pipeline

The ETL layer consists of three specialized Python repositories that ingest, transform, and load climate data into the AClimate v3 database.

Pipeline Overview

flowchart LR
    subgraph "Data Sources"
        ERA5[ERA5]
        CHIRPS[CHIRPS]
        AGERA5[AgERA5]
        STATIONS[Weather Stations<br/>IDEAM, METOS, etc.]
    end

    subgraph "ETL Processing"
        CSD[cut_spatial_data<br/>Spatial Clipping]
        HSE[historical_spatial_etl<br/>Gridded Data]
        HLE[historical_location_etl<br/>Station Data]
    end

    subgraph "Output"
        DB[(PostgreSQL<br/>AClimate DB)]
        SDB[(AClimate<br/>Spatial DB)]
    end

    ERA5 --> CSD
    CHIRPS --> CSD
    CSD --> HSE
    HSE --> SDB
    STATIONS --> HLE
    HLE --> DB

Repository Overview

Repository Purpose Processing Type
aclimate_cut_spatial_data Spatial clipping of climate datasets (NetCDF, GeoTIFF) using shapefiles, GeoServer, or PostGIS geometries Raster preprocessing
aclimate_v3_historical_spatial_etl Gridded climate data processing (ERA5, CHIRPS, local data) Spatial indicators and aggregation
aclimate_v3_historical_location_etl Station-based climate data processing Point indicators, quality control

Data Quality Controls

All ETLs implement the following quality controls:

Check Description Action on Failure
Schema validation Data matches ORM schema Record rejected, logged
Range validation Values within expected bounds Flagged, logged
Completeness Required fields present Record rejected
Duplicate detection Existing records checked Skipped or updated

Idempotency

All ETLs are designed to be idempotent: same input plus same configuration equals same output. Duplicate runs do not create duplicate records. Reprocessing can be triggered for specific time ranges or locations.

Section Contents