Up-to date measurements of surface meteorological variables are essential to monitor weather conditions, their spatio-temporal variability and the potential effects on a wide range of sectors and applications. Moreover, when included in continuous records of long historical observations spanning several decades, they become essential for assessing long-term climate variability and change locally and on a regional level. Automated pipelines capable of retrieving and processing near-real time meteorological data satisfy the primary prerequisites towards the development and advancement of effective and operational climate services. With a public and operational near real-time monitoring web platform in mind, we present automated pipelines to collect and process up-to-date daily temperature and precipitation records for Trentino South Tyrol (Italy) and surrounding areas, and to derive their spatially interpolated fields at sub-km scale. Our pipelines are composed by multiple steps including data download, sanity checks, reconstruction of missing daily records, integration into the historical archive, spatial interpolation and publication onto online FAIR catalogues as (openEO) “datacubes”. The different APIs, data formats and structure across the various data sources, and the need to merge the data onto harmonized meteorological layers, make this a typical case of the so-called Extract, Transform and Load (ETL) pipelines, and, in order to follow the principles of data reproducibility and Open Science, we embraced open-source automated workflow management through GitLab’s Continuous Integration / Continuous Development (CI/CD) capabilities. CI/CD workflows greatly help the management of the relatively complex graphs of tasks required for our climate application, ensuring seamless orchestration with thorough flow monitoring, application logs, transactions rollbacks, and exception handling in general. Native pipeline-oriented software development also fosters a clean separation of roles among the tasks, and a more modular architecture. This effectively reduces barriers to collaborative development and paves the way for robust operational climate services for researchers and decision makers in the face of the changing climate.

1. Elena Maines
11.11.23
Embracing CI/CD workflows
for building ETL pipelines
how we will gather and monitor multi-source spatially-
interpolated meteorological parameters in near-real time

2. Who
PIERO CAMPALANI
Software Engineer
ALICE CRESPI
Senior Researcher
ELENA MAINES
Junior Researcher

3. Pipeline: motivation
Monitoring current
meteorological conditions
at regional level
Monitoring climate
variability at local level
Having reliable and up-to-date long-term meteorological data is crucial for
Providing spatially distributed
meteorological inputs for
impact models (e.g., for
hydrological simulations)

4. Meteorological data and data preparation
Time series of key weather variables:
e.g., precipitation, min/max
temperatures.
These data are recorded at daily
and sub-daily resolution by
weather stations

5. Meteorological data and data preparation
Time series of key weather variables:
e.g., precipitation, min/max
temperatures.
These data are recorded at daily
and sub-daily resolution by
weather stations

6. Meteorological data and data preparation
Comprehensive and
updated time-series
dataset: a robust
compilation of at-site
meteorological conditions
Interpolation
for obtaining a
gridded
dataset
Irregular spatial coverage
of weather stations
PostgreSQL

7. Pipeline: motivation
Focus on analysis
Accurate data

8. Pipeline: implementation
➢ GitLab CI/CD pipeline
about.gitlab.com

9. Pipeline: implementation
➢ GitLab CI/CD pipeline
➢ Dockerized jobs runners
about.gitlab.com

10. Pipeline: implementation
➢ GitLab CI/CD pipeline
➢ Dockerized jobs runners
➢ R modules + geo-spatial libraries
about.gitlab.com

11. Pipeline: implementation
➢ GitLab CI/CD pipeline
➢ Dockerized jobs runners
➢ R modules + geo-spatial libraries
➢ testthat + shell-based unit test suites
about.gitlab.com

12. Pipeline: implementation
➢ GitLab CI/CD pipeline
➢ Dockerized jobs runners
➢ R modules + geo-spatial libraries
➢ testthat + shell-based unit test suites
➢ renv for reproducibile environment
about.gitlab.com

13. Pipeline: implementation
➢ GitLab CI/CD pipeline
➢ Dockerized jobs runners
➢ R modules + geo-spatial libraries
➢ testthat + shell-based unit test suites
➢ renv for reproducibile environment
➢ bash scripts and wrappers
about.gitlab.com

14. ETL pipeline for
updating
meteorological datasets
in near real-time
Data retrieval
Integration
with historical
data
Gap filling
Gridding
Publishing
Sanity checks
Harmonization

15. ETL pipeline for
updating
meteorological
datasets in near
real-time
Data retrieval
Integration
with historical
data
Gap filling
Gridding
Publishing
Sanity checks
Harmonization

16. 01 : Data extraction
parameter
list of selected
stations code
time interval

17. ETL pipeline for
updating
meteorological
datasets in near
real-time
Sanity checks
Harmonization
Integration
with historical
data
Gap filling
Gridding
Publishing
Data retrieval

18. 02 : Sanity checks and harmonization

19. 02 : Sanity checks and harmonization
Daily mean
temperature
calculated as
mean(min, max)

20. 03 : Integration with historical data
t

21. 04 : Gap filling
Only isolated gaps

22. 04 : Gap filling
Only isolated gaps
Interpolation based on
surrounding
stations

23. 04 : Gap filling
Only isolated gaps
Interpolation based on
surrounding
stations
Considering common
historical data

24. 04 : Gap filling
Only isolated gaps
Interpolation based on
surrounding correlated
stations
Considering common
historical data

25. 04 : Gap filling
Only isolated gaps
Interpolation based on
surrounding correlated
stations
Considering common
historical data
Missing datum
estimated as the
weighted average of
rescaled values

26. 04 : Gap filling
Only isolated gaps
Interpolation based on
surrounding correlated
stations
Considering common
historical data
Missing datum
estimated as the
weighted average of
rescaled values
Simulated values are checked for consistency and
mean temperature missing values are determined
from reconstructed values of min and max
temperatures

27. ETL pipeline for
updating
meteorological
datasets in near
real-time
Gap filling
Gridding
Publishing
Data retrieval
Integration
with historical
data
Sanity checks
Harmonization

28. 05 : Interpolation/gridding
daily at-site time series
daily grids
250m x 250m
NetCDF

29. 05 : Interpolation/gridding: temperature

30. 05 : Interpolation/gridding: temperature

31. 05 : Interpolation/gridding: temperature

32. 05 : Interpolation/gridding: temperature

33. 06 : Publishing
Updated local
timeseries
datacubes
PostgreSQL

34. Automating the pipeline

35. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
r-base:4.2.2
docker image
Makefile LICENSE
cache and
artifacts
ScientificNet

36. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
r-base:4.2.2
docker image
Makefile LICENSE
✓ develop / install / test / …
cache and
artifacts
ScientificNet

37. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
✓ Reproducible R library
r-base:4.2.2
docker image
Makefile LICENSE
✓ develop / install / test / …
cache and
artifacts
ScientificNet

38. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
✓ Reproducible R library
r-base:4.2.2
docker image
Makefile LICENSE
✓ develop / install / test / …
cache and
artifacts
ScientificNet
download sanity-checks integration
…

39. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
✓ Reproducible R library
r-base:4.2.2
docker image
Makefile LICENSE
✓ develop / install / test / …
cache and
artifacts
ScientificNet
✓ registered GitLab Runner
download sanity-checks integration
…

40. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
✓ Reproducible R library
r-base:4.2.2
docker image
Makefile LICENSE
✓ develop / install / test / …
cache and
artifacts
ScientificNet
every 2
weeks
✓ registered GitLab Runner
download sanity-checks integration
…

41. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
✓ Reproducible R library
r-base:4.2.2
docker image
Makefile LICENSE
✓ develop / install / test / …
JOB 1
meteo-pipeline/
cache and
artifacts
ScientificNet
every 2
weeks
✓ registered GitLab Runner
download sanity-checks integration
…

42. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
✓ Reproducible R library
r-base:4.2.2
docker image
Makefile LICENSE
✓ develop / install / test / …
JOB 1
meteo-pipeline/
cache and
artifacts
JOB 2
meteo-pipeline/
ScientificNet
every 2
weeks
✓ registered GitLab Runner
download sanity-checks integration
…

43. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
✓ Reproducible R library
r-base:4.2.2
docker image
Makefile LICENSE
✓ develop / install / test / …
JOB 1
meteo-pipeline/
cache and
artifacts
JOB 2
meteo-pipeline/
JOB N
meteo-pipeline/
ScientificNet
every 2
weeks
✓ registered GitLab Runner
download sanity-checks integration
…

44. Automating the pipeline
GitLab
https://gitlab.inf.unibz.it/
“REMSEN” Group
meteo-pipeline.git
VM/Ubuntu
src/ tests/
renv.lock .gitlab-ci.yml
✓ Reproducible R library
r-base:4.2.2
docker image
Makefile LICENSE
✓ develop / install / test / …
JOB 1
meteo-pipeline/
cache and
artifacts
JOB 2
meteo-pipeline/
JOB N
meteo-pipeline/
ScientificNet
every 2
weeks
✓ registered GitLab Runner
✓ jobs parallelization
download sanity-checks integration
…

45. GitLab’s out of the box:
Monitoring & Troubleshooting

46. GitLab’s out of the box:
✓ UI-based pipeline status monitoring
Monitoring & Troubleshooting

47. GitLab’s out of the box:
✓ UI-based pipeline status monitoring
✓ Jobs logs visualization
Monitoring & Troubleshooting

48. GitLab’s out of the box:
✓ UI-based pipeline status monitoring
✓ Jobs logs visualization
✓ Timing information
Monitoring & Troubleshooting

49. GitLab’s out of the box:
✓ UI-based pipeline status monitoring
✓ Jobs logs visualization
✓ Timing information
✓ Job’s manual re-runs
Monitoring & Troubleshooting

50. GitLab’s out of the box:
✓ UI-based pipeline status monitoring
✓ Jobs logs visualization
✓ Timing information
✓ Job’s manual re-runs
✓ API also available
Monitoring & Troubleshooting

51. GitLab’s out of the box:
The Pipeline Editor

52. GitLab’s out of the box:
The Pipeline Editor
job

53. GitLab’s out of the box:
The Pipeline Editor
job
dependency

54. GitLab’s out of the box:
The Pipeline Editor
stage
job
dependency

55. GitLab’s out of the box:
The Pipeline Editor
real-time syntax validation
stage
job
dependency

56. What's next

57. • Finalized implementation of the
GitLab pipeline
• Automating the connectors with
online catalogues
• Set up of webhooks for reporting to
MS Teams channel(s)
• Implementation of a monitoring
dashboard with GIS-based
visualization of the data store and of
near-real time updated fields
What’s next

58. Thank you!
Contact us:
alice.crespi@eurac.edu
elena.maines@eurac.edu
piero.campalani@eurac.edu