1. Cylc: the Python
workflow engine for
cycling systems
Declan Valters
Scientific Software Engineer
@dvalts
Modelling Infrastructure Support Systems
Met Office
@MetOffice_Sci
www.metoffice.gov.uk © Crown Copyright 2017, Met Office

2. Cylc team
MetOffice
Exeter, UK
Dave Matthews
Matthew Shin
Oliver Sanders
Declan Valters
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
NIWA
Wellington, New Zealand
Hilary Oliver
(original developer)

3. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
What is a workflow?
A workflow consists of an orchestrated and
repeatable pattern of business activity enabled
by the systematic organization of resources
into processes that transform materials,
provide services, or process information.
—Wikipedia

4. Cylc – a workflow engine
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• Python software fororchestrating workflows
• Workflow:a series of dependenttasks or actions.
• Cylc automatically executes tasks according to complexschedules and
dependencies
• Scientific research& operations workflows
• Business logic workflows

5. www.metoffice.gov.uk © Crown Copyright 2017, Met Office

6. Cylc – a workflow engine
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• A series of dependent tasks
executed according to a logical
order or schedule.
• Workflow may be cycling or non-
cycling
• Cylc handles a range of workflow
complexities
• Cylc deals with task failures if they
occur

7. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
An example workflow:
weather forecasting

8. www.metoffice.gov.uk © Crown Copyright 2017, Met Office

9. www.metoffice.gov.uk © Crown Copyright 2017, Met Office

10. www.metoffice.gov.uk © Crown Copyright 2017, Met Office

11. www.metoffice.gov.uk © Crown Copyright 2017, Met Office

12. Cylc – a workflow engine
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Cylc can handle simple workflows like
this one…

13. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
…to more complex workflows

14. www.metoffice.gov.uk © Crown Copyright 2017, Met Office

15. www.metoffice.gov.uk © Crown Copyright 2017, Met Office

16. Cylc history
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• Pre-Cylc (at the Met Office):
• SCS “Suite Control System”
• Developed 1996/7
• Tcl/Tk, C, Shell scripts
• Strongly coupled to specific models
• Needed a system that could manage:
• Different runtime configurations
• Different compute resources
• Easily deployable
• Multiple applications
• Cylc:
2009/10 – development began
2012 – First deployed in Met Office,
use within R&D
2013 – Switched operational
forecasting over to using Cylc
2017 – now at version 7.5

17. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
How does Cylc manage workflows?

18. Cylc workflows in more depth
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
To recap, a Cylc workflow may:
• Be cycling or non-cycling
• Cycle indefinitely, or have an endpoint
• Cycle according to integers (Cycle 1, Cycle 2, etc…)
• Cycle according to date-times (2017-01-01,2017-02-01,… )
• Date times in Cylc may be simulated or connected to real time.
For example, simulating historic case studies.

19. A ‘Classic’ approach to workflows
www.metoffice.gov.uk © Crown Copyright 2017, Met Office

20. How Cylc handles workflows
www.metoffice.gov.uk © Crown Copyright 2017, Met Office

21. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
How Cylc handles workflows

22. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Cylc can recover from delays and task
failures efficiently

23. Cylc runs tasks in the most efficient order possible,
given task and cycle dependencies are met
www.metoffice.gov.uk © Crown Copyright 2017, Met Office

24. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Using Cylc

25. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• Operational Forecasting
• Started using Cylc
operationally in 2013
• Climate modelling
• Research NWP
• Physics simulations
• Data processing
Cylc: Uses

26. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• Met Office
• NIWA
• Bureau of Meteorology
• Max-Planck-Institut für Meteorologie
• Geophysical Fluid Dynamics Laboratory
• Korean Meteorological Administration
• South African Weather Service
• NCAR (National Centers for
Atmospheric Research)
• Many others…
• Your organisation?
Cylc: Users

27. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• Three supercomputers in the
Top 50
• Many other smaller clusters in
use
• Generate a lot of data every
day (200TB daily)
• 480,000 cores
• How does Cylc help us?
Supercomputer use
at the Met Office

28. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• Tasks can be run as soon as
dependencies are met
• No need to run tasks
sequentially
• Run tasks in parallel
• More efficient use of HPC
resources
Cylc keeps our
supercomputer busy

29. Cylc features: configuration
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Workflows are implemented
with ‘suite’ configuration files
• INI-like nested structure
• Easy fornon-developers to
use and configure their
suites.
• Syntax easy(ish) to grasp
for new users

30. Cylc: graphing example
www.metoffice.gov.uk © Crown Copyright 2017, Met Office

31. www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Cylc features: GUI
Graph view
PyGTK
(for now…)

32. Cylc features: GUI
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Tree view

33. Cylc features: GUI
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Monitor all running suites from one view

34. www.metoffice.gov.uk © Crown Copyright 2017, Met Office

35. Cylc features: remote hosts
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Cylc can be configured to send tasks to remote hosts and supports a range of
job submissionschedulers:
• at
• loadleveler
• lsf
• pbs
• sge
• slurm
• moab

36. Python in Cylc
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Cylc developmentalmost entirely in Python (save for a few shell scripts)
Python has enabled rapid developmentof Cylc, prototyping new features,
testing.
Aided by already available Python packages:
• Jinja2 – Python-like templating language for suite config files
• CherryPy– web framework
• Requests – http(s) communications
• (Pyro in earlier versions of cylc)
• Pygraphviz– workflow diagrams (graphs)
• OpenSSL

37. Cylc community
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• Cylc is open source,GNU GPL licenced
• Cylc is released on GitHub at https://github.com/cylc

38. Cylc community
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
We welcome contributions!
Extensive user guide
Email helpdesk:
metomi@metoffice.gov.uk
Google groups: “cylc”
GitHub: github.com/cylc

39. Summary
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• Cylc is a general-purposeworkflowengine,written in Python
• Cylc automatically executes tasks according to complex schedules and
dependencies,dealing with task failures if they occur
• Cylc is ideal for cycling workflows (workflows that repeat according to a
pattern or schedule)
…but can just as easily be used with non-cycling workflows
• Cylc works on a rangeof computingplatforms from your desktop or
laptop to supercomputers
• Cylc scales well to highly complex,large workflows.
…but no workflow is too small for Cylc!

40. Summary
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
Cylc homepage: http://cylc.github.io/cylc
Contact us at the Met Office: metomi@metoffice.gov.uk
Follow code development, contribute, see issues and pull requests
at: http://github.com/cylc
Ask us questions!

41. Cylc future development
www.metoffice.gov.uk © Crown Copyright 2017, Met Office
• Python API
• Web-based single GUI
• More efficiency in Cylc internals
• Future suites may contain tens of thousands of tasks
• Challenge to maintain scalability as complexity grows