Cylc - the Python workflow engine for cycling systems.

Cylc: the Python
workflow engine for
cycling systems
Declan Valters
Scientific Software Engineer
@dvalts
Modelling Infrastructure Support Systems
Met Office
@MetOffice_Sci
www.metoffice.gov.uk © Crown Copyright 2017, Met Office

Cylc team
MetOffice
Exeter, UK
Dave Matthews
Matthew Shin
Oliver Sanders
Declan Valters
NIWA
Wellington, New Zealand
Hilary Oliver
(original developer)

What is a workflow?
A workflow consists of an orchestrated and
repeatable pattern of business activity enabled
by the systematic organization of resources
into processes that transform materials,
provide services, or process information.
—Wikipedia

Cylc – a workflow engine
• Python software fororchestrating workflows
• Workflow:a series of dependenttasks or actions.
• Cylc automatically executes tasks according to complexschedules and
dependencies
• Scientific research& operations workflows
• Business logic workflows

• A series of dependent tasks
executed according to a logical
order or schedule.
• Workflow may be cycling or non-
cycling
• Cylc handles a range of workflow
complexities
• Cylc deals with task failures if they
occur

An example workflow:
weather forecasting

Cylc can handle simple workflows like
this one…

…to more complex workflows

Cylc history
• Pre-Cylc (at the Met Office):
• SCS “Suite Control System”
• Developed 1996/7
• Tcl/Tk, C, Shell scripts
• Strongly coupled to specific models
• Needed a system that could manage:
• Different runtime configurations
• Different compute resources
• Easily deployable
• Multiple applications
• Cylc:
2009/10 – development began
2012 – First deployed in Met Office,
use within R&D
2013 – Switched operational
forecasting over to using Cylc
2017 – now at version 7.5

How does Cylc manage workflows?

Cylc workflows in more depth
To recap, a Cylc workflow may:
• Be cycling or non-cycling
• Cycle indefinitely, or have an endpoint
• Cycle according to integers (Cycle 1, Cycle 2, etc…)
• Cycle according to date-times (2017-01-01,2017-02-01,… )
• Date times in Cylc may be simulated or connected to real time.
For example, simulating historic case studies.

A ‘Classic’ approach to workflows

How Cylc handles workflows

Cylc can recover from delays and task
failures efficiently

Cylc runs tasks in the most efficient order possible,
given task and cycle dependencies are met

Using Cylc

• Operational Forecasting
• Started using Cylc
operationally in 2013
• Climate modelling
• Research NWP
• Physics simulations
• Data processing
Cylc: Uses

• Met Office
• NIWA
• Bureau of Meteorology
• Max-Planck-Institut für Meteorologie
• Geophysical Fluid Dynamics Laboratory
• Korean Meteorological Administration
• South African Weather Service
• NCAR (National Centers for
Atmospheric Research)
• Many others…
• Your organisation?
Cylc: Users

• Three supercomputers in the
Top 50
• Many other smaller clusters in
use
• Generate a lot of data every
day (200TB daily)
• 480,000 cores
• How does Cylc help us?
Supercomputer use
at the Met Office

• Tasks can be run as soon as
dependencies are met
• No need to run tasks
sequentially
• Run tasks in parallel
• More efficient use of HPC
resources
Cylc keeps our
supercomputer busy

Cylc features: configuration
Workflows are implemented
with ‘suite’ configuration files
• INI-like nested structure
• Easy fornon-developers to
use and configure their
suites.
• Syntax easy(ish) to grasp
for new users

Cylc: graphing example

Cylc features: GUI
Graph view
PyGTK
(for now…)

Cylc features: GUI
Tree view

Cylc features: GUI
Monitor all running suites from one view

Cylc features: remote hosts
Cylc can be configured to send tasks to remote hosts and supports a range of
job submissionschedulers:
• at
• loadleveler
• lsf
• pbs
• sge
• slurm
• moab

Python in Cylc
Cylc developmentalmost entirely in Python (save for a few shell scripts)
Python has enabled rapid developmentof Cylc, prototyping new features,
testing.
Aided by already available Python packages:
• Jinja2 – Python-like templating language for suite config files
• CherryPy– web framework
• Requests – http(s) communications
• (Pyro in earlier versions of cylc)
• Pygraphviz– workflow diagrams (graphs)
• OpenSSL

Cylc community
• Cylc is open source,GNU GPL licenced
• Cylc is released on GitHub at https://github.com/cylc

Cylc community
We welcome contributions!
Extensive user guide
Email helpdesk:
metomi@metoffice.gov.uk
Google groups: “cylc”
GitHub: github.com/cylc

Summary
• Cylc is a general-purposeworkflowengine,written in Python
• Cylc automatically executes tasks according to complex schedules and
dependencies,dealing with task failures if they occur
• Cylc is ideal for cycling workflows (workflows that repeat according to a
pattern or schedule)
…but can just as easily be used with non-cycling workflows
• Cylc works on a rangeof computingplatforms from your desktop or
laptop to supercomputers
• Cylc scales well to highly complex,large workflows.
…but no workflow is too small for Cylc!

Summary
Cylc homepage: http://cylc.github.io/cylc
Contact us at the Met Office: metomi@metoffice.gov.uk
Follow code development, contribute, see issues and pull requests
at: http://github.com/cylc
Ask us questions!

Cylc future development
• Python API
• Web-based single GUI
• More efficiency in Cylc internals
• Future suites may contain tens of thousands of tasks
• Challenge to maintain scalability as complexity grows

Cylc - the Python workflow engine for cycling systems.

More Related Content

What's hot

Similar to Cylc - the Python workflow engine for cycling systems.

Recently uploaded

Cylc - the Python workflow engine for cycling systems.