• Like
Mistral Hong Kong Unconference track
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Mistral Hong Kong Unconference track


Mistral Unconference slides from Hong Kong OpenStack Design Summit

Mistral Unconference slides from Hong Kong OpenStack Design Summit

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Mistral Task Orchestration and Scheduling service Task Orchestration and Scheduling service • Home: launchpad.net/mistral • Wiki: wiki.openstack.org/wiki/Mistral • IRC: #openstack-mistral • Me: Renat Akhmerov < rakhmerov@mirantis.com>
  • 2. Background • Realized by Murano team as a generic external state machine with HA and scalability • Discussed with and supported by the authors of the Convection proposal, Joshua Harlow and TaskFlow team • Joshua, Thanks!
  • 3. How is Mistral related to TaskFlow & Convection • Mistral implements the Convection (Workflow as a Service) and goes far beyond the initial ideas • TaskFlow library will be used for implementation • TaskFlow library will have Mistral engine to execute tasks over Mistral in a distributed manner
  • 4. Mistral Mission • Build a Task Orchestration and Scheduling service for OpenStack • Provide an easy and flexible mechanism for executing workflows that consist of interrelated tasks in a cloud environment • Help other OpenStack projects automate their internal workflows • Provide HA and Scalability for task/workflow execution
  • 5. Mistral Users • Infrastructure OpenStack developers : Nova, Heat, Murano, Solum etc. • System administrators : to use capabilities like Cloud Cron to be able to schedule helper cloudwide tasks (local VM processes, RESTful services, creating/terminating VMs, etc.) • Enterprise application developers : automate business processes consisting of multiple distributed processing steps
  • 6. Reasons to use Mistral 1. High Availability : ability to continue a workflow from the point it crashed at 2. Scalability: Mistral knows what parts of a workflow can be executed in parallel 3. Scheduling: workflows/tasks can be scheduled for periodical execution (e.g. every Tuesday at 3 am) 4. Observable state: can always see the details of the current execution state and history 5. Offloading dependency management : given what needs to be completed Mistral executes whatever tasks needed for that 6. Additional integration points : a 3rd party application can hook into deployment processes represented as Mistral workflows using events 7. Formalized task graphs are just easier to manage and understand : they can be visualized, analyzed and optimized
  • 7. Use Cases • Cloud Cron: a system administrator can schedule cloud tasks for periodical execution • Cloud Environment Deployment : a deployment tool or a system can represent deployment steps as a Mistral workflow and use Heat for every step as a software orchestration tool • Event Based Scheduling : e.g. Live Migration on CPU 100% from Ceilometer
  • 8. Cloud Cron • Single point of configuration and control • Wide range of tasks : local processes, RESTful services etc. • Monitoring & Management • HA & Scalability
  • 9. Cloud Environment Deployment • Formalized deployment workflow • Individual tasks use Heat to install applications • Dependency Management offloaded to Mistral • Multiple flows in parallel • Mistral provides HA and scalability • No need to use locking
  • 10. Event Based Scheduling • Live migration is an example • Workflow execution on external events • Various event types • Various event sources
  • 11. Key Mistral Features • Uploading custom graphs of tasks • Driven by simple DSL: YAML/JSON/XML • Task Scheduling: using UI or Cron patterns • Generic service: doesn’t do the actual work itself and acts as an HA coordinator signaling to workers about what needs to be done
  • 12. Main Concepts • Task Graph - a set of related tasks • Workflow - a subgraph involved into one execution • Flow - a path that can be processed independently • Target task - a task that needs to get completed in a particular case • Action - a particular piece of work associated with a task (signal)
  • 13. DSL # Tasks # Actions tasks: Task1: action: action1 actions: action1: transport: rest1 Task2: action: action2 action2: transport: amqp1 Task3: requires: Task1, Task2 action: action3 action3: transport: poll # Transports transports: rest1: type: REST url: http://some.url amqp1: type: AMQP host: my.host.org port: 5672 routing-key: a_queue … poll: type: POLL
  • 14. Data Flow (Concept) 1. Select a subset of input data 2. Process it 3. Merge it back in 4. Pass it on 5. Merge with parallel branches
  • 15. What about Mistral workers? • Core engine doesn’t start any workers • Workers are dynamic, configured in DSL • Types of workers: • • • Client application nodes REST services Other processes accessible by AMQP, SSH etc. • BUT! • Dedicated workers can be allocated using toolsets created on top of Mistral core engine
  • 16. Can Mistral do anything else? • Easy to build additional toolsets and frameworks • Potential capabilities include: • Domains (Namespaces) • Event subscription • Role Based Access Control • Task priorities • Task collocation • Language-oriented frameworks (Python, Java, etc.) • Plugin system to introduce new DSL keywords
  • 17. Is Mistral similar to Amazon Simple Workflow • In many ways Mistral is similar to Amazon SWF • Amazon SWF is oriented to language bindings (Java, Ruby, Python) • Conceptual differences in underlying model • Mistral is targeting to be much simpler • Mistral may later have Amazon SWF adapter
  • 18. Current Status • Documented all the core ideas and concepts • Prepared a draft of DSL/API specification • Planned the Roadmap • Started prototyping core ideas • Introduced at HK OpenStack design summit! :)
  • 19. Roadmap • Nov 25th, 2013. • • Dec 10th, 2013. • • DSL v1.0 and API v1.0 implementation Feb 20th, 2014. Release 0.2: • • Mistral PoC: Basic task orchestration, DSL, API and Scheduling Jan 20th, 2014. Release 0.1 • • DSL/API v1.0 specifications Task engine HA & scalability, Web UI for monitoring & management March 20th, 2014. Release 0.3: • Triggers (Cron, Events), Web UI for triggers (Cron, Events)
  • 20. Thanks! Q&A Q&A Q&A