Mistral Hong Kong Unconference track


Published on

Mistral Unconference slides from Hong Kong OpenStack Design Summit

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Mistral Hong Kong Unconference track

  1. 1. Mistral Task Orchestration and Scheduling service Task Orchestration and Scheduling service • Home: launchpad.net/mistral • Wiki: wiki.openstack.org/wiki/Mistral • IRC: #openstack-mistral • Me: Renat Akhmerov < rakhmerov@mirantis.com>
  2. 2. Background • Realized by Murano team as a generic external state machine with HA and scalability • Discussed with and supported by the authors of the Convection proposal, Joshua Harlow and TaskFlow team • Joshua, Thanks!
  3. 3. How is Mistral related to TaskFlow & Convection • Mistral implements the Convection (Workflow as a Service) and goes far beyond the initial ideas • TaskFlow library will be used for implementation • TaskFlow library will have Mistral engine to execute tasks over Mistral in a distributed manner
  4. 4. Mistral Mission • Build a Task Orchestration and Scheduling service for OpenStack • Provide an easy and flexible mechanism for executing workflows that consist of interrelated tasks in a cloud environment • Help other OpenStack projects automate their internal workflows • Provide HA and Scalability for task/workflow execution
  5. 5. Mistral Users • Infrastructure OpenStack developers : Nova, Heat, Murano, Solum etc. • System administrators : to use capabilities like Cloud Cron to be able to schedule helper cloudwide tasks (local VM processes, RESTful services, creating/terminating VMs, etc.) • Enterprise application developers : automate business processes consisting of multiple distributed processing steps
  6. 6. Reasons to use Mistral 1. High Availability : ability to continue a workflow from the point it crashed at 2. Scalability: Mistral knows what parts of a workflow can be executed in parallel 3. Scheduling: workflows/tasks can be scheduled for periodical execution (e.g. every Tuesday at 3 am) 4. Observable state: can always see the details of the current execution state and history 5. Offloading dependency management : given what needs to be completed Mistral executes whatever tasks needed for that 6. Additional integration points : a 3rd party application can hook into deployment processes represented as Mistral workflows using events 7. Formalized task graphs are just easier to manage and understand : they can be visualized, analyzed and optimized
  7. 7. Use Cases • Cloud Cron: a system administrator can schedule cloud tasks for periodical execution • Cloud Environment Deployment : a deployment tool or a system can represent deployment steps as a Mistral workflow and use Heat for every step as a software orchestration tool • Event Based Scheduling : e.g. Live Migration on CPU 100% from Ceilometer
  8. 8. Cloud Cron • Single point of configuration and control • Wide range of tasks : local processes, RESTful services etc. • Monitoring & Management • HA & Scalability
  9. 9. Cloud Environment Deployment • Formalized deployment workflow • Individual tasks use Heat to install applications • Dependency Management offloaded to Mistral • Multiple flows in parallel • Mistral provides HA and scalability • No need to use locking
  10. 10. Event Based Scheduling • Live migration is an example • Workflow execution on external events • Various event types • Various event sources
  11. 11. Key Mistral Features • Uploading custom graphs of tasks • Driven by simple DSL: YAML/JSON/XML • Task Scheduling: using UI or Cron patterns • Generic service: doesn’t do the actual work itself and acts as an HA coordinator signaling to workers about what needs to be done
  12. 12. Main Concepts • Task Graph - a set of related tasks • Workflow - a subgraph involved into one execution • Flow - a path that can be processed independently • Target task - a task that needs to get completed in a particular case • Action - a particular piece of work associated with a task (signal)
  13. 13. DSL # Tasks # Actions tasks: Task1: action: action1 actions: action1: transport: rest1 Task2: action: action2 action2: transport: amqp1 Task3: requires: Task1, Task2 action: action3 action3: transport: poll # Transports transports: rest1: type: REST url: http://some.url amqp1: type: AMQP host: my.host.org port: 5672 routing-key: a_queue … poll: type: POLL
  14. 14. Data Flow (Concept) 1. Select a subset of input data 2. Process it 3. Merge it back in 4. Pass it on 5. Merge with parallel branches
  15. 15. What about Mistral workers? • Core engine doesn’t start any workers • Workers are dynamic, configured in DSL • Types of workers: • • • Client application nodes REST services Other processes accessible by AMQP, SSH etc. • BUT! • Dedicated workers can be allocated using toolsets created on top of Mistral core engine
  16. 16. Can Mistral do anything else? • Easy to build additional toolsets and frameworks • Potential capabilities include: • Domains (Namespaces) • Event subscription • Role Based Access Control • Task priorities • Task collocation • Language-oriented frameworks (Python, Java, etc.) • Plugin system to introduce new DSL keywords
  17. 17. Is Mistral similar to Amazon Simple Workflow • In many ways Mistral is similar to Amazon SWF • Amazon SWF is oriented to language bindings (Java, Ruby, Python) • Conceptual differences in underlying model • Mistral is targeting to be much simpler • Mistral may later have Amazon SWF adapter
  18. 18. Current Status • Documented all the core ideas and concepts • Prepared a draft of DSL/API specification • Planned the Roadmap • Started prototyping core ideas • Introduced at HK OpenStack design summit! :)
  19. 19. Roadmap • Nov 25th, 2013. • • Dec 10th, 2013. • • DSL v1.0 and API v1.0 implementation Feb 20th, 2014. Release 0.2: • • Mistral PoC: Basic task orchestration, DSL, API and Scheduling Jan 20th, 2014. Release 0.1 • • DSL/API v1.0 specifications Task engine HA & scalability, Web UI for monitoring & management March 20th, 2014. Release 0.3: • Triggers (Cron, Events), Web UI for triggers (Cron, Events)
  20. 20. Thanks! Q&A Q&A Q&A