• Like
Mistral Hong Kong Unconference track
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Mistral Hong Kong Unconference track

  • 625 views
Published

Mistral Unconference slides from Hong Kong OpenStack Design Summit

Mistral Unconference slides from Hong Kong OpenStack Design Summit

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
625
On SlideShare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
23
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Mistral Task Orchestration and Scheduling service Task Orchestration and Scheduling service • Home: launchpad.net/mistral • Wiki: wiki.openstack.org/wiki/Mistral • IRC: #openstack-mistral • Me: Renat Akhmerov < rakhmerov@mirantis.com>
  • 2. Background • Realized by Murano team as a generic external state machine with HA and scalability • Discussed with and supported by the authors of the Convection proposal, Joshua Harlow and TaskFlow team • Joshua, Thanks!
  • 3. How is Mistral related to TaskFlow & Convection • Mistral implements the Convection (Workflow as a Service) and goes far beyond the initial ideas • TaskFlow library will be used for implementation • TaskFlow library will have Mistral engine to execute tasks over Mistral in a distributed manner
  • 4. Mistral Mission • Build a Task Orchestration and Scheduling service for OpenStack • Provide an easy and flexible mechanism for executing workflows that consist of interrelated tasks in a cloud environment • Help other OpenStack projects automate their internal workflows • Provide HA and Scalability for task/workflow execution
  • 5. Mistral Users • Infrastructure OpenStack developers : Nova, Heat, Murano, Solum etc. • System administrators : to use capabilities like Cloud Cron to be able to schedule helper cloudwide tasks (local VM processes, RESTful services, creating/terminating VMs, etc.) • Enterprise application developers : automate business processes consisting of multiple distributed processing steps
  • 6. Reasons to use Mistral 1. High Availability : ability to continue a workflow from the point it crashed at 2. Scalability: Mistral knows what parts of a workflow can be executed in parallel 3. Scheduling: workflows/tasks can be scheduled for periodical execution (e.g. every Tuesday at 3 am) 4. Observable state: can always see the details of the current execution state and history 5. Offloading dependency management : given what needs to be completed Mistral executes whatever tasks needed for that 6. Additional integration points : a 3rd party application can hook into deployment processes represented as Mistral workflows using events 7. Formalized task graphs are just easier to manage and understand : they can be visualized, analyzed and optimized
  • 7. Use Cases • Cloud Cron: a system administrator can schedule cloud tasks for periodical execution • Cloud Environment Deployment : a deployment tool or a system can represent deployment steps as a Mistral workflow and use Heat for every step as a software orchestration tool • Event Based Scheduling : e.g. Live Migration on CPU 100% from Ceilometer
  • 8. Cloud Cron • Single point of configuration and control • Wide range of tasks : local processes, RESTful services etc. • Monitoring & Management • HA & Scalability
  • 9. Cloud Environment Deployment • Formalized deployment workflow • Individual tasks use Heat to install applications • Dependency Management offloaded to Mistral • Multiple flows in parallel • Mistral provides HA and scalability • No need to use locking
  • 10. Event Based Scheduling • Live migration is an example • Workflow execution on external events • Various event types • Various event sources
  • 11. Key Mistral Features • Uploading custom graphs of tasks • Driven by simple DSL: YAML/JSON/XML • Task Scheduling: using UI or Cron patterns • Generic service: doesn’t do the actual work itself and acts as an HA coordinator signaling to workers about what needs to be done
  • 12. Main Concepts • Task Graph - a set of related tasks • Workflow - a subgraph involved into one execution • Flow - a path that can be processed independently • Target task - a task that needs to get completed in a particular case • Action - a particular piece of work associated with a task (signal)
  • 13. DSL # Tasks # Actions tasks: Task1: action: action1 actions: action1: transport: rest1 Task2: action: action2 action2: transport: amqp1 Task3: requires: Task1, Task2 action: action3 action3: transport: poll # Transports transports: rest1: type: REST url: http://some.url amqp1: type: AMQP host: my.host.org port: 5672 routing-key: a_queue … poll: type: POLL
  • 14. Data Flow (Concept) 1. Select a subset of input data 2. Process it 3. Merge it back in 4. Pass it on 5. Merge with parallel branches
  • 15. What about Mistral workers? • Core engine doesn’t start any workers • Workers are dynamic, configured in DSL • Types of workers: • • • Client application nodes REST services Other processes accessible by AMQP, SSH etc. • BUT! • Dedicated workers can be allocated using toolsets created on top of Mistral core engine
  • 16. Can Mistral do anything else? • Easy to build additional toolsets and frameworks • Potential capabilities include: • Domains (Namespaces) • Event subscription • Role Based Access Control • Task priorities • Task collocation • Language-oriented frameworks (Python, Java, etc.) • Plugin system to introduce new DSL keywords
  • 17. Is Mistral similar to Amazon Simple Workflow • In many ways Mistral is similar to Amazon SWF • Amazon SWF is oriented to language bindings (Java, Ruby, Python) • Conceptual differences in underlying model • Mistral is targeting to be much simpler • Mistral may later have Amazon SWF adapter
  • 18. Current Status • Documented all the core ideas and concepts • Prepared a draft of DSL/API specification • Planned the Roadmap • Started prototyping core ideas • Introduced at HK OpenStack design summit! :)
  • 19. Roadmap • Nov 25th, 2013. • • Dec 10th, 2013. • • DSL v1.0 and API v1.0 implementation Feb 20th, 2014. Release 0.2: • • Mistral PoC: Basic task orchestration, DSL, API and Scheduling Jan 20th, 2014. Release 0.1 • • DSL/API v1.0 specifications Task engine HA & scalability, Web UI for monitoring & management March 20th, 2014. Release 0.3: • Triggers (Cron, Events), Web UI for triggers (Cron, Events)
  • 20. Thanks! Q&A Q&A Q&A