Mistral OpenStack Meetup Feb 5

4,725 views

Published on

Presentation about Mistral (Workflow Service) from SF Bay Area OpenStack meetup held at Yahoo! on Feb 6th.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,725
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
83
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Mistral OpenStack Meetup Feb 5

  1. 1. Mistral 
 Workflow as a Service Renat Akhmerov
 Senior Software Engineer © MIRANTIS 2014 !1
  2. 2. Agenda • • • • • • • • Project Mission Project users Use Cases Main Features DSL, Data Flow concept, Architecture Current status Live Demo Roadmap and Contribution © MIRANTIS 2014 !2
  3. 3. Workflows all Around Find a job Stanford, Harvard, etc. Make money © MIRANTIS 2014 !3
  4. 4. Project Mission Provide a mechanism to define and execute tasks and workflows without writing the code © MIRANTIS 2014 !4
  5. 5. Who are Mistral Users? • OpenStack developers: Nova, Heat, Murano, Solum etc. • Administrators: capabilities like Cloud Cron to schedule periodic helper cloud tasks (local VM processes, backups etc.) • Enterprise application developers: automate business processes consisting of multiple distributed processing steps © MIRANTIS 2014 !5
  6. 6. Use Cases • Cloud Cron: a system administrator can schedule cloud tasks for periodical execution • Cloud Environment Deployment: tools like Heat can represent deployment steps as a workflow and use Mistral to run it • Live Migration: on CPU close to 100% run specific migration workflow • • Long-Running Business Process Big Data Analysis & Reporting © MIRANTIS 2014 !6
  7. 7. Cloud Cron • • Start a workflow on event (timer) • Wide range of tasks: local Single point of configuration and control for helper periodic jobs processes, OpenStack services (Nova, Cinder, Swift etc.), email, SMS, custom actions • • Monitoring & Management HA & Scalability © MIRANTIS 2014 !7
  8. 8. Main Features • • • RESTful Web service • • • • • Tasks associated with pluggable actions Based on Domain Specific Language (DSL) Runs workflows consisting of synchronous and asynchronous tasks Built-in actions: HTTP request, AMQP message Runs workflows on event (timer, ceilometer alarm) Provides history and state of currently running workflows HA and Scalability for workflow executions © MIRANTIS 2014 !8
  9. 9. Mistral Workflow • • • • • • • Graph of tasks T7 Task Actions (HTTP, AMQP, SSH, email, etc.) T9 T6 T8 Transitions depends Dependencies T3 Data Flow T2 T1 T4 Conditions start Events (triggers) © MIRANTIS 2014 T5 !9
  10. 10. Mistral DSL Capabilities • • Task dependencies (task3 depends on task2 and task1) • Conditional expressions in dependencies and direct transitions (based on Data Flow) • • • Task action declarations (signals over HTTP, AMQP) Direct transitions (on task1 success execute task2, on error task3) Standard actions (std:repeat, std:email) Cron events (run workflow hourly) © MIRANTIS 2014 !10
  11. 11. DSL Snippet Workflow: tasks: runJob: requires: createVM: $.vm_ip != null action: DemoApp:runJob on-success: deleteVM on-error: sendJobError ! ! deleteVM: action: Nova:deleteVM on-finish: end sendJobError: action: std:email parameters: address: admin@mycompany.com subject: Workflow error body: Error occurred while running a VM job in execution {$.execution.id} on-finish: deleteVM © MIRANTIS 2014 !11
  12. 12. Data Flow (concept) “image_id”: “123”, “person”: { “name”: “John”, “email”: “me@me.com” }, “vm_ip”: “10.0.0.3” IsMale input: person.name RunVM “image_id”: “123”, “person”: { “name”: “John”, “email”: “me@me.com” }, “vm_ip”: “10.0.0.3”, “male”: “true” GoOn input: “image_id”: “123”, image_id “person”: { “name”: “John”, “email”: “me@me.com” } SendMsg “image_id”: “123”, “person”: { “name”: “John”, “email”: “me@me.com” }, “vm_ip”: “10.0.0.3” - task input input: person.email “image_id”: “123”, “person”: { “name”: “John”, “email”: “me@me.com” }, “vm_ip”: “10.0.0.3”, “sent_msg”: “true” “image_id”: “123”, “person”: { “name”: “John”, “email”: “me@me.com” }, “vm_ip”: “10.0.0.3”, “male”: “true”, “sent_msg”: “true” - task result (added into context) © MIRANTIS 2014 !12
  13. 13. Architecture Workflow Queue Engine Scheduler API Server workflows executions tasks events Task Queue Task Executor ! ... Task Executor © MIRANTIS 2014 !13
  14. 14. Current Status • • • • Basic task dependency based model works Pilot is currently being worked on Launchpad home page o Code in stackforge code repos o o • • https://launchpad.net/mistral o https://github.com/stackforge/mistral https://github.com/stackforge/mistral-extra https://github.com/stackforge/python-mistralclient Description on OpenStack Wiki o wiki.openstack.org/wiki/Mistral Contributors: Mirantis, StackStorm © MIRANTIS 2014 !14
  15. 15. Live Demo Demo App 1. Upload workbook 4) sk (ta 4 Put Service on hold w REST endpoint rk flo 3 Mistral n Ru 5 Backup User Data Backup Service Data 2. HTTP wo 4 Execute Backup © MIRANTIS 2014 !15
  16. 16. Mistral Icehouse Roadmap • • End of February, 2014. Pilot. o Basic DSL capabilities (dependencies, direct transitions, conditionals, cron events) o Keystone Integration o REST API o Python Client April, 2014. Release 0.1 o Overall stability o Refined DSL o Mistral CLI o Horizon Dashboard (Workbook CRUD operations, workflow execution state, history) © MIRANTIS 2014 !16
  17. 17. Contribution Opportunities • • • • • • • • • Blueprints for additional capabilities Seed initial dev team Web UI (history, workflow lifecycle management) Ceilometer integration Integration with deployment tools (Puppet, Chef) Advanced capabilities of Python Client (building workflow step-by-step locally, callbacks etc.) Workflow rollback CLI for Mistral API CLI for Mistral action handlers © MIRANTIS 2014 !17
  18. 18. How to Become a Contributor? Install Python, git etc. Learn Python Register at Launchpad Make changes clone mistral repos Sign OpenStack CLA tor life -cy cle Buy a computer Co ntr ibu no Send patches Have coffee Tired? yes © MIRANTIS 2014 !18
  19. 19. Thanks for Attending! Project: launchpad.net/mistral IRC: #openstack-mistral Me: Renat Akhmerov Email: rakhmerov@mirantis.com © MIRANTIS 2014 Q&A !19

×