BPM in the Cloud: Early Research Works and Challenges

748 views

Published on

With an estimated market volume of US$ 130 billion and double-figure growth rates, cloud computing is on the path to change many aspects of our digital lives. In this talk I consider the question of potential benefits and open challenges when bringing BPM research into the cloud, such as (i) how can BPM benefit from the cloud, (ii) what should BPM in the cloud look like, and (iii) what can BPM bring to cloud computing practices? In this talk I will present three works around those questions, namely (a.) end-user-friendly process modelling in the cloud, (b.) process-aware auto-scaling of cloud resources, and (c.) process mining for error diagnostics in cloud management processes.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
748
On SlideShare
0
From Embeds
0
Number of Embeds
51
Actions
Shares
0
Downloads
23
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

BPM in the Cloud: Early Research Works and Challenges

  1. 1. NICTA Copyright 2012 From imagination to impact BPM in the Cloud: Early Research Works and Challenges Dr. Ingo Weber Senior Researcher NICTA, Sydney (ATP) September 2013
  2. 2. NICTA Copyright 2012 From imagination to impact Cloud Service Market Growth vs. BPM Market Gartner (2013) • The cloud is big, and here to stay
  3. 3. NICTA Copyright 2012 From imagination to impact Cloud Service Market Growth vs. BPM Market Gartner (2013)BPM market estimated at $2.6B in 2013, $7B in 2018 (PRWeb 16/6/2013) • The cloud is big, and here to stay • By comparison, BPM is a dwarf
  4. 4. NICTA Copyright 2012 From imagination to impact Why BPM in the Cloud? • BPM as a standalone system may not take off… – But there are opportunities to use BPM thinking, techniques, and tools in other domains / applications • BPM in the cloud will happen – If something provides value, the industry will build it – This is an opportunity for the BPM research community to have practical impact on a promising area • Many avenues to explore… – This talk aims to provide a short overview of where we are at, what is needed, and some of my works. 4
  5. 5. NICTA Copyright 2012 From imagination to impact Cloud Computing Overview 5 Service Model SaaS Software as a Service Examples: CRM, email, online games, vitual desktops, etc. PaaS Platform as a Service Examples: web server, database, execution runtime, development tools, etc. IaaS Infrastructure as a Service Examples: virtual machines, storage, load balancers, network, etc. Examples in BPM In-browser modelling, e.g., Signavio, Oryx Specialized environments, e.g., IFTTT, Effektif Complete environments, e.g., Intalio|Create, Cordys (now OpenText) BPM execution platforms, e.g., Intalio Workflow tools, e.g., AWS Simple Workflow Service - Left-hand side based on Wikipedia article Cloud Computing, 09/2013
  6. 6. NICTA Copyright 2012 From imagination to impact SaaS-level BPM for end-users? • Characteristics needed – Limit functionality to something small but useful – User experience – Simplicity – Social aspects for easy connection to colleagues/friends – Short learning curve: people need to understand the tool very quickly • Are we there yet? – ...or should we try and re-think what BPM in the cloud can be? 6 Top part inspired by T. Baeyen„s talk at BPM, 08/2013
  7. 7. NICTA Copyright 2012 From imagination to impact Example: IFTTT does this quite well 7 www.ifttt.com, 09/2013
  8. 8. NICTA Copyright 2012 From imagination to impact Example recipes in IFTTT 8 www.ifttt.com, 09/2013 Whenever a game of the Boston Red Sox ends, send me an email
  9. 9. NICTA Copyright 2012 From imagination to impact Example recipes in IFTTT 9 www.ifttt.com, 09/2013 Whenever a game of the Boston Red Sox ends, send me an email Whenever someone tags me on a facebook photo, download that photo to my dropbox account
  10. 10. NICTA Copyright 2012 From imagination to impact Example: IFTTT 10 www.ifttt.com, 09/2013 If there is a birthday of a friend in my Google calendar for today, post „happy birthday“ on their wall
  11. 11. NICTA Copyright 2012 From imagination to impact Are we there yet? • Questions to drive research directions – How can BPM benefit from the cloud? – What should BPM in the cloud look like? – What can BPM bring to cloud computing practices? • Next: some of my early works 11
  12. 12. NICTA Copyright 2012 From imagination to impact How can BPM benefit from the cloud? • SaaS – No installation, very little burden for people to get started – if the tool fulfils the characteristics while being useful – My work: FormSys Process Designer [1,2] • PaaS / IaaS – Use cloud resources to enable process execution – Benefit from elasticity to stay within SLAs, but not over-pay for over-provisioning – My work: process-aware auto-scaling of cloud resources (on the IaaS level) [3] 12
  13. 13. NICTA Copyright 2012 From imagination to impact Process-aware auto-scaling of cloud resources • Assumption – Executing process model that calls Web services (WSs) – WSs are fully automated and perform compute-intensive tasks – Horizontal scaling of the VMs running the WSs is possible – Complexity of a task can depend on the input • Example – Integration logic around proprietary systems (e.g., in COBOL) – Translation, transformation, aggregation can create high utilization of resources (CPU, disk I/O, etc.) and slow down process execution • Problem – Say we have external SLAs for the whole process, e.g., to complete 95% of instances within 300s – How many resources to schedule for each task, when the requests change (complexity, frequency)? 13
  14. 14. NICTA Copyright 2012 From imagination to impact • Setup of our experiment 14 Process-aware auto-scaling of cloud resources Elastic Load Balancer 1 Auto Scaling Group 1 Instances for Service 1 Elastic Load Balancer 2 Auto Scaling Group 2 Instances for Service 2 Elastic Load Balancer 3 Auto Scaling Group 3 Instances for Service 3 ServiceProvider Service 1 Service 2 Service 3
  15. 15. NICTA Copyright 2012 From imagination to impact • Early work: use a fuzzy controller for process-aware auto-scaling • Process backlog on cloud IaaS: process-aware vs. standard auto-scaling Process-aware auto-scaling of cloud resources Load profile (different scale) Complexity of the requests increases, plateaus, then decreases, and finally no new instances are created
  16. 16. NICTA Copyright 2012 From imagination to impact • Early work: use a fuzzy controller for process-aware auto-scaling • Process backlog on cloud IaaS: process-aware vs. standard auto-scaling Process-aware auto-scaling of cloud resources Backlog: in a given minute, how many more process instances are started than completed? Load profile (different scale)
  17. 17. NICTA Copyright 2012 From imagination to impact • Early work: use a fuzzy controller for process-aware auto-scaling • Process backlog on cloud IaaS: process-aware vs. standard auto-scaling Process-aware auto-scaling of cloud resources Load profile (different scale) “Cut-offs” show where time-outs happened, caused by the ELBs.
  18. 18. NICTA Copyright 2012 From imagination to impact What should BPM in the cloud look like? • SaaS – My work: FormSys Process Designer (2010-2011) [1,2] – Metaphor: forms – ideally the ones known by users – Approach: scripting (not Programming by Demonstration), but made easy – For execution & to enable technical editing of end-user-created process models: translation to BPEL – Features: • user-editable forms are linked to Web services • restricted, yet powerful generic composition language • immediate automated process verification • process simulation in the design phase • automatic code generation: parallelized orchestrations – Evaluated in a user study 18
  19. 19. NICTA Copyright 2012 From imagination to impact FormSys Process Designer - core ideas • Service: name + icon – Visualize Data Service • Messages to / from services – Example: Sirca, computing GARCH Daily Volatility 19 Form through which the user usually controls a service
  20. 20. NICTA Copyright 2012 From imagination to impact FormSys Process Designer - core ideas • Service: name + icon – Visualize Data Service • Messages to / from services – Example: Sirca, computing GARCH Daily Volatility 20 Form through which the user usually controls a service Fields in the form can be linked to parts of WSDL messages (by a developer)
  21. 21. NICTA Copyright 2012 From imagination to impact FormSys Process Designer - example • Financial data analysis process: 1. Find news data: e.g., news data on the company „BHP‟ 2. Find performance data: e.g., on company code „BHP.AX‟ 3. Aggregate the performance data, e.g., avg. hourly stock prices 4. Merge datasets: e.g., merge the result data sets from the first and third steps 5. Visualize dataset: e.g., to see influence of news on prices 21 Control flow modelling Data flow modelling, see [1,2] for details
  22. 22. NICTA Copyright 2012 From imagination to impact FormSys Process Designer - User study [1] • Ability to solve tasks in FormSys Process Designer – Tech.: users with technical expertise – Fin.: users with financial expertise – Gen.: users with expertise in both 22 0 1 2 3 4 5 6 7 8 Tech. Gen. Fin. Solved Not solved Not attempted
  23. 23. NICTA Copyright 2012 From imagination to impact • Times by expertise 23 0:00 0:14 0:28 0:43 0:57 1:12 1:26 Gen. Tech. Fin. Time(h:mm,successonly) Expertise FormSys Process Designer - User study [1]
  24. 24. NICTA Copyright 2012 From imagination to impact • Times by expertise 24 0:00 0:14 0:28 0:43 0:57 1:12 1:26 Gen. Tech. Fin. Time(h:mm,successonly) ExpertiseMean time using Intalio|bpms FormSys Process Designer - User study [1]
  25. 25. NICTA Copyright 2012 From imagination to impact What can BPM bring to cloud computing practices? • My work: using process mining in systems operations (a.k.a. cloud administration / management) – Disclaimer: started only 2 months ago • Core question: can a process view help in operations? 1. Understanding how a tool is running 2. Understanding what the best practice is (manual processes) 3. Detecting when process steps are supposedly complete, but failed 4. Detecting deviations from best-practice / usual path 5. Detecting unusual log messages 25
  26. 26. NICTA Copyright 2012 From imagination to impact What can BPM bring to cloud computing practices? • My work: using process mining in systems operations (a.k.a. cloud administration / management) – Disclaimer: started only 2 months ago • Core question: can a process view help in operations? 1. Understanding how a tool is running 2. Understanding what the best practice is (manual processes) 3. Detecting when process steps are supposedly complete, but failed 4. Detecting deviations from best-practice / usual path 5. Detecting unusual log messages 26 Process discovery Assertion checks Conformance checks Classification Approaches:
  27. 27. NICTA Copyright 2012 From imagination to impact Process Discovery for System Operations • Challenges: – How to get the right logs – How to correlate events from several logs • Especially in a large-scale distributed system • What is a process instance – How to handle concurrent sub-processes – How to deal with low-level log messages – How to analyze incomplete log messages – How to deal with 1000s of log lines per instance 27
  28. 28. NICTA Copyright 2012 From imagination to impact Process Discovery – Example: Asgard • Netflix Asgard is a cloud management tool – Open source – For Amazon Web Services (AWS) – Sunny day example: 1 log source, high quality logs, ... • Example process: rolling upgrade – “Rolling”: upgrade one virtual machine (VM) at a time – Auto-scaling group (ASG): set of VMs from the same Amazon machine image (AMI), usually behind an elastic load balancer (ELB) – Upgrade by replacing AMI for the ASG, then de- register a VM from the ELB, terminate the VM, wait for ASG to create a new VM from the new AMI 28
  29. 29. NICTA Copyright 2012 From imagination to impact Process Discovery – Asgard logs: 3 lines [2013-07-18 15:37:31,369] [Task:Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135: {Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast- 2} [Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] Started on thread Task:Pushing ami-ad059597 into group hadoopcluster--firstASG for app hadoopcluster. [2013-07-18 15:37:31,996] [Task:Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135: {Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast- 2} [Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] Updating launch from hadoopcluster--firstASG-20130718120001 with ami-ad059597 into hadoopcluster--firstASG-20130718153731 [2013-07-18 15:37:31,998] [Task:Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135: {Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast- 2} [Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] Create Launch Configuration 'hadoopcluster--firstASG- 20130718153731' with image 'amiad059597' 29
  30. 30. NICTA Copyright 2012 From imagination to impact Process Discovery – Asgard logs: 3 lines [2013-07-18 15:37:31,369] [Task:Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135: {Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast- 2} [Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] Started on thread Task:Pushing ami-ad059597 into group hadoopcluster--firstASG for app hadoopcluster. [2013-07-18 15:37:31,996] [Task:Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135: {Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast- 2} [Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] Updating launch from hadoopcluster--firstASG-20130718120001 with ami-ad059597 into hadoopcluster--firstASG-20130718153731 [2013-07-18 15:37:31,998] [Task:Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135: {Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast- 2} [Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] Create Launch Configuration 'hadoopcluster--firstASG- 20130718153731' with image 'amiad059597' 30 „Instance ID“
  31. 31. NICTA Copyright 2012 From imagination to impact Process Discovery – Asgard logs: 3 lines [2013-07-18 15:37:31,369] [Task:Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135: {Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast- 2} [Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] Started on thread Task:Pushing ami-ad059597 into group hadoopcluster--firstASG for app hadoopcluster. [2013-07-18 15:37:31,996] [Task:Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135: {Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast- 2} [Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] Updating launch from hadoopcluster--firstASG-20130718120001 with ami-ad059597 into hadoopcluster--firstASG-20130718153731 [2013-07-18 15:37:31,998] [Task:Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135: {Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast- 2} [Pushing ami-ad059597 into group hadoopcluster-- firstASG for app hadoopcluster] Create Launch Configuration 'hadoopcluster--firstASG- 20130718153731' with image 'amiad059597' 31 Payload AMI ID ASG ID
  32. 32. NICTA Copyright 2012 From imagination to impact Process Discovery – Example: Asgard 32
  33. 33. NICTA Copyright 2012 From imagination to impact Process Discovery – Approach 33
  34. 34. NICTA Copyright 2012 From imagination to impact Open Challenges 34 Analysis Design Implementation Enactment Monitoring Evaluation On-demand self service Broad network access Resource pooling Rapid elasticity Measured service SaaS PaaS IaaS Private cloud Community cloud Public cloud Hybrid cloud BPMlife-cycle (Mendling 2007) NISTDefinitionofCloudComputing EssentialCharacteristics Service Model Deployment Models • Some dimensions when thinking about solutions Disclaimer: likely incomplete and needs per-tool analysis
  35. 35. NICTA Copyright 2012 From imagination to impact Summary & Outlook • BPM in the cloud will happen – If something provides value, the industry will build it – This is an opportunity for the BPM research community to have practical impact on a promising area • Presented some early work – Process-aware auto-scaling of cloud resources [3] – Domain-user BPM [1,2] – Process view on system operations using mining • Contact us for more information – http://ssrg.nicta.com.au/people/?cn=Ingo+Weber 35
  36. 36. NICTA Copyright 2012 From imagination to impact References 1. Ingo Weber, Hye young Paik, and Boualem Benatallah. Forms- based web service composition for domain experts. ACM Transactions on the Web (TWEB), conditionally accepted, 2013. 2. Ingo Weber, Hye young Paik, Boualem Benatallah. Forms-based Service Composition. Accepted for publication in ICSOC'11: 9th International Conference on Service Oriented Computing, short paper, Paphos, Cyprus, December 2011. 3. Christian Janiesch, Ingo Weber, Michael Menzel, and Jörn Kuhlenkamp. Optimizing the performance of automated business processes executed on virtualized resources. In HICSS'14: Hawaiian International Conference on System Science, January 2014, to appear. Acknowledgements: Thanks to Christian Janiesch and Len Bass for their input to the slides. And thanks to many colleagues for the discussions and feedback. 36

×