Next-Generation BPM - How to create intelligent Business Processes thanks to Big Data and Apache Hadoop
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Next-Generation BPM - How to create intelligent Business Processes thanks to Big Data and Apache Hadoop

  • 1,369 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,369
On Slideshare
1,091
From Embeds
278
Number of Embeds
3

Actions

Shares
Downloads
41
Comments
0
Likes
0

Embeds 278

http://www.kai-waehner.de 276
http://translate.googleusercontent.com 1
http://www.slideee.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Next-Generation BPM – How to create intelligent Business Processes thanks to Big Data Talend, Global Leader in Open Source Integration Solutions Kai Wähner kontakt@kai-waehner.de @KaiWaehner Xing / LinkedIn www.kai-waehner.de
  • 2. Kai Wähner Main Tasks Requirements Engineering Enterprise Architecture Management Business Process Management Architecture and Development of Applications Service-oriented Architecture Integration of Legacy Applications Cloud Computing Big Data Contact Consulting Developing Coaching Speaking Writing © Talend 2013 Email: kontakt@kai-waehner.de Blog: www.kai-waehner.de/blog Twitter: @KaiWaehner Social Networks: Xing, LinkedIn “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 3. Key messages BPM should be used (just) for optimizing business processes! Intelligent business processes need big data and integration! Big data will reduce human interactions in BPM further! © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 4. Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 5. Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 6. Why should you care about big data? “If you can't measure it, you can't manage it.” William Edwards Deming (1900 –1993) American statistician, professor, author, lecturer and consultant © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 7. Why should you care about big data? „Silence the HiPPOs“ (highest-paid person‘s opinion)  Being able to interpret unimaginable large data stream, the gut feeling is no longer justified!  © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 8. Why does big data exist? Changing Scale Sensors Changing Expectations Cloud Changing Interactions © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 9. Three shifts in the way we analyze information • Messiness: Using ALL data, not just samples • Also bad data (e.g. Word spell checker, Google autocomplete and „did you mean...“ recommendation • Correlations: Instead of causalities • May not tell us WHY something is happening, but THAT it is happening • In many situations, this is good enough • What drug substance cures cancer? When should I buy an airplane ticket? • Datafication: Store, process, combine, reuse, enhance all data! • Digitalisation (Amazon Kindle  Read) vs. Datafication (Google Books  Read, Search, Process, ...) • Words becomes data: Google books: not just read, but also search, analyse, etc. • Locations becomes data: GPS: not just navigation, but also insurance costs, economic routes, etc. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 10. The Vs of big data Volume (terabytes, petabytes) Velocity (realtime or nearrealtime) Variety (social networks, blog posts, logs, sensors, etc.) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Value
  • 11. Big data tasks to solve - before analysis Big Data Integration – Land data in a Big Data cluster – Implement or generate parallel processes Big Data Manipulation – Simplify manipulation, such as sort and filter – Computational expensive functions Big Data Quality & Governance – Identify linkages and duplicates, validate big data – Match component, execute basic quality features Big Data Project Management – Place frameworks around big data projects – Common Repository, scheduling, monitoring © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 12. Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 13. Storage  Reduce costs Global Parcel Service A lot of data must be stored „forever“ ➜ Numbers increase exponentially ➜ Goal: As cheap as possible ➜ Problem: (Fast) queries must still be possible ➜ Solution: Commodity servers and „Hadoop querying“ ➜ http://archive.org/stream/BigDataImPraxiseinsatz-SzenarienBeispieleEffekte/Big_Data_BITKOM-Leitfaden_Sept.2012#page/n0/mode/2up © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 14. Replace ETL Improve performance “The advantage of their new system is that they can now look at their data [from their log processing system] in anyway they want: ➜ Nightly MapReduce jobs collect statistics about their mail system such as spam counts by domain, bytes transferred and number of logins. ➜ When they wanted to find out which part of the world their customers logged in from, a quick [ad hoc] MapReduce job was created and they had the answer within a few hours. Not really possible in your typical ETL system.” http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 15. Risk management  Customer success Deduce Customer Defections http://hkotadia.com/archives/5021 © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 16. Flexible pricing  Increase revenue ➜ ➜ ➜ ➜ With revenue of almost USD 30 billion and a network of 800 locations, Macy's is considered the largest store operator in the USA Daily price check analysis of its 10,000 articles in less than two hours Whenever a neighboring competitor anywhere between New York and Los Angeles goes for aggressive price reductions, Macy's follows its example If there is no market competitor, the prices remain unchanged http://www.t-systems.com/about-t-systems/examples-of-successes-companies-analyze-big-data-in-record-time-l-t-systems/1029702 © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 17. Great big data use cases, but ... ➜ ➜ ➜ ➜ How do you put this big data easily in the hands of the people that need it? Making the data “actionable” is the real challenge. Seeing the information that helps make a decision on a composite dashboard is just the first step and where too many companies stop. A business must be able to fire off the business process to execute the decision made regarding the data. http://smartdatacollective.com/matt-davies/104576/data-driven-bpm-making-big-data-actionable © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 18. Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 19. Intelligent business processes Humans have to interpret large data to make decision. Using gut feeling is nothing but gambling. ➜ Just doing big data analytics is not enough. Systematic and monitored human interactions are as important to get best outcomes. ➜ An intelligent business process combines big data and BPM. This enables humans to make data-driven decisions based on big data analytics. ➜ © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 20. Intelligent business processes ➜ Process starts action (PULL Big Data) • Manual or automated • Faster responses (e.g. „spam by domain“) • Better outcomes (e.g. „recommendation engine“) ➜ Data starts action (Big Data PUSH) • (Usually) automated • Predictive processes (e.g. „preventing flu epidemic“) • Handle before it happens (e.g. „customer deduction“) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 21. Combination of big data and BPM How are they related? ➜ How to combine? ➜ How to realize this technically? ➜ © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 22. How BPM? Script Task Groovy JavaScript etc. Service Task SOAP Web Service Everything from Cobol to Ruby... © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner ... or a „big data service“
  • 23. Challenge Separation of Concerns © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 24. Building Blocks for „Intelligent Business Processes“ Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 25. Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 26. Building Blocks for „Intelligent Business Processes“ Integration • Extract Transform Load (ETL) • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 27. Enterprise Integration Patterns © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 28. Do not write all that “glue code”! AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials( S3Sample.class.getResourceAsStream("AwsCredentials.properties"))); String bucketName = "my-first-s3-bucket-" + UUID.randomUUID(); String key = "MyObjectKey"; try { s3.createBucket(bucketName); s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile())); S3Object object = s3.getObject(new GetObjectRequest(bucketName, key)); ObjectListing objectListing = s3.listObjects(new ListObjectsRequest() .withBucketName(bucketName) .withPrefix("My")); s3.deleteObject(bucketName, key); s3.deleteBucket(bucketName); } catch (AmazonServiceException ase) { // error handling... } catch (AmazonClientException ace) { // error handling... } © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 29. Integration framework (e.g. Apache Camel) // Producer from(“ftp:toS3") .setHeader(S3Constants.KEY, simple(“order.txt")) .to("aws-s3://myBucket?accessKey=" + a+ "&secretKey= " + s) // Consumer from(„salesforce://orders__c?user=dummy1“) .filter(„customer == ${dummyCustomer}) .to(“ibm-database:orderData") © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 30. Enterprise Service Bus (e.g. Talend ESB) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 31. Alternatives for integration Integration Suite Enterprise Service Bus Integration Framework Low Connectivity Routing Transformation © Talend 2013 High + INTEGRATION Tooling Monitoring Support + BUSINESS PROCESS MGT. BIG DATA / MDM REGISTRY / REPOSITORY RULES ENGINE „YOU NAME IT“ “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Complexity of Integration
  • 32. Building Blocks for „Intelligent Business Processes“ Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 33. Technology perspective How to process big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 34. How to process big data? The defacto standard for big data processing © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 35. How to process big data? “A big part of [the company’s strategy] includes wiring SQL Server 2012 (formerly known by the codename “Denali”) to the Hadoop distributed computing platform, and bringing Hadoop to Windows Server and Azure” Even Microsoft (the .NET house) relies on Hadoop since 2011 © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 36. What is Hadoop? Apache Hadoop, an open-source software library, is a framework that allows for the distributed processing of large data sets across clusters of commodity hardware using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 37. How to process big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 38. Hadoop alternatives Integration Suite Hadoop Distribution Apache Hadoop few MapReduce HDFS Ecosystem © Talend 2013 many + Packaging Deployment-Tooling Support Tooling / Modeling Code Generation Scheduling Other Tools (ESB, BPM, ...) + “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Features included
  • 39. (Near) Realtime? ! Hadoop cannot solve every big data problem. Complex event processing and real-time analytics have to be solved in another way (at least today). In-memory computing and streaming platforms are good alternatives or complements to Hadoop for processing and analyzing big data. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 40. Building Blocks for „Intelligent Business Processes“ Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 41. Standards jPDL BPEL BPM BPMN XPDL WF-XML ARIS EPC BPEL4People © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 42. BPMN „Business Process Model and Notation (BPMN) is a graphical representation for specifying business processes in a business process model.“  BPMN 2.0 is also executable! Wikipedia © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 43. Alternatives for BPM Integration Suite BPM Suite BPM Framework Low High Coding Service Tasks Human Interaction GUI © Talend 2013 + BPM Tooling Monitoring Support + ESB BIG DATA / MDM REGISTRY / REPOSITORY RULES ENGINE „YOU NAME IT“ “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner Complexity of Orchestration
  • 44. Building Blocks for „Intelligent Business Processes“ Let‘s realize it !!! Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 45. Frameworks vs. Tools Suite of Tools Specific Tools Frameworks Low High e.g. Camel (Integration) Hadoop (Big Data) Activiti (BPM) © Talend 2013 e.g. Mule ESB (Integration) MapR (Big Data) Camunda (BPM) e.g. Talend Unified Platform i.e ALL-IN-ONE (Integration, Big Data, BPM) “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner Complexity of Orchestration
  • 46. Custom combination of integration, big data and BPM? • A lot of glue code • Testing • Bugfixing • No support Some other people already had the problems you would have! © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner
  • 47. Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 48. Flexible pricing  Increase revenue ➜ ➜ ➜ ➜ With revenue of almost USD 30 billion and a network of 800 locations, Macy's is considered the largest store operator in the USA Daily price check analysis of its 10,000 articles in less than two hours Whenever a neighboring competitor anywhere between New York and Los Angeles goes for aggressive price reductions, Macy's follows its example If there is no market competitor, the prices remain unchanged http://www.t-systems.com/about-t-systems/examples-of-successes-companies-analyze-big-data-in-record-time-l-t-systems/1029702 © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 49. Implementation of an use case Suite of Tools Specific Tools Frameworks Low High e.g. Camel (Integration) Hadoop (Big Data) Activiti (BPM) © Talend 2013 e.g. Mule ESB (Integration) MapR (Big Data) Camunda (BPM) Complexity of Orchestration e.g. Talend Unified Platform i.e ALL-IN-ONE (Integration, Big Data, BPM) “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 50. Talend Unified Platform Big Data Data Quality Data Integration MDM ESB BPM  Commercial license  Subscription model  Support included  Open source license  Free of charge Big Data Data Quality Data Integration MDM ESB  Optional support  Based on open source projects such as Eclipse or Apache Camel, CXF, Hadoop © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 51. Example (Talend): Integration Connect to data sources from competitors, for example via REST service, Twitter API, or custom scripts. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 52. Example (Talend): Big Data Processing Move data to HDFS for processing, as your classic servers and data warehouses are not able to process this semi-structured data fast enough (and cheap), probably. Manipulate the data, in other words, filter relevant information, sort it, and compare it to prices of your products. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 53. Example (Talend): Business Process Start a new instance of a business process to review the result and continue with further tasks, such as calling a web service which does the price reduction in selected locations. Reviews can be done by human interaction or via automated tasks depending on the proposed price reduction. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 54. Implementation of an use case „Talend Unified Platform“ in action... © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 55. Did you get the key message? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 56. Key messages BPM should be used (just) for optimizing business processes! Intelligent business processes need big data and integration! Big data will reduce human interactions in BPM further! © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 57. Did you get the key message? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
  • 58. Thank you for your attention. Questions? KAI WÄHNER kontakt@kai-waehner.de www.kai-waehner.de LinkedIn / Xing @KaiWaehner