Big Data Architectures @ JAX / BigDataCon 2016
•
5 likes•1,483 views
Mit der Architektur steht und fällt jedes IT-Projekt. Das gilt in noch stärkerem Maße für Big-Data-Projekte, denn hier konnten noch keine Standards über Jahrzehnte ihre Tauglichkeit beweisen. Dennoch verbreiten und etablieren sich auch hier gute und effektive Lösungen. Der Vortrag erklärt, welche Bausteine wichtig für die verschiedenen Einsatzmöglichkeiten im Big-Data-Umfeld sind, und wie sie in konkrete Lösungen gegossen werden können. Dabei beleuchtet er sowohl traditionelle Big-Data-Architekturen als auch aktuelle Ansätze, wie z. B. die Lambda- und die Kappa-Architektur. Ebenfalls ein Thema sind Stream-Processing-Infrastrukturen und ihre Kombination mit Big-Data-Technologien. Ausgehend von einer produkt- und technologieunabhängigen Referenzarchitektur stellt dieser Vortrag verschiedene Lösungsmöglichkeiten auf Basis von Open-Source-Komponenten vor.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Big Data Architectures @ JAX / BigDataCon 2016
Similar to Big Data Architectures @ JAX / BigDataCon 2016 (20)
More from Guido Schmutz
More from Guido Schmutz (20)
Recently uploaded
Recently uploaded (20)
Big Data Architectures @ JAX / BigDataCon 2016
- 1. Guido Schmutz | Trivadis Architektur von Big Data Lösungen
- 2. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Architektur von Big Data Lösungen Guido Schmutz
- 3. Guido Schmutz Working for Trivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Co-Author of different books Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Twitter: gschmutz
- 4. Agenda 1. Introduction 2. Traditional Architecture for Big Data 3. Streaming Analytics Architecture for Fast Data 4. Lambda/Kappa/Unifed Architecture for Big Data 5. Summary
- 5. Introduction
- 6. How to do Big Data?
- 7. Big Data is still “work in progress” Choosing the right architecture is key for any (big data) project Big Data is still quite a young field and therefore there are no standard architectures available which have been used for years In the past few years, a few architectures have evolved and have been discussed online Know the use cases before choosing your architecture To have one/a few reference architectures can help in choosing the right components
- 8. Building Blocks for (Big) Data Processing Data Acquisition Format File System Stream Processing Batch SQL Graph DBMS Document DBMS Relational DBMS Visualization IoT Messaging Analytics OLAP DBMS Query Federation Table-Style DBMS Key Value DBMS Batch Processing In-Memory
- 9. Big Data Ecosystem – many choices ….
- 10. Important Properties to choose a Big Data Architecture Latency Keep raw and un-interpreted data “forever” ? Volume, Velocity, Variety, Veracity Ad-Hoc Query Capabilities needed ? Robustness & Fault Tolerance Scalability …
- 11. From Volume and Variety to Velocity Big Data has evolved … and the Hadoop Ecosystem as well …. Past Big Data = Volume & Variety Present Big Data = Volume & Variety & Velocity Past Batch Processing Time to insight of Hours Present Batch & Stream Processing Time to insight in Seconds Adapted from Cloudera blog article
- 12. Traditional Architecture for Big Data
- 13. “Traditional Architecture” for Big Data Data Collection (Analytical) Data Processing Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine Batch compute Stage Result Store Query Engine Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 14. Use Case 1) – Log File Analysis (Audit, Compliance, Root Cause Analysis) Data Collection (Analytical) Data Processing Result StoreData Sources Data Consumer Channel Batch compute Computed Information Raw Data (Reservoir) Result Store Query Engine Reports Analytic Tools Logfiles
- 15. Use Case 1a) – Log File Analysis (Audit, Compliance, Root Cause Analysis) Data Collection (Analytical) Data Processing Result StoreData Sources Data Consumer Channel Batch compute Computed Information Raw Data (Reservoir) Result Store Query Engine Reports Analytic Tools Logfiles
- 16. Use Case 2) – Customer Data Hub (360 degree view of customer) Data Collection (Analytical) Data Processing Result StoreData Sources Data Consumer RDBMS Batch compute Computed Information Raw Data (Reservoir) Result Store Query Engine Reports Service Analytic Tools Alerting Tools
- 17. Use Case 2a) – Customer Data Hub (360 degree view of customer) Data Collection (Analytical) Data Processing Result StoreData Sources Data Consumer RDBMS (CDC) Batch compute Computed Information Raw Data (Reservoir) Result Store Query Engine Reports Service Analytic Tools Alerting Tools Channel
- 18. “Hadoop Ecosystem” Technology Mapping Data Collection (Analytical) Data Processing Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine Batch compute Staging Result Store Query Engine Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 19. Apache Spark – the new kid on the block Apache Spark is a fast and general engine for large-scale data processing • The hot trend in Big Data! • Originally developed 2009 in UC Berkley’s AMPLab • Can run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk • One of the largest OSS communities in big data with over 200 contributors in 50+ organizations • Open Sourced in 2010 – since 2014 part of Apache Software foundation • Supported by many vendors
- 20. Motivation – Why Apache Spark? Apache Hadoop MapReduce: Data Sharing on Disk Apache Spark: Speed up processing by using Memory instead of Disks map reduce . . . Input HDFS read HDFS write HDFS read HDFS write op1 op2 . . . Input Output Output
- 21. Apache Spark “Ecosystem” Spark SQL (Batch Processing) Blink DB (Approximate Querying) Spark Streaming (Real-Time) MLlib, Spark R (Machine Learning) GraphX (Graph Processing) Spark Core API and Execution Model Spark Standalone MESOS YARN HDFS Elastic Search NoSQL S3 Libraries Core Runtime Cluster Resource Managers Data / Data Stores
- 22. “Spark Ecosystem” Technology Mapping Data Collection (Analytical) Data Processing Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine Batch compute Stage Result Store Query Engine Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 23. Use Case 3) – (Batch) Recommendations on Historic Data Data Collection (Analytical) Data Processing Result StoreData Sources Data Consumer Machine Batch compute Computed Information Raw Data (Reservoir) Result Store Query Engine Reports Service Analytic Tools Alerting Tools DB StagingFile Channel
- 24. Traditional Architecture for Big Data • Batch Processing • Not for low latency use cases • Spark can speed up, but if positioned as alternative to Hadoop Map/Reduce, it’s still Batch Processing • Spark Ecosystems offers a lot of additional advanced analytic capabilities (machine learning, graph processing, …)
- 25. Streaming Analytics Architecture for Big Data
- 26. Streaming Analytics Architecture for Big Data aka. (Complex) Event Processing) Data Collection Batch compute Data Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social Logfiles Sensor RDBMS ERP Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store = Data in Motion = Data at Rest
- 27. Use Case 4) Algorithmic Trading Data Collection Batch compute Data Sources Channel Data Consumer Analytic Tools Alerting Tools Sensor Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store
- 28. Use Case 5) Real-Time Analytics / Dashboard Data Collection Batch compute Data Sources Channel Data Consumer Analytic Tools Sensor Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store
- 29. Native Streaming vs. Micro-Batching Native Streaming • Events processed as they arrive • + low-latency • - throughput • - fault tolerance is expensive Micro-Batching • Splits incoming stream in small batches • + high(er) throughput • + easier fault tolerance • - lower latency Source: Distributed Real-Time Stream Processing: Why and How by Petr Zapletal
- 30. Unified Log (Event) Processing Stream processing allows for computing feeds off of other feeds Derived feeds are no different than original feeds they are computed off Single deployment of “Unified Log” but logically different feeds Meter Readings Collector Enrich / Transform Aggregate by Minute RawMeter Readings Meter with Customer Meter by Customer by Minute Customer Aggregate by Minute Meter by Minute Persist Meter by Minute Persist Raw Meter Readings
- 31. Streaming Analytics Technology Mapping Data Collection Batch compute Data Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social Logfiles Sensor RDBMS ERP Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store = Data in Motion = Data at Rest
- 32. Streaming Analytics Architecture for Big Data The solution for low latency use cases Process each event separately => low latency Process events in micro-batches => increases latency but offers better reliability Previously known as “Complex Event Processing” Keep the data moving / Data in Motion instead of Data at Rest => raw events were not stored
- 33. Keep raw event data Data Collection Batch compute Data Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social Logfiles Sensor RDBMS ERP Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store = Data in Motion = Data at Rest (Analytical) Batch Data Processing Raw Data (Reservoir)
- 34. Lambda Architecture for Big Data
- 35. “Lambda Architecture” for Big Data Data Collection (Analytical) Batch Data Processing Batch compute Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 36. Use Case 6) Social Media Analysis Data Collection (Analytical) Batch Data Processing Batch compute Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 37. Use Case 7) Customer Event Hub Data Collection (Analytical) Batch Data Processing Batch compute Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest Social RDBMS (CDC) Sensor ERP Logfiles
- 38. Lambda Architecture for Big Data Combines (Big) Data at Rest with (Fast) Data in Motion Closes the gap from high-latency batch processing Keeps the raw information forever Makes it possible to rerun analytics operations on whole data set if necessary => because the old run had an error or => because we have found a better algorithm we want to apply Have to implement functionality twice • Once for batch • Once for real-time streaming
- 39. „Kappa“ Architecture for Big Data
- 40. “Kappa Architecture” for Big Data Data Collection “Raw Data Reservoir” Batch compute Data Sources Messaging Data Consumer Reports Service Analytic Tools Alerting Tools Social Logfiles Sensor RDBMS ERP Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store Raw Data (Reservoir) = Data in Motion = Data at Rest Computed Information
- 41. Use Case 8) Social Network Analysis Data Collection “Raw Data Reservoir” Batch compute Data Sources Messaging Data Consumer Reports Service Analytic Tools Alerting Tools Social Logfiles Sensor RDBMS ERP Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store Raw Data (Reservoir) = Data in Motion = Data at Rest Computed Information
- 42. Kappa Architecture for Big Data Today the stream processing infrastructure are as scalable as Big Data processing architectures • Some using the same base infrastructure, i.e. Hadoop YARN Only implement processing / analytics logic once Can Replay historical events out of an historical (raw) event store • Provided by either the Messaging or Raw Data (Reservoir) component Updates of processing logic / Event replay are handled by deploying new version of logic in parallel to old one • New logic will reprocess events until it caught up with the current events and then the old version can be de-commissioned.
- 43. „Unified“ Architecture for Big Data
- 44. “Unified Architecture” for Big Data Data Collection (Analytical) Batch Data Processing (Calculate Models of incoming data) Batch compute Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest Prediction Models
- 45. Use Case 8) Industry 4.0 => Predictive Maintainance Data Collection (Analytical) Batch Data Processing (Calculate Models of incoming data) Batch compute Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools RDBMS Sensor ERP Logfiles Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) Prediction Models J-PMML
- 46. Summary
- 47. Summary Know your use cases and then choose your architecture and the relevant components/products/frameworks You don’t have to use all the components of the Hadoop Ecosystem to be successful Big Data is still quite a young field and therefore there are no standard architectures available which have been used for years Lambda, Kappa Architecture & Unified Architecture are best practices architectures which you have to adapt to your environment => Combination of traditional Big Data Batch & Complex Event Processing (CEP)