Building Scalable Big Data Pipelines

6,737 views

Published on

Talk held at the NoSQL Roadshow on 19.09.2013 in Zurich.

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,737
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
169
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Building Scalable Big Data Pipelines

  1. 1. Building Scalable Big Data Pipelines Christian Gügi, Solution Architect 19.09.2013 NOSQL SEARCH ROADSHOW ZURICH
  2. 2. AGENDA  Opportunities & Challenges  Integrating Hadoop  Lambda Architecture  Lambda in Practice  Recommendations
  3. 3. ABOUT ME  Solution Architect @ YMC  Founder and organizer Swiss Big Data User Group   http://www.bigdata-usergroup.ch/  Contact   christian.guegi@ymc.ch   http://about.me/cguegi   @chrisgugi
  4. 4. ABOUT YMC  Founded in 2001  Based in Kreuzlingen, Switzerland  Big Data Analytics, Web Solutions and Mobile Applications  24 experts   Consulting, creation, engineering
  5. 5. OPPORTUNITIES &
  6. 6. A.  New sources and types from inside & outside organisations   “Internet of things”, sensors, RFID, intelligent devices, etc.   Unstructured information – documents, web logs, email, social media, etc.   Trusted 3rd party sources – industry provider & aggregators, governments “Open Data”, weather, etc. B.  Technology innovations to exploit new world of data   Low cost storage and process power (cloud, on-premise & hybrid)   New software patterns to handle speed & volume, structured and unstructured (In-memory computation, Hadoop, Mapreduce, etc.)   Revolution in user experience, analytics, recommendations BIG DATA – WHAT IS THE BIG DEAL?
  7. 7. BIG DATA – CHALLENGES • Align business strategy • Data Management • Privacy protection • Lack of skilled and experienced people • Volume • Velocity • Variety • Veracity Character of data Overwhelming landscape & integration Available talent Organisational issues
  8. 8. INTEGRATING
  9. 9. TYPICAL RDBMS SZENARIO Data Sources Data Systems Apps DWH RDBMS RDBMS NFS Others BI Web Mobile ETL
  10. 10. BIG DATA SZENARIO Data Sources Data Systems Apps DWH RDBMS RDBMS NFS Logs BI Web Mobile Social Media Sensors Hadoop 1) Recommendations, etc. 1)
  11. 11. HADOOP ECOSYSTEM
  12. 12. LAMBDA
  13. 13. ARCHITECTURE  Credits Nathan Marz  Former Engineer at Twitter  Storm, Cascalog, ElephantDB LAMBDA http://www.manning.com/marz/
  14. 14. DESIGN PRINCIPLES Lambda Architecture  Human fault-tolerance  Data immutability  Re-computation
  15. 15. HUMAN FAULT-TOLERANCE Lambda Architecture  Design for human error   Bugs in code   Accidental data loss   Data corruption  Protect good data, so you can always fix what went wrong
  16. 16. DATA IMMUTABILIY Lambda Architecture  Store data in it’s rawest form  Create and read but no update  No data can be lost   To fix the system just delete bad data   Can always revert to a true state
  17. 17. DATA IMMUTABILIY Lambda Architecture Name Location Time Alice Zurich 2009/03/29 Bob Lucerne 2012/04/12 Tom Bern 2010/04/09 Name Location Time Alice Zurich 2009/03/29 Bob Lucerne 2012/04/12 Tom Bern 2010/04/09 Alice Basel 2013/08/20 Name Location Alice Zurich Bob Lucerne Tom Bern Name Location Alice Basel Bob Lucerne Tom Bern Capturing change traditionally (mutability) Capturing change (immutability)
  18. 18. RE-COMPUTATION Lambda Architecture  Always able to re-compute from historical data  Basis for all data systems   query = function(all data) All Data Pre-computed views Query
  19. 19. LAYERS Lambda Architecture http://www.ymc.ch/en/lambda-architecture-part-1
  20. 20. Lambda in Practice
  21. 21. ONLINE MARKETING  Tracking and analytics solution  Improve customer targeting and segmentation  Various reports  Real-time not required
  22. 22. OVERVIEW HBase FTP HDFS AdServer Flume log HDFS Campaign Database Sqoop csv csv Up- & Download Hive fs -put Aggregated Data Web PigImpala DWH BI apps Oozie ZooKeeper Cloudera Manager
  23. 23. DATA PIPELINE FTP HDFS AdServer Flume log HDFS Campaign Database Sqoop csv csv fs -put M/R Avro Avro Avro Extracting Transformation M/R M/R Loading Tracking Profiles Bulk Importer DWH
  24. 24. ADVANTAGES  Extensible – easily add speed layer later on  Complements existing DWH/BI system  ETL phases are decoupled  Reliable   Infrastructure   Each step can be replayed  Scalable   Storage   Processing  Highly available  Ad-hoc analysis right from the beginning
  25. 25. RECOMMENDATIONS
  26. 26. RECOMMENDATIONS  Not a fixed, one-size-fits-all approach   Adopt to your needs/requirements  Hadoop complements existing systems  How real-time do I need to be?  Immutability and pre-computation are just good ideas!   Store information in rawest format possible   Use a serialization framework (Avro, Thrift, Protocol Buffers)
  27. 27. THANK YOU!
  28. 28. YMC AG Sonnenstrasse 4 CH-8280 Kreuzlingen Switzerland Photo Credits: Slide 05: Success opportunity achieve by Stephen McCulloch Slide 08: Matrix by Gamaliel Espinoza Macedo. Slide 12: Layers by Katelyn Leblanc Slide 20: Mining For Information by JD Hancock Slide 27: Warning Question by longzijun @chrisgugi CONTACT US christian.guegi@ymc.ch Tel. +41 (0)71 508 24 76 www.ymc.ch

×