Big data architectures

1,612 views
1,451 views

Published on

Presentation given at @DamnData discussing different architecture types for BigData environments

Published in: Technology

Big data architectures

  1. 1. BigData Architectures Daan Gerits Dasos
  2. 2. Volume IOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOO OIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOI OIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOII IOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIII OIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOII OIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOII We already have that: - NAS/SAN - High Performance Computing
  3. 3. Variety IOII IOIIIOIIIOII IOII IOII IOII IOII IOII We already have that: - Meta-modeling - NAS/SAN
  4. 4. Velocity OIII IOII IOII OO OIII We already have that: - Complex Event Processing
  5. 5. But do you have all of that in 1 platform?
  6. 6. But How??
  7. 7. Architectures (Thx Nathan Marz!)
  8. 8. Analytical Big Data Analysis Oriented Optimize Non-intrusive
  9. 9. Delta Apps Dashboards Distributed Database Data Sources Ingestion Engine Enrich Data Systems
  10. 10. Delta Impala, Hive, ... Apps Dashboards Distributed Database Data Sources Flume, Sqoop, Scribe, ... MR, Pig, Crunch, Mahout, ... MR, Pig, Crunch, ... Data Systems
  11. 11. Delta Analytical Big Data architecture for enriching mostly structured data with the goal to optimize business processes.
  12. 12. Delta Apps Dashboards Distributed Database Data Sources Ingestion Engine Enrich Overload! Data Systems
  13. 13. Delta Be write-heavy or read-heavy NOT both!
  14. 14. Operational Big Data Focussed on Day-today business Innovate (Non-)intrusive (Thx Nathan Marz!)
  15. 15. Lambda Realtime View A Realtime Processing Apps Realtime View B Dashboard Realtime View C Data Sources Batch View A Fact Store Just In Time Combiner Batch View B Batch View C Reports
  16. 16. Lambda Cassandra* Storm Apps Cassandra* Dashboard Cassandra* Custom Code* Data Sources ElephantDB HDFS ElephantDB ElephantDB Reports
  17. 17. Lambda Operational Big Data architecture for storing and processing multi-structured and immutable data with the goal to Innovate business
  18. 18. Technologies to use Pick your stack!
  19. 19. Advice Pilots, PoC, PoT, … do them! Be pragmatic, start skinny In Belgium: Variety > Volume Be prepared to pivot on technologies
  20. 20. Questions? Thoughts? Ideas? Disagreements? ... daan.gerits@dasos.be www.dasos.be @daangerits All images are used merely for illustrational means. In no way was it my purpose to violate any rights by using
  21. 21. BigData Architectures Backup Slides
  22. 22. Variety Velocity Volume
  23. 23. Lambda Multistructured Unstructured Restructured

×