The document discusses the integration of real-time processing with big data, emphasizing the importance of the lambda architecture and the relay model. It highlights the value of real-time data analytics for businesses, describing the differences between batch processing and real-time implementations using various technologies like Hadoop and MongoDB. The author concludes by encouraging the adoption of scalable big data solutions without the need for extensive upfront investments.

1. REAL-TIME AND BIG DATA
Mahmoud M. Jalajel

2. OUTLINE
• Intro: Real-time with Big Data
• The Lambda Architecture
• The Relay Model

3. WHY SOLVE FOR REAL-TIME
• Real-time offers more business value
• Live Web Analytics
• Recommendations
• Real-time = (semi-) realtime
• Event to index ~ single digit minutes
• Query duration ~ single digit seconds

4. REAL-TIME IMPLEMENTATION
• Incremental Implementation
• Stream processing / No full data context
• A real-time implementation is:
• Far more useful
• Faster
• Easily adaptable to batch mode

5. REAL-TIME IN HADOOP
MongoDb QueryTime
(optimized, single-node)
Hive QueryTime
(5 nodes)
Hangs, crashes, starts begging for
mercy then commits suicide and
weepingly dies
A few hours
2 Seconds 15 Minutes

6. LAMBDA ARCHITECTURE
Created by: Nathan Marz
lambda-architecture.net

7. LAMBDA ARCHITECTURE

8. BASIC ASSUMPTIONS
1. Query = Function(All Data)
2. Data are immutable timely facts
3. Append-Only (CRUD becomes CR)
4. Human Fault-Tolerance

9. THE BATCH LAYER
• Accepts stream of data
• Appends to master dataset
• Uses: HDFS

10. THE SERVING LAYER
• Precomputes different views
• Works on full dataset
• Refreshes regularly offline
• Batch views are usually stored
in a key-value store

13. CHECKPOINT
• Typical Hadoop Setup
• Slow, inefficient
• Outdated. usually lagging by hours or days
• Although accurate for surveyed data
• Costly to re-run. Real-time is not an option

14. THE SPEED LAYER
• Works with recent data
• Complements results
• Incremental implementation

16. THE FULL PICTURE
Query Merging

17. EXAMPLETECHNOLOGIES

18. DRUID EXAMPLE

19. REVIEWING LA
PROs
• Modular
• Flexible
• Self-Auditing
• Proven components
CONs
• Complex
• Maintainability
• Query Merging

20. THE RELAY MODEL

21. RELAY MODEL
Query Merging

22. THE WORKFLOW

23. REVIEWING RM
PROs
• Coherent, Simpler
than LA
• Extensible to full LA
• Cheaper
CONs
• Master Data Storage
• Query flexibility

24. WHY NOT HADOOP NOW?
• Too much time, no capacity
• Too soon or too late
• Too expensive
• Hammer/nail problem

25. CONCLUSIONS
• Think big data, now!
• No need to invest years of development to perfect a
big data system.
• Start now! gradually grow system requirements and
engineering skill-set
• Select scalable components

26. Mahmoud Jalajel – @mjalajel
Questions ?

27. APPENDIX

28. Apache Kafka

29. Apache Storm

30. Apache Storm with external systems