1. POLYGLOT PROCESSING –
AN INTRODUCTION
Dr. Mohan K. Bavirisetty
Chief Scientist
Modern Renaissance
2. Agenda
1. Big Data Landscape
2. Lambda vs. Kappa Architecture
3. Spark vs. Storm vs. Flink
4. Demo 1 – Apache Spark
5. Demo 2 – Storm, Kafka and Redis
6. Demo 3 – Flink with Data Stream API?
7. Summary
8. Questions
The purpose of computing is insight not data – Richard Hamming
4. What is Big Data?
Big data is high-volume, high-velocity and high-
variety information assets that demand cost-
effective, innovative forms of information processing
for enhanced insight and decision making.
Source: Gartner Research
7. • Batch Operations1
• Micro batch Operations2
• Real-time Streaming3
3 Common Kinds of Workloads
“Evidence-based decision-making (aka Big Data) is not just the latest fad, it's
the future of how we are going to guide and grow business.”
– Kristen Hammond, CTO, Narrative Sciences
8. 8 Requirements of Real-time Computing
Keep Data Moving
Allow SQL Queries
Handle Stream Imperfections
Generate Predictable Outcomes
Integrate Streaming Data and Stored Data
Guarantee Data Safety and Availability
Partition and Scale Applications Automatically
Process and Respond Instantaneously
12. Polyglot …..
•One who is versed in many languages …Polyglot
•Different languages, frameworks and services
•Example Java with Scala, Clojure inside Trident
Polyglot
Programming
•Capacity to store data in multiple formats
•Structured, document, Log, GPS
Polyglot
Persistence
•Refers to capability to process any kind of data,
any kind of workload, any kind of workflow
Polyglot
Processing
16. What is Apache Storm?
Apache Storm is a free and open source
distributed real-time computation system it
makes it easy to reliably process unbounded
streams of data.
17. Why Apache Storm?
Storm is fast, horizontally scalable,
fault-tolerant, easy to setup and
operate and programming language
agnostic
28. Summary
• Big Data Challenges are being met with new and
innovative approaches and architectures.
• Lambda Architecture is a pragmatic near-term
solution. Fidelity is already implementing it.
• Kappa Architecture could turn out to be long-term
elegant solution to Polyglot Processing.
• Apache Spark, Strom and Flink have their strengths
and niche areas of applicability.
• Apache Samoa, Apache Zappelin and Tacheon add
value further by providing additional capabilities
31. QUESTIONS?
We do not learn by inference and deduction and the application of mathematics to
philosophy, but by direct intercourse …
- Henry David Thoreau