Pig, Hive, Flink, Kafka, Zeppelin... if you now wonder if someone just tried to offend you or are those just Pokemon names, then this talk is just for you! Big Data is everywhere and new tools for it are released almost at the speed of new JavaScript frameworks. During this entry level presentation I will walk though the challenges which Big Data presents, reflect how big is big and introduce currently most fancy and popular (mostly open source) tools. I'll try to spark off interest in Big Data by showing application areas and by throwing ideas where you can later dive into.

1. Small Intro to Big Data
Michał Matłoka @mmatloka

3. What
is
Seahorse?
?

4. What
is
Seahorse?

5. What
is
Azurill?
?

6. What
is
Azurill?
https://pixelastic.github.io/pokemonorbigdata/

7. HTTPS://GITHUB.COM/
ONURAKPOLAT/AWESOME-BIGDATA

9. What
is
Big Data?
3Vs
Volume Velocity
Variety

10. HOW BIG IS BIG?

11. –ThoughtWorks Technology Radar
“We continue to see organizations chasing "cool" technologies,
taking on unnecessary complexity and risk when a simpler
choice would be better. One particular theme is using distributed,
Big Data systems for relatively small data sets. (…)”

12. PROBLEM: AIR POLUTION DATA

13. STORING

14. VERTICAL SCALING

15. Example limitations:
• Max 48 nodes
• Read-only nodes
• Cross-shard joins
• Auto-increments
• Distributed transactions,
possible but…
It can work!
Relational
scaling
(horizontal)

16. In distributed system
you can only have two
of three guarantees:
• Consistency
• Availability
• Partition Tolerance
CAP theorem
(Brewer’s theorem)

17. • Key-value (Redis, Dynamo, ...)
• Column (Cassandra, HBase, ...)
• Document (MongoDB, … )
• Graph (Neo4J, … )
• Multi-model (OrientDB, …)
Apple - 115k Cassandra nodes with
over 10PB of data!
NoSQL
(Not only SQL)

18. Basically
Available
Soft state
Eventual consistency
Atomicity
Consistency
Isolation
Durability

19. THERE IS NO SINGLE CHOICE

20. BATCH PROCESSING

21. • Can take hours = not real-time
• May include many data sources
• Different types of data
• Big period of time (month, year?)
Batch
processing

22. Map Reduce (Word Count)

23. Batch
processing
(tools)

24. STREAM PROCESSING

25. • Real-time or near real-time
• Processing (usually) does not end
• Source is a “stream” of messages/
file lines/packets …
• Event time vs processing time
• Windows - fixed, sliding, session
• Watermarks
• State
Stream
processing

26. Stream
processing
(tools)

27. Stream
processing
(tools)

28. • Native vs mico-batch
• Latency
• Throughput
• Delivery guarantees
• APIs - compositional/declarative
Stream
processing
(differences)

29. Lambda Architecture

30. Kappa Architecture

31. FINDING VALUE IN DATA

32. Analysis
(SQL Engines & Querying)

33. Machine
Learning

34. RESOURCE MANAGERS

35. Resources
Managers

37. WHERE TO START?

38. • Find a problem to solve
• https://github.com/
onurakpolat/awesome-
bigdata
• SMACK? (Spark Mesos
Akka Cassandra Kafka)
Where to
start?

39. Articles & references
• http://lambda-architecture.net/
• http://milinda.pathirage.org/kappa-
architecture.com/
• https://www.cakesolutions.net/teamblogs/
comparison-of-apache-stream-processing-
frameworks-part-1
• https://www.oreilly.com/ideas/the-world-beyond-
batch-streaming-101
• https://www.oreilly.com/ideas/the-world-beyond-
batch-streaming-102

40. Thank you, Q&A?
@mmatloka
mmatloka
softwaremill.com/blog