This document discusses Kappa Architecture 2.0 and tools for real-time data processing. It introduces Jay Kreps, who coined the term "Kappa Architecture" and has worked on several related projects. The document then contrasts the usual batch processing data flow with the Kappa Architecture approach of combining real-time and batch processing. It provides examples of using Spark and Kafka for real-time analytics and discusses taking the Kappa Architecture to version 3.0 with additional tools like Apache Druid, Kafka Streams, and Google Cloud.

1. Juantomás García - Open Sistemas
Kappa Architecture 2.0

2. київ хороші днів!!
(kyyiv khoroshi dniv)
first

3. Juantomás García
• Data Solutions Manager @ OpenSistemas
• GDE (Google Developer Expert) for cloud
Others
• Co-Author of the first Spanish free software book “La Pastilla
Roja”
• President of Hispalinux (Spanish Linux User Group)
• Organizer of the Machine Learning Spain and GDG Cloud
Madrid.
Who I am

4. What’s Kappa Architecture?
July 2, 2014 Jay Kreps coined the term Kappa
Architecture in an article for O’reilly Radar
“Maybe we could call this the Kappa Achitecture, though it may
be too simple of an idea to merit a Greek letter”

5. Jay has been involved in lots
of projects:
✓ Author of the essay: The
Log: What every software
engineer should know about
real-time data's unifying
abstraction (12/16/2013)
✓ Author of the book I love
Logs
Who is Jay Kreps?

6. •Involved with projects as:
✓ Apache Kafka
✓ Apache Samza
✓ Voldemort
✓ Azkaban
✓ Ex-Linkedin
✓ Now co-founder and CEO of Confluent
Who is Jay Kreps?

7. Usual Data Flow

8. Kappa Architecture Way

9. Tools we use

10. Tools we use

11. Tools we use

12. ✓ If you have an schema spark SQL, is
perfect.
✓ Spark streaming works very fine with spark
and almost each streaming sources.
✓ Structured queries will be a huge advance.
✓ We love Scala, the spirit of Spark.
Some Favorite Spark Features

13. We love code like this:
Some Favorite Spark Features

14. • One of our clients wanted to monitor all the
car's information via OBD II
• OBD II is a car interface with the car
electronics.
• Our client developed an app for reading all
the car information throw ODB II with
bluetooth
A Real Use Case

15. A Real Use Case

16. • We needed to scale the rest interfaces.
There were too many requests.
• MySQL don’t scale
• Client wanted to do realtime expensive
queries.
First Problems

17. Some metrics

18. Architecture v 2.0

19. Architecture v 3.0

20. We can have queries like:
“What are the drivers that are not client
of the X gas brand, has a few gas and
are near of gas station of the brand X and
if true, send a notification with a discount
coupon and a link with the route."
Now we’re more flexible!!

21. • Kappa architecture is not a silver bullet but helps
with a lot of solutions.
• Kafka + spark streaming are our favorite tools
• There are a lots of improvements:
Takeaways
✓ OLAP like Apache Druid
✓ Graph databases like neo4j
✓ Kafka streams and
compacts logs
✓ Apache Beams
✓ Scio Scala bindings

22. Takeaways: Apache Beam

23. Takeaways: Scio Scala Binding

24. Think Big

25. Think Big
• Forget Legacy Architectures
• Forget Old Tools
• Use Light Technologies / Serverless
• Use pieces of Lego
• Mix different technologies from diverse sources

26. Spark Use Cases
Not to do list
•Avoid install & config a server even a
VM.
•Avoid installs tools instead use
containers and/or cloud services.
•In general: think if there is a simpler
way to do it and needs less effort

27. Spark Use Cases
Architecture & Tools
•To use Cloud Services is not a brainer
decision.
•Git + Containers + Kubernetes
•Use the best language* for each
module.
•Use Notebooks: Jupyter, Zeppelin,
DSX
(*) Even java might be an option - unprovable

28. Google Cloud Version

29. How: Make a Knowledge Graph
Questions?
•email: juantomas@opensistemas.com
•twitter: @juantomas
This talk have a free questions lifetime warranty: If you have any questions or concerns
about this talk, feel free to contact me anytime.
Selfie Time: If you like the talk just smile while I take
the selfie ;-)