Juan Tomás García, Open Sistemas

1. Kappa Architecture:
IoT of the cars
Juantomás García

2. Who am I
• Data Solutions Manager at Open Sistemas
• Co-author of the first spanish book about free
software “La pastilla roja”
• Former president of Hispalinux (Spanish Linux User
Group).
• Co-organizer of Machine Learning Spain Meetup

3. What’s Kappa Architecture
July 2, 2014 Jay Kreps coined the term
Kappa Architecture in an article for
O’reilly Radar

4. Who is Jay Kreps
Jay has been involved in lots of projects:
Author of the essay:
The Log: What every software engineer
should know about real-time data's
unifying abstraction (12/16/2013)
https://engineering.linkedin.com/distributed-systems/log-what-every-software-
engineer-should-know-about-real-time-datas-unifying

5. Who is Jay Kreps
Author of the book: I ♥ Logs

6. Who is Jay Kreps
Involved with projects as:
Apache Kafka
Apache Samza
Voldemort
Azkaban
Ex-Linkedin
Now co-founder and CEO of Confluent

7. Usual data flow

8. Kappa Architecture Way

9. Tools we use to implement
Kappa Architecture

10. Tools we use to implement
Kappa Architecture

11. Tools we use to implement
Kappa Architecture
• Our favorite kafka features.
• Easy to use
• Robust & Resilient
• Kafka connectors
• Kafka streams
• Compact logs
• Very fast writes and reads

12. Tools we use to implement
Kappa Architecture

13. Some favorite Spark features
• If you have an schema spark SQL is
perfect.
• Spark streaming work very fine with spark
and almost all streaming sources.
• Structured queries will be a huge advance.
• We love scala and scala is the spirit of
spark.

14. Some favorite Spark features
• We love code like this:

15. A real case
• One of our clients want to monitor all the
cars informations via OBD II
• OBD II is a car interface with the car
electronics.
• Client develop an app to read all the car
information throw a ODB II with bluetooth

16. A real case

17. First Problems
• We need to scale the rest interfaces too
many requests.
• MySQL don’t scale
• Client wants to make realtime expensive
queries.

18. Some Numbers

19. Architecture V2

20. Architecture KappaV3

21. Now we’re more flexible
• We can have queries like:
• What are the drivers that are not client
of the X gas brand, has a few gas and
are near of gas station of the brand X
and if true, send a notification with a
discount coupon and a link with the
route.

22. Takeaways
• Kappa architecture is not a silver bullet but
helps in a lot of solutions.
• Kafka + spark streaming is our favorite
tools
• There a lots of improvements:
• OLAP like Apache Druid
• Graph databases like neo4j
• Kafka streams and compacts logs

23. ¿Questions?
• My contact info:
• Juantomás García
• juantomas@opensistemas.com
• @juantomas