Slides from Coding serbia meetup held on 29.09.2015.

1. Why go distributed
Matija Gobec
SmartCat.io
@mad_max0204
@SmartCat_io

2. Agenda
Introduction to distributed systems
History
Monolithic vs distributed
Pros vs Cons
SMACK stack
Spark
Mesos
Akka

3. History
“A distributed system is one in which the
failure of a computer you didn’t even know
existed can render your own computer
unusable.”
Leslie Lamport
28.03.1987.

4. Definition
“Distributed system is a collection of independent computers
that appears to its users as a single coherent system.”

5. Goals of distributed systems
Resource sharing
Scalability
Fault tolerance
Concurrency
Transparency

6. Common delusions
Network is reliable
Latency is zero
Transport cost is zero
Network is homogeneous

7. Problems of distributed systems
Hardware failure
Network failure and splits
Network latency
Availability zone and region outages
Data corruption
Inconsistency
Security

8. Two generals problem

9. Monolithic architecture

10. Asynchronous fault tolerant architecture
https://github.com/killrweather/killrweather

11. Pros
Economic (price/performance)
Reliability (fault tolerance)
Speed and computational power
Incremental growth
Cons
Complexity
Security
Network limitations

12. SMACK stack

13. General purpose data processing engine
Data abstraction as RDD (Resilient Distributed Dataset)
Batch, iterative and streaming analysis
Faster than MapReduce: 10x on disk / 100x in memory
Runs on YARN, Mesos or standalone
Spark
Word count
val textFile = spark.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

14. Mesos
Server kernel with resource management and scheduling APIs
Hardware abstraction
Fault tolerant with master election
Scalable
Native container isolation
Docker support

15. Akka
Toolbox and runtime for CDR systems
Actor model
Mathematical model of concurrent computation
Message based local decisions
High level of abstraction
Fault tolerant system
“Let it crash” model
Supervisor hierarchies
Thread
Actor
Behavior
State
mailbox

16. Cassandra
Amazon Dynamo & Google Bigtable
Open sourced by Facebook in 2008.
Share nothing architecture
Fast, scalable, distributed partitioned row store
Multi data center deployments
Masterless replication

17. Kafka
Publish-subscribe messaging
Cluster-centric design
Fast, scalable and fault tolerant
Partitioned data streams
Data replication
Great for building data pipelines

18. https://github.com/killrweather/killrweather

19. Spark + Cassandra
How to do analytics on Cassandra ?
No JOIN
No GROUP BY
Filter by PRIMARY KEY only (map of maps)

21. Spark + Cassandra
Spark-cassandra-connector (https://github.com/datastax/spark-cassandra-connector)
Cassandra table as RDD or DStream
Data locality aware (performance)
Server-side filtering
Cross-table operations (JOIN, UNION)
Batch and streaming operations

22. Why go distributed ?

23. Why go distributed ?
To solve problems that monolithic systems have
To easily gain performance at scale
To get near-realtime analytics
IT’S FUN!!!

25. Thank you
Matija Gobec
@mad_max0204
matija0204@gmail.com
matija.gobec@smartcat.io
SmartCat
www.smartcat.io
@SmartCat_io
https://github.com/smartcat-labs