Common patterns and anti-patterns to consider when integrating Kafka, Cassandra and Storm for a real-time streaming analytics platform.

1. REAL-TIME ANALYTICS
WITH KAFKA, CASSANDRA
& STORM
Dr. John Georgiadis
Modio Computing

2. Modio Computing
USE CASES
• Collecting/processing measurements from large
sensor networks (e.g. weather data).
• Aggregated processing of financial trading streams.
• Customer activity monitoring for advertising
purposes, fraud detection, etc.
• Real-time security log processing.

3. Modio Computing
SOLUTION APPROACH
• Real-time Updates: Employ streaming instead of batch analytics.
• Apache Storm: Large installation base. Streaming & micro-batch.
• Apache Spark: Uniform API for batch & micro-batch. On top of
YARN/HDFS. Micro-batch less mature but catching-up quickly.
• Large data sets +Time Series + Write-Intensive + Data Expiration =
Apache Cassandra

4. Modio Computing
ARCHITECTURE

5. Modio Computing
APACHE KAFKA
• N Nodes
• TTopics
• Replication Factor: Defines high availability
• Partitions:They define parallelism level.A single consumer per
partition.
• Consumer discovers cluster nodes through Zookeeper
• Consumer partition state is just an integer: the partition offset.

6. Modio Computing
STORM
• Storm is a distributed computing platform. In Storm a distributed
computation is a directed graph of interconnected processors (topology)
that exchange messages.
• Spouts: Processors that inject messages into the topology.
• Bolts: Processors that process messages including sending to 3rd parties
(.e.g persistence).
• Trident: High-level operations on message batches. Support batch replay
in case of failure.Translates to a graph of low-level spouts and bolts.

7. Modio Computing
STORM :: NIMBUS
• A single controller (Nimbus)
where topologies are submitted.
Nimbus breaks topologies in tasks and
forwards to supervisors which spawn
one or more workers(processes) per
task.
Nimbus redistributes tasks in case a supervisor fails.
Nimbus is not HA. If Nimbus fails, running topologies are not
affected.

8. Modio Computing
STORM :: SUPERVISOR
• 1 supervisor per host.
• Supervisor registers with ZK at startup and thus it’s discoverable
by Nimbus.
• Supervisor spawns Worker JVMs: one process per topology.
• JAR submitted to Nimbus is copied to Worker classpath.
• When a Supervisor dies, all Worker tasks are migrated to the
remaining Supervisors.

9. Modio Computing
STORM ::TRIDENT
• When to use Micro-batch (akaTrident) instead
of Streaming.
• Millisecond latency not required.Typical
Trident latency threshold: 500ms.
Allows batch mode persistence operations.
High-level abstractions: partitionBy, partitionAggregate, stateQuery,
partitionPersist.
Batch processing timeout/exception will cause a replay of the batch
provided the Spout supports replays (Kafka does):At-least-once semantics.

10. Modio Computing
STORM :: PARALLELISM
• Parallelism = Number of threads executing a topology cluster-wide.
• Parallelism <= CPU threads/worker x Workers
• Define per-topology max #workers (explicitly) and max parallelism (implicitly).
• Define explicitly topology step parallelism.
Max parallelism = Σ(step parallelism).
• Trident merges multiple steps into the same thread/node. Last parallelism statement is the
effective parallelism.
• Repartition operations define step merging boundaries.
• Repartition operations (shuffle, partitionBy, broadcast, etc.) imply network transfer and are
expensive. In some cases they are disastrous to performance!

11. Modio Computing
STORM :: PERFORMANCETUNING
• Spouts must match upstream parallelism: one spout per Kafka partition.
• Little’s Law: Batch Size =Throughput x Latency
• Adjust batch size: (Kafka Partitions) x (Kafka fetch size)
Larger batch size = {higher throughput, higher latency}
Increase batch size gradually until latency starts increasing sharply.
• Identify the slowest stage (I/O or CPU bound):
• You can’t have better throughput than the throughput of your slowest stage.
• You can’t have better latency than the sum of individual latencies.
• If CPU bound, increase parallelism. If I/O bound increase downstream (i.e storage) capacity.

12. Modio Computing
CASSANDRA ::THE GOOD
• Great write performance. Decent read performance.
• Write latency: 20μs-120μs
• 15K writes/sec on a single node
• Extremely stable.Very low record of data corruption.
• Decentralized setup: all cluster nodes have the same setup.
• Multi-datacenter setups.
• Configurable consistency of updates: ONE, QUORUM,ALL.
• TTL per cell (row & column).
• Detailed metrics: #operations, latencies, thread pools, memory, cache performance.

13. Modio Computing
CASSANDRA ::THE “FEATURES”
• All partition keys must be set in queries.
• All primary keys preceding an initialized primary key with a value must also be initialized in
queries.
• TTL is not allowed on cells containing counters.
• NULL values are not supported on primary keys.
• Range queries can only be applied on the last column of the composite primary key that
appears in the query.
• Disjunction operator (OR) is not available.The IN keyword can be used in some cases instead.
• Row counting is a very expensive operation.

14. Modio Computing
CASSANDRA :: PERFORMANCE
• Design the schema around the partition key.
• Keep each partition size small (no more than a few 100s entry) as reading will fetch the whole partition.
• Leverage Key cache
• Avoid making time fragments part of the partition key as this will direct all activity to the node that is the
partition owner at a given date.
• Query/Update Plan:
• Avoid range queries and the IN operator as it requires contacting multiple nodes and assembling the
results at the coordinator node.
• Use prepared statements to avoid repeated statement parsing.
• Prefer async writes combined with a max pending statements threshold.
• Best performance out of batches containing statements with the same partition key.

15. Modio Computing
CASSANDRA :: CLUSTER
• One or more data nodes are also “seed”nodes
acting as the membership gatekeepers.
• Table sharding across the cluster based on the
partition key hash (token).
• Table replication according to replication
factor (RF). Configurable per keyspace (database).
• The Java driver has several load balancing approaches:
• Token-aware: sends each statement to the node that actually will store it. Random selection
amongst the nodes for a given replica set.
• Latency-aware: sends each statement to the node with the fastest response.
• Round-robin & custom load balancers supported.

16. Modio Computing
• Kafka
• N-way replication: N-1 node failures.
• Clients dynamically reconfigured if accessing through Zookeeper.
• Storm
• For cluster size N, X supervisor failures provided (N-X) nodes have memory
to accommodate X JVM worker processes.
• Incomplete batches replayed:At-least-once semantics.
• Cassandra
• N-way replication: N-1 node failures if using ONE consistency, if using QUORUM consistency.
• Clients require a list of all cluster nodes.
• Zookeeper
• Majority voting is required:At most F failures in cluster with 2F+1 nodes. Leader re-election very fast (200ms).
• Sizes bigger than 3-5 not recommended due to decreasing write performance.
• If majority voting is lost, Storm will stop. Kafka will fail to commit client offsets. If majority is regained Storm will resume. Kafka
brokers will resume in most cases.
FAILURE SCENARIOS