A Tale of Two
Technologies
A Story of the IoT “revolution”
It was the age of connectedness
It was the age of disconnectedness
Failure tolerance
is not optional
It was the age of wisdom
It was the age of foolishness
It was the epoch of complexity
Did you just tell me to go
#@$#@$ myself?
It was the epoch of simplicity
We brought as much simplicity as
we could to the menagerie known
as Hadoop, but it’s still ...
What is the Internet of Things?
“The Internet of Things (IoT) refers to uniquely identifiable objects and their
virtual re...
No really what is IoT?
● It’s literally the act of connecting “things” to
the Internet
● It predates the World Wide Web
● ...
So IoT is old news?
Most definitely
So IoT is just hype?
Home security and automation
Fitness trackers
Your (driverless?) car
Medical devices
Sensors
Industri...
What is Cassandra?
● A massively scalable distributed database
● Chooses availability over strong consistency
(yes, that r...
What is Spark?
● DAG is a logical superset of M/R
● Adopts much of the Hadoop ecosystem,
without being bound by it
● Intel...
What does IoT need from big data?
● Log time-series events -- at scale
● Gather meaning from that data -- at scale
● Repor...
Logging events at scale
Gathering meaning at scale
Reporting at scale
● Canned reports
● Ad-hoc querying and reporting
● Drill down / exploratory
● Alerting
● Aggregation
● ...
Take action at scale
● Stateless application servers
● Horizontally scaled and co-located with
Cassandra and Spark in each...
The architecture...
Spark Streaming to Cassandra
Cassandra to Spark Streaming
Multi-DC
DC1 DC2
Write anywhere.
Things that go together
Things that go together
A Tale of Two Summits
Upcoming SlideShare
Loading in …5
×

Cassandra Day NY 2014: Using Spark Streaming for High Velocity Analytics on Cassandra

726 views

Published on

0.) With each device comes an implicit contract with the end user: you give us the data, we give you the results. Now. Not tomorrow. Not even fifteen minutes from now.

1.) The flip side of getting data in real time is that users expect results in real time.

2.) In return for being wired into the internet 24x7 customers demand a similar level of responsiveness and even better availability.

__

Spark Streaming and Cassandra form the ideal combination of high velocity CEP and analytics with a high velocity and always on database.

Today’s solutions don’t scale to the Internet of tomorrow. The always-on nature of the emerging Internet of Things space means you need to process information at previously unseen scale and, more difficult, make sense out of that data.

Cassandra is the leader in large scale, high velocity, time series data workloads. While the Hadoop world has been stuck with legacy “batch analytics” technology, Cassandra users have been increasingly focused on the “now”. Fast answers to easy questions about your data, at any velocity, and any scale. But Cassandra has always been weak on the “complex questions” problem. DataStax integrated with Hadoop to overcome this limitation, but it was always an awkward fit. Slow batch analytics on top of fast moving data really doesn’t do you much good.

But Spark, and in this case, Spark Streaming, make high velocity streaming analytics at scale easier than ever, similar to how Cassandra pioneered high-velocity data management at scale.

Hadoop is the right choice for batch analytics. Until recently, nobody really knew what the right solution is for real-time processing. We believe that Spark and Cassandra are the clear answer.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
726
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Rant about strong consistency
  • 1990: John Romkey created the first Internet ‘device’, a toaster that could be turned on and off over the Internet.
    1991: The first web page was created by Tim Berners-Lee
  • Cassandra powers devices in all of these spaces today
  • We can scale to whatever you need.
  • Spark streaming is the real time data enrichment layer. Data augmentation, anomaly detection, denormalization, etc.
  • Even without augmentation, individual device timelines are stored in wide partitions, hence fast to use Spark to
  • Application logic/servers co-located with each Spark/Cassandra DC. Output of any Spark job can be a message to the application servers to take action.
  • So what does the deployment look like?
  • Talk about message queues, and methods of getting data to spark
  • Talk about the Cassandra API, triggers, and the path to write back to Cassandra
  • Write locally, interact globally. Replication factor per keyspace per dc.
  • Look for an announcement at the Spark Summit on Monday the 30th. Much more to come at the Cassandra Summit in September.
  • Cassandra Day NY 2014: Using Spark Streaming for High Velocity Analytics on Cassandra

    1. 1. A Tale of Two Technologies A Story of the IoT “revolution”
    2. 2. It was the age of connectedness
    3. 3. It was the age of disconnectedness Failure tolerance is not optional
    4. 4. It was the age of wisdom
    5. 5. It was the age of foolishness
    6. 6. It was the epoch of complexity Did you just tell me to go #@$#@$ myself?
    7. 7. It was the epoch of simplicity We brought as much simplicity as we could to the menagerie known as Hadoop, but it’s still slow.
    8. 8. What is the Internet of Things? “The Internet of Things (IoT) refers to uniquely identifiable objects and their virtual representations in an Internet-like structure.” -Wikipedia
    9. 9. No really what is IoT? ● It’s literally the act of connecting “things” to the Internet ● It predates the World Wide Web ● It shouldn’t be surprising to anybody
    10. 10. So IoT is old news? Most definitely
    11. 11. So IoT is just hype? Home security and automation Fitness trackers Your (driverless?) car Medical devices Sensors Industrial equipment NO!
    12. 12. What is Cassandra? ● A massively scalable distributed database ● Chooses availability over strong consistency (yes, that really is a fundamental tradeoff) ● With its wide partitions it is able to take advantage of data locality even at “web scale” Chocolate!
    13. 13. What is Spark? ● DAG is a logical superset of M/R ● Adopts much of the Hadoop ecosystem, without being bound by it ● Intelligent use of caching (RDDs) for massive performance gains ● Incorporate Streaming to make ingestion- time processing a first class citizen Peanut butter!
    14. 14. What does IoT need from big data? ● Log time-series events -- at scale ● Gather meaning from that data -- at scale ● Report on that data -- at scale ● Take action on that data -- at scale
    15. 15. Logging events at scale
    16. 16. Gathering meaning at scale
    17. 17. Reporting at scale ● Canned reports ● Ad-hoc querying and reporting ● Drill down / exploratory ● Alerting ● Aggregation ● Clustering (K-means, et al) ● Generalized machine learning
    18. 18. Take action at scale ● Stateless application servers ● Horizontally scaled and co-located with Cassandra and Spark in each DC ● Any platform with a CQL driver
    19. 19. The architecture...
    20. 20. Spark Streaming to Cassandra
    21. 21. Cassandra to Spark Streaming
    22. 22. Multi-DC DC1 DC2 Write anywhere.
    23. 23. Things that go together
    24. 24. Things that go together
    25. 25. A Tale of Two Summits

    ×