Follow
Join Us!
< meta description=“Aerospike is a
Rule CaptainWord {
strings:
$header = {D0 CF 11 E0 A1 B1 1A E1}
$author...
What is Storm?
➤  Storm is an Apache open-source project
(Hadoop of real time)
§  Created by Twitter
§  No “Cloudera”-li...
Why Storm?
➤  Same management framework as
Hadoop
➤  Many Spouts and Bolts available
https://github.com/nathanmarz/storm-
...
But ---
➤  Storm has problems
➤  0mq -> Netty TCP transition
§  So much easier!
➤  Trident not good enough
§  But closer...
Streaming architecture
© 2014 Aerospike. All rights reserved. | Pg. 5
Data
Warehouse,
Hadoop
Cluster
Real-time
Interaction...
Integration with Hadoop
➤  Example Bolt connectors for
§  Hdfs
§  Hadoop
➤  Example pattern
§  Bolt writes to HDFS
§  ...
Examples
➤  TrendingWords
§  Slice time into “slots” (1 minute / 5 minute)
for each word
§  On seeing a word, increment ...
Examples
➤  TrendingWords – Database analysis – single words
§  Twitter at … 5k tps?
§  20 indexable words per tweet
§ ...
Examples
➤  Recommendations
§  Multiple recommendation systems
§  Multi-arm bandit
§  https://github.com/tdunning/
stor...
Follow
Join Us!
< meta description=“Aerospike is a
Rule CaptainWord {
strings:
$header = {D0 CF 11 E0 A1 B1 1A E1}
$author...
Development installs (easy)
➤  Everything on one server
( no ZeroMQ )
➤  Zookeeper, Nimbus, Supervisors (slaves)
all on on...
Real-world install
( distributed or go home )
➤  Zookeeper
§  A small (1 to 5) set of nodes to hold configuration. It’s l...
Follow
Join Us!
< meta description=“Aerospike is a
Rule CaptainWord {
strings:
$header = {D0 CF 11 E0 A1 B1 1A E1}
$author...
Real-world install
➤  Ubuntu 12.04 LTS
➤  In Amazon, there’s a variety of scripts and
helpers
§  ( now you can build your...
Prerequisites
➤  Packages
§  sudo apt-get install git-core libtool
autoconf g++ build-essential uuid-dev
maven
➤  Java
§...
Prerequisites
➤  0mq – only required in 0.8
§  Native, high performance messaging
§  Storm 0.9.0 avoids 0mq, uses HTTP (...
Zookeeper server
➤  Packages
§  sudo apt-get install zookeeperd
‘d’ is for servers, without is for clients
§  export PAT...
Zookeeper server
➤  Validate
§  zkCli.sh -à ( whines a bit, ends up [CONNECTED] )
§  ls /
create /foo mydata
get /foo
s...
Follow
Join Us!
< meta description=“Aerospike is a
Rule CaptainWord {
strings:
$header = {D0 CF 11 E0 A1 B1 1A E1}
$author...
Storm
➤  No packages in Ubuntu
§  Download http://storm-project.net/downloads
§  Into /opt/storm, unzip, add to path
§ ...
Storm
➤  Try running components stand alone
sudo su - storm
cd /opt/storm/storm-0.8.2/
bin/storm nimbus
bin/storm supervis...
Storm
➤  storm-master.conf
[program:storm-nimbus]
command=/opt/storm/storm-0.8.2/bin/storm nimbus
user=storm
autostart=tru...
Follow
Join Us!
< meta description=“Aerospike is a
Rule CaptainWord {
strings:
$header = {D0 CF 11 E0 A1 B1 1A E1}
$author...
Run some storm !
➤  Copy storm.yaml to a local user’s directory
mkdir ~/.storm
sudo cp /opt/storm/storm-0.8.2/conf/storm.y...
Run some storm !
➤  Submit the topology
storm jar target/storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.Exclama...
Follow
Join Us!
< meta description=“Aerospike is a
Rule CaptainWord {
strings:
$header = {D0 CF 11 E0 A1 B1 1A E1}
$author...
Creating topologies
➤  Say it with code
public static void main(String[] args) throws Exception {
TopologyBuilder builder ...
Spouts
➤  RandomSentenceSpout
  public void nextTuple() {
Utils.sleep(100);
String[] sentences = new String[]{
"the cow ju...
Spouts
➤  Easy interaction with queues
§  https://github.com/nathanmarz/storm-contrib
➤  storm-kafka
§  Popular persiste...
Bolts
➤  Simple exclamation bolt
public static class ExclamationBolt extends BaseRichBolt {
OutputCollector _collector;
@O...
Persisting Bolts
➤  Persistence is good ! Scale the data layer
§  Shared required for counters like TrendingWords
§  Sta...
Aerospike Bolts
➤  Bolts available on github
§  https://github.com/aerospike/storm-aerospike
➤  EnrichBolt
§  Add fields...
Consider Aerospike
➤  If you need the speed of Storm, you need the speed
of Aerospike
§  That’s why we talk about Storm!
...
Interesting Bolts
➤  MovingAverageWithSpikeDetection
§  https://github.com/stormprocessor/storm-examples
➤  TrendingTopic...
Follow
Join Us!
< meta description=“Aerospike is a
Rule CaptainWord {
strings:
$header = {D0 CF 11 E0 A1 B1 1A E1}
$author...
Why not other databases?
➤  Low latency - database requests in bolts
➤  Proven in production with large-scale deployments
...
Aerospike: the gold standard for high throughput,
low latency, high reliability transactions
Performance
• Over ten trilli...
2 Trillion Transactions per month
➤  100,000 attributes each for 160 Million users
§  Data used by recommendation engines...
➤ Hard to
Maintain
➤ Performance
Better than the Competition
➤ Latency
➤ Number of
Servers
➤ Stability
➤ Cost of
RAM
➤ Cos...
OHIO
1)  No Hotspots
– DHT with RIPEMD160
simplifies data
partitioning
2)  Smart Client – 1 hop to
data, no load balancers...
DISTRIBUTED QUERIES
1.  “Scatter” requests to all nodes
2.  Indexes in DRAM for fast map of secondary à primary keys
3.  ...
Follow
Join Us!
< meta description=“Aerospike is a
Rule CaptainWord {
strings:
$header = {D0 CF 11 E0 A1 B1 1A E1}
$author...
Combining Real-time and Batch Analytics with NoSQL, Storm and Hadoop - NoSQL Matters, Cologne, 2014
Upcoming SlideShare
Loading in...5
×

Combining Real-time and Batch Analytics with NoSQL, Storm and Hadoop - NoSQL Matters, Cologne, 2014

3,329
-1

Published on

From financial services, to digital advertising, omni-channel marketing and retail, companies are pushing to grow revenue by personalizing the customer experience in real-time based on knowing what they care about, where they are, and what they are doing now. For growing numbers of these businesses, this means developing applications that combine the historical analysis provided by Hadoop with real-time analysis through Storm and within NoSQL databases, themselves. This session will examine the design considerations and development approaches for successfully delivering interactive applications that incorporate real-time and batch analysis using a combination of Hadoop, Storm and NoSQL. Key topics will include:· A review of the respective roles that Hadoop, Storm and NoSQL databases play.· Considerations in choosing which technology to use in areas where their capabilities overlap.· An overview of a typical solution architecture.· Strategies for addressing the diverse data types required for providing a complete view of the customers.· Approaches to managing large data types to ensure reliable real-time responses.Throughout the discussion, concepts will be illustrated by use cases of businesses that have implemented real-time applications using Hadoop, Storm and NoSQL, which are in production today.

This presentation was given at the 2014 NoSQL Matters conference in Cologne, Germany.

Published in: Technology
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,329
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
75
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

Combining Real-time and Batch Analytics with NoSQL, Storm and Hadoop - NoSQL Matters, Cologne, 2014

  1. 1. Follow Join Us! < meta description=“Aerospike is a Rule CaptainWord { strings: $header = {D0 CF 11 E0 A1 B1 1A E1} $author = {00 00 00 63 61 70 74 61 69 6E 00} condition: $header at 0 and $author Content=“malware,Exec Code, Overflow, ExecCode Bypass” Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027” Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili” tag <head> <meta name=“”> <meta desc=“”> </head> © 2014 Aerospike. All rights reserved. Pg. 1 “keywords” <TITLE> ✓ I’M ATTENDING @Aerospikedb Real-time analytics with Storm, NoSQL, and Hadoop Brian Bulkowski Founder and CTO Aerospike
  2. 2. What is Storm? ➤  Storm is an Apache open-source project (Hadoop of real time) §  Created by Twitter §  No “Cloudera”-like vendor §  Return of “Message Oriented Middleware” ➤  Overview §  “Spouts” connect with data sources (example: web log tail ingest) §  “Bolts” analysis and data manipulation of any sort §  “Nimbus” control entity re-balances and re-configures without downtime, add servers seamlessly §  “Topology” Ordering of bolts and spouts §  “Tuple” A Storm message §  “Trident” Higher level abstraction supporting real-time joins, reliability © 2014 Aerospike. All rights reserved. Confidential | Pg. 2
  3. 3. Why Storm? ➤  Same management framework as Hadoop ➤  Many Spouts and Bolts available https://github.com/nathanmarz/storm- contrib ➤  Simple database integration ➤  High performance & reliability (1MM/S/S ++) ➤  Great documentation ➤  Multiple languages ➤  Excellent framework for event oriented implementation © 2014 Aerospike. All rights reserved. Confidential | Pg. 3
  4. 4. But --- ➤  Storm has problems ➤  0mq -> Netty TCP transition §  So much easier! ➤  Trident not good enough §  But closer than most things ➤  Auto-rebalance framework ➤  Good news: Twitter doubling down © 2014 Aerospike. All rights reserved. Confidential | Pg. 4
  5. 5. Streaming architecture © 2014 Aerospike. All rights reserved. | Pg. 5 Data Warehouse, Hadoop Cluster Real-time Interactions Server Batch Analytics •  User segmentation •  Location patterns •  Similar audience Real-time Interactions •  Frequency caps •  Recent ads served •  Recent search terms User Data Streaming (Storm) Hadoop
  6. 6. Integration with Hadoop ➤  Example Bolt connectors for §  Hdfs §  Hadoop ➤  Example pattern §  Bolt writes to HDFS §  Bolt reads from HDFS ➤  Problem: §  Storm is AT LEAST ONCE ➤  Solution: §  write from your app to Hadoop §  Emit from Hadoop to an edge database §  Or read from Hdfs directly © 2014 Aerospike. All rights reserved. Confidential | Pg. 6
  7. 7. Examples ➤  TrendingWords §  Slice time into “slots” (1 minute / 5 minute) for each word §  On seeing a word, increment bucket © 2014 Aerospike. All rights reserved. Confidential | Pg. 7 http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/ §  Use Storm Ticks (0.8) for emitting top events §  Determine most popular words in a category §  Clean up old data on Ticks or on reads Medium easy ! ( Google ‘trending topics storm’ )
  8. 8. Examples ➤  TrendingWords – Database analysis – single words §  Twitter at … 5k tps? §  20 indexable words per tweet §  10k writes per second §  50,000 words in English §  A reasonable RAM problem Requires queries & kvs ➤  Tuples §  Perhaps 30 tuples per tweet §  600K tps writes §  10M tuples ? Flash might be better Problem is much harder now! © 2014 Aerospike. All rights reserved. Confidential | Pg. 8 http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/
  9. 9. Examples ➤  Recommendations §  Multiple recommendation systems §  Multi-arm bandit §  https://github.com/tdunning/ storm-counts/wiki/Bayesian-Bandit ➤  Simple fraud counts §  Store recent requests for payment §  Store recent users §  Calculate fraud scores, drop events if past threshold © 2014 Aerospike. All rights reserved. Confidential | Pg. 9
  10. 10. Follow Join Us! < meta description=“Aerospike is a Rule CaptainWord { strings: $header = {D0 CF 11 E0 A1 B1 1A E1} $author = {00 00 00 63 61 70 74 61 69 6E 00} condition: $header at 0 and $author Content=“malware,Exec Code, Overflow, ExecCode Bypass” Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027” Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili” #HashTags tag <head> <meta name=“”> <meta desc=“”> </head> © 2014 Aerospike. All rights reserved. Pg. 10 “keywords” <TITLE> ✓ I’M ATTENDING @Aerospikedb Storm overview
  11. 11. Development installs (easy) ➤  Everything on one server ( no ZeroMQ ) ➤  Zookeeper, Nimbus, Supervisors (slaves) all on one machine ➤  Write your Spouts and Bolts, but what next? ➤  Resources: ➤  https://github.com/nathanmarz/storm/wiki/ Setting-up-development-environment © 2014 Aerospike. All rights reserved. Confidential | Pg. 11
  12. 12. Real-world install ( distributed or go home ) ➤  Zookeeper §  A small (1 to 5) set of nodes to hold configuration. It’s like DNS. Hopefully you have this running for Hadoop. ➤  Nimbus §  Single server where “Topologies” are managed §  Communicates via Zookeeper, “storm” namespace ➤  Supervisor ( slave ) §  Needs the addresses of zookeeper nodes §  Needs the Nimbus address §  Then simply waits for work ➤  Best guide – this is really good ! http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/ © 2012 Aerospike. All rights reserved. Confidential | Pg. 12
  13. 13. Follow Join Us! < meta description=“Aerospike is a Rule CaptainWord { strings: $header = {D0 CF 11 E0 A1 B1 1A E1} $author = {00 00 00 63 61 70 74 61 69 6E 00} condition: $header at 0 and $author Content=“malware,Exec Code, Overflow, ExecCode Bypass” Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027” Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili” #HashTags tag <head> <meta name=“”> <meta desc=“”> </head> © 2014 Aerospike. All rights reserved. Pg. 13 “keywords” <TITLE> ✓ I’M ATTENDING @Aerospikedb Let’s Begin
  14. 14. Real-world install ➤  Ubuntu 12.04 LTS ➤  In Amazon, there’s a variety of scripts and helpers §  ( now you can build your own ) §  You can scale back to 1 machine (zoo & nimbus) ➤  In a laptop VM environment, use NAT §  Share your /etc/hosts file © 2014 Aerospike. All rights reserved. Confidential | Pg. 14
  15. 15. Prerequisites ➤  Packages §  sudo apt-get install git-core libtool autoconf g++ build-essential uuid-dev maven ➤  Java §  OpenJDK 6 “is unfortunate”, OracleJDK 6 ok §  7 can be OpenJDK or Oracle §  Can’t mix and match 5, 6, 7 §  Will be building & submitting “fat jars” §  JAVA_HOME everywhere © 2014 Aerospike. All rights reserved. Confidential | Pg. 15
  16. 16. Prerequisites ➤  0mq – only required in 0.8 §  Native, high performance messaging §  Storm 0.9.0 avoids 0mq, uses HTTP (!) §  2.1.x (2.1.6 -> 2.1.11) ONLY §  sudo apt-get install libzmq-dev ➤  Storm’s Java 0mq client (ugly http://stackoverflow.com/questions/12115160/compiling-jzmq-on-ubuntu git clone https://github.com/nathanmarz/jzmq.git cd jzmq ./autogen.sh ./configure # overcoming flaw in marz’s repo cd src touch classdist_noinst.stamp CLASSPATH=.:./.:$CLASSPATH javac -d . org/zeromq/ZMQ.java org/ zeromq/ZMQException.java org/zeromq/ZMQQueue.java org/zeromq/ ZMQForwarder.java org/zeromq/ZMQStreamer.java cd .. make sudo make install © 2014 Aerospike. All rights reserved. Confidential | Pg. 16
  17. 17. Zookeeper server ➤  Packages §  sudo apt-get install zookeeperd ‘d’ is for servers, without is for clients §  export PATH=$PATH:/user/share/zookeeper/bin §  sudo update-rc.d zookeeper defaults ➤  Config §  All zookeepers know about all other zookeepers §  /etc/zookeeper/conf/myid -> 1 through 255 §  /etc/zookeeper/conf/zoo.cfg list IP or DNS for all other zookeepers §  Service is up only if a majority is up © 2014 Aerospike. All rights reserved. Confidential | Pg. 17
  18. 18. Zookeeper server ➤  Validate §  zkCli.sh -à ( whines a bit, ends up [CONNECTED] ) §  ls / create /foo mydata get /foo set /foo otherdata get /foo ➤  Notes Data directory is /var/lib/zookeeper Config file is /etc/zookeeper/conf/zoo.cfg “client port” is 2181 © 2014 Aerospike. All rights reserved. Confidential | Pg. 18
  19. 19. Follow Join Us! < meta description=“Aerospike is a Rule CaptainWord { strings: $header = {D0 CF 11 E0 A1 B1 1A E1} $author = {00 00 00 63 61 70 74 61 69 6E 00} condition: $header at 0 and $author Content=“malware,Exec Code, Overflow, ExecCode Bypass” Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027” Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili” #HashTags tag <head> <meta name=“”> <meta desc=“”> </head> © 2014 Aerospike. All rights reserved. Pg. 19 “keywords” <TITLE> ✓ I’M ATTENDING @Aerospikedb Let’s Install Storm
  20. 20. Storm ➤  No packages in Ubuntu §  Download http://storm-project.net/downloads §  Into /opt/storm, unzip, add to path §  export PATH=$PATH:/opt/storm/storm-0.8.2/bin ➤  Create user “storm” ; remember Path + Java sudo chown –R storm:storm /opt/storm sudo chown –R storm:storm /var/data/storm ➤  Update config file # update the storm configuration file - create a data directory, point storm at it sudo mkdir -p /var/data/storm sudo chmod a+w /var/data/storm sudo vi /opt/storm/storm-0.8.2/conf/storm.yaml storm.local.dir: "/var/data/storm" : storm.zookeeper.servers: - “192.168.62.149” ➤  YAML is touchy. Make sure to have a space after the ‘:’ © 2014 Aerospike. All rights reserved. Confidential | Pg. 20
  21. 21. Storm ➤  Try running components stand alone sudo su - storm cd /opt/storm/storm-0.8.2/ bin/storm nimbus bin/storm supervisor bin/storm ui ➤  Reliability through low state, quick restart – supervisor ! sudo apt-get install supervisord cp storm.conf /etc/supervisor/conf.d ➤  Extra hint sudo unlink /var/run/supervisor.sock ➤  Storm.conf [program:storm-supervisor] command=/opt/storm/storm-0.8.2/bin/storm supervisor user=storm autostart=true autorestart=true startsecs=10 startretries=999 log_stdout=true log_stderr=true logfile=/var/log/storm/supervisor.out logfile_maxbytes=20MB logfile_backups=10 © 2014 Aerospike. All rights reserved. Confidential | Pg. 21
  22. 22. Storm ➤  storm-master.conf [program:storm-nimbus] command=/opt/storm/storm-0.8.2/bin/storm nimbus user=storm autostart=true autorestart=true startsecs=10 startretries=999 log_stdout=true log_stderr=true logfile=/var/log/storm/nimbus.out logfile_maxbytes=20MB logfile_backups=10 [program:storm-ui] command=/opt/storm/storm-0.8.2/bin/storm ui user=storm autostart=true autorestart=true startsecs=10 startretries=999 log_stdout=true log_stderr=true logfile=/var/log/storm/ui.out logfile_maxbytes=20MB logfile_backups=10 © 2014 Aerospike. All rights reserved. Confidential | Pg. 22
  23. 23. Follow Join Us! < meta description=“Aerospike is a Rule CaptainWord { strings: $header = {D0 CF 11 E0 A1 B1 1A E1} $author = {00 00 00 63 61 70 74 61 69 6E 00} condition: $header at 0 and $author Content=“malware,Exec Code, Overflow, ExecCode Bypass” Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027” Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili” #HashTags tag <head> <meta name=“”> <meta desc=“”> </head> © 2014 Aerospike. All rights reserved. Pg. 23 “keywords” <TITLE> ✓ I’M ATTENDING @Aerospikedb Let’s Run Storm
  24. 24. Run some storm ! ➤  Copy storm.yaml to a local user’s directory mkdir ~/.storm sudo cp /opt/storm/storm-0.8.2/conf/storm.yaml ~/.storm/ storm.yaml sudo chown brian:brian ~/.storm/storm.yaml ➤  Install ‘leiningen’ (clojure tools) Package version too old http://leiningen.org/ download ‘lein’ into /usr/local/bin – chmod +x /usr/local/bin/lein ➤  storm-starter git clone http://github.com/nathanmarz/storm-starter.git cd storm-starter lein deps ; lein compile ; lein uberjar © 2014 Aerospike. All rights reserved. Confidential | Pg. 24
  25. 25. Run some storm ! ➤  Submit the topology storm jar target/storm-starter-0.0.1-SNAPSHOT-standalone.jar storm.starter.ExclamationTopology exclamation-topology ➤  Watch the output tail –f /opt/storm/storm-0.8.2/logs/worker-xxxx.log ➤  View the job $ storm list Topology_name Status Num_tasks Num_workers Uptime_secs ------------------------------------------------------------ exclamation-topology ACTIVE 16 3 798 © 2014 Aerospike. All rights reserved. Confidential | Pg. 25
  26. 26. Follow Join Us! < meta description=“Aerospike is a Rule CaptainWord { strings: $header = {D0 CF 11 E0 A1 B1 1A E1} $author = {00 00 00 63 61 70 74 61 69 6E 00} condition: $header at 0 and $author Content=“malware,Exec Code, Overflow, ExecCode Bypass” Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027” Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili” #HashTags tag <head> <meta name=“”> <meta desc=“”> </head> © 2014 Aerospike. All rights reserved. Pg. 26 “keywords” <TITLE> ✓ I’M ATTENDING @Aerospikedb Coding to Storm
  27. 27. Creating topologies ➤  Say it with code public static void main(String[] args) throws Exception { TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("word", new TestWordSpout(), 10); builder.setBolt("exclaim1", new ExclamationBolt(), 3).shuffleGrouping("word"); builder.setBolt("exclaim2", new ExclamationBolt(), 2).shuffleGrouping("exclaim1"); Config conf = new Config(); conf.setDebug(true); if (args != null && args.length > 0) { conf.setNumWorkers(3); StormSubmitter.submitTopology(args[0], conf, builder.createTopology()); } else { LocalCluster cluster = new LocalCluster(); cluster.submitTopology("test", conf, builder.createTopology()); Utils.sleep(10000); cluster.killTopology("test"); cluster.shutdown(); } } From the ExclaimationTopology © 2014 Aerospike. All rights reserved. Confidential | Pg. 27
  28. 28. Spouts ➤  RandomSentenceSpout   public void nextTuple() { Utils.sleep(100); String[] sentences = new String[]{ "the cow jumped over the moon", "an apple a day keeps the doctor away", "four score and seven years ago", "snow white and the seven dwarfs", "i am at two with nature" }; String sentence = sentences[_rand.nextInt(sentences.length)]; _collector.emit(new Values(sentence)); } © 2014 Aerospike. All rights reserved. Confidential | Pg. 28
  29. 29. Spouts ➤  Easy interaction with queues §  https://github.com/nathanmarz/storm-contrib ➤  storm-kafka §  Popular persistent, topic-based queue ➤  storm-jms §  Java message service ➤  storm-sqs §  EC2’s queue service ➤  more! © 2014 Aerospike. All rights reserved. Confidential | Pg. 29
  30. 30. Bolts ➤  Simple exclamation bolt public static class ExclamationBolt extends BaseRichBolt { OutputCollector _collector; @Override public void prepare(Map conf, TopologyContext context, OutputCollector collector) { _collector = collector; } @Override public void execute(Tuple tuple) { _collector.emit(tuple, new Values(tuple.getString(0) + "!!!")); _collector.ack(tuple); } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); } } © 2014 Aerospike. All rights reserved. Confidential | Pg. 30
  31. 31. Persisting Bolts ➤  Persistence is good ! Scale the data layer §  Shared required for counters like TrendingWords §  Stateless Bolts are better ➤  Easy persistence through examples §  https://github.com/nathanmarz/storm-contrib ➤  storm-rdbms §  Stores to a database table ➤  storm-cassandra §  Stores to a cassandra cluster ➤  storm-redis ➤  storm-memcache © 2014 Aerospike. All rights reserved. Confidential | Pg. 31
  32. 32. Aerospike Bolts ➤  Bolts available on github §  https://github.com/aerospike/storm-aerospike ➤  EnrichBolt §  Add fields from column after looking up a key ➤  PersistBolt §  Store fields based on a key © 2014 Aerospike. All rights reserved. Confidential | Pg. 32
  33. 33. Consider Aerospike ➤  If you need the speed of Storm, you need the speed of Aerospike §  That’s why we talk about Storm! ➤  Free version at http://aerospike.com/ ➤  Internap – free high performance SSD servers for trial ➤  Benefits §  In memory with FLASH §  Clustered for high performance §  HA state matches Storm’s stateless model © 2014 Aerospike. All rights reserved. Confidential | Pg. 33
  34. 34. Interesting Bolts ➤  MovingAverageWithSpikeDetection §  https://github.com/stormprocessor/storm-examples ➤  TrendingTopics §  http://www.michael-noll.com/blog/2013/01/18/ implementing-real-time-trending-topics-in-storm/ ➤  Baysian Bandit §  https://github.com/tdunning/storm-counts © 2014 Aerospike. All rights reserved. Confidential | Pg. 34
  35. 35. Follow Join Us! < meta description=“Aerospike is a Rule CaptainWord { strings: $header = {D0 CF 11 E0 A1 B1 1A E1} $author = {00 00 00 63 61 70 74 61 69 6E 00} condition: $header at 0 and $author Content=“malware,Exec Code, Overflow, ExecCode Bypass” Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027” Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili” #HashTags tag <head> <meta name=“”> <meta desc=“”> </head> © 2014 Aerospike. All rights reserved. Pg. 35 “keywords” <TITLE> ✓ I’M ATTENDING @Aerospikedb Why Aerospike ?
  36. 36. Why not other databases? ➤  Low latency - database requests in bolts ➤  Proven in production with large-scale deployments ➤  Flash optimized §  Do you need more than 30G ? ➤  Read / write optimized ➤  Faster & more reliable than than Kafka (Cassandra based) ➤  Faster than Mongo ➤  More scale than Redis © 2012 Aerospike. All rights reserved. Confidential | Pg. 36
  37. 37. Aerospike: the gold standard for high throughput, low latency, high reliability transactions Performance • Over ten trillion transactions per month • 99% of transactions faster than 2 ms • 150K TPS per server Scalability • Billions of Internet users • Clustered Software • Automatic Data Rebalancing Reliability • 50 customers; zero service down- time • Immediate Consistency • Rapid Failover; Data Center Replication Price/Performance • Makes impossible projects affordable • Flash-optimized • 1/10 the servers required
  38. 38. 2 Trillion Transactions per month ➤  100,000 attributes each for 160 Million users §  Data used by recommendation engines, e-commerce, RTB, mobile, video and display ad platforms ➤  16 TB of data on 400 Million users §  60 Billion Transactions per month §  4 data centers for geographic proximity & high availability §  “Scale. Real-time performance. Real-time replication at 4 datacenters. Aerospike delivered.”  - Elad Efraim, CTO © 2013 Aerospike. All rights reserved. Proprietary Pg. 39
  39. 39. ➤ Hard to Maintain ➤ Performance Better than the Competition ➤ Latency ➤ Number of Servers ➤ Stability ➤ Cost of RAM ➤ Cost of RAM ➤ Scalability
  40. 40. OHIO 1)  No Hotspots – DHT with RIPEMD160 simplifies data partitioning 2)  Smart Client – 1 hop to data, no load balancers 3)  Shared Nothing Architecture, every node identical 7) XDR – asynch replication across data centers ensures Zero Downtime 4)  Single row ACID – synch replication in cluster 5)  Smart Cluster, Zero Touch – auto-failover, rebalancing, rolling upgrades.. 6)  Transactions and long running tasks prioritized real-time Simpler Scaling: Fewer Servers, ACID, Zero Touch
  41. 41. DISTRIBUTED QUERIES 1.  “Scatter” requests to all nodes 2.  Indexes in DRAM for fast map of secondary à primary keys 3.  Indexes co-located with data to guarantee ACID, manage migrations 4.  Records read in parallel from all SSDs using lock free concurrency control 5.  Aggregate results on each node 6.  “Gather” results from all nodes on client STREAM AGGREGATIONS 1.  Push Code/ Security Policies/ Rules to Data with UDFs 2.  Pipe Query results through UDFs to Filter, Transform, Aggregate.. Map, Reduce REAL-TIME ANALYTICS on OPERATIONAL DATA (No ETL) ➤  In Database, within the same Cluster ➤  On the same Data, on XDR Replicated Clusters Real-time Analytics on Operational Data
  42. 42. Follow Join Us! < meta description=“Aerospike is a Rule CaptainWord { strings: $header = {D0 CF 11 E0 A1 B1 1A E1} $author = {00 00 00 63 61 70 74 61 69 6E 00} condition: $header at 0 and $author Content=“malware,Exec Code, Overflow, ExecCode Bypass” Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027” Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili” #HashTags tag <head> <meta name=“”> <meta desc=“”> </head> © 2014 Aerospike. All rights reserved. Pg. 43 “keywords” <TITLE> ✓ I’M ATTENDING @Aerospikedb Questions ?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×