Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - James Stanier

2,807 views

Published on

JAX London Presentation 2014

  • Hey guys! Who wants to chat with me? More photos with me here 👉 http://www.bit.ly/katekoxx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - James Stanier

  1. 1. Dr. James Stanier | Head of Analytics | Brandwatch.com | jamess@brandwatch.com
  2. 2. Coming Up/ • Me, Brandwatch and new problems • Apache Kafka • Processing data in Java • Distributing work with Zookeeper • Managing state © 2014 Brandwatch | www.brandwatch.com 2
  3. 3. Who? © 2014 Brandwatch | www.brandwatch.com 3
  4. 4. Dr. James Stanier Head of Analytics, Brandwatch @jstanier | jamess@brandwatch.com © 2014 Brandwatch | www.brandwatch.com 4
  5. 5. Brandwatch © 2014 Brandwatch | www.brandwatch.com 5
  6. 6. Where we are/ • Brighton • New York • San Francisco • Berlin • Stuttgart © 2013 Brandwatch | www.brandwatch.com 6
  7. 7. © 2014 Brandwatch | www.brandwatch.com 7
  8. 8. What Brandwatch does/ Crawl Store and Index Analyse 3 Present 4 • Crawl 70M+ sites including key social networks • 27 languages • Powerful search operators • 20Bn + indexed URLs • Years of historical data • Automated topic & sentiment analysis in all 27 languages • Automate common tasks including alerts • Advanced analytics modules • Automatic categorisation with rules • Custom dashboards • Reporting & alerts © 2014 Brandwatch | www.brandwatch.com 8
  9. 9. Brandwatch Analytics © 2014 Brandwatch | www.brandwatch.com 9
  10. 10. 10 Data/ Presentation
  11. 11. Data/ Aggregation © 2014 Brandwatch | www.brandwatch.com 11
  12. 12. Data/ Classification © 2014 Brandwatch | www.brandwatch.com 12
  13. 13. Data/ Not just top level metrics © 2014 Brandwatch | www.brandwatch.com 13
  14. 14. Development/ What do we use? © 2014 Brandwatch | www.brandwatch.com 14
  15. 15. Data/ The numbers • 50+ Java Web Crawlers • 10+ Historical crawlers for new queries • Twitter via GNIP (now Twitter) • 70M+ query matches per day © 2014 Brandwatch | www.brandwatch.com 15
  16. 16. The speed of social © 2014 Brandwatch | www.brandwatch.com 16
  17. 17. © 2014 Brandwatch | www.brandwatch.com 17
  18. 18. © 2014 Brandwatch | www.brandwatch.com 18
  19. 19. © 2014 Brandwatch | www.brandwatch.com 19
  20. 20. A new challenge © 2014 Brandwatch | www.brandwatch.com 20
  21. 21. 21 The challenge/ The signal from the noise
  22. 22. The challenge/ at scale • 100K+ user queries • 70M+ mentions per day • Polling the database for mentions would take 8hrs for one pass © 2014 Brandwatch | www.brandwatch.com 22
  23. 23. The Problem/ How we handled it… Crawler 1 Crawler 2 Crawler n-1 Crawler N Kafka Cluster Signals Signals Signals Processing cluster Signals handler JVM DB Mentions Mentions © 2014 Brandwatch | www.brandwatch.com 23
  24. 24. Kafka © 2014 Brandwatch | www.brandwatch.com 24
  25. 25. Step 1/ Kafka Crawler 1 Crawler 2 Crawler n-1 Crawler N Kafka Cluster Mentions © 2014 Brandwatch | www.brandwatch.com 25
  26. 26. Kafka/ What is it? • Apache Kafka is a publish-subscribe messaging system rethought as a distributed commit log • Apache top level project November 2013 • Started at LinkedIn © 2014 Brandwatch | www.brandwatch.com 26
  27. 27. Kafka/ is… • Fast: hundreds of MBs read/write per second from thousands of clients • Scalable: clustered, partitioned over many machines, expanded without downtime • Durable: messages persisted to disk and replicated in cluster © 2014 Brandwatch | www.brandwatch.com 27
  28. 28. Kafka/ Terminology • Kafka maintains feeds of messages called topics • Programs that publish messages are called producers • Programs that subscribe to messages are called consumers • Kafka is a cluster of servers called brokers © 2014 Brandwatch | www.brandwatch.com 28
  29. 29. Kafka/ How it’s used... Producer Producer Producer Kafka Cluster Consumer Consumer Consumer © 2014 Brandwatch | www.brandwatch.com 29
  30. 30. Kafka/ Written to disk? http://q.acm.org/detail.cfm?id=1563874 © 2014 Brandwatch | www.brandwatch.com 30
  31. 31. Kafka/ Bending, not breaking http://engineering.gnip.com/tag/kafka/ © 2014 Brandwatch | www.brandwatch.com 31
  32. 32. Kafka/ The anatomy of a topic 0 1 2 3 4 5 6 0 1 2 3 4 0 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Old New Writes © 2014 Brandwatch | www.brandwatch.com 32
  33. 33. Kafka/ Warning: ordering Kafka guarantees a total ordering per partition, not per whole topic © 2014 Brandwatch | www.brandwatch.com 33
  34. 34. Kafka/ Try it out! > tar -xzf kafka_2.9.2-0.8.1.tgz! > cd kafka_2.9.2-0.8.1! > bin/zookeeper-server-start.sh config/ zookeeper.properties! > bin/kafka-server-start.sh config/ server.properties! ! © 2014 Brandwatch | www.brandwatch.com 34
  35. 35. Kafka/ Try it out! > bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test! > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test! Hello JAX!! > bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning! Hello JAX!! © 2014 Brandwatch | www.brandwatch.com 35
  36. 36. Kafka/ With Java <dependency>! !<groupId>org.apache.kafka</groupId>! !<artifactId>kafka_2.10</artifactId>! !<version>0.8.1</version>! </dependency>! © 2014 Brandwatch | www.brandwatch.com 36
  37. 37. Kafka/ With Java Properties props = new Properties();! props.put("metadata.broker.list", "broker1:9092,broker2:9092");! props.put("serializer.class", "kafka.serializer.StringEncoder");! props.put("partitioner.class", "example.producer.SimplePartitioner");! props.put("request.required.acks", "1");! ProducerConfig config = new ProducerConfig(props);! Producer<String, String> producer = new Producer<String, String>(config);! © 2014 Brandwatch | www.brandwatch.com 37
  38. 38. Kafka/ Partitioning public class SimplePartitioner implements Partitioner<String> {! public SimplePartitioner (VerifiableProperties props) {! ! public int partition(String key, int numberOfPartitions) {! return md5hash(key) % numberOfPartitions;! }! }! © 2014 Brandwatch | www.brandwatch.com 38
  39. 39. Kafka/ Sending from the crawlers String message = toJson(...);! KeyedMessage<String, String> message = new! !KeyedMessage<String, String>("query.mentions", !queryId, message);! producer.send(message);! © 2014 Brandwatch | www.brandwatch.com 39
  40. 40. Step 1/ Done Crawler 1 Crawler 2 Crawler n-1 Crawler N Kafka Cluster Mentions © 2014 Brandwatch | www.brandwatch.com 40
  41. 41. Processing © 2014 Brandwatch | www.brandwatch.com 41
  42. 42. 42 Processing/ What’s happening now?
  43. 43. Step 2.1/ One processing JVM Crawler 1 Crawler 2 Crawler n-1 Crawler N Kafka Cluster Mentions Signals processor © 2014 Brandwatch | www.brandwatch.com 43
  44. 44. Processing/ A wild tweet appears! Mention date: 14/10/2014 6:10PM pageType: twitter author: @javadude hashtags: [#jaxlondon, #amazingtalk, #greatshoes] mentionedTweeters: [@jstanier] text: “@jstanier is at #jaxlondon tonight #amazingtalk #greatshoes” © 2014 Brandwatch | www.brandwatch.com 44
  45. 45. Processing/ Storing hashtags Map<Date, Multiset<String>>! ! Initialise with the last 24 hours © 2014 Brandwatch | www.brandwatch.com 45
  46. 46. Processing/ Storing hashtags Map<Date, Multiset<String>>! ! Mention date: 14/10/2014 6:10PM hashtags: [#jaxlondon, #amazingtalk, #greatshoes] © 2014 Brandwatch | www.brandwatch.com 46
  47. 47. Processing/ Storing hashtags Map<Date, Multiset<String>>! ! Mention date: 14/10/2014 6:10PM hashtags: [#jaxlondon, #amazingtalk, #greatshoes] add(“#jaxlondon”)! add(“#amazingtalk”)! add(“#greatshoes”)! © 2014 Brandwatch | www.brandwatch.com 47
  48. 48. Processing/ Cycling the buckets @Scheduled(cron = "0 0 * * * *")! public void cycleBuckets() {! Date oldest = buckets.lastKey();! removeBucket(oldest);! DateTime newest = new! ! ! !DateTime(buckets.firstKey());! addBucket(newest.plusHours(1).toDate());! }! © 2014 Brandwatch | www.brandwatch.com 48
  49. 49. Processing/ Detecting spikes • At regular intervals • For each hashtag • Convert to a timeseries [5, …. 1002, 5499] • Use our super secret detection algorithm • Give a score to it • If score > threshold, it’s interesting • Send it on a new Kafka topic © 2014 Brandwatch | www.brandwatch.com 49
  50. 50. Processing/ What we just did #hashtag data model © 2014 Brandwatch | www.brandwatch.com 50
  51. 51. Processing/ But we also track… author data model sentiment data model page type data model link share data model #hashtag data model country data model volume data model © 2014 Brandwatch | www.brandwatch.com 51
  52. 52. Processing/ …for one query “JAX London” query author data model sentiment data model page type data model link share data model #hashtag data model country data model volume data model © 2014 Brandwatch | www.brandwatch.com 52
  53. 53. Processing/ 100K+ queries and rising © 2014 Brandwatch | www.brandwatch.com 53
  54. 54. Processing/ We need more JVMs But how do we share the workload? © 2014 Brandwatch | www.brandwatch.com 54
  55. 55. Distribution of work © 2014 Brandwatch | www.brandwatch.com 55
  56. 56. Step 2.2/ A cluster of processing JVMs Crawler 1 Crawler 2 Crawler n-1 Crawler N Mentions Signals © 2014 Brandwatch | www.brandwatch.com Kafka Cluster Signals Processing cluster Mentions
  57. 57. Distribution/ An atomic unit of work © 2014 Brandwatch | www.brandwatch.com Signals Processing cluster ?
  58. 58. Distribution/ Leader election A way of deciding who is the leader for a task in a group of distributed nodes © 2014 Brandwatch | www.brandwatch.com 58
  59. 59. Distribution/ Zookeeper A way of coordinating and managing distributed applications © 2014 Brandwatch | www.brandwatch.com 59
  60. 60. Zookeeper/ It’s like a file system /brandwatch /feature_1 /feature_2 © 2014 Brandwatch | www.brandwatch.com 60
  61. 61. Zookeeper/ At the command line © 2014 Brandwatch | www.brandwatch.com 61
  62. 62. Distribution/ In Java http://curator.apache.org/curator-framework © 2014 Brandwatch | www.brandwatch.com 62
  63. 63. Distribution/ Recipes © 2014 Brandwatch | www.brandwatch.com 63
  64. 64. Distribution/ Instantiating Curator ! CuratorFrameworkFactory! .builder()! .connectString(zkQuorum)! .namespace(namespace)! .build()! ! ! ! !.start();! © 2014 Brandwatch | www.brandwatch.com 64
  65. 65. Distribution/ Offering jobs /brandwatch /signals /queries /15846 /1268589 Manager JVM DB © 2014 Brandwatch | www.brandwatch.com 65
  66. 66. Distribution/ Adding nodes ! public void createZNode(String queryId) {! try {! client.create().forPath(ZK_NODE_PREFIX + queryId);! } catch (NodeExistsException e) {! log.debug("Node {} was already created.”, queryId);! }! }! © 2014 Brandwatch | www.brandwatch.com 66
  67. 67. Distribution/ Deleting nodes ! public void removeZNode(String queryId) {! try {! client.delete().forPath(ZK_NODE_PREFIX + queryId);! } catch (NoNodeException e) {! log.debug("Node {} was already deleted.”, queryId);! }! }! © 2014 Brandwatch | www.brandwatch.com 67
  68. 68. Distribution/ Leader election 101 /brandwatch /signals /queries /15846 /1268589 Processing JVM 1 Processing JVM 2 Processing JVM 3 © 2014 Brandwatch | www.brandwatch.com 68
  69. 69. Distribution/ Leader election 101 /brandwatch /signals /queries /15846 /1268589 1 2 3 Processing JVM 1 Processing JVM 2 Processing JVM 3 © 2014 Brandwatch | www.brandwatch.com 69
  70. 70. Distribution/ The leader dies /brandwatch /signals /queries /15846 /1268589 2 3 Processing JVM Processing JVM © 2014 Brandwatch | www.brandwatch.com 70
  71. 71. Distribution/ The dead rises again /brandwatch /signals /queries /15846 /1268589 2 3 4 Processing JVM Processing JVM Processing JVM © 2014 Brandwatch | www.brandwatch.com 71
  72. 72. Distribution/ Curator: LeaderLatch ! public LeaderLatch(CuratorFramework client, String latchPath)! Parameters:! ! client - the client! latchPath - the path for this leadership group! © 2014 Brandwatch | www.brandwatch.com 72
  73. 73. Distribution/ LeaderLatch recipe public class WorkerManager implements PathChildrenCacheListener {! ! private Map<Integer, LeaderLatch> leaderLatches = newHashMap();! © 2014 Brandwatch | www.brandwatch.com 73 ! @Override! public void childEvent(CuratorFramework client, ! PathChildrenCacheEvent event) {! // Handle adds and removes here!! }! }!
  74. 74. Distribution/ Curator: Starting up @PostConstruct! public void initialise() throws Exception {! List<ChildData> currentData = newArrayList(initialisePathChildrenCache());! log.info("Pre creating workers for {} existing queries", currentData.size());! for (ChildData childData : currentData) {! int queryId = parseQueryIdFromPath(childData.getPath());! startLeaderElection(queryId);! }! }! © 2014 Brandwatch | www.brandwatch.com 74
  75. 75. Distribution/ Curator: PathChildrenCache private List<ChildData> initialisePathChildrenCache() throws Exception {! pathChildrenCache.start(StartMode.BUILD_INITIAL_CACHE);! pathChildrenCache.getListenable().addListener(this);! List<ChildData> currentData = pathChildrenCache.getCurrentData();! return currentData;! }! © 2014 Brandwatch | www.brandwatch.com 75
  76. 76. Distribution/ Curator: Adding a node @Override! public void childEvent(CuratorFramework client, PathChildrenCacheEvent event) { ! ChildData childData = event.getData();! switch (event.getType()) {! case CHILD_ADDED:! queryId = parseQueryIdFromPath(childData.getPath());! if (!haveLeaderLatchForQuery(queryId)) {! startLeaderElection(queryId);! }! break;! © 2014 Brandwatch | www.brandwatch.com 76
  77. 77. Distribution/ Curator: Deleting a node // Continued...! case CHILD_REMOVED:! queryId = parseQueryIdFromPath(childData.getPath());! removeLeaderLatchForQuery(queryId);! break;! default:! break;! }! }! © 2014 Brandwatch | www.brandwatch.com 77
  78. 78. Distribution/ Almost there? We are processing long running jobs What about workers getting overloaded? © 2014 Brandwatch | www.brandwatch.com 78
  79. 79. Distribution/ After leader election 1. Take leadership 2. Hit max queries? a. No – go to 3 b. Yes – give up leadership, try again 3. Start working © 2014 Brandwatch | www.brandwatch.com 79
  80. 80. Distribution/ Now we’re almost there? Actually, no… © 2014 Brandwatch | www.brandwatch.com 80
  81. 81. Distribution/ Infinite election At capacity! Processing JVM Elected for 1328 Elected for 1328 At capacity! At capacity! Elected for 1328 Processing JVM Processing JVM © 2014 Brandwatch | www.brandwatch.com 81
  82. 82. Distribution/ Solution Processing JVM Elected for 1328 At capacity! Elected for 1328 Processing JVM Processing JVM Refused 1328 © 2014 Brandwatch | www.brandwatch.com 82
  83. 83. Distribution/ Solution At capacity! Processing JVM Elected for 1328 Elected for 1328 Refused 1328 At capacity! At capacity! Elected for 1328 Processing JVM Processing JVM Refused 1328 Refused 1328 © 2014 Brandwatch | www.brandwatch.com 83
  84. 84. State © 2014 Brandwatch | www.brandwatch.com 84
  85. 85. State/ CAP theorem Availability CA AP CP © 2014 Brandwatch | www.brandwatch.com 85
  86. 86. State/ Snapshotting of worker data If one worker dies, we want the other to pick up where it left off Regular snapshotting to HBase © 2014 Brandwatch | www.brandwatch.com 86
  87. 87. State/ Serialisation and compression Serialise and compress using Kryo ~ 0.5MB per query, but a lot are very small © 2014 Brandwatch | www.brandwatch.com 87
  88. 88. Step 2.2/ Done! Crawler 1 Crawler 2 Crawler n-1 Crawler N Kafka Cluster Signals Signals Signals Processing cluster Signals handler JVM DB Mentions Mentions © 2014 Brandwatch | www.brandwatch.com 88
  89. 89. Monitoring © 2014 Brandwatch | www.brandwatch.com 89
  90. 90. Monitoring/ Statsd and Graphite © 2014 Brandwatch | www.brandwatch.com 90
  91. 91. Closing remarks © 2014 Brandwatch | www.brandwatch.com 91
  92. 92. © 2014 Brandwatch | www.brandwatch.com 92
  93. 93. Summary/ Using this architecture Now • Smarter alerts (email, push, in-browser) • Monitoring crises/events as they happen Underway • Automatic clustering of spikes into events • Historical analysis of trends © 2014 Brandwatch | www.brandwatch.com 93
  94. 94. Say hello/ jamess@brandwatch.com UK: +44 (0)1273 358 635 @brandwatch | @jstanier www.brandwatch.com © 2014 Brandwatch | www.brandwatch.com 94
  95. 95. Q&A © 2014 Brandwatch | www.brandwatch.com 95

×