Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scalable Architectures - Taming the Twitter Firehose

33,807 views

Published on

Published in: Technology
  • ZeroMQ is right solution for given use.. Excellent choice of tools. Right bus architecture.
    I am sure you must have considered AMQP based RabbitMQ..

    Did you not choose, AMQP based messaging due to performance overhead ( AMQP header overhead )
    I would appreciate your answer and thought process
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Never thought so much is behind Datasift platform. Very information. Thanks for the share.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Scalable Architectures - Taming the Twitter Firehose

  1. Lorenzo Alberton @lorenzoalbertonScalable Architectures: Taming the Twitter Firehose Patterns for scalable real-time platforms PHPDay 2012 Verona, 18th May 2012 1
  2. Outline1) SOAs scaling the platform 2
  3. Outline1) SOAs scaling the platform 2) Message Queues scaling the communication 2
  4. Outline1) SOAs scaling the platform 2) Message Queues scaling the communication 3) Monitoring scaling the maintainability 2
  5. DataSift Architecture High-level overview 3
  6. DataSift Architecture http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 4
  7. 1/4) Ingestion of Input Streams http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 5
  8. 2/4) Filtering http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 6
  9. 3/4) Delivery / Frontend http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 7
  10. 4/4) Monitoring / Historics / Analytics http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 8
  11. DataSift 350+ Million input messages/day * ~330 Million from the Twitter Firehose alone 9
  12. DataSift 2 Terabyte messages processed in real time and stored every day ~1 Petabyte of storage available 10
  13. DataSift Thousands of concurrent, custom output streams all crafted with tender love and surgical precision 11
  14. SOAService-Oriented Architectures 12
  15. Service-Oriented Architectures Service Service Service A B C Loose Coupling - Separation of Responsibilities http://en.wikipedia.org/wiki/Service-oriented_architecture 13
  16. Service-Oriented Architectures Consumer Service Service Service A B C Separate ConsumersSeparation of Responsibilities Loose Coupling - From Service Implementation http://en.wikipedia.org/wiki/Service-oriented_architecture 13
  17. Service-Oriented Architectures Consumer Consumer Proxy Cache Service Service Service A B C Separate ConsumersSeparation of Responsibilities Loose Couplingcaching atService Implementation Aggressive - From application level http://en.wikipedia.org/wiki/Service-oriented_architecture 13
  18. Service-Oriented Architectures Orchestrator Service Service Service A B COrchestration of distinctFrom ServiceResponsibilities Separate ConsumersSeparation of Implementation Loose Couplingcaching at accessible over a network Aggressive - units application level http://en.wikipedia.org/wiki/Service-oriented_architecture 13
  19. Service-Oriented Architectures Orchestrator JSON Thrift XML Service Service Service A B CCommunication distinctFrom Service Implementation Separate ConsumersSeparation interoperablenetworkOrchestration of via a -well-definedof Responsibilities Loose Couplingcaching at accessible over a format Aggressive units application level http://en.wikipedia.org/wiki/Service-oriented_architecture 13
  20. Independent Horizontal Scaling Service A Orchestrator Service B 14
  21. Independent Horizontal Scaling Service A Orchestrator Service B 14
  22. Independent Horizontal Scaling Service A Orchestrator Load Balancer Service B1 Load balancing - Service Service Multiple nodes B B2 14
  23. Independent Horizontal Scaling Rev.Proxy Better single-node Service performances with A application-level caching Orchestrator Load Balancer Service B1 Load balancing - Service Service Multiple nodes B B2 14
  24. Cell Architectures Ensure that everything +1 you develop has at least one additional instance N + 1 design of that system in the event of failure. Have multiple live, isolated nodes of the multiple same type to distribute live nodes the load. http://highscalability.com/blog/2012/5/9/cell-architectures.html 15
  25. Cardinality of Nodes on Each Service 3 2 2 5 2 2 2 8 8 5 7 60+ 7 7 7 7 7 http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 16
  26. Load-Balancing Example 17
  27. Load-Balancing with HAProxy # /etc/haproxy.cfg global frontend http-in    daemon    bind *:80    maxconn 256    default_backend mysvc     defaults backend mysvc    mode http    server s1 10.0.1.10:7474 maxconn 32    timeout connect 5000ms    server s2 10.0.1.11:7474 maxconn 32    timeout client 50000ms      timeout server 50000ms listen admin      bind *:8080    stats enableStart by running/usr/sbin/haproxy -f /etc/haproxy.cfg http://haproxy.1wt.eu/ 18
  28. Load-Balancing with Varnishbackend node01 { backend node02 { .host = "svc01.myhost.com"; .host = "svc02.myhost.com"; .probe = { .probe = { .url = "/"; .url = "/"; .interval = 1s; .interval = 1s; .timeout = 50 ms; .timeout = 50 ms; .window = 2; .window = 2; .threshold = 2; .threshold = 2; } }} }director mysvcdir round-robin { {.backend = node01;} mysvc {.backend = node02;} Request 50% node 01 Varnish}sub vcl_recv { set req.backend = mysvcdir; 50% mysvc return(pass); round-robin node 02} http://varnish-cache.org 19
  29. Caching Example 20
  30. Caching with VarnishNo special directives required to cache normal requests.Just use the defaults, and set Cache-Control headers. <?php $ttl = 300; //cache for 5 minutes $ts = new DateTime(@ . (time() + $ttl)); header("Expires: " . $ts->format(DateTime::RFC1123)); header("Cache-Control: max-age=$ttl, must-revalidate"); ?> Warning: by default, pages with cookies are not cached 21
  31. Application Programming Interfaces APIs Software-to-Software Contract 22
  32. API Docs GuidelinesSimple (RESTful verbs, actions on resources)Well defined (action, endpoint, parameters, response)Discoverable (self-describing endpoint)Working documentation 23
  33. APIs everywhere: Internal & External http://mashery.com/solution/iodocs http://console.datasift.com/ 24
  34. Service API discoveryGET /<servicename>/api 25
  35. Service Host Discovery - Config MgrGET /configuration/<servicename>/hostsHTTP/1.1 200 OKContent-Type: application/json; charset=UTF-8{ “service”: “<servicename>”, “hosts”:[ “10.0.1.33:80”, “10.0.1.34:80” ], “base_path”: “/svc/xyz/”} 26
  36. Service Host Discovery - Zookeeper ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. http://zookeeper.apache.org/<?php$zk = new Zookeeper();$zk->connect(localhost:2181); //server$params = array(array( perms => Zookeeper::PERM_ALL, scheme => world, id => anyone));if (!$zk->exists(/services/mysvc/host) { $zk->create(/services, config for internal services, $params); $zk->create(/services/mysvc, config for mysvc, $params); $zk->create(/services/mysvc/host, http://my.site.com, $params);} 27
  37. Service Host Discovery - Zookeeper ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. http://zookeeper.apache.org/<?php$zk = new Zookeeper();$zk->connect(localhost:2181); //server$params = array(array( perms => Zookeeper::PERM_ALL, scheme => world, id => anyone));if (!$zk->exists(/services/mysvc/host) { $zk->create(/services, config for internal services, $params); $zk->create(/services/mysvc, config for mysvc, $params); $zk->create(/services/mysvc/host, http://my.site.com, $params);} 27
  38. Service Host Discovery - Zookeeper ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. http://zookeeper.apache.org/<?php$zk = new Zookeeper();$zk->connect(localhost:2181); //client$host = $zk->get(/services/mysvc/host);... 28
  39. SOA - Scale Each Component http://www.thisnext.com/item/647CD0BE/Matryoshkas-Nesting-Dolls 29
  40. SOA - Scale Each Component http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 30
  41. SOA - Scale Each Component SOA: Independently scalable services. Example on distributing processing load: http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 30
  42. Workers for sharing processing load 31
  43. Workers for sharing processing load Distribute processing load among workers. Lightweight orchestration, heavy lifting in separate, asynchronous processes 31
  44. Scale all things! http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 32
  45. Scale all things! Example on scaling large data volumes: http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 32
  46. In Case of “Big Data”... 33
  47. In Case of “Big Data”... With lots of data, move the processing logic itself to the storage nodes (I/O is expensive) Map/Reduce, Parallel Processing 33
  48. Message Queues Asynchronous Communication 34
  49. MessagingZeroMQ: PUSH-PULL, REQ-REP, PUB-SUB (multicast, broadcast) Internal communication: pass messages to the next processing stage in the pipeline, control events, monitoring. Very high throughput. Socket library.Kafka/Redis: PUSH-PULL with persistence* Internal message / workload buffering and distributionNode.js: WebSockets / HTTP Streaming Message delivery (output) 35
  50. Message queues as Buffers (Decoupling) P C Unpredictable load spikes 36
  51. Message queues as Buffers (Decoupling) P C Unpredictable load spikes P C Load normalisation / smoothing 36
  52. Message queues as Buffers (Decoupling) P C Unpredictable load spikes P C Load normalisation / smoothing Batching ⇒ higher throughput 36
  53. Redis Buffer Example 37
  54. Redis Processing Queue<?php //producer(s)$redis = new Redis();$redis->connect(127.0.0.1, 6379, 1.5); // timeout 1.5 seconds...// push items to the queue as they are produced$redis->lPush(queue:xyz, $item);...<?php... //consumer(s)while (true) { // read items off the queue as they are available // block for up to 2 seconds (timeout) $item = redis->brPop(queue:xyz, 2); ...} https://github.com/nicolasff/phpredis https://github.com/chrisboulton/php-resque 38
  55. Kafka Buffer Example 39
  56. Kafka Processing QueueProducer<?php$host = 127.0.0.1;$port = 9092;$producer = new Kafka_Producer($host, $port);$messages = array( aaa, bbb, ccc,);$topic = test;// send a batch of messages (MessageSet)$bytes_sent = $producer->send($messages, $topic); https://github.com/apache/kafka/tree/trunk/clients/php/src/examples 40
  57. Kafka Processing QueueConsumer<?php$timeout = 2; $maxSize = 1000000;$host = 127.0.0.1; $port = 9092;$partition = 0; $offset = 0;$topic = test;$consumer = new Kafka_SimpleConsumer($host, $port, $timeout, $maxSize);while (true) { $request = new Kafka_FetchRequest($topic, $partition, $offset, $maxSize); $messages = $consumer->fetch($request); foreach ($messages as $msg) { echo $msg->payload(); } $offset += $messages->validBytes();} 41
  58. 0mq PUSH-PULL (Workload Distribution) Consumer 1 p ull push pull Producer Consumer 2 (blocking operation, pu ll until delivered to one worker) Consumer 3 42
  59. Workload Distribution Example 43
  60. ZeroMQ Producer (PUSH)<?php$context = new ZMQContext();$producer = $context->getSocket(ZMQ::SOCKET_PUSH);$producer->bind(tcp://*:5555);// send tasks to workers.foreach ($tasks as $task) { // Blocking operation until the message // is received by one (and only one) worker $producer->send($task);}... http://zguide.zeromq.org/php:all 44
  61. ZeroMQ Consumers (PULL)<?php$context = new ZMQContext();$worker = $context->getSocket(ZMQ::SOCKET_PULL);$worker->connect(tcp://myhost:5555);// process tasks foreverwhile (true) { // receive a message (blocking operation) $task = $worker->recv(); ...} 45
  62. 0mq PUSH-PULL (Mux) Producer 1 pus hR 1, R 2, R 3 push R4 Producer 2 pull Consumer 5 , R6 fair-queuing: ushR p R1, R4, R5, R2, R6, R3 Producer 3 46
  63. 0mq PUB-SUB (High Availability) Listener 1Publisher 1 Listener 2Publisher 2 Listener 3 [Broadcast] [Dynamic Subscriptions] 47
  64. 0mq PUB-SUB (High Availability) DC 1Publisher 1Publisher 2 DC 2 48
  65. High Availability - Replication Example 49
  66. ZeroMQ Producer (PUB)<?php$context = new ZMQContext();$producer = $context->getSocket(ZMQ::SOCKET_PUB);$producer->bind(tcp://*:5555);$messages = array( // topic => msg array(painters => Michelangelo), array(painters => Raffaello), array(sculptors => Donatello),);// send messages to listeners.foreach ($messages as $msg) { // Non-blocking operation. No ACK. // Message sent to ALL subscribers $producer->sendMulti($msg);} 50
  67. ZeroMQ Consumer (SUB)<?php$context = new ZMQContext();$producer = $context->getSocket(ZMQ::SOCKET_SUB);$producer->connect(tcp://myhost:5555);$topic = painters; // ignore sculptors$producer->setSockOption( ZMQ::SOCKOPT_SUBSCRIBE, $topic);// Listen to messages with given topicwhile (true) { list($t, $m) = $producer->recvMulti(); // $t is the topic (‘painters.*’)} 51
  68. Interesting Ideas Some Architecture Ideas 52
  69. Internal “Firehose” Publishers Subscribers Alice’s John’s Y Z timeline Inbox X subscribe to topic X Data Bus subscribe to topic Y System Fred’s Tech Monitor Followers Blog Feed 53
  70. Internal “Firehose” Publishers Data Feeds, Subscribers User-generated Alice’s John’s content, timeline Inbox X Y Z System events, ... subscribe to topic X Data Bus subscribe to topic Y System Fred’s Tech Monitor Followers Blog Feed 53
  71. Internal “Firehose” Publishers Applications, Subscribers Services, Monitors, Alice’s John’s Y Z Routers, timeline Inbox X Repeaters, subscribe ... to topic X Data Bus subscribe to topic Y System Fred’s Tech Monitor Followers Blog Feed 53
  72. Internal “Firehose” Publishers Subscribers Alice’s John’s Y Z timeline Inbox X subscribe to topic X Data Bus subscribe to topic Y Everyone connected to System Fred’s Tech the data bus, no Monitor Followers Blog Feed directed graph 53
  73. Internal “Firehose” Publishers Subscribers Alice’s John’s Y Z timeline Inbox X subscribe to topic X Data Bus subscribe to topic Y System Fred’s Tech Monitor Followers Blog Feed 53
  74. Monitoring Measure Anything, Measure Everythinghttp://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ 54
  75. Monitoring: Measure Everything 55
  76. Monitoring: Measure Everything 1. Is there a problem? User experience / Business metrics monitors 2. Where is the problem? System monitors (threshold - variance) 3. What is the problem? Application monitors 55
  77. Monitoring: Measure Everything 1. Is there a problem? User experience / Business metrics monitors 2. Where is the problem? System monitors (threshold - variance) 3. What is the problem? Application monitors Keep Signal vs. Noise ratio high 55
  78. Monitoring: Measure Everything StatsD 1. Is there a problem? User experience / Business metrics monitors 2. Where is the problem? System monitors (threshold - variance) 3. What is the problem? Application monitors Keep Signal vs. Noise ratio high 55
  79. Instrumentation https://play.google.com/store/apps/details?id=net.networksaremadeofstring.rhybudd 56
  80. Look! Monitors! 57
  81. Look! Monitors! 57
  82. StatsD + Graphite Example StatsD: Node.JS daemon. Listens for messages over a UDP port and extracts metrics, which are dumped to Graphite for further processing and visualisation. Graphite: Real-time graphing system. Data is sent to carbon (processing back-end) which stores data into Graphite’s db. Data visualised via Graphite’s web interface. 58
  83. StatsD Metrics<?php ; statsd.ini$statsTypePrefix = workerX.received.type.; [statsd] host = yourhost$statsTimeKey = workerX.processing_time; port = 8125while (true) { $batch = $worker->getBatchOfWork(); foreach ($batch as $item) { // time how long it takes to process this item... $time_start = microtime(true); // ... process item here ... $time = (int)(1000 * (microtime(true) - $time_start)); StatsD::timing($statsTimeKey, $time); // time in ms // count items by type StatsD::increment($statsTypePrefix . $item[type]);} https://github.com/etsy/statsd/ 59
  84. StatsD Metrics<?php ; statsd.ini$statsTypePrefix = workerX.received.type.; [statsd] host = yourhost$statsTimeKey = workerX.processing_time; port = 8125while (true) { $batch = $worker->getBatchOfWork(); foreach ($batch as $item) { // time how long it takes to process this item... $time_start = microtime(true); // ... process item here ... $time = (int)(1000 * (microtime(true) - $time_start)); StatsD::timing($statsTimeKey, $time); // time in ms // count items by type StatsD::increment($statsTypePrefix . $item[type]);} https://github.com/etsy/statsd/ 59
  85. Graphite Output workerX.processing_time.mean workerX.processing_time.upper_90 http://graphite.wikidot.com/ 60
  86. Graphite Output monitor average, percentiles, standard deviation workerX.processing_time.mean workerX.processing_time.upper_90 http://graphite.wikidot.com/ 60
  87. Look! Rib cages! Network Load Viz http://www.network-weathermap.com/ http://cacti.net 61
  88. Look! Rib cages! Network Load Viz Not enough! Contextualise metrics http://www.network-weathermap.com/ http://cacti.net 61
  89. Cacti + WeatherMap Example Cacti: Network graphing solution harnessing the power of RRDTool’s data storage and graphing functionality. Provides a fast poller, graph templating, multiple data acquisition methods. Weathermap: Cacti plugin to integrate network maps into the Cacti web UI. Includes a web-based map editor. 62
  90. Network Load Visualisation 345/s 8432/s 225/s 296/s 335/s 7312/s 311/s 289/s 145/s 4410/s 5320/s 80/s 1331/s 5320/s 5320/s 13/s 2954/s 44/s 3296/s 4322/s 219/s 2954/s 5320/s 832/s 5320/s Graphite datasource for Weathermap: https://github.com/alexforrow/php-weathermap-graphite 63
  91. Network Load Visualisation 345/s 8432/s 225/s 296/s 335/s 7312/s 311/s 289/s 145/s 5320/s augmentation 4410/s 80/s service 1331/s timing out? 5320/s 5320/s 13/s 2954/s 44/s 3296/s 4322/s 219/s 2954/s 5320/s 832/s 5320/s Graphite datasource for Weathermap: https://github.com/alexforrow/php-weathermap-graphite 63
  92. Network Load Visualisation filtering server 8432/s 345/s slightly 225/s 296/s 335/s overloaded? 7312/s 311/s 289/s 145/s 4410/s 5320/s 80/s 1331/s 5320/s 5320/s 13/s 2954/s 44/s 3296/s 4322/s 219/s 2954/s 5320/s 832/s 5320/s Graphite datasource for Weathermap: https://github.com/alexforrow/php-weathermap-graphite 63
  93. Network Load Visualisation 345/s 8432/s 225/s 296/s 335/s 7312/s 311/s 289/s 145/s 4410/s 5320/s 1331/s consumer 80/s 5320/s 5320/s slower than 13/s producer? 2954/s 44/s 3296/s 4322/s 219/s 2954/s 5320/s 832/s 5320/s Graphite datasource for Weathermap: https://github.com/alexforrow/php-weathermap-graphite 63
  94. Monitoring Reporting GuidelinesMake the subtle obviousMake the complex/busy simple/cleanGroup information by contextDetect anomalies/deviation from normTurn raw numbers into graphsAppeal to intuition 64
  95. We’re Hiring!http://datasift.com/whoweare/jobs 65
  96. References http://www.slideshare.net/quipo/the-art-of-scalability-managing-growth https://bitly.com/vCSd49 (DataSift architecture on HighScalability) http://www.slideshare.net/combell/varnish-in-action-phpday2011 https://vimeo.com/couchmode/chariottechcast/videos/sort:date/40988625 http://blog.stuartherbert.com/php/2011/09/21/real-time-graphing-with- graphite/ http://zguide.zeromq.org/page:allImage credits:http://accidental-entrepreneur.com/wp-content/uploads/2011/04/fire-hose.jpghttp://www.alibaba.com/product-free/103854677/Q_FIRE_FIRE_HOSE.html 66
  97. Lorenzo Alberton @lorenzoalberton Thank you! lorenzo@alberton.infohttp://www.alberton.info/talks https://joind.in/6372 67

×