مدخل إلى تعلم الآلة
Your SlideShare is downloading.
×
Check these out next
Recommended
More Related Content
Slideshows for you (20)
Viewers also liked (6)
Similar to Scalable Architectures - Taming the Twitter Firehose (20)
Recently uploaded (20)
Scalable Architectures - Taming the Twitter Firehose
- Lorenzo Alberton @lorenzoalberton Scalable Architectures: Taming the Twitter Firehose Patterns for scalable real-time platforms PHPDay 2012 Verona, 18th May 2012 1
- Outline 1) SOAs scaling the platform 2
- Outline 1) SOAs scaling the platform 2) Message Queues scaling the communication 2
- Outline 1) SOAs scaling the platform 2) Message Queues scaling the communication 3) Monitoring scaling the maintainability 2
- DataSift Architecture High-level overview 3
- DataSift Architecture http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 4
- 1/4) Ingestion of Input Streams http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 5
- 2/4) Filtering http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 6
- 3/4) Delivery / Frontend http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 7
- 4/4) Monitoring / Historics / Analytics http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 8
- DataSift 350+ Million input messages/day * ~330 Million from the Twitter Firehose alone 9
- DataSift 2 Terabyte messages processed in real time and stored every day ~1 Petabyte of storage available 10
- DataSift Thousands of concurrent, custom output streams all crafted with tender love and surgical precision 11
- SOA Service-Oriented Architectures 12
- Service-Oriented Architectures Service Service Service A B C Loose Coupling - Separation of Responsibilities http://en.wikipedia.org/wiki/Service-oriented_architecture 13
- Service-Oriented Architectures Consumer Service Service Service A B C Separate ConsumersSeparation of Responsibilities Loose Coupling - From Service Implementation http://en.wikipedia.org/wiki/Service-oriented_architecture 13
- Service-Oriented Architectures Consumer Consumer Proxy Cache Service Service Service A B C Separate ConsumersSeparation of Responsibilities Loose Couplingcaching atService Implementation Aggressive - From application level http://en.wikipedia.org/wiki/Service-oriented_architecture 13
- Service-Oriented Architectures Orchestrator Service Service Service A B C Orchestration of distinctFrom ServiceResponsibilities Separate ConsumersSeparation of Implementation Loose Couplingcaching at accessible over a network Aggressive - units application level http://en.wikipedia.org/wiki/Service-oriented_architecture 13
- Service-Oriented Architectures Orchestrator JSON Thrift XML Service Service Service A B C Communication distinctFrom Service Implementation Separate ConsumersSeparation interoperablenetwork Orchestration of via a -well-definedof Responsibilities Loose Couplingcaching at accessible over a format Aggressive units application level http://en.wikipedia.org/wiki/Service-oriented_architecture 13
- Independent Horizontal Scaling Service A Orchestrator Service B 14
- Independent Horizontal Scaling Service A Orchestrator Service B 14
- Independent Horizontal Scaling Service A Orchestrator Load Balancer Service B1 Load balancing - Service Service Multiple nodes B B2 14
- Independent Horizontal Scaling Rev.Proxy Better single-node Service performances with A application-level caching Orchestrator Load Balancer Service B1 Load balancing - Service Service Multiple nodes B B2 14
- Cell Architectures Ensure that everything +1 you develop has at least one additional instance N + 1 design of that system in the event of failure. Have multiple live, isolated nodes of the multiple same type to distribute live nodes the load. http://highscalability.com/blog/2012/5/9/cell-architectures.html 15
- Cardinality of Nodes on Each Service 3 2 2 5 2 2 2 8 8 5 7 60+ 7 7 7 7 7 http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 16
- Load-Balancing Example 17
- Load-Balancing with HAProxy # /etc/haproxy.cfg global frontend http-in daemon bind *:80 maxconn 256 default_backend mysvc defaults backend mysvc mode http server s1 10.0.1.10:7474 maxconn 32 timeout connect 5000ms server s2 10.0.1.11:7474 maxconn 32 timeout client 50000ms timeout server 50000ms listen admin bind *:8080 stats enable Start by running /usr/sbin/haproxy -f /etc/haproxy.cfg http://haproxy.1wt.eu/ 18
- Load-Balancing with Varnish backend node01 { backend node02 { .host = "svc01.myhost.com"; .host = "svc02.myhost.com"; .probe = { .probe = { .url = "/"; .url = "/"; .interval = 1s; .interval = 1s; .timeout = 50 ms; .timeout = 50 ms; .window = 2; .window = 2; .threshold = 2; .threshold = 2; } } } } director mysvcdir round-robin { {.backend = node01;} mysvc {.backend = node02;} Request 50% node 01 Varnish } sub vcl_recv { set req.backend = mysvcdir; 50% mysvc return(pass); round-robin node 02 } http://varnish-cache.org 19
- Caching Example 20
- Caching with Varnish No special directives required to cache normal requests. Just use the defaults, and set Cache-Control headers. <?php $ttl = 300; //cache for 5 minutes $ts = new DateTime('@' . (time() + $ttl)); header("Expires: " . $ts->format(DateTime::RFC1123)); header("Cache-Control: max-age=$ttl, must-revalidate"); ?> Warning: by default, pages with cookies are not cached 21
- Application Programming Interfaces APIs Software-to-Software Contract 22
- API Docs Guidelines Simple (RESTful verbs, actions on resources) Well defined (action, endpoint, parameters, response) Discoverable (self-describing endpoint) Working documentation 23
- APIs everywhere: Internal & External http://mashery.com/solution/iodocs http://console.datasift.com/ 24
- Service API discovery GET /<servicename>/api 25
- Service Host Discovery - Config Mgr GET /configuration/<servicename>/hosts HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 { “service”: “<servicename>”, “hosts”:[ “10.0.1.33:80”, “10.0.1.34:80” ], “base_path”: “/svc/xyz/” } 26
- Service Host Discovery - Zookeeper ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. http://zookeeper.apache.org/ <?php $zk = new Zookeeper(); $zk->connect('localhost:2181'); //server $params = array(array( 'perms' => Zookeeper::PERM_ALL, 'scheme' => 'world', 'id' => 'anyone' )); if (!$zk->exists('/services/mysvc/host') { $zk->create('/services', 'config for internal services', $params); $zk->create('/services/mysvc', 'config for mysvc', $params); $zk->create('/services/mysvc/host', 'http://my.site.com', $params); } 27
- Service Host Discovery - Zookeeper ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. http://zookeeper.apache.org/ <?php $zk = new Zookeeper(); $zk->connect('localhost:2181'); //server $params = array(array( 'perms' => Zookeeper::PERM_ALL, 'scheme' => 'world', 'id' => 'anyone' )); if (!$zk->exists('/services/mysvc/host') { $zk->create('/services', 'config for internal services', $params); $zk->create('/services/mysvc', 'config for mysvc', $params); $zk->create('/services/mysvc/host', 'http://my.site.com', $params); } 27
- Service Host Discovery - Zookeeper ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. http://zookeeper.apache.org/ <?php $zk = new Zookeeper(); $zk->connect('localhost:2181'); //client $host = $zk->get('/services/mysvc/host'); ... 28
- SOA - Scale Each Component http://www.thisnext.com/item/647CD0BE/Matryoshkas-Nesting-Dolls 29
- SOA - Scale Each Component http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 30
- SOA - Scale Each Component SOA: Independently scalable services. Example on distributing processing load: http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 30
- Workers for sharing processing load 31
- Workers for sharing processing load Distribute processing load among workers. Lightweight orchestration, heavy lifting in separate, asynchronous processes 31
- Scale all things! http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 32
- Scale all things! Example on scaling large data volumes: http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html 32
- In Case of “Big Data”... 33
- In Case of “Big Data”... With lots of data, move the processing logic itself to the storage nodes (I/O is expensive) Map/Reduce, Parallel Processing 33
- Message Queues Asynchronous Communication 34
- Messaging ZeroMQ: PUSH-PULL, REQ-REP, PUB-SUB (multicast, broadcast) Internal communication: pass messages to the next processing stage in the pipeline, control events, monitoring. Very high throughput. Socket library. Kafka/Redis: PUSH-PULL with persistence* Internal message / workload buffering and distribution Node.js: WebSockets / HTTP Streaming Message delivery (output) 35
- Message queues as Buffers (Decoupling) P C Unpredictable load spikes 36
- Message queues as Buffers (Decoupling) P C Unpredictable load spikes P C Load normalisation / smoothing 36
- Message queues as Buffers (Decoupling) P C Unpredictable load spikes P C Load normalisation / smoothing Batching ⇒ higher throughput 36
- Redis Buffer Example 37
- Redis Processing Queue <?php //producer(s) $redis = new Redis(); $redis->connect('127.0.0.1', 6379, 1.5); // timeout 1.5 seconds ... // push items to the queue as they are produced $redis->lPush('queue:xyz', $item); ... <?php ... //consumer(s) while (true) { // read items off the queue as they are available // block for up to 2 seconds (timeout) $item = redis->brPop('queue:xyz', 2); ... } https://github.com/nicolasff/phpredis https://github.com/chrisboulton/php-resque 38
- Kafka Buffer Example 39
- Kafka Processing Queue Producer <?php $host = '127.0.0.1'; $port = 9092; $producer = new Kafka_Producer($host, $port); $messages = array( 'aaa', 'bbb', 'ccc', ); $topic = 'test'; // send a batch of messages (MessageSet) $bytes_sent = $producer->send($messages, $topic); https://github.com/apache/kafka/tree/trunk/clients/php/src/examples 40
- Kafka Processing Queue Consumer <?php $timeout = 2; $maxSize = 1000000; $host = '127.0.0.1'; $port = 9092; $partition = 0; $offset = 0; $topic = 'test'; $consumer = new Kafka_SimpleConsumer($host, $port, $timeout, $maxSize); while (true) { $request = new Kafka_FetchRequest($topic, $partition, $offset, $maxSize); $messages = $consumer->fetch($request); foreach ($messages as $msg) { echo $msg->payload(); } $offset += $messages->validBytes(); } 41
- 0mq PUSH-PULL (Workload Distribution) Consumer 1 p ull push pull Producer Consumer 2 (blocking operation, pu ll until delivered to one worker) Consumer 3 42
- Workload Distribution Example 43
- ZeroMQ Producer (PUSH) <?php $context = new ZMQContext(); $producer = $context->getSocket(ZMQ::SOCKET_PUSH); $producer->bind('tcp://*:5555'); // send tasks to workers. foreach ($tasks as $task) { // Blocking operation until the message // is received by one (and only one) worker $producer->send($task); } ... http://zguide.zeromq.org/php:all 44
- ZeroMQ Consumers (PULL) <?php $context = new ZMQContext(); $worker = $context->getSocket(ZMQ::SOCKET_PULL); $worker->connect('tcp://myhost:5555'); // process tasks forever while (true) { // receive a message (blocking operation) $task = $worker->recv(); ... } 45
- 0mq PUSH-PULL (Mux) Producer 1 pus hR 1, R 2, R 3 push R4 Producer 2 pull Consumer 5 , R6 fair-queuing: ushR p R1, R4, R5, R2, R6, R3 Producer 3 46
- 0mq PUB-SUB (High Availability) Listener 1 Publisher 1 Listener 2 Publisher 2 Listener 3 [Broadcast] [Dynamic Subscriptions] 47
- 0mq PUB-SUB (High Availability) DC 1 Publisher 1 Publisher 2 DC 2 48
- High Availability - Replication Example 49
- ZeroMQ Producer (PUB) <?php $context = new ZMQContext(); $producer = $context->getSocket(ZMQ::SOCKET_PUB); $producer->bind('tcp://*:5555'); $messages = array( // topic => msg array('painters' => 'Michelangelo'), array('painters' => 'Raffaello'), array('sculptors' => 'Donatello'), ); // send messages to listeners. foreach ($messages as $msg) { // Non-blocking operation. No ACK. // Message sent to ALL subscribers $producer->sendMulti($msg); } 50
- ZeroMQ Consumer (SUB) <?php $context = new ZMQContext(); $producer = $context->getSocket(ZMQ::SOCKET_SUB); $producer->connect('tcp://myhost:5555'); $topic = 'painters'; // ignore sculptors $producer->setSockOption( ZMQ::SOCKOPT_SUBSCRIBE, $topic ); // Listen to messages with given topic while (true) { list($t, $m) = $producer->recvMulti(); // $t is the topic (‘painters.*’) } 51
- Interesting Ideas Some Architecture Ideas 52
- Internal “Firehose” Publishers Subscribers Alice’s John’s Y Z timeline Inbox X subscribe to topic X Data Bus subscribe to topic Y System Fred’s Tech Monitor Followers Blog Feed 53
- Internal “Firehose” Publishers Data Feeds, Subscribers User-generated Alice’s John’s content, timeline Inbox X Y Z System events, ... subscribe to topic X Data Bus subscribe to topic Y System Fred’s Tech Monitor Followers Blog Feed 53
- Internal “Firehose” Publishers Applications, Subscribers Services, Monitors, Alice’s John’s Y Z Routers, timeline Inbox X Repeaters, subscribe ... to topic X Data Bus subscribe to topic Y System Fred’s Tech Monitor Followers Blog Feed 53
- Internal “Firehose” Publishers Subscribers Alice’s John’s Y Z timeline Inbox X subscribe to topic X Data Bus subscribe to topic Y Everyone connected to System Fred’s Tech the data bus, no Monitor Followers Blog Feed directed graph 53
- Internal “Firehose” Publishers Subscribers Alice’s John’s Y Z timeline Inbox X subscribe to topic X Data Bus subscribe to topic Y System Fred’s Tech Monitor Followers Blog Feed 53
- Monitoring Measure Anything, Measure Everything http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ 54
- Monitoring: Measure Everything 55
- Monitoring: Measure Everything 1. Is there a problem? User experience / Business metrics monitors 2. Where is the problem? System monitors (threshold - variance) 3. What is the problem? Application monitors 55
- Monitoring: Measure Everything 1. Is there a problem? User experience / Business metrics monitors 2. Where is the problem? System monitors (threshold - variance) 3. What is the problem? Application monitors Keep Signal vs. Noise ratio high 55
- Monitoring: Measure Everything StatsD 1. Is there a problem? User experience / Business metrics monitors 2. Where is the problem? System monitors (threshold - variance) 3. What is the problem? Application monitors Keep Signal vs. Noise ratio high 55
- Instrumentation https://play.google.com/store/apps/details?id=net.networksaremadeofstring.rhybudd 56
- Look! Monitors! 57
- Look! Monitors! 57
- StatsD + Graphite Example StatsD: Node.JS daemon. Listens for messages over a UDP port and extracts metrics, which are dumped to Graphite for further processing and visualisation. Graphite: Real-time graphing system. Data is sent to carbon (processing back-end) which stores data into Graphite’s db. Data visualised via Graphite’s web interface. 58
- StatsD Metrics <?php ; statsd.ini $statsTypePrefix = 'workerX.received.type.'; [statsd] host = yourhost $statsTimeKey = 'workerX.processing_time'; port = 8125 while (true) { $batch = $worker->getBatchOfWork(); foreach ($batch as $item) { // time how long it takes to process this item... $time_start = microtime(true); // ... process item here ... $time = (int)(1000 * (microtime(true) - $time_start)); StatsD::timing($statsTimeKey, $time); // time in ms // count items by type StatsD::increment($statsTypePrefix . $item['type']); } https://github.com/etsy/statsd/ 59
- StatsD Metrics <?php ; statsd.ini $statsTypePrefix = 'workerX.received.type.'; [statsd] host = yourhost $statsTimeKey = 'workerX.processing_time'; port = 8125 while (true) { $batch = $worker->getBatchOfWork(); foreach ($batch as $item) { // time how long it takes to process this item... $time_start = microtime(true); // ... process item here ... $time = (int)(1000 * (microtime(true) - $time_start)); StatsD::timing($statsTimeKey, $time); // time in ms // count items by type StatsD::increment($statsTypePrefix . $item['type']); } https://github.com/etsy/statsd/ 59
- Graphite Output workerX.processing_time.mean workerX.processing_time.upper_90 http://graphite.wikidot.com/ 60
- Graphite Output monitor average, percentiles, standard deviation workerX.processing_time.mean workerX.processing_time.upper_90 http://graphite.wikidot.com/ 60
- Look! Rib cages! Network Load Viz http://www.network-weathermap.com/ http://cacti.net 61
- Look! Rib cages! Network Load Viz Not enough! Contextualise metrics http://www.network-weathermap.com/ http://cacti.net 61
- Cacti + WeatherMap Example Cacti: Network graphing solution harnessing the power of RRDTool’s data storage and graphing functionality. Provides a fast poller, graph templating, multiple data acquisition methods. Weathermap: Cacti plugin to integrate network maps into the Cacti web UI. Includes a web-based map editor. 62
- Network Load Visualisation 345/s 8432/s 225/s 296/s 335/s 7312/s 311/s 289/s 145/s 4410/s 5320/s 80/s 1331/s 5320/s 5320/s 13/s 2954/s 44/s 3296/s 4322/s 219/s 2954/s 5320/s 832/s 5320/s Graphite datasource for Weathermap: https://github.com/alexforrow/php-weathermap-graphite 63
- Network Load Visualisation 345/s 8432/s 225/s 296/s 335/s 7312/s 311/s 289/s 145/s 5320/s augmentation 4410/s 80/s service 1331/s timing out? 5320/s 5320/s 13/s 2954/s 44/s 3296/s 4322/s 219/s 2954/s 5320/s 832/s 5320/s Graphite datasource for Weathermap: https://github.com/alexforrow/php-weathermap-graphite 63
- Network Load Visualisation filtering server 8432/s 345/s slightly 225/s 296/s 335/s overloaded? 7312/s 311/s 289/s 145/s 4410/s 5320/s 80/s 1331/s 5320/s 5320/s 13/s 2954/s 44/s 3296/s 4322/s 219/s 2954/s 5320/s 832/s 5320/s Graphite datasource for Weathermap: https://github.com/alexforrow/php-weathermap-graphite 63
- Network Load Visualisation 345/s 8432/s 225/s 296/s 335/s 7312/s 311/s 289/s 145/s 4410/s 5320/s 1331/s consumer 80/s 5320/s 5320/s slower than 13/s producer? 2954/s 44/s 3296/s 4322/s 219/s 2954/s 5320/s 832/s 5320/s Graphite datasource for Weathermap: https://github.com/alexforrow/php-weathermap-graphite 63
- Monitoring Reporting Guidelines Make the subtle obvious Make the complex/busy simple/clean Group information by context Detect anomalies/deviation from norm Turn raw numbers into graphs Appeal to intuition 64
- We’re Hiring! http://datasift.com/whoweare/jobs 65
- References http://www.slideshare.net/quipo/the-art-of-scalability-managing-growth https://bitly.com/vCSd49 (DataSift architecture on HighScalability) http://www.slideshare.net/combell/varnish-in-action-phpday2011 https://vimeo.com/couchmode/chariottechcast/videos/sort:date/40988625 http://blog.stuartherbert.com/php/2011/09/21/real-time-graphing-with- graphite/ http://zguide.zeromq.org/page:all Image credits: http://accidental-entrepreneur.com/wp-content/uploads/2011/04/fire-hose.jpg http://www.alibaba.com/product-free/103854677/Q_FIRE_FIRE_HOSE.html 66
- Lorenzo Alberton @lorenzoalberton Thank you! lorenzo@alberton.info http://www.alberton.info/talks https://joind.in/6372 67
