Asynchronous Architectures for Implementing Scalable Cloud Services - Evan Cooke - Gluecon 2012

AsynchronousArchitectures forImplementingScalable CloudServicesDesigning for GracefulDegradationEVAN COOKECO-FOUNDER & CTO twilio CLOUD COMMUNICATIONS

Cloud services power the apps that arethe backbone of modern society. How we work, play, and communicate.

Cloud Workloads Can Be Unpredictable

SMS API Usage 6x spike in 5 mins

Danger! Load higher than instantaneous throughput Load FAILRequestLatency Time

Don’t FailRequests

Incoming Requests Load Balancer Worker Pool AAA ... AAA AAAThrottling Throttling Throttling W W App App App W App WServer Server Server Server W W W WThrottling Throttling Throttling

Worker Pools e.g., Apache/Nginx Failed Requests 100%+ 70%10% Time

Problem Summary• Cloud services often use worker pools to handle incoming requests• When load goes beyond size of the worker pool, requests fail

What next?A few observations based on workimplementing and scaling the Twilio APIover the past 4 years... • Twilio Voice/SMS Cloud APIs • 100,000 Twilio Developers • 100+ employees

Observation 1For many APIs, taking more time toservice a request is better than failing thatrequestImplication: in many cases, it is betterto service a request with some delayrather than failing it

Observation 2Matching the amount of availableresources precisely to the size of incomingrequest worker pools is challengingImplication: under load, it may bepossible delay or drop only thoserequests that truly impact resources

What are we going to do?Suggestion: if request concurrency wasvery cheap, we could implement delayand ﬁner-grained resource controls muchmore easily...

Event-driven programming and the Reactor Pattern

Event-driven programming and the Reactor Pattern Worker Time req = ‘GET /’; 1 req.append(‘/r/n/r/n’); 1 socket.write(req); 10000x resp = socket.read(); 10000000x print(resp); 10

Event-driven programming and the Reactor Pattern Time req = ‘GET /’; 1 req.append(‘/r/n/r/n’); 1 socket.write(req); 10000x resp = socket.read(); 10000000x print(resp); 10 Huge IO latency blocks worker

Event-driven programming and the Reactor Pattern req = ‘GET /’; req.append(‘/r/n/r/n’); socket.write(req, fn() { Make IO socket.read(fn(resp) { operations print(resp); async and }); “callback” }); when done

Event-driven programming and the Reactor Patternreq = ‘GET /’;req.append(‘/r/n/r/n’);socket.write(req, fn() { socket.read(fn(resp) { print(resp); }); Central dispatch}); to coordinatereactor.run_forever(); event callbacks

Event-driven programming and the Reactor Pattern Timereq = ‘GET /’; 1req.append(‘/r/n/r/n’); 1socket.write(req, fn() { 10 socket.read(fn(resp) { 10 print(resp); 10 }); Result: we}); don’t blockreactor.run_forever(); the worker

(Some)Reactor Pattern Frameworks c/libevent c/libev java/nio/netty js/node.js ruby/eventmachine python/twisted python/gevent

The Callback Mess Python Twistedreq = ‘GET /’req += ‘/r/n/r/n’def r(resp): print respdef w(): socket.read().addCallback(r)socket.write().addCallback(w)

The Callback Mess Python Twistedreq = ‘GET /’req += ‘/r/n/r/n’yield socket.write()resp = yield socket.read()print resp Use deferred generators and inline callbacks

The Callback Mess Python Twistedreq = ‘GET /’req += ‘/r/n/r/n’yield socket.write()resp = yield socket.read()print resp Easy sequential programming with mostly implicit async IO

Enter gevent “gevent is a coroutine-based Python networking library that uses greenletto provide a high-level synchronous API on top of the libevent event loop.” Natively Async socket.write() resp = socket.read() print resp

Enter gevent Simple Echo Serverfrom gevent.server import StreamServerdef echo(socket, address): print (New connection from %s:%s % address) socket.sendall(Welcome to the echo server!rn) line = fileobj.readline() fileobj.write(line) fileobj.flush() print ("echoed %r" % line)if __name__ == __main__: server = StreamServer((0.0.0.0, 6000), echo) server.serve_forever() Easy sequential model Fully async

Async Services with Ginkgo Ginkgo is a simple framework for composing async gevent services with common conﬁguration, logging, demonizing etc. https://github.com/progrium/ginkgo Let’s look a simple example that implements a TCP and HTTP server...

Async Services with Ginkgoimport geventfrom gevent.pywsgi import WSGIServerfrom gevent.server import StreamServerfrom ginkgo.core import Servicedef handle_http(env, start_response): start_response(200 OK, [(Content-Type, text/html)]) print new http request! return ["hello world"]def handle_tcp(socket, address): print new tcp connection! while True: socket.send(hellon) gevent.sleep(1)app = Service()app.add_service(StreamServer((127.0.0.1, 1234),handle_tcp))app.add_service(WSGIServer((127.0.0.1, 8080), handle_http))app.serve_forever()

Async Services with Ginkgoimport geventfrom gevent.pywsgi import WSGIServerfrom gevent.server import StreamServer Import WSGI/TCP Serversfrom ginkgo.core import Servicedef handle_http(env, start_response): start_response(200 OK, [(Content-Type, text/html)]) print new http request! return ["hello world"]def handle_tcp(socket, address): print new tcp connection! while True: socket.send(hellon) gevent.sleep(1)app = Service()app.add_service(StreamServer((127.0.0.1, 1234),handle_tcp))app.add_service(WSGIServer((127.0.0.1, 8080), handle_http))app.serve_forever()

Async Services with Ginkgoimport geventfrom gevent.pywsgi import WSGIServerfrom gevent.server import StreamServerfrom ginkgo.core import Service HTTP Handlerdef handle_http(env, start_response): start_response(200 OK, [(Content-Type, text/html)]) print new http request! return ["hello world"]def handle_tcp(socket, address): print new tcp connection! while True: socket.send(hellon) gevent.sleep(1)app = Service()app.add_service(StreamServer((127.0.0.1, 1234),handle_tcp))app.add_service(WSGIServer((127.0.0.1, 8080), handle_http))app.serve_forever()

Async Services with Ginkgoimport geventfrom gevent.pywsgi import WSGIServerfrom gevent.server import StreamServerfrom ginkgo.core import Servicedef handle_http(env, start_response): start_response(200 OK, [(Content-Type, text/html)]) print new http request! return ["hello world"]def handle_tcp(socket, address): TCP Handler print new tcp connection! while True: socket.send(hellon) gevent.sleep(1)app = Service()app.add_service(StreamServer((127.0.0.1, 1234),handle_tcp))app.add_service(WSGIServer((127.0.0.1, 8080), handle_http))app.serve_forever()

Async Services with Ginkgoimport geventfrom gevent.pywsgi import WSGIServerfrom gevent.server import StreamServerfrom ginkgo.core import Servicedef handle_http(env, start_response): start_response(200 OK, [(Content-Type, text/html)]) print new http request! return ["hello world"]def handle_tcp(socket, address): print new tcp connection! while True: socket.send(hellon) Service gevent.sleep(1) Compositionapp = Service()app.add_service(StreamServer((127.0.0.1, 1234),handle_tcp))app.add_service(WSGIServer((127.0.0.1, 8080), handle_http))app.serve_forever()

Incoming Requests Load Balancer Async Server Async Server ... Async ServerUsing our async reactor-basedapproach let’s redesign our servinginfrastructure

Incoming Requests Load Balancer AAA ... AAA AAA Async Async Async Server Server ServerStep 1: deﬁne an authentication andauthorization layer that will identifythe user and the resource beingrequested

Incoming Requests Load Balancer AAA ... AAA AAA Throttling Throttling Throttling Async Async Async Concurrency Server Server Server ManagerStep 2: add a throttling layer andconcurrency manager

Concurrency Admission Control• Goal: limit concurrency by delaying or selectively failing requests• Common metrics- By Account- By Resource Type- By Availability of Dependent Resources• What we’ve found useful- By (Account, Resource Type)

Delay - delay responses without failingrequests LoadLatency

Deny - deny requests based on resource usage LoadLatency /x FailLatency /*

Incoming Requests Load Balancer AAA ... AAA AAA Throttling Throttling Throttling App App App Concurrency Server Server Server Manager Throttling Throttling ThrottlingStep 3: allow backend resources tothrottle requests Dependent Services

SummaryAsync frameworks like gevent allow youto easily decouple a request from accessto constrained resources Service-wide FailureRequestLatency Time

Don’t Fail Requests DecreasePerformance

Asynchronous Architectures for Implementing Scalable Cloud Services - Evan Cooke - Gluecon 2012

by Twilio on Jun 09, 2012

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 11

Statistics

Asynchronous Architectures for Implementing Scalable Cloud Services - Evan Cooke - Gluecon 2012 Presentation Transcript

http://us-w1.rockmelt.com	6
http://www.twylah.com	5