Applying Evolutionary Architecture on a Popular API

Wednesday, March 6, 13

Applying Evolutionary Architecture on a Popular API phil calçado @pcalcado http://philcalcado.com QCon London Session Code: 1603Wednesday, March 6, 13

> 50K registered client apps ~ 180 million users / month ~5 years oldWednesday, March 6, 13

API controllers UI controllers domain logicWednesday, March 6, 13

http://bit.ly/building-the-next-soundcloudWednesday, March 6, 13

API controllers UI controllers domain logicWednesday, March 6, 13

UI controllers internet API controllers domain logicWednesday, March 6, 13

a new platform while keeping the old one running?Wednesday, March 6, 13

WARNING: THIS IS WORK IN PROGRESS and you can help us ;) http://bit.ly/work-at-soundcloudWednesday, March 6, 13

DO NOT BREAK CLIENT APPSWednesday, March 6, 13

separate application from domainWednesday, March 6, 13

Classic Next SoundCloud SoundCloud A set is a A set is a collection of collection of your own your own sounds soundsWednesday, March 6, 13

Classic Next SoundCloud SoundCloud A set is a A set is a collection of collection of your own your own sounds sounds SoundCloud API A set is a collection of soundsWednesday, March 6, 13

UI controllers internet API controllers domain logicWednesday, March 6, 13

UI controllers internet API controllers domain logic domain logic domain logicWednesday, March 6, 13

applications enforce their own business rules domain enforces “global” rulesWednesday, March 6, 13

applications enforce their own business rules domain enforces “global” rules (accidents happen...)Wednesday, March 6, 13

how to add/change features on the fly?Wednesday, March 6, 13

attempt #1: identify “official” apps - too easy to spoof - abused by spammers - Android and iOS can’t easily rotate secretsWednesday, March 6, 13

attempt #2: namespace endpoints - “experimental” endpoints would link to regular ones but never be linked back - toyed with “lifted” hyperlinks, not enough time for launchWednesday, March 6, 13

attempt #3: just use feature flags - works ok for logged in users - system is a little monster now, and ruby is not very good with numbers. rewriting in scalaWednesday, March 6, 13

SPLITTING THE MOTHERSHIPWednesday, March 6, 13

define internal protocolsWednesday, March 6, 13

micro-services - couple thousands lines of code - mostly JVM-based, JRuby, Clojure and Scala - more about this at the SoundCoud pub night tomorrow: http:// lanyrd.com/2013/soundcloudpub/ (Walrus Pub Waterloo)Wednesday, March 6, 13

HTTP+JSON & AMQP for most stuff - moving AMQP usage for event broadcasting, as opposed to messaging - no internal HTTP caching (yet?)Wednesday, March 6, 13

make it easy to get stuff in prod - heroku-like deployment system - tending towards twitter stack (Finagle) for I/OWednesday, March 6, 13

URNs, not IDs soundcloud:sounds:123Wednesday, March 6, 13

URNs, not IDs - everything addressable has a URN, some have URLs - Mapping depends on context, e.g. given object may have a public URL and an internal URLWednesday, March 6, 13

<3 hypermedia - can easily remove things from the main app without breaking clients, just some HTTP 303sWednesday, March 6, 13

OPTIMISE ALL THE THINGS!!11!Wednesday, March 6, 13

twitter moving back to server rendering...?Wednesday, March 6, 13

not too bad, actually - besides load balancers vs. increase in number of requests per page http://backstage.soundcloud.com/2012/08/evolution-of-soundclouds- architecture/Wednesday, March 6, 13

the actual problems - number of requests from Next keeps increasing, page load affected - very different behaviour depending on the client appWednesday, March 6, 13

UI controllers internet API controllers domain logic domain logic domain logicWednesday, March 6, 13

UI controllers internet API controllersWednesday, March 6, 13

attempt #1: optmising proxy for clients UI controllers internet Proxy API controllers domain logic domain logic domain logicWednesday, March 6, 13

attempt #1: optmising proxy for clients - still leverage public API - API has to know about how things behave in different clientsWednesday, March 6, 13

attempt #2: specialised back-end for official clients UI controllers internet Specialised API controllers Back-end domain logic domain logic domain logicWednesday, March 6, 13

attempt #2: specialised back-end for official clients - each official app has its own back- end facade - no rendering, just data - public API becomes just one of them - front-end heavy teams can just script a common core using JRubyWednesday, March 6, 13

NEXT STEPSWednesday, March 6, 13

what we are looking at - dataflow programming for the specialised back-ends - ProtocolBuffers for Android - make the mothership just a back- end service, not exposed to the InternetWednesday, March 6, 13

phil calçado http://philcalcado.com @pcalcado www.soundcloud.comWednesday, March 6, 13

http://crypticmantra.com	224
http://www.tien-chang.com	98
https://twitter.com	56
http://eventifier.co	32
http://flashattackcrew.blogspot.com	1
http://www.onlydoo.com	1
http://www.tiktest.com	1

Applying Evolutionary Architecture on a Popular API

by Phil Calçado on Mar 06, 2013

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 413

Statistics

Applying Evolutionary Architecture on a Popular API Presentation Transcript