1. Wednesday, March 6, 13

2. Applying Evolutionary
Architecture on a
Popular API
phil calçado
@pcalcado
http://philcalcado.com
QCon London Session Code: 1603
Wednesday, March 6, 13

3. Wednesday, March 6, 13

4. Wednesday, March 6, 13

5. > 50K registered
client apps
~ 180 million users / month
~5 years old
Wednesday, March 6, 13

6. Wednesday, March 6, 13

7. API controllers UI controllers
domain logic
Wednesday, March 6, 13

8. http://bit.ly/building-the-next-soundcloud
Wednesday, March 6, 13

9. API controllers UI controllers
domain logic
Wednesday, March 6, 13

10. UI controllers
internet
API controllers
domain logic
Wednesday, March 6, 13

11. a new platform while
keeping the old one
running?
Wednesday, March 6, 13

12. WARNING:
THIS IS
WORK IN PROGRESS
and you can help us ;)
http://bit.ly/work-at-soundcloud
Wednesday, March 6, 13

13. DO NOT BREAK
CLIENT APPS
Wednesday, March 6, 13

14. separate
application
from domain
Wednesday, March 6, 13

15. Classic Next
SoundCloud SoundCloud
A set is a A set is a
collection of collection of
your own your own
sounds sounds
Wednesday, March 6, 13

16. Classic Next
SoundCloud SoundCloud
A set is a A set is a
collection of collection of
your own your own
sounds sounds
SoundCloud
API
A set is a
collection of
sounds
Wednesday, March 6, 13

17. UI controllers
internet
API controllers
domain logic
Wednesday, March 6, 13

18. UI controllers
internet
API controllers
domain logic domain logic domain logic
Wednesday, March 6, 13

19. applications enforce their
own business rules
domain enforces “global”
rules
Wednesday, March 6, 13

20. applications enforce their
own business rules
domain enforces “global”
rules
(accidents happen...)
Wednesday, March 6, 13

21. how to add/change
features on the fly?
Wednesday, March 6, 13

22. attempt #1:
identify “official” apps
- too easy to spoof
- abused by spammers
- Android and iOS can’t easily
rotate secrets
Wednesday, March 6, 13

23. attempt #2:
namespace endpoints
- “experimental” endpoints
would link to regular ones but
never be linked back
- toyed with “lifted”
hyperlinks, not enough time
for launch
Wednesday, March 6, 13

24. attempt #3:
just use feature flags
- works ok for logged in users
- system is a little monster
now, and ruby is not very
good with numbers. rewriting
in scala
Wednesday, March 6, 13

25. SPLITTING THE
MOTHERSHIP
Wednesday, March 6, 13

26. define internal protocols
Wednesday, March 6, 13

27. micro-services
- couple thousands lines of code
- mostly JVM-based, JRuby, Clojure
and Scala
- more about this at the SoundCoud
pub night tomorrow: http://
lanyrd.com/2013/soundcloudpub/
(Walrus Pub Waterloo)
Wednesday, March 6, 13

28. HTTP+JSON & AMQP
for most stuff
- moving AMQP usage for
event broadcasting, as
opposed to messaging
- no internal HTTP caching
(yet?)
Wednesday, March 6, 13

29. make it easy to get stuff in
prod
- heroku-like deployment
system
- tending towards twitter
stack (Finagle) for I/O
Wednesday, March 6, 13

30. URNs, not IDs
soundcloud:sounds:123
Wednesday, March 6, 13

31. URNs, not IDs
- everything addressable has
a URN, some have URLs
- Mapping depends on
context, e.g. given object may
have a public URL and an
internal URL
Wednesday, March 6, 13

32. <3 hypermedia
- can easily remove things
from the main app without
breaking clients, just some
HTTP 303s
Wednesday, March 6, 13

33. OPTIMISE ALL
THE THINGS!!11!
Wednesday, March 6, 13

34. twitter moving back to
server rendering...?
Wednesday, March 6, 13

35. not too bad, actually
- besides load balancers vs.
increase in number of
requests per page
http://backstage.soundcloud.com/2012/08/evolution-of-soundclouds-
architecture/
Wednesday, March 6, 13

36. the actual problems
- number of requests from
Next keeps increasing, page
load affected
- very different behaviour
depending on the client app
Wednesday, March 6, 13

37. UI controllers
internet
API controllers
domain logic domain logic domain logic
Wednesday, March 6, 13

38. UI controllers
internet
API controllers
Wednesday, March 6, 13

39. attempt #1:
optmising proxy for clients
UI controllers
internet
Proxy
API controllers
domain logic domain logic domain logic
Wednesday, March 6, 13

40. attempt #1:
optmising proxy for clients
- still leverage public API
- API has to know about how
things behave in different
clients
Wednesday, March 6, 13

41. attempt #2:
specialised back-end for official
clients
UI controllers
internet
Specialised
API controllers
Back-end
domain logic domain logic domain logic
Wednesday, March 6, 13

42. attempt #2:
specialised back-end for official
clients
- each official app has its own back-
end facade
- no rendering, just data
- public API becomes just one of
them
- front-end heavy teams can just
script a common core using JRuby
Wednesday, March 6, 13

43. NEXT STEPS
Wednesday, March 6, 13

44. what we are looking at
- dataflow programming for the
specialised back-ends
- ProtocolBuffers for Android
- make the mothership just a back-
end service, not exposed to the
Internet
Wednesday, March 6, 13

45. phil calçado
http://philcalcado.com
@pcalcado
www.soundcloud.com
Wednesday, March 6, 13