A presentation I gave at Vancouver's Scala Meetup in February 2015 documenting a modernization effort from PHP/Gearman to Scala/Akka/async-http-client. Abstract: Hootsuite issues millions of requests to social networks on behalf of its users on a daily basis. These requests incur network latency costs which can accumulate rapidly and affect user experience and operations in general. This talk will review the modernization of Hootsuite's infrastructure for automated publishing via RSS/Atom feeds and how Scala, Akka, and Ning's asynchronous HTTP client were combined to effectively handle network latency. It will explore the motivations behind this effort, the approaches used, the benefits, and the lessons learned from developing and deploying this reactive service.

2. Scala + Akka
+
ning/async-http-client
Or: How I stopped worrying about network
latency and drank the reactive cool-aid

3. Intro
$ whoami

4. RSS is dead, long live RSS!
● In constant decline over the years as
consumer-facing tech
● Still very popular between “content services”
● RSS/Atom feeds @ HS
o Follow feeds on behalf of users
o Post to social networks based on content
o ~250K active feeds, ~400K messages daily

5. Scaling webfeeds using PHP

6. But latency isn’t the problem; blocking is.

7. Enter Scala + Akka + async HTTP client

8. Akka: a short detour for the uninitiated
“Toolkit and runtime for building highly
concurrent, distributed, and fault tolerant
applications on the JVM”
● Implementation of the Actor model
● Scala, Java
● Adheres strictly to the Reactive Manifesto:
o Responsive, Resilient, Elastic, Message-driven

9. Akka (continued)
Source: http://www.scottlogic.com/blog/rdoyle/assets/ActorModel.png

10. Ning’s async-http-client

11. … in an actor

12. Actors in Scala webfeed service

13. So, what did this buy us?

14. Now
BeforeBefore ● 120 processes
● 6 X m3.xlarge
● 2 processes
● 2 X m3.xlarge

15. 90 webfeed/sec 300 webfeed/sec

16. CPU utilization & load average - barely breaking a sweat!

17. Some lessons learned
● You can't ignore back pressure
o Pull-based approach
o Akka Streams to the rescue?
● scala.xml.pull.XMLEventReader is not
production ready
● Be weary of the resulting load on the rest of
your infrastructure!

18. Some lessons learned (continued)
● Actors should only ever do one thing (and do
it well)
o Easy to modify the actor topology (and it will
happen!)
o Easier to describe supervisor’s responsibilities
● Let it crash!

19. Thanks!
http://akka.io/
https://github.com/AsyncHttpClient/async-http-client
We’re hiring!
https://hootsuite.com/about/careers
yanik.berube@hootsuite.com
@yanik_berube