This document discusses using reactive programming with Scala and Akka to build distributed, concurrent systems. It describes using the actor model and message passing between actors to develop scalable and resilient applications. Key points covered include using actors to build a web scraping system, handling failures through supervision strategies, and testing actor systems.

1. © 2014 VMware Inc. All rights reserved.
Scale Up Your Thinking
Reactive Programming with Scala and Akka
Lior Shapsa
Yardena Meymann
February 24, 2014

2. About us
• Lior Shapsa – Leads Benchmark Development (lshapsa@vmware.com)
• Yardena Meymann - Group Architect (ymeymann@vmware.com)
2

3. The Project
• New Generation Business Management Products
• Manage Financial Aspects of the Cloud
• Help CFOs to optimize the cloud cost
• Benchmarking – Compare to Industry Best Practices
3

4. What Do We Need
• Web Scraping – naïve way isn’t good enough
• Crowd Sourcing
• Analyze Data
– Build a reliable price list
– How efficient is my virtual infra vs. industry? (cost, capacity)
– How fast am I responding to business challenges
• Start With Single Instance and Scale Out Fast!

5. Why to Scale Out
Moore’s Law
• The Clock Speed Has
Reached Its Limit in 2006
• Free Lunch Is Over

6. Why to Scale Out
Amdahl’s Law
• The New Reality
http://en.wikipedia.org/wiki/Amdahl's_law

7. The Challenge
Shared mutable state -> race conditions
Threads are
expensive
Locks are hard
(and expensive)

8. The Challenge
8 Fallacies of Distributed Computing:
– The network is reliable
– Latency is zero
– Bandwidth is infinite
– The network is secure
– Topology doesn't change
– There is one administrator
– Transport cost is zero
– The network is homogeneous

9. Reactive Programming
• React to Events
– Pushing rather than pulling (No Shared State)
• React to Load
– Focus on scalability rather than single-user performance.
• React to Failure
– Resilient systems with the ability to recover quickly at all levels.
• React to Users
– Combine the above traits for an interactive user experience
9

10. Chosen Solution: Actor Model
• Formalized in 1973 by Carl Hewitt and refined by
Gul Agha in mid 80s.
– The first major adoption is done by Ericsson in mid 80s.
• Invented Erlang and later open-sourced in 90s.
• Built a distributed, concurrent, and fault-tolerant
telecom system which has 99.9999999% uptime

11. Chosen Solution: Actor Model
• Actor Instead of Object, Asynch Message Instead of Method Call
• Keep Mutable State Internal, Share Nothing
• Communicate via Immutable Message Passing
• Messages are Kept in Mailbox and Processed Sequentially
• Asynchronous and Non-blocking
Actor Actor
Mailbox
state
Mailbox behavior
behavior state

12. Actor Example (Scala)
• Define
case class Greeting(who: String) //message
class GreetingActor extends Actor {
def receive = {
case Greeting(who) => sender ! “Hello ” + who
}
}
• Create
val sys = ActorSystem("MySystem")
val greeter = sys.actorOf(Props[GreetingActor], "greeter")
• Send
greeter ! Greeting(“Akka”);

13. Actor Libs For the JVM
• Akka (Java/Scala)
• Kilim (Java)
• Jetlang (Java)
• Actor’s Guild (Java)
• Actorom (Java)
• FunctionalJava (Java)
• GPar (Groovy)


14. Web Scraping - Example

15. Dispatch
Crawling
library
Netty
Web Scraping Actors Design
Curiosity
Site …
Pages
Level1 Level1
Level2
…
Level2
File
writer
Site
…
k/v

16. Example Site
class Servers extends Site {
override def start(): Unit = goto[ListPage] “…”
}
class ListPage extends Page {
override def processResult(result: Response): Unit = {
val links = listProductLinks(result.content)
links foreach goto[ProductPage](_, pageType = Child)
val next = getNextPage(result.content)
next foreach goto[ListPage](_, pageType = Sibling)
}
}
class ProductPage extends Page {
override def processResult(result: Response): Unit = {
val spec = productTitle(result.content) ++
generalSpec(result.content) ++
pricingInfo(result.content)
writeToFile(spec)
}
}

17. Supervision
• Let It Crash!
– Errors are encapsulated as messages and isolated from the BL
– The caller is not necessarily available
• The Supervisor Reacts Upon Failure
– Resume, keeping its accumulated internal state
– Restart, clearing out its accumulated internal state
– Terminate
– Escalate

18. Supervision – Example (optional)
override val supervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 10,
withinTimeRange = 1 minute) {
case _: ArithmeticException => Resume
case _: NullPointerException => Restart
case _: Exception => Escalate
}

19. Akka Distributable by Design
• Superior model for detecting and recovering from errors
• Location transparent - scale UP and OUT for free
• Adaptive load-balancing, cluster rebalancing & actor
migration
• Build extremely loosely coupled and dynamic systems
that can change and adapt at runtime

20. Under the hood

21. Receive Method
• Pattern match on the message
class Site… extends Actor … {
override def receive: Receive = {
case Start =>
log.info(s"Starting $siteName ...")
start()
…
}
• Defined for some (but not necessarily all) messages
• Unhandled messages are sent to a special location

22. Splitting Behavior Between Methods
• Group related messages in a receive function
• Compose receive functions with orElse operator
private def process: Receive = {
case res: Response => …
case Status.Failure(e) => …
}
private def monitor: Receive = {
case Create… => …
case Terminated(actor) => …
Curiosity
Site …
Crawler
Crawling
library
}
protected override def receive: Receive = process orElse monitor
Netty
Pages
Page
Page
Page
…
Page
File writer
Site
…

23. Using Traits to Compose a Behavior
trait Percolator { myself: Actor =>
protected def percolate: Receive = …
}
trait Composite { myself: Actor =>
protected def monitor: Receive = …
}
class Page extends Actor with Percolator with Composite {
private def process: Receive = …
protected override def receive =
process orElse percolate orElse monitor
}
Composite Percolator
Actor
Page
c
t t
c

24. Web Scraping Actors Design
Curiosity
Dispatch
Crawling
library
Netty
Site …
Pages
Level1 Level1
Level2
…
Level2
File
writer
Site
…

25. Propagate Messages
You can forward messages without understanding them, “methodMissing”
trait Percolator { myself: Actor =>
protected def percolate: Receive = {
case _ => parent forward msg
}
class Site… extends Actor {
protected def override receive: Receive = {
case msg: Goto => crawler forward msg
case msg: WriteFile => writer forward msg
}

26. Futures
• Future holds a value which may become available at some point
– completed when the computation results in a value or an exception
• java.util.concurrent has futures, but we can do better…
• Scala’s futures are asynchronously compose-able
– Callbacks: onSuccess, onFailure, onComplete
val f: Future[List[String]] = future { session.getRecentPosts }
f onFailure {
case e => println("An error has occurred: " + e.getMessage)
}
f onSuccess {
case posts => for (post <- posts) println(post)
}
• More and more API’s are non-blocking - returning futures

27. Tell vs Ask
• Tell, a.k.a. !
– Fire & forget
– Best concurrency and scalability
• Ask, a.k.a. ?
– sends and (optionally) wait for reply
– uses delayed computation (Futures)
– combining futures and actors – pipeTo trick
import akka.pattern.pipe
val page: ActorRef = …
val f: Future[xml.NodeSeq] = Http.browse(url(…) OK as.tagsoup.NodeSeq)
f pipeTo page
• Why we decided to stick with “bare actors”

28. A Closer Look at Akka Actor
Reference
Instance
Instance
factory
Context
Mailbox
Receive function
Receive function
Behavior

29. Actor State
It’s Ok to have mutable state inside actor
trait Composite { myself: Actor =>
private var childrenCount = 0
override def receive: Receive = {
case Created… =>
childrenCount = childrenCount + 1
case Terminated… =>
childrenCount = childrenCount – 1
if (childrenCount == 0) context stop self
}

30. Lightweight State-Machine with become
class Curiosity extends Actor {
private def free: Receive = {
case Start =>
crawlers foreach (_ ! Start)
context become busy
case Status | Stop => sender ! Free
}
private def busy: Receive = {
case Stop =>
crawlers foreach (_ ! Stop)
context become awaitingTermination
case Status | Start => sender ! Busy
case AllFinished => context become free
}
private def awaitingTermination: Receive = {
case AllFinished => context become free
case Status | Start | Stop => sender ! Busy
}
override def receive: Receive = free
}
Free
Busy
Start
Awaiting
Termination
Stop
Finished
Finished

31. Actor Hierarchy DO’s and DON’Ts
• DO give actors meaningful names if you can
• DON’T be afraid to create lots of short-living actors
• DON’T nest actor classes in one another
• DO note that restart that results from failure is not the same as a
graceful stop
• DON’T define public methods on Actors
• DO choose supervision strategy wisely, build your hierarchy according
to the strategy
• DON’T confuse actor.Status.Failure with util.Failure
• DO make sure messages are immutable
• DON’T rely on using generics in messages, as they will be erased in
bytecode (unless you reify with ClassTag/TypeTag)

32. Stopping Actors
• Watching another actor – Death Watch
– Not same as supervision
– Use context.watch(actorReference)
– Terminated(actorReference) events
• Stop vs. PoisonPill and how we used both
– context.stop – terminate after current message
– context.kill – terminate now
– PoisonPill message – terminate after processing all messages

33. Web Scraping - Termination
Curiosity
Dispatch
Crawling
library
Netty
Site …
Pages
Page Page
Page
…
Page
File
writer
Site
…
k/v k/v k/v

34. ETL – Decrypt Translate & Load (optional)
34
Decrypt Load Router Router
Load Manager
Decrypt
Worker
Translate
Worker
Archiver
Mongo
Loader
Decrypt Manager
Translate
WTorarknesrlate
Worker
Decrypt
WDorekcerrypt
Worker
Directory
Watcher
Decrypt
WDorekcerrypt
Worker
Directory
Watcher

35. Exposing Actors with Spray
• Spray – lightweight HTTP server, non-blocking, 100% Akka
• Making your actors respond to HTTP requests
import akka.io.IO
import spray.can.Http
import spray.routing.HttpService
val Handler = new Actor with HttpService {
def actorRefFactory = context
def receive = runRoute {
path("start") { … { crawlers ! Start ; …}} ~
path("stop") { … { crawlers ! Stop ; …}} ~
…
}
}
IO(Http)(system) ! Http.Bind(handler, interface = … , port = …)

36. Testing Akka
• ScalaTest – BDD (Given-When-Then)
• ScalaMock & Mockito
• Measuring Scala code coverage with sbt (scct + cobertura + some
tricks)
• Using Akka TestKit + Spray
• ScalaCheck

37. Testing Akka - Example
import org.scalatest._
import akka.testkit.{ImplicitSender , TestActorRef, TestKit}
class CuriosityTest extends TestKit(new ActorSystem(“…”) … {
val myTestedActor = TestActorRef[Curiosity]
“Curiosity" must {
"display free status when not working" in {
myTestedActor ! Status
expectMsg(Free)
}
}
…
}

38. Testing Akka – Running Tests

39. Debugging Akka
• Typesafe console
• Logs, watching for DeadLetter
• Using Scala clone of Yammer Metrics

40. Typesafe Console

41. Q&A

42. Our Amazing Team
If you want to join, find us at http://vmware.jobs/

43. Thank You!