Demonstration of a web crawler implemented using Akka. Link to the sources coming soon.

1. Florian Hopf
http://blog.florian-hopf.de

2. Download
Page
Link
downloaded
already?
Parse Links
& Content
Index
Content

3. Sequential
Example

4. 1. Scale Up

5. Thread A Page Thread B
Store

6. Synchronize
Thread A Page Thread B
and Suffer
Store

11. Actor-Framework

12. Message Passing
Concurrency

13. Thread A Thread B
Actor A Actor B

14. Master
Page
Fetch Store Index
Page Page
Page
Indexed
Page Content
Indexing
Parsing
Actor
Actor

15. Simple Example

16. Actors are accessed
by references

17. Reference
Actor
Instance

18. Reference
Actor
Actor
n
Instance
Instance
Actor
Instances

19. Fetching Pages
in parallel

20. 2. Scale Out

21. References can point
to remote machines

22. akka {
actor {
deployment {
/serviceA/aggregation {
router = "round­robin"
nr­of­instances = 10
target {
nodes =
["akka://app@10.0.0.2:2552",
"akka://app@10.0.0.3:2552"]
}
}
}
}
}

23. 3. Fault Tolerance

24. Actors are supervised
by their parent

25. Master
Page
Store
Page
Indexing
Parsing
Actor
Actor

26. Reference
Actor
Instance

27. Master
Page
Store
Page
Indexing
Parsing
Actor
Actor

28. Reference
Actor
Instance

29. Actions:
Escalate
Resume
Restart
Stop

30. Network fail on download

31. Master
Page
Store
Page
Indexing
Parsing
Actor
Actor

32. Resume
PageParsingActor,
resend message

33. Indexing Failure, I/O Error

34. Master
Page
Store
Page
Indexing
Parsing
Actor
Actor

35. Restart Master

36. Scaling Up
Scaling Out
Fault Tolerance

37. Software Transactional
Memory
Durable Mailboxes

38. http://akka.io