1. Konrad `@ktosopl` Malawski
need for async
hot pursuit
for Scalable apps

2. Konrad `ktoso` Malawski
Akka Team,
Reactive Streams TCK
(we’re renaming soon!)

3. Konrad `@ktosopl` Malawski
akka.io
typesafe.com
geecon.org
Java.pl / KrakowScala.pl
sckrk.com / meetup.com/Paper-Cup @ London
GDGKrakow.pl
lambdakrk.pl
(we’re renaming soon!)

4. Nice to meet you!
Who are you guys?

5. High Performance Software Development
For the majority of the time,
high performance software development
is not about compiler hacks and bit twiddling.
It is about fundamental design principles that are
key to doing any effective software development.
Martin Thompson
practicalperformanceanalyst.com/2015/02/17/getting-to-know-martin-thompson-...

6. Agenda
• Why?
• Async and Synch basics / definitions
• Async where it matters: Scheduling
• How NOT to measure Latency
• Concurrent < lock-free < wait-free
• I/O: IO, AIO, NIO, Zero
• C10K: select, poll, epoll / kqueue
• Distributed Systems: Where Async is at Home
• Wrapping up and Q/A

7. Why?

8. Why this talk?
“The free lunch is over”
Herb Sutter
(A Fundamental Turn Toward Concurrency in Software)

9. Why this talk?
“The free lunch is over”
Herb Sutter
(A Fundamental Turn Toward Concurrency in Software)
How and why
Asynchronous Processing
comes into play for
Scalability and Performance.

10. Why this talk?
“The free lunch is over”
Herb Sutter
(A Fundamental Turn Toward Concurrency in Software)
How and why
Asynchronous Processing
comes into play for
Scalability and Performance.
I want you guys make
well informed decisions,
not buzzword driven development.

11. Sync / Async Basics

12. Sync / Async

13. Sync / Async

14. Sync / Async

15. Sync / Async

16. Sync / Async

17. Sync / Async

18. Sync / Async

19. Sync / Async

20. Sync / Async

21. Highly parallel systems
Event loops
Actors
Async where it matters:

22. Async where it matters: Scheduling

23. Async where it matters: Scheduling

24. Async where it matters: Scheduling

25. Async where it matters: Scheduling

26. Async where it matters: Scheduling

27. Async where it matters: Scheduling

28. Async where it matters: Scheduling

29. Async where it matters: Scheduling

30. Async where it matters: Scheduling

31. Scheduling (notice they grey sync call)

32. Scheduling (notice they grey sync call)

33. Scheduling (now with Async db call)

34. Scheduling (now with Async db call)

35. Scheduling (now with Async db call)

36. Scheduling (now with Async db call)

37. Lesson learned:
Blocking is indistinguishable from being very slow.

38. Latency

39. Latency Quiz #1: Which one is Latency?
By the queueing theory definitions:

40. Latency Quiz #1: Which one is Latency?
By the queueing theory definitions:

41. Latency Quiz #1: Which one is Latency?
By the queueing theory definitions:

42. Latency Quiz #1: Which one is Latency?
By the queueing theory definitions:

43. Latency Quiz #1: Which one is Latency?

44. Latency Quiz #1: Which one is Latency?
Sometimes devs use Latency and Response Time as the same thing.
That’s OK, as long as both sides know which one they are talking about.

45. Latency Quiz #2
Gil Tene style, see:“How NOT to Measure Latency”
Is 10s latency acceptable in your app?
Is 200ms latency acceptable?
How about most responses within 200ms?
So mostly 20ms and some 1 minute latencies is OK?
Do people die when we go above 200ms?
So 90% below 200ms, 99% bellow 1s, 99.99% below 2s?

46. Latency in the “real world”
Gil Tene style, see:“How NOT to Measure Latency”
“Our response time is 200ms average,
stddev is around 60ms”
— a typical quote

47. Latency in the “real world”
Gil Tene style, see:“How NOT to Measure Latency”
“Our response time is 200ms average,
stddev is around 60ms”
— a typical quote
Latency does NOT behave like normal distribution!
“So yeah, our 99,99%’ is…”

48. http://hdrhistogram.github.io/HdrHistogram/
Hiccups

49. Gil Tene style, see:“How NOT to Measure Latency”
Hiccups

50. Gil Tene style, see:“How NOT to Measure Latency”
Hiccups

51. Gil Tene style, see:“How NOT to Measure Latency”
Hiccups

52. Gil Tene style, see:“How NOT to Measure Latency”
Hiccups

53. Gil Tene style, see:“How NOT to Measure Latency”
Hiccups

54. Lesson learned:
Use precise language when talking about latencies.
Measure the right thing!
“Trust no-one, bench everything!”
Verify the results!
Ask on groups or forums.

55. Concurrent < lock-free < wait-free

56. Concurrent < lock-free < wait-free

57. Concurrent < lock-free < wait-free

58. Concurrent < lock-free < wait-free
“concurrent” data structure

59. Concurrent < lock-free < wait-free
What can happen in concurrent data structures:
A tries to write; B tries to write; B wins!
A tries to write; C tries to write; C wins!
A tries to write; D tries to write; D wins!
A tries to write; B tries to write; B wins!
A tries to write; E tries to write; E wins!
A tries to write; F tries to write; F wins!
…
Moral?
1) Thread A is a complete loser.
2) Thread A may never make progress.

60. What can happen in concurrent data structures:
A tries to write; B tries to write; B wins!
C tries to write; C wins!
D tries to write; D wins!
B tries to write; B wins!
E tries to write; E wins!
F tries to write; F wins!
…
Moral?
1) Thread A is a complete loser.
2) Thread A may never make progress.
Concurrent < lock-free < wait-free

61. Concurrent < lock-free < wait-less
Remember: Concurrency is NOT Parallelism.
Rob Pike - Concurrency is NOT Parallelism (video)
def offer(a: A): Boolean // returns on failure
def add(a: A): Unit // throws on failure
def put(a: A): Boolean // blocks until able to enqueue

62. Concurrent < lock-free < wait-free
concurrent data structure
<
lock-free* data structure
* lock-free a.k.a. lockless

63. What lock-free programming looks like:

64. An algorithm is lock-free if it satisfies that:
When the program threads are run sufficiently long,
at least one of the threads makes progress.
Concurrent < lock-free < wait-free

65. * Both versions are used: lock-free / lockless
class CASBackedQueue[A] {
val _queue = new AtomicReference(Vector[A]())
// does not block, may spin though
@tailrec final def put(a: A): Unit = {
val queue = _queue.get
val appended = queue :+ a
if (!_queue.compareAndSet(queue, appended))
put(a)
}
}
Concurrent < lock-free < wait-free

66. class CASBackedQueue[A] {
val _queue = new AtomicReference(Vector[A]())
// does not block, may spin though
@tailrec final def put(a: A): Unit = {
val queue = _queue.get
val appended = queue :+ a
if (!_queue.compareAndSet(queue, appended))
put(a)
}
}
Concurrent < lock-free < wait-free

67. class CASBackedQueue[A] {
val _queue = new AtomicReference(Vector[A]())
// does not block, may spin though
@tailrec final def put(a: A): Unit = {
val queue = _queue.get
val appended = queue :+ a
if (!_queue.compareAndSet(queue, appended))
put(a)
}
}
Concurrent < lock-free < wait-free

68. Concurrent < lock-free < wait-free

69. Concurrent < lock-free < wait-free

70. Concurrent < lock-free < wait-free

71. Concurrent < lock-free < wait-free

72. Concurrent < lock-free < wait-free

73. Concurrent < lock-free < wait-free

74. Concurrent < lock-free < wait-free
“concurrent” data structure
<
lock-free* data structure
<
wait-free data structure
* Both versions are used: lock-free / lockless

75. Concurrent < lock-free < wait-free
Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms
Maged M. Michael Michael L. Scott
An algorithm is wait-free if every operation has a bound on
the number of steps the algorithm will take before the
operation completes.

76. wait-free: j.u.c.ConcurrentLinkedQueue
Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms
Maged M. Michael Michael L. Scott
public boolean offer(E e) {
checkNotNull(e);
final Node<E> newNode = new Node<E>(e);
for (Node<E> t = tail, p = t;;) {
Node<E> q = p.next;
if (q == null) {
// p is last node
if (p.casNext(null, newNode)) {
// Successful CAS is the linearization point
// for e to become an element of this queue,
// and for newNode to become "live".
if (p != t) // hop two nodes at a time
casTail(t, newNode); // Failure is OK.
return true;
}
// Lost CAS race to another thread; re-read next
}
else if (p == q)
// We have fallen off list. If tail is unchanged, it
// will also be off-list, in which case we need to
// jump to head, from which all live nodes are always
// reachable. Else the new tail is a better bet.
p = (t != (t = tail)) ? t : head;
else
// Check for tail updates after two hops.
p = (p != t && t != (t = tail)) ? t : q;
}
}
This is a modification of the Michael & Scott algorithm,
adapted for a garbage-collected environment, with support
for interior node deletion (to support remove(Object)).
For explanation, read the paper.

77. Lesson learned:
Different ways to construct concurrent data structures.
Pick the right impl for the right job.

78. I / O

79. IO / AIO

80. IO / AIO / NIO

81. IO / AIO / NIO / Zero

82. Synchronous I / O
— Havoc Pennington
(HAL, GNOME, GConf, D-BUS, now Typesafe)
When I learned J2EE about 2008 with some of my
desktop colleagues our reactions included something like:
”wtf is this sync IO crap,
where is the main loop?!” :-)

83. Interruption!
CPU: User Mode / Kernel Mode

84. Kernels and CPUs

85. Kernels and CPUs

86. Kernels and CPUs

87. Kernels and CPUs

88. Kernels and CPUs
http://wiki.osdev.org/Context_Switching

89. Kernels and CPUs
[…] switching from user-level to kernel-level
on a (2.8 GHz) P4 is 1348 cycles.
[…] Counting actual time, the P4 takes 481ns […]
http://wiki.osdev.org/Context_Switching

90. I / O

91. I / O

92. I / O

93. I / O

94. I / O

95. I / O
“Don’t worry.
It only gets worse!”

96. I / O
“Don’t worry.
It only gets worse!”
Same data in 3 buffers!4 mode switches!

97. Asynchronous I / O [Linux]
Linux AIO = JVM NIO

98. Asynchronous I / O [Linux]
NewIO… since 2004!
(No-one calls it “new” any more)
Linux AIO = JVM NIO

99. Asynchronous I / O [Linux]
Less time wasted waiting.
Same amount of buffer copies.

100. ZeroCopy = sendfile [Linux]
“Work smarter.
Not harder.”
http://fourhourworkweek.com/

101.