Multipathed, Multiplexed, Multilateral Transport Protocols - Decoupling transport protocols from what's below, by Catherine Pearce. A presentation given at APRICOT 2016’s APOPS Plenary 1 session on 22 February 2016.

1. Catherine (Kate) Pearce
Sr Security Consultant, Cisco Security Services
Feb 22, 2016
Decoupling transport protocols from what's below
Multipathed, Multiplexed, Multilateral Transport Protocols

2. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?

3. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce

4. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce
• @secvalve

5. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce
• @secvalve
• Sr. Security Consultant
(Customer Focused)

6. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce
• @secvalve
• Sr. Security Consultant
(Customer Focused)
• Break & Report

7. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce
• @secvalve
• Sr. Security Consultant
(Customer Focused)
• Break & Report
• Coach the builders

8. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce
• @secvalve
• Sr. Security Consultant
(Customer Focused)
• Break & Report
• Coach the builders
• Research what’s ahead

9. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce
• @secvalve
• Sr. Security Consultant
(Customer Focused)
• Break & Report
• Coach the builders
• Research what’s ahead
• Distinguishing Features:

10. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce
• @secvalve
• Sr. Security Consultant
(Customer Focused)
• Break & Report
• Coach the builders
• Research what’s ahead
• Distinguishing Features:
• Loud, Yellow

11. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce
• @secvalve
• Sr. Security Consultant
(Customer Focused)
• Break & Report
• Coach the builders
• Research what’s ahead
• Distinguishing Features:
• Loud, Yellow
• Or is that “Loud Yellow”?

12. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 2
Who?
• Catherine (Kate) Pearce
• @secvalve
• Sr. Security Consultant
(Customer Focused)
• Break & Report
• Coach the builders
• Research what’s ahead
• Distinguishing Features:
• Loud, Yellow
• Or is that “Loud Yellow”?
ALL OPINIONS ENTIRELY MY OWN.
NO official Cisco representations of any kind

13. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 3
MPTCP changes
fundamental assumptions
about
how TCP works*

14. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 3
MPTCP changes
fundamental assumptions
about
how TCP works*
Use it to break
things today

15. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 3
MPTCP changes
fundamental assumptions
about
how TCP works*
Use it to break
things today
Adapt to it for
tomorrow

16. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 4

17. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 4
QUIC also changes
fundamental assumptions
about
how HTTP works*

18. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 4
QUIC also changes
fundamental assumptions
about
how HTTP works*
Use it to break
things today

19. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 4
QUIC also changes
fundamental assumptions
about
how HTTP works*
Use it to break
things today
Adapt to it for
tomorrow

20. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 5
2 Simple Examples: #1

21. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 5
2 Simple Examples: #1

22. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 6
2 Simple Examples: #1

23. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 7
2 Simple Examples: #1

24. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 7
2 Simple Examples: #1
Wait,
What!?!!

25. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 8
2 Simple Examples: #1

26. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 8
2 Simple Examples: #1
Wait,
What!?!!

27. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 9
2 Simple Examples: #2

28. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 10
2 Simple Examples: #2

29. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 10
2 Simple Examples: #2

30. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 10
2 Simple Examples: #2
Err?

31. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 11
What’s going on here?

32. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 11
What’s going on here?
à Let’s talk about upcoming transport
protocols

33. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 12
Introduction ✔
Background (TCP, Why Change it?)
MPTCP
Background Redux (Why NOT to change TCP)
QUIC
Implications
Conclusion and Takeaways

34. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 13
Multiplexing
~ Layer Technique Endpoint Endpoint Definition
1 Uniplex Wire Implicit
1/2 Circuit
Switching
Physical Wire Address Local Network Address
3 Packet
Switching
Routed Logical Address Routable Network Address (Mac/IP)
4 Packet
Switched
Transport
Software Logical Port
(linked to a single logical
network address)
Routable Network Address
+ Transport Protocol
+ Port
4+ Multipath
Networking
Process with a Logical
Connection Identifier.
(Linked to n logical network
addresses + transports)
+ Multipath Protocol Identifier
+ Connection Identifier
(Transport & Network Agnostic – as long as one given)
5 / 6 / 7 N/A URI/URL URI/URL (Transport-agnostic)

35. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 14
Internet Scale Explosion

36. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 14
• Host count
Internet Scale Explosion

37. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 14
• Host count
• Interfaces / host
Internet Scale Explosion

38. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 14
• Host count
• Interfaces / host
• Applications / host
Internet Scale Explosion

39. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 14
• Host count
• Interfaces / host
• Applications / host
• Connections /
application
Internet Scale Explosion

40. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 14
• Host count
• Interfaces / host
• Applications / host
• Connections /
application
Internet Scale Explosion
More Addresses!
(Carrier NAT if that doesn't work)

41. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 14
• Host count
• Interfaces / host
• Applications / host
• Connections /
application
Internet Scale Explosion
More Addresses!
(Carrier NAT if that doesn't work)
More Compute

42. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 14
• Host count
• Interfaces / host
• Applications / host
• Connections /
application
Internet Scale Explosion
More Addresses!
(Carrier NAT if that doesn't work)
More Compute
Workarounds & New
Application protocols

43. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 14
• Host count
• Interfaces / host
• Applications / host
• Connections /
application
Internet Scale Explosion
More Addresses!
(Carrier NAT if that doesn't work)
More Compute
Workarounds & New
Application protocols
?????

44. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 15
Why do you care?

45. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 15
• Familiar Problems
• Address Space Exhaustion
• Route Table Explosion
Why do you care?

46. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 15
• Familiar Problems
• Address Space Exhaustion
• Route Table Explosion
• “New” Problems
• NAT Table Explosion
• Client-Controlled routing - Route Arbitrage and Swarming
Why do you care?

47. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 16
TO BE CLEAR:

48. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 16
TO BE CLEAR:
These technologies are more culture shock
than direct vulnerability / concern

49. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 16
TO BE CLEAR:
These technologies are more culture shock
than direct vulnerability / concern
Personally, I like them, and want them to
succeed

50. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 16
TO BE CLEAR:
These technologies are more culture shock
than direct vulnerability / concern
Personally, I like them, and want them to
succeed
Network tools and operators, need to be ready

51. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Current TCP is rather limited

52. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Doesn’t support use cases for:
Current TCP is rather limited

53. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Doesn’t support use cases for:
• High Availability
Current TCP is rather limited

54. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Doesn’t support use cases for:
• High Availability
• Link Aggregation
Current TCP is rather limited

55. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Doesn’t support use cases for:
• High Availability
• Link Aggregation
• Multihoming
Current TCP is rather limited

56. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Doesn’t support use cases for:
• High Availability
• Link Aggregation
• Multihoming
• Mesh networking
Current TCP is rather limited

57. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Doesn’t support use cases for:
• High Availability
• Link Aggregation
• Multihoming
• Mesh networking
Makes a lot of round trips
Current TCP is rather limited

58. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Doesn’t support use cases for:
• High Availability
• Link Aggregation
• Multihoming
• Mesh networking
Makes a lot of round trips
Blocks stream on retransmits
Current TCP is rather limited

59. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Doesn’t support use cases for:
• High Availability
• Link Aggregation
• Multihoming
• Mesh networking
Makes a lot of round trips
Blocks stream on retransmits
Current TCP is rather limited
MPTCP

60. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 17
Doesn’t support use cases for:
• High Availability
• Link Aggregation
• Multihoming
• Mesh networking
Makes a lot of round trips
Blocks stream on retransmits
Current TCP is rather limited
QUIC
MPTCP

61. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 18
Why is this happening?

62. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 18
Networks need multipath, but there’s more
than one way to do it
Why is this happening?

63. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 18
Networks need multipath, but there’s more
than one way to do it
1. MPTCP Extends TCP to Multiplex
Why is this happening?

64. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 18
Networks need multipath, but there’s more
than one way to do it
1. MPTCP Extends TCP to Multiplex
2. QUIC IGNORES TCP to handle it itself
Why is this happening?

65. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 18
Networks need multipath, but there’s more
than one way to do it
1. MPTCP Extends TCP to Multiplex
2. QUIC IGNORES TCP to handle it itself
But, these technologies change the way the
internet behaves
Why is this happening?

66. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 19
Introduction ✔
Background (Why Change TCP) ✔
MPTCP
Background Redux (Why NOT to change TCP)
QUIC
Implications
Conclusion and Takeaways

67. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 20
Earlier:
Current TCP is rather limited

68. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 20
Doesn’t support use cases for:
• High Availability
• Link Aggregation
• Multihoming
• Mesh networking
Makes a lot of round trips
Earlier:
Current TCP is rather limited

69. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 21
Multipath TCP

70. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 21
Multipath TCP
Multipath TCP is an extension to TCP that adds
the above functionality

71. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 21
Multipath TCP
Multipath TCP is an extension to TCP that adds
the above functionality
AND: it works over existing infrastructure
• (it IS TCP… just more so)

72. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 21
Multipath TCP
Multipath TCP is an extension to TCP that adds
the above functionality
AND: it works over existing infrastructure
• (it IS TCP… just more so)

73. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 21
Multipath TCP
Multipath TCP is an extension to TCP that adds
the above functionality
AND: it works over existing infrastructure
• (it IS TCP… just more so)
BUT: nothing much else understands it

74. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 22
Motivations and Advantages

75. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 22
Motivations and Advantages
• TCP implements connections between

76. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 22
Motivations and Advantages
• TCP implements connections between
IP:PORT & IP:PORT, “without regard to
path”

77. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 22
Motivations and Advantages
• TCP implements connections between
IP:PORT & IP:PORT, “without regard to
path”
• NOT between endpoint A and endpoint B

78. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 22
Motivations and Advantages
• TCP implements connections between
IP:PORT & IP:PORT, “without regard to
path”
• NOT between endpoint A and endpoint B
• In the past this was a distinction without a
difference, but not any more

79. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 23
Riding atop of TCP

80. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 23
Riding atop of TCP
•An MPTCP Connection is defined by a
connection ID

81. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 23
Riding atop of TCP
•An MPTCP Connection is defined by a
connection ID
•It is composed of multiple streams, where each
stream is a regular TCP connection (with an
option strapped on)

82. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 24
MPTCP Characteristics

83. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 24
MPTCP Characteristics
•Backwards compatibility

84. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 24
MPTCP Characteristics
•Backwards compatibility
•Performance >= now

85. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 24
MPTCP Characteristics
•Backwards compatibility
•Performance >= now
•Security >= now

86. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 25
MPTCP – Simple Case
MPTCP connection looks like TCP so
far…
Client Server
Network A

87. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 26
MPTCP – Simple Case
Client Server
Network A

88. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 26
MPTCP – Simple Case
Client Server
Network A
Network B

89. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 26
MPTCP – Simple Case
Client Server
Network A
Network B

90. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 26
MPTCP – Simple Case
N different TCP connections,
contributing to ONE logical data flow
Client Server
Network A
Network B

91. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 26
MPTCP – Simple Case
N different TCP connections,
contributing to ONE logical data flow
Client Server
Network A
Network B

92. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 27
MPTCP – Simple Case
N different TCP connections, contributing to ONE
logical data flow… data flows through any/all
Client Server
Network A
Network B
DATA[0]
DATA[1]
DATA[2]

93. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 28
MPTCP – Simple Case
Client Server
Network A
Network B
DATA[0]
DATA[1]
DATA[2]
Sender of a packet can choose to use
any flow (this will be important)

94. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 29
MPTCP – Simple Case
Any subset of connections can drop,
overall flow continues.
Client Server
Network A
Network B
DATA[3]

95. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 30
MPTCP – Simple Case
Connections can be re-added at any time
Client Server
Network A
Network B

96. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 31
MPTCP – Simple Case
Un-ACK’d data can be quickly resent over a
different flow… first ACK is good enough!
Client Server
Network A
Network B
DATA[4]
ß
DATA[4]
ACK
DATA[4]

97. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 32
MPTCP – Basic Use Cases
Client Server
Network A
Network B
For seamless roaming For high
availability

98. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 33
How is MPTCP implemented? – TCP Option

99. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 34
What does it look like?
•Packet Breakdown - WireShark
• [WireShark / TCPdump Capture
screenshot]

100. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 34
What does it look like?
•Packet Breakdown - WireShark
• [WireShark / TCPdump Capture
screenshot]

101. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 34
What does it look like?
•Packet Breakdown - WireShark
• [WireShark / TCPdump Capture
screenshot]
TCP Options
field

102. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 34
What does it look like?
•Packet Breakdown - WireShark
• [WireShark / TCPdump Capture
screenshot]
TCP Options
field
Option 30
(0x1E)

103. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 35
Path Management - Linux

104. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 35
Path Management - Linux
• The Linux Path Manager has two
primary path managers at present
• Fullmesh – n:n (all to all)
• Ndiffports – 1-1 interfaces, n-1 ports

105. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 35
Path Management - Linux
• The Linux Path Manager has two
primary path managers at present
• Fullmesh – n:n (all to all)
• Ndiffports – 1-1 interfaces, n-1 ports
• This is in the TCP stack… application
layers get MPTCP for free (mostly)

106. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 36
Path Management - ndiffports
N different source ports,
1 destination port
Client Server

107. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 36
Path Management - ndiffports
N different source ports,
1 destination port
Client Server
N = 1

108. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 36
Path Management - ndiffports
N different source ports,
1 destination port
Client Server
N = 2

109. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 36
Path Management - ndiffports
N different source ports,
1 destination port
Client Server
N = 3

110. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 37
Path Management - fullmesh
All possible paths used
Client Server

111. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 37
Path Management - fullmesh
All possible paths used
Client Server

112. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 37
Path Management - fullmesh
All possible paths used
Client Server

113. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 37
Path Management - fullmesh
All possible paths used
Client Server

114. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 37
Path Management - fullmesh
All possible paths used
Client Server

115. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 38
Introduction ✔
Background (Why Change TCP) ✔
MPTCP ✔
Background Redux (Why NOT TO change TCP)
QUIC
Implications
Conclusion and Takeaways

116. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 39
Why NOT Change TCP?

117. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 39
Why NOT Change TCP?
Lessons from MPTCP

118. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 39
Why NOT Change TCP?
Lessons from MPTCP
•Slow moving

119. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 39
Why NOT Change TCP?
Lessons from MPTCP
•Slow moving
•Meddleboxes limit protocol deployability

120. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 39
Why NOT Change TCP?
Lessons from MPTCP
•Slow moving
•Meddleboxes limit protocol deployability
•Chicken and egg deployment

121. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 40
Why NOT Change TCP?

122. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 40
Why NOT Change TCP?
TCP Characteristics:

123. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 40
Why NOT Change TCP?
TCP Characteristics:
•Handshake design

124. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 40
Why NOT Change TCP?
TCP Characteristics:
•Handshake design
•Outside user-space

125. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 40
Why NOT Change TCP?
TCP Characteristics:
•Handshake design
•Outside user-space
•End-of-line Blocking

126. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 41
Why NOT Change TCP?

127. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 41
Why NOT Change TCP?
If you can’t change TCP what’s left?

128. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 41
Why NOT Change TCP?
If you can’t change TCP what’s left?
•SCTP?
• Same problems, but amplified

129. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 41
Why NOT Change TCP?
If you can’t change TCP what’s left?
•SCTP?
• Same problems, but amplified

130. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 41
Why NOT Change TCP?
If you can’t change TCP what’s left?
•SCTP?
• Same problems, but amplified
•UDP?
• But it doesn’t do ANYTHING fancy?
• Exactly.

131. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 42
Introduction ✔
Background (Why Change TCP) ✔
MPTCP ✔
Background Redux (Why NOT TO change TCP) ✔
QUIC
Implications
Conclusion and Takeaways

132. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 43
QUIC
(Quick UDP Internet Connections)

133. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 43
•You thought MPTCP developed fast?
QUIC
(Quick UDP Internet Connections)

134. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 43
•You thought MPTCP developed fast?
•QUIC was even QUIC-ker
• Already in use on many Google properties
• Youtube, Google search, and more
• Likely several percent of your traffic
QUIC
(Quick UDP Internet Connections)

135. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 44
Current TCP is rather limited

136. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 44
Makes a lot of round trips
Current TCP is rather limited

137. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 44
Makes a lot of round trips
Blocks stream on retransmits
Current TCP is rather limited

138. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 44
Makes a lot of round trips
Blocks stream on retransmits
Current TCP is rather limited
QUIC

139. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)

140. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)
UDP transport
protocol

141. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)
UDP transport
protocol
- Google championed
successor to SPDY

142. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)
UDP transport
protocol
- Google championed
successor to SPDY
- Latency Optimized

143. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)
UDP transport
protocol
- Google championed
successor to SPDY
- Latency Optimized
- Reliable, Multiplexed

144. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)
UDP transport
protocol
- Google championed
successor to SPDY
- Latency Optimized
- Reliable, Multiplexed
- Always Encrypted

145. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)
UDP transport
protocol
- Google championed
successor to SPDY
- Latency Optimized
- Reliable, Multiplexed
- Always Encrypted
Open Source

146. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)
UDP transport
protocol
- Google championed
successor to SPDY
- Latency Optimized
- Reliable, Multiplexed
- Always Encrypted
Open Source
User Space

147. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)
UDP transport
protocol
- Google championed
successor to SPDY
- Latency Optimized
- Reliable, Multiplexed
- Always Encrypted
Open Source
User Space
- No OS requirements

148. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 45
QUIC
(Quick UDP Internet Connections)
UDP transport
protocol
- Google championed
successor to SPDY
- Latency Optimized
- Reliable, Multiplexed
- Always Encrypted
Open Source
User Space
- No OS requirements
- Fast Evolving

149. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 46
QUIC connection – Latency Reduction

150. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 46
• 0-RTT Connection establishment (1 sometimes)
QUIC connection – Latency Reduction

151. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 46
• 0-RTT Connection establishment (1 sometimes)
• FEC-based packet-loss recovery
QUIC connection – Latency Reduction

152. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 46
• 0-RTT Connection establishment (1 sometimes)
• FEC-based packet-loss recovery
• Flow Control at both connection and stream level
QUIC connection – Latency Reduction

153. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 46
• 0-RTT Connection establishment (1 sometimes)
• FEC-based packet-loss recovery
• Flow Control at both connection and stream level
• Certificate and header compression
QUIC connection – Latency Reduction

154. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 47
• Google claims:
• 30% reduction in video rebuffers
• 10% reduction in page load times
https://peering.google.com/#/learn-more/quic
QUIC connection – Benefits

155. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 48
QUIC connection - Overview

156. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 48
• UDP Port 80 and 443
QUIC connection - Overview

157. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 48
• UDP Port 80 and 443
• Upgrade headers in HTTP:
• Alternate-Protocol: [port]:quic
• Alternate-Protocol: 443:quic
QUIC connection - Overview

158. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 49
QUIC Infodump 0% [Starting…]

159. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 49
• Quic isn’t simple.
• Read the RFC, docs, and source
QUIC Infodump 0% [Starting…]

160. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 49
• Quic isn’t simple.
• Read the RFC, docs, and source
• QUIC is way more intricate than MPTCP
QUIC Infodump 0% [Starting…]

161. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 49
• Quic isn’t simple.
• Read the RFC, docs, and source
• QUIC is way more intricate than MPTCP
QUIC Infodump 0% [Starting…]

162. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 49
• Quic isn’t simple.
• Read the RFC, docs, and source
• QUIC is way more intricate than MPTCP
• I’m going to RACE through some details to
show the complexity of QUIC
QUIC Infodump 0% [Starting…]

163. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 50
QUIC Infodump 5% [x-------------------] Packet
Types

164. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 50
• Regular Packets
• Frame Packets
QUIC Infodump 5% [x-------------------] Packet
Types

165. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 50
• Regular Packets
• Frame Packets
• Special packet types
• Version negotiation,
• Public reset
QUIC Infodump 5% [x-------------------] Packet
Types

166. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview

167. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
UDP Packet UDP Packet

168. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
UDP Packet UDP Packet
QUIC Packet

169. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
UDP Packet
AEAD Data
UDP Packet
QUIC Packet

170. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
UDP Packet
AEAD Data
QUIC Private Header
UDP Packet
QUIC Packet

171. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
QUIC Frame Packet
UDP Packet
AEAD Data
QUIC Private Header
UDP Packet
QUIC Packet
Frame Packet

172. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
QUIC Frame Packet
QUIC Frame 1
UDP Packet
AEAD Data
QUIC Private Header
UDP Packet
QUIC Packet
Frame Packet
Frame

173. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
QUIC Frame Packet
QUIC Frame 1
UDP Packet
AEAD Data
QUIC Private Header
UDP Packet
QUIC Packet
Frame Packet
FrameQUIC Frame 2
Frame

174. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
QUIC Frame Packet
QUIC Frame 1
UDP Packet
AEAD Data
QUIC Private Header
UDP Packet
QUIC Packet
Frame Packet
Frame
FrameQUIC Frame 2
Frame

175. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
QUIC Frame Packet
QUIC Frame 1
UDP Packet
AEAD Data
QUIC Private Header
UDP Packet
QUIC Packet
Frame Packet
Frame
FrameQUIC Frame 2
QUIC Frame n
Frame

176. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
QUIC Frame Packet
QUIC Frame 1
UDP Packet
AEAD Data
QUIC Private Header
UDP Packet
QUIC Packet
Frame Packet
Frame
FrameQUIC Frame 2
QUIC Frame n
Frame
Unencrypted
Authenticated

177. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 51
QUIC Infodump 10%[--x-----------------]
Packet Structure- Overview
UDP
QUIC Public Header
QUIC Frame Packet
QUIC Frame 1
UDP Packet
AEAD Data
QUIC Private Header
UDP Packet
QUIC Packet
Frame Packet
Frame
FrameQUIC Frame 2
QUIC Frame n
Frame
Unencrypted
Authenticated
Encrypted
Authenticated

178. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 52
QUIC Infodump 14%[--x-----------------]
Packet Structure- Overview

179. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 52
• ONE QUIC Connection
QUIC Infodump 14%[--x-----------------]
Packet Structure- Overview

180. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 52
• ONE QUIC Connection
QUIC Infodump 14%[--x-----------------]
Packet Structure- Overview
• ONE QUIC Packet

181. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 52
• ONE QUIC Connection
QUIC Infodump 14%[--x-----------------]
Packet Structure- Overview
• ONE QUIC Packet
Contains

182. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 52
• ONE QUIC Connection
QUIC Infodump 14%[--x-----------------]
Packet Structure- Overview
• ONE QUIC Packet
Contains
• 0-1 Frame Packets

183. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 52
• ONE QUIC Connection
QUIC Infodump 14%[--x-----------------]
Packet Structure- Overview
• ONE QUIC Packet
Contains
• 0-1 Frame Packets
Each containing

184. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 52
• ONE QUIC Connection
Contains
• N Streams
QUIC Infodump 14%[--x-----------------]
Packet Structure- Overview
• ONE QUIC Packet
Contains
• 0-1 Frame Packets
Each containing
• N frames

185. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 52
• ONE QUIC Connection
Contains
• N Streams
QUIC Infodump 14%[--x-----------------]
Packet Structure- Overview
• ONE QUIC Packet
Contains
• 0-1 Frame Packets
Each containing
• N frames

186. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 53
QUIC Infodump 17%[---x----------------]
Packet Structure- Overview

187. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 53
• Two primary parts to a [regular] QUIC packet
QUIC Infodump 17%[---x----------------]
Packet Structure- Overview

188. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 53
• Two primary parts to a [regular] QUIC packet
• Public Header (Authenticated, NOT Encrypted)
QUIC Infodump 17%[---x----------------]
Packet Structure- Overview

189. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 53
• Two primary parts to a [regular] QUIC packet
• Public Header (Authenticated, NOT Encrypted)
• Private (Authenticated, Encrypted)
QUIC Infodump 17%[---x----------------]
Packet Structure- Overview

190. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 53
• Two primary parts to a [regular] QUIC packet
• Public Header (Authenticated, NOT Encrypted)
• Private (Authenticated, Encrypted)
• AEAD (authenticated encryption and
associated data) data directly after Public
header, used to interpret Private data
QUIC Infodump 17%[---x----------------]
Packet Structure- Overview

191. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 54
QUIC Infodump 25%[-----x--------------]
Headers

192. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 54
• Public Header (Authenticated, NOT Encrypted)
• Flags
• Connection ID (Variable length, optional)
• Packet Number (Variable length)
• Version Header (If flag set)
QUIC Infodump 25%[-----x--------------]
Headers

193. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 54
• Public Header (Authenticated, NOT Encrypted)
• Flags
• Connection ID (Variable length, optional)
• Packet Number (Variable length)
• Version Header (If flag set)
• Private (Authenticated, Encrypted)
• Flags
• FEC (Optional)
• Frame or FEC Payload
QUIC Infodump 25%[-----x--------------]
Headers

194. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 55
QUIC Infodump 40%[--------x-----------]
Frames

195. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 55
• Frame Packet (contains frames)
• Type
• Special
• Regular
• N Frames
QUIC Infodump 40%[--------x-----------]
Frames

196. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 55
• Frame Packet (contains frames)
• Type
• Special
• Regular
• N Frames
• FEC Packet
• NULL-padded XOR of the data packets in the payload group
• Payload group details in the Private header
QUIC Infodump 40%[--------x-----------]
Frames

197. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 56
QUIC Infodump 47%[----------x---------]
Frame Types

198. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 56
• Control Frames
• Padding
• RST Stream
• Connection Close
• Goaway
• Window Update
• Blocked
• Stop_waiting
• Ping
QUIC Infodump 47%[----------x---------]
Frame Types

199. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 56
• Control Frames
• Padding
• RST Stream
• Connection Close
• Goaway
• Window Update
• Blocked
• Stop_waiting
• Ping
QUIC Infodump 47%[----------x---------]
Frame Types
• Special Frames
• Stream
• Ack
• Congestion feedback
• Most relevant:
• Stream frame
• ACK Frame

200. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 57
QUIC Infodump 52%[-----------x--------]
Stream Frame

201. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 57
• Contained inside a frame packet
QUIC Infodump 52%[-----------x--------]
Stream Frame

202. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 57
• Contained inside a frame packet
• Fields
• Type (Frame header: flags about lengths of other
items below)
• Stream ID
• Offset
• Length (0+, optional, omission means full packet)
QUIC Infodump 52%[-----------x--------]
Stream Frame

203. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 58
QUIC Infodump 57%[------------x-------]
ACK Frame

204. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 58
• Contained inside a frame packet
QUIC Infodump 57%[------------x-------]
ACK Frame

205. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 58
• Contained inside a frame packet
• Fields
• Type (Frame header: flags about forms of other items below)
• Largest Observed packet #
• ACK Delay time
• Timestamp Section (Used as congestion indicator)
• Missing Packet Section (~NACK)
• Revived Packet Section (Which did FEC revive)
QUIC Infodump 57%[------------x-------]
ACK Frame

206. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 59
QUIC Infodump 65% [-------------x------]
Addressing

207. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 59
• Connection ID
• 64 bit, client chosen
• Independent of Network Address
• Source Address token (from previous connection)
QUIC Infodump 65% [-------------x------]
Addressing

208. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 59
• Connection ID
• 64 bit, client chosen
• Independent of Network Address
• Source Address token (from previous connection)
• Stream ID
• Data flow WITHIN a connection
QUIC Infodump 65% [-------------x------]
Addressing

209. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 60
QUIC Infodump 77% [---------------x----]
Cryptographic

210. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 60
• Replaces, and Comparable to, TLS
• Client reuses encryption parameters from previous
connection
QUIC Infodump 77% [---------------x----]
Cryptographic

211. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 60
• Replaces, and Comparable to, TLS
• Client reuses encryption parameters from previous
connection
• IP Spoofing protection
• Requires an IP-validated Source Address Token, or
negotiates a new one (0/1 RTT)
• No trivial reflection attacks as in NTP/DNS
QUIC Infodump 77% [---------------x----]
Cryptographic

212. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 61
QUIC Infodump 84%[----------------x---]
Forward Error Correction

213. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 61
• XOR of all packets in FEC block
QUIC Infodump 84%[----------------x---]
Forward Error Correction

214. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 61
• XOR of all packets in FEC block
• FEC block size is variable
QUIC Infodump 84%[----------------x---]
Forward Error Correction

215. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 61
• XOR of all packets in FEC block
• FEC block size is variable
• If “revived by FEC” then indicate this ID as
NACK and REVIVED
QUIC Infodump 84%[----------------x---]
Forward Error Correction

216. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 62
QUIC Infodump 93%[-------------------x]
Connection Management

217. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 62
• All connections assumed to be left in open
state until user leaves page
QUIC Infodump 93%[-------------------x]
Connection Management

218. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 62
• All connections assumed to be left in open
state until user leaves page
• Address mobility supported
QUIC Infodump 93%[-------------------x]
Connection Management

219. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 62
• All connections assumed to be left in open
state until user leaves page
• Address mobility supported
• Multipath planned, but not yet there
QUIC Infodump 93%[-------------------x]
Connection Management

220. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 63
• QUIC intends to handle most HTTP/2
stream management
QUIC Infodump 99%[-------------------x]
HTTP/2 Integration

221. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 64
• UDP Unicast unimpeded
• No filters
• No Rate Limiters
• UDP NAT good practice
• E.g. Timeouts set reasonably
QUIC Infodump 100% [ Finishing… ]
Network Requirements

222. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 65
• Chrome:
• chrome://net-internals/#quic
Debugging UIC

223. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 66
• Wireshark:
QUIC dissector =>
Debugging QUIC

224. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 67
Introduction ✔
Background (Why Change TCP) ✔
MPTCP ✔
Background Redux (Why NOT TO change TCP) ✔
QUIC ✔
Implications
Conclusion and Takeaways

225. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 68
Was anyone thinking about the network environment?

226. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 68
Was anyone thinking about the network environment?
•The security and stability of
MPTCP itself

227. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 68
Was anyone thinking about the network environment?
•The security and stability of
MPTCP itself

228. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 68
Was anyone thinking about the network environment?
•The security and stability of
MPTCP itself
•What changes like this could
mean for network security

229. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 68
Was anyone thinking about the network environment?
•The security and stability of
MPTCP itself
•What changes like this could
mean for network security
… not so much

230. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 69
Was anyone thinking about the network environment?
•The security of MPTCP itself
•What changes like this could mean for
network security
… not so much
QUIC put more work into
this
• Google also operates the network
• But they also put in “Protection”
against network operator
interference

231. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 70
What does it look like?
•On the network: If you don’t
understand
•…

232. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 71
What does it look like?
•On the network: If you don’t
understand
Each yellow blob is actually part of an address label

233. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 72
What does it look like?
•On the
network: If
you don’t
understand it,
but you
cluster IPs

234. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 73
What does it look like?
• On the network: If you do understand
▪ But you can only do this when you can see & correlate all related flows…

235. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 74
MPTCP and … Network Management

236. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 74
MPTCP and … Network Management
• These protocols are agnostic of IPv4 and
IPv6, happily use both

237. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 74
MPTCP and … Network Management
• These protocols are agnostic of IPv4 and
IPv6, happily use both
• If tool doesn’t understand MPTCP, flows
look like unrelated TCP streams

238. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 74
MPTCP and … Network Management
• These protocols are agnostic of IPv4 and
IPv6, happily use both
• If tool doesn’t understand MPTCP, flows
look like unrelated TCP streams
• If tool doesn’t understand QUIC, flows
look like unrelated UDP flows

239. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 75
IDS
Monitoring – Cross-path fragmentation
Client Server
SEND
ABCD
RECV
ABCD
AC

240. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 75
IDS
Monitoring – Cross-path fragmentation
Client Server
SEND
ABCD
RECV
ABCD
AC
BD

241. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 75
IDS
Monitoring – Cross-path fragmentation
Client Server
IDS
SEND
ABCD
RECV
ABCD
AC
BD

242. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 76
Multipath and … Network monitoring

243. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 76
Multipath and … Network monitoring
• How would your CGNAT handle an exponential
increase in connections as well as clients?

244. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 76
Multipath and … Network monitoring
• How would your CGNAT handle an exponential
increase in connections as well as clients?
• Two devices:
• One interface each: 1 flow
• Four interfaces each: 16 flows

245. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 76
Multipath and … Network monitoring
• How would your CGNAT handle an exponential
increase in connections as well as clients?
• Two devices:
• One interface each: 1 flow
• Four interfaces each: 16 flows
• What about virtual interfaces, VPNs, proxies, load
balancers..

246. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 77
D
Monitoring – Combined Cross-path and Cross-Stream fragmentation
Client Server
Server Client
A
B
C
Insert Route redundancy
and/or Proxies to be mean

247. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 77
D
Monitoring – Combined Cross-path and Cross-Stream fragmentation
Client Server
Server Client
A
B
C
Insert Route redundancy
and/or Proxies to be mean

248. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 77
D
IDS
Monitoring – Combined Cross-path and Cross-Stream fragmentation
Client Server
IDS
Server Client
A
B
Proxy
Proxy
C
Insert Route redundancy
and/or Proxies to be mean

249. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 78
Multipath and … Network monitoring
• How would your CGNAT handle an
exponential increase in connections as
well as clients?
• Two Devices:
• One interface each: 1 flow
• Four Interfaces each: 16 flows
• What about virtual interfaces, VPNs,
proxies, load balancers..
In trying to solve address
exhaustion with NAT do we risk
state processing exhaustion?

250. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
Because of
these…

251. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
Because of
these…

252. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
Because of
these…

253. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
• Connection
resilience
Because of
these…

254. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
• Connection
resilience
• Reverse
connections
Because of
these…

255. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
• Connection
resilience
• Reverse
connections
Because of
these…

256. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
• Connection
resilience
• Reverse
connections
Because of
these…
… if your approach to network
management requires any of these…

257. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
• Connection
resilience
• Reverse
connections
• See all app layer data in a TCP stream
Because of
these…
… if your approach to network
management requires any of these…

258. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
• Connection
resilience
• Reverse
connections
• See all app layer data in a TCP stream
• Associate logical sessions to IP
addresses
Because of
these…
… if your approach to network
management requires any of these…

259. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
• Connection
resilience
• Reverse
connections
• See all app layer data in a TCP stream
• Associate logical sessions to IP
addresses
• Tamper with or close "bad" connections
mid-stream
Because of
these…
… if your approach to network
management requires any of these…

260. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
• Connection
resilience
• Reverse
connections
• See all app layer data in a TCP stream
• Associate logical sessions to IP
addresses
• Tamper with or close "bad" connections
mid-stream
• Differentiate clients from servers based
on connection direction
Because of
these…
… if your approach to network
management requires any of these…

261. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Cross-path
• Moving target
• Connection
resilience
• Reverse
connections
• See all app layer data in a TCP stream
• Associate logical sessions to IP
addresses
• Tamper with or close "bad" connections
mid-stream
• Differentiate clients from servers based
on connection direction
Because of
these…
… if your approach to network
management requires any of these…
…then something is probably going to break

262. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 80
Key Effects

263. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 80
Key Effects
• Cross-path traffic fragmentation
• That’s the whole point!

264. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 80
Key Effects
• Cross-path traffic fragmentation
• That’s the whole point!
• Moving target
• Ability to change source and destination addresses in the middle of a connection

265. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 80
Key Effects
• Cross-path traffic fragmentation
• That’s the whole point!
• Moving target
• Ability to change source and destination addresses in the middle of a connection

266. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 80
Key Effects
• Cross-path traffic fragmentation
• That’s the whole point!
• Moving target
• Ability to change source and destination addresses in the middle of a connection
• Connection resilience
• Has additional checksums that require capture of the initial packet to reliably fake
• Until every subflow is dead the overall connection keeps going

267. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 80
Key Effects
• Cross-path traffic fragmentation
• That’s the whole point!
• Moving target
• Ability to change source and destination addresses in the middle of a connection
• Connection resilience
• Has additional checksums that require capture of the initial packet to reliably fake
• Until every subflow is dead the overall connection keeps going
• “Reverse” connections

268. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 81
Key Effects

269. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 81
Key Effects
A few lot of slides back…
• The packet sender decides which data goes down which path.

270. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 82
John Gilmore, 1993
“The Net interprets censorship
as damage and routes
around it.”

271. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 83
Kate Pearce, 2014
“Multipath interprets
modification as damage and
routes around it.”

272. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 84
Key Effects

273. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 84
Key Effects
A few lot of slides back…
• The packet sender decides which data goes down which path.
• Normal/benign clients won’t choose pathological fragmentation
schemes

274. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 84
Key Effects
A few lot of slides back…
• The packet sender decides which data goes down which path.
• Normal/benign clients won’t choose pathological fragmentation
schemes

275. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 85
MPTCP and … Firewalls

276. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 85
MPTCP and … Firewalls
•MPTCP has address
advertisement
• This changes things for perimeters

277. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 85
MPTCP and … Firewalls
•MPTCP has address
advertisement
• This changes things for perimeters

278. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 85
MPTCP and … Firewalls
•MPTCP has address
advertisement
• This changes things for perimeters
•How’d you like an outbound
incoming connection?

279. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 86
MPTCP and … Firewalls
• MPTCP connection looks like TCP so
far…
Client ServerFirewall
1. SYN (MP_CAPABLE)

280. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 87
MPTCP and … Firewalls
• Still seems pretty standard, albeit with
extra TCP OPTIONS
Client ServerFirewall
2. SYN, ACK (MP_CAPABLE)

281. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 88
MPTCP and … Firewalls
• OK, so it’s a TCP connection with an
additional options… so what?
Client ServerFirewall
3. ACK…

282. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 89
MPTCP and … Firewalls
• Well, what if the client tells the server
about a new address?
Client ServerFirewall
Pre-established
MPTCP Connection
4. ADD_ADDR (IP#2)

283. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 90
MPTCP and … Firewalls
• Now, the “Internal” host may set up a
connection to the advertised address
Client ServerFirewall
Pre-established
MPTCP Connection
4. ADD_ADDR (IP#2)

284. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 91
MPTCP and … Firewalls
Client ServerFirewall
Pre-established
MPTCP Connection
5. SYN (MP_JOIN)

285. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 91
MPTCP and … Firewalls
Client ServerFirewall
Pre-established
MPTCP Connection
5. SYN (MP_JOIN)
▪ Is this new connection incoming or outgoing?

286. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 92
MPTCP and … Firewalls
• Is this new connection incoming or outgoing?
Client ServerFirewall
Pre-established
MPTCP Connection
5. SYN (MP_JOIN)

287. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 93
Multipath and … Privacy

288. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 93
Multipath and … Privacy
• Multipath shifts power towards endpoints,
and away from infrastructure & ISP’s

289. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 93
Multipath and … Privacy
• Multipath shifts power towards endpoints,
and away from infrastructure & ISP’s
• I don’t fully trust my ISP or Cellular
company…

290. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 93
Multipath and … Privacy
• Multipath shifts power towards endpoints,
and away from infrastructure & ISP’s
• I don’t fully trust my ISP or Cellular
company…
• But they probably don’t trust each other
either!

291. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 94
Introduction ✔
Background (Why Change TCP) ✔
MPTCP ✔
Background Redux (Why NOT TO change TCP) ✔
QUIC ✔
Implications ✔
Conclusion and Takeaways

292. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 95
So… things people took away from my MPTCP work

293. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 95
So… things people took away from my MPTCP work
•It’s a problem of tools (NO)

294. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 95
So… things people took away from my MPTCP work
•It’s a problem of tools (NO)
•It’s a problem that only exists because of
MPTCP

295. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 95
So… things people took away from my MPTCP work
•It’s a problem of tools (NO)
•It’s a problem that only exists because of
MPTCP
•NO

296. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 95
So… things people took away from my MPTCP work
•It’s a problem of tools (NO)
•It’s a problem that only exists because of
MPTCP
•NO
•NO

297. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 95
So… things people took away from my MPTCP work
•It’s a problem of tools (NO)
•It’s a problem that only exists because of
MPTCP
•NO
•NO
•NO

298. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 95
So… things people took away from my MPTCP work
•It’s a problem of tools (NO)
•It’s a problem that only exists because of
MPTCP
•NO
•NO
•NO
•NO
▪NO
▪NO
▪NO
▪NO
▪NO
▪NO
▪NO
▪NO

299. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 96
Key Insights- Multipath
•The (IP) network’s job is to transmit data from
end to end. (?)

300. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 97
[CERF, VINTON G., and ROBERT E. KAHN. "A Protocol for Packet Network
Intercommunication." (1974).]
“Within each network, communication may be
disrupted due to unrecoverable mutation of
the data or missing data. End-to-end
restoration procedures are desirable to allow
complete recovery from these conditions.”

301. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 98
Key Insights – Content
Inspection

302. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 98
Key Insights – Content
Inspection
Many network management approaches rely
too much on

303. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 98
Key Insights – Content
Inspection
Many network management approaches rely
too much on
- Metadata

304. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 98
Key Insights – Content
Inspection
Many network management approaches rely
too much on
- Metadata
- Data inspection

305. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 98
Key Insights – Content
Inspection
Many network management approaches rely
too much on
- Metadata
- Data inspection
- Context Inspection

306. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 98
Key Insights – Content
Inspection
Many network management approaches rely
too much on
- Metadata
- Data inspection
- Context Inspection
The Internet wasn’t designed to validate your
control over network data flows.

307. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 99
Technical Takeaways

308. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 99
• Address != connection endpoint
Technical Takeaways

309. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 99
• Address != connection endpoint
• State exhaustion could become the new
address exhaustion
Technical Takeaways

310. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 99
• Address != connection endpoint
• State exhaustion could become the new
address exhaustion
• Multipath tech breaks assumptions many
didn’t even know we were making
Technical Takeaways

311. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 100
Conclusions

312. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 100
Conclusions
• Multipath communications are awesome,
and they’re coming

313. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 100
Conclusions
• Multipath communications are awesome,
and they’re coming
• Multipath communication confounds
business & security models relying on
stateful inspection or lookup

314. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 100
Conclusions
• Multipath communications are awesome,
and they’re coming
• Multipath communication confounds
business & security models relying on
stateful inspection or lookup
• Now is the time for network security and
network management to prepare

315. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 101
• MPTCP:
• Raiciu, C. et al., 2012. How hard can it be? designing and implementing a deployable multipath TCP.
NSDI, (1). Available at: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final125.pdf.
• ACM Queue - Multipath TCP, Decoupled from IP, TCP is at last able to support multihomed hosts -
Christoph Paasch and Olivier Bonaventure, UCL - http://queue.acm.org/detail.cfm?id=2591369
• IETF Working group - http://datatracker.ietf.org/wg/mptcp/
• QUIC
• https://peering.google.com/#/learn-more/quic
• https://www.chromium.org/quic
• https://tools.ietf.org/html/draft-tsvwg-quic-protocol-02
• https://tools.ietf.org/html/draft-tsvwg-quic-loss-recovery-01
References and Links

316. Questions?
Twitter: @secvalve
katpearc@cisco.com
Catherine (Kate) Pearce

317. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential2016 103
All possible
paths can be used at the same time
Path Management – mesh networks
Internet