Reliability Extensions and Multi-Hop Evaluation of
Distributed Protocol Stacks
IEEE International Conference on Cyber, Physical and Social Computing (CPSCom 2013)
2013 IEEE 国际信息物理社会计算大会
August 20-23, 2013, Beijing, China
Peter Rothenpieler
Institute of Telematics, University of Lübeck
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
2
Content
 Distributed Protocol Stacks
 Motivation & Concept
 Protocol overview
 Reliability extensions
 Multi-hop evaluation
 Handshake & connection duration
 Packet reception & duplicate packets rate
 Round-trip time
 Summary
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
Motivation: From WSNs to the Internet of Things
WSN = resource constrained devices (memory, processing, energy)
 problem specific, custom tailored solutions  standardized protocol stacks
Advantages:
 Change individual layers  adapt to scenario
 Interoperability & heterogeneity
Disadvantages:
 Additional protocols/layers increase code size
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
Application
6LoWPAN
IPv6
UDP ICMP
IEEE 802.15.4
MAC & PHY
Application
6LoWPAN
IPv6
UDP ICMP
IEEE 802.15.4
MAC & PHY
3
Motivation: Code Size of 6LoWPAN & IPv6
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
4
61%
50%
53%
26%
24%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Contiki (JN5148, 128KB)
iSense (JN5148, 128KB)
Contiki (MSP430, 48KB)
Contiki (JN5139, 96KB)
iSense (JN5139, 96KB)
Platform
Code size [% of available program memory]
Source: Instant Contiki 2.6, Jennisense (07/2012), iSense SVN (04/2012)
Cheap*
(6.25€ / $8.30 / ¥50.90)
Expensive*
(8.95€ / $11.90 / ¥72.80) +43%
*Prices from Farnell/element14 (08/2013)
 Not included:
 OS, hardware / sensor drivers, …
 Duty cycle protocol, security mechanisms, …
 Application logic and protocols
 IPv6 not possible on JN5139: IPv6+OS+Application = 99KB > 96KB available
Distributed Protocol Stacks
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
Distributed Protocol Stack
 Cooperation between layers  cooperation between nodes
 Share implementations of layers with neighboring nodes
 Asynchronous RPC calls (“message passing”)
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
Application
6LoWPAN
IPv6
UDP ICMP
IEEE 802.15.4
MAC & PHY
Application
6LoWPAN
IPv6
UDP ICMP
IEEE 802.15.4
MAC & PHY
IPv6 stub
UDP ICMP
Application
IEEE 802.15.4
MAC & PHY
DPS
IEEE 802.15.4
MAC & PHY
IPv6 skeleton
UDP ICMP
Application
DPS
6LoWPAN
RPC
Server (Expensive node) Client (Cheap node)
6
Protocol Overview
 Connection establishment
A1) Discovery & advertise
 Metric: LQI, RSSI, ETX
A2) Handshake
 Runtime
B1) RPC messages
B2) Monitoring of connections
 Heartbeat messages
 Timeout
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
Server Server
Client
Advertise
Advertise
Discovery
7
Handshake
RPC Messages
Server
Client
Server
Client
Heartbeat
Server
Client
Heartbeat
A1) A2)
B1) B2)
Multi-Hop Evaluation
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
Experimental Setup
 Wisebed Testbed located at University of Lübeck
 Total: 43 nodes (2:1 ratio of Clients : Servers)
 4 Experiments with a total runtime of 20h
 Scenario: Building monitoring application
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
9
EU FP-7 Project (2008-2011): http://www.wisebed.eu
Questions
 Does the DPS protocol work?
 Does the DPS protocol have negative impact on
 Packet reception rate (PRR)?
 Latency (“Ping”)?
Testbed Topology (IP routing protocol + DPS)
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
10
213c
2138
2134
212c
2128
2100
2104
2108
2110
2114
2118
2124
2120
2144 2140
2130
(SINK)
2039
2040
2041
2044
2045 2031 2030
2025
2014
2019
2018 2010
2011
203C
203D
200C
2008
2009
2001
2000
2005
2004
2038
2035
2034
202D
2029
2028
202C
Link between Servers (IP routing protocol)Server node
Client node Link between Server and Client (DPS protocol)
Visualization of routing path: All nodes send data to sink
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
11
Link between Servers (IP routing protocol)
Link between Server and Client (DPS protocol)
Server node
Client node
All nodes send their gathered sensor data (48 byte) to the
Sink using UDP every 30 seconds.
Example:
• Client A sends it‘s data to the sink
• Client A is attached to Server S1
• S1 forwards data to sink using IP-routing protocol
• Routing path:
• A  S1  S2  S3  S4  Sink
• Distance: 5 hops
SINK
Client A
Server S1
Server S2
Server S3
Server S4
How long does the Discovery and Handshake take?
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
12
 Time between first DISCOVERY message and connection successfully established
 Clients send at least DISCOVERY messages every 30ms until
 At least 3 DISCOVERY messages have been sent
 At least 1 ADVERTISE was received
 Afterwards, Three-way-Handshake starts (CONNECT, ALLOW, FINISH)
>90ms
(90+x) ms
(Transfer time
+ CSMA random backoff)
+(30+x) ms
(One additional DISCOVERY)
How long do DPS-connections last?
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
13
 Clients and Servers exchange heartbeat messages every second
 DPS connections are closed if no heartbeat messages was received for more than 4 seconds
 If one node closes the connection, the other node closes the connection as well (within 4 seconds)
Most connections last 10-20 min
8% last longer than 2 hours
How many connections does each node establish?
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
14
 A new connection is established after the heartbeat mechanism detected a disconnection
 Clients establish exactly 1 connection to 1 Server
 Servers establish several connections (1 to each Client)
Large differences between nodes (factor 3-4) are a result
of different positions (obstacles, interference, density, …)
Average: 1.1 Average: 2.0
Packet Reception Rate and Duplicate Packets
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
15
 70% of all nodes have average PRR >95%
 Node with lowest average PRR: 0x2100 89.2% (1 hop from sink)
 Duplicate packets increase with distance
 PRR does not increase with distance (obstacles, interference, density, …)
Far from sink Close to sink
Round-trip time: “Ping” all nodes in the network
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
16
Link between Servers (IP routing protocol)
Link between Server and Client (DPS protocol)
Server node
Client node
SINK
Client A
Server S1
Server S2
Server S3
Server S4
Client B
ICMPv6 Echo Request
ICMPv6 Echo Response
What is the round-trip time between each node and the sink?
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
17
 100 ICMP Echo Request/Response Packets to each node in the network (48byte payload, same as UDP)
 DPS has a negative influence on RTT (see paper from CPScom 2012)
 But this influence in negligible in a multi-hop scenario (or even positive)
Upper quartile
Median
Lower quartile
Negative impact of DPS
Positive impact of DPS?
Summary
 Motivation & Concept: Distributed Protocol Stacks
 Protocol overview (incl. Reliability extensions)
 Multi-hop evaluation
 Connection establishment takes ~100ms
 Connections last ~10-20min (or even more than 2h)
 No negative impact on packet reception rate
 Negative impact on round-trip time only measurable on
short distances
 DPS Protocol works!
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
18
Available on slideshare:
http://bit.ly/1752ZIY
Summary
 Motivation & Concept: Distributed Protocol Stacks
 Protocol overview (incl. Reliability extensions)
 Multi-hop evaluation
 Connection establishment takes ~100ms
 Connections last ~10-20min (or even more than 2h)
 No negative impact on packet reception rate
 Negative impact on round-trip time only measurable on
short distances
 DPS Protocol works!
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
19
Available on slideshare:
http://bit.ly/1752ZIY
Thank you for your attention!
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
Code Size
 Code size of native IPv6 implementation on JN5139 > 96 KB Flash
 Use of IPv6 on JN5139 now possible (26.5 KB reduction)
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
Native IPv6
JN5139
DPS Client
JN5139
Native IPv6
JN5148
DPS Server
JN5148
Application 5.5 KB 3.8 KB
Os 43.9 KB 36.4 KB
IPv6 Stack 50.2 KB 10.1 KB 27.6 KB 27.6 KB
DPS - 13.6 KB - 8.0 KB
Σ 99.6 KB 73.1 KB 67.8 KB 75.8 KB
Relative
100 % -27 %
-26.5 KB
100 % + 12 %
+8.0 KB
ServerClient
21
What is the average packet reception rate (PRR)?
Dipl.-Inf. Peter Rothenpieler
rothenpieler@itm.uni-luebeck.de
http://www.itm.uni-luebeck.de/users/rothenpieler
22
 Nodes send data via UDP  No reliability!
 Network size: 1-5 hops (average: 4.5 hops)
 For Clients, the first hop uses the DPS protocol
 All remaining hops use IPv6/6LoWPAN
57.1% of the time, PRR is higher than 95%

Reliability extensions and multi hop evaluation of distributed protocol stacks

  • 1.
    Reliability Extensions andMulti-Hop Evaluation of Distributed Protocol Stacks IEEE International Conference on Cyber, Physical and Social Computing (CPSCom 2013) 2013 IEEE 国际信息物理社会计算大会 August 20-23, 2013, Beijing, China Peter Rothenpieler Institute of Telematics, University of Lübeck Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler
  • 2.
    2 Content  Distributed ProtocolStacks  Motivation & Concept  Protocol overview  Reliability extensions  Multi-hop evaluation  Handshake & connection duration  Packet reception & duplicate packets rate  Round-trip time  Summary Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler
  • 3.
    Motivation: From WSNsto the Internet of Things WSN = resource constrained devices (memory, processing, energy)  problem specific, custom tailored solutions  standardized protocol stacks Advantages:  Change individual layers  adapt to scenario  Interoperability & heterogeneity Disadvantages:  Additional protocols/layers increase code size Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler Application 6LoWPAN IPv6 UDP ICMP IEEE 802.15.4 MAC & PHY Application 6LoWPAN IPv6 UDP ICMP IEEE 802.15.4 MAC & PHY 3
  • 4.
    Motivation: Code Sizeof 6LoWPAN & IPv6 Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 4 61% 50% 53% 26% 24% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Contiki (JN5148, 128KB) iSense (JN5148, 128KB) Contiki (MSP430, 48KB) Contiki (JN5139, 96KB) iSense (JN5139, 96KB) Platform Code size [% of available program memory] Source: Instant Contiki 2.6, Jennisense (07/2012), iSense SVN (04/2012) Cheap* (6.25€ / $8.30 / ¥50.90) Expensive* (8.95€ / $11.90 / ¥72.80) +43% *Prices from Farnell/element14 (08/2013)  Not included:  OS, hardware / sensor drivers, …  Duty cycle protocol, security mechanisms, …  Application logic and protocols  IPv6 not possible on JN5139: IPv6+OS+Application = 99KB > 96KB available
  • 5.
    Distributed Protocol Stacks Dipl.-Inf.Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler
  • 6.
    Distributed Protocol Stack Cooperation between layers  cooperation between nodes  Share implementations of layers with neighboring nodes  Asynchronous RPC calls (“message passing”) Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler Application 6LoWPAN IPv6 UDP ICMP IEEE 802.15.4 MAC & PHY Application 6LoWPAN IPv6 UDP ICMP IEEE 802.15.4 MAC & PHY IPv6 stub UDP ICMP Application IEEE 802.15.4 MAC & PHY DPS IEEE 802.15.4 MAC & PHY IPv6 skeleton UDP ICMP Application DPS 6LoWPAN RPC Server (Expensive node) Client (Cheap node) 6
  • 7.
    Protocol Overview  Connectionestablishment A1) Discovery & advertise  Metric: LQI, RSSI, ETX A2) Handshake  Runtime B1) RPC messages B2) Monitoring of connections  Heartbeat messages  Timeout Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler Server Server Client Advertise Advertise Discovery 7 Handshake RPC Messages Server Client Server Client Heartbeat Server Client Heartbeat A1) A2) B1) B2)
  • 8.
    Multi-Hop Evaluation Dipl.-Inf. PeterRothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler
  • 9.
    Experimental Setup  WisebedTestbed located at University of Lübeck  Total: 43 nodes (2:1 ratio of Clients : Servers)  4 Experiments with a total runtime of 20h  Scenario: Building monitoring application Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 9 EU FP-7 Project (2008-2011): http://www.wisebed.eu Questions  Does the DPS protocol work?  Does the DPS protocol have negative impact on  Packet reception rate (PRR)?  Latency (“Ping”)?
  • 10.
    Testbed Topology (IProuting protocol + DPS) Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 10 213c 2138 2134 212c 2128 2100 2104 2108 2110 2114 2118 2124 2120 2144 2140 2130 (SINK) 2039 2040 2041 2044 2045 2031 2030 2025 2014 2019 2018 2010 2011 203C 203D 200C 2008 2009 2001 2000 2005 2004 2038 2035 2034 202D 2029 2028 202C Link between Servers (IP routing protocol)Server node Client node Link between Server and Client (DPS protocol)
  • 11.
    Visualization of routingpath: All nodes send data to sink Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 11 Link between Servers (IP routing protocol) Link between Server and Client (DPS protocol) Server node Client node All nodes send their gathered sensor data (48 byte) to the Sink using UDP every 30 seconds. Example: • Client A sends it‘s data to the sink • Client A is attached to Server S1 • S1 forwards data to sink using IP-routing protocol • Routing path: • A  S1  S2  S3  S4  Sink • Distance: 5 hops SINK Client A Server S1 Server S2 Server S3 Server S4
  • 12.
    How long doesthe Discovery and Handshake take? Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 12  Time between first DISCOVERY message and connection successfully established  Clients send at least DISCOVERY messages every 30ms until  At least 3 DISCOVERY messages have been sent  At least 1 ADVERTISE was received  Afterwards, Three-way-Handshake starts (CONNECT, ALLOW, FINISH) >90ms (90+x) ms (Transfer time + CSMA random backoff) +(30+x) ms (One additional DISCOVERY)
  • 13.
    How long doDPS-connections last? Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 13  Clients and Servers exchange heartbeat messages every second  DPS connections are closed if no heartbeat messages was received for more than 4 seconds  If one node closes the connection, the other node closes the connection as well (within 4 seconds) Most connections last 10-20 min 8% last longer than 2 hours
  • 14.
    How many connectionsdoes each node establish? Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 14  A new connection is established after the heartbeat mechanism detected a disconnection  Clients establish exactly 1 connection to 1 Server  Servers establish several connections (1 to each Client) Large differences between nodes (factor 3-4) are a result of different positions (obstacles, interference, density, …) Average: 1.1 Average: 2.0
  • 15.
    Packet Reception Rateand Duplicate Packets Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 15  70% of all nodes have average PRR >95%  Node with lowest average PRR: 0x2100 89.2% (1 hop from sink)  Duplicate packets increase with distance  PRR does not increase with distance (obstacles, interference, density, …) Far from sink Close to sink
  • 16.
    Round-trip time: “Ping”all nodes in the network Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 16 Link between Servers (IP routing protocol) Link between Server and Client (DPS protocol) Server node Client node SINK Client A Server S1 Server S2 Server S3 Server S4 Client B ICMPv6 Echo Request ICMPv6 Echo Response
  • 17.
    What is theround-trip time between each node and the sink? Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 17  100 ICMP Echo Request/Response Packets to each node in the network (48byte payload, same as UDP)  DPS has a negative influence on RTT (see paper from CPScom 2012)  But this influence in negligible in a multi-hop scenario (or even positive) Upper quartile Median Lower quartile Negative impact of DPS Positive impact of DPS?
  • 18.
    Summary  Motivation &Concept: Distributed Protocol Stacks  Protocol overview (incl. Reliability extensions)  Multi-hop evaluation  Connection establishment takes ~100ms  Connections last ~10-20min (or even more than 2h)  No negative impact on packet reception rate  Negative impact on round-trip time only measurable on short distances  DPS Protocol works! Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 18 Available on slideshare: http://bit.ly/1752ZIY
  • 19.
    Summary  Motivation &Concept: Distributed Protocol Stacks  Protocol overview (incl. Reliability extensions)  Multi-hop evaluation  Connection establishment takes ~100ms  Connections last ~10-20min (or even more than 2h)  No negative impact on packet reception rate  Negative impact on round-trip time only measurable on short distances  DPS Protocol works! Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 19 Available on slideshare: http://bit.ly/1752ZIY Thank you for your attention!
  • 20.
  • 21.
    Code Size  Codesize of native IPv6 implementation on JN5139 > 96 KB Flash  Use of IPv6 on JN5139 now possible (26.5 KB reduction) Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler Native IPv6 JN5139 DPS Client JN5139 Native IPv6 JN5148 DPS Server JN5148 Application 5.5 KB 3.8 KB Os 43.9 KB 36.4 KB IPv6 Stack 50.2 KB 10.1 KB 27.6 KB 27.6 KB DPS - 13.6 KB - 8.0 KB Σ 99.6 KB 73.1 KB 67.8 KB 75.8 KB Relative 100 % -27 % -26.5 KB 100 % + 12 % +8.0 KB ServerClient 21
  • 22.
    What is theaverage packet reception rate (PRR)? Dipl.-Inf. Peter Rothenpieler rothenpieler@itm.uni-luebeck.de http://www.itm.uni-luebeck.de/users/rothenpieler 22  Nodes send data via UDP  No reliability!  Network size: 1-5 hops (average: 4.5 hops)  For Clients, the first hop uses the DPS protocol  All remaining hops use IPv6/6LoWPAN 57.1% of the time, PRR is higher than 95%