Practical Wireless Mesh Networks and Their Applications

Practical
Wireless Mesh Networks
and Their Applications

Raluca Musăloiu-E.
Johns Hopkins University
November 11, 2009

Joint work with:
Yair Amir, Claudiu Danilov, Michael Hilsdale, Michael Kaplan, Nilo Rivera

Deploying a wireless
network...

... using access points.

Follows client-server paradigm.
For more coverage, install more
access points.
Each access point is connected to
the Internet.

That’s a lot of wires
for a wireless
network!

Mesh networks are a paradigm shift.

Two classes of participants:
Clients, which are mobile.
Nodes, which are relatively stationary.

Only a few nodes are connected to
the Internet (Internet gateways).
They are multi-hop networks.

Lots of research on wireless
networks relies on simulations.
The Mistaken Axioms of Wireless-Network Research:
(Kotz et al., Darmouth College, Tech Report, 2003)

“The world is ﬂat.”
“A radio's transmission area is circular.”
“All radios have equal range.”
“If I can hear you, you can hear me (symmetry).”
“If I can hear you at all, I can hear you perfectly.”
“Signal strength is a simple function of distance.”

We try to bridge the gap
between theory and practice
and build a real mesh system.

Effort has already been made to
make mesh networks a reality.
However...

Some of them are experimental testbeds.
Use expensive hardware for mesh nodes.
Have limited support (or none) for mobility.

For example,
In academia: In industry:
MIT Roofnet Metricom Ricochet
Microsoft MCL Nokia Rooftop
UCSB MeshNet Firetide
Rice Taps Meraki
Rutgers ORBIT Lab Tropos Networks
Stony Brook iMesh

Community Networks:
Champaign-Urbana Community Wireless Network (CUWiN)
NYCwireless
Freyfunk (Germany)

What we are looking for is
Seamless access for mesh users.
Fast handoff while roaming.
Rapid deployment.
Robustness.
Low cost.
Security.
Applications.

This thesis introduces

The architecture of the ﬁrst high-throughput wireless mesh
network with fast handoff using off-the-shelf routers.

The ﬁrst robust Push-To-Talk service for wireless mesh
networks.

The entire mesh network is
seen by the client as an
omniscient access point!

We use 802.11 IBSS mode
(ad-hoc).

MobiSys 2006
WoWMoM 2007
WiMesh 2008

SMesh communication
infrastructure uses the Spines
Messaging System.

Spines was created by Yair Amir and Claudiu Danilov (DSN03, NOSSDAV05,
TOM06).
Spines daemons create overlay networks on the ﬂy.
We run a Spines daemon on each mesh node.
Spines provides unicast, anycast, multicast communication.

Mesh topology formation

Direct links are created Client C

between nearby nodes via an
auto-discovery mechanism.
Client B

Virtual links are created
between Internet gateways
(they form a fully connected
graph, communicating over
an overlay multicast group).

Client A

Mesh topology is hybrid: we use
both wireless and wired links.
The routing metric gives preference to wired links.

SMesh provides a seamless
interface to the mobile clients.

Standard DHCP protocol.

Client always gets the same IP address
(private IP in 10.0.0.0/8 address space;
based on the MAC address).

Client routes all packets through a Virtual
Default Gateway.

NAT

We associate two overlay multicast groups for
each client.

Control
Group

26
30

14

NAT

each client.

Control
Group

26 Data
30
Group

14

NAT

each client.

Fast intra-domain handoff.

The handoff is controlled from the mesh infrastructure!
We use multiple access points during the handoff, to avoid
losing packets.

Fast inter-domain handoff.
SMesh runs in a private address space (NAT is performed at
each Internet gateway).
Connection oriented protocols expect packets to come
from the same source.

Solution: Route each stream through the Internet gateway
used during connection establishment.

We deployed SMesh in our campus.

18-nodes testbed, covering three buildings in
our campus (NEB/Shaffer, Maryland, Barton).

Linksys WRT54G 802.11b/g routers,
BCM947XX radios, omni-directional
antennas, 16 MB RAM, 4 MB ﬂash memory,
200 Mhz CPU.

Available as open-source software
(smesh.org).

1 2
Node 5 routing rules
3 Source Destination Ne
Node 1 client 1 6
… …
4 Node 2 client 1 6, 7
5 … …
6

7

Client 1

Fig. 1. The routes to a mobile client (multipath routin
To route, we need to use source based multicast trees.
Multipath routing is not supported by current operating
systems. an overlay network to increase the reliability of
to-end path. End-System-Multicast [14] and Spine

For flexibility in routing we
could use user-space routing,
but...

With user-space routing all the
packets are moved to
application level.
Spines

Spines Spines
User space
Kernel space
Spines

We need flexibility in
routing... but without losing
performance.
We developed a new architecture that maintains the control in
user space, while the data is routed at the kernel level.

Node’s 5 kernel routing tables:
1 2
3 Source Destination Next-Hop(s)
Node 1 client 1 6
… …
4 Node 2 client 1 6, 7
5 … … Node 3
6
Node 2
7 Node 1
Client 1

Fig. 1. The routes to a mobile client (multipath routing). Fig. 2

an overlay network to increase the reliability of the end- space to user space in orde
to-end path. End-System-Multicast [14] and Spines [3] also routing decision is made, th
Each route through an application router to support overlay multicast spaceone for sent on th
node maintains multiple kernel routing tables, where it is
each without infrastructure support.
node in the network. boundary must be crossed
Other work has looked into operating system support for We describe next a me
wireless ad-hoc routing protocols. Chakeres and Belding dundant multipath routing
showed in [9] an in-kernel design and implementation of the is simple: each node maint
ad-hoc AODV protocol using Netﬁlter modules, and showed one for each node in the

One of node’s 5 routing tables:
1 2
3 Source Destination
Node 2
Next-Hop(s)
Node 1 client 1 6
… …
4
5
Node 2 Destination
client 1 6, 7 Next-hops
… …
6 client 1 6, 7
client 2 3
7
... ...
Client 1

Fig. 1. The routes to a mobile client (multipath routing). Fig. 2

an overlay network to increase the reliability of the end- space to user space in orde
Eachto-end path. End-System-Multicast [14] and Spines [3] also
route may have multiple next-hops. routing decision is made, th
route through an application router to support overlay multicast space where it is sent on th
without infrastructure support. boundary must be crossed
Other work has looked into operating system support for We describe next a me
wireless ad-hoc routing protocols. Chakeres and Belding dundant multipath routing
showed in [9] an in-kernel design and implementation of the is simple: each node maint
ad-hoc AODV protocol using Netﬁlter modules, and showed one for each node in the

Some interesting
implementation
details...

1. Encode entry node in the
packet’s IP header.
0 7 8 15 16 23 24 31

Version IHL TOS Total length

Identiﬁcation (IPID) Flags Fragment offset

TTL Protocol Header checksum

Source IP

Destination IP

Options and padding

0 7 8 15 16 23 24 31

Version IHL TOS Total length

Identiﬁcation (IPID)


Source IP

Destination IP

Options and padding

0 7 8 15 16 23 24 31

Version IHL TOS
TOS Total length

Identiﬁcation (IPID)


Source IP

Destination IP

Options and padding

2. Use policy routing and define
multiple routing tables.

# iptables -A PREROUTING -t mangle
-m u32 --u32 "2&0xFFFF=35"
-j MARK --set-mark 35

# ip rule add fwmark 35 table 35

3. Build a kernel module to
deliver packets to multiple next
hops.
Use CONFIG_IP_ROUTE_MULTIPATH kernel option.

# ip route add 10.233.59.169/32 table 35
nexthop via 10.0.11.32 dev eth1
nexthop via 10.0.11.33 dev eth1

# iptables -A POSTROUTING
-t mangle
-j MULTIHOP

A packet path in the kernel..
entry point: set IPID
set TOS

all routers: set fwmark fwmark MULTIHOP module

Routing
NF_IP_PRE_ROUTING NF_IP_FORWARD NF_IP_POST_ROUTING
Decision

Routing
Decision

NF_IP_LOCAL_IN NF_IP_LOCAL_OUT

... and a packet path in the mesh.

Node 5


IPID: 5
TOS set
Node 5


IPID: 5
TOS set

Node 5

IPID: 5
TOS set

Node 5

IPID: 5
TOS set

IPID: 5
TOS set

IPID: 5 Node 5
TOS set

We used three configurations.

1. One router

2. 5 nodes wireless
“line” setup

3. 17 nodes wireless
testbed

Rate 24 Mbps

Transmission power 50 mW

Retransmission limit 7

VoIP stream 64 Kbps

1. Single router
Packets / s (sent + received) Packets / s (sent + received)
100 800 1600 3200 4000 5000 6000 7000 100 800 1600 3200 4000 5000 6000 7000

100
1.0

Overlay routing Overlay routing

80
0.8

60
0.6
CPU load

Loss rate
Native
SMesh kernel
routing

40
0.4

SMesh

20
0.2

Native kernel routing
0.0

0
1 4 8 16 32 40 50 55 60 70 1 4 8 16 32 40 50 55 60 70
Number of VoIP streams (each direction) Number of VoIP streams (each direction)

We route up to 50 duplex VoIP streams, before the CPU
is saturated.
User space overlay routing 4 streams 512 Kbps
SMesh 50 streams 6.4 Mbps
Native kernel routing 60 streams 7.6 Mbps

2. “Line” topology of 5 nodes
TCP Throughput (Mbps)

10.1

With one wireless
5.1 hop, we get 10 Mbps.
3.3
2.7
2.1

2.1 2.1 2.0 1.8 1.8

1 hop 2 hops 3 hops 4 hops 5 hops

Overlay routing SMesh

3. Testbed of 17 nodes
User-space overlay routing SMesh kernel routing
Access Access
Point ID Point ID
10 5 hops 28 10 5 hops 28

4 hops 26 4 hops 26

3 hops 25 3 hops 25

8 24 8 24
2 hops 2 hops
Throughput (Mbps)

Throughput (Mbps)
36 36
33 33
1 hop 32 1 hop 32
6 6
31 31

4 4

2 2

0 0

0 80 160 240 320 0 40 80 120 160
Time (s) Time (s)

Results are close to the “line” topology (8.5 Mbps for one hop).

Application:
Push-To-Talk
System for First
Responders

What is PTT?

Half-duplex communication
system with multiple
participants.
While one person speaks, the
others listen.

Push-To-Talk systems require an arbitration
mechanism (“floor control”) that
determines the order in which participants
speak.

In cellular networks, Push-To-Talk systems are centralized.

We need a robust, distributed
Push-To-Talk system that works
even when

Mesh nodes crash.
The network is partitioned or it merges.

Our Push-To-Talk system allows regular
phone users to remotely join a PTT
session established inside the mesh!

Mobile Client with VoIP Software

SIP RTP

PTT Controller
Distributed RTP Proxy Mobile, Fault-Tolerant
SIP 3PCC DTMF Voice
Floor
Monitor
Management

Mobile Client
Mesh Node

State sip_call_id, PTT Session
sip_cseq, rtp_port, Manager
ptt_group, ptt_state

Client PTT PTT
PTT Data
Control CMonitor Controller
Group
Group Group Group

Routing Daemon (Discovery, Topology Management, Group Management)

Wireless Mesh Network

Interface with Mobile Client

We use the standard Session Initiation Protocol (SIP) to interact
with users.
The entire mesh is seen by the user as a single, distributed third
party call controller (3pcc).

A user interacts with the system
using a VoIP application.

sip : ptt @ 192.168.1.10
How to connect:
(that’s a virtual SIP server)

How to join a group: type # 12 #

How to request to speak: type 5

How is notiﬁed when he
receive a “beep-beep” audio signal
has permission to speak:

We use multicast groups to
manage the client and the PTT
sessions.

These are overlay multicast groups.

Client management

Session management

Client management

Control
Group

Session management

Client management

Control
Group

Session management

Controller
Group

Client management

Control
Group

Session management

Controller
Group

Monitoring
Group

Client management

Control
Group

Session management

Controller
Group

Monitoring
Group

Data
Group

How are
users’ requests
arbitrated?

Sending Mesh
Controller
client node

Sending Mesh
Controller
client node

Sends “Request to speak”
to the access point.

Sending Mesh
Controller
client node

Floo
r Re
Sends “Request to speak” Con ques
t
trol
to the access point. ler g
roup

Sending Mesh
Controller
client node

Floo
r Re
t
trol
roup

k
e st Ac
Requ
st
u nica

Sending Mesh
Controller
client node

Floo
r Re
t
trol
roup

k
e st Ac
Requ
st
u nica

k
to Spea
sion up
Pe rmis ol gro
C ontr
Receives “Permission lient
Mesh C
granted” from the node
access point.

This can be a different node!

Sending Mesh
Controller
client node

Floo
r Re
t
trol
roup

k
e st Ac
Requ
st
u nica

k
to Spea
sion up
Pe rmis ol gro
C ontr
Receives “Permission lient
Mesh C
granted” from the node
access point. PTS
Ac k
unic
This can be a different node! ast

The controller is rotated periodically,
according to participants’ locations in
the network.

We want to maintain the controller node in the “center of
gravity” of the nodes handling PTT clients on a certain group.

Mesh
Controller
node

When a “better” controller is available
stop handling and queueing requests.

Mesh
Controller
node

stop handling and queueing requests. INVI
TE (
Grou
p, Q
unic ueue
ast )

Join Controller group
Join Monitoring group

Start handling
requests

Mesh
Controller
node

stop handling and queueing requests. INVI
TE (
Grou
p, Q
unic ueue
ast )

Join Controller group
Join Monitoring group

Start handling
)
requests
(G roup
k
TE Ac
INVI st
u nica

Controller change succeeded

Leave Controller group
Leave Monitoring group (if necessary)

The system needs to withstand
node crashes, network partitions
and merges.

Sending
Network Controller
node

Sending
Network Controller
node

Sending
client

PING_CMON Sending
Network Controller
Monitoring group node

On timeout: Controller is lost

Join Controller group (if lowest IP).
Start handling requests.

Sending
client

PING_CMON PTS_PING (client) Sending
Network Controller
Monitoring group Controller group node

On timeout: Controller is lost On timeout: Sending node is lost

Join Controller group (if lowest IP). Handle the next client in the
Start handling requests. queue.

Sending
client

PING_CMON PTS_PING (client) Sending
Network Controller
Monitoring group Controller group node

On timeout: Controller is lost On timeout: Sending node is lost

Join Controller group (if lowest IP). Handle the next client in the Voice
Start handling requests. queue.

Sending
client

In summary,

We instantiate a controller for each PTT group that arbitrates the
requests on that group.
The controller is monitored and rotated if a more suitable node is
available.
We achieve high availability with a monitoring mechanism that uses
overlay multicasts groups.

Nodes 14

Rate 18 Mbps

Transmission power 50 mW

Retransmission limit 7

VoIP stream 64 Kbps

Speak duration 20 sec

We evaluated the system when

1. The network is stable (normal operation).

2. The number of users in a PTT group increases.

3. The number of groups increases.

4. Large-scale scenario (40 clients, 10 groups, network partitions
and merges).

1. Normal operation
Throughput

80 Kbps

Sender node 2 Sender node 12 Sender node 14

14 nodes,
60 Kbps

40 Kbps 4 users, each speaks
for 20 seconds.
20 Kbps
Overhead

0 Kbps

0 10 20 30 40 50 60 70

Time (s)

Sender node 2 Sender node 12

21.7 21.8 21.9 22.0 22.1 22.2 22.3 22.4

Packet Arrival time (s)

2. Scalability with the #clients
Average latency Average loss rate
1.2 %
120 ms 114

1 %
100 ms 0.92

0.8 %
80 ms No PTT support

60 ms 0.6 % No PTT support

40 ms 0.4 %
27
Single radio 28 Single radio
23 26
0.23
18 16 21 0.18 0.19
25 0.2 %
20 ms 17
21
19
16 Dual radio 0.08 0.09 0.1
0.15
0.05 0.13
Dual radio
0 ms 0 %
2 4 8 14 28 42 2 4 8 14 28 42
Number of clients Number of clients

Using dual radios, it scales to at least 42 users in a single PTT
session, in our testbed.

3. Scalability with the #groups
Average latency Average loss rate
2500 ms 20 % Dual radio
Single radio 18
2227

2000 ms Single radio
15 % 14
Dual radio Dual radio+
1541
1500 ms packing
1270
10 % Single radio+ Dual radio+
packing packing
1000 ms 6
6

Single radio+ 5 %
500 ms 360 packing 2
2
244 1
178 183 1
0 0
74 53 0 0 0
24 26 28 24 26 30 37
0 ms 0 %
1 2 4 6 8 10 12 14 16 18 20 1 2 4 6 8 10 12 14 16 18 20
Number of groups Number of groups

Using dual radios and packing, it scales up to 18 groups, with 4
users in each PTT group.

4. Large-scale scenario
(10 PTT groups, 4 clients on each group)

Data & overhead trafﬁc:
5000
Data
Traffic (kbps)

4000
C D F
PTT control traffic
Routing control traffic
3000
A B E
2000
1000
G
0
0 50 100 150 200 250 300 350 400 450
Time (s)

Overhead trafﬁc only:
100
C D F
PTT control traffic
Traffic (kbps)

80
A B E
Routing control traffic
60
40 G
20
0
0 50 100 150 200 250 300 350 400 450
Time (s)

(A) clients join (D) the network partitions
(B) clients request to speak (E) stable network after partition
(C) regular operation (F) the network merges
(G) clients stop speaking

In conclusion, we presented

The architecture of the ﬁrst high-throughput wireless mesh
network with fast handoff using off-the-shelf routers.

The ﬁrst robust Push-To-Talk service for wireless mesh
networks.

Turning research ideas into
practical systems is not an
easy task, but it’s definitely
a lot of fun.

Wireless Mesh Networks
is a promising technology.

Killer applications are yet to come. They may
require infrastructure support.

The SMesh system is
available as open-source at
http://smesh.org
Try it for yourself and let us know what you think! :-)

Practical Wireless Mesh Networks and Their Applications

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Practical Wireless Mesh Networks and Their Applications

Similar to Practical Wireless Mesh Networks and Their Applications (20)

Recently uploaded

Recently uploaded (20)

Practical Wireless Mesh Networks and Their Applications

Editor's Notes