VoWiFi testing challenges

1Spirent Communications PROPRIETARY AND CONFIDENTIAL
VoWi-Fi Testing
Challenges
Measuring the
Quality of Experience
in the field and lab

Agenda
Part 1: VoWi-Fi technology landscape
 Drivers for Wi-Fi adoption
 VoWi-Fi QoE Challenges
Part 2: SRG VoWi-Fi field-based study results
 Wi-Fi study background and motivation
 Study setup and measurement methodology
 VoWi-Fi Field test results
Part 3: Evaluating VoWi-Fi performance in the lab
 Advantages of lab-based testing
 Evaluating Key Performance Indicators (KPIs) in the lab
 Future deployment and test considerations
 Audience Q&A

Drivers for Wi-Fi adoption
Why Wi-Fi is an attractive choice as a technology

VoWi-Fi QoE Challenges
Offloading Algorithms
• When and why?
• When to go back?
Assessing Signal Strength and Integrity
• RSSI measurements before and
during offload
IP Layer
• Best effort delivery
• Access Point variations
Authorization
• ePDG authentication for
unsecured Wi-Fi access

KPIs used to measure VoWi-Fi Quality of Experience (QoE)
 Voice Quality:
 MOS Score – Uplink and Downlink
 Packet (Audio) loss percentage
 Conversational (Ear-to-mouth) delay
 Call reliability:
 Percentage of successful handovers in both directions under “expected” circumstances
 Percentage of call drop under various ePDG adversarial scenarios (lab-based simulation)
 Percentage of successful handovers when access point is loaded
 Time taken for call setup (for Wi-Fi originated calls)
 Time taken for LTE<-> Wi-Fi handover

 Audience Q&A

Background and rationale for the study
 Spirent and SRG have collaborated on all-things-VoLTE for the last 15 months
 VoWi-Fi is the rage
 Although operators are rushing to embrace the technology – sometimes in
advance of VoLTE – there isn’t a clear understanding how the technology
performs
 VoWi-Fi is a natural extension of VoLTE, given the interoperability that should
exist between the two voice solutions – also doesn’t hurt that HD Voice is
supported across all 3 RAN components of the operator’s network

Study setup and test methodology
 Spirent Communications provided Nomad User Experience system
 Accuver X-Cal M data collection tool and Accuver XCAP post-processing software
for logging all lower layer parameters
 The testing relied on mobile-to-mobile calls – implies that the uplink MOS of one
device can affect the downlink MOS of the other
 Testing included VoLTE-VoWi-Fi handovers and vice versa, as well as call quality
with different device configurations (3G only, Wi-Fi- only, etc.)
 VoWi-Fi voice quality measured under realistic traffic loading at residential and
public Wi-Fi access points

Procedures:
 Introduced additional background traffic onto same Wi-Fi access point while performing voice quality test
Observations:
 Streaming HD Video (2x) had very little impact on voice quality when the downlink consisted of two video
streams
Field test results: residential Wi-Fi access point
How does the
performance look
when a device using
VoWi-Fi calling
on a residential Wi-Fi
access point is loaded
on the downlink

Field test results: residential Wi-Fi access point
How does the
performance look
when a device using
VoWi-Fi calling
on a residential Wi-Fi
access point is loaded
on the uplink
Procedures:
 Transferred emails with ~10 MB files attached over same Wi-Fi AP being use to support voice calls
Observations:
 Relatively modest amounts of uplink traffic can impact call quality, including no voice detection
 As implemented, VoWi-Fi does not support QoS so all traffic is treated as best effort

Field test results: public Wi-Fi access point
How does the
performance look
when two
smartphones are
using VoWi-Fi
in a public Wi-Fi
access point
Procedures:
 Stationary testing at airport with both smartphones using VoWi-Fi and the same SSID
– network seemed to be lightly loaded
Observations:
 Bigger variances in the results and lower scores, despite lightly-loaded network
– VoWi-Fi on both ends increases impact
 Quality of network has a major influence on call quality

Field test results: public Wi-Fi access point
How does the
performance look
when a device
using VoWiFi calls
another device on
VoWiFi on a public
Wi-Fi Access point
Procedures:
 Comparable to airport test but done at San Diego Convention Center – 18,000 attendees for a conference
Observations:
 Less variance in the results compared with the airport test
 Quality of network has a major influence on call quality – in this case a good thing

Field test results: VoWi-Fi to VoLTE handovers
How does the
performance look
when a device using
VoWi-Fi hands over
from a residential
Wi-Fi Access Point
to VoLTE
Procedures:
 Stationary device, which is placed near Wi-Fi AP, uses VoLTE; Second device was mobile – started on VoWi-Fi
 Walked down the street pushing the “little red wagon” until the call switched to VoLTE
 Returned back to Wi-Fi AP and waited for handover to Wi-Fi before walking again of Wi-Fi coverage
Observations:
 No impact to MOS Score during handover; also hard to detect handover by listening to the call
 Handovers back into VoWi-Fi take longer (no surprise) while outbound handovers aren’t predictable

Field test results: VoWi-Fi to VoLTE handovers
How does the network
and device activity
look when a device
switches between
VoLTE and VoWi-Fi
Procedures:
 One device configured for 3G voice only; second device configured for VoLTE / VoWi-Fi
 Originate call within Wi-Fi AP coverage and then proceed to walk out of coverage with both phones in hand
 Continue until handover occurs and then walk back into coverage; repeat as necessary
Observations:
 Everything consistent with expectations; lack of QoS with VoWi-Fi is evident

Field test results: VoWi-Fi to VoLTE handovers, II
How does the
network and device
activity look when a
device switches
between VoLTE and
VoWi-Fi
Procedures:
 One device (S5) configured for 3G voice only; second device (S6) configured for VoLTE / VoWi-Fi
 Originate call within Wi-Fi AP coverage and then proceed to walk out of coverage with both phones in hand
 Continue until handover occurs and then walk back into coverage; repeat as necessary
Observations:
 S5 used UMTS as intended but S6 didn’t use VoWi-Fi, instead using a combination of UMTS and 2G (NB-AMR)
 Call drops when we returned to Wi-Fi coverage (repeated with same results)

Key observations
 The benefits of VoWi-Fi are clear but it isn’t a panacea
 VoWi-Fi is as good or as bad as the underlying Wi-Fi network
 VoWi-Fi handovers with the LTE network generally worked but there were
some exceptions
 Problems, which we attribute to network/IMS registration and device
interoperability, exist

 Audience Q&A

Advantages of evaluating VoWi-Fi in the lab
 Audio quality issues related to low MOS scores, audio (packet) loss and codec
(AMR-NB/AMR-WB/EVS) negotiation failure
 SIP registration issues
 ePDG authentication failure
 Handover at power levels higher/lower than that mandated by operator test plans
 Poor QoE resulting from UE toggling between LTE – Wi-Fi networks
 Does not require a wagon

Advantages of evaluating VoWi-Fi in the lab
VoLTE
VoWi-FieAP AKA
IPSec tunnel
VoWi-Fi
VoWi-Fi

Evaluating VoWi-FI KPIs in the lab
Test scenario Measurable KPIs QoE impact
Voicequality
Test voice quality when the Wi-Fi
power is in the edge condition of
making a decision on handover
from LTE-Wi-Fi or vice-versa and
is toggling between the two
 MOS Score
 Packet loss percentage
 Network latency
 Jitter
 Lower MOS Score => lower
voice quality
 High packet loss =>
Intermittent drops in speech
 High network latency =>
Variable delays in speech
Test voice quality in the
presence of loading on the Wi-Fi
access point and tolerance of
UE to larger network delays
Test voice quality of two test
devices against a reference
“soft” client, both on the Uplink
and the Downlink

Evaluating VoWi-Fi KPIs in the lab
Test scenario Measurable KPIs QoE impact
Callreliability
Test the reliability of the call
through the handover process
and time for handover by control
of Wi-Fi power over multiple
iterations
 Percentage of successful call
completion
 Handover time from
LTE – Wi-Fi
 Larger handover time =>
periods of voice discontinuity
 Lower percentage of call
completion => Increased
dropped calls during handover
 Improper response to ePDG
adversarial scenarios => call
drops during handover
when the UE is sent adversarial
responses by ePDG
through the handover process
and time for handover by control
of Wi-Fi power over multiple
iterations

Test case 1
Test voice quality
between two
devices Device A
and Device B and
measure MOS
score on both
Uplink and
Downlink
Result
Low MOS
score
measured
on Uplink of
Device A
Test Case 1: Voice Quality
Low MOS score on Downlink of Device B in device-to-device call
traced to low Uplink MOS score on Device A

Test case 2
ePDG
adversarial
testing scenario
Result
LTE-Wi-Fi
Handover
failure
UE fails to handover and drops call when ePDG issues error code
Test Case 2: ePDG responses

Test case 3
Test voice quality when the UE is
toggling between Wi-Fi and
LTE network
Result
Lower MOS Score
Test Case 3: Handover
Packet loss
Root cause identified: Packet loss as a result of device “toggling” between LTE-Wi-Fi

Test case 4
Call initiation on Wi-Fi
originated call
Result
Low MOS score
Test Case 4: Codec negotiation
UE1 supporting AMR-WB UE2 supporting only AMR-NB
Root cause identified: Disagreement of codec between UE1 and UE2

VoWi-Fi test capabilities and requirements

Bringing field study into the lab
SRG and Spirent partner on evaluating VoWi-Fi QoE in the lab
 LTE EPC Configuration
 IMS registration call flow
 Wi-Fi authentication
 ePDG Adversarial test condition
 IMS test agent simulation

Additional VoWi-Fi QoE considerations
 Service request type
 Wi-Fi / LTE power level
 Capacity/loading on Wi-Fi radio link
 Measuring impact on voice quality while introducing fading and noise on radio
interface of Wi-Fi
 Testing audio quality and call performance with different classes of access points
(Home, Enterprise, public Wi-Fi hotspots)
 Testing performance in the presence of introducing IP impairments on the IP
backhaul
 Testing the impact of performance in the presence of interference from other
adjacent bands such as LTE and/or adjacent Wi-Fi hotspots
 ANDSF, Hotspot 2.0…

 Audience Q&A

Questions?

VoWiFi testing challenges

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to VoWiFi testing challenges

Similar to VoWiFi testing challenges (20)

Recently uploaded

Recently uploaded (7)

VoWiFi testing challenges