NFV technology is moving from a PoCs and trails stage to real live deployment. One of the major gaps of OPNFV with the existing technologies is the OA&M toolset offering. These toolsets are required to provide the carrier grade class of service the customers expect. In this talk we will highlight the status of fault management in NFV, describe in more details the collected requirements of "Pinpoint", OPNFV fault management project. We will also describe the first use case we have implemented - providing dynamic Root Cause Analysis (RCA). We will introduce A-CORD project, and present how we have integrated "Pinpoint" into the A-CORD architecture.

1. June 20–23, 2016 | Berlin, Germany

2. Providing Root Cause Analysis to OPNFV Using
Pinpoint -the A-CORD Architecture Use Case
Nir Halachmi, Huawei

3. NFV Fault Localization Is Challenging
>>>
>>>
>>>
>>>
Physical switch
down
Switch
port down
Host’s NIC
Down
Damaged
host cable
OVS crash
Hypervisor
crash
MTU
misconfigure
L2 agent
crash
IP address
misconfigure

4. NFV Fault Localization Vision
Automated process of pinpointing the exact source of failure (RCA) from a set of
observed indications in the domain of NFV, SDN & Future Networks
A set of algorithms
A set of APIs
Extendable compute and
storage

5. Immediate Business Benefits
• Carrier Grade Cloud Environment - Carriers are required to ensure the
move to a virtualized environment but maintain the quality and
performance of the legacy networks
• Reduce OPEX of the NFV environment- There is lack of operational
tools to support the NFV technology adoption
• Shorter Mean Time To Repair (MTTR) - Promote the introduction of
network fault localization and additional monitoring and SLA assurance
applications.

6. Pinpoint Project Overview
Project Goal: Provide a fault
localization framework for NFV
Deduce the exact source of the failure
from a set of observed failure indications
and reduce the analysis time (RCA)
Pinpoint is aligned with ETSI-NFV -
REL 001 V1.1.1 - Resiliency
RequirementsNeutron/
Nova
Ceilometer/
Monasca
Neutron/
Nova
Neutron/Nova
SDN Controller
User / System
Fault Localization System
(set of analysis methods)
Find root cause(s)
Find correlated failures
root cause(s)
correlated failures
Set test
Get test-info
Get info Get info
Set config
Get info
System OAM tools Fault/ Performance System configuration System models
• Events
• Alarms
• Statistics
• Logs
Expected/desired
Configuration as known
By the CMS
• Layering
• Dependencies
• Topology
• Connectivity
Active tools like
ping, trace etc.
>>>
>>>
>>>

7. What is A-CORD?
Analytics platform designed
to support multiple analytic
applications for the CORD
architecture
Real time feedback - Allow analytic
apps to customize measurement as
well as modify appropriate flows
and VNFs using ONOS + XOS
Supports multiple
apps from multiple
vendors

8. Pinpoint Integration Into A-CORD
Dynamic Root Cause Analytics based on A-CORD Architecture
Customer Care
(app)
Security Diagnosis RCA
Pinpoint gets all the events from
ceilometer as well as
configuration ,topology and
layering information and writes
inferred events back to
ceilometer
RCA application gets events in the
standard A-CORD events distribution
mechanism and performs alarms
suppression
Analytic
Apps
A-CORD Analytics Platform
(XOS + Services)
ROADM
(Core)
Apps Apps Apps
ONOS + XOS + OpenStack
LOG Pinpoint
GPON
(Access)
ROADM
(Core)

9. NFV Fault Localization In Action
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
App App App App App App App App App App App App
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
1 2 3 4 5 6 7 8 9 10 11 12
Host 3 NIC
down
OVS 3 is
down
VMs 7,8,9
disconnected
Apps 4,5
disconnected
1 2 3 4 5 6
1 2 3 4
Apps 4 & 5 disconnected
Host 3 NIC Down
OVS 3 is down
VMs 7,8,9 disconnected
Host 3 NIC failure

10. Fault Localization Using Dynamic OAM Tools
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
V
M
App App App App App App App App App App App App
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
1 2 3 4 5 6 7 8 9 10 11 12
VMs 7,8,9
disconnected
VM 13
disconnected
Trace route to
Router 2
1 2 3 4 5 6
1 2 3 4
Host 3 NIC Down
OVS 3 is down
VMs 7,8,9 disconnected
Link failure between Router 2 and NIC 3
VM 13 disconnected
OAM tool OAM tool
Trace route to
NIC 3

11. Summary
Health Management, Reliability, High-Availability and Resiliency are
critical to the success of NFV
The complexity of the Cloud Environment and the transition to NFV
requires a new set of tools and methods to ensure carrier grade service
Pinpoint is the framework to build Fault Localization/Root Cause
Analysis capabilities
Community is welcomed to join the OPNFV Pinpoint project