CGE IVT Master Test Plan

Date Modified: August-18-2014

CGE IVT Master Test Plan
Copyright 2014 Cisco Systems 1
Document Number EDCS-1291267, Rev 14
Based on Template EDCS-206096 Rev.40
Created By Anuj Dewangan (adewanga) –
TME (IOTG), Malay Thaker
(mathaker) – PLM (IOTG)
Project Cisco DevNet – Cisco Enabled
Grid Devices
Reviewers/Approvers
Department Name/Title
IOTG Product Management/Business
Development
Patrick Grossetete (pgrosset)/TME, Marty Collins
(mcollins)/TME, Gaurav Agarwal
(gauraaga)/TME, Felix Davis (felixdav)/PLM,
Brian Mansfield (bmansfie)/PM, Nitin Nayar
(nnayar)/PM, Joe Huber (joehuber)/BDM
IOTG Engineering (Development) Wei Hong (wehong)/Director.Engineering,
Jonathan Hui (johui)/Technical Leader, Phil
Buonadonna (pbuonado)/Technical Leader,
Chris Wilson (chwilso3)/Hardware Engineer,
Yuwen Lan (yulan)
IOTG Engineering (Support/Test) Arun Annavarapu (aannavar)/Software Engineer,
Jonathon Paul (jopaul)/Technical Leader, Linda
Yang (linyan)/Technical Leader


Modification History
Revision Name User Id Date Comments
0.1 Anuj Dewangan adewanga 09/11/2013 Added test cases without detailed
procedure
0.2 Anuj Dewangan adewanga 09/18/2013 Added test case details + edits
1.0 Anuj Dewangan adewanga 09/19/2013 Malay edits + Added for Internal
Review (EDCS-1291267 Rev 8)
1.1 Anuj Dewangan
Malay Thaker
adewanga
mathaker
12/04/2013 • Review comments
• Additional details in test
cases based on
requirements document
1.2 Anuj Dewangan
Malay Thaker
adewanga
mathaker
01/23/2014 • Re-arranged sections
• Added IVT recertification
requirements
1.3 Anuj Dewangan adewanga 01/31/2014 • Added new test case
CGE_IVT_APP_TC010
2.0 Anuj Dewangan adewanga 05/16/2014 • Added multi-hop PAN
migration topology
• Added requirement for pre-
IVT testing as entry criteria
for IVT
• Added test cases for two
phases of certification:
A] Pre-IVT Testing
B] IVT Testing
3.0 Linda Yang
Anuj Dewangan
linyan
adewanga
08/18/2014 • Added dual-phy topology
and test cases
• Added PLC test cases for
modulation and LMAC OTA
• Modifications to PON/PRN
test cases
• Added test case to verify
high QoS markings are not
present
• Added test case to verify
maximum 800 byte L3
application packet size


• Modified test case to verify
relative Tx-power versus
iTron meters and Tx-power
configurability
(CGE_IVT_SIG_TC001)
• Removed QoS
considerations, as they are
part of the requirements
document
• Updated content for dual-phy
design


Table of Contents
1
Introduction............................................................................................................................................ 7

1.1
Connected Grid Endpoints and Field Area Network....................................................................... 7

1.2
System Development Kit (SDK) and Cisco DevNet ....................................................................... 9

1.3
CG-REDI and partner development infrastructure ....................................................................... 11

2
Test Coverage..................................................................................................................................... 13

2.1
What Will Be Tested..................................................................................................................... 14

2.2
What Will Not Be Tested .............................................................................................................. 15

3
Test Setups ......................................................................................................................................... 16

3.1
MCU Diagnostics Setup ............................................................................................................... 16

3.2
One CGE in a PAN....................................................................................................................... 17

3.3
Four (at least) CGEs in a PAN ..................................................................................................... 17

3.4
Large scale, multi-hop, multi-vendor CGE PAN........................................................................... 18

3.5
Multi-hop PAN Migration .............................................................................................................. 18

3.6
Four (at least) multi-vendor CGEs in a PAN ................................................................................ 19

3.7
Multi-phy CGEs in a PAN............................................................................................................. 19

4
Test and Certification Approach and Considerations.......................................................................... 21

4.1
Test Phase Entry Criteria ............................................................................................................. 21

4.2
IVT Considerations specific to a certification................................................................................ 22

4.3
Test Execution tracking and reporting.......................................................................................... 23

4.4
Defect Tracking ............................................................................................................................ 23

4.5
Requirements Traceability Approach ........................................................................................... 23

4.6
IVT Certification Elements............................................................................................................ 23

4.7
IVT Recertification criteria ............................................................................................................ 24

4.8
IVT Certification Numbering scheme............................................................................................ 25

4.9
IVT Certification Report................................................................................................................ 25

4.10
IVT Deliverables......................................................................................................................... 26

4.11
Test Exit/Certification Criteria..................................................................................................... 27

4.12
Performance test (PR) and Scalability test (SC) ........................................................................ 27

4.13
Pre-IVT Testing Matrix ............................................................................................................... 28

4.14
IVT Testing Matrix ...................................................................................................................... 31

5
Test Resources ................................................................................................................................... 36

5.1
People .......................................................................................................................................... 36

5.2
Hardware...................................................................................................................................... 36

5.3
Firmware ...................................................................................................................................... 36

5.4
Software ....................................................................................................................................... 37

5.5
Test Equipment ............................................................................................................................ 37

5.6
Test Tools (software).................................................................................................................... 37

6
Risks, Mitigations, and Contingencies................................................................................................. 39

7
Detailed Test Cases (Pre-IVT) ............................................................................................................ 40

7.1
CGE HW Schematics and diagnostics (Pre-IVT): ........................................................................ 40

7.2
Workforce enablement (Pre-IVT): ................................................................................................ 41

7.3
CG-Mesh Connectivity (Pre-IVT):................................................................................................. 45

7.4
CG-Mesh Security (Pre-IVT): ....................................................................................................... 54

7.5
Remote Management and Notifications (Pre-IVT) ....................................................................... 56

7.6
CGE Application (Pre-IVT): .......................................................................................................... 63

7.7
Fault resiliency and availability:.................................................................................................... 70

8
Detailed Test Cases (IVT) ................................................................................................................... 73

8.1
CGE HW Schematics and diagnostics (IVT):............................................................................... 73

8.2
RF/PLC physical and MAC layer (IVT):........................................................................................ 74

8.3
Workforce enablement (IVT): ....................................................................................................... 78

8.4
CG-Mesh Connectivity (IVT): ....................................................................................................... 83

8.5
CG-Mesh Security (IVT):.............................................................................................................. 88


8.6
Remote Management and Notifications (IVT): ............................................................................. 92

8.7
CGE Application (IVT):................................................................................................................. 98

8.8
Fault resiliency and availability (IVT):......................................................................................... 105

8.9
Soak and Performance/Stress: .................................................................................................. 107

9
References ........................................................................................................................................ 110


Table of Figures
Figure 1.1 Protocol Stack for Connect Grid Endpoint SDK – IEEE 802.15.4g/e ......................................................... 7

Figure 1.2 Protocol Stack for Connect Grid Endpoint SDK – IEEE 1901.2 PLC......................................................... 8

Figure 1.4 CGE SDK – NIC (PPP) model ................................................................................................................... 10

Figure 1.5 CGE SDK – Integrated Execution (IE) model ........................................................................................... 10

Figure 1.6 CG-REDI architecture for partner development......................................................................................... 11

Figure 1.7 Partner Premises – “Thin Starter Kit” topology ......................................................................................... 12

Figure 3.1 IVT Topology 1: CGE diagnostics............................................................................................................. 16

Figure 3.2 IVT Topology 2: 1 CGE in a PAN ............................................................................................................. 17

Figure 3.3 IVT Topology 3: 4 CGEs in a PAN............................................................................................................ 17

Figure 3.4 IVT Topology 4: Large scale, multi-hop, multi-vendor CGE PAN........................................................... 18

Figure 3.5 IVT Topology 5: Multi-hop PAN Migration.............................................................................................. 18

Figure 3.6 IVT Topology 6: 4 CGEs in a multi-vendor CGE PAN............................................................................. 19

Figure 3.7.1 IVT Topology 7.1: 4 CGEs in a multi-Phy CGE PAN........................................................................... 19

Figure 3.7.2 IVT Topology 7.2: 4 CGEs in a multi-Phy CGE PAN........................................................................... 20

Table of Tables
Table 4.6.1: IVT elements:........................................................................................................................................... 23

Table 4.14.1: CGE diagnostic tests (Pre IVT): ............................................................................................................ 28

Table 4.14.2: Workforce enablement (Pre IVT):......................................................................................................... 28

Table 4.14.3: CG-Mesh Connectivity (Pre IVT): ........................................................................................................ 28

Table 4.14.4: Mesh Security (Pre IVT):....................................................................................................................... 29

Table 4.14.5: Remote Management and Notifications (Pre IVT):............................................................................... 29

Table 4.14.6: CGE Application (Pre IVT):.................................................................................................................. 30

Table 4.14.7: Fault resiliency and availability (Pre IVT): ........................................................................................... 30

Table 4.15.1: CGE HW Schematics and diagnostics (IVT): ....................................................................................... 31

Table 4.15.2: RF/PLC physical and MAC layer (IVT):............................................................................................... 31

Table 4.15.3: Workforce enablement (IVT): ............................................................................................................... 31

Table 4.15.4: CG-Mesh Connectivity (IVT):............................................................................................................... 32

Table 4.15.5: Mesh Security (IVT):............................................................................................................................. 32

Table 4.15.6: Remote Management and Notifications (IVT): ..................................................................................... 33

Table 4.15.7: CGE Application (IVT): ........................................................................................................................ 33

Table 4.15.8: Fault resiliency and availability (IVT):.................................................................................................. 34

Table 4.15.9: Soak and Performance/Stress (IVT): ..................................................................................................... 35

Table 5.2.1: Hardware requirement for certification: .................................................................................................. 36

Table 5.3.1: Firmware requirements for certification:................................................................................................. 36

Table 5.4.1: Software requirements for certification: .................................................................................................. 37

Table 5.5.1: Test equipment requirements for certification:........................................................................................ 37

Table 5.6.1: Test tool requirements for certification: .................................................................................................. 37

Table 6.1: Risks............................................................................................................................................................ 39


1 Introduction
This document serves as the Master Test Plan for inter-operability verification testing (IVT) of
Connected Grid Endpoints (CGE) developed by Cisco Connected Grid ecosystem partners, engaged
through the Cisco-Enabled Grid Devices Technology Center within Cisco DevNet (henceforth,
referred to as ‘DevNet’). It defines the scope, approach, resources, schedule, risks/mitigations,
entry/exit criteria and certification requirements for the IVT performed on the CGE and the Field Area
Network implementation. Detailed test plans for each IVT certification testing for a CGE partner will
be derived from this document and will add or remove test cases based on implementation specifics.
This plan, including test cases, will be reviewed and approved to assure completeness of the testing
and to determine the testing schedule. Once testing starts all test cases can be managed within a test
case management system. If major changes to testing scope are made, once execution starts, then
the test plan, including test cases, will be re-reviewed and approved. The test results review will
become the final validation of the test plan along with any other exit criteria.
1.1 Connected Grid Endpoints and Field Area Network
Cisco supports two major communication technologies in the connected grid endpoint space for
the last mile: Radio Frequency (RF) mesh and Power Line Communications (PLC) mesh.
Vendor partners also have an option to create an endpoint supporting both the technologies.
Such an endpoint is referred to as ‘Dual-phy’ (as a reference to having two different physical
and MAC layers for communication, but a single layer 3 instance).
Cisco’s implementation of these technologies is based on and driven by industry standards. The
following diagrams illustrate the protocol stack implemented for each of the two technologies
and the associated standards:
Figure 1.1 Protocol Stack for Connect Grid Endpoint SDK – IEEE 802.15.4g/e


Figure 1.2 Protocol Stack for Connect Grid Endpoint SDK – IEEE 1901.2 PLC
A Cisco Connected Grid Endpoint (CGE) needs to implement one or both of the above protocol
stacks and needs to be appropriately configured to be able to join and communicate with a
Neighborhood Area Network (NAN) rooted at a Cisco’s Connected Grid Router (CGR) 1000
series.
A CGE connected to a NAN/CG mesh (RF, PLC or dual-phy) is capable of end-to-end layer 3
communication using IPv6. When a CGE attempts to join a CG-Mesh network, it is required to
authenticate itself to the network, get an IPv6 address along with options and prefix-delegation if
required, register itself to network management services using CoAP Simple Management
Protocol (CSMP), and communicate with required application servers (collection engines,
SCADA master, MDMS etc) to deliver grid functionalities. Typically the network services are
hosted in a utility Network Operations Center (NOC). The following topology diagram shows the
end-to-end solution architecture of a Field Area Network (FAN) with the Wide Area Network
(WAN) and Neighborhood Area Network (NAN) tiers:
Figure 1.3 Multi-Service Field Area Network Architecture with tiers:


In the diagram above, the Cisco Connected Grid Router (CGR) 1000 Series acts as a Field
Area Router (FAR). Each FAR advertises a unique Personal Area Network (PAN) – recognized
by a combination of a SSID and PAN ID. CGEs are programmed to join a PAN with a given
SSID. CGEs can migrate between PANs based on a set of metrics for the PAN (very rarely) and
for fault tolerance.
The testing scenarios and use-cases described in this document are focused on the NAN tier
and also deal with interactions between head-end elements and the CGEs.
1.2 System Development Kit (SDK) and Cisco DevNet
Cisco supports integration of Cisco’s endpoint technologies in partner devices through a System
Development Kit (SDK). Cisco provides the following to a CGE partner to enable the
development of CGE devices that interact with the NAN:
• HW Reference design
• Bill of Materials (BOM)
• SDK firmware binary
• CG-REDI cloud services
• Reference guides and training material
• Requirements, guidelines and best practices document
• CGE IVT Master test plan (this document)
The delivery of the elements listed above is handled through the Cisco Connected Grid DevNet
program. More specifically, the development of CGE is supported through the ‘Cisco-Enabled Grid
Devices’ Technology Center within the DevNet program.
Cisco provides several versions of Hardware reference designs to its partners for development along
with a BOM. For RF, the hardware reference designs are CGEREF1 (100-pin reference design,
formerly known as ‘Banshee’. This reference design is now deprecated.), CGEREF2 (64-pin
reference design) and CGEREF3 (144-pin reference design). For PLC implementations, the hardware
reference designs are CGEPLCREF1 (100-pin reference design, formerly known as ‘Alamo’. This
reference design is now deprecated), CGEPLCREF2 (64-pin reference design) and CGEPLCREF3
144-pin reference design). It is recommended to implement either the 64-pin or the 144-pin variants
of the MCU for either of the communication technologies. Newer features and firmware will only be
released for these two variants, and firmware for the 100-pin MCU will not receive these updates in
the future.
Along the HW reference design, the partner may choose to develop their endpoints using one of the
two supported endpoint software models:


Figure 1.4 CGE SDK – NIC (PPP) model
The NIC (PPP) model, also known as dual MCU model, has Cisco’s Reference design being
implemented as a Network Interface Card (NIC) or Communications Module (CM) providing CG
communication services to the CGE. The partner also develops an Application Module (AM) that
communicates with the CM over a PPP link. The AM, typically along with a micro-kernel Operating
System (OS), hosts the partner AMI/DA application like DNP3 client, ANSI C12.22 with C12.19,
DLMS/COSEM server etc. Cisco provides the firmware binary for the CM, which the partner installs
and runs on the CM MCU.
Figure 1.5 CGE SDK – Integrated Execution (IE) model


The Integrated Execution (IE) model implements a single Micro-controller Unit (MCU), which runs
both the communications stack as well as the partner application. Cisco provides a firmware binary
OS kernel with integrated protocols stack. The firmware also exposes APIs for communication with
MCU peripherals (GPIO and I2C). The partner develops applications using the APIs to communicate
with the sensor devices connected as peripherals. The application along with the OS kernel is
packaged into a single image for deployment.
Note that the firmware binary provided by Cisco is specific to the communication technology and the
software model. So there are different binaries for each hardware reference platform (64, 100 or 144
pin) as well as for each communication technology (RF, PLC or dual-phy).
1.3 CG-REDI and partner development infrastructure
To enable rapid development of CGE, Cisco provides cloud-based network infrastructure services –
Connected Grid – Rapid Endpoint Development Infrastructure-as-a-service or CG-REDI. This enables
the CGE partners to focus on the key development steps while using the required network services
from cloud-hosted facility at Cisco’s DMZ facility. The following diagram illustrates the CG-REDI
architecture to support partner development:
Figure 1.6 CG-REDI architecture for partner development


The following diagram illustrates the partner premises topology (‘Thin Starter Kit’) required for
development and IVT:
Figure 1.7 Partner Premises – “Thin Starter Kit” topology
The starter kit connects to the CG-REDI via IPv6 GRE over IPv4 IPSec VPN. Also as seen, the
Application Server, which hosts the endpoint management and collection software like SCADA
master, MDMS, Collection Engine etc., resides at the partner premises and connects to the CG-REDI
through a SSL VPN connection. More details on CG-REDI and the starter kit topology can be found in
the web-link here.


2 Test Coverage
The testing defined within this test plan is focused on features and functionality related to the
Connected Grid Endpoints. This test plan encompasses implementations of the NIC model and
the IE model across the RF and PLC HW reference designs. To this affect, some test cases are
only applicable to a particular implementation model or a SDK technology. Each test case will
point out its applicability to a model and/or the technology.


2.1 What Will Be Tested
The testing would encompass the following areas:
• CGE HW Schematics and diagnostics: HW Schematic review and MCU diagnostics
• RF/PLC characterization: signal strengths and quality, comparative tests
• Local debug ability and manageability/Workforce enablement: Local CSMP, field
CG-Mesh firmware upgrades, reboot of CM/IE MCU, field tool security (partially)
• CG-Mesh Connectivity: CG-Mesh L2 connectivity, IPv6 Configuration (managed), IPv6
Routing (RPL) and forwarding over mesh topologies, end-to-end bidirectional communication,
multicast forwarding and reachability, Quality of Service (QoS), Time synchronization
• Mesh Security: Access control, data confidentiality and privacy, data authenticity,
firmware authenticity
• Remote Manageability and Notifications: CSMP queries, CG-NMS manageability,
Firmware management and Over-The-Air firmware update (OTA) through CG-NMS, Power
Outage Notifications (PON)/Power Restoration Notifications (PRN)
• Fault resiliency and availability: PAN migration, long duration testing, recovery and
reachability after power failure, node failure
• CGE Application: CGE management and functionality (eg. Meter reads), Application
QoS markings, OTA AM firmware upgrades, Secure field AM firmware upgrades, Partial
application certification for “mesh-friendliness”, Time synchronization of AM application,
Remote upgrade of CGE, server virtualization
• Multi-vendor device interoperability: A subset of the above test cases would be
executed in a multi-vendor PAN environment
• Performance and Stress: Latencies, performance and reachability with scaled traffic
forwarding, ping and meter reads in multi-device, multi-hop network


2.2 What Will Not Be Tested
The following elements will not be tested:
• Application functionality and security:
This includes any functionality that the CGE device provides like metering, SCADA
control, transformer/feeder metering, voltage/amp regulation etc. Also no features on the
Application Server(s) like UI/CLI interface, performance/scale or stress will be tested.
Test cases defined here also do not test the digital security of the CGE device and the
application server. These include attacks like flooding, denial-of-service, man-in-the-
middle, intrusion detection and prevention. IVT certification also does not test authenticity
and privacy implementations of application traffic. End-to-end application and data
security/integrity may be partially tested and the appropriate test cases have been
documented here (see Test Suite – CGE Applications).
• No/Partial coverage of application compatibility over CG-Mesh:
Low power and Lossy Networks (LLNs) require applications designed to operate in these
environments. These include several design considerations like algorithmic efficiencies of
the application protocol (for less intensive CPU and IO operations), protocol design
considerations like notification instead of polling, “randomization” of notification to avoid
medium access, connectionless transport services with lightweight acknowledgements
etc. Application compatibility and mesh-friendliness will not be tested in all cases. The
related test cases are documented here (see Test Suite – CGE Applications).
• Physical security:
The testing does not take into consideration the physical security of the CGE.
• Security of data stored or exchanged within the CGE:
The IVT process does not test for security of instructions, data, certificates or other data
stored in the MCU of CM/IE, AM or any other storage devices. It also does not test for
security of data being exchanged between components on the HW.
• Manufacture time programmability and configurations:
Many of the test cases require the capability to upgrade the boot-loader, CG-Mesh
application firmware and configurations on the CGE device. Many of these functions are
required at manufacturing or staging time, with configurations/firmware specific to a
customer.


However, Cisco does not recommend that the boot-loader and the configuration binary
files be updated by any means in a device once it is deployed in the field, as they raise
security concerns. The test cases document these capabilities if present in the field tool,
and more specifically features that may raise security concerns. However IVT certification
does not mandate compliance to these recommendations
• Any other use-cases identified outside this document
3 Test Setups
The test setups in all cases (except the test setup described in Section 3.4) use the CG-REDI
infrastructure as described earlier: The partner premises hosts the PAN and the endpoints, whereas
CG-REDI hosts all head-end applications except the Application Server which is typically hosted at
the partner premises and connects to the CGEs through a VPN connection via the CG-REDI
infrastructure. It will be required for the partner to also provide Cisco with 5 CGE DUTs for IVT
certification. These CGEs would be integrated in multi-vendor and large-scale topologies.
For the purpose of certification, the PAN will need to be configured for the following topologies for
different test scenarios. The required topologies are described in the test cases:
3.1 MCU Diagnostics Setup
The MCU diagnostic setup requires connecting the CM’s debug UART port with Serial Console
application on a PC (like Hyperterminal, Putty) via a UART-to-USB converter:
Figure 3.1 IVT Topology 1: CGE diagnostics


The topology and connections are detailed here (link requires access to ‘Cisco-Enabled Grid Devices’
Technology Center).
3.2 One CGE in a PAN
This simple one-device PAN topology is recommended for validation of basic features before testing of
the features in more complex topologies:
Figure 3.2 IVT Topology 2: 1 CGE in a PAN
3.3 Four (at least) CGEs in a PAN
This is the recommended topology for most feature, functionality, security, management and fault
resiliency tests. It should be noted that four is the minimum suggested number for CGEs in this topology.
It is a requirement to have at least 3-hops in the mesh network. Also note that the CGEs in this topology
are the partner devices in test i.e. not a multi-device topology.
Figure 3.3 IVT Topology 3: 4 CGEs in a PAN


3.4 Large scale, multi-hop, multi-vendor CGE PAN
In this topology, the CGE DUT (Device Under Test) is one of many multi-vendor devices connected to the
same PAN. The test topology requires the presence of multiple CGE DUTs in this topology in the roles of
mesh nodes and leaf nodes. The details are noted in the test cases.
Figure 3.4 IVT Topology 4: Large scale, multi-hop, multi-vendor CGE PAN
3.5 Multi-hop PAN Migration
In this topology, multiple PANs with same SSIDs are advertised. Some CGE DUTs (Devices 1, 2 and 3)
are capable of reaching and joining either PAN, while some others cannot join both PANs directly (Device
4). The CGEs should choose a PAN as primary (shown in blue below) and after failure would migrate to a
secondary PAN (show in dotted red below).
Figure 3.5 IVT Topology 5: Multi-hop PAN Migration


3.6 Four (at least) multi-vendor CGEs in a PAN
This topology is similar to Section 3.3, except that multi-vendor devices are present in the topology. The
exact placements of the CGE DUTs are noted in the test cases. This topology is a subset of the topology
described in Section 3.4, and that topology may be used for the test cases referencing this section.
Figure 3.6 IVT Topology 6: 4 CGEs in a multi-vendor CGE PAN
3.7 Multi-phy CGEs in a PAN
This topology is focusing on multiple Phys in the same network.
Figure 3.7.1 IVT Topology 7.1: 4 CGEs in a multi-Phy CGE PAN


Figure 3.7.2 IVT Topology 7.2: 4 CGEs in a multi-Phy CGE PAN


4 Test and Certification Approach and Considerations
A CGE partner will have access to the test cases that the final product needs to pass for Cisco
certification (this document) while reviewing the development process, along with the requirements,
guidelines and best practices document (see References). The partner would thus be aware of all
requirements for development and certification, in addition to the AM application and functionality.
The partner is also required to perform and submit pre-IVT testing, submit test and endpoint product
documentation and provide CGE DUTs to Cisco for the certification (identified in the following sub-
section)
4.1 Test Phase Entry Criteria
Each of the criteria identified below need to be met before IVT certification testing can begin:
• Partner has reviewed this test plan and understands the requirements for certification
• Partner has completed development (all intended features) of the CGE including CM/IE (and
AM) hardware and firmware, product integration is complete and server application has been
developed and integrated with the CGE
• HW Schematics for the CM have been shared with, and reviewed/approved by Cisco
• The vendor partner performs all pre-IVT testing. The test results are available to Cisco for
review in an agreed format. All high priority test cases identified for pre-IVT testing have
passed.
• All software including field tools and head-end applications are made available, including
installation of the applications if applicable, for IVT testing
• Partner has submitted all the requirements needed for certification (identified below in
‘Documentation Requirements’)
• Partner has submitted the product for IVT testing through their DevNet ‘Developer
Dashboard’ portal
• Partner premises ‘Starter Kit’ is deployed and VPN connections from CGR and Application
Server to CG-REDI are working. All services and infrastructure has been provisioned in CG-
REDI. If applicable, remote access to Application Server as well as other software required
for IVT is made available to Cisco IVT team for IVT certification.
Documentation requirements from vendor partner before IVT entry (versioning is vendor partner
specific):
• Product name and version


• Product Hardware versions (Communication Module, Application Module and versions other
relevant electronic components in the CGE)
• Product firmware versions (including CM boot-loader and application; and if applicable all
firmware on AM)
• Product data sheets, clearly specifying the application protocol (C12.22, DLMS/COSEM,
DNP3 etc.) and intended use-cases
• End-to-end communication architecture and network topology for product deployment
• Product documentation (version information, data sheets, use cases and topology of
deployment, and product installation instructions, known issues)
• Application Server product name, version, platform, version and deployment topology
• Field tool name, version and deployment details
• 7 CGEs to Cisco for local testing and debugging
• Accompanying instructions for the CGE, for setup, configuration and local management
• Known issues with the product
4.2 IVT Considerations specific to a certification
Detailed Test Plan document specific to an IVT certification may be required. The testing strategy for
each certification needs to be identified in the Detailed Test Plan. New test cases may be identified
and existing test cases may be removed, based on design and implementation of the CGE.


4.3 Test Execution tracking and reporting
• Cisco DevNet team will use the test plan identified for IVT certification (this document or a
derived detailed test plan document specific to an IVT) and create a Cisco TIMS Results
(Database Name - SGBU)
• Vendor partner will perform pre-IVT testing at vendor premises using CG-REDI based services.
The vendor partner will share the test results with Cisco DevNet team. Once all IVT entry criteria
have been met, Cisco DevNet team will update the pre-IVT test case results in TIMS.
• All test results executed as part of IVT will also be tracked in TIMS database
• Detailed Test result summary will be captured in the final IVT certification report. Intermediate
IVT test reports can also be generated to track progress, defects and open issues, and shared
with the vendor partners.
4.4 Defect Tracking
Defect tracking to be performed through shared documentations between Cisco and partner. Details
TBD.
4.5 Requirements Traceability Approach
The requirements for the CGE certification in terms of features and functionality are traced from the
use-cases and subsystems recognized in the Zanzibar project EC and DIG documents. New test
cases are added to the test plan based on availability and implementation of new requirements and
features.
4.6 IVT Certification Elements
The following elements are tied to a certification process and recorded in the certification report
Table 4.6.1: IVT elements:
Category Description Device/Location
Hardware Connected Grid Endpoint
Hardware version (CM and AM
if applicable)
Connected Grid Endpoint
Firmware Connected Grid Endpoint
CM/IE secure boot loader
firmware version
Firmware Connected Grid Endpoint
CM/IE application firmware


4.7 IVT Recertification criteria
The vendor CGE product will require recertification if any of the following conditions are met:
• Certification is required with a different version of Cisco SDK firmware – bootloader or application
(including bug fixes or feature enhancements) which are determined to have dependencies on
hardware implementation or include software/host interface (PPP) changes
• Any changes by the vendor partner to the CGE hardware components, schematics, I/O interfaces
or layouts, specifically to the CG-Mesh communications hardware, or its interfacing with other
components
• Any changes by the vendor partner to the AM hardware or firmware, which is determined to
impact mesh communications
• Recertification may be required to establish interoperability between a specific version of CGR
PAN firmware and CGE firmware
version
Firmware Connected Grid Endpoint AM
firmware version(s)
Firmware Connected Grid Router PAN
firmware version
Connected Grid Router
Firmware Connected Grid Router CG-
OS/IOx software version
Software CG-NMS version CG-REDI
Network infrastructure Certificate authority platform,
software and version
CG-REDI
Network infrastructure AAA server platform, software
and version
CG-REDI
Network infrastructure DHCPv6 Server platform,
software and version
CG-REDI
Network infrastructure HSM device and firmware
version
CG-REDI
Network infrastructure NTP Server CG-REDI
Application Server(s) Application server elements,
software, platform, version and
deployment topology
Application Server(s)


• Recertification may also be required to establish interoperability between FAN Head-end software
and hardware components, and their interactions with CGE products
• Any other changes to elements recognized in Section 4.7, as determined by Cisco’s DevNet and
engineering teams
4.8 IVT Certification Numbering scheme
The following numbering scheme will be used for certification:
• The Certification number will start with ‘CGE-IVT’, recognizing the certification of CGE
through the DevNet IVT process.
• ‘CGE-IVT’ will be appended by the EDCS document number of the Certification Report
(Defined in Section 4.11 of this document)
A sample certification number would thus be ‘CGE-IVT-1291267’ where 1291267 is the EDCS
document number of the Certification Report (described in Section 4.10). This scheme allows us to
track the details of the certification from the number.
4.9 IVT Certification Report
The following items are required to be part of the certification report:
• Certification Program: CGE IVT Certification
• Partner Name, product name and product version
• Product description and deployment topology
• Certification number: Defined in Section 4.8 of this document
• Certification date
• Certification personnel
• Cisco EDCS document number of the certification report
• Certification Elements as defined in Section 4.6 of this document
• Any deviations to the recommended topology including supplemental devices used during
certification testing
• Reference to the Approved Test Plan (this document or supplemental documents derived
from this document) and version of the Test Plan used for certification
• Test summary including failed and dropped test cases, and the reason for dropping
• Any deviations from Cisco’s recommendations as noted in the test cases
• Reference to TIMS database


4.10 IVT Deliverables
The following items are deliverables to be shared with the vendor partner after successful IVT
certification:
• Test Plan (this document or supplemental documents derived from this document)
• Test Execution results: Will be archived in TIMS database, and will also be part of the IVT
certification
• Configuration files used for CGR WPAN interface and the CGEs
• Certification Report: Described in Section 4.9, checked into EDCS: This is made available
to the partner.


4.11 Test Exit/Certification Criteria
All of the following conditions should be met for successful IVT certification:
• 100% execution of high priority test cases.
• All high priority test cases should pass (100%)
• If any medium or low priority tests were dropped, the reason for dropping will need to be reviewed
and approved by stakeholders. These need to be documented in the certification report.
• If medium or low priority tests failed, appropriate documentation should be provided in
certification report.
• Test results uploaded into TIMS and test summary provided in the certification report.
4.12 Performance test (PR) and Scalability test (SC)
A stress test for a CGE would include performing the following operations:
• CGE device would be positioned in a large scale, multi-hop, multi-device PAN (Section 3.4)
as root of a sub tree
• CGE performs forwarding of large volumes of IPv6 datagrams downstream and upstream
(i.e. using both source and default routing) across multiple destinations
• A traffic intensive remote operation over the mesh to the CGE is performed, like file/firmware
transfer
• A combination of the above, where several operations are performed together.
The application performance (meter reads), forwarding efficiency (end-to-end packet drops and
latency values for upstream and downstream traffic flows), FAN features (eg. PON/PRN),
Manageability (CSMP queries), long duration/soak testing should be tested in the proposed stress
test environment. Criteria for passing a stress test will be identified in the associated test cases.


4.13 Pre-IVT Testing Matrix
The vendor partner runs the test cases described below at their premises
Table 4.14.1: CGE diagnostic tests (Pre IVT):
Logical ID Test Setup Priority Title
CGE_PRE_IVT_DIAG_TC001 Figure 3.1 High Run MCU diagnostics and
review results
CGE_PRE_IVT_DIAG_TC002 Figure 3.1 High RF MCU Diagnostic test
Table 4.14.2: Workforce enablement (Pre IVT):
Logical ID Test
Setup
Priority Title
CGE_PRE_IVT_WFE_TC001 N/A High User and device
authentication in the field tool
CGE_PRE_IVT_WFE_TC002 N/A High Capability to upgrade CM/IE
application firmware and
LMAC firmware (for PLC and
dual-phy) in the field
CGE_PRE_IVT_WFE_TC003 N/A High Data privacy for field tool
CGE_PRE_IVT_WFE_TC004 N/A High Debugability through CSMP
CGE_PRE_IVT_WFE_TC005 N/A High Network debugability through
ICMP pings
CGE_PRE_IVT_WFE_TC006 N/A High Capability to query
networking configuration
Table 4.14.3: CG-Mesh Connectivity (Pre IVT):
CGE_PRE_IVT_MESH_TC001 Figure 3.2 High Layer 2 Mesh connectivity
CGE_PRE_IVT_MESH_TC002 Figure 3.2 High Managed IPv6 configuration
CGE_PRE_IVT_MESH_TC003 Figure 3.3 High RPL advertisements
CGE_PRE_IVT_MESH_TC004 Figure 3.3 High IPv6 mesh connectivity and
forwarding
CGE_PRE_IVT_MESH_TC005 Figure 3.3 High Bidirectional connectivity to
head-end servers
CGE_PRE_IVT_MESH_TC006 Figure 3.3 High Time Synchronization
CGE_PRE_IVT_MESH_TC007 N/A High Organizationally Unique


EUI64 values for CG-Mesh
communications
CGE_PRE_IVT_MESH_TC008 Figure 3.7.1 High Preference of physical
medium (dual-phy only)
CGE_PRE_IVT_MESH_TC009 Figure 3.7 High Multi-phy multi-hop test
(dual-phy only)
Table 4.14.4: Mesh Security (Pre IVT):
CGE_ PRE_IVT_SEC_TC001 Figure 3.3 High Access Control with 802.1X
EAP-TLS
CGE_ PRE_IVT_SEC_TC002 Figure 3.3 High 802.11i handshake, key
exchange and data privacy
for all data
CGE_ PRE_IVT_SEC_TC003 Figure 3.3 High Firmware upgrade with
secure boot-loader
Table 4.14.5: Remote Management and Notifications (Pre IVT):
CGE_PRE_IVT_MGMT_TC001 Figure 3.3 High CG-NMS registration,
reachability, configuration
template and periodic
updates
CGE_PRE_IVT_MGMT_TC002 Figure 3.3 High CG-NMS CSMP data integrity
and authenticity
CGE_PRE_IVT_MGMT_TC003 Figure 3.2 High CG-NMS firmware upgrade
over unicast CSMP
messages
CGE_PRE_IVT_MGMT_TC004 Figure 3.3 High CG-NMS firmware upgrade
over multicast CSMP
messages
CGE_PRE_IVT_MGMT_TC005 Figure 3.3 Medium PON/PRN notifications (RF
and dual-phy only)
CGE_PRE_IVT_MGMT_TC006 Figure 3.3 High Capability to derive
Application Server(s) address
from DHCP Option 17
CGE_PRE_IVT_MGMT_TC007 Figure 3.3 High LMAC FWDL (PLC and dual-
phy only)
CGE_PRE_IVT_MGMT_TC008 Figure 3.7 High Dual-phy CSMP TLVs (dual-
phy only)


Table 4.14.6: CGE Application (Pre IVT):
CGE_PRE_IVT_APP_TC001 Figure 3.3 High Application Server
functionality (Meter reads,
SCADA Comms etc)
CGE_PRE_IVT_APP_TC002 Figure 3.3 High Application QoS markings
(bidirectional)
CGE_PRE_IVT_APP_TC003 Figure 3.3 High Time synchronization of
Application Server and AM
firmware for event correlation
and DA use-cases
CGE_PRE_IVT_APP_TC004 Figure 3.3 High Application protocol “mesh-
friendliness”
CGE_PRE_IVT_APP_TC005 Figure 3.1 High IPv6 MTU and fragmentation
CGE_PRE_IVT_APP_TC006 Figure 3.3 Medium Conformance to 800 byte L3
packet size
Table 4.14.7: Fault resiliency and availability (Pre IVT):
CGE_PRE_IVT_FRA_TC001 Figure 3.3 High Recovery after power failure
of CGE
CGE_PRE_IVT_FRA_TC002 Figure 3.3 High Recovery after power failure
of FAR
CGE_PRE_IVT_FRA_TC003 Figure 3.3 High RPL convergence after node
failures


4.14 IVT Testing Matrix
All test cases described below are run in a multi-vendor environment in IVT test labs
Table 4.15.1: CGE HW Schematics and diagnostics (IVT):
CGE_IVT_HW_TC001 N/A High Review HW Schematics
Table 4.15.2: RF/PLC physical and MAC layer (IVT):
CGE_IVT_SIG_TC001 Figure 3.2 High Transmit Power
CGE_IVT_SIG_TC002 Figure 3.2 High RSSI/LQI Comparative test
CGE_IVT_SIG_TC003 Figure 3.2 High PLC Modulation and
throughput (PLC and dual-
phy only)
CGE_IVT_SIG_TC004 Figure 3.2 High Throughput comparison test
(dual-phy only)
Table 4.15.3: Workforce enablement (IVT):
CGE_IVT_WFE_TC001 N/A High User and device
authentication in the field tool
CGE_IVT_WFE_TC002 N/A High Capability to upgrade CM/IE
application firmware and
LMAC firmware (for PLC and
dual-phy) in the field
CGE_IVT_WFE_TC003 N/A Medium Document capability to
upgrade CM/IE boot loader in
the field
CGE_IVT_WFE_TC004 N/A Medium Document capability to
upgrade CM/IE configuration
in the field
CGE_IVT_WFE_TC005 N/A Medium Data Privacy for the field tool
CGE_IVT_WFE_TC006 N/A High Debugability through CSMP
CGE_IVT_WFE_TC007 N/A High Network debugability through
ICMP pings


CGE_IVT_WFE_TC008 N/A Low Capability to reboot/reset
communication HW
CGE_IVT_WFE_TC009 N/A High Capability to query
networking configuration
CGE_IVT_WFE_TC010 N/A Medium Capability to upgrade AM
boot-loader, application
firmware and configuration in
the field
CGE_IVT_WFE_TC011 N/A Low Capability to reboot/reset
communication module HW
Table 4.15.4: CG-Mesh Connectivity (IVT):
Logical ID Test
Setup
Priority Title
CGE_IVT_MESH_TC001 Figure
3.6
Medium CSMP QoS markings
CGE_IVT_MESH_TC002 N/A High Organizationally Unique EUI64
values for CG-Mesh
communications
3.7.1
High Preference of physical medium
(dual-phy only)
3.7.1
Medium Use of advanced metrics for
physical medium choice (dual-
phy only)
3.7.1
Medium RPL advertisements
(dual-phy only)
3.7.1
Medium Use of advanced RPL metrics
(dual-phy only)
3.7
High Multi-phy multi-hop test
(dual-phy only)
Table 4.15.5: Mesh Security (IVT):
CGE_IVT_SEC_TC001 Figure 3.6 High Access Control with 802.1X
EAP-TLS
CGE_IVT_SEC_TC002 Figure 3.6 Low Access control for
unauthenticated/untrusted
certificate on device (device
rejection)
CGE_IVT_SEC_TC003 Figure 3.6 High Firmware upgrade with
secure boot-loader


CGE_IVT_SEC_TC004 Figure 3.6 High Failed untrusted firmware
upgrade with secure boot-
loader
CGE_IVT_SEC_TC005 Figure 3.7 High Secure mesh with dual-phy
(dual-phy only)
Table 4.15.6: Remote Management and Notifications (IVT):
CGE_IVT_MGMT_TC001 Figure 3.6 High CSMP over CG-Mesh
CGE_IVT_MGMT_TC002 Figure 3.6 High CG-NMS registration,
reachability, configuration
template and periodic
updates
CGE_IVT_MGMT_TC003 Figure 3.6 High CG-NMS CSMP data integrity
and authenticity
CGE_IVT_MGMT_TC004 Figure 3.6 High CG-NMS CSMP
missing/invalid signatures
CGE_IVT_MGMT_TC005 Figure 3.6 High CG-NMS firmware upgrade
over unicast CSMP
messages
CGE_IVT_MGMT_TC006 Figure 3.6 High CG-NMS firmware upgrade
over multicast CSMP
messages
CGE_IVT_MGMT_TC007 Figure 3.6 Medium PON/PRN notifications (RF
and dual-phy only)
CGE_IVT_MGMT_TC008 Figure 3.6 High Capability to derive
Application Server(s) address
from DHCP Option 17
CGE_IVT_MGMT_TC009 Figure 3.6 High LMAC FWDL (PLC and dual-
phy only)
CGE_IVT_MGMT_TC010 Figure 3.7 High Dual-phy CSMP TLVs (dual-
phy only)
Table 4.15.7: CGE Application (IVT):
CGE_IVT_APP_TC001 Figure 3.6 High Application Server
functionality (Meter reads,
SCADA Comms etc)
CGE_IVT_APP_TC002 Figure 3.6 High Application QoS markings
(bidirectional)


CGE_IVT_APP_TC003 Figure 3.6 High Time synchronization of
Application Server and AM
firmware for event correlation
and DA use-cases
CGE_IVT_APP_TC004 Figure 3.6 Low Document capability to
perform AM application
firmware, boot-loader and
configuration updates over-
the-air
CGE_IVT_APP_TC005 Figure 3.6 High Application protocol “mesh-
friendliness”
CGE_IVT_APP_TC006 Figure 3.6 Low Document application data
integrity and authenticity
checks for Application
messages
CGE_IVT_APP_TC007 Figure 3.6 Low Document capability and
method to securely update
(CM/IE) boot-loader and
configuration over-the-air
CGE_IVT_APP_TC008 N/A Low Document device inventory,
firmware and configuration
management capabilities of
Application Server(s)
CGE_IVT_APP_TC009 N/A Medium Document server
virtualization capability
CGE_IVT_APP_TC010 Figure 3.6 High IPv6 MTU and fragmentation
CGE_IVT_APP_TC011 Figure 3.6 Medium Conformance to 800 byte L3
packet size
Table 4.15.8: Fault resiliency and availability (IVT):
CGE_IVT_FRA_TC001 Figure 3.6 High Recovery after power failure
of CGE
CGE_IVT_FRA_TC002 Figure 3.6 High Recovery after power failure
of FAR
CGE_IVT_FRA_TC003 Figure 3.5 High PAN migration after DAG
failure
CGE_IVT_FRA_TC004 Figure 3.6 High RPL convergence after node
failures


Table 4.15.9: Soak and Performance/Stress (IVT):
CGE_IVT_SOAK_TC001 Figure 3.6 High Soak Test
CGE_IVT_STRESS_TC001 Figure 3.4 Optional App performance with long
duration large volume packet
forwarding


5 Test Resources
5.1 People
All testing will be performed as a joint activity between the partner development and test team (at the
partner premises setup – Figure 1.6) and Cisco’s DevNet team (at IVT test labs).
5.2 Hardware
Table 5.2.1: Hardware requirement for certification:
5.3 Firmware
Table 5.3.1: Firmware requirements for certification:
Hardware Location
Connected Grid Endpoint device Partner premises and/or at Cisco test labs
Connected Grid Router Partner premises and/or at Cisco test labs
Application Server(s) Partner premises or CG-REDI PaaS cloud
Head-end Router (HER) ASR1K CG-REDI
SSL VPN Gateway CG-REDI
Server hosting CG-NMS CG-REDI
Servers hosting Network-
infrastructure Services
CG-REDI
Hardware Security Module (HSM) CG-REDI
VPN router (optional) Partner premises
Firmware Device
Connected Grid Endpoint CM/IE
firmware
Connected Grid Endpoint AM
firmware
Connected Grid Router WPAN
firmware


5.4 Software
Table 5.4.1: Software requirements for certification:
5.5 Test Equipment
Table 5.5.1: Test equipment requirements for certification:
5.6 Test Tools (software)
Table 5.6.1: Test tool requirements for certification:
Connected Grid Router CG-OS
release
Software Location
CG-NMS CG-REDI
Public key infrastructure
(PKI)/Certificate authority (CA)
CG-REDI
AAA server CG-REDI
DHCPv6 Server CG-REDI
NTP Server CG-REDI
Application Server(s) Partner Premises or CG-REDI PaaS cloud
Equipment Location
Test Traffic Generator (TBD) TBD
Tool Comments
MCU Diagnostics Firmware provided by Cisco through DEVNET
portal


CSMP CLI/GUI Java based tool provided by Cisco for CSMP
functionality
Partner Field tool(s) Installed in hardened field device for workforce
enablement
Cisco cfgwriter For configuration generation
Segger JLink Pro Programming CGE through JTAG


6 Risks, Mitigations, and Contingencies
Table 6.1: Risks
Id Risk Probability Impact Mitigation/Contingency Plans
1 Incorrect
implementation of
HW Reference
Design
Medium High Review HW Schematic before IVT
Certification (Test Entry Criteria);
Mandate MCU diagnostics
2 CM/IE firmware code
quality
Low High Cisco Engineering provides new
firmware for CM/IE and certification
3 Defects identified in
HW Reference
Design
Medium High Cisco Engineering provides new
reference design and suggested
fixes to existing hardware
implementations
4 Missing/Incorrect
implementation of
high priority features
in CGE
Medium High High priority features would require
fix from CGE partner
5 Missing/Incorrect
implementation of
medium priority
features in CGE
Medium Medium Conditional certification and clearly
marked deficiencies in the
certification report
6 Lack of available
CGEs for certification
Medium Medium Test cases involving larger number
of CGEs can be substituted with
mixed device topologies
7 Delay in fixing critical
defects in firmware
Low High -


7 Detailed Test Cases (Pre-IVT)
7.1 CGE HW Schematics and diagnostics (Pre-IVT):
Test Case Details
Title Run MCU diagnostics and review results
ID
Logical ID CGE_PRE_IVT_DIAG_TC001
Owner(s) mathaker
Description Run MCU diagnostics firmware tests on the MCU of the CGE
Test Setup Figure 3.1
Procedure • Download MCU diagnostics firmware tests based on HW
Reference design (PLC or RF) from DevNet Web portal
• Load and run the MCU diagnostic on CM/IE MCU as per the
instructions posted on DevNet Web portal
• Make sure all tests pass with the exception of the RF test.
• Record results
Test Phase Pre-IVT System Test
Test Suite CGE HW Schematics and diagnostics
Pass/Fail Criteria All tests pass with the exception of the RF test should pass
Priority High
Test Case Details
Title RF MCU Diagnostic test
ID
Logical ID CGE_PRE_IVT_DIAG_TC002
Owner(s) mathaker
Description Run RF MCU diagnostics firmware tests on the MCU of the CGE
w
i
t
h
T
I
M
S
w
i
t
h
T
I
M
S


Procedure • Download MCU diagnostics firmware tests based on HW
Reference design (PLC or RF or Dual Phy) from DevNet Web
portal
• Load and run the MCU diagnostic on CM/IE MCU as per the
instructions posted on DevNet Web portal
• Two nodes will be required.
• Make sure RF test passes and record the results.
Test Suite CGE HW Schematics and diagnostics
Pass/Fail Criteria As noted in the procedure
Priority High
7.2 Workforce enablement (Pre-IVT):
Test Case Details
Title User and device authentication in the field tool
ID
Logical ID CGE_PRE_IVT_WFE_TC001
Owner(s) adewanga
Description The CGE device management field tool access should be secured
through user and/or device authentication
Test Setup N/A
Procedure • Access control can be in the form of user based
authentication and/or device authentication
• Access control maybe tied to only critical functionality like
IE/CM firmware upgrades
• The access control mechanism should be documented in the
test report
• For user based authentication, document mechanism of
authentication (local vs remote server) and method of storage
of passwords if local (hashed, encrypted, plain-text)
w
i
t
h
T
I
M
S


• Document the access control mechanism including software
components protected. Provide screenshots demonstrating
the mechanisms.
Test Suite Workforce Enablement
Pass/Fail Criteria All criteria described in procedure should pass for the test case to
pass. Exceptions should be noted.
Priority High
Test Case Details
Title Capability to upgrade CM/IE application firmware and LMAC firmware
(for PLC and dual-phy) in the field
ID
Owner(s) adewanga
Description Capability to upgrade CM/IE firmware and LMAC firmware (for PLC
and dual-phy) in the field
Test Setup N/A
Procedure • The CGE should provide a mechanism to upgrade CM/IE
application firmware in the field
This enables field technicians to upgrade the communications
firmware in the field if the CGE is not online.
• For PLC and dual-phy CGEs, the field tool should also
provide a mechanism to update the LMAC firmware.
• The mechanism should be documented. Provide supporting
screenshots.
Pass/Fail Criteria Described in procedure.
Priority High
Test Case Details
w
i
t
h
T
I
M
S
w
i
t
h
T
I
M


Title Data privacy for field tool
ID
Owner(s) adewanga
Description Communication between field tool and the CGE should be secure
Test Setup N/A
Procedure • The medium of communication between the field tool and the
CGE should be secure i.e. implement data privacy:
-If using wireless medium, then the wireless connection
should be secure through the use of wireless encryption
This is required if secure data is being transferred between
the field tool and the CGE like 802.1x EAP-TLS X.509
certificate with private key (see CGE_IVT_WFE_TC004)
Pass/Fail Criteria Described in procedure. Exceptions should be noted.
Priority High
Test Case Details
Title Debugability through CSMP
ID
Owner(s) adewanga
Description The CGE should provide a mechanism to query CSMP TLV from the
CM/IE kernel
Test Setup N/A
Procedure • For PPP model, the capability to query CSMP TLVs via the
host port over the PPP connection should be available. The
AM may relay/pass-through/route the CSMP messages from
the field tool to the CM, without any processing on CSMP
message contents
• For IE, tool to query CSMP TLVs via API calls to the kernel
should be available
• All CSMP TLVs values should be available to query.
w
i
t
h
T
I
M
S


Document the procedure and screenshots.
Pass/Fail Criteria All criteria described in procedure should pass for the test case to
pass.
Priority High
Test Case Details
Title Network debugability through ICMP pings
ID
Owner(s) adewanga
Description The CGE should provide a mechanism to ping IPv6 nodes in the
network
Test Setup N/A
Procedure • For PPP model, the capability to ping from the AM to any IPv6
destination should be available
• For IE, the capability to ping any IPv6 destination should be
available
• Make sure IPv6 pings from AM App/IE App to the CGR
WPAN interface is successful.
• Make sure IPv6 ping to the AM of another CGE in the same
WPAN is successful.
• Provide test logs – topology (RPL atree, DUT location),
procedure to initiate ping and successful ping response
Pass/Fail Criteria As described in the procedure
Priority High
Test Case Details
Title Capability to query networking configuration
w
i
t
h
T
I
M
S
w
i
t
h
T
I
M
S


ID
Owner(s) adewanga
Description Capability to query networking configuration should be available
Test Setup N/A
Procedure • The CGE should provide tools to query or display the
following networking configuration on the AM/IE kernel:
-For PPP model, PPP interface IPv6 addresses (link local and
global) and prefix-length on the AM
-Routing table on the AM/IE kernel
-Application Server address: This is derived from contents of
DHCP Option 17. Each vendor will be assigned a unique
Option index for Option 17, and the IPv6 address
corresponding to the Option index will contain the Application
server address. This assignment will be specific to the utility
customer. The CGE must not “hard-code” this index as it may
vary from utility to utility.
• CG-NMS IPv6 address: This is derived from DHCP Option 17
– Option index 1 value.
• Make sure that the tool reports expected values for the above
parameters
• Document procedure to access the networking tools and
execute the tests. Provide screenshots and/or test logs
Pass/Fail Criteria As described in the procedure
Priority High
7.3 CG-Mesh Connectivity (Pre-IVT):
Test Case Details
Title Layer 2 Mesh connectivity
ID
Logical ID CGE_PRE_IVT_MESH_TC001
Owner(s) adewanga
w
i
t
h
T
I
M
S


Description The CGE should join the FAR PAN at Layer 2
Procedure • Program the CGE CM/IE MCU flash with the boot loader,
application image and the configuration
• Deploy the CGE at the intended distance and environment
from the FAR as per intended deployment specifications
• Reboot the CGE
• Monitor and document the CGE time to join the PAN mesh at
layer 2. This should be comparable to the Cisco reference
implementation of the same HW Reference design within the
same environment with the same configuration (attenuation,
distance, same PAN, and configuration). Make note of any
deviations.
• Commands for on CGR:
CG-OS
show wpan in-hardware linknbrs
show wpan linknbrs table
IOX
show wpan x/1 hardware link-neighbors
show wpan x/1 link-neighbors table
• Query CSMP TLV 52 on CM/IE kernel and make sure CGR is
seen as a CG-Mesh neighbor
• Provide the times documented and the following CLI outputs
from CGR–
CG-OS
show wpan in-hardware linknbrs
show wpan linknbrs table
IOX
show wpan x/1 hardware link-neighbors
show wpan x/1 link-neighbors table
Test Suite CG-Mesh Connectivity
Priority High


Test Case Details
Title Managed IPv6 configuration
ID
Owner(s) adewanga
Description The CGE nodes should be able to Solicit and configure IPv6
addresses
Procedure • Make sure provisioning with CG-REDI is completed and the
connection is working. Make sure CGR is able to reach the
DHCP Server through pings
• Program the CGE CM/IE MCU flash with the boot loader,
• Deploy the CGE at the intended distance and environment
• Reboot the CGE
• Monitor and document the CGE time to the first DHCP solicit
request from the CGE. Use in-built ‘ethanalyzer’ tool in CGR
to capture and decode packets in the PAN network. Make
note of any deviations. CLI for ethanalyzer:
ethanalyzer local interface wpan limit-
captured-frames 0
• This should be comparable to the Cisco reference
implementation of the same HW Reference design, within the
same environment with the same configuration (attenuation,
distance, same PAN, and configuration) and with the same
DHCP Solicit interval configuration. (Skip this step for Pre-IVT
testing)
• Verify that the CGE received a DHCPv6 Reply from the CG-
REDI DHCP server. Note the address offered in the DHCPv6
Reply message
• For PPP model, verify that AM makes DHCPv6 requests
every 60 seconds on the PPP link.
• Verify that the AM configures the IPv6 address to its PPP
interface with a /128 mask.
• Verify that the default gateway on AM module is pointing to
the PPP interface.
• Provide test logs – Successfully configured IPv6 address as
verified from the field tool, times documented above.
w
i
t
h
T
I
M
S


Priority High
Test Case Details
Title RPL advertisements
ID
Owner(s) adewanga
Description RPL should advertise all IPv6 destinations as per the deployed
topology to the CGR
Procedure • Make sure provisioning with CG-REDI is completed and the
connection is working. Make sure CGR is able to reach the
DHCP Server through pings
• Program the CGEs CM/IE MCU flash with the boot loader,
• Deploy the CGEs at the intended distance and environment
• Make sure to place the CGE (or use appropriate attenuations)
so that they join the CG-Mesh in the topology as shown in
Figure 3.3
• Make sure that all CGEs have joined the CG Mesh and DHCP
server has leased an IPv6 address to all the CGEs
• Make sure that all CGE RPL instances advertise the IPv6
addresses to the CGR as per the leases granted by DHCP
Server:
CG-OS
show rpl dag 1 table
show rpl dag 1 tree
IOX
show wpan x/1 rpl table
show wpan x/1 rpl tree
• The RPL tree should reflect the topology in Figure 3.3
• For PPP model, the AM IPv6 address should also be
advertised through RPL and displayed in these CLI:
w
i
t
h
T
I
M
S


CG-OS
show rpl dag 1 table
show rpl dag 1 tree
IOX
show wpan x/1 rpl atable
show wpan x/1 rpl atree
• Document outputs of the above CLIs
Priority High
Test Case Details
Title IPv6 mesh connectivity and forwarding
ID
Owner(s) adewanga
Description All IPv6 destinations in the CG-Mesh should be inter-reachable
Procedure • Follow the procedure noted in CGE_PRE_IVT_MESH_TC003
• Make sure all the CGE and AM (if applicable) are reachable
from CGR through pings
• For IE model, make sure that all CGEs within the CG-Mesh
can reach each other through pings
• For PPP model, make sure all AMs are able to reach each
other through pings
• Document the RPL topology and ping outputs
Priority High
w
i
t
h
T
I
M
S


Test Case Details
Title Bidirectional connectivity to head-end servers
ID
Owner(s) adewanga
Description The endpoints should have bidirectional connectivity to head-end
management servers
Procedure • Follow the procedure noted in CGE_PRE_IVT_MESH_TC004
• Bidirectional reachability to CG-NMS and all Application
Server(s) should be verified using pings from AM/IE and/or
the head-end servers
• Document the RPL topology and ping outputs
Priority High
Test Case Details
Title Time Synchronization
ID
Owner(s) adewanga
Description Make sure all CGEs are time synchronized with the CGR
Procedure • Make sure provisioning of DHCP Server with CG-REDI is
completed and the GRE/IPSec connection is working
between CGR and the CG-REDI.
• Make sure CGR is configured with NTP client pointing to the
NTP server in CG-REDI, and is synchronized. Use the
following CLIs:
CG-OS
w
i
t
h
T
I
M
S
w
i
t
h
T
I
M
S


show ntp peer-status
IOX
show ntp associations
show clock
• Make sure all CGEs have joined the PAN.
• Time is distributed in CG-Mesh using 802.15.4e MAC Time
Distribution Protocol. The accuracy of the time is expected to
be in 10ms to 100ms range from CGR time.
• Check the time on the CGE CM/IE using CSMP query for TLV
18
• Verify that the time reported is synchronized with the system
time on CGR to the required accuracy (100s of msec)
(Method TBD)
• Document the CLI outputs from CGR and the corresponding
TLV outputs from the CGE.
Priority High
Test Case Details
Title Organizationally Unique EUI64 values for CG-Mesh Communication
ID
Owner(s) adewanga
Description EUI64 values for the CGE should be globally unique
Test Setup N/A
Procedure • Partner should use an IEEE Registration Authority assigned
Organizationally Unique Identifier (OUI) for the first 3-bytes of
the EUI-64 for all their devices.
The remaining 40 bits of the EUI-64 should be unique for
every device that is manufactured.
• Document the EUI64 derivation procedure for the CGEs:
Through xml configuration or algorithmic derivation
• Document the procedure for algorithmic derivation of the
w
i
t
h
T
I
M
S


EUI64 value and make sure that IEEE assigns the OUI being
used.
Priority High
Test Case Details
Title Preference of physical medium (dual-phy only)
ID
Owner(s) linyan
Description RF is the preferred medium for dual-phy endpoints
Test Setup Figure 3.7.1
between CGR and the CG-REDI. Make sure CGR is able to
reach the DHCP Server using the WPAN intf address(es).
• Setup the bridges with RF module as a master and PLC as a
slave:
http://solutionpartner.cisco.com/web/cegd/dual-phy-
configuration
• When the CGE comes up, check to ensure that CGE tries to
use RF media before the PLC media to form the mesh
network.
• Make sure all the CGEs have joined the mesh in Layer 2 and
Layer 3 and are reachable via the preferred RF medium.
Verify using the counters below.
• Query TLVs for the network stats:
LowpanPhyCounters
LowpanRfStats
LowpanPlcStats
w
i
t
h
T
I
M
S


Priority High
Test Case Details
Title Multi-phy multi-hop test (dual-phy only)
ID
Owner(s) linyan
Description Mesh network with different mediums at various hop level
between CGR and the CG-REDI. Make sure CGR is able to
reach the DHCP Server using the WPAN intf address(es).
• Setup the bridges with RF module as a master and PLC as a
slave (Figure 3.7.1):
http://solutionpartner.cisco.com/web/cegd/dual-phy-
configuration
• When the CGE comes up, check to ensure that CGE tries to
use RF media before the PLC media to form mesh as the
primary medium.
• Make sure all the CGEs have joined the mesh in Layer 2 and
Layer 3 and are reachable via the preferred RF medium.
Verify using the counters below.
• Ensure that the network can ping the CGEs using 1- 1280
packet size
• Next, setup the bridges with PLC module as a master and RF
as a slave
• Ensure that the network can ping the CGEs using 1- 1280
packet size
Priority High
w
i
t
h
T
I
M
S


7.4 CG-Mesh Security (Pre-IVT):
Test Case Details
Title Access Control with 802.1X EAP-TLS
ID
Logical ID CGE_PRE_IVT_SEC_TC001
Owner(s) adewanga
Description Make sure client functionality for access control using 802.1x EAP-
TLS is working
between CGR and the CG-REDI.
• Follow procedure documented here to setup security:
http://solutionpartner.cisco.com/web/cegd/configure-security-
in-starter-kit
• Make sure devices at one hop from CGR were able to
authenticate (EAP + Radius)
• Make sure multi-hop devices were able to authenticate with
the Radius server (EAPoUDP + Radius)
• CGR CLI:
show dot1x all details
• Make sure all CGEs get configured with IPv6 addresses and
are reachable via pings from CGR
• Document CLI output for dot1x and RPL topology
Test Suite CG-Mesh Security
Pass/Fail Criteria As noted in procedure
Priority High
Test Case Details
Title 802.11i handshake, key exchange and data privacy for all data
ID
w
i
t
h
T
I
M
S
w
i
t
h
T
I
M
S


Owner(s) adewanga
Description Trusted CGEs should be able to derive PMK, and exchange PTK and
GTK keys through handshake for CG-Mesh security
Procedure • Follow procedure documented in
CGE_PRE_IVT_SEC_TC001
• Make sure that CGEs were able to derive PMK, and
download PTK and GTK keys using a 4-way handshake from
CGR, by querying CSMP TLV 34 – which should show the
keys and their lifetimes.
• Make sure pings to the CGEs are working
• Document the TLV values and the ping logs
Priority High
Test Case Details
Title Firmware upgrade with secure boot loader
ID
Owner(s) adewanga
Description Trusted CM/IE application firmware should be successfully verified by
secure boot loader and the upgrade should work
Procedure • Perform a firmware upgrade of a trusted CM/IE application
firmware using the field tool
Trusted CM application firmware is signed by Cisco’s CA and verified
by the CM secure boot-loader, which recognizes Cisco’s CA as a
trustpoint
A trusted IE application firmware is signed by the partner CA and
validated by the secure boot-loader, which recognizes the partner CA
as a trustpoint
w
i
t
h
T
I
M
S


• Make sure the firmware upgrade is successful by verifying
CSMP TLV 11 on the CM/IE kernel
• Modify 802.11i GTK (mesh-security keys) on FAR to ensure
cold start and reboot the CGE
• The device should join, authenticate and communicate over
the CG-Mesh
Priority High
7.5 Remote Management and Notifications (Pre-IVT)
Test Case Details
Title CG-NMS registration, reachability, configuration template and periodic
updates
ID
Logical ID CGE_PRE_IVT_MGMT_TC001
Owner(s) adewanga
Description CGE CM/IE kernel should register with CG-NMS and should be
manageable
Procedure • Make sure all provisioning for the partner setup is complete
and working between CG-REDI and partner premises,
including CGR and Application server connectivity.
• Make sure that the PAN environment is configured for
security including 802.1x and 802.11i.
• Make sure that Option 17 in DHCP Reply message from
DHCP Server to the PAN contains the NMS address as its
first option
• Reboot the CGEs to make sure that the correct Option 17 is
received, if required.
• Make sure all CGEs are deployed and have IPv6 addresses
assigned and are reachable through the CGR. Query CSMP
w
i
t
h
T
I
M
S


TLV 43 from the host port/Field application to make sure the
correct CG-NMS IPv6 address is received
• Document the time taken to register node with CG-NMS
• Make sure ping is working from CG-NMS to the CGEs
• Update the ‘Report Interval’ from the configuration templates
for mesh devices and Push the configuration to the mesh
devices
• Make sure in CG-NMS that properties are reported
periodically like IP address, serial number, model number,
firmware version, SSID PAN ID, Mesh statistics like Tx/Rx
speeds, network interfaces, network routes etc.
• Document CG-NMS registration of CGEs through
screenshots
Test Suite Remote Management and Notifications
Priority High
Test Case Details
Title CG-NMS CSMP data integrity and authenticity
ID
Owner(s) adewanga
Description The CGE should successfully verify CSMP message signature from
CG-NMS
• Make sure CG-NMS is signing all CSMP messages sent to
the CGEs. Browse to Admin > Certificates > Certificate for
CSMP in CG-NMS.
• Configure the CGEs for security mode and signature
validation for Post and Responses through xml configuration:
http://solutionpartner.cisco.com/web/cegd/configure-security-
w
i
t
h
T
I
M
S


in-starter-kit
• Reboot the CGEs and make sure they register with CG-NMS
• Query TLV 78 and 79 on the CGE CM App to make sure that
the certificate is being verified and the certificate contents
• Perform a CSMP Post (or a firmware download) and make
sure that the operation is successful
• Document the successful post operation/firmware download
Priority High
Test Case Details
Title CG-NMS firmware upgrade over unicast CSMP messages
ID
Owner(s) adewanga
Description Firmware upgrade of CM/IE firmware from CG-NMS should work over
unicast CSMP messages
Procedure • Repeat the procedure documented in
CGE_PRE_IVT_MGMT_TC002
• Perform a firmware upgrade through CG-NMS. Follow
procedure documented here:
http://solutionpartner.cisco.com/web/cegd/tutorial-
cgnms#OTA
• Make sure that the firmware download to the CGE is using
unicast messages
• Make sure the firmware download is successful
• After the firmware download is completed, schedule a reload
time
• Make sure that the CGE CM/IE board reloads at the
scheduled time
• After CGE reloads, make sure that it joins the CG-Mesh and
requests and receives IPv6 address successfully
w
i
t
h
T
I
M
S


• Make sure the CGR registers with CG-NMS
• Verify the statistics reported in CG-NMS and make sure that
the new firmware version is reported
• Also query TLV 11 from the Field tool to make sure that the
firmware download was successful
• Query TLV 75 to verify the contents of the downloading,
backup and running firmware slots
other through pings
• Document the successful firmware download through
screenshot in CG-NMS and logs.
Priority High
Test Case Details
Title CG-NMS firmware upgrade over multicast CSMP messages
ID
Owner(s) adewanga
Description Firmware upgrade of CM/IE firmware from CG-NMS should work over
multicast CSMP messages
Procedure • Repeat the procedure documented in
CGE_PRE_IVT_MGMT_TC003
• Make sure that the firmware download to the CGE is using
multicast messages
other through pings
w
i
t
h
T
I
M
S


• Document the successful firmware download through
screenshot in CG-NMS and logs.
Priority High
Test Case Details
Title PON/PRN notifications (RF and dual-phy only)
ID
Owner(s) adewanga
Description Verify PON/PRN notifications
security including 802.1x and 802.11i, and the CGEs are
configured with the credentials as well as all required
certificates (for 802.1x, CG-NMS and Application Server(s) if
applicable)
• Make sure the CGEs join the mesh and can communicate
using pings. Make sure CGEs have registered to CG-NMS
and CG-NMS is configured for CSMP signing.
• Make sure CGR is configured to relay the PON/PRN
messages to the CG-NMS.
• Introduce Power Outage event in the multi-hop leaf CGE for a
period of at least 30 seconds
• The CGE should broadcast 3 PONs over a 15 second period
to its neighbors and the CG-NMS would forward the PON
along with the EUI64 and timestamp to CG-NMS
• Make sure PON is seen in CGR: show wpan outage table
• Make sure PON is notified in CG-NMS. The CGE should be
marked with ‘Outage’
• Restore power on the CGE.
w
i
t
h
T
I
M
S


• Restoration messages would be broadcast in Layer 2 and
CG-NMS will be notified by CGR
• Make sure PRN is seen in CGR: show wpan restoration table
• CG-NMS should indicate that power is restored successfully
• Repeat the test with one hop CGE
• Repeat the test will all the CGEs in the topology
• Document the CGR CLI outputs as well as CG-NMS
screenshots
Priority Medium
Test Case Details
Title Capability to derive Application Server(s) address from DHCP Option
17
ID
Owner(s) adewanga
Description While configuration of IPv6 address through DHCPv6, the DHCP
server returns Option 17 values to the CGE.
The Option 17 contains a list of applications and the related IPv6
addresses of the Application Server in the Head-End. The Option 17
list including indexes and service names, is utility specific (and can be
PAN specific) and should be programmable in the CGE
applicable)
w
i
t
h
T
I
M
S


• The CGE should have the capability to either program the
parsing algorithm to derive the Application Server Address
from Option 17 OR should mandate the use of a partner
specific Option 17 service name which the customer will use.
• Depending on the solution above, program the CGEs to
derive the Application server address from Option 17
• Using field tool, validate that the correct Application Server
Address was derived
• Document the results
Priority High
Test Case Details
Title LMAC FWDL (PLC and dual-phy only)
ID
Owner(s) linyan
Description Verify FWDL capability of LMAC using unicast and multicast
Procedure • Follow same procedure as CGE_PRE_IVT_MGMT_TC003
for setup, but upgrade LMAC image instead of the CG-Mesh
App
• Repeat the procedure as described in
CGE_PRE_IVT_MGMT_TC004 for LMAC firmware upgrade
using multicast.
Test Phase System Test
Priority High
Test Case Details
w
i
t
h
T
I
M
S
w
i
t
h
T
I


Title Dual-phy CSMP TLVs (dual-phy only)
ID
Owner(s) linyan
Description Test the Dual Phy CSMP TLV added, namely:
• LowpanPhyCounters
• LowpanRfStats
• LowpanPlcStat
• LowpanPhyStats
• Make sure all CGEs are deployed and have IPv6 addresses
assigned and are reachable through the CGR. Query CSMP
TLV 43 from the host port/Field application to make sure the
correct CG-NMS IPv6 address is received
• Query the following CSMP TLVs specific to dual-phy:
LowpanPhyCounters
LowpanRfStats
LowpanPlcStats
LowpanPhyStats
• Verify the counter values are as expected
Test Phase System Test
Priority High
7.6 CGE Application (Pre-IVT):
Test Case Details
Title Application Server functionality (Meter reads, SCADA Comms etc)
w
i
t
h
T
I
M
S


ID
Logical ID CGE_PRE_IVT_APP_TC001
Owner(s) adewanga
Description Application Server should be able to communicate with and manage
the endpoints
Procedure • Make sure the Application Server is connected to CG-REDI
using VPN and has a valid and reachable IPv4 and IPv6
address
• Make sure routing is configured between the CGR and the
Application Server for reachability
• Make sure that the DHCP server Option 17 serves out the
option for the partner specific Application server IPv6
address.
applicable)
• Using the field tool, make sure that the CGEs have the correct
Application Server address configured
• If applicable, make sure that the CGEs register themselves to
the Application Server
• Make sure that the Application Server learns about the CGEs
(through registration or configuration) and is able to manage
the endpoint
• Make sure the connection/connectivity between the
Application Server and the CGE is successful (TCP/UDP)
• Make sure that Application server functionality is successful
eg., SCADA Master commands, on-demand DA or Meter
reads, periodic notifications from CGEs etc
Test Suite CGE Application
Priority High


Test Case Details
Title Application QoS markings (bidirectional)
ID
Owner(s) adewanga
Description • The Application should mark the correct QoS values
• No packet should be marked for High queue
applicable)
and Application Server(s) if applicable, and CG-NMS is
configured for CSMP signing.
• Refer Table 4.2 for reference of QoS priority for traffic
• Create on-demand application traffic event eg. Meter reads, if
applicable
• Using tools on HER and CGR like ACLs, Policy-maps note
that the QoS values in the IPv6 packets
• Make sure that packets from Application Server to the CGEs
have the suggested QoS markings of Medium (AF21)
• Make sure that packets from the CGEs to the Application
Server have the suggested markings of Normal (AF11)
• Measure and validate the QoS values in periodic meter
reads/reports as Normal (AF11) for traffic in both directions, if
applicable
• Measure and validate the QoS values for (meter) notification
events as Medium (AF21) for traffic in both directions, if
applicable
• Validate the QoS markings for HAN traffic as Low (0)
• QoS for any configuration or firmware upgrade should also be
Low (0)
• No application traffic should be marked for High queue (AF3x)
• Test DSCP values for any other major traffic types for the
w
i
t
h
T
I
M
S


CGE and validate
• Document results and QoS markings for different traffic types
Test Suite CGE Application
Priority High
Test Case Details
Title Time synchronization of Application Server and AM firmware for event
correlation and DA use-cases
ID
Owner(s) adewanga
Description Application Server should time synchronize with head-end servers for
event correlation and DA use-cases
applicable),
and Application Server(s) if applicable, and CG-NMS is
configured for CSMP signing.
• Time synchronization is required across the CG-Mesh and the
CGEs for AMI time stamping in Meter reads, notification, log
entries etc. (10 ms accuracy)
• Time synchronization is also required for fault detection,
classification, location, isolation and restoration services in
DA (10 us accuracy)
• Time synchronization for other use-cases are identified in
Zanzibar and Zanzibar should be used as reference for time
accuracy needs
• Verify that the time on the AM/IE Application is synchronized
w
i
t
h
T
I
M
S

CGE IVT Master Test Plan

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (12)

Similar to CGE IVT Master Test Plan

Similar to CGE IVT Master Test Plan (20)

More from Anuj Dewangan

More from Anuj Dewangan (9)

CGE IVT Master Test Plan