Psi multi accessgateway_casestudy


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Psi multi accessgateway_casestudy

  1. 1. PrimeSoft Solutions, Inc. Multi Access Gateway: Case Study August 26, 2010This document contains information proprietary to PrimeSoft Solutions, Inc.Reproduction, release, or use of the information contained in this document isstrictly prohibited without prior written permission from PrimeSoft Solutions,Inc.
  2. 2. Multi Access Gateway : Case StudyMulti Access GatewayClient Offshore Development Center (ODC)This Santa Clara, CA, company develops and markets industrys first multi-accessconvergence gateway designed for stateful management of tens of thousands of secure,reliable, high-quality subscriber sessions over multiple network access types including Wi-Fi and WiMAX.To date, the company has raised over $50M and is backed by Sequoia Capital and KleinerPerkins among other venture capital firms.Relationship ModelPrimeSoft is the chosen Outsourced Product Development partner for the client. PrimeSoftdeveloped this relationship by providing the following at our offshore development center 1. Building testing, automation and development teams 2. Identifying an Project Manager 3. Setting a Lab and setting it up with Client specific software and hardware components 4. Providing supported from HR, Admin and Facilities Management functionsThe offshore team consists of 18 engineering resources divided into QA, Automation andDevelopment. The teams function as an extension to the client’s team in USA, andparticipate in various projects during a release.Weekly status reports are provided by the Project Manager on all active offshore projects,followed by conference calls to discuss any open issues.UMA Handset SimulatorTechnology BriefUnlicensed Mobile Access (UMA) technology provides access to GSM and GPRS mobileservices over unlicensed spectrum technologies, including Bluetooth and 802.11. Bydeploying UMA technology, service providers can enable subscribers to roam and performhandover between cellular networks and unlicensed wireless networks using dual-modemobile handsets. With UMA, subscribers receive a consistent user experience for theirmobile voice and data services as they transition between networks.Project ObjectivesThe objective of this project was to develop a UMA Handset Simulator. UMA simulatorcame out as a requirement from the development and automation engineers to help themperform unit testing and automation of UMA functionality.PrimeSoft Confidential Page 2 of 11
  3. 3. Multi Access Gateway : Case StudyUMA simulator was required support functional, to help automate more than 90% of allfunctional tests and help development engineers run pre-commit unit tests to ensure basicfunctionality.PrimeSoft Confidential Page 3 of 11
  4. 4. Multi Access Gateway : Case StudyOur ApproachThe TeamPrimeSoft identified a technical lead, development & test engineers to staff this project.The team had prior experience in working with telecom and data communicationsprotocols and call processing.Project Execution Requirements Analysis A team from PrimeSoft traveled to US client site for gathering requirements. The team had brainstorming sessions with client’s technical team and end users of the UMA simulator. The requirements were documented in a Functional Requirements Document Template and shared with all the stake holders, followed by a formal review. Review comments were incorporated and the updated document was checked into CVS and posted on internal Wiki. Functional Specifications & Software Design Functional Requirements Document was an input for developing a Functional Specifications Document. The HLD (High Level Design) and LLD (Low Level Design) documents were developed based on the functional specifications. The design document identified modules, interfaces, and presented pseudo code. Coding, Unit Testing & Acceptance Testing Coding of the tool involved setting up the environment and developing code based on the low level design. The code was unit tested by development engineers. System testing of the simulator was carried out by an independent team that developed system test plan document based on the requirements document and executed multiple test cycles and logged bugs using Bugzilla. User acceptance test plan was developed by PrimeSoft QA team and shared with the client. Acceptance testing was performed jointly by PrimeSoft and client teams.Salient Features Object oriented Design approach provides well defined programming interfaces to the scripting layer Provides interfaces to build and parse and UMA and GSM Layer3 messages Multithreaded implementation for RTP sends and receives for multiple handset instances Python extensions provided for abstracting the complex scripting functionality and reduce scripting effort Supports complex handover scenarios by simulating two UMA connections from a single HandsetPrimeSoft Confidential Page 4 of 11
  5. 5. Multi Access Gateway : Case StudyProject Deliverables Deliverable Details Functional Requiremnts Document Functional Specifications Document Documents Design Document Users Guide Release Notes UMA Simulator Source Code (C/C++, Makefiles, Shell Source Code Scripts) Unit Test Plan Test Documents Unit Test Matrix Bugs BUGS Logged into Bugzilla Sample Scripts Sample Automated Scripts Users Guide Automation suite and scripts users guide Traces all the requirements against Functional Requirements Traceability Specifications, Design, System Testing, Unit Testing Matrix and Users GuideTools & Technologies Title Details Protocols UMA, GSM, TCP/IP Operating Systems Redhat Linux Programming & Scripting C, C++, Python, Shell Scripts, Makefiles Configuration Management CVSPrimeSoft Confidential Page 5 of 11
  6. 6. Multi Access Gateway : Case StudyMulti Access Gateway TestingPrimeSoft houses the largest testing team for our client. Our testing team is involved in theentire test life cycle of the product including test strategy development, test plansdevelopment, tests cases development, manual testing, filing bugs and running regressioncycles.Project ObjectivesThe main objective of the testing group is to work as an extension to client’s onsiteengineering teams, and be responsible for test and automation activities involved in theproject.The offshore team is responsible of the following project activities 1. Defining QA and Automation Strategy for a identified release 2. Developing QA plan for the identified release 3. Developing Test Plans and Test Case documents based on a. Marketing and Product Requirements Documents b. Functional Specifications c. Design Documents d. RFC’s Supported 4. Performing Functional, System, Sanity, Regression, Load and Stress, Security and Interop testing 5. Identifying and logging bugs, coordinating with developers in re-producing and analyzing bugs 6. Developing test matrix and reporting on manual tests progress on a daily/weekly basisTesting ProjectsPrimeSoft team worked with client on multiple releases of this product and was involved intesting and owning QA and Automation for the following modules  IPSec IKE v2, IKEv1  MOBIKE (Mobile IKE)  RIP/OSPF  VLAN (Virtual LAN)  Session Initiation Protocol (SIP)  QoS (Quality of Service)  AAA (Authentication, Authorization and Accounting) and RADIUS  Change of Authorization (CoA)  Internet Protocol Slow PathPrimeSoft Confidential Page 6 of 11
  7. 7. Multi Access Gateway : Case Study  Point to Point over Ethernet (PPPoE)  Dynamic Host Control Protocol (DHCP)  PDIF (Protocol Data Interworking Function)  CDR (Call Detail Records)  IP-IP Encapsulation  DOS Attacks  ACL (Access Control List)  SNMP (Simple Network Management Protocol)  RTC (Real Time Charging)  HA (High Availability)  System Testing  Performance TestingOur ApproachThe TeamWe created a dedicated test team who were technically competent in networkingtechnologies. Project Manager and a core group of engineers started the project byperforming manual testing on one of the releases and logged bugs. The team developedits competency on the product by organizing Internal training seminars Technical hands on sessions Developing internal wiki pages Going through past bugs reports Communicating with developers and PLM teamsTest BedsA Site-to-Site VPN was setup between PrimeSoft, and Client location in USA, allowingengineers to reserve, configure and access remote test setups.PrimeSoft also has a lab setup with client specific software and hardware available locallyfor engineers.Project ExecutionTest Projects execution followed the following phases. Analysis The aim of this phase was to technically be competent in the technology and enhance product understanding and gather all the information necessary to develop a test plan document and completely define the scope of testing activities. The team initiated the projects by going through RFC’s and product documentation including, Requirements, Functional Specifications and High Level Design. EmailPrimeSoft Confidential Page 7 of 11
  8. 8. Multi Access Gateway : Case Study exchanges and conference calls were conducted to understand the nuances of implementation on client’s product. Test Plan Development Test plan development involved, identification and documenting 1. Test Topology 2. Software and Hardware Tools 3. Entry and Exit Criteria 4. Conformance scenarios from RFC’s 5. Test cases against Requirements 6. CLI Related test cases 7. Functional test cases 8. Negative test cases 9. End-to-End tests 10. SNMP related test cases 11. High availability test cases Internal & External Reviews Internal peer reviews (board review) is performed and review comments are updated into the document. An internal review checklist is used to ensure the quality of the test plans is consistent. The update document is sent out for an external/client review followed by a walk through of the test plan through a conference call. Review comments received from architects, designers, implementers and PLM are consolidated and updated into the document. Updated document is checked into CVS and posted on internal Wiki. Manual Test Cycle Manual test cycles involved setting up test bed and executing test cases against the SUT (System Under Test) and tracking progress. A Test Matrix is filled for each cycle of manual test cycles across multiple builds, the test matrix captures testing results documented as PASS, FAIL or BLOCKED. The matrix also provides key metrics of the testing cycle including testing effort, bugs logged and passes percentage. Bugs were logged into client maintained bugzilla, with information including 1. Bug description 2. Build version 3. Steps to reproducePrimeSoft Confidential Page 8 of 11
  9. 9. Multi Access Gateway : Case Study 4. Logs and core files path 5. Initial diagnosis findings 6. Observation and CommentsTools & Technologies Title Detail Protocols IPSec, IKE v1/v2, AAA, CoA, PPPoE, DHCP, SIP, SNMP Operating Systems REDHAT Linux, QNX Scripting Python, Shell Scripts Ethereal (Network Analyzer), Scapy (Packet Generator), Tools Silvercreek, SIPp, QA Robot, Smartbits, Landslide Avalanche, IXIA, IKEv1/v2 clients, DOS Attacks tools Configuration Management CVS, BugzillaProject Deliverables Deliverable Details Test Plan Test Plan Document Test Matrix Test Cycle Summary in a Test Matrix Document Bugs BUGS Logged into Bugzilla Checklist Test Plan Release Checklist Requirements Traceability Traces all the requirements against the test cases and Matrix bugsMulti Access Gateway AutomationWith a growing feature list in the product we sent out with a target of automating 50% ofour test cases, and this could significantly reduce the manual test cycle timeline.PrimeSoft was involved in test automation from the very beginning and participated indeveloping an automation framework, developing automated test scripts and suites.Project ObjectivesThe objective of the automation group at PrimeSoft is to Design, Develop and Enhance an Object Oriented Automated Framework Develop automated test scripts Run automated regression on daily/weekly buildsPrimeSoft Confidential Page 9 of 11
  10. 10. Multi Access Gateway : Case Study Develop test suites for pre-commit, sanity, weekly regression and feature specific suites Identify and log bugs Fix scripts and maintain them across release and buildsOur ApproachThe TeamWe created a dedicated automation group within PrimeSoft who were technicallycompetent in scripting languages and had experience with relevant networkingtechnologies. Framework Enhancements Our team participated in design and development activities of the automation framework. An object oriented framework was designed using Python as the scripting language. Enhancements to the framework were performed and tested (using automated unit tests); and the framework was released to carry out the development of the scripts. Automated Scripts A team of automation engineers was selected to automate the test plans using the enhanced automation framework. Automation engineers focused on developing automated scripts and scripts development involved the following scripts Manual testing Develop and automated test script Running the test on a stable build to verify the behavior of the script Fixing all issues with the scripts Adding the script in a suite and running the suite Every developed script went through a through review with at least two reviewers and focused on functionality and coding guidelines. Functional Review: Review focused on the functionality of the scripts and compared them with what is described in the test cases Coding Guidelines Review: The review focused on usability, maintainability and adherence to coding guidelines Checked into CVSProject Salient Features Python based Object Oriented Automation Framework o Can load/clear and check configuration on system under test o Can load/clear and check configuration on Juniper Netscreen hardware o Can initiate IKE v1 and v2 sessions from end clients o Can initiate PPPoE sessions from Linux based end clients o Can handle High Availability cases Exposes library functions for script developersPrimeSoft Confidential Page 10 of 11
  11. 11. Multi Access Gateway : Case Study Automated Tests o Topology File: Can be changed to point to a new topology o Configuration File: Captures all common configuration for a test o Suite file: Can include multiple test cases to be run in a suiteTools & Technologies Title Detail SIP, SDP, RTP Protocols IPSec IKE v1/v2 RIP, OSPF, BGP, DHCP, DNS, Radius, TCP/IP, PPPoE Clients Hardware/Software Netscreen 5GT, XpressVPN, PPPoE Linux Client, SIPp Used test tool, Cisco Terminal Server Operating Systems Redhat Linux, QNX Programming & Scripting Python, Perl, Shell Scripts Configuration Management CVS, BugzillaProject Deliverables Deliverable Details Automated Scripts Developed as part of each Scripts functionality Test Suits Automated test suites for various types of testing Framework Libraries Framework Libraries and Documentation Checklist Scripts Development and Review Checklist Requirements Traceability Traces all the requirements against the test cases and Matrix bugsPrimeSoft Confidential Page 11 of 11