Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Yarnlab Testmate


Published on

Yarnlab Testmate Overview
Cisco HCS collaboration test automation

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Yarnlab Testmate

  1. 1. Unwinding Telco Zane England – Product Director & co-Founder
  2. 2. Making testing friendly
  3. 3. Testing is boring!
  4. 4. Manual Testing Problem Statement Manual Long lead time High Cost Error prone Automated Quick results Fixed costs Repeatable TestMate Once a testing process is understood it can be automated and repeated easily which enables changing the approach to how testing can be done Testmate is an flexible multi-protocol testing platform for UC that automates the testing process. It is designed to be integrated into various process such as: • Lab testing • System commissioning • Cutover • Post upgrade
  5. 5. Methodology How we think testing should be done • Automated • One person can test hundreds or potentially thousands of devices • Repeatable • Tests can be done as often as required • Reportable • Easily find the status of test results • Simple to use
  6. 6. Testing process and principals How to improve change management • Baseline testing • Test before making a change so that the current environment is understood and that there is are not pre- existing issues (that would otherwise be identified after change) • Execute change • Post change verification • Generate test artefacts for internal use and for end customers
  7. 7. Testmate Engine FSM/TSM The Test State Machine is the fundamental component of the Testmate. It is what provided the test result functionality Initialisation Collect test input data and setup for testing Testing Execute tests Clean-up Collect test results and return system to starting point Repeat Re run whole test plans or specific tests as required Initialization Testing Clean-up Results Repeat
  8. 8. What it actually does The details Testmate uses a combination APIs to perform actions and collection information from validation actions that includes • CTI (JTAPI) • Web services (AXL/RisPort/EMAPI) • Phone Services The Testmate interface provides real time feedback of test execution and will also allows for manual test case execution for example calls to emergency services (000) which should not be automated. Testmate JTAPI
  9. 9. Dial Plan Testing How we do it? List of CSSs to test CSS Group List of Test numbers for a country mapped to call type Number Group CUCM (arm) Device Pool Phone Line Select Device Map expected results to call type and CSS Result Mapping Run Tests Repeat End
  10. 10. Testing Dial Plans Without getting calluses! To test a dial plan Testmate makes from phone A to phone B via back to back trunks or another CUCM. Process 1. Find device 2. Set CSS 3. Make test calls 4. Change CSS 5. Repeat 6. Results Support for HCS dial plans is done by adding the site ID suffix Testmate JTAPI AXL RisPort Phone BPhone A CUBE CUCM b2b Trunks Outbound calls are routed to test number by adjusting route select and phone B is set to Auto Answer
  11. 11. Expected results COS instantiation The instantiation of expected results greatly simplifies setting up a test plan and expected results It is represented as an editable table that enables copy and paste from other sources such as excel
  12. 12. Dial plan View (in progress) Find out where something has gone wrong Dial plan view helps troubleshooting when an issue with the dial plan is found It uses a combination of APIs to • Determine dial plan in use • Trace the call end to end • Highlight which pattern and loop • Does not match • Is blocked It can also be used for simulated dial plan testing
  13. 13. Testing dial plans continued… This is only part of the challenge
  14. 14. The remote What it enables PSTN +61212341200 TM System under test +61312341200 Test network CTI RP The remote enables making end to end calls between two separate systems. It can either use CTI RP or dedicated test phone(s) to originate and terminate calls This enables: • End to end call path verification • Inbound call testing • CLIP/CLIR verification • DTMF verification • Verification of numbers during call states (Called, Connected, Calling and Redirecting) **To enable network traversal Arms are used when required
  15. 15. Number Range Test How we do it? Select what type of UAT Type of test CUCM device pool Or Manual import via excel Input Site What % of phones Test Level Select remote to use Remote system Run Tests End
  16. 16. Number Range what do we test? What features and functions The number range test plan utilises the remote and additional CTI actions that covers the following • End to end PSTN calling (outbound/Inbound) • CLIP/CLIR verification • Extension mobility • MOH • DTMF Planned support Media services (CONF/XCD)
  17. 17. Test Plans in Development Concepts and ideas
  18. 18. System Acceptance Simply upgrades and changes…. and be sure it is working after The feature verification test plan enables quick and easy testing of various features such as • Supplementary services • MOH • TFTP (config, images, ILS, etc) • Voicemail • Hunt groups It can be used both in the planning phase for pre upgrade testing in a lab or post upgrade acceptance testing Plan Test Adjust Launch
  19. 19. SAT How is can be used The SAT is a modular test plan where test cases can be added and removed as required. On starting the SAT plan it collects extensive status and configuration state information which is stored with the test run, including software version. Additional test case will be added over time Testing Scenarios • Supplementary services • Hold • Transfer (blind/consultative) • Call Forward • Conference • Voice mail • Long running calls • Content sharing • URI dialling
  20. 20. Configuration Management Coverage Setting up the environment to test It is planned to cover the creation of a new customer and adding users to a test CUCM then cleaning up the test environment Initially CUCDM 8 will be targeted with plans to add CUCDM10/11 Configuration management • Creation of customer, division and location • Add user • Add devices • Add EM/SNR/EM • Add number ranges Validation After the environment has been setup Testmate will execute a test plan to verify Next steps • Build test CUCM and boot strap configuration • Possible configuration automation of aggregation layer using Perimate • Expressway configuration for MRA and URI dialling test cases • Self care and admin portal testing
  21. 21. SATP More options Select type of testing Test definition Existing test system or system to be built Inputs Setup environment to test Environment setup Select remote to use Remote system Run Tests End
  22. 22. User Acceptance Testing (guided) How to automate during a migration
  23. 23. Guiding the UAT process Eliminating excel and word test plans The UAT test plan is intended to provide a portal for end customers to perform a guided UAT process that covers the user facing features via a web portal The end users performing the UAT will select a test device to be used and step through the test plan and Testmate will collect the results in real time It will also provide a mechanism for users to manually fail test cases and escalate to support teams Initial test cases • Extension mobility • PSTN calling • Emergency services • CLI • Redial • Hold/Resume • Transfer • Conferencing • SNR • Call park
  24. 24. Video Testing Interop! The video test plan provides a way to test end to end video focusing on identifying interoperability issues The focus of the testing is to make and receive calls on your video infrastructure, This does require test equipment but intelligent scheduling can be used so that high value internal systems can be used out of hours for testing purposes Testmate Video Infrastructure Call control Endpoints
  25. 25. On-Net Jabber Testing How to test softphones, what OS, what version? Provides a method for testing the various jabber softphones testing both on-net and via MRA (if required) Testing includes • Login • Media (audio and video) • Instant messaging and presence Analytics and Telemetry Analytics and problem report data is collected Testmate Virtual Machine DMZ
  26. 26. Testing globally How to test to and from around the world I +1 +44 +91 +27 +49 +86 Its time consuming and expensive to maintain test numbers around the world in each geography Where possible we will have on demand test services in each geography
  27. 27. Testmate Hosted How to test with real numbers across the world 1 2 4 1. User starts global test plan with AUS test numbers 2. Testmate service allocates numbers for testplan run 3. Hosted service sets up answer sets for connection from testmate 4. Testmate starts making calls to test numbers TM Service PSTN 3 TM Testmate hosted is a service that Testmate can connect to for end to end testing to test number hosted by Yarnlab. It provides test numbers for various calls types in different countries (where possible). The hosted service answers the call, play back media to verify speech path and reports calling number. It can also be used to make inbound calls from various geographies +61212341234 +498899776655 There is an option for customers to host this service also!
  28. 28. Test Plan Designer How can I write my own tests, do I really want to? Current work in progress Writing test plans that are valid and reusable but simple is hard Design Test (the testplan) Adjust Deploy
  29. 29. Test Plan Designer What we are working on The test plan designer will be the interface to create customised test plans that end users can create Plans will be defined in a script with running state available in a UI System acceptance and accreditation testing will use this due to the highly customised and complicated nature of this testing APIs • AXL • RisPort • JTAPI • EMAPI Planned Systems • EXPW • UCXN • IOS (CUBE/VG) • CNDR Support is mandatory for the test plan designer component and you will have a dedicated Yarnlab
  30. 30. Custom Test Plans built by Yarnlab If you need something Different As an interim measure until the test plan designer is ready or if you want a fully supported Yarnlab can build test plans on request using our team of experts Talk to us for more details on this process
  31. 31. Dial plan Demo Run Through Setup (pre canned) 1. Create A. Number group B. CSS group C. Expected results D. Geographic locations Demo 1. Create test plan 2. Execute test plan Screens to show • Status • Live view • Logs • Reports • Re run x 3 • Clone
  32. 32. Screenshot Number Groups
  33. 33. Screenshot CSS Groups
  34. 34. Screenshot Real time view
  35. 35. Screenshot Logging Console
  36. 36. Our platform
  37. 37. Architecture What and where the modules are Yarnman is a modular platform that allows for components to deployed as a single server or as a distributed system to support multi tenancy Core Hosts the engine for workflows and any workers Datastore Hosts the database and message bus Arm/Interconnect Hosts the interfaces for web management, services. It provides the mechanism to traverse the networks and hosts the interconnects Core Datastore Arm Customer 1 Customer 2
  38. 38. Yet another platform The reasons why We built the Yarnman platform to with the following in mind • Multi tenancy from the ground up • Secure network traversal • Centralised management • Central and distributed components • Credential and data security • Data separation (sovereignty) • Resiliency What the Yarnman platform provides • Single centralised RBAC • CMDB (device inventory) • Centralised Logging • Data store Message bus • Yarnapp hosting • Centralised upgrade
  39. 39. 321 Pitt St Sydney Australia Wattstraße 11 Berlin, Germany Contact [at] Demo videos coming soon KEEP IN TOUCH WITH US We don’t bite Images Courtesy of Cisco Systems, Inc. Unauthorized use not permitted.