“Applying agile test to complex embedded systems”                © 2012 Intune Networks Ltd. Confidential   1
How hard can it be?                      © 2011 Intune Networks Ltd. Confidential   2
Distributed Ethernet optical switch – initial release    Up to 10 nodes per ring    80Gbps input and output ports per no...
Sometimes it felt more like..                      © 2011 Intune Networks Ltd. Confidential   4
The $1,000,000 Question• How do we test  increasingly  complex systems in  increasingly shorter  time intervals?   • Compo...
Integration Order VTE                                                                             System   Application    ...
Integration methodology                   Backlog   NMRC SW ___________________     TCS HW___________________      NPC SW_...
Programme Map with SPIKES                                                                                                 ...
Milestone Tracking and Traceability to RequirementsAREA                                    Pass/Fail CommentsUse of Develo...
© 2011 Intune Networks Ltd. Confidential   10
Extensive Use Automation    60% Integration team are automation primes.     1000 Automated Tests nightly against mainlin...
Automation Example : int_nightly_hw_regr Tests are organised into Test Umbrella’s that test specific  areas.   • int_nigh...
Automation Trending Automation allows specific test areas to be investigated or trended  over time. E.g OLC Sprint 32  ol...
Hardware Test Environment               •   Processors                     • 256/512M                     • PowerQUICC III...
Virtual Test Environment                                                           1     • 35 Processors                  ...
Utilisation NPC Card                       © 2011 Intune Networks Ltd. Confidential   16
Automation and Virtual Test EnvironmentAREA                                                      CommentsVirtual Test Envi...
Conclusions   Drive programme from clearly defined integration strategy, with    staged integration points between comple...
Upcoming SlideShare
Loading in...5
×

Intune Agile Testing Talk with www.softtest.ie

758

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
758
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Intune Agile Testing Talk with www.softtest.ie

  1. 1. “Applying agile test to complex embedded systems” © 2012 Intune Networks Ltd. Confidential 1
  2. 2. How hard can it be? © 2011 Intune Networks Ltd. Confidential 2
  3. 3. Distributed Ethernet optical switch – initial release Up to 10 nodes per ring 80Gbps input and output ports per node– ≡ 7.2TBps of fully meshed capacity 12Gbit/s INEX Peak Internet traffic Use in 2011. 500K Lines of code Distributed Architecture – 6 Architecturally distinct Subsystems – 3 Primary Software Loads iVX 8000 * In practice 500Gbps due to collision detection overhead © 2011 Intune Networks Ltd. Confidential 3
  4. 4. Sometimes it felt more like.. © 2011 Intune Networks Ltd. Confidential 4
  5. 5. The $1,000,000 Question• How do we test increasingly complex systems in increasingly shorter time intervals? • Compounded.. Integration led Extensive R&D strategy Automation & VTE © 2011 Intune Networks Ltd. Confidential 5
  6. 6. Integration Order VTE System Application Application Hardware Abstraction Layer Proto 1W HW Device Drivers Device Drivers Device Drivers Hardware Demo Hardware © 2011 Intune Networks Ltd. Confidential 6
  7. 7. Integration methodology Backlog NMRC SW ___________________ TCS HW___________________ NPC SW___________________ NPC HW___________________ DSSE HW___________________ OFS SW___________________ OFS HW___________________ Integration SPIKE ___________________ Spike 1 Spike 3 Integration Spikes: Identified high risk (complex, impact) interfaces R&D backlog and teams aligned to achieve spikes © 2011 Intune Networks Ltd. Confidential 7
  8. 8. Programme Map with SPIKES Integration CR-1 Test Strategy Q2 09 Q3 09 Q4 09 Q1 10 Mar/Apr June/July Sept/Oct Dec/Jan HA1 HA1 HA1 HA1 TCSOptics B DSSE CME 24/3 24/3 22/5 HA1 ??? LA1 NPC 21/6 iNX 8000 Shelf OAM &BI Rollout BI modelling three BI Modelled in Real nodes with real Shelf with IPsubnet BI Robusteness Shelf TCS Restarting Port All Cards Ring Testing SU_A1 SU_A2 SU_A3 SU_A4 Ring Protected Shelf Comm Shelf Level Ring IP Services Prototype IP SUbnet SUBNET IP IP IP IP SU_A1 SU_A2 SU_A3 SU_A4 CC, Topology, IPSUBNET Marvell DX167 CC GFP 1 Devel Board 2 Connect 4 X Y 3 Switches CC Physical CC CC CC GigE GigE Layer (optics) SU_A2 SU_A3 SU_A1 Freescale Freescale CC Scheduler 8548 TCS 8548 TCS & Robustness QNX BSP Management BT Simulated Discovery ? Maintenance Adj BT BT SU_A1 SU_A2 APR Ring Power Cal Ring Level Lambda Assign I/P Port 90/10 Delay C.A. 80/20 EDF C.L. WSS VOA = Assembly (within WSS) 2x2 95/5 O/P Port Ring OLC OLC OLC A switch 95/5 C.C. PD_CC FOA Mon Port 1st stage VOA =  VOA =  C.C. Bringup SU_A4 VOA_CC C.C. SFP VOA =  (within WSS) PD_O SU_A2 SU_A3 Scenarios OLC DCM (40 km) C.D. PD_D1 C.Off. CIRC EDF A VOA_O within EDFA OLC Section OLC PD_A SOA GFF PD_D2 4 Star Coupler VOA_A 2nd stage CC , EDFA/WSS, SU_A4 PD_D3 Arm uP Carrier Gain Adaption Sense PD_D4 I2C Ethernet MII I2C CS, SFP Control C.C. SFP Simple Test Card 1200 Laser JTAG Alarms/ Interrupts Ethernet MII Physical Shelf RS232 TCS Module L2 Switch TCS-DSSE Interconnect Ring Data Path Cont Mode, Physical Shelf PAT Cont Mode AURA iNX 8000 Loopback Simple OPR Aura IF & Mem Shelf Burst Mode Simple SCheduler PHYSICAL/DD TCS-TXP DATA PATH HS Control Loops DSSE DP DP DP Line Loopback HA2a SU_A1 SU_A2 SU_A3 TXP HA2 NPC TCS-NPC Cont Mode HA2a iNX 8000 Ethernet Services Aura IF & Mem SU_A2 Shelf (No Marvel) Marvel, Eline Eth Serv © 2011 Intune Networks Ltd. Confidential 8
  9. 9. Milestone Tracking and Traceability to RequirementsAREA Pass/Fail CommentsUse of Development SPIKES Provided R&D focus, Investor Visable Milestones, andto enable cross subsystem enabled key risks to be proven earlyearly integration  Agile methodology made it difficult to accurately define. Initially only happy path focus. Strengthen Q3, Q4 Test Regime Clear Architectural definition of Hardware Abstraction Layer. Logging Capability & Treating HW and Device Drivers together Prototype hardware using reference board to host early software loads (pre-hardware availability)  Automation of early milestones & spikes. To enable n-1 regression All System UC and Subsystem UC Required for each milestone is maintained & tracked within Database Virtual Test Environment and formalised Creation HAL interface to enable distributed system test without hardware © 2011 Intune Networks Ltd. Confidential 9
  10. 10. © 2011 Intune Networks Ltd. Confidential 10
  11. 11. Extensive Use Automation 60% Integration team are automation primes.  1000 Automated Tests nightly against mainline (trunk): o Execution time 12 h, 207 HW, 700 VTE Significantly greater coverage than normal Sanity  Tests explore both System functionality and Subsystem Daily Inspection of all test automation results in 9:30 Am meeting  Daily Bug discussion of critical issues preventing performance © 2011 Intune Networks Ltd. Confidential 11
  12. 12. Automation Example : int_nightly_hw_regr Tests are organised into Test Umbrella’s that test specific areas. • int_nightly_hw_regr: SummaryHW - 20111031-104218 Campaign No. Category Total Run % Run Pass Fail % Pass ID 1 BI 85462 45 45 100% 42 3 93% 2 CCS 85462 22 22 100% 22 0 100% 3 ECS 85462 32 32 100% 32 0 100% 4 ECS_OPST 85462 32 32 100% 32 0 100% 5 OLC 85462 10 10 100% 10 0 100% 6 OPST_DP 85462 29 29 100% 28 1 97% 7 SANITY 85462 7 7 100% 6 1 86% 8 SYSTEM 85462 30 30 100% 27 3 90% TOTAL 207 207 100% 199 8 96% Other packs include ECS_OPST, OLC_DP, OLC_Bringup etc Some Test Packs only suitable for Weekend runs © 2011 Intune Networks Ltd. Confidential 12
  13. 13. Automation Trending Automation allows specific test areas to be investigated or trended over time. E.g OLC Sprint 32 olc_Sprint32 Example for one Testpack, being extended for individual tests © 2011 Intune Networks Ltd. Confidential 13
  14. 14. Hardware Test Environment • Processors • 256/512M • PowerQUICC III • 35 Processors 1.3Gbits • 60GBit RAM • Total • 45GHz Processing • 7GHz Processor • 10 GB RAM © 2011 Intune Networks Ltd. Confidential 14
  15. 15. Virtual Test Environment 1 • 35 Processors Ring • 10GBit RAM • 45GHz Processing • vSphere Virtualises Server • total 210 virtual processors HP DL380 Server • 60Gbits RAM Dual Hex Core 2.8GHz • Approx 0.5TB Storgae (HD Image 2Gbits/CPU) 64GBits RAM • VTE Costs= 10-12K • Hardware Cost £xxMillion 1 1 1 1 1 1 © 2011 Intune Networks Ltd. Confidential 15
  16. 16. Utilisation NPC Card © 2011 Intune Networks Ltd. Confidential 16
  17. 17. Automation and Virtual Test EnvironmentAREA CommentsVirtual Test Environment Needs Controlled and as much attention as Hardware Environment. Firewall server/resources and ensure source control of VM machine/memory requirements  Clarify Ownership of Test Areas Initial organisational separation made structured re-use procedures, classes difficult Prototype hardware using reference board to host early software loads (pre-hardware availability)Automation General  Be prepared to Retire Tests Maintenance early agile tests often not worth lifetime cost. Allow to die & replace with better  Consider Lifetime costs Focused initially on automation for device drivers and FPGA streams, chose TCL language without long term focus for software (moved Python) All System UC and Subsystem UC Required Object Orientated, structured test environment. for each milestone is maintained & tracked within Database Classes. Design Authority as per Software. Common Clearly defined logging levels. Too much debug as destructive as too little 17 © 2011 Intune Networks Ltd. Confidential
  18. 18. Conclusions Drive programme from clearly defined integration strategy, with staged integration points between complex subsystems  Treat as full milestone review and apply “waterfall”rigour to these points  Track programme overall according to milestones vs 4 week agile sprints Invest in Virtual Test Environment  Massively cost efficient  Build into process and architecture, clean hardware abstraction layer  Apply source control to VM’s Next Steps: Automated Assistance Manual Testing NBI Interface Compliance vs WADL © 2011 Intune Networks Ltd. Confidential 18

×