Your SlideShare is downloading. ×
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Intune Agile Testing Talk with www.softtest.ie
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Intune Agile Testing Talk with www.softtest.ie

745

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
745
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. “Applying agile test to complex embedded systems” © 2012 Intune Networks Ltd. Confidential 1
  • 2. How hard can it be? © 2011 Intune Networks Ltd. Confidential 2
  • 3. Distributed Ethernet optical switch – initial release Up to 10 nodes per ring 80Gbps input and output ports per node– ≡ 7.2TBps of fully meshed capacity 12Gbit/s INEX Peak Internet traffic Use in 2011. 500K Lines of code Distributed Architecture – 6 Architecturally distinct Subsystems – 3 Primary Software Loads iVX 8000 * In practice 500Gbps due to collision detection overhead © 2011 Intune Networks Ltd. Confidential 3
  • 4. Sometimes it felt more like.. © 2011 Intune Networks Ltd. Confidential 4
  • 5. The $1,000,000 Question• How do we test increasingly complex systems in increasingly shorter time intervals? • Compounded.. Integration led Extensive R&D strategy Automation & VTE © 2011 Intune Networks Ltd. Confidential 5
  • 6. Integration Order VTE System Application Application Hardware Abstraction Layer Proto 1W HW Device Drivers Device Drivers Device Drivers Hardware Demo Hardware © 2011 Intune Networks Ltd. Confidential 6
  • 7. Integration methodology Backlog NMRC SW ___________________ TCS HW___________________ NPC SW___________________ NPC HW___________________ DSSE HW___________________ OFS SW___________________ OFS HW___________________ Integration SPIKE ___________________ Spike 1 Spike 3 Integration Spikes: Identified high risk (complex, impact) interfaces R&D backlog and teams aligned to achieve spikes © 2011 Intune Networks Ltd. Confidential 7
  • 8. Programme Map with SPIKES Integration CR-1 Test Strategy Q2 09 Q3 09 Q4 09 Q1 10 Mar/Apr June/July Sept/Oct Dec/Jan HA1 HA1 HA1 HA1 TCSOptics B DSSE CME 24/3 24/3 22/5 HA1 ??? LA1 NPC 21/6 iNX 8000 Shelf OAM &BI Rollout BI modelling three BI Modelled in Real nodes with real Shelf with IPsubnet BI Robusteness Shelf TCS Restarting Port All Cards Ring Testing SU_A1 SU_A2 SU_A3 SU_A4 Ring Protected Shelf Comm Shelf Level Ring IP Services Prototype IP SUbnet SUBNET IP IP IP IP SU_A1 SU_A2 SU_A3 SU_A4 CC, Topology, IPSUBNET Marvell DX167 CC GFP 1 Devel Board 2 Connect 4 X Y 3 Switches CC Physical CC CC CC GigE GigE Layer (optics) SU_A2 SU_A3 SU_A1 Freescale Freescale CC Scheduler 8548 TCS 8548 TCS & Robustness QNX BSP Management BT Simulated Discovery ? Maintenance Adj BT BT SU_A1 SU_A2 APR Ring Power Cal Ring Level Lambda Assign I/P Port 90/10 Delay C.A. 80/20 EDF C.L. WSS VOA = Assembly (within WSS) 2x2 95/5 O/P Port Ring OLC OLC OLC A switch 95/5 C.C. PD_CC FOA Mon Port 1st stage VOA =  VOA =  C.C. Bringup SU_A4 VOA_CC C.C. SFP VOA =  (within WSS) PD_O SU_A2 SU_A3 Scenarios OLC DCM (40 km) C.D. PD_D1 C.Off. CIRC EDF A VOA_O within EDFA OLC Section OLC PD_A SOA GFF PD_D2 4 Star Coupler VOA_A 2nd stage CC , EDFA/WSS, SU_A4 PD_D3 Arm uP Carrier Gain Adaption Sense PD_D4 I2C Ethernet MII I2C CS, SFP Control C.C. SFP Simple Test Card 1200 Laser JTAG Alarms/ Interrupts Ethernet MII Physical Shelf RS232 TCS Module L2 Switch TCS-DSSE Interconnect Ring Data Path Cont Mode, Physical Shelf PAT Cont Mode AURA iNX 8000 Loopback Simple OPR Aura IF & Mem Shelf Burst Mode Simple SCheduler PHYSICAL/DD TCS-TXP DATA PATH HS Control Loops DSSE DP DP DP Line Loopback HA2a SU_A1 SU_A2 SU_A3 TXP HA2 NPC TCS-NPC Cont Mode HA2a iNX 8000 Ethernet Services Aura IF & Mem SU_A2 Shelf (No Marvel) Marvel, Eline Eth Serv © 2011 Intune Networks Ltd. Confidential 8
  • 9. Milestone Tracking and Traceability to RequirementsAREA Pass/Fail CommentsUse of Development SPIKES Provided R&D focus, Investor Visable Milestones, andto enable cross subsystem enabled key risks to be proven earlyearly integration  Agile methodology made it difficult to accurately define. Initially only happy path focus. Strengthen Q3, Q4 Test Regime Clear Architectural definition of Hardware Abstraction Layer. Logging Capability & Treating HW and Device Drivers together Prototype hardware using reference board to host early software loads (pre-hardware availability)  Automation of early milestones & spikes. To enable n-1 regression All System UC and Subsystem UC Required for each milestone is maintained & tracked within Database Virtual Test Environment and formalised Creation HAL interface to enable distributed system test without hardware © 2011 Intune Networks Ltd. Confidential 9
  • 10. © 2011 Intune Networks Ltd. Confidential 10
  • 11. Extensive Use Automation 60% Integration team are automation primes.  1000 Automated Tests nightly against mainline (trunk): o Execution time 12 h, 207 HW, 700 VTE Significantly greater coverage than normal Sanity  Tests explore both System functionality and Subsystem Daily Inspection of all test automation results in 9:30 Am meeting  Daily Bug discussion of critical issues preventing performance © 2011 Intune Networks Ltd. Confidential 11
  • 12. Automation Example : int_nightly_hw_regr Tests are organised into Test Umbrella’s that test specific areas. • int_nightly_hw_regr: SummaryHW - 20111031-104218 Campaign No. Category Total Run % Run Pass Fail % Pass ID 1 BI 85462 45 45 100% 42 3 93% 2 CCS 85462 22 22 100% 22 0 100% 3 ECS 85462 32 32 100% 32 0 100% 4 ECS_OPST 85462 32 32 100% 32 0 100% 5 OLC 85462 10 10 100% 10 0 100% 6 OPST_DP 85462 29 29 100% 28 1 97% 7 SANITY 85462 7 7 100% 6 1 86% 8 SYSTEM 85462 30 30 100% 27 3 90% TOTAL 207 207 100% 199 8 96% Other packs include ECS_OPST, OLC_DP, OLC_Bringup etc Some Test Packs only suitable for Weekend runs © 2011 Intune Networks Ltd. Confidential 12
  • 13. Automation Trending Automation allows specific test areas to be investigated or trended over time. E.g OLC Sprint 32 olc_Sprint32 Example for one Testpack, being extended for individual tests © 2011 Intune Networks Ltd. Confidential 13
  • 14. Hardware Test Environment • Processors • 256/512M • PowerQUICC III • 35 Processors 1.3Gbits • 60GBit RAM • Total • 45GHz Processing • 7GHz Processor • 10 GB RAM © 2011 Intune Networks Ltd. Confidential 14
  • 15. Virtual Test Environment 1 • 35 Processors Ring • 10GBit RAM • 45GHz Processing • vSphere Virtualises Server • total 210 virtual processors HP DL380 Server • 60Gbits RAM Dual Hex Core 2.8GHz • Approx 0.5TB Storgae (HD Image 2Gbits/CPU) 64GBits RAM • VTE Costs= 10-12K • Hardware Cost £xxMillion 1 1 1 1 1 1 © 2011 Intune Networks Ltd. Confidential 15
  • 16. Utilisation NPC Card © 2011 Intune Networks Ltd. Confidential 16
  • 17. Automation and Virtual Test EnvironmentAREA CommentsVirtual Test Environment Needs Controlled and as much attention as Hardware Environment. Firewall server/resources and ensure source control of VM machine/memory requirements  Clarify Ownership of Test Areas Initial organisational separation made structured re-use procedures, classes difficult Prototype hardware using reference board to host early software loads (pre-hardware availability)Automation General  Be prepared to Retire Tests Maintenance early agile tests often not worth lifetime cost. Allow to die & replace with better  Consider Lifetime costs Focused initially on automation for device drivers and FPGA streams, chose TCL language without long term focus for software (moved Python) All System UC and Subsystem UC Required Object Orientated, structured test environment. for each milestone is maintained & tracked within Database Classes. Design Authority as per Software. Common Clearly defined logging levels. Too much debug as destructive as too little 17 © 2011 Intune Networks Ltd. Confidential
  • 18. Conclusions Drive programme from clearly defined integration strategy, with staged integration points between complex subsystems  Treat as full milestone review and apply “waterfall”rigour to these points  Track programme overall according to milestones vs 4 week agile sprints Invest in Virtual Test Environment  Massively cost efficient  Build into process and architecture, clean hardware abstraction layer  Apply source control to VM’s Next Steps: Automated Assistance Manual Testing NBI Interface Compliance vs WADL © 2011 Intune Networks Ltd. Confidential 18

×