Typical Interview Question• Do you know programming• Answer: Yes, a little• Which language do you know the best• Answer: I know C a little• How many lines of code have you written• Answer : Umm.., about 300 lines• Do you know Java or .NET,• Answer : No• Do you know Linux• Answer : No• Do you know networking• Answer : I installed Windows XP on my PC….• Ok, you can work for us as testing engineer
Tester’s Dilemma• Testing is not cool• Tester is second class when compared withdeveloper• Developer using me as servant• I don’t see I can make difference in our company• Testers are keep coming, they are young andbetter than me.• 我在吃青春饭， I don’t know where my careerpath is• OK ， I need to be a developer and write code!
Top Managements Dilemma也知道， 在招到好的 工程 是有多你 现 测试 师 难- 北京研究所副所华为 长 2008It took Juniper 6 month plus to find a new QAdirector
Career Story of Ting• Graduate at 1990, only job found is the tester at Sun• Join Cisco as tester in 1993, then test automationengineer (design Cisco’s automation framework)• Cisco grows from 2000 to 40000 from 1993 to 1998• Join Netscreen as the 1st testing engineer at 1998• NetScreen went to Nasdaq on 2001• Grow with NetScreen as testing lead, testing manager,testing senior manager and testing director• Found Sigma at 2004, served as CEO as today
Lesson• You may have to stay long enough to becompetitive.
What is a testing expert?• Ability to find critical bugs in given time frame(hot gun)• Ability to build a comprehensive testingstrategy in given time frame (expert)• Ability to manage the release process (topexpert)
What is a testing expert Looks like?• Play the video of James Bach
Bugs in the News• A Cisco Secure Access Control Server (ACS) that isconfigured to use Extensible AuthenticationProtocol-Transport Layer Security (EAP-TLS) toauthenticate users to the network will allowaccess to any user that uses a cryptographicallycorrect certificate which can be expired, or comefrom an untrusted Certificate Authority (CA) andstill be cryptographically correct.• CSCse58195. The WLC contains a bug whenprocessing WLAN ACLs that causes theWLANvACL configuration to be saved with aninvalid checksum. When the configuration issubsequently reloaded at boot time, thechecksum fails and the WLAN ACLs are notinstalled.
Bugs in the News• CSCdv24925 It is possible to read storedconfiguration file from the Storage Routerwithout any authorization.• CSCdu45417 It is possible to halt the StorageRouter by sending a fragmented packet overthe Gigabit interface.• CSCdv24925 An unauthorized person mayread the configuration of the Storage Router.That may lead to unauthorized access of astorage space.
Bugs in the News• Versions of the Cisco ACE 4710 ApplicationControl Engine appliance prior to softwareversion A1(8a) use default administrator, webmanagement, and device managementaccount credentials. The appliance andmodule do not prompt users to modify systemaccount passwords during the initialconfigurationprocess.• Crafted SSH Packet Vulnerability• Crafted SNMPv2c Packet Vulnerability
2 Factors of a hot gun• Technical Expertise• Thinking methodology
Hot Gun’s Bug PercentageP4P2P1P3CosmeticFunctionalityMajor FunctionalityCritical Functionality, Crash, Hang10%30%40%20%
Develop Testing Strategy• Be the review person• 知其然也知其所以然
Develop Testing Strategy• What is your goal first?– Find more bugs?– Find more critical bugs?– Ensure product or feature has no critical defects– Ensure customers will be ok after the release?• The strategy– Bug oriented?– Coverage oriented?– Customer oriented?
What is Coverage Strategy?• How to thoroughly test OSPF Hello protocol?– Function points? (tester)– User Scenarios? (test expert)– Scalability? (test expert)– Performance?– Security? (test expert)– ………
What is Coverage Strategy?• 7 platforms• 6 different line cards• 2 modes (main/aggressive)• AH/ESP• CA/No CA• HA/No HA• Hub Spoke/Partial Mesh/Full Mesh7x6x2x2x2x3 = 1088
Types of Testing Covered on Different ReleaseMajorReleaseMinorReleasePatchReleasePlatformReleaseSFR CSPNew Feature Test Full Full TBD FullRegression Test Full Partial Partial Partial PartialSystem Test Full Full Full FullInteroperability Test Full TBD TBD TBDPerformance Test Full TBD Full TBDCapacity Test Full Full TBDSecurity Test Full Full Full Full FullAutomation Test(partial regression)Full Full Full Full Full FullSFR – Special Feature Release CSP – Customer SpecificPatch
Advise to the New Expert• Practice, Practice, Practice• Don’t confuse experience with Expertise• Don’t trust the folklore – but learn it anyway• Take nothing on faith, own your methodology• Drive your own education, no one else will• Reputation = Money: Build and protect yourreputation• Relentlessly gather resources, materials and tools.• Associate with demanding colleagues.• Write, speak
An Expert’s Vision• An Expert’s Vision– I can test anything– Under any condition– On any given time frame.