Lessons Learned in Software DevelopmentQA Infrastructure – Maintaining Robustness in Commercial Software<br />Marcus Lager...
About the Speaker<br />Marcus Lagergren holds a master’s degree from KTH, major in Theoretical Computer Science<br />Marcu...
Agenda<br />Robustness in commercial apps with tight release schedules.<br />Utopian vision: perpetual stable bits so we c...
Agenda<br />Result databases<br />Automatic Testing<br />Complex and not-so-standard testing<br />Development aspects<br />
Why listen to me?<br />We’vespent the last 10 yearsdeveloping a JVM and the last 3 yearsdeveloping a Guest Operating syste...
BEA Confidential.  |   6<br />QA infrastructure<br />QA infrastructure is harder and probably even more important than dev...
QA infrastructure<br />QA and Dev, if separate deparments or roles, shouldalways work together. <br />Preferably as physic...
Build System<br />Build system, test system and sourcecontrol are parts of the same distributed system. <br />Mobility - B...
Source Control and Development<br />Needgood support for distributeddevelopment<br />Should be able to handledirectories a...
Test System<br />Also under sourcecontrol<br />Distributed system – veryimportant.<br />Virtualizeifpossible. Maximizereso...
Building Blocks – Tests<br />Many tests, especially regression tests, for an appneedn’t be morethan a mainclass with a ret...
Building Blocks – Tests<br />Easy-to-write tests make it possible for the test suite to grownaturally. <br />If 10 minutes...
Building Blocks – Result Database<br />Store results in cheapdatabase with sensible layout somewhere.<br />Any freeware is...
Building Blocks - Tests<br />Use ”terror harnesses” that attack the cross sectionsbetweenmodules. <br />AllocAndRun<br />R...
Building Blocks - Performance<br />Anythingcaneffects performance.<br />EVERYTHING affects performance.<br />Weneedautomat...
Testing – The need for continuous automatic testing<br />Needcontinuousautomatic testing.<br />Example from real life: JRo...
Testing – So What About the Other 50%?<br />Simple Java programs with main functions may not be enough for all the bugs.<b...
Testing – So What About the Other 50%?<br />Examples:<br />Create a very special heap with a fewobjects in nastyplaces.  L...
Testing – So What About the Other 50%?<br />But of courseit’s not as simple as that.<br />Whataboutmultithreadedapps? Race...
Testing – So What About the Other 50%?<br />Disclaimer: Sometimeswe just need to crunch a lot of code for a long, long tim...
Testing – Retrofitting a framework	<br />You willprobablyhave to do this, sincepeopledon’tunderstand the importance of fun...
Development – The platform matrix<br />Try to keep the amount of common code as large as possible.<br />It is always a cho...
Development – The platform matrix<br />Otherseemlinglyplatformdependentthingscan be madeplatform independent.<br />Example...
Development<br />Don’tlosefocus. Modularity first.<br />Example: ”the fastest server side JVM”, ”startup time is an issue”...
Development - Policy<br />Don’t be toomuch of a quality fascist whencode is written.<br />If you spend all your time preve...
Lessons Learned <br />Summary – The important stuff to bring with you<br />Build the test infrastructure in parallel with ...
Lessons Learned in Software Development: QA Infrastructure – Maintaining Robustness in Commercial Software
Upcoming SlideShare
Loading in …5
×

Lessons Learned in Software Development: QA Infrastructure – Maintaining Robustness in Commercial Software

2,415
-1

Published on

1st DSV-PhD Workshop: Keynote speech by Marcus Lagergren, Oracle Inc.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,415
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Lessons Learned in Software Development: QA Infrastructure – Maintaining Robustness in Commercial Software

  1. 1. Lessons Learned in Software DevelopmentQA Infrastructure – Maintaining Robustness in Commercial Software<br />Marcus Lagergren<br />Consulting Member of Technical Staff<br />Oracle Corporation<br />
  2. 2. About the Speaker<br />Marcus Lagergren holds a master’s degree from KTH, major in Theoretical Computer Science<br />Marcus was one of the founders of Appeal Virtual Machines that was acquired by BEA in 2002, which in turn was acquired by Oracle in 2008.<br />Marcus has worked on almost all aspects of the JRockit Virtual Machine and is now working with Virtualization technology<br />Marcus likes power tools and scuba diving.<br />
  3. 3. Agenda<br />Robustness in commercial apps with tight release schedules.<br />Utopian vision: perpetual stable bits so we can spin off a release at any time<br />Build Systems<br />Source control and Development<br />Tests<br />Functionality<br />Performance<br />Regression testing<br />
  4. 4. Agenda<br />Result databases<br />Automatic Testing<br />Complex and not-so-standard testing<br />Development aspects<br />
  5. 5. Why listen to me?<br />We’vespent the last 10 yearsdeveloping a JVM and the last 3 yearsdeveloping a Guest Operating system for twocommercial hypervisors.<br />Hundreds of thousands on man hoursspent on robustnessalone<br />Harder-to-debug software hardlyexists. <br />We’vehad to invent stuff from dayone.<br />Ok, from day 365 or so, lessonslearned<br />We’vemademanymistakesalong the way.<br />No one gets to Utopia, but at leastwehave a reasonablygoodidea of in whichdirection to go<br />
  6. 6. BEA Confidential. | 6<br />QA infrastructure<br />QA infrastructure is harder and probably even more important than development infrastructure.<br />The most valuable lesson we have learned is that it must be developed parallel to the application and significant effort must be spent on it. <br />It is at least as important as the application itself.<br />Sometimes the boundaries between app and test infrastructure aren’t even clear.<br />
  7. 7. QA infrastructure<br />QA and Dev, if separate deparments or roles, shouldalways work together. <br />Preferably as physicallyclose to eachother as possible<br />Theyshould be able to fill in for eachother and be encouraged to doeachothers’ work.<br />Verydangerous to have a separate QA department on anotherfloor.<br />Verydangerous for QA to just do blackbox testing withoutunderstandingwhat’s in the box.<br />QA staffshould be treated as anyotherdeveloper<br />
  8. 8. Build System<br />Build system, test system and sourcecontrol are parts of the same distributed system. <br />Mobility - Buildanythinganywhere, locally or globally (distributed) - ”Adistributed cross compiler”<br />Build system should be selfcontained & part of sourcecontrol. <br />Do a sync on a fresh laptop, have all the details.<br />We chose to putbinariesthere as well to producedeterministic bits and provide selfsufficience<br />Not always a goodidea, butmostly a goodidea<br />
  9. 9. Source Control and Development<br />Needgood support for distributeddevelopment<br />Should be able to handledirectories as separate sourcecontrolentitites.<br />Gatekeepers of mainbranches, distributed team baseddevelopment.<br />Sourcecontrol, builds and developmentshouldonlyrequire vi + prompt<br />Morecomplexenvironments on top for ease of use.<br />Easier to extend with different UIs.<br />
  10. 10. Test System<br />Also under sourcecontrol<br />Distributed system – veryimportant.<br />Virtualizeifpossible. Maximizeresourceusage.<br />Local and remote test runspossible.<br />Submitjobs ”crunchthroughthese tests”<br />”Check in ifpasses tests”. <br />Test Machines<br />Performance test machines (dedicated)<br />Functionality test machines (not necessarilydedicated)<br />Anymachinecanvolounteer CPU cycles for functional testing.<br />
  11. 11. Building Blocks – Tests<br />Many tests, especially regression tests, for an appneedn’t be morethan a mainclass with a returnvalue.<br />Keep it simple!<br />”I spent a fewhoursdistilling this huge program down to a reproducer for BUG123456”<br />Claim:ifit’s simple enough to write and submit a test, &gt; 50% of the bugscan get regression tests as part of the original bugfix. <br />I willaddress the other 50% later.<br />
  12. 12. Building Blocks – Tests<br />Easy-to-write tests make it possible for the test suite to grownaturally. <br />If 10 minutes of spare time canlead to a new test beingwritten, checked in and enabled as part of the global test suite, you havesucceeded. <br />Encouragedevelopers to check in unit tests for new functionalitytogether with the functionality.<br />Need the infrastructure for it in the app<br />Mightwant to enforce this strictly, but it might hinder developmenttoo.<br />
  13. 13. Building Blocks – Result Database<br />Store results in cheapdatabase with sensible layout somewhere.<br />Any freeware is fine – get it up and running. <br />Easy to maintain and backup<br />Query from localmachinesabouthistorical test results.<br />”Whenexactlydid this performance regression appear?” <br />”List all benchmarkscores on this machine for this benchmarksinceJanuary 1”<br />”Has this functional test failedbefore? Whatwere the bugfixes?”<br />
  14. 14. Building Blocks - Tests<br />Use ”terror harnesses” that attack the cross sectionsbetweenmodules. <br />AllocAndRun<br />RedefineClasses<br />ExceptionInClinit<br />
  15. 15. Building Blocks - Performance<br />Anythingcaneffects performance.<br />EVERYTHING affects performance.<br />Weneedautomatic regression warnings. Anyone who submits a performance regression will get an e-mail from the test system.<br />Continuously make it easy to addmorebenchmarks.<br />Automation: Deviations, baselines, invariants. <br />
  16. 16. Testing – The need for continuous automatic testing<br />Needcontinuousautomatic testing.<br />Example from real life: JRockit Solaris has beenmadeavailableoff and on over the years. Bit rot sets in immediatelywhenremoved from automated testing. <br />Release version may break debug version and vice versa.<br />Linux version may break Windows version and vice versa.<br />Useextremelystrict and pickycompilerflags.<br />
  17. 17. Testing – So What About the Other 50%?<br />Simple Java programs with main functions may not be enough for all the bugs.<br />How do we test for a specific optimization bug in the code generator? <br />How do we test for a strange boundary case that crashes the GC, that happens after two weeks in production?<br />Key observation: We need to export a state.<br />
  18. 18. Testing – So What About the Other 50%?<br />Examples:<br />Create a very special heap with a fewobjects in nastyplaces. Load it and trigger a garbagecollection. Save it and compare to reference.<br />Serialize an IR from just before an offendingoptimization. Load it and trigger the optimization. Save the resulting IR and compare it to reference.<br />Comparewould be more of an ”equals” than a ”memcmp”<br />Weneed a level of modularizationthat’sgoodenough for this. <br />The collection of tests shouldgrownaturally, but the VM design shouldallow the ways of testing the VM to grownaturally as well.<br />
  19. 19. Testing – So What About the Other 50%?<br />But of courseit’s not as simple as that.<br />Whataboutmultithreadedapps? Race conditions? <br />Plenty of threadsoperate on the same memory – e.g. Multithreaded GC. Howcanwe make test cases?<br />Synchronization points. <br />Randomized input, randomizedsleeps. Try to cover the malicioussideeffects of parallelism. <br />ThingsliketheRaceTrackalgorithmcanfindsome (not all) races in staticcode, but the world is dynamic. Testing needs to be.<br />
  20. 20. Testing – So What About the Other 50%?<br />Disclaimer: Sometimeswe just need to crunch a lot of code for a long, long time. Nothingelsesuffices to reproduce a problem or the framework that would make it possibledoesn’texist.<br />So make sure the distributed system burnsthosefree CPU cycles<br />And make the dumps full and comprehensible. <br />Don’tlosethem, dammit! No wipingthem after 24h. Disk is cheap. <br />”Phonehome”<br />Suprisinglyeffectiveif you haveenough beta testers. <br />
  21. 21. Testing – Retrofitting a framework <br />You willprobablyhave to do this, sincepeopledon’tunderstand the importance of fundamental QA from day 1.<br />Situation: Weneed the QA infrastructurebutdon’thave it. Our app has come a longway.<br />Learn from history<br />For example, go over 500 bug parade entries for HotSpot.<br />Howmanycan be tested by small deterministicreproducers? <br />Whatabout the rest - brainstormwhatfunctionality the VM wouldneedifwehad to write a simple reproducer for each problem.<br />
  22. 22. Development – The platform matrix<br />Try to keep the amount of common code as large as possible.<br />It is always a choice between platform specific features and test matrix growth.<br />Initially, our performance critical code was native. As our JIT got better, we would write more and more in Java. Native is much worse. ”premethods”<br />Augmented Java – intrinsics, ”pd_addr”, preprocessed Java files. <br />
  23. 23. Development – The platform matrix<br />Otherseemlinglyplatformdependentthingscan be madeplatform independent.<br />Example: Native stubs. The bulk of the work is parameter marshalling, the register allocatorcando that already. <br />Beware of ”falseabstraction”. <br />That extra parameter that is NULL on all platformsexcept IA64.<br />Implementationlanguage: Debugging is an issue<br />Powerful C/C++ debuggersexist. Meta-debugging is usuallyharder. <br />
  24. 24. Development<br />Don’tlosefocus. Modularity first.<br />Example: ”the fastest server side JVM”, ”startup time is an issue”, ”clientapplications are an issue” ”weneedzero overhead runtime instrumentation”. Runfool! Run! <br />It is importantwhenoptimizing for performance not just too look at e.g.SPECjbb™and SPECjvm98™ Real world applicationsdo a lot of otherthings. <br />”There is no genericcommutative plus operator”. At leastnobodycares.<br />
  25. 25. Development - Policy<br />Don’t be toomuch of a quality fascist whencode is written.<br />If you spend all your time preventinglargercheckins or demand 100% testing on everythingnothingwillever get checked in.<br />If you demand a strictlydocumented process with specifications for everything, all anyonewilleverdo is to writespecifications and holdmeetings.<br />Both of the above are good in smalleramounts.<br />It’smore of an awarenessthing.<br />And the infrastructursshouldquickly and mercilesslyraise the alarm as soon as something breaks to preventfurtherdamage.<br />
  26. 26. Lessons Learned <br />Summary – The important stuff to bring with you<br />Build the test infrastructure in parallel with the application<br />Start at the same time! Don’tput it off. It is part of the appdevelopment process and should be in the time budget. <br />IdeallyDevand QA teams should be fused and be able to doeachother’sjobs. No separate compartments.<br />Don’t be afraid to couple it tightly in placesif that is what is required to maintainstability.<br />Use all available CPU cycles for testing<br />

×