The Evolution and Development of Quality Assurance in Gaming

The Evolution and Development of Quality Assurance in Gaming



Much of QA is about common sense and the ability to see risks and weaknesses in the project chain, but while testing for bugs is a core part - and obviously the original purpose - of QA, today that is ...

Much of QA is about common sense and the ability to see risks and weaknesses in the project chain, but while testing for bugs is a core part - and obviously the original purpose - of QA, today that is just one of many aspects that the QA department can provide vital feedback on; from level design and playability through to more subjective aspects like overall player experience. Read more...



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds


Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

The Evolution and Development of Quality Assurance in Gaming The Evolution and Development of Quality Assurance in Gaming Document Transcript

  • A BRIEF HISTORY OFTHE EVOLUTION AND DEVELOPMENTOF QUALITY ASSURANCE IN GAMINGin association withCONTENTSPart I: The Birth of Video Games and the Early Days of QAPart II: The Shift in Responsibilities: from Programmers to QA ExpertsPart III: The Birth of Outsourced QAPart IV: The Move to Managing Simultaneous In-House and Outsourced QA TeamsPart V: Embedded QA and Agile Methodologies
  • IntroductionPart I: The Birth of Video Games and the Early Days of QAAs video games have become increasingly advanced over the years, the merits of strong Quality Assurance (QA) beforea game release have become widely acknowledged by the industry, in order to ensure developers retain credibilityand, of course, maximise sales. Validating and verifying that new games meet the requirements during the design anddevelopment process has never been more important, as customer demand has increased and mobile devices proliferatefurther.Much of QA is about common sense and the ability to see risks and weaknesses in the project chain, but while testingfor bugs is a core part – and obviously the original purpose – of QA, today that is just one of many aspects that the QAdepartment can provide vital feedback on; from level design and playability through to more subjective aspects like overallplayer experience.In the early days, QA was often conducted by theprogrammers themselves. Back in 1971, Nolan Bushnelland Ted Dabney founded an engineering firm, SyzygyEngineering, that designed and built the first arcade videogame - Computer Space for Nutting Associates. In June of thefollowing year this company became Atari, Inc.Atari was a pioneer in arcade games, home video gameconsolesalmost single-handedly responsible for the explosionin popularity of video games. Unsurprisingly though, iconicgames like Pong (1972), not being overly complex, did notrequire much in the way of QA testing. Indeed from the 1970sthrough to the mid-1980s, the relatively straightforward gamesbeing produced meant that the developers could test thegames as they went (which in the right light could be viewedas an early concept of agile testing) - a far cry from the farmore complicated AAA titles of today.As the popularity of video games increased, more gamesbegan to be mass-produced in order to meet the growingdemand, often cutting corners in order to meet shortdevelopment deadlines and seasonal release dates. As atime-consuming and non-revenue generating function, testingwas often the area where most shortcuts were taken, withlittle thought given to how that may impact the overall qualityof the finished game.A great example of this is 1982’s E.T. the Extra-Terrestrial.Often cited as one of the worst games ever made, Atariskipped user playtesting in order to meet a very rushedChristmas release date. Aside from the numerous bugs thatwere not caught by the overstretched developers, playtestingwould have highlighted the poor gameplay and overall userexperience; who knows how things might have been for E.T.had Atari prioritised game quality and thorough testing overmeeting a seasonal deadline.In 1984 the industry fell into a major recession; partlyattributed to the this steady super-saturation of the marketwith hundreds of low-quality games. By the time the industrycame back to life a few years later with the release of theNintendo Entertainment System in 1985, new measureshad been put in place in order to control the third partdevelopment of software. The long term effect of this was thebirth of the platform-specific approval system that we knowtoday, beginning then with the widely recognised NintendoSeal of Quality, but for developers it meant more stringentquality control and testing. This then began to catalyse achange in the way QA was handled, with responsibilitiesstarting to shift away from the programmers and on to smallteams of test specialists – the birth of the QA team.
  • Part II: The Shift in Responsibilities: from Programmers to QA ExpertsCase Study #1: BioShock 2 (2010)Case Study #2: The Elder Scrolls V: Skyrim (2012)At this point, QA had become too much for programmers to manage by themselves, so companies needed dedicated QAteams. Nintendo began to embed QA teams as part of its business model, and as a result QA slowly became a mandatorypart of the internal game development process; the same model used today for game development across the globe.However, despite the introduction of more dedicated QA teams and more rigorous procedures, as the complexity ofgames increases, and the variety of platforms released on grows, it becomes less and less likely that a developer (orpublisher) will catch every bug. In fact, it is almost impossible to have a 100% bug-free game. That said, there have beensome examples in recent years of games that have a few more bugs left than they really should…BioShock 2, a sequel to the 2007 major international hit Bioshock, is a first-person shootersurvival horror video game developed by 2K Marin for Microsoft Windows, PlayStation 3 andXbox 360. It was released worldwide in February 2010 for Windows XP, Vista and Windows 7, butexperienced considerable problems with the game due to insufficient Quality Assurance testing.Although the Windows version of BioShock 2 had received some patches and downloadablecontent shortly after release, 2K Games announced in October 2010 that they would not providea final patch to fix remaining issues with the game. Furthermore, 2K Games announced theywould not offer the Protector Trials or Minerva’s Den content on the platform due to ‘timing andtechnical issues’ that they could not work around. 2K later explained that in trying to prepareadditional patches with support for the additional content, they encountered bugs under specificcircumstances that they felt would not be acceptable to the Microsoft certification process, andwhile they had tried to correct those bugs, they were unable to do so in a timely manner.The announcement was met with heavy criticism from some players, who had been promised game and DLC updates. In lateOctober, 2K Games reversed their decision based on the players’ responses, and have recommitted resources to complete thegame patch and support for both content packs on the Windows version; the patch and Protector Trial would likely be available inSpring 2011, while more work is necessary to complete the Minerva’s Den content.But then Protector Trials DLC for the PC version of BioShock 2 was made available prematurely via Games for Windows Livein February 2011 for 400 MS Points. However when users paid for and downloaded the update, they were unable to access it.In March 2011 Protector Trials was officially released for PC. The patch to the main game which enabled the DLC to run wasreleased after the DLC itself. Though 2K was originally reluctant to confirm a release date for the Minerva’s Den DLC, eventuallythey confirmed that it would be released at the end of May 2011.The Elder Scrolls V: Skyrim is an action role-playing open world video game developed by BethesdaGame Studios, published by Bethesda Softworks, and released in November 2011, for MicrosoftWindows, PlayStation 3 and Xbox 360.This is another prime example of a game plagued by a multitudeof technical issues, stemming from poor QA.As soon as the game was released players reported a range of issues, mainly connected with the gamecrashing in various circumstances. Most notably, the PS3 version of the game suffered a fatal bugwhereby once the player’s save file becomes larger than 6 MB players experience frame rate issues andslowdown to the point of being completely unplayable. Other issues included a texture down-scalingissue on the Xbox 360 version when the game was run from the hard drive, unexplained crashes andslowdown problems, as well as various crashes and slowdowns on the Windows version.It’s prequel, Elder Scrolls IV: Oblivion, also had numerous bugs, reported upon release of the Xbox 360version. Similarly, the problems were related to loading speed and lack of visual clarity, but technicalimprovements were made for the PS3 release. Load times were reduced, fewer framerate drops were experienced and several bugswere fixed. Draw distance was increased, and new shaders were included to render the foreground cleanly and sharply, leading to rockylandscapes with craggy appearances rather than smooth, non-distinct surfaces. The new shader sets blended near detail and far detailonscreen, removing the harsh line that cut between them in previous releases.Remarkably, despite Skyrim’s game-breaking PS3 issues and plethora of problems and the less-than-perfect reputation for bugs that aBethesda title carries with it, Skyrim still won multiple Game of the Year awards in 2012. Maybe there is hope for bug-ridden games yet…Ultimately, though, as consumers or industry professionals, we rarely hear about case studies of exceptional QA, and instead only hearabout the games that suffer from problems. Why? It’s because those games have few problems. By the very nature of QA, we only hearabout the bug-filled games, which is why getting it right is just so critical.
  • Part III: The Birth of Outsourced QAThe Pros and Cons of Outsourcing QA (vs. In-House teams)By the mid-2000s, higher consumer demands led to the rise of more complex games and ultimately an explosion of gamedevelopment. With high definition video an undeniable hit and longstanding gamers seeking immersive experiences,expectations for visuals in games along with the increasing complexity of games have resulted in a spike in thedevelopment budgets of gaming companies.Seventh-generation consoles came in, such as the Xbox 360, Play Station 3 and the Wii, accompanied by a range ofever-more elaborate games. For the Nintendo Wii, a higher level of QA was needed for games including Wii Sportsand Wii Fit, Super Mario Galaxy, The Legend of Zelda; for the Xbox 360 groundbreaking games included Call of Duty2, The Elder Scrolls IV: Oblivion, Gears of War and Halo 3; and for the PS3 the complex new range of games includedResistance: Fall of Man, The Elder Scrolls IV: Oblivion, and MotorStorm.Just as in the 80s-90s the demands of QA became too much for the programmers to manage alone, with the need formore extensive game testing the in-house QA teams became so big that there came a point where they proved toomuch to manage internally, and so once again a new solution was needed. This led to the rise in popularity of externaldedicated QA providers, and the birth of outsourced testing and hybrid models.PRO 4Hidden costs of In-House teams: Supporting an in-house QAdepartment not only encompasses basic QA costs investment,but also hidden expenses that companies must budget for,including:• Salary costs (recruiting, regular salary, employer contribution (FUTA, Medicare, SUI), employer tax, vacation and sick leave, 401(k) plan, workers’ compensation insurance)• Employee benefits (medical and dental care, life insurance)• IT and infrastructure costs (computer, fax, software/licenses, internet, network infrastructure, helpdesk, repairs, phone lines, overseas calls, email, web hosting, voicemail.)• Training costs (internal, external training, tuition reimbursement)• Office costs (office space, common areas, parking/transit, furniture, personal property tax, personal property insurance, liability insurance, refreshments, office supplies, paper, toner)PRO 4The right talent is hard to find: Good QA engineers, automationspecialists, functional testers, etc, are not easy to find, sodedicated 3rd parties are able to offer a ready-made expert team,without the need to go through extensive (and expensive) hiringperiods – particularly if there is a need for specialist expertise.PRO 4Gives developers more time to focus on other important aspectsof game development: developers and publishers are both oftenstretched with multiple projects simultaneously, so allowing a 3rdparty to take some responsibility means that the teams can focuson bigger business priorities.CON 7Knowledge Loss through Offshoring: Outsourcing QA can be avery dangerous trap if not managed correctly. Companies thatshed internal QA resources and offshore their QA operationtypically pay the price in knowledge loss and ultimately indegradation of the product quality. QA engineers in well-runteams often have better product knowledge than any other part ofthe organisation.CON 7Cost Advantage Myth: Companies in US often consideroutsourcing QA for all wrong reasons, primarily driven by costadvantage. In India, China and especially Eastern Europe, thehourly rates for contractors continue to be much lower than thosein the US. However, the benefits are compromised by lowerproductivity: on average productivity of resources in India wouldbe 50% of what you would get with local mid-level resources,for senior developers it would be at best 75%, and for juniorsup to 80%. To achieve that level of productivity you would needto put above average efforts in shaping your team and haveconsiderable amount of luck. To ensure a high quality of service,it is better to use a nearshore provider, which often comes withmuch higher costs.VERDICTUltimately, clients frequently find outsourced QA rates prove morethan competitive, and the reduction of in-house staff managementfrees up time to pursue other business goals, but obviously putsthe QA team more at arms reach than an in-house team. Thereis no easy answer one way or the other; however by adopting ahybrid model, wherein outsourced QA supports an existing in-house team, it is possible to get the best of both worlds, althoughcareful management is necessary to coordinate between the twoteams simultaneously.
  • Part IV: The Move to Managing Simultaneous In-House and OutsourcedQA TeamsSelecting the Right Vendor The Steps to Successful OutsourcingAs the game development industry has evolved, and QA has become more important, simultaneous in house andoutsourced QA teams have become regarded as a good halfway house, allowing companies the right balance of controland manpower.Some of the key considerations when adopting this modern day QA model include: management of teams in different timezones, agile methodologies, test plans and test automation. Selecting the right vendor is also a vital step at the start of theprocess.One of the first things to evaluate in a possiblegame vendor is what QA processes they havein place. The best vendors have their own QAteam in place that works in conjunction with thedevelopers to implement a software test plan.It is essential to check the following key pointsbeforehand: What is the system for trackingissues/bugs or system changes? What arethe processes for fixing bugs? What are thestandards for monitoring and quality compliance?Do the developers use industry-standard unittests and regression tests to test each build? Isthe game being tested for additional factors suchas level design, playability and experience?It is important to test cases, based upon thecarefully documented system requirementsdeveloped for any game. This can make thedifference between a “great” beta version, or onethat is bug-filled. Once game development iscomplete, the QA team will step in, checking thatall functionality, scalability and security issueshave been addressed based upon the initial testplan developed from the requirements gathered.The test plan covers all regression, load andvolume testing, and conduct user acceptancetesting, with specific performance criteria foreach. Of course, ideally, QA would be embeddedearlier in the game development cycle and notjust at the end!Another way to improve the quality of thecompleted deliverable is to conduct inspectionsof the work products. Inspections are detailedtechnical peer reviews of games underdevelopment. It has been estimated that eachhour spent on QA activities, such as designreviews, can save a firm from 3-10 hours indownstream costs. Ideally, offshore vendorshould conduct inspections at each stage of thegame development or the maintenance process.By conducting regular peer review inspections,the vendor will be able to detect and correctdefects rapidly in upstream work products.This allows them to better control the costs andprevent schedule delays during the project.Successfully executed outsourcing can yield very positiveresults. In order to ensure a high-quality return, the followingsteps are advisable:• A rigorous vendor selection process with the focus on ‘the match’ between your organization and vendors’ and searching for the match on multiple dimensions.• Resource augmentation and joined teams rather than complete outsourcing.• Abundant communication in all forms with face-to-face meetings and on-site / offshore swaps.• Ongoing preventive maintenance and control in all aspects of the engagement.• Adjusting Systems Development Life Cycle to cater for idiosyncrasies introduced by offshore.• Disposal outsourcing models can be developed to help structure processes.• Working with near-shore partners is easier in many aspects especially when running agile projects, and often produces higher quality results• Resource augmentation and joined teams rather than complete outsourcing as a compromise.There are two other considerations when deciding how toimplement the software QA function. One approach keeps QAindependent of the development team, where the independentQA team (externally or internally) reports directly to the Head ofQA. The embedded approach is where QA runs as a part of theexisting development team, in this case, the QA team reports tothe game development project manager, who has the final sayon all questions.• Independent QA: History suggests that independent QA teams deliver better results, and that following this approach the customer always gets an accurate picture of the quality of the game under development. Furthermore, independent QA is clearly focused on revealing all issues as early as possible.• Embedded QA: Here, with the dependent QA set-up, the development project manager may be overly optimistic about the current status of the project. This approach frequently results in hidden problems, which may be revealed only at the final stages of product development or even after product release. On the other hand, as this method becomes more popular, developers are finding that by integrating this approach with agile techniques the final game quality could be vastly improved.
  • SummaryProblems are commonly seen in games where the firstreal chance that testers have to test the game once allthe features are implemented – the ‘Alpha’ stage. Thefailure to take advantage of the help at hand from QAexperts during the earlier stages of the process can costtime and money. There are different types of EmbeddedQA – the key ones being Agile Methodologies, mostnotably Scrum.Agile: a set of practices which can help the developersto know where they are up to all the way through theproject, and where the testers actually help to drivethe project. Agile projects push testers into leadershiproles - they might, for instance, define the acceptancetests and help the customer define their requirements,they might even explain the risks to the customer.Projects that don’t require tests before the code existscan easily push the testers into a background role.On an ideal Agile project, the project team discussesthe ‘user stories’ (options and outcomes) to roughlyestimate the time it will take to implement each one.The customer ranks the user stories to determinewhich user stories the project team will approach in thisiteration. Then, before a single line of code is written,the testers write acceptance tests with the customer. The developers can discuss how they might implementthe user story, they may even prototype to discover anyof the system-level problems before implementation,but they can’t write any production code untilthe acceptance tests are complete. Testers writesystem level acceptance tests to drive the productdevelopment. The developers implement only enoughcode to pass the tests.Scrum: A specific form of agile practices. The term istaken from rugby where the team moves the ball asa group and doesn’t divide up the team into roles asstrictly as other sports teams.In game testing, using this process involves embeddingindividuals from QA into the Scrum team. They sitin the same bullpen as the rest of the team and getmore involved than previously. Some of them can doa bit of coding, others layout and others take on parttime associate producer roles, but their core job is toexercise functionality as the game comes online.QA has come a long way since the early days of flawed releasesbeing rushed to market by publishers very much aware of the growingappetite for new games. These days far more care is taken to forward-plan release dates to avoid the problems experienced by the likes ofAtari’s E.T.More and more emphasis has been placed on QA in recent years, asgame quality and playability becomes more critical in the reception ofthe game – and thus its sales figures. There was a time when videogames were so novel (and niche) that gamers were more forgivingof problems – many gamers look back at iconic N64 games like 007:Goldeneye and Perfect Dark with much fondness (and a certaindegree of reverence), but they certainly weren’t perfect.These days, games released with bugs are more subject to scrutinyby critics and media channels alike – just look at the recent exampleof EA/Maxis Sim City V, in which QA has frequently been blamed byill-informed consumers and the occasional journalist for the myriadof server and connection problems experienced at launch. Here’s asituation where QA had no involvement in these issues, and yet comeunder major international fire – just imagine the scenario where thefinger could legitimately be pointed at QA.For all this, it is interesting to note that the number of games beingreleased today with an immediate “day zero” patch has also increased– is this a sign that many games are still being rushed and beingreleased unfinished? Or that they are tested more thoroughly now andas such require more frequent patches and updates? Only time willtell what this trend reveals.The key point here though is that QA has become increasingly moreimportant in the industry and is now recognised as a critical functionin its own right, as opposed to being a mere stepping stone into more‘desirable’ functions within game development.The industry stepping stone perception can never be truly lost underthe current model, as QA testing requires the least specialist skills andknowledge, and therefore has the lowest barrier to entry. But, as therole of the QA tester continues to evolve far beyond simply looking forbugs and filing report more and more skills will be developed and thatbarrier will begin to rise.It’s not difficult to picture a world in the not-too-distant future in whichstudents and graduates need extra skills in order to begin a career inQA. At this point, testers may begin to garner more industry respect,and the ‘stepping stone’ stigma will be slowly be eroded.Ultimately though, the key take-home is that the industry has nowreached a turning point in QA. When major developers and publisherslike EA, Sony, Microsoft, Sega and Ubisoft etc. are publically supportingand speaking out at forums like the Game QA & Localisation event inLondon this June it demonstrates just how seriously QA is now beingtaken; not only to improve the quality of games being released, butto help catalyse a shift in the way QA is perceived inside the industryand out.Great strides are being taken in the right direction – let’s hope QA cankeep up the pace.Part V: Embedded QA andAgile Methodologies
  • Pestana Chelsea Bridge Hotel & Spa, London, UK • June 25th-26th 2013The World’s First Two-Day QA & Loc-Dedicated Interactive Forumwww.gamingqa.comPLAY YOUR PART IN CHANGING THE WAY QUALITY IS PERCEIVED AROUND THE WORLD> Develop best practices for test plans, QA budgeting and strategy> Work more effectively with the development team to improve game quality and playability earlier in the development process - the need for embedded QA> Automation - how to automate testing, assimilate it into your existing strategy and when to use it> Selecting the right technology tools and outsourced solutions for your QA> Working on different platforms - the minefield of platform-specific requirements> Establishing a clear QA career path and team structure to encourage internal asset retention and developmentBenefits of Attending:This event is a good opportunity for games QA to put a stakein the ground and show how much it has matured. We’ve trulybecome a discipline in our own right.Chris Rowley, Senior Director of Certification, EAThe inaugural game QA and Test forum promises to be a key event in2013. The gathering of industry leaders provides a unique opportunityto openly share and exchange ideas, whilst discussing the challengesand opportunities we face in our organisations. I’m looking forward toseeing a few familiar faces as well as making new contacts.Dave Parkinson, Director of Global 1st Party QA, SCEE