Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. (Êîììåíòàðèé "Âûäåëåíèå" Gunchenko15.02.2013 17:28:44ïóñòî)9-610--032JANUARY 1 55, 2010KARI MM R. LAKHANIDAVID A. GARVINERIC L ONSTEINToppCoderr (A): DDevelopping Sooftwaree througghCroowdsouurcingIn December 22009, Jack Huughes, CEO and founderr of TopCodeer Inc., enterredhis comppany’sheadqquarters in Glastonbury,, Connecticuut, eager to review a pparticularlyccomplex sofftwaredevelopment projeect for an eneergy firm’s dyynamic poweer pricing systtem.Eight yeears after founndingTopCoder, Hughees still enjoyyed detailed project reviews. He waas particularllyproud thaat hiscompany could prroduce high-qquality softwaare solutions for which hiis ownemplooyees did nott haveto wrrite a single liine of code. IInstead, the ffirm nurturedd a globalcommmunity of mmore than 2225,000progrrammers whoo competed too design and create softwaare modules for TopCoderclients, a prrocessthat thhe popular prress called croowdsourcing.1Huughes smiledd at the projecct’s success. TThe resulting software codde wasbug-freee and opera tionalon itss first day, a rrarity in the ssoftware induustry. Especiially impressiveto Hughees was that inn fourmonthhs, 65 particippants from 1 1 countries inn 6 continents had competed in 57conntests to creatte thiscriticaal pricing syystem for thee client (see Exhibit 1). AAs of 2009, TopCoderrooutinely prodducedsoftwware solutionss for over 45 clients, incluuding AOL, Best Buy, Elii Lilly,ESPNN, GEICO, annd theRoyall Bank of Scottland.In the past eighht years, Hughes had refinned TopCoderr’s business mmodel toaccoommodate onngoingchangges in the sooftware induustry, while also pursuinng its uniquuecompetitioon-based sofftwaredevelopment apprroach. He haad transitioneed his busine ss from a moodel thathelpped other sofftwarefirms identify ’topp coders’ to aa company thhat developedd custom softtwarethroughh a combinatiion oftradittional IT conssulting servicces and comppetitions, pitting developeersworld-widde to solve cllientsprobleems.Thhe shift to a greater emmphasis on competitionns, encompasssing all asppectsof softwaredevelopment, howwever, meant that project volume was a "growing" issue--forToopCoder. Huugheshad too think throuugh how a commpetition-bassed business model, whichhincreasinglyy stressed conntestsas ann organizing,, as well as money-makking approacch, could hanndle increaseesin numbeers of
  2. 2. competitions, cliennts, and partiicipants. Huughes consideered his own goal:attaininng $200 milliion inrevennue from a higgh of just overr $18 million in 2008. He ffundamentalllybelieved thhat contest demmandwouldd spur the suupply of ToppCoder particcipants who wwould in turrn createhighh quality sofftware1 Jeff HHowe, ’The Rise oof Crowdsourcinng,’ Wired Magazzine, Issue 14.06, June2006.Professoors Karim R. Lakhaani and David A. GGarvin and Researrch Associate EricLLonstein prepared this case. HBS casses are developed ssolely asthe basiss for class discussioon. Cases are not inntended to serve assendorsements, soources of primary ddata, or illustrationns of effective orineeffectivemanagement.Copyrigght © 2010 Presidennt and Fellows of HHarvard College. TTo order copiesor request permissionn to reproduce matterials, call 1-800-5545-7685,write Haarvard Business Scchool Publishing, Booston, MA 02163, oor go towww.hbspp.harvard.edu/edu cators. This publicaation may not be digitized,photocoopied, or otherwise reproduced, posteed, or transmitted, wwithout thepermisssion of Harvard Buusiness School.
  3. 3. TopCoder (A): Developing Software through Crowdsourcingsolutions. But, was $200 million in revenue a reasonable goal? Did hisassumptions make sense? Ifso, what would it take to increase revenues by over an order of magnitude?Background and Current OperationsBefore he founded TopCoder in 2001, Hughes had built a custom softwaredevelopment2company, Business Data Services, in 1985; renamed Tallan in 1991. Tallanemployed some 600 peoplebefore being sold to CMGI in 2000.3 As he was completing the transaction, Hughesreflected on whathe had learned from his experiences at Tallan--these experiences would inspirethe core tenets of theTopCoder business model. Although Hughes enjoyed his time at Tallan, there wereareas in whichthe company struggled. For example, recruitment was an expensive and frustratingprocess becausefinding qualified programmers was time consuming and talent was difficult toassess. Due toconstantly evolving technologies, programmers’ skill sets often became obsoleteafter only a fewyears of productive service, leading to high levels of employee turnover.Furthermore, despiteTallan’s goal of maximizing billable hours, Hughes believed there wereopportunities to save clientstime and money by, for example, reusing computer programs’ basic componentsinstead of buildingeach application from scratch.Drawing upon these and other insights, Hughes set about creating a new kind oforganization thatwould build a "community" of programmers to help address the issues he hadidentified. Theseprogrammers would compete--as well as affiliate’by building and using componentsthat hadalready been tested and found workable. The idea of reusing software componentsfor new projectswould become the core of the solutions the new company, called TopCoder,provided. Hughesenvisioned the company as a ’two-sided platform’ for software development. Oneside of theplatform would be clients, firms that needed software developed, who would workwith his staff tospecify programming challenges. The other side would be community members whowould competein contests to create solutions to the challenges for money and skill ratings.TopCoder would be inthe middle as the platform host, designing and enforcing the rules of engagementbetween clientsand the community members. Pete Bourdon, TopCoder’s CFO, explained the companyneeded toexcel at five core tasks: breaking down large client software projects intocomponents, taking in andprocessing client project specifications, determining appropriate contestprizes, having a consistentand unbiased way of picking contest winners, and fixing bugs at the back end ofdevelopment.Setting out to amass an initial collection of highly skilled programmers, from2001 to 2003
  4. 4. TopCoder asked established software development companies to sponsor world-wideweb-basedprogramming competitions. The sponsorships increased the popularity andlegitimacy of TopCoder’scompetition platform and provided the company with access to talentedprogrammers from aroundthe world. In return, the sponsors, including Sun Microsystems and Google, usedthe contests toadvertise and recruit new talent. Tanya Horgan, TopCoder’s V.P. of Finance,explained that duringthe sponsorship phase, TopCoder offered unusually large prizes’as much as $5,000to $10,000 permatch for tournament winners’to attract competitors and expand the community. Inaddition, everycontestant that participated was given an objective numerical rating for theirperformance against theglobal talent pool, providing a clear signal to TopCoder and others about thetalent in thecommunity.2 Custom software development done by specialist firms in the global ITconsulting and services sector (for example,Accenture and IBM) was an over $500 billion segment in 2008. (Source: ’Global ITConsulting & Other Services: IndustryProfile,’ Data Monitor, March 2009.)3 Clint Boulton, ’CMGI Acquires Tallan for $920 million,’ February 14, 2000,http://www.internetnews.com/ecnews/article.php/303771 8/1/2009.
  5. 5. TopCoder (A): Developing Software through CrowdsourcingBy the end of 2004, the TopCoder community was 50,000 members strong. In itsearly efforts touse the community to generate revenue, TopCoder acted as a placement firm,matching top ratedcommunity members with firms seeking new talent. Hughes, however, wasdissatisfied: ’I cringed atthe idea of TopCoder becoming a placement firm. That was not my end-vision forthe company.’As of 2005, TopCoder began to use its community to develop software componentsandapplications. Hughes first tested this model by having highly rated communitymembers compete toredesign and rewrite the firm’s own software platform. The resulting code washigher quality andmuch less expensive than TopCoder’s own internally developed solution. Hughesnow had positiveproof that complex software systems could be built through competitions.Initially, TopCoder adopted a model to create solutions for clients bycontracting with communitymembers, running competitions, and providing consulting services. The companybroke down thesoftware development process into seven distinct but interrelated tasks; 1)Conceptualization, 2)Specification, 3) Architecture, 4) Component Production, 5) ApplicationAssembly, 6) Certification,and 7) Deployment. Most revenue came from consulting services: TopCoder billedclients for the timethe company’s platform managers spent conceptualizing and specifying clientproblems, setting upcomponent design and development competitions, assembling components, anddelivering finishedsolutions.Shortly after TopCoder started developing software for clients, the companyidentified reusablecomponents from the software it was creating and collected the components in acatalog. Thesesoftware components became an important part of TopCoder’s value proposition toits clients. Manyof the custom applications could be produced by combining existing catalogcomponents with newcomponents built through competition. TopCoder’s hybrid consulting model led to large increases in revenues. However,Hughes wasstill dissatisfied: ’I viewed the hours-based services approach to be a broken,inefficient model.’ In2007 and 2008, TopCoder produced nearly $20 million in revenue but platformmanager costsremained high (see Exhibit 2 for information on revenue and platform managercosts). Attempting toalleviate costs, in 2007 TopCoder introduced competition tracks for componentarchitecture andassembly. With these new competition tracks in place, the work traditionallydone by platformmanagers would now be done by the community. In 2008, the company also addedcompetitions insoftware development tasks such as conceptualization and specification, as wellas deployment and
  6. 6. bug fixing.By early 2009, TopCoder had moved increasingly away from the hybrid consultingmodel. It nowfocused on completing all tasks in software development through competitions.Instead of paying fortime and materials for TopCoder platform managers, clients paid a monthlyplatform fee, based onthe complexity of their software requirements and the estimated number ofcompetitions they wouldrun through the TopCoder platform each month. The platform fee also providedclients withunlimited access to the over 1400 components in TopCoder’s catalog. Roughly 60%of most clients’projects could be accomplished through reusing components from the catalog. Themove from thehybrid consultancy model to a competition model was coupled with the reductionof many platformmanager positions, leaving the company with 16 project managers servicing 35clients by the end of2009.As of late 2009, TopCoder ran two different types of competitions on itsplatform: algorithm andclient software development. Algorithm competitions served as the primary meansfor attracting newmembers and retaining existing members. These competitions required members todevelop creativesoftware solutions to relatively difficult programming challenges. All memberswere assessed against
  7. 7. TopCoder (A): Developing Software through Crowdsourcingeach other through an automated computer scoring system; they then received aTopCoder rating fortheir performance. Some algorithm competitions also had cash prizes for winners.The second type of competition was targeted at developing software applicationsfor specificclient needs. A TopCoder platform manager initially worked with the client staffto develop a ’GamePlan’ (see Exhibit 3 for a representative Game Plan) or a project road map forbuilding the software.The first step typically involved a contest where the general client problem waspresented to theTopCoder community in a conceptualization contest. Here contestants publiclycross-examined theclient staff as to their actual needs and then submitted a business requirementsdocument and highleveluse cases. The client picked the submission(s) that best represented their needsas the basis forfurther development. Then a series of specification contests were held to createthe applicationsrequirements documents, application wireframes (i.e. the logical flow ofapplication) and storyboards(detailed cases of the user experience). Next, the output of the specificationcontests were fed intoseveral architecture contests where the overall system and component leveldesigns would becreated. At this point the TopCoder platform manager would work with the clientto either selectcomponents from the catalog or commission the creation of new components throughdesign anddevelopment competitions. After the component production phase, all the relevantcomponents wereput together through an assembly competition with the objective of creating aworking system.Assembly was then followed by certification and testing contests and theneventually deployment.Throughout the execution of the Game Plan, TopCoder retained flexibility indevelopment byrunning ’Bug Races’ to accommodate changing client specification or unforeseenerrors. TopCoderalso had been issued eight U.S. patents on various aspects of running onlineprogramming contests ina distributed community setting and had other patents pending domestically andinternationally.To determine winners and assess quality in client software development, TopCoderused acommunity-based peer review system. In particular, expert and experiencedTopCoder communitymembers were paid to grade and comment on all contest submissions using detailedscorecards,ultimately picking the contest winners. The winning competitors for each contestwere then awardedmonetary prizes and all participants were given updated ratings for theirperformance. TopCoderalso ran Studio contests if an application required logos or graphics; in thosecases, clients chose thewinners.Evolution of the TopCoder Community
  8. 8. Growth and CompositionFrom 2001 to 2009, TopCoder added an average of 25,000 new computer programmersto itscommunity each year. After filling out a short online registration form, anybodyin the world couldparticipate in a software development competition; by spring 2009, the TopCodercommunity hadover 200,000 members (see Exhibit 4 for community growth). Although the size ofthe overallcommunity was large, the number of people within that community who activelyparticipated incontests and posted in forums was much smaller. The majority of communitymembers at TopCoderregistered as members of the community but never competed in any contests. Infact, by 2009, only35,000 unique individuals had competed in contests. To Mike Lydon, TopCoder’sChief TechnologyOfficer, the remaining 82.5% of the community was the ’latent pool’: people whowere interestedenough in the TopCoder platform to register and had the potential to provideTopCoder withincreased development under the right conditions.A second group within the TopCoder community comprised those members who at onetimeparticipated in TopCoder contests but then stopped participating. Lydon notedthat, after TopCoderdecreased prize values in 2008, many competitors from the United States andCanada left the
  9. 9. TopCoder (A): Developing Software through CrowdsourcingTopCoder community. Yet another group included people who participated inTopCoder contestsbut did not win. TopCoder saw those competitors as the ’long tail’’people whoprimarilycompeted for the sake of learning. One of TopCoder’s main goals was to cultivatethe long tail so thatlesser skilled competitors could improve over time and increase their levels ofcontribution. Lastly,TopCoder’s most valuable group of competitors included the everyday winners. Thetalent ofTopCoder’s elite programmers was equal to the best in the world, but suchmembers only accountedfor 0.5% of the total TopCoder population.The core of TopCoder’s community was made up of single, highly competitive malesin their 20s.According to Michael Paweska, a six-year veteran at TopCoder: ’To be successfulat TopCoder, youmust ask yourself, ’Are you a competitor?’ You need to be able to thrive oncompetition; you can’t bescared of it. You also need the flexibility to work long hours. TopCoder is abachelor’s sport: themoment you become involved with someone else, it becomes a point of friction.’TopCoder attractedcompetitors from developed nations such as the United States, Canada, SouthKorea, and Japan, aswell as from emerging economies such as China, Russia, Poland, India, andUkraine. Wu Yanbo, aChinese TopCoder community member studying abroad in Australia, explained thatmostcompetitors in the lower-paid contests were from developing countries. Accordingto Wu, the prizeswere not large enough for many individuals from developed countries to compete,since they couldspend their time better elsewhere.Justin Gasper, a member since 2001, began experimenting with the TopCoderplatform whileworking for a traditional software engineering company. After winningsignificant money withTopCoder, Gasper decided to quit his job in 2005 and devote forty to fifty hoursa week to TopCoder.Gasper explained: ’TopCoder is my full time job; I don’t have a day job.’ Gasperwas one ofTopCoder’s regular winners, a member of the ’global elite’ of programmers. Inarchitecturecompetitions, Gasper came in at least 2nd place 95% of the time and had a winpercentage of 69.23%.Competitors at TopCoder could choose which contests and what type of contests tojoin (see Exhibit5 for participation and prize data by contest type).Profiles and RatingsEach programmer in the TopCoder community maintained a public profile thatdisplayed his orher user name, contest history, and basic personal information. Another part ofthe member profiledisplayed a competitor’s numeric rating for each type of contest. The ratingsystem was modeled onthe one used to rank grandmaster chess players engaged in worldwide competition.
  10. 10. A ’red colorrating,’ or a rating of over 2,200, represented elite status within thecommunity and a high skill level.Yellow, blue, and green color ratings represented descending skill levels. Eachcompetitor’s countryrank, total community rank, success rates for contests, and reliability’orpercentage of times thecontestant joined a competition and submitted a passing solution’were featuredin their profiles.TopCoder members could also choose whether or not they wanted to display theirtotal earnings ontheir profiles (see Exhibit 6 for an example member profile).Motivating MembersBetween 2001 and 2009, TopCoder paid out over $20 million dollars in prizes andpeer reviewmoney to its community of developers. However, prize money was not evenlydistributedthroughout the TopCoder community. The top 5% of prize earners receivedapproximately 80% ofthe total prize pool, while the majority of TopCoder community members earnedlittle or no moneyfrom competitions. Some competitors were extremely successful. For example, from2006 to 2008,Paweska earned $200,000 to $300,000 per year, while Gasper averaged over$100,000 annually. Wucommented: ’I have to say money is the most attractive thing. The prize is verygood compared to the
  11. 11. TopCoder (A): Developing Software through Crowdsourcingincome of my friends who are working in some local companies in China. Eventhough the economyis not very good and TopCoder reduced its prizes, I can still earn around $1000per month in myspare time.’ TopCoder typically awarded prizes to the top two submissions ineach contest, with thelion’s share of the prize money going to the top performer.Besides prizes awarded on a contest-by-contest basis, another main source ofincome for memberswas the Digital Run. In the Digital Run system, the top five ranked competitorsfor each contest wereawarded points based on contest rank and performance. At the end of each month,TopCoder talliedcompetitors’ total points and awarded the top point earners thousands of dollarsin bonus prizes.Paweska explained that success in the Digital Run was not all about who was thebest programmerbut more about who could handle the most all-nighters. Other competitors, suchas Gasper, alsomade money through contracted projects that TopCoder assigned.In addition to their cash earnings, many community members reported that theirTopCoder ratingwas very important because it provided an objective assessment of ability. Wucommented that it wasnot easy to maintain a very high rating as it required familiarity with manykinds of technologies,quick thinking, the ability to learn independently, a strong work ethic, andattention to detail.According to Wu, a TopCoder rating could be important for a programmer’s futurecareer. Forexample, a high TopCoder rating helped one of Wu’s friends earn a job at Google.Gasper noted thatTopCoder ratings were also symbols of status and prestige for many programmers:’If you have redratings, people look up to you.’ Indeed, many prestigious software firms askedpotential recruits toget a TopCoder rating before applying for a job. To others, however, the ratingsystem was lessimportant. Gasper, for example, explained that winning and making money meantmore to him thanratings.Although there were differences of opinion regarding the importance of ratings,almost allcommunity members agreed that competing at TopCoder provided numerousopportunities to learnand improve. In fact, for many programmers, a TopCoder career often began withfailure, but postcontestevaluation and peer review of each submission helped them grow and improve.Gaspernoted: ’I totally failed in my first competition. But the reviewers were reallygood at pointing me inthe right direction, saying ’here’s where you went wrong’’. You can’t fake itbecause you’re gettingpeer reviewed by people who are better at programming than you are. Thereviewers don’t care ifthey hurt your feelings; they are direct. If they see a bad design, they rip itapart.’ Paweska agreedthat getting feedback from reviewers was crucial and added that community
  12. 12. members could alsolearn from acting as a reviewer for contests. For scientists and developers, Wubelieved thatalgorithm contests were particularly helpful at sharpening research skills andimproving criticalthinking abilities. In all cases, continual learning opportunities from peerswere an important reasonfor participation.Gasper described the appeal of working at home on a web-based platform insteadof in atraditional ’cubicle farm’ setting: ’I like the flexibility that TopCoder givesme. I don’t need to drivea half an hour to work each day and can do the same work at home. If I want totake off a day to playgolf, I just do it. I also don’t have to work from 2 to 6 pm, my mostunproductive hours.’ Sharingsimilar sentiments, Paweska liked that while working at TopCoder he did not havea supervisorlooking over his shoulder.Setting one’s hours was convenient but also challenging, as competitors had toactively managetheir individual levels of participation. Gasper constantly balanced effort andreward to maximizeincome while still living a sustainable lifestyle. ’If there’s something that isway too much work forthe payment, I won’t do it’ that’s a super power I’ve developed. I know when thespec is clean andworthwhile to solve. It’s a skill that comes from doing tons and tons ofcontests.’
  13. 13. TopCoder (A): Developing Software through CrowdsourcingA ’Community’ of CompetitorsWu noted that, although the firm was competitive in sprit, competition atTopCoder was neverdisrespectful or nasty and that people liked to help each other, even when theycompeted in the samearena. TopCoder forums were the main source for collaboration. In the forums,less experiencedcommunity members asked for assistance on certain problems and received instantfeedback frommore experienced competitors.At TopCoder, conversations and relationships extended beyond the scope ofsoftwaredevelopment. Hughes reflected on a particularly remarkable exhibit of communalstrength andcaring for fellow community members outside of software development. ’When oneof thecommunity members died,’ he said, ’the outpouring of support was such that anumber of thecommunity members took all of their winnings for a few weeks and gave it to thedeceased’s wife. Itended up being tens of thousands of dollars.’Once a year, TopCoder paid for all of the best talent from the community totravel to Las Vegas,Nevada, to compete in the TopCoder Open (TCO). In additional to serving as aproving ground forthe best programmers in the world, the TCO provided community members with theopportunity tonetwork professionally and socially.The TopCoder community had a distinctive culture, with identifiablepersonalities. Wu explained:’I believe this community, like all others, has its own culture. Clearly, themembers built it upcontinuously. When I joined the community, there were already some leadingmembers who wereactive in competitions and forums, brought out good suggestions, and started upinteresting andimportant discussions.’ In some cases, the fame of community leaders extendedwell beyondTopCoder. For example, Tomasz Czajka, from Poland, achieved ’rock star’ statusand had his pictureplastered on billboards throughout Warsaw after he won the TopCoder Open in2006.The Client’s PerspectiveClients came to TopCoder to have high quality software developed in a costeffective and timeefficient manner. TopCoder positioned itself to serve both large firms andmedium to small-sizedbusiness that wanted to see systems developed. Keith Moore, a TopCoder clientand former SeniorVice President at LendingTree.com, believed that, regardless of size, anycompany could takeadvantage of TopCoder, whether it was a five man operation or large outsourcingvendor. For manyCIOs, the process of software development and talent recruitment was a majorheadache, and missed
  14. 14. deadlines and large cost overruns were common worries. According to StephenLaster, the CIO atHarvard Business School and a TopCoder client, ’A typical IT shop will turn over48% of itsemployees every three years. This process is very costly. The same problemexists with ouroutsourcing consultants. When selecting consultant teams, we tried out 60programmers beforefinding our team of 20. With TopCoder, I pay for performance and the CIO seesNirvana.’As of 2009, TopCoder had developed a strong relationship with existing clientsfor delivering highquality software solutions and superior customer service. After completing theirfirst project withTopCoder, 82% of clients signed up for a second round of contests. They citedseveral advantages.BenefitsBetter Ideas Before sinking thousands of dollars into a project, a client couldrun aconceptualization contest through which TopCoder members helped identify badideas and generatebetter approaches early in the development cycle. When the client introduced abusiness problem to
  15. 15. TopCoder (A): Developing Software through Crowdsourcingthe community, members asked hundreds of questions. Nic Perez, a formerTechnical Director atAOL, explained that the community’s questions ’gave us insights into problems Ididn’t even reallyknow I had’ and ’saved us money by doing all of those questions upfront.’ Usingonline forums,clients answered questions for all competitors only once, avoiding repeatedefforts. In some cases,clients scrapped product ideas entirely after the community raised concernsabout the product’slikely success or usability in the marketplace.TopCoder’s contest-based development system consistently produced highlycreative ideas andsolutions. According to Darren Smith, a Solution Architect for the e-commercedivision at FergusonEnterprises, North America’s largest plumbing supplies wholesaler anddistributor: ’The communitycomes back with many options. It really has surprised us. You never know whatyou are you going toget. The creative side allows us to go to the marketing management team and say,’We could do x, y,and z that we may not have previously considered.’ They’re adding value to ourbusiness becausethey bring us solutions that quite frankly we may not have considered or werenot resourced todeliver.’(Êîììåíòàðèé "Âûäåëåíèå" Gunchenko15.02.2013 17:33:01ïóñòî)Superior Quality, Cost, Speed and Flexibility Clients praised TopCoder’srigorousevaluation and documentation process for being well above industry standards.Reflecting on hisexperience working on the Google Talk interface to AOL Instant Messenger, Perezstated thatTopCoder and its community had a strong desire to deliver bug-free code and thateven the mostcomplex systems always had fewer than 100 identified bugs. According to Perez,the same sizedprojects, developed internally, at AOL would have had 5-8 times that number ofbugs.Another TopCoder client, a web-based startup business, noted that it would havehad to pay$350,000 to a large IT consulting firm, $200,000 to a small IT consulting firm,or $80,000 to individualcontractors to build the company’s website. Using TopCoder, the client onlyspent $35,000. This sameclient proclaimed: ’At $35,000, it’s priceless. There is no other game outthere.’ A different clientnoted that based on its experience working with almost every type of softwaredevelopmentcompany, TopCoder charged approximately half of the fee of a large, tier one ITconsulting firm.Using the community for parallel problem solving, TopCoder marketed itself asfaster thanother software development shops. This was true for back end bug races and
  16. 16. system checks, asTopCoder took 72 hours to complete the same bug testing that a traditionaldevelopment firmfinished in 10 business days. However, for other steps in the softwaredevelopment process, reportson speed were mixed as some clients said that TopCoder worked at about the samespeed as a largeIT consulting firm while others lauded TopCoder for speed of completion.Especially appealing to clients was TopCoder’s ability to supply flexiblesoftware developmentcapacity. In particular, a TopCoder client could expand or reduce its businessrequirements anddevelopment capabilities without having to hire or fire programmers. Accordingto one client, a basicin-house computer programmer cost $120,000 a year, after accounting forbenefits, sick time, andvacation. Working with TopCoder, clients did not have to spend as much onemployees’ benefits anddowntime.ConcernsAlthough CIOs were impressed by TopCoder’s technical capabilities and costsaving potential,many often had initial reservations about working with TopCoder’s unusualsoftware developmentmodel.Intellectual Property and Security According to Ira Heffan, TopCoder’s ChiefLegalCounsel, ’For new clients unfamiliar with TopCoder’s model, IP and securityconcerns can be an
  17. 17. TopCoder (A): Developing Software through Crowdsourcinginitial point of resistance. Until they understand the documentation andprocesses we have in placewith the community members, they see IP and security as potential barriers toworking with acommunity.’ For example, some clients were apprehensive that a TopCodercommunity membermight divulge proprietary ideas, business plans, or operations to theircompetitors. In addition, someclients worried that once a component became an integral part of their ITsystems, the communitymember who built the component might attempt to prohibit its use or ask theclient to payconsiderable royalties. Lastly, some clients were concerned that a solutionsubmitted by a communitymember could be stolen, copyrighted, or taken from open source softwareprojects, thus potentiallyopening the door for intellectual property disputes.TopCoder had in place a number of initiatives targeted at addressing theseconcerns and reducingthe risk level for clients, and also took steps to communicate its processes. Toease client’s intellectualproperty and security concerns, TopCoder produced a white paper that detailedconfidentialitypolicies, intellectual property assignment rules, and TopCoder’s modularapproach to softwaredevelopment. In addition, TopCoder allowed clients to keep their company namesanonymousduring competitions and helped clients generate test data sets to avoid theexposure of sensitiveinformation. At the client’s request, before a community member was allowedentry into acompetition, all competitors could be required to sign a standard CompetitionConfidentialityAgreement.The peer review process was another means to ensure code security and quality.Peer reviewerswere selected and vetted by TopCoder employees based on their superiorperformance on priorcompetitions. TopCoder clients also had the option of running testingcompetitions at the back end ofsoftware production, serving as an additional means of checking code securityand quality.TopCoder’s compartmentalized software development process also made it difficultfor a singlecompetitor to insert harmful code into a program, since individual contests onlyaddressed one smallpiece of the overall program.Cultural Change Many clients realized that working with TopCoder would bedifficultculturally for their company. In particular, CIOs believed that internalemployees would viewTopCoder as a threat to their job security. One new client observed: ’TopCoderis a CIO’s dream buta programmer’s worst nightmare. I fully expect that if this goes well and if myprogrammers seegood quality work coming out of TopCoder, fear will quickly explode throughoutthe building.’Although using TopCoder could help a company scale and reduce the programming
  18. 18. staff costs,companies still had to retain the ’big thinking’ people’the employees who couldguide theTopCoder development process. The managers at TopCoder clients also had toadjust to a perceivedloss of control over the software development process. Smith commented: ’We setthe competitions,but they manage the whole process. Our project management group works with theTopCodermanager to ensure delivery according to pre-determined SLAs (Service LevelAgreement).’ Someclients even found a few community members to be pushy and rude during pre-competition questionand answer sessions.Coding Challenges Even if TopCoder did all of a company’s internal developmentwork, thecompany still needed to have an internal staff to integrate the deliverable intothe client’s existingsystems, review the code for security issues, and adjust and fix code as systemschanged over time.For example, Smith’s team at Ferguson spent a significant amount of timeinspecting, testing, andprocessing TopCoder’s work to make sure that there were no security threats orbugs. In somecontests, TopCoder clients also spent time evaluating ideas and approaches frommultiple winningsolutions.Another ongoing issue for clients was finding the right types of problems andproviding theappropriate amount of problem detail for the TopCoder community. As Mooredescribed it, ’You
  19. 19. TopCoder (A): Developing Software through Crowdsourcingwant neither too much nor too little detail. You do not want to quell innovationbut also want asolution that makes sense in your system.’ Clients discovered that contestparticipation decreased ifthey were unclear about what problems they wanted to solve or presented problemsthat were toocomplex or vast in scope; in those cases, the TopCoder community struggled toproduce anacceptable solution. Clients also found that community members worked best whencontests lastedless than two weeks. If projects took too long to complete, contestants wouldlose interest and notmake submissions.Managing TopCoderThe Supply SideA management job at TopCoder was unique. Along with supervising internalTopCoderemployees, managers at the firm had to oversee a community of over 200,000members and direct theprocess of competition-based software development. According to Senior VicePresident GeorgeTsipolitis, the key to success was effective process management: ’When you’remanaging acommunity, you are no longer managing individuals, you’re managing a whole. Wecan’t controlindividuals. We can only control the process of their participation.’ Clientsand employees alikebelieved that the sustainable value of the company was dependent on TopCoder’sability to facilitatecommunity participation and foster community growth. Lydon described the risks:’From thebeginning, we focused on the community. We knew they could be unforgiving. Ifyou did the wrongthing, you got crucified.’Attraction To run many competitions simultaneously and produce solutions formany clientsat the same time, TopCoder needed to have access to a critical mass of talentand coding capacity.TopCoder’s primary means of attracting new members into the community was theappeal andchallenge of the algorithm challenges. In addition, TopCoder occasionallyadvertised its onlinecompetition platform by paying for Google keyword searches using terms such as’design contests.’A third mechanism for attracting talent was ’member development days.’ Organizedby a smallteam of TopCoder employees, member development days were held at Chinese andotherinternational universities. At a member development day, a studentrepresentative would post signsaround the school and explain the TopCoder system. A primary goal of thesemember developmentdays was to encourage participation in the higher revenue producing developmentand designcontests. During one member development day in China, TopCoder registered overone thousandnew community members. Bourdon noted that TopCoder had achieved critical mass
  20. 20. once it crossedthe 200,000 member threshold as there were now many members with deep and narrowskills over arange of software development challenges (See Exhibit 7 for the number ofparticipants by contesttype).Norms As the community grew, TopCoder paid close attention to establishingcommunitynorms. As contest administrator, the company had to maintain the higheststandards of contestintegrity, fairness, transparency, and quality. For example, TopCoder personnelstrictly monitoredcompetitions and tolerated no form of cheating. Community members who peeked atothercompetitors’ solutions, shared ideas during competition, or used unauthorizedcode wereimmediately eliminated from the contest. Often they were kicked out of thecommunity entirely.If any uncertainty or disagreement arose about which competitor won a particularcontest,TopCoder would spend the extra money to re-run the competition. Another part ofcontest integrity,Tsipolitis explained, was TopCoder’s emphasis on maintaining consistency ofrules and procedures:’The second that participants can’t figure out how to win, they’ll stopparticipating. So we can’tchange the rules of a competition mid-stream.’ TopCoder also guaranteed completecontest
  21. 21. TopCoder (A): Developing Software through Crowdsourcingtransparency by storing all contest and competitor statistics, peer reviews, andsolutions in a datawarehouse. The data were publicly available to registered community members,accessed via theTopCoder website.Integrity and fairness also extended to TopCoder’s corporate motivations andcommunitycompensation philosophy. In particular, TopCoder was upfront with the communityover itsintention to make money. When TopCoder made a decision to change corporatedirection orcompetition procedures, Hughes posted the information in the forums andexplained the businessreasons behind his decisions. Hughes also believed that, since the companybenefited from thecommunity’s hard work, adequately compensating community members was essential.Governance Although TopCoder executives were responsible for final decisions,theyfrequently incorporated community members’ views into the process. Lydonexplained: ’We treat thecommunity as the driver for everything we do. If we have enough dissent frommembers, we alwaystake that into account. The problem is that when we don’t know what to do, ourmembers will also besplit.’ Community member Gasper, shared a similar perspective: ’TopCoder tendsto push out ideasinto the forums to get feedback. Seventy-five percent of the time, they listento the community. ButTopCoder also has its own business interests to consider. Sometimes thecommunity and businessinterests don’t line-up.’Similarly, if competitors were unhappy with a peer review scoring outcome,TopCoder allowedthem to appeal the decision. Over 90% of contests featured at least one appeal.If a member appealed,peer reviewers had to provide specific reasons why the appeal was accepted orrejected. Ifdisagreement remained between contestant and reviewer, TopCoder employees ofteninvestigated.Contestants could also appeal directly and privately to TopCoder personnel orpost complaintspublicly on the TopCoder forums.TopCoder managers inevitably made decisions that sometimes disturbed and upsetthe TopCodercommunity. For example, facing a very difficult economic environment in thesummer and fall of2008, TopCoder reduced the contest prize amounts, cut payments to peerreviewers, and reduced thenumber of algorithm competitions. During this period, some TopCoder competitorsleft thecommunity entirely and others dramatically reduced their participation levels.Gasper argued thatthe payment cuts also led to many superficial reviews because the best reviewerswere no longerdoing the work, which then required additional cycles to achieve acceptablequality.
  22. 22. Resource Allocation Another part of the TopCoder managerial role was allocatingcommunity resources and controlling contest participation. Lydon explained, ’Wehave to figure outhow to distribute the number of people who want to participate across the numberof contests thathave to be solved.’ Unlike a typical software development firm, TopCoder couldnot assign specificpeople to a task or project. As participation manager, TopCoder’s goal was tominimize the costs ofevaluation, stimulate effort through competition, and get at least one solutionthat was acceptable tothe client. To achieve the ideal number of submissions and participants,TopCoder adjusted the prizeamount, the duration and timing of the contest, the number of other contestsrunning concurrently,and the problem’s complexity and scope. When deciding on the ideal number ofcompetitors,TopCoder also considered contestants’ reliability. As a last resort, TopCoderemployees reached outdirectly to individual community members if other methods did not lead to thedesired participationlevels.Although TopCoder managers could pull many levers to influence contestparticipation, theybelieved it was important not to act like the community’s boss. Hughes explainedhis communitymanagement philosophy: ’We don’t own this community. We want people to be herewhen they
  23. 23. TopCoder (A): Developing Software through Crowdsourcingwant to be here. You are just going to get much better results when you letpeople do what theyreally want to do.’Retention At the same time, TopCoder executives worked to retain communitymembers andencourage future contest participation. At least one client raised concerns inthis area: ’I think thatcommunities are fickle. Community members could start to ask, why do I needthem? For example,what happens if an imitator comes along and offers 2x the prize amount?’ Toavoid such problems,TopCoder tried to supply community members with consistent work streams andprize money.TopCoder also encouraged community members to engage in the community as much aspossible bydedicating significant resources to facilitating forum discussions and invitingcontestants toparticipate in peer reviews, write problems for contests, and develop TopCoder’sinternal systems.TopCoder community members differed on their level of loyalty to the TopCodercommunity.Paweska, appreciative of all the opportunities TopCoder had provided, reflected,’I have someloyalty. I think it would take a lot for me to leave. Only if there were noprojects would I leave.’Gasper viewed his position in the community in a different light. ’I’m not superloyal to TopCoder oranything,’ he said. ’To defect, the payment and work would have to outweigh thepayment andflexibility I have at TopCoder.’The Demand SidePlatform Managers The other side of management at TopCoder was guiding clientsthroughthe contest-based software development process. This was the responsibility ofthe company’splatform managers, whose job was to induce the appropriate amount of communityparticipation,make suggestions for contest prize amounts, gather feedback between contests,and provide projectstatus updates to clients. Before starting the next step in the Game Plan,platform managers alsoadjusted contest requirements based on the work already completed. Once theproduct was deliveredto the client, TopCoder platform managers were required to act in a support andservice role. If therewas a technical problem with a solution, the platform manager often contactedthe communitymembers who developed the component and worked with the community members to fixthe issue.Most enterprise-level clients believed the platform manager was pivotal to aproject’s success. At alarge client like LendingTree, the platform manager was onsite three to fourdays a week, conductingdaily meetings with the internal teams. A large part of the platform manager’srole was managingclient expectations and serving as a sounding board for client concerns. At the
  24. 24. backend of projects,although the community often assembled a project’s components throughcompetition, the platformmanager was also an expert at combining the small software pieces. The componentintegration rolesaved the client hours of work trying to figure out how all the pieces fittogether. At Ferguson, Smithconsidered the TopCoder personnel working onsite to be an integral part of histeam.TopCoder Direct However, each platform manager added to TopCoder’s overheadcosts andnarrowed profit margins. As of 2009, a typical platform manager at TopCoder cost$100,000 a yearincluding benefits. To Tsipolitis, the platform manager’s time was not alwayswell spent. ’Our projectmanagers spend a lot of time babysitting,’ he said.To avoid a potentially large increase in expenses as TopCoder added clients andprojects, Hughescame up with the concept of ’TopCoder Direct,’ in which the client used thecompany’s platformwith little to no intervention from its employees. Under this self-servicemodel, platform managerswould educate clients on how to use the TopCoder platform to manage the contest-based softwaredevelopment process themselves. Hughes envisioned shifting the platformmanager’s responsibilitiesto an experienced community member or an external consultant familiar withTopCoder’s platform’someone who would serve as a ’co-pilot’ to assist the client staff. With co-pilots taking the role of
  25. 25. TopCoder (A): Developing Software through Crowdsourcingplatform managers, Hughes estimated that the platform manager’s weekly time on aproject wouldshrink from forty hours to two, thus saving the client and TopCoder considerabletime and money.The FutureAs of December 2009, no competitors had elected to copy TopCoder’s businessmodel by offeringfull service software development through a competition-based approach. Instead,companies such asRentACoder, Elance, and oDesk served as online liaisons between clients andfreelance softwaredevelopers. Unlike TopCoder, whose clients only paid for solutions, clients atthese firms used a ’buytalent’ approach: they selected one or more programmers to solve their problem.More similar toTopCoder, uTest used crowdsourcing to find bugs and check the functionalusability of web, mobile,desktop, and gaming applications, but did not engage in software development.According toHughes, this lack of direct competition reflected the technical difficulties andcosts associated withbuilding a full-fledged community and platform.After a significant downturn in the global economy in 2008 and 2009, Hughesbelieved thatTopCoder was primed for growth. Sales staff were forecasting aggressive targetsfor the volume ofcompetitions and revenues and there were several strategic partnerships underconsideration.However, significant challenges and uncertainty remained. In particular, Hugheswondered whetherthe community, as well as the company, could grow to meet increasing demand.Stakeholders had divergent views. Mike Morris, Vice-President of Sales, sawunlimited potential:’If sales grow at a linear rate, membership grows at an exponential rate. Thesupply of communitymembers is not going to limit growth. If you throw enough money out there, youwill get enoughprogrammers.’ Community members Paweska and Wu agreed that offering more moneyper contestwould increase participation among existing members. Paweska also believed,however, that holdingmany more contests than usual in a given week would result in inexperiencedcompetitors competingactively for the prizes, possibly reducing code quality. Furthermore, Lydonnoted that that duringTopCoder’s last large scale up in 2007, review quality suffered during atransitional period of a fewmonths. As more contests became available, the usual reviewers wanted to competein the contests,rather than review them, leaving TopCoder scrambling to find replacements. Inaddition, a fewclients worried that as the number of avenues of competition at TopCoder grew,attracting the samegroup of competitors would prove much more difficult, reducing contestconsistency and continuity,which were especially critical for addressing legacy systems.
  26. 26. Hughes also worried about client service. If the number of TopCoder clientsexpandedsignificantly, TopCoder’s staff might face increasing difficulties responding toall those clients’questions and concerns. For large clients, expansion might require adding moreplatform managers,but Rob Hughes, TopCoder’s COO, was concerned that too many platform managersmight make thefirm appear to be like any other large IT consulting company, with the risk oflosing its uniquebusiness model.Even if Hughes succeeded at growing TopCoder, he was unsure about the company’scompetitiveposition. In particular, Hughes wondered if community members would stick withTopCoder if anew competition-based software development company emerged. What would happen ifa companylike Accenture started to develop software in the same way as TopCoder? Wouldthe TopCodercommunity remain intact?
  27. 27. 610-032-14Exhibit 1TopCoder Members Involvedin Makinga PowerPricing Systemfor anEnergyCompanySource:Company documents;DevelopedviaaTopCoderStudio competition.
  28. 28. 610-032-15Exhibit 2Number of Clients, Revenue,Number ofPlatformManagersandPlatformManagerCosts by Quarter 200720082009Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Number of Clients323234252438374746473835Total Revenue($MM)4.664.503.805.355.805.504.852.602.251.921.822.45Number of Platform Managers515144
  29. 29. 525246443620191816Cost of Platform Managers ($MM)a1. Company statistics.aIncludesPlatformManagers salaries, benefitsand other expenses.
  31. 31. TFConceptualizationLogo-Tournament0-$ConceptContest-Tournamentxxxxx14,500.00$Wireframes-Tournamentxxxxx13,000.00$Storyboard -Tournamentxxxxxxxxx13,500.00$ApplicationBuildSystemArchitectureSystemArchitecturexxxxxxxxxx14,500.00$ComponentDesignxxxx
  32. 32. xxx12,300.00$ComponentDevelopmentxxxxxxxx12,050.00$CatalogComponents0-$SystemAssemblyxxxxxxxxxx14,900.00$<Functional>ModuleModuleSpecificationxxxxxxxxxx14,000.00$Module Architecturexxxxxxxxxx14,000.00$ComponentDesignxxx
  33. 33. xxxxx36,900.00$ComponentDevelopmentxxxxxxxxx36,600.00$CatalogComponents0-$ModuleAssemblyxxxxxxxxx14,900.00$Prototype Assemblyxxxxxxxxxx14,900.00$TestingTestScenariosxxxxxxxxxx23,900.00$TestCasesxx
  34. 34. xxxxxxxx23,900.00$Bug Huntxxxxxxxx12,750.00$DeploymentDeploymentxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  35. 35. xxxxxxxxxxx204,000.00$UpdatesBug Racesxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  36. 36. xxxxxxxxxxxxxxxxxxxxxxxxxxx304,200.00$Total74,800.00$Timeline4/275/45/115/185/256/16/87/278/38/106/156/226/297/67/137/20Source:Company documents.
  37. 37. TopCoder (A): Developing Software through Crowdsourcing610-032Exhibit 4 Community Growth250,000050,000100,000150,000200,000TotalCommunityMembersSource: Company statistics.Exhibit 5 Average Contest Registration, Submission and Prize Amount for ClientContests in 2008and 20092008 2009Contest Type # of Submissions Prize $ per # of Submissions Prize $ perRegistrants per Contest Contesta Registrants per Contest Contestaper Contest per ContestConceptualization n/abn/ab n/ab 17.57 3.60 $1,314Specification n/abn/ab n/ab 14.20 1.94 $1,017Architecture 16.3 1.64 $1,590 19.34 1.75 $1,095ComponentDesign 9.83 2.85 $899 16.26 1.94 $559ComponentDevelopment 15.4 2.69 $733 25.59 2.56 $465Assembly 16.09 1.12 $1,628 18.38 1.18 $913Studio 27.55 14.57 $795 27.57 20.04 $1,015Source: Company statistics.aPrize per contest= Prize for 1st & 2nd places and reserve for Digital Run.bn/a- Data not available for most of 2008.17
  38. 38. TopCoder (A): Developing Software through CrowdsourcingExhibit 6 Example Community ProfileSource: http://www.topcoder.com/tc?module=MemberProfile&cr=287614 - 12/23/2009.
  39. 39. 610-032-19Exhibit 7Number ofunique participants by contest type peryearand month/Total number of official contests per year20052006200720082009Contest TypeAverage Submittersper Monthper YearContests per YearAverage Submittersper Monthper YearContests per YearAverage Submittersper Monthper YearContests per YearAverage Submittersper Monthper YearContests per YearAverage Submittersper Monthper YearContests per YearAlgorithm ContestsSingle Rounda Marathon MatchbClientSoftware Development Contests ConceptualizationSpecification Architecture Component Design Component Dev. Assembly DesignContests StudioTotald1,3195,287500c500c33157583161,3705,56581 1c 362 287 7301,9307,5252731,53245
  40. 40. 2258845165c223c2,1468,517105 20 615 484 17c1,2242,6388,9942811,5881c4c61243135605747814532,86710,072113 30 4c 698 780 86 118 1,8252,94510,4332531,6217c11c5c7c42938144914341059662793,19811,48766 29 9c16c 65 488 733 191 451 2,0232,5589,6163142,15063463293620934120420104102
  41. 41. 4292,91111,1225135 7071145300337416 456 1,881Source: Company statistics.a75minuteprogramming contest.bProgrammingcontestthatrunfrom3-30days.cPartialyeardata’contest trackofficiallydid notofficiallystartuntil themiddle orend of the year.dRepresentsthe unique numberofparticipantsduring agiventime period. Columnsare not additive.