(Êîììåíòàðèé "Âûäåëåíèå" Gunchenko15.02.2013 17:28:44ïóñòî)9-610--032JANUARY 1 55, 2010KARI MM R. LAKHANIDAVID A. GARVINERIC L ONSTEINToppCoderr (A): DDevelopping Sooftwaree througghCroowdsouurcingIn December 22009, Jack Huughes, CEO and founderr of TopCodeer Inc., enterredhis comppanysheadqquarters in Glastonbury,, Connecticuut, eager to review a pparticularlyccomplex sofftwaredevelopment projeect for an eneergy firms dyynamic poweer pricing systtem.Eight yeears after founndingTopCoder, Hughees still enjoyyed detailed project reviews. He waas particularllyproud thaat hiscompany could prroduce high-qquality softwaare solutions for which hiis ownemplooyees did nott haveto wrrite a single liine of code. IInstead, the ffirm nurturedd a globalcommmunity of mmore than 2225,000progrrammers whoo competed too design and create softwaare modules for TopCoderclients, a prrocessthat thhe popular prress called croowdsourcing.1Huughes smiledd at the projeccts success. TThe resulting software codde wasbug-freee and opera tionalon itss first day, a rrarity in the ssoftware induustry. Especiially impressiveto Hughees was that inn fourmonthhs, 65 particippants from 1 1 countries inn 6 continents had competed in 57conntests to creatte thiscriticaal pricing syystem for thee client (see Exhibit 1). AAs of 2009, TopCoderrooutinely prodducedsoftwware solutionss for over 45 clients, incluuding AOL, Best Buy, Elii Lilly,ESPNN, GEICO, annd theRoyall Bank of Scottland.In the past eighht years, Hughes had refinned TopCoderrs business mmodel toaccoommodate onngoingchangges in the sooftware induustry, while also pursuinng its uniquuecompetitioon-based sofftwaredevelopment apprroach. He haad transitioneed his busine ss from a moodel thathelpped other sofftwarefirms identify topp coders to aa company thhat developedd custom softtwarethroughh a combinatiion oftradittional IT conssulting servicces and comppetitions, pitting developeersworld-widde to solve cllientsprobleems.Thhe shift to a greater emmphasis on competitionns, encompasssing all asppectsof softwaredevelopment, howwever, meant that project volume was a "growing" issue--forToopCoder. Huugheshad too think throuugh how a commpetition-bassed business model, whichhincreasinglyy stressed conntestsas ann organizing,, as well as money-makking approacch, could hanndle increaseesin numbeers of
TopCoder (A): Developing Software through Crowdsourcingsolutions. But, was $200 million in revenue a reasonable goal? Did hisassumptions make sense? Ifso, what would it take to increase revenues by over an order of magnitude?Background and Current OperationsBefore he founded TopCoder in 2001, Hughes had built a custom softwaredevelopment2company, Business Data Services, in 1985; renamed Tallan in 1991. Tallanemployed some 600 peoplebefore being sold to CMGI in 2000.3 As he was completing the transaction, Hughesreflected on whathe had learned from his experiences at Tallan--these experiences would inspirethe core tenets of theTopCoder business model. Although Hughes enjoyed his time at Tallan, there wereareas in whichthe company struggled. For example, recruitment was an expensive and frustratingprocess becausefinding qualified programmers was time consuming and talent was difficult toassess. Due toconstantly evolving technologies, programmers skill sets often became obsoleteafter only a fewyears of productive service, leading to high levels of employee turnover.Furthermore, despiteTallans goal of maximizing billable hours, Hughes believed there wereopportunities to save clientstime and money by, for example, reusing computer programs basic componentsinstead of buildingeach application from scratch.Drawing upon these and other insights, Hughes set about creating a new kind oforganization thatwould build a "community" of programmers to help address the issues he hadidentified. Theseprogrammers would compete--as well as affiliateby building and using componentsthat hadalready been tested and found workable. The idea of reusing software componentsfor new projectswould become the core of the solutions the new company, called TopCoder,provided. Hughesenvisioned the company as a two-sided platform for software development. Oneside of theplatform would be clients, firms that needed software developed, who would workwith his staff tospecify programming challenges. The other side would be community members whowould competein contests to create solutions to the challenges for money and skill ratings.TopCoder would be inthe middle as the platform host, designing and enforcing the rules of engagementbetween clientsand the community members. Pete Bourdon, TopCoders CFO, explained the companyneeded toexcel at five core tasks: breaking down large client software projects intocomponents, taking in andprocessing client project specifications, determining appropriate contestprizes, having a consistentand unbiased way of picking contest winners, and fixing bugs at the back end ofdevelopment.Setting out to amass an initial collection of highly skilled programmers, from2001 to 2003
TopCoder asked established software development companies to sponsor world-wideweb-basedprogramming competitions. The sponsorships increased the popularity andlegitimacy of TopCoderscompetition platform and provided the company with access to talentedprogrammers from aroundthe world. In return, the sponsors, including Sun Microsystems and Google, usedthe contests toadvertise and recruit new talent. Tanya Horgan, TopCoders V.P. of Finance,explained that duringthe sponsorship phase, TopCoder offered unusually large prizesas much as $5,000to $10,000 permatch for tournament winnersto attract competitors and expand the community. Inaddition, everycontestant that participated was given an objective numerical rating for theirperformance against theglobal talent pool, providing a clear signal to TopCoder and others about thetalent in thecommunity.2 Custom software development done by specialist firms in the global ITconsulting and services sector (for example,Accenture and IBM) was an over $500 billion segment in 2008. (Source: Global ITConsulting & Other Services: IndustryProfile, Data Monitor, March 2009.)3 Clint Boulton, CMGI Acquires Tallan for $920 million, February 14, 2000,http://www.internetnews.com/ecnews/article.php/303771 8/1/2009.
TopCoder (A): Developing Software through CrowdsourcingBy the end of 2004, the TopCoder community was 50,000 members strong. In itsearly efforts touse the community to generate revenue, TopCoder acted as a placement firm,matching top ratedcommunity members with firms seeking new talent. Hughes, however, wasdissatisfied: I cringed atthe idea of TopCoder becoming a placement firm. That was not my end-vision forthe company.As of 2005, TopCoder began to use its community to develop software componentsandapplications. Hughes first tested this model by having highly rated communitymembers compete toredesign and rewrite the firms own software platform. The resulting code washigher quality andmuch less expensive than TopCoders own internally developed solution. Hughesnow had positiveproof that complex software systems could be built through competitions.Initially, TopCoder adopted a model to create solutions for clients bycontracting with communitymembers, running competitions, and providing consulting services. The companybroke down thesoftware development process into seven distinct but interrelated tasks; 1)Conceptualization, 2)Specification, 3) Architecture, 4) Component Production, 5) ApplicationAssembly, 6) Certification,and 7) Deployment. Most revenue came from consulting services: TopCoder billedclients for the timethe companys platform managers spent conceptualizing and specifying clientproblems, setting upcomponent design and development competitions, assembling components, anddelivering finishedsolutions.Shortly after TopCoder started developing software for clients, the companyidentified reusablecomponents from the software it was creating and collected the components in acatalog. Thesesoftware components became an important part of TopCoders value proposition toits clients. Manyof the custom applications could be produced by combining existing catalogcomponents with newcomponents built through competition. TopCoders hybrid consulting model led to large increases in revenues. However,Hughes wasstill dissatisfied: I viewed the hours-based services approach to be a broken,inefficient model. In2007 and 2008, TopCoder produced nearly $20 million in revenue but platformmanager costsremained high (see Exhibit 2 for information on revenue and platform managercosts). Attempting toalleviate costs, in 2007 TopCoder introduced competition tracks for componentarchitecture andassembly. With these new competition tracks in place, the work traditionallydone by platformmanagers would now be done by the community. In 2008, the company also addedcompetitions insoftware development tasks such as conceptualization and specification, as wellas deployment and
bug fixing.By early 2009, TopCoder had moved increasingly away from the hybrid consultingmodel. It nowfocused on completing all tasks in software development through competitions.Instead of paying fortime and materials for TopCoder platform managers, clients paid a monthlyplatform fee, based onthe complexity of their software requirements and the estimated number ofcompetitions they wouldrun through the TopCoder platform each month. The platform fee also providedclients withunlimited access to the over 1400 components in TopCoders catalog. Roughly 60%of most clientsprojects could be accomplished through reusing components from the catalog. Themove from thehybrid consultancy model to a competition model was coupled with the reductionof many platformmanager positions, leaving the company with 16 project managers servicing 35clients by the end of2009.As of late 2009, TopCoder ran two different types of competitions on itsplatform: algorithm andclient software development. Algorithm competitions served as the primary meansfor attracting newmembers and retaining existing members. These competitions required members todevelop creativesoftware solutions to relatively difficult programming challenges. All memberswere assessed against
TopCoder (A): Developing Software through Crowdsourcingeach other through an automated computer scoring system; they then received aTopCoder rating fortheir performance. Some algorithm competitions also had cash prizes for winners.The second type of competition was targeted at developing software applicationsfor specificclient needs. A TopCoder platform manager initially worked with the client staffto develop a GamePlan (see Exhibit 3 for a representative Game Plan) or a project road map forbuilding the software.The first step typically involved a contest where the general client problem waspresented to theTopCoder community in a conceptualization contest. Here contestants publiclycross-examined theclient staff as to their actual needs and then submitted a business requirementsdocument and highleveluse cases. The client picked the submission(s) that best represented their needsas the basis forfurther development. Then a series of specification contests were held to createthe applicationsrequirements documents, application wireframes (i.e. the logical flow ofapplication) and storyboards(detailed cases of the user experience). Next, the output of the specificationcontests were fed intoseveral architecture contests where the overall system and component leveldesigns would becreated. At this point the TopCoder platform manager would work with the clientto either selectcomponents from the catalog or commission the creation of new components throughdesign anddevelopment competitions. After the component production phase, all the relevantcomponents wereput together through an assembly competition with the objective of creating aworking system.Assembly was then followed by certification and testing contests and theneventually deployment.Throughout the execution of the Game Plan, TopCoder retained flexibility indevelopment byrunning Bug Races to accommodate changing client specification or unforeseenerrors. TopCoderalso had been issued eight U.S. patents on various aspects of running onlineprogramming contests ina distributed community setting and had other patents pending domestically andinternationally.To determine winners and assess quality in client software development, TopCoderused acommunity-based peer review system. In particular, expert and experiencedTopCoder communitymembers were paid to grade and comment on all contest submissions using detailedscorecards,ultimately picking the contest winners. The winning competitors for each contestwere then awardedmonetary prizes and all participants were given updated ratings for theirperformance. TopCoderalso ran Studio contests if an application required logos or graphics; in thosecases, clients chose thewinners.Evolution of the TopCoder Community
Growth and CompositionFrom 2001 to 2009, TopCoder added an average of 25,000 new computer programmersto itscommunity each year. After filling out a short online registration form, anybodyin the world couldparticipate in a software development competition; by spring 2009, the TopCodercommunity hadover 200,000 members (see Exhibit 4 for community growth). Although the size ofthe overallcommunity was large, the number of people within that community who activelyparticipated incontests and posted in forums was much smaller. The majority of communitymembers at TopCoderregistered as members of the community but never competed in any contests. Infact, by 2009, only35,000 unique individuals had competed in contests. To Mike Lydon, TopCodersChief TechnologyOfficer, the remaining 82.5% of the community was the latent pool: people whowere interestedenough in the TopCoder platform to register and had the potential to provideTopCoder withincreased development under the right conditions.A second group within the TopCoder community comprised those members who at onetimeparticipated in TopCoder contests but then stopped participating. Lydon notedthat, after TopCoderdecreased prize values in 2008, many competitors from the United States andCanada left the
TopCoder (A): Developing Software through CrowdsourcingTopCoder community. Yet another group included people who participated inTopCoder contestsbut did not win. TopCoder saw those competitors as the long tailpeople whoprimarilycompeted for the sake of learning. One of TopCoders main goals was to cultivatethe long tail so thatlesser skilled competitors could improve over time and increase their levels ofcontribution. Lastly,TopCoders most valuable group of competitors included the everyday winners. Thetalent ofTopCoders elite programmers was equal to the best in the world, but suchmembers only accountedfor 0.5% of the total TopCoder population.The core of TopCoders community was made up of single, highly competitive malesin their 20s.According to Michael Paweska, a six-year veteran at TopCoder: To be successfulat TopCoder, youmust ask yourself, Are you a competitor? You need to be able to thrive oncompetition; you cant bescared of it. You also need the flexibility to work long hours. TopCoder is abachelors sport: themoment you become involved with someone else, it becomes a point of friction.TopCoder attractedcompetitors from developed nations such as the United States, Canada, SouthKorea, and Japan, aswell as from emerging economies such as China, Russia, Poland, India, andUkraine. Wu Yanbo, aChinese TopCoder community member studying abroad in Australia, explained thatmostcompetitors in the lower-paid contests were from developing countries. Accordingto Wu, the prizeswere not large enough for many individuals from developed countries to compete,since they couldspend their time better elsewhere.Justin Gasper, a member since 2001, began experimenting with the TopCoderplatform whileworking for a traditional software engineering company. After winningsignificant money withTopCoder, Gasper decided to quit his job in 2005 and devote forty to fifty hoursa week to TopCoder.Gasper explained: TopCoder is my full time job; I dont have a day job. Gasperwas one ofTopCoders regular winners, a member of the global elite of programmers. Inarchitecturecompetitions, Gasper came in at least 2nd place 95% of the time and had a winpercentage of 69.23%.Competitors at TopCoder could choose which contests and what type of contests tojoin (see Exhibit5 for participation and prize data by contest type).Profiles and RatingsEach programmer in the TopCoder community maintained a public profile thatdisplayed his orher user name, contest history, and basic personal information. Another part ofthe member profiledisplayed a competitors numeric rating for each type of contest. The ratingsystem was modeled onthe one used to rank grandmaster chess players engaged in worldwide competition.
A red colorrating, or a rating of over 2,200, represented elite status within thecommunity and a high skill level.Yellow, blue, and green color ratings represented descending skill levels. Eachcompetitors countryrank, total community rank, success rates for contests, and reliabilityorpercentage of times thecontestant joined a competition and submitted a passing solutionwere featuredin their profiles.TopCoder members could also choose whether or not they wanted to display theirtotal earnings ontheir profiles (see Exhibit 6 for an example member profile).Motivating MembersBetween 2001 and 2009, TopCoder paid out over $20 million dollars in prizes andpeer reviewmoney to its community of developers. However, prize money was not evenlydistributedthroughout the TopCoder community. The top 5% of prize earners receivedapproximately 80% ofthe total prize pool, while the majority of TopCoder community members earnedlittle or no moneyfrom competitions. Some competitors were extremely successful. For example, from2006 to 2008,Paweska earned $200,000 to $300,000 per year, while Gasper averaged over$100,000 annually. Wucommented: I have to say money is the most attractive thing. The prize is verygood compared to the
TopCoder (A): Developing Software through Crowdsourcingincome of my friends who are working in some local companies in China. Eventhough the economyis not very good and TopCoder reduced its prizes, I can still earn around $1000per month in myspare time. TopCoder typically awarded prizes to the top two submissions ineach contest, with thelions share of the prize money going to the top performer.Besides prizes awarded on a contest-by-contest basis, another main source ofincome for memberswas the Digital Run. In the Digital Run system, the top five ranked competitorsfor each contest wereawarded points based on contest rank and performance. At the end of each month,TopCoder talliedcompetitors total points and awarded the top point earners thousands of dollarsin bonus prizes.Paweska explained that success in the Digital Run was not all about who was thebest programmerbut more about who could handle the most all-nighters. Other competitors, suchas Gasper, alsomade money through contracted projects that TopCoder assigned.In addition to their cash earnings, many community members reported that theirTopCoder ratingwas very important because it provided an objective assessment of ability. Wucommented that it wasnot easy to maintain a very high rating as it required familiarity with manykinds of technologies,quick thinking, the ability to learn independently, a strong work ethic, andattention to detail.According to Wu, a TopCoder rating could be important for a programmers futurecareer. Forexample, a high TopCoder rating helped one of Wus friends earn a job at Google.Gasper noted thatTopCoder ratings were also symbols of status and prestige for many programmers:If you have redratings, people look up to you. Indeed, many prestigious software firms askedpotential recruits toget a TopCoder rating before applying for a job. To others, however, the ratingsystem was lessimportant. Gasper, for example, explained that winning and making money meantmore to him thanratings.Although there were differences of opinion regarding the importance of ratings,almost allcommunity members agreed that competing at TopCoder provided numerousopportunities to learnand improve. In fact, for many programmers, a TopCoder career often began withfailure, but postcontestevaluation and peer review of each submission helped them grow and improve.Gaspernoted: I totally failed in my first competition. But the reviewers were reallygood at pointing me inthe right direction, saying heres where you went wrong. You cant fake itbecause youre gettingpeer reviewed by people who are better at programming than you are. Thereviewers dont care ifthey hurt your feelings; they are direct. If they see a bad design, they rip itapart. Paweska agreedthat getting feedback from reviewers was crucial and added that community
members could alsolearn from acting as a reviewer for contests. For scientists and developers, Wubelieved thatalgorithm contests were particularly helpful at sharpening research skills andimproving criticalthinking abilities. In all cases, continual learning opportunities from peerswere an important reasonfor participation.Gasper described the appeal of working at home on a web-based platform insteadof in atraditional cubicle farm setting: I like the flexibility that TopCoder givesme. I dont need to drivea half an hour to work each day and can do the same work at home. If I want totake off a day to playgolf, I just do it. I also dont have to work from 2 to 6 pm, my mostunproductive hours. Sharingsimilar sentiments, Paweska liked that while working at TopCoder he did not havea supervisorlooking over his shoulder.Setting ones hours was convenient but also challenging, as competitors had toactively managetheir individual levels of participation. Gasper constantly balanced effort andreward to maximizeincome while still living a sustainable lifestyle. If theres something that isway too much work forthe payment, I wont do it thats a super power Ive developed. I know when thespec is clean andworthwhile to solve. Its a skill that comes from doing tons and tons ofcontests.
TopCoder (A): Developing Software through CrowdsourcingA Community of CompetitorsWu noted that, although the firm was competitive in sprit, competition atTopCoder was neverdisrespectful or nasty and that people liked to help each other, even when theycompeted in the samearena. TopCoder forums were the main source for collaboration. In the forums,less experiencedcommunity members asked for assistance on certain problems and received instantfeedback frommore experienced competitors.At TopCoder, conversations and relationships extended beyond the scope ofsoftwaredevelopment. Hughes reflected on a particularly remarkable exhibit of communalstrength andcaring for fellow community members outside of software development. When oneof thecommunity members died, he said, the outpouring of support was such that anumber of thecommunity members took all of their winnings for a few weeks and gave it to thedeceaseds wife. Itended up being tens of thousands of dollars.Once a year, TopCoder paid for all of the best talent from the community totravel to Las Vegas,Nevada, to compete in the TopCoder Open (TCO). In additional to serving as aproving ground forthe best programmers in the world, the TCO provided community members with theopportunity tonetwork professionally and socially.The TopCoder community had a distinctive culture, with identifiablepersonalities. Wu explained:I believe this community, like all others, has its own culture. Clearly, themembers built it upcontinuously. When I joined the community, there were already some leadingmembers who wereactive in competitions and forums, brought out good suggestions, and started upinteresting andimportant discussions. In some cases, the fame of community leaders extendedwell beyondTopCoder. For example, Tomasz Czajka, from Poland, achieved rock star statusand had his pictureplastered on billboards throughout Warsaw after he won the TopCoder Open in2006.The Clients PerspectiveClients came to TopCoder to have high quality software developed in a costeffective and timeefficient manner. TopCoder positioned itself to serve both large firms andmedium to small-sizedbusiness that wanted to see systems developed. Keith Moore, a TopCoder clientand former SeniorVice President at LendingTree.com, believed that, regardless of size, anycompany could takeadvantage of TopCoder, whether it was a five man operation or large outsourcingvendor. For manyCIOs, the process of software development and talent recruitment was a majorheadache, and missed
deadlines and large cost overruns were common worries. According to StephenLaster, the CIO atHarvard Business School and a TopCoder client, A typical IT shop will turn over48% of itsemployees every three years. This process is very costly. The same problemexists with ouroutsourcing consultants. When selecting consultant teams, we tried out 60programmers beforefinding our team of 20. With TopCoder, I pay for performance and the CIO seesNirvana.As of 2009, TopCoder had developed a strong relationship with existing clientsfor delivering highquality software solutions and superior customer service. After completing theirfirst project withTopCoder, 82% of clients signed up for a second round of contests. They citedseveral advantages.BenefitsBetter Ideas Before sinking thousands of dollars into a project, a client couldrun aconceptualization contest through which TopCoder members helped identify badideas and generatebetter approaches early in the development cycle. When the client introduced abusiness problem to
TopCoder (A): Developing Software through Crowdsourcingthe community, members asked hundreds of questions. Nic Perez, a formerTechnical Director atAOL, explained that the communitys questions gave us insights into problems Ididnt even reallyknow I had and saved us money by doing all of those questions upfront. Usingonline forums,clients answered questions for all competitors only once, avoiding repeatedefforts. In some cases,clients scrapped product ideas entirely after the community raised concernsabout the productslikely success or usability in the marketplace.TopCoders contest-based development system consistently produced highlycreative ideas andsolutions. According to Darren Smith, a Solution Architect for the e-commercedivision at FergusonEnterprises, North Americas largest plumbing supplies wholesaler anddistributor: The communitycomes back with many options. It really has surprised us. You never know whatyou are you going toget. The creative side allows us to go to the marketing management team and say,We could do x, y,and z that we may not have previously considered. Theyre adding value to ourbusiness becausethey bring us solutions that quite frankly we may not have considered or werenot resourced todeliver.(Êîììåíòàðèé "Âûäåëåíèå" Gunchenko15.02.2013 17:33:01ïóñòî)Superior Quality, Cost, Speed and Flexibility Clients praised TopCodersrigorousevaluation and documentation process for being well above industry standards.Reflecting on hisexperience working on the Google Talk interface to AOL Instant Messenger, Perezstated thatTopCoder and its community had a strong desire to deliver bug-free code and thateven the mostcomplex systems always had fewer than 100 identified bugs. According to Perez,the same sizedprojects, developed internally, at AOL would have had 5-8 times that number ofbugs.Another TopCoder client, a web-based startup business, noted that it would havehad to pay$350,000 to a large IT consulting firm, $200,000 to a small IT consulting firm,or $80,000 to individualcontractors to build the companys website. Using TopCoder, the client onlyspent $35,000. This sameclient proclaimed: At $35,000, its priceless. There is no other game outthere. A different clientnoted that based on its experience working with almost every type of softwaredevelopmentcompany, TopCoder charged approximately half of the fee of a large, tier one ITconsulting firm.Using the community for parallel problem solving, TopCoder marketed itself asfaster thanother software development shops. This was true for back end bug races and
system checks, asTopCoder took 72 hours to complete the same bug testing that a traditionaldevelopment firmfinished in 10 business days. However, for other steps in the softwaredevelopment process, reportson speed were mixed as some clients said that TopCoder worked at about the samespeed as a largeIT consulting firm while others lauded TopCoder for speed of completion.Especially appealing to clients was TopCoders ability to supply flexiblesoftware developmentcapacity. In particular, a TopCoder client could expand or reduce its businessrequirements anddevelopment capabilities without having to hire or fire programmers. Accordingto one client, a basicin-house computer programmer cost $120,000 a year, after accounting forbenefits, sick time, andvacation. Working with TopCoder, clients did not have to spend as much onemployees benefits anddowntime.ConcernsAlthough CIOs were impressed by TopCoders technical capabilities and costsaving potential,many often had initial reservations about working with TopCoders unusualsoftware developmentmodel.Intellectual Property and Security According to Ira Heffan, TopCoders ChiefLegalCounsel, For new clients unfamiliar with TopCoders model, IP and securityconcerns can be an
TopCoder (A): Developing Software through Crowdsourcinginitial point of resistance. Until they understand the documentation andprocesses we have in placewith the community members, they see IP and security as potential barriers toworking with acommunity. For example, some clients were apprehensive that a TopCodercommunity membermight divulge proprietary ideas, business plans, or operations to theircompetitors. In addition, someclients worried that once a component became an integral part of their ITsystems, the communitymember who built the component might attempt to prohibit its use or ask theclient to payconsiderable royalties. Lastly, some clients were concerned that a solutionsubmitted by a communitymember could be stolen, copyrighted, or taken from open source softwareprojects, thus potentiallyopening the door for intellectual property disputes.TopCoder had in place a number of initiatives targeted at addressing theseconcerns and reducingthe risk level for clients, and also took steps to communicate its processes. Toease clients intellectualproperty and security concerns, TopCoder produced a white paper that detailedconfidentialitypolicies, intellectual property assignment rules, and TopCoders modularapproach to softwaredevelopment. In addition, TopCoder allowed clients to keep their company namesanonymousduring competitions and helped clients generate test data sets to avoid theexposure of sensitiveinformation. At the clients request, before a community member was allowedentry into acompetition, all competitors could be required to sign a standard CompetitionConfidentialityAgreement.The peer review process was another means to ensure code security and quality.Peer reviewerswere selected and vetted by TopCoder employees based on their superiorperformance on priorcompetitions. TopCoder clients also had the option of running testingcompetitions at the back end ofsoftware production, serving as an additional means of checking code securityand quality.TopCoders compartmentalized software development process also made it difficultfor a singlecompetitor to insert harmful code into a program, since individual contests onlyaddressed one smallpiece of the overall program.Cultural Change Many clients realized that working with TopCoder would bedifficultculturally for their company. In particular, CIOs believed that internalemployees would viewTopCoder as a threat to their job security. One new client observed: TopCoderis a CIOs dream buta programmers worst nightmare. I fully expect that if this goes well and if myprogrammers seegood quality work coming out of TopCoder, fear will quickly explode throughoutthe building.Although using TopCoder could help a company scale and reduce the programming
staff costs,companies still had to retain the big thinking peoplethe employees who couldguide theTopCoder development process. The managers at TopCoder clients also had toadjust to a perceivedloss of control over the software development process. Smith commented: We setthe competitions,but they manage the whole process. Our project management group works with theTopCodermanager to ensure delivery according to pre-determined SLAs (Service LevelAgreement). Someclients even found a few community members to be pushy and rude during pre-competition questionand answer sessions.Coding Challenges Even if TopCoder did all of a companys internal developmentwork, thecompany still needed to have an internal staff to integrate the deliverable intothe clients existingsystems, review the code for security issues, and adjust and fix code as systemschanged over time.For example, Smiths team at Ferguson spent a significant amount of timeinspecting, testing, andprocessing TopCoders work to make sure that there were no security threats orbugs. In somecontests, TopCoder clients also spent time evaluating ideas and approaches frommultiple winningsolutions.Another ongoing issue for clients was finding the right types of problems andproviding theappropriate amount of problem detail for the TopCoder community. As Mooredescribed it, You
TopCoder (A): Developing Software through Crowdsourcingwant neither too much nor too little detail. You do not want to quell innovationbut also want asolution that makes sense in your system. Clients discovered that contestparticipation decreased ifthey were unclear about what problems they wanted to solve or presented problemsthat were toocomplex or vast in scope; in those cases, the TopCoder community struggled toproduce anacceptable solution. Clients also found that community members worked best whencontests lastedless than two weeks. If projects took too long to complete, contestants wouldlose interest and notmake submissions.Managing TopCoderThe Supply SideA management job at TopCoder was unique. Along with supervising internalTopCoderemployees, managers at the firm had to oversee a community of over 200,000members and direct theprocess of competition-based software development. According to Senior VicePresident GeorgeTsipolitis, the key to success was effective process management: When youremanaging acommunity, you are no longer managing individuals, youre managing a whole. Wecant controlindividuals. We can only control the process of their participation. Clientsand employees alikebelieved that the sustainable value of the company was dependent on TopCodersability to facilitatecommunity participation and foster community growth. Lydon described the risks:From thebeginning, we focused on the community. We knew they could be unforgiving. Ifyou did the wrongthing, you got crucified.Attraction To run many competitions simultaneously and produce solutions formany clientsat the same time, TopCoder needed to have access to a critical mass of talentand coding capacity.TopCoders primary means of attracting new members into the community was theappeal andchallenge of the algorithm challenges. In addition, TopCoder occasionallyadvertised its onlinecompetition platform by paying for Google keyword searches using terms such asdesign contests.A third mechanism for attracting talent was member development days. Organizedby a smallteam of TopCoder employees, member development days were held at Chinese andotherinternational universities. At a member development day, a studentrepresentative would post signsaround the school and explain the TopCoder system. A primary goal of thesemember developmentdays was to encourage participation in the higher revenue producing developmentand designcontests. During one member development day in China, TopCoder registered overone thousandnew community members. Bourdon noted that TopCoder had achieved critical mass
once it crossedthe 200,000 member threshold as there were now many members with deep and narrowskills over arange of software development challenges (See Exhibit 7 for the number ofparticipants by contesttype).Norms As the community grew, TopCoder paid close attention to establishingcommunitynorms. As contest administrator, the company had to maintain the higheststandards of contestintegrity, fairness, transparency, and quality. For example, TopCoder personnelstrictly monitoredcompetitions and tolerated no form of cheating. Community members who peeked atothercompetitors solutions, shared ideas during competition, or used unauthorizedcode wereimmediately eliminated from the contest. Often they were kicked out of thecommunity entirely.If any uncertainty or disagreement arose about which competitor won a particularcontest,TopCoder would spend the extra money to re-run the competition. Another part ofcontest integrity,Tsipolitis explained, was TopCoders emphasis on maintaining consistency ofrules and procedures:The second that participants cant figure out how to win, theyll stopparticipating. So we cantchange the rules of a competition mid-stream. TopCoder also guaranteed completecontest
TopCoder (A): Developing Software through Crowdsourcingtransparency by storing all contest and competitor statistics, peer reviews, andsolutions in a datawarehouse. The data were publicly available to registered community members,accessed via theTopCoder website.Integrity and fairness also extended to TopCoders corporate motivations andcommunitycompensation philosophy. In particular, TopCoder was upfront with the communityover itsintention to make money. When TopCoder made a decision to change corporatedirection orcompetition procedures, Hughes posted the information in the forums andexplained the businessreasons behind his decisions. Hughes also believed that, since the companybenefited from thecommunitys hard work, adequately compensating community members was essential.Governance Although TopCoder executives were responsible for final decisions,theyfrequently incorporated community members views into the process. Lydonexplained: We treat thecommunity as the driver for everything we do. If we have enough dissent frommembers, we alwaystake that into account. The problem is that when we dont know what to do, ourmembers will also besplit. Community member Gasper, shared a similar perspective: TopCoder tendsto push out ideasinto the forums to get feedback. Seventy-five percent of the time, they listento the community. ButTopCoder also has its own business interests to consider. Sometimes thecommunity and businessinterests dont line-up.Similarly, if competitors were unhappy with a peer review scoring outcome,TopCoder allowedthem to appeal the decision. Over 90% of contests featured at least one appeal.If a member appealed,peer reviewers had to provide specific reasons why the appeal was accepted orrejected. Ifdisagreement remained between contestant and reviewer, TopCoder employees ofteninvestigated.Contestants could also appeal directly and privately to TopCoder personnel orpost complaintspublicly on the TopCoder forums.TopCoder managers inevitably made decisions that sometimes disturbed and upsetthe TopCodercommunity. For example, facing a very difficult economic environment in thesummer and fall of2008, TopCoder reduced the contest prize amounts, cut payments to peerreviewers, and reduced thenumber of algorithm competitions. During this period, some TopCoder competitorsleft thecommunity entirely and others dramatically reduced their participation levels.Gasper argued thatthe payment cuts also led to many superficial reviews because the best reviewerswere no longerdoing the work, which then required additional cycles to achieve acceptablequality.
Resource Allocation Another part of the TopCoder managerial role was allocatingcommunity resources and controlling contest participation. Lydon explained, Wehave to figure outhow to distribute the number of people who want to participate across the numberof contests thathave to be solved. Unlike a typical software development firm, TopCoder couldnot assign specificpeople to a task or project. As participation manager, TopCoders goal was tominimize the costs ofevaluation, stimulate effort through competition, and get at least one solutionthat was acceptable tothe client. To achieve the ideal number of submissions and participants,TopCoder adjusted the prizeamount, the duration and timing of the contest, the number of other contestsrunning concurrently,and the problems complexity and scope. When deciding on the ideal number ofcompetitors,TopCoder also considered contestants reliability. As a last resort, TopCoderemployees reached outdirectly to individual community members if other methods did not lead to thedesired participationlevels.Although TopCoder managers could pull many levers to influence contestparticipation, theybelieved it was important not to act like the communitys boss. Hughes explainedhis communitymanagement philosophy: We dont own this community. We want people to be herewhen they
TopCoder (A): Developing Software through Crowdsourcingwant to be here. You are just going to get much better results when you letpeople do what theyreally want to do.Retention At the same time, TopCoder executives worked to retain communitymembers andencourage future contest participation. At least one client raised concerns inthis area: I think thatcommunities are fickle. Community members could start to ask, why do I needthem? For example,what happens if an imitator comes along and offers 2x the prize amount? Toavoid such problems,TopCoder tried to supply community members with consistent work streams andprize money.TopCoder also encouraged community members to engage in the community as much aspossible bydedicating significant resources to facilitating forum discussions and invitingcontestants toparticipate in peer reviews, write problems for contests, and develop TopCodersinternal systems.TopCoder community members differed on their level of loyalty to the TopCodercommunity.Paweska, appreciative of all the opportunities TopCoder had provided, reflected,I have someloyalty. I think it would take a lot for me to leave. Only if there were noprojects would I leave.Gasper viewed his position in the community in a different light. Im not superloyal to TopCoder oranything, he said. To defect, the payment and work would have to outweigh thepayment andflexibility I have at TopCoder.The Demand SidePlatform Managers The other side of management at TopCoder was guiding clientsthroughthe contest-based software development process. This was the responsibility ofthe companysplatform managers, whose job was to induce the appropriate amount of communityparticipation,make suggestions for contest prize amounts, gather feedback between contests,and provide projectstatus updates to clients. Before starting the next step in the Game Plan,platform managers alsoadjusted contest requirements based on the work already completed. Once theproduct was deliveredto the client, TopCoder platform managers were required to act in a support andservice role. If therewas a technical problem with a solution, the platform manager often contactedthe communitymembers who developed the component and worked with the community members to fixthe issue.Most enterprise-level clients believed the platform manager was pivotal to aprojects success. At alarge client like LendingTree, the platform manager was onsite three to fourdays a week, conductingdaily meetings with the internal teams. A large part of the platform managersrole was managingclient expectations and serving as a sounding board for client concerns. At the
backend of projects,although the community often assembled a projects components throughcompetition, the platformmanager was also an expert at combining the small software pieces. The componentintegration rolesaved the client hours of work trying to figure out how all the pieces fittogether. At Ferguson, Smithconsidered the TopCoder personnel working onsite to be an integral part of histeam.TopCoder Direct However, each platform manager added to TopCoders overheadcosts andnarrowed profit margins. As of 2009, a typical platform manager at TopCoder cost$100,000 a yearincluding benefits. To Tsipolitis, the platform managers time was not alwayswell spent. Our projectmanagers spend a lot of time babysitting, he said.To avoid a potentially large increase in expenses as TopCoder added clients andprojects, Hughescame up with the concept of TopCoder Direct, in which the client used thecompanys platformwith little to no intervention from its employees. Under this self-servicemodel, platform managerswould educate clients on how to use the TopCoder platform to manage the contest-based softwaredevelopment process themselves. Hughes envisioned shifting the platformmanagers responsibilitiesto an experienced community member or an external consultant familiar withTopCoders platformsomeone who would serve as a co-pilot to assist the client staff. With co-pilots taking the role of
TopCoder (A): Developing Software through Crowdsourcingplatform managers, Hughes estimated that the platform managers weekly time on aproject wouldshrink from forty hours to two, thus saving the client and TopCoder considerabletime and money.The FutureAs of December 2009, no competitors had elected to copy TopCoders businessmodel by offeringfull service software development through a competition-based approach. Instead,companies such asRentACoder, Elance, and oDesk served as online liaisons between clients andfreelance softwaredevelopers. Unlike TopCoder, whose clients only paid for solutions, clients atthese firms used a buytalent approach: they selected one or more programmers to solve their problem.More similar toTopCoder, uTest used crowdsourcing to find bugs and check the functionalusability of web, mobile,desktop, and gaming applications, but did not engage in software development.According toHughes, this lack of direct competition reflected the technical difficulties andcosts associated withbuilding a full-fledged community and platform.After a significant downturn in the global economy in 2008 and 2009, Hughesbelieved thatTopCoder was primed for growth. Sales staff were forecasting aggressive targetsfor the volume ofcompetitions and revenues and there were several strategic partnerships underconsideration.However, significant challenges and uncertainty remained. In particular, Hugheswondered whetherthe community, as well as the company, could grow to meet increasing demand.Stakeholders had divergent views. Mike Morris, Vice-President of Sales, sawunlimited potential:If sales grow at a linear rate, membership grows at an exponential rate. Thesupply of communitymembers is not going to limit growth. If you throw enough money out there, youwill get enoughprogrammers. Community members Paweska and Wu agreed that offering more moneyper contestwould increase participation among existing members. Paweska also believed,however, that holdingmany more contests than usual in a given week would result in inexperiencedcompetitors competingactively for the prizes, possibly reducing code quality. Furthermore, Lydonnoted that that duringTopCoders last large scale up in 2007, review quality suffered during atransitional period of a fewmonths. As more contests became available, the usual reviewers wanted to competein the contests,rather than review them, leaving TopCoder scrambling to find replacements. Inaddition, a fewclients worried that as the number of avenues of competition at TopCoder grew,attracting the samegroup of competitors would prove much more difficult, reducing contestconsistency and continuity,which were especially critical for addressing legacy systems.
Hughes also worried about client service. If the number of TopCoder clientsexpandedsignificantly, TopCoders staff might face increasing difficulties responding toall those clientsquestions and concerns. For large clients, expansion might require adding moreplatform managers,but Rob Hughes, TopCoders COO, was concerned that too many platform managersmight make thefirm appear to be like any other large IT consulting company, with the risk oflosing its uniquebusiness model.Even if Hughes succeeded at growing TopCoder, he was unsure about the companyscompetitiveposition. In particular, Hughes wondered if community members would stick withTopCoder if anew competition-based software development company emerged. What would happen ifa companylike Accenture started to develop software in the same way as TopCoder? Wouldthe TopCodercommunity remain intact?
610-032-15Exhibit 2Number of Clients, Revenue,Number ofPlatformManagersandPlatformManagerCosts by Quarter 200720082009Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Number of Clients323234252438374746473835Total Revenue($MM)4.664.503.805.355.805.504.852.602.251.921.822.45Number of Platform Managers515144
525246443620191816Cost of Platform Managers ($MM)a126.96.36.199.301.3188.8.131.520.530.490.460.40Source: Company statistics.aIncludesPlatformManagers salaries, benefitsand other expenses.
TopCoder (A): Developing Software through Crowdsourcing610-032Exhibit 4 Community Growth250,000050,000100,000150,000200,000TotalCommunityMembersSource: Company statistics.Exhibit 5 Average Contest Registration, Submission and Prize Amount for ClientContests in 2008and 20092008 2009Contest Type # of Submissions Prize $ per # of Submissions Prize $ perRegistrants per Contest Contesta Registrants per Contest Contestaper Contest per ContestConceptualization n/abn/ab n/ab 17.57 3.60 $1,314Specification n/abn/ab n/ab 14.20 1.94 $1,017Architecture 16.3 1.64 $1,590 19.34 1.75 $1,095ComponentDesign 9.83 2.85 $899 16.26 1.94 $559ComponentDevelopment 15.4 2.69 $733 25.59 2.56 $465Assembly 16.09 1.12 $1,628 18.38 1.18 $913Studio 27.55 14.57 $795 27.57 20.04 $1,015Source: Company statistics.aPrize per contest= Prize for 1st & 2nd places and reserve for Digital Run.bn/a- Data not available for most of 2008.17
TopCoder (A): Developing Software through CrowdsourcingExhibit 6 Example Community ProfileSource: http://www.topcoder.com/tc?module=MemberProfile&cr=287614 - 12/23/2009.
610-032-19Exhibit 7Number ofunique participants by contest type peryearand month/Total number of official contests per year20052006200720082009Contest TypeAverage Submittersper Monthper YearContests per YearAverage Submittersper Monthper YearContests per YearAverage Submittersper Monthper YearContests per YearAverage Submittersper Monthper YearContests per YearAverage Submittersper Monthper YearContests per YearAlgorithm ContestsSingle Rounda Marathon MatchbClientSoftware Development Contests ConceptualizationSpecification Architecture Component Design Component Dev. Assembly DesignContests StudioTotald1,3195,287500c500c33157583161,3705,56581 1c 362 287 7301,9307,5252731,53245