Crowdsourcing for Customer Experience Management Success
Steven W Beauchem http://www.linkedin.com/in/stevenbeauchem http://yetanothervaliantattempt.blogspot.com/ Phone: 847-224-8346 E-Mail: firstname.lastname@example.org Crowdsourcing for Customer Experience Management Success Fall 2012Why Read This ReportAs the importance of effective digital customer experience continues to grow, poor content quality will become amajor limiter in digital business success. Interactive Marketing Technology (IMT) professionals must expand their focusbeyond the technology platform, and provide solutions that improve the impact of content to the business. Recentdevelopments in crowdsourcing, specifically in the area of cloud labor, provide unique opportunities for IMT pros totake a proactive approach to content quality, both during content migration and after the Customer ExperienceManagement (CXM) platform is operational.
Crowdsourcing for Customer Experience Management Success 1Report HighlightsAn Effective Customer Experience Solution Requires Content QualityCustomer Experience Management (CXM) platforms have evolved well beyond managing html, images and keywords. Adynamic, personalized customer experience depends on content that is wrapped with rich metadata, structured forpublishing across multiple channels, and assigned to relevant business categories. As customer experience capabilitiesexpand, the amount of effort required to maintain content quality grows in parallel.Cloud Labor is Emerging as a Solution to Maintain Content QualityContent producers are overstretched trying to keep up with new content, and they simply lack the bandwidth to keepup with ongoing management. By leveraging a globally distributed workforce of task-based workers (i.e., Cloud Labor)to offload content quality maintenance, content producers can focus on building and launching new content whilecrowd workers keep existing content in excellent condition.IMT is the Architect of Cloud Labor IntegrationIn order for cloud labor solutions to function effectively, IMT pros need to collaborate from the start with contentproducers to identify best uses for cloud labor resources, design cloud labor processes and work activities, and developthe integration points between the CXM and cloud labor platforms.
Crowdsourcing for Customer Experience Management Success 2Firms Scramble to Support Rich, Meaningful Customer ExperiencesAs leading edge companies have demonstrated the business value of focusing on customer experience quality, so toohave customers become accustomed to digital touchpoints that are easy, useful and enjoyablei. Personalization, forexample, makes or breaks customer experiences, and winning companies make this a key value, rather than just afeature.”ii This resulting population of increasingly savvy customers has left many firms struggling to keep up, bothwith their direct competitors and with expectations being created by digital innovators (i.e., the “Amazon Effect”).And, as business-to-business (B2B) firms are starting to learn, customer experience is not just a consumer play;effective customer experience has the potential to drive measurable business benefit regardless of audienceiii.Interactive marketing technology (IMT) professionals need to play a critical role in their firms’ customer experiencestrategies. As experiences grow in sophistication, so does the level of pre and post-delivery content support required torealize the value of the delivered solution. Companies build rich, highly engaging customer experiences by combiningcustomer experience management (CXM) solutions and the content that populates them. In order for CXM features(e.g., search, personalization, analytics, marketing automation, commerce) to function as advertised, content quality(Figure 1) is a must.FIGURE 1: ATTRIBUTES OF CONTENT QUALITY—INHERENT VALUE, STRUCTURE, METADATA, AND TAXONOMY Is the content accurate, useful and utilized? Are the content Inherent descriptive attributes complete and Value accurately populated? Structure MetadataIs the content structureconsistently abstractedfrom presentation to Taxonomysupport ease of reuse? Has the content been categorized appropriately to support customer experience features?What’s frustrating to many IMT professionals is that they are fully aware of the content quality issues gumming up theirsolutions, and it’s not pretty. Firms port content into their shiny “next gen” CXM solution from multiple publishingsystems, each with its own metadata structures, quality standards and levels of quality enforcement. As a result,dynamic features and personalization fall prey to inconsistent, suboptimal content quality. Or, project time and costincreases exponentially as each additional content store is manually audited and optimizediv. And, like the second lawof thermodynamics, entropy starts kicking in the moment the CXM solution is launched; unless there have beendramatic cultural and process changes in the content producing organization, content quality will continue to degrade.IMT Professionals Deal With the Content Quality MessIt’s easy to point fingers when it comes to issues of content quality. IMT professionals blame content producers, who inturn complain about hard-to-use content management systems, and everyone takes a shot at creative agencies. Inreality, the situation is much more complex, emerging from a number of prior decisions and constraints into the hotmess we have today:
Crowdsourcing for Customer Experience Management Success 3Interactive budgets trend upwards, but interactive teams remain under-resourcedThe effort required to effectively optimize content for effective customer experience is frequently beyond thecapacity of most interactive teams to effectively managev. Companies have attempted a variety of “magic bullet”solutions (e.g., auto-classification, dynamic transformation, syndication) to reduce workload, but with mixed results.And, with every minor tweak to content architecture, legacy content recedes further away from optimal. Ideal or not,content teams tend to ignore yesterday’s issues to focus on the burning platform.Indiscriminate use of outsourcing to manage content exacerbate the issueThe quality of vendors managing content varies widely, as does the ability of interactive teams to define and managevendor relationships. This creates challenges ranging from misunderstandings regarding content requirements toproject delays to unacceptable deliverablesvi. Vendor agreements also tend to focus on specific quality attributes (e.g.,browser compatibility), with little attention paid to elements that drive a personalized customer experience (e.g.,metadata, structure, taxonomy). With the firm’s interactive team focused on getting content deployed, they may limitor skip content optimization activities altogether.Interactive teams swim in a torrent of user-generated contentOn top of all of this, interactive teams have new mandates to: encourage the creation of user-generated content(UGC); screen the volumes of assets being generated; and, identify items with the potential for extended use. Onceacquired, UGC has to go through the same content optimization processesvii as owned content, and we already knowhow likely those are to be completed.Management of content as a strategic asset has been lackingLet’s face it: firms just aren’t very good at managing content. Instead, they focus on short-term gains: investing invisual appeal at the expense of content architecture; gold-plating initial releases of content management solutionsbecause of funding uncertainties for follow-on work; and avoiding the perceived drudgery of content maintenance andoptimization in favor of deploying new content. While the rapid pace of change in content management technologieswill continue to create optimization challenges, solution owners haven’t done themselves any favors by focusing onimmediate needs at the expense of future possibilities.Cloud Labor: Now Ready to Support Customer ExperiencesFortunately, recent trends in crowdsourcing create potential opportunities to systematically and cost-effectivelyimprove existing content quality and optimize the quality of new content as it is deployed. Cloud labor is differentfrom outsourcing: “Cloud Labor is the leveraging of a distributed virtual labor pool, available on-demand to fulfill a range of tasks from simple to complex. Crowdsourcing is used to connect labor demand and supply. Virtual workers perform activities that range from simple to specialized tasks.”viiiOne cloud worker working for a client recently sent this email to the document author, apologizing for errors in hercompleted assignments: “Hi – Im really sorry – I completed 2 of your tasks before I realized that they were the same. Ihad sorted differently and yours appeared twice for me. I am really sorry and I dont expect to be paid.” While cloudlabor changes the nature of the relationship between workers and managers, individuals generally take their cloudlabor responsibilities as seriously as they would full-time employment.Over the past 24 months, cloud labor has emerged as a viable option for getting work done within the enterprise.Organizations are still exploring potential uses for a skilled, on-demand workforce to complete activities requiring toomuch human judgment to automate, yet insufficiently valuable to cost-justify dedicated resources. However, it is clearfrom cloud labor industry growth (75% in 2011 to $375 million) and the arrival of enterprise early adopters (e.g.,Amazonix, Target Stores, Microsoft, eBay) that firms are responding positively to the cloud labor value propositionx.
Crowdsourcing for Customer Experience Management Success 4In comparison to outsourcing, cloud labor provides greater flexibility at lower cost; and, when well designed andmanaged, can achieve and exceed outsourcing quality levels. The structural differences between the two extendedworkforce models are the source of crowdsourcing’s relative benefits (Figure 2).FIGURE 2: CLOUD LABOR VS OUTSOURCING Cloud Labor OutsourcingUnit of Work Work is managed and compensated at the Work is generally managed by task, but task level. As a result, workers are compensated on an hourly basis. Unless rewarded for completing more tasks per specified in the outsourcing agreement, time period. workers do not have an incentive to increase their task output.Business Relationship Workers function as independent Workers are employees of a contracting contractors, and generally self-select tasks organization. To meet delivery commitments, for which they are qualified and which hold workers may be assigned to activities for interestxi. which they are not qualified.Vendor Management Simple: since work is managed at the task Complex: quality standards must be clearly level, work can be screened and approved / defined in a legal agreement and managed rejected as it is completed. Rejected tasks with the outsourcing vendor. Issues in delivery are not compensated. Task design can be quality are often not identified until a major tested, revised and validated in small review milestone. Rejected work will typically batches using agile principles. result in additional costs for rework. Firms have limited input into work design beyond requirements definition.Scalability Dynamic: with some platforms hosting Flexible: new projects require time for vendor +100k cloud workers, new projects can get identification, contract negotiation and moving quickly. Note that projects requiring approval. Additional time is often required to specific domain expertise or up-front scale the project team to production levels. learning and certification will take longer to staff.What Types Of Content Management Activities Are Well Suited To Cloud Labor?When considering what content management activities to support with cloud labor, organizations should start with thelist of content maintenance tasks that keep getting delayed into next week/month/quarter (see Figure 3). While cloudlabor solutions will evolve to address more complex activities, you can achieve consistent results today given that yourcloud labor assignments meet the following criteria:• The tasks require human judgment and intuition. While automated solutions such as Autonomy’s MetaTagger provide effective textual analysis and content classification, results vary based on input quality. An experienced cloud worker can be more effective than a technology-based solution at judging relevance and classifying content, especially where images, audio or video are involved, or where there is limited content of high quality to “train” a
Crowdsourcing for Customer Experience Management Success 5 technology solutionxii. Activities requiring technical knowledge (e.g., classifying content on medical conditions for a healthcare website) will require additional time to identify appropriate workers and/or provide additional worker training.• The tasks are discrete. a cloud labor task needs to be a self-contained activity, with clearly defined inputs and outputs. A worker should be able to complete a selected task without external coordination or assistance (e.g., calling a support desk for clarification or coordinating with upstream / downstream participants in a process). Cloud laborers generally seek to compress as much work into as little time as possible, either because they’re seeking to generate as much revenue as possible; or, because it’s a secondary source of income and they have limited time in which to complete cloud labor tasks. In addition, cloud labor platforms aren’t sufficiently mature to support multiple workers collaborating within a single activity.• Task fulfillment cost is lower than equivalent internal or outsourced resources. As firms gain experience with the cloud labor approach, it will become fairly straightforward to build cost models for cloud labor tasks. Performing detailed cost analysis will help to justify investments in development and infrastructure required to support ongoing leverage of cloud labor.FIGURE 3: EXAMPLE CONTENT MANAGEMENT ACTIVITIES SUITED FOR CLOUD LABORTask Description VendorsTagging / Categorization Selection of categories or taxonomy Amazon, CloudFactory, CrowdFlower, attributes relevant to a content item (text CrowdSource, TagCow (images), Tagasauris or media). Note that cloud labor is (images) currently less effective for highly technical vocabularies.Transcription Creation of transcripts from audio or CastingWords, Speechpad video sources for use with search engine optimization (SEO) or ADA compliance.Content Descriptions Creation of teaser content, product Servio, CrowdFlower, CrowdSource descriptions or other summary content to support alternate display formats (e.g., list views, syndication).Content Moderation Review of new posts to communities, CrowdSource, CrowdFlower product forums, review sites, etc., to ensure suitability of content, compliance with behavior policies.Relevance Evaluation Determination of whether a given Amazon, CloudFactory, CrowdFlower, description or attribute is relevant to the CrowdSource content asset to which it is assigned (e.g., “is this a picture of a Ford Explorer?”).Content Extraction Scraping html from an existing URL and Amazon transferring into a structured content template.Translation Translation of content from source Smartling, Amara language into one or more target languages. Note that cloud labor is
Crowdsourcing for Customer Experience Management Success 6Task Description Vendors currently less effective for highly technical vocabularies, and may require additional review.Usability Testing Feedback from site or application testers TryMyUI on design patterns, navigation, feature validation, etc. Primarily useful for generic usability testing and quality assurance. As vendors expand and refine their crowd labor segmentation, it should be possible to recruit cloud workers specific to a firm’s target audience.Key Considerations for Planning Your Cloud Labor SolutionOrganizations can group content management processes c into two categories: batch processes (e.g., contentextraction to support a web content management (WCM) migration, or bulk update of content with new metadataattributes) and workflow processes (e.g., review of a new customer forum post for appropriateness, or confirmationthat a personalization feature is displaying the correct content). While cloud labor is effective for both process types,different approaches are required.For batch processes (especially those with limited reuse), organizations should design with a primary focus onefficiency, i.e., building the minimum solution that accomplishes the desired results. In particular, they should:• Develop worker user experience leveraging “out-of-the-box” platform capabilities. Most platforms provide basic task design and editing tools, as well as templates for common tasks. Leverage these to get up and running quickly.• Invest lightly in worker training and certification resources, assuming limited reuse. Unless there is significant detail or nuance you need to communicate to your cloud workers, stick with the basics (html, screen captures, screencasts, etc.).• Invest in pilot processes. Organizations should compare results from pilots to a pre-defined valid response set to troubleshoot worker training and task design issues. Look to the cloud worker community for feedback on issues and opportunities for improvement.• Bulk load task results to content repository via scripting. Most cloud labor platforms are capable of exporting results in CSV and other data formats.• Consider SaaS solutions that are specific to the batch process. The vendors identified in Figure 3 have already done a lot of the heavy lifting to get you moving quickly on certain content management processes.For workflow processes, firms should build for effectiveness, with the primary objective of delivering consistent resultsover time. In particular, they should:• Design and develop reusable worker user experience leveraging the firm’s CXM stack. The CXM toolset should be capable of providing a much more effective user experience than the crowd labor platform. Given that the crowd labor output would end up here anyway, it makes sense to build here in the first place.• Explore opportunities to encapsulate external tasks within CM workflow processes. Most content management vendors provide capabilities to externalize workflow activities (generally via structured XML output and input). This creates opportunities to initiate crowd labor tasks based on events within the CM solution (e.g., a new product description is created).
Crowdsourcing for Customer Experience Management Success 7• Integrate to the cloud labor platform for specific worker and task management services. The cloud labor vendor should provide web services APIs for the capabilities a workflow solution will require (e.g., authentication, authorization, worker task selection, task approval).• Consider third-party solutions for ongoing workforce management. A vendor ecosystem is emerging to manage worker qualifications and maintain consistent result quality over time. This adds to the ongoing solution cost, but reduces up-front investment in quality control expertise.• Invest in robust worker training and certification. Given the need for consistent result quality and repeatability over time, targeted investments in richer training assets (e.g., modular, just-in-time training) are justified.RecommendationsLeverage Cloud Labor for Content Quality ImprovementsWhile cloud labor is still climbing the maturity curve, organizations can recognize clear value in using it to addressspecific quality issues with customer experience-based content. It’s likely that your content professionals are havingsimilar thoughts, and a coordinated approach to piloting and implementing crowd labor solutions will help to establishgood practices for future expansion of cloud labor applications. Specifically, IMT professionals need to:• Understand the role of compensation in cloud labor performance. One of the biggest challenges firms face when implementing cloud labor solutions is figuring out task pricing. Recent research has indicated that task pricing has a limited impact on result quality—more compensation doesn’t buy better outcomes. However, higher task prices do result in faster completion; workers tend to chase better payouts and will complete as many as they can.xiii Explore how other organizations are pricing similar tasks, and adjust based on your priority (i.e., speed vs. cost).• Focus on user experience. Firms getting started in the use of cloud labor often assume they can shortchange experience design—that they can “throw something together”. Effective task design is the primary factor in achieving result quality from crowd labor, and poor task design will result in additional time and expense to get to desired results. Engage design resources and user test your tasks with crowd workers to ensure that the objectives of the task are easy to understand and efficient to complete.• Clearly define accountability for cloud worker management. It’s easy for stakeholders to forget that someone has to be responsible for care and feeding of the cloud workforce. Your author recommends that content producers maintain responsibility for cloud workforce management, given that the role has a direct impact on quality of content. Regardless of who owns the activity, adequate resources and funding must be allocated.• Get paid to explore the possibilities. One of the best ways to get a better understanding of cloud labor is to spend a few hours as a cloud worker. The experience will help to ground you regarding what works (or doesn’t) in task design and user experience, as well as the type of attention and discipline required to be successful as a cloud worker. And, you can always use your earnings to buy yourself lunch (or coffee, depending on how successful you are).
Crowdsourcing for Customer Experience Management Success 8Endnotesi Burns, Megan. "The Business Impact Of Customer Experience, 2011." Forrester.com. Forrester Research, 07 July 2011. Web. 06 June 2012. <http://www.forrester.com/The Business Impact Of Customer Experience 2011/-/E-RES59072?objectid=RES59072>.ii Brave, Scott. "Personalization Is Not A Feature." TechCrunch. TechCrunch, 18 May 2012. Web. 06 June 2012. <http://techcrunch.com/2012/05/18/personalization-is-not-a-feature/>.iii Hagen, Paul. "How B2B Firms Improve Customer and Partner Experience." Forrester.com. Forrester Research, 14 Mar. 2012. Web. 06 June 2012. <http://www.forrester.com/home#/How+B2B+Firms+Improve+Customer+And+Partner+Experience/quickscan/-/E-RES60388>.iv For the purposes of this document, “content optimization” is the collection of content provider activities performed to ensure content quality. These activities include: editorial review, search engine optimization, metadata and taxonomy population, presentation testing in target environments, etc.v VanBoskirk, Shar. "The Future of Interactive Marketing." Forrester.com. Forrester Research, 04 Apr. 2011. Web. 06 June 2012. <http://www.forrester.com/home#/The+Future+Of+Interactive+Marketing/fulltext/-/E-RES59137>.vi The Current State of Digital Content. Rep. ValueNotes, Mar. 2011. Web. 6 June 2012. <http://www.artisarkapps.com/admin/s4carlisle/wp- content/uploads/2012/01/Market-Study-Report.pdf>.vii For the purposes of this document, “content optimization” is the collection of content provider activities performed to ensure content quality. These activities include: editorial review, search engine optimization, metadata and taxonomy population, presentation testing in target environments, etc.viii "Cloud Labor." www.crowdsourcing.org. Crowdsourcing LLC, 06 June 2012. Web. 06 June 2012. <http://www.crowdsourcing.org/community/cloud-labor/6>.ix While a number of cloud labor platform providers are in the process of establishing themselves, the vendor that continues to receive the most attention is Amazon, with their Mechanical Turk (AMT). Initially built to support Amazon’s own needs for content maintenance, AMT now offers a robust, extensible cloud labor Platform as a Service (PaaS) that can be leveraged across a wide range of use cases. As a result, multiple solution providers (including most of the vendors identified in Figure 3) leverage the AMT services framework to deliver some portion of their feature set . For IMT professionals seeking a low-barrier means to explore cloud labor and perform pilot testing, AMT offers a fully-featured sandbox environment to build and deploy worker tasks.x "Enterprise Crowdsourcing Research Report." Www.crowdsourcing.org. Crowdsourcing LLC, 10 Mar. 2012. Web. 06 June 2012. <http://www.crowdsourcing.org/editorial/enterprise-crowdsourcing-research-report-by-massolution/11736>.xi A significant factor constraining more rapid adoption is the lack of clarity around the employment status of cloud laborers. For more information, see: Wolfson, Stephen, and Matthew Lease. "Look before You Leap: Legal Pitfalls of Crowdsourcing." Wiley Online Library. John Wiley & Sons, Inc., 11 Jan. 2012. Web. 06 June 2012. <http://onlinelibrary.wiley.com/doi/10.1002/meet.2011.14504801135/abstract>.xii In November 2012, the National Institute of Standards and Technology will be holding its 21st annual Text Retrieval Conference (TREC). For the last two (2) years, the conference has included a Crowdsourcing track whose focus is on leveraging human relevance judgments to inform / train automated classification solutions. For more information see: http://trec.nist.gov/.xiii Ipeirotis, Panos, and Praveen Paritosh. "Managing Crowdsourced Human Computation." Research at Google. Google, 2011. Web. 06 June 2012. <http://research.google.com/pubs/pub36946.html>.