Overview of the use of crowdsourcing tactics and capabilities to improve and maintain the quality of your interactive marketing content.

1. Steven W Beauchem
http://www.linkedin.com/in/stevenbeauchem
http://yetanothervaliantattempt.blogspot.com/
Phone: 847-224-8346
E-Mail: swbeauchem@sbcglobal.net
Crowdsourcing for
Customer
Experience
Management
Success
Fall 2012
Why Read This Report
As the importance of effective digital customer experience continues to grow, poor content quality will become a
major limiter in digital business success. Interactive Marketing Technology (IMT) professionals must expand their focus
beyond the technology platform, and provide solutions that improve the impact of content to the business. Recent
developments in crowdsourcing, specifically in the area of cloud labor, provide unique opportunities for IMT pros to
take a proactive approach to content quality, both during content migration and after the Customer Experience
Management (CXM) platform is operational.

2. Crowdsourcing for
Customer Experience Management Success
1
Report Highlights
An Effective Customer Experience Solution Requires Content Quality
Customer Experience Management (CXM) platforms have evolved well beyond managing html, images and keywords. A
dynamic, personalized customer experience depends on content that is wrapped with rich metadata, structured for
publishing across multiple channels, and assigned to relevant business categories. As customer experience capabilities
expand, the amount of effort required to maintain content quality grows in parallel.
Cloud Labor is Emerging as a Solution to Maintain Content Quality
Content producers are overstretched trying to keep up with new content, and they simply lack the bandwidth to keep
up with ongoing management. By leveraging a globally distributed workforce of task-based workers (i.e., Cloud Labor)
to offload content quality maintenance, content producers can focus on building and launching new content while
crowd workers keep existing content in excellent condition.
IMT is the Architect of Cloud Labor Integration
In order for cloud labor solutions to function effectively, IMT pros need to collaborate from the start with content
producers to identify best uses for cloud labor resources, design cloud labor processes and work activities, and develop
the integration points between the CXM and cloud labor platforms.

3. Crowdsourcing for
Customer Experience Management Success
2
Firms Scramble to Support Rich, Meaningful Customer Experiences
As leading edge companies have demonstrated the business value of focusing on customer experience quality, so too
have customers become accustomed to digital touchpoints that are easy, useful and enjoyablei. Personalization, for
example, makes or breaks customer experiences, and winning companies make this a key value, rather than just a
feature.”ii This resulting population of increasingly savvy customers has left many firms struggling to keep up, both
with their direct competitors and with expectations being created by digital innovators (i.e., the “Amazon Effect”).
And, as business-to-business (B2B) firms are starting to learn, customer experience is not just a consumer play;
effective customer experience has the potential to drive measurable business benefit regardless of audienceiii.
Interactive marketing technology (IMT) professionals need to play a critical role in their firms’ customer experience
strategies. As experiences grow in sophistication, so does the level of pre and post-delivery content support required to
realize the value of the delivered solution. Companies build rich, highly engaging customer experiences by combining
customer experience management (CXM) solutions and the content that populates them. In order for CXM features
(e.g., search, personalization, analytics, marketing automation, commerce) to function as advertised, content quality
(Figure 1) is a must.
FIGURE 1: ATTRIBUTES OF CONTENT QUALITY—INHERENT VALUE, STRUCTURE, METADATA, AND TAXONOMY
Is the content accurate,
useful and utilized?
Are the content
Inherent descriptive attributes
complete and
Value accurately populated?
Structure Metadata
Is the content structure
consistently abstracted
from presentation to Taxonomy
support ease of reuse?
Has the content been
categorized appropriately
to support customer
experience features?
What’s frustrating to many IMT professionals is that they are fully aware of the content quality issues gumming up their
solutions, and it’s not pretty. Firms port content into their shiny “next gen” CXM solution from multiple publishing
systems, each with its own metadata structures, quality standards and levels of quality enforcement. As a result,
dynamic features and personalization fall prey to inconsistent, suboptimal content quality. Or, project time and cost
increases exponentially as each additional content store is manually audited and optimizediv. And, like the second law
of thermodynamics, entropy starts kicking in the moment the CXM solution is launched; unless there have been
dramatic cultural and process changes in the content producing organization, content quality will continue to degrade.
IMT Professionals Deal With the Content Quality Mess
It’s easy to point fingers when it comes to issues of content quality. IMT professionals blame content producers, who in
turn complain about hard-to-use content management systems, and everyone takes a shot at creative agencies. In
reality, the situation is much more complex, emerging from a number of prior decisions and constraints into the hot
mess we have today:

4. Crowdsourcing for
Customer Experience Management Success
3
Interactive budgets trend upwards, but interactive teams remain under-resourced
The effort required to effectively optimize content for effective customer experience is frequently beyond the
capacity of most interactive teams to effectively managev. Companies have attempted a variety of “magic bullet”
solutions (e.g., auto-classification, dynamic transformation, syndication) to reduce workload, but with mixed results.
And, with every minor tweak to content architecture, legacy content recedes further away from optimal. Ideal or not,
content teams tend to ignore yesterday’s issues to focus on the burning platform.
Indiscriminate use of outsourcing to manage content exacerbate the issue
The quality of vendors managing content varies widely, as does the ability of interactive teams to define and manage
vendor relationships. This creates challenges ranging from misunderstandings regarding content requirements to
project delays to unacceptable deliverablesvi. Vendor agreements also tend to focus on specific quality attributes (e.g.,
browser compatibility), with little attention paid to elements that drive a personalized customer experience (e.g.,
metadata, structure, taxonomy). With the firm’s interactive team focused on getting content deployed, they may limit
or skip content optimization activities altogether.
Interactive teams swim in a torrent of user-generated content
On top of all of this, interactive teams have new mandates to: encourage the creation of user-generated content
(UGC); screen the volumes of assets being generated; and, identify items with the potential for extended use. Once
acquired, UGC has to go through the same content optimization processesvii as owned content, and we already know
how likely those are to be completed.
Management of content as a strategic asset has been lacking
Let’s face it: firms just aren’t very good at managing content. Instead, they focus on short-term gains: investing in
visual appeal at the expense of content architecture; gold-plating initial releases of content management solutions
because of funding uncertainties for follow-on work; and avoiding the perceived drudgery of content maintenance and
optimization in favor of deploying new content. While the rapid pace of change in content management technologies
will continue to create optimization challenges, solution owners haven’t done themselves any favors by focusing on
immediate needs at the expense of future possibilities.
Cloud Labor: Now Ready to Support Customer Experiences
Fortunately, recent trends in crowdsourcing create potential opportunities to systematically and cost-effectively
improve existing content quality and optimize the quality of new content as it is deployed. Cloud labor is different
from outsourcing:
“Cloud Labor is the leveraging of a distributed virtual labor pool, available on-demand to fulfill a
range of tasks from simple to complex. Crowdsourcing is used to connect labor demand and
supply. Virtual workers perform activities that range from simple to specialized tasks.”viii
One cloud worker working for a client recently sent this email to the document author, apologizing for errors in her
completed assignments: “Hi – I'm really sorry – I completed 2 of your tasks before I realized that they were the same. I
had sorted differently and yours appeared twice for me. I am really sorry and I don't expect to be paid.” While cloud
labor changes the nature of the relationship between workers and managers, individuals generally take their cloud
labor responsibilities as seriously as they would full-time employment.
Over the past 24 months, cloud labor has emerged as a viable option for getting work done within the enterprise.
Organizations are still exploring potential uses for a skilled, on-demand workforce to complete activities requiring too
much human judgment to automate, yet insufficiently valuable to cost-justify dedicated resources. However, it is clear
from cloud labor industry growth (75% in 2011 to $375 million) and the arrival of enterprise early adopters (e.g.,
Amazonix, Target Stores, Microsoft, eBay) that firms are responding positively to the cloud labor value propositionx.

5. Crowdsourcing for
Customer Experience Management Success
4
In comparison to outsourcing, cloud labor provides greater flexibility at lower cost; and, when well designed and
managed, can achieve and exceed outsourcing quality levels. The structural differences between the two extended
workforce models are the source of crowdsourcing’s relative benefits (Figure 2).
FIGURE 2: CLOUD LABOR VS OUTSOURCING
Cloud Labor Outsourcing
Unit of Work Work is managed and compensated at the Work is generally managed by task, but
task level. As a result, workers are compensated on an hourly basis. Unless
rewarded for completing more tasks per specified in the outsourcing agreement,
time period. workers do not have an incentive to increase
their task output.
Business Relationship Workers function as independent Workers are employees of a contracting
contractors, and generally self-select tasks organization. To meet delivery commitments,
for which they are qualified and which hold workers may be assigned to activities for
interestxi. which they are not qualified.
Vendor Management Simple: since work is managed at the task Complex: quality standards must be clearly
level, work can be screened and approved / defined in a legal agreement and managed
rejected as it is completed. Rejected tasks with the outsourcing vendor. Issues in delivery
are not compensated. Task design can be quality are often not identified until a major
tested, revised and validated in small review milestone. Rejected work will typically
batches using agile principles. result in additional costs for rework. Firms
have limited input into work design beyond
requirements definition.
Scalability Dynamic: with some platforms hosting Flexible: new projects require time for vendor
+100k cloud workers, new projects can get identification, contract negotiation and
moving quickly. Note that projects requiring approval. Additional time is often required to
specific domain expertise or up-front scale the project team to production levels.
learning and certification will take longer to
staff.
What Types Of Content Management Activities Are Well Suited To Cloud Labor?
When considering what content management activities to support with cloud labor, organizations should start with the
list of content maintenance tasks that keep getting delayed into next week/month/quarter (see Figure 3). While cloud
labor solutions will evolve to address more complex activities, you can achieve consistent results today given that your
cloud labor assignments meet the following criteria:
• The tasks require human judgment and intuition. While automated solutions such as Autonomy’s MetaTagger
provide effective textual analysis and content classification, results vary based on input quality. An experienced
cloud worker can be more effective than a technology-based solution at judging relevance and classifying content,
especially where images, audio or video are involved, or where there is limited content of high quality to “train” a

6. Crowdsourcing for
Customer Experience Management Success
5
technology solutionxii. Activities requiring technical knowledge (e.g., classifying content on medical conditions for a
healthcare website) will require additional time to identify appropriate workers and/or provide additional worker
training.
• The tasks are discrete. a cloud labor task needs to be a self-contained activity, with clearly defined inputs and
outputs. A worker should be able to complete a selected task without external coordination or assistance (e.g.,
calling a support desk for clarification or coordinating with upstream / downstream participants in a process).
Cloud laborers generally seek to compress as much work into as little time as possible, either because they’re
seeking to generate as much revenue as possible; or, because it’s a secondary source of income and they have
limited time in which to complete cloud labor tasks. In addition, cloud labor platforms aren’t sufficiently mature to
support multiple workers collaborating within a single activity.
• Task fulfillment cost is lower than equivalent internal or outsourced resources. As firms gain experience with
the cloud labor approach, it will become fairly straightforward to build cost models for cloud labor tasks.
Performing detailed cost analysis will help to justify investments in development and infrastructure required to
support ongoing leverage of cloud labor.
FIGURE 3: EXAMPLE CONTENT MANAGEMENT ACTIVITIES SUITED FOR CLOUD LABOR
Task Description Vendors
Tagging / Categorization Selection of categories or taxonomy Amazon, CloudFactory, CrowdFlower,
attributes relevant to a content item (text CrowdSource, TagCow (images), Tagasauris
or media). Note that cloud labor is (images)
currently less effective for highly
technical vocabularies.
Transcription Creation of transcripts from audio or CastingWords, Speechpad
video sources for use with search engine
optimization (SEO) or ADA compliance.
Content Descriptions Creation of teaser content, product Servio, CrowdFlower, CrowdSource
descriptions or other summary content to
support alternate display formats (e.g.,
list views, syndication).
Content Moderation Review of new posts to communities, CrowdSource, CrowdFlower
product forums, review sites, etc., to
ensure suitability of content, compliance
with behavior policies.
Relevance Evaluation Determination of whether a given Amazon, CloudFactory, CrowdFlower,
description or attribute is relevant to the CrowdSource
content asset to which it is assigned (e.g.,
“is this a picture of a Ford Explorer?”).
Content Extraction Scraping html from an existing URL and Amazon
transferring into a structured content
template.
Translation Translation of content from source Smartling, Amara
language into one or more target
languages. Note that cloud labor is

7. Crowdsourcing for
Customer Experience Management Success
6
Task Description Vendors
currently less effective for highly
technical vocabularies, and may require
additional review.
Usability Testing Feedback from site or application testers TryMyUI
on design patterns, navigation, feature
validation, etc. Primarily useful for
generic usability testing and quality
assurance. As vendors expand and refine
their crowd labor segmentation, it should
be possible to recruit cloud workers
specific to a firm’s target audience.
Key Considerations for Planning Your Cloud Labor Solution
Organizations can group content management processes c into two categories: batch processes (e.g., content
extraction to support a web content management (WCM) migration, or bulk update of content with new metadata
attributes) and workflow processes (e.g., review of a new customer forum post for appropriateness, or confirmation
that a personalization feature is displaying the correct content). While cloud labor is effective for both process types,
different approaches are required.
For batch processes (especially those with limited reuse), organizations should design with a primary focus on
efficiency, i.e., building the minimum solution that accomplishes the desired results. In particular, they should:
• Develop worker user experience leveraging “out-of-the-box” platform capabilities. Most platforms provide
basic task design and editing tools, as well as templates for common tasks. Leverage these to get up and running
quickly.
• Invest lightly in worker training and certification resources, assuming limited reuse. Unless there is significant
detail or nuance you need to communicate to your cloud workers, stick with the basics (html, screen captures,
screencasts, etc.).
• Invest in pilot processes. Organizations should compare results from pilots to a pre-defined valid response set to
troubleshoot worker training and task design issues. Look to the cloud worker community for feedback on issues
and opportunities for improvement.
• Bulk load task results to content repository via scripting. Most cloud labor platforms are capable of exporting
results in CSV and other data formats.
• Consider SaaS solutions that are specific to the batch process. The vendors identified in Figure 3 have already
done a lot of the heavy lifting to get you moving quickly on certain content management processes.
For workflow processes, firms should build for effectiveness, with the primary objective of delivering consistent results
over time. In particular, they should:
• Design and develop reusable worker user experience leveraging the firm’s CXM stack. The CXM toolset should
be capable of providing a much more effective user experience than the crowd labor platform. Given that the
crowd labor output would end up here anyway, it makes sense to build here in the first place.
• Explore opportunities to encapsulate external tasks within CM workflow processes. Most content management
vendors provide capabilities to externalize workflow activities (generally via structured XML output and input). This
creates opportunities to initiate crowd labor tasks based on events within the CM solution (e.g., a new product
description is created).

8. Crowdsourcing for
Customer Experience Management Success
7
• Integrate to the cloud labor platform for specific worker and task management services. The cloud labor vendor
should provide web services APIs for the capabilities a workflow solution will require (e.g., authentication,
authorization, worker task selection, task approval).
• Consider third-party solutions for ongoing workforce management. A vendor ecosystem is emerging to manage
worker qualifications and maintain consistent result quality over time. This adds to the ongoing solution cost, but
reduces up-front investment in quality control expertise.
• Invest in robust worker training and certification. Given the need for consistent result quality and repeatability
over time, targeted investments in richer training assets (e.g., modular, just-in-time training) are justified.
Recommendations
Leverage Cloud Labor for Content Quality Improvements
While cloud labor is still climbing the maturity curve, organizations can recognize clear value in using it to address
specific quality issues with customer experience-based content. It’s likely that your content professionals are having
similar thoughts, and a coordinated approach to piloting and implementing crowd labor solutions will help to establish
good practices for future expansion of cloud labor applications. Specifically, IMT professionals need to:
• Understand the role of compensation in cloud labor performance. One of the biggest challenges firms face when
implementing cloud labor solutions is figuring out task pricing. Recent research has indicated that task pricing has
a limited impact on result quality—more compensation doesn’t buy better outcomes. However, higher task prices
do result in faster completion; workers tend to chase better payouts and will complete as many as they can.xiii
Explore how other organizations are pricing similar tasks, and adjust based on your priority (i.e., speed vs. cost).
• Focus on user experience. Firms getting started in the use of cloud labor often assume they can shortchange
experience design—that they can “throw something together”. Effective task design is the primary factor in
achieving result quality from crowd labor, and poor task design will result in additional time and expense to get to
desired results. Engage design resources and user test your tasks with crowd workers to ensure that the objectives
of the task are easy to understand and efficient to complete.
• Clearly define accountability for cloud worker management. It’s easy for stakeholders to forget that someone
has to be responsible for care and feeding of the cloud workforce. Your author recommends that content producers
maintain responsibility for cloud workforce management, given that the role has a direct impact on quality of
content. Regardless of who owns the activity, adequate resources and funding must be allocated.
• Get paid to explore the possibilities. One of the best ways to get a better understanding of cloud labor is to spend
a few hours as a cloud worker. The experience will help to ground you regarding what works (or doesn’t) in task
design and user experience, as well as the type of attention and discipline required to be successful as a cloud
worker. And, you can always use your earnings to buy yourself lunch (or coffee, depending on how successful you
are).

9. Crowdsourcing for
Customer Experience Management Success
8
Endnotes
i
Burns, Megan. "The Business Impact Of Customer Experience, 2011." Forrester.com. Forrester Research, 07 July 2011. Web. 06 June 2012.
<http://www.forrester.com/The Business Impact Of Customer Experience 2011/-/E-RES59072?objectid=RES59072>.
ii
Brave, Scott. "Personalization Is Not A Feature." TechCrunch. TechCrunch, 18 May 2012. Web. 06 June 2012.
<http://techcrunch.com/2012/05/18/personalization-is-not-a-feature/>.
iii
Hagen, Paul. "How B2B Firms Improve Customer and Partner Experience." Forrester.com. Forrester Research, 14 Mar. 2012. Web. 06 June 2012.
<http://www.forrester.com/home#/How+B2B+Firms+Improve+Customer+And+Partner+Experience/quickscan/-/E-RES60388>.
iv
For the purposes of this document, “content optimization” is the collection of content provider activities performed to ensure content quality.
These activities include: editorial review, search engine optimization, metadata and taxonomy population, presentation testing in target
environments, etc.
v
VanBoskirk, Shar. "The Future of Interactive Marketing." Forrester.com. Forrester Research, 04 Apr. 2011. Web. 06 June 2012.
<http://www.forrester.com/home#/The+Future+Of+Interactive+Marketing/fulltext/-/E-RES59137>.
vi
The Current State of Digital Content. Rep. ValueNotes, Mar. 2011. Web. 6 June 2012. <http://www.artisarkapps.com/admin/s4carlisle/wp-
content/uploads/2012/01/Market-Study-Report.pdf>.
vii
For the purposes of this document, “content optimization” is the collection of content provider activities performed to ensure content quality.
These activities include: editorial review, search engine optimization, metadata and taxonomy population, presentation testing in target
environments, etc.
viii
"Cloud Labor." www.crowdsourcing.org. Crowdsourcing LLC, 06 June 2012. Web. 06 June 2012.
<http://www.crowdsourcing.org/community/cloud-labor/6>.
ix
While a number of cloud labor platform providers are in the process of establishing themselves, the vendor that continues to receive the most
attention is Amazon, with their Mechanical Turk (AMT). Initially built to support Amazon’s own needs for content maintenance, AMT now offers a
robust, extensible cloud labor Platform as a Service (PaaS) that can be leveraged across a wide range of use cases. As a result, multiple solution
providers (including most of the vendors identified in Figure 3) leverage the AMT services framework to deliver some portion of their feature set .
For IMT professionals seeking a low-barrier means to explore cloud labor and perform pilot testing, AMT offers a fully-featured sandbox
environment to build and deploy worker tasks.
x
"Enterprise Crowdsourcing Research Report." Www.crowdsourcing.org. Crowdsourcing LLC, 10 Mar. 2012. Web. 06 June 2012.
<http://www.crowdsourcing.org/editorial/enterprise-crowdsourcing-research-report-by-massolution/11736>.
xi
A significant factor constraining more rapid adoption is the lack of clarity around the employment status of cloud laborers. For more information,
see: Wolfson, Stephen, and Matthew Lease. "Look before You Leap: Legal Pitfalls of Crowdsourcing." Wiley Online Library. John Wiley & Sons, Inc.,
11 Jan. 2012. Web. 06 June 2012. <http://onlinelibrary.wiley.com/doi/10.1002/meet.2011.14504801135/abstract>.
xii
In November 2012, the National Institute of Standards and Technology will be holding its 21st annual Text Retrieval Conference (TREC). For the last
two (2) years, the conference has included a Crowdsourcing track whose focus is on leveraging human relevance judgments to inform / train
automated classification solutions. For more information see: http://trec.nist.gov/.
xiii
Ipeirotis, Panos, and Praveen Paritosh. "Managing Crowdsourced Human Computation." Research at Google. Google, 2011. Web. 06 June 2012.
<http://research.google.com/pubs/pub36946.html>.