Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Crowdsourcing Social Media Scoring
Evaluating An Alternative Approach for Text Analytics
The First Crowdsourcing Project

1], . mmi nx ‘ ‘Hm.  ,, _.ku. ».m H.  n. :.. .1 1).  my 1.:  1). »: Hum!  m  ‘mm:  w, ,.....
Agenda

- Why Crowdsourcing? 

- Our Experiment

- The Results

- Implications for Social Media Intelligence

 

_E4_N_A/ ...
Crowdsourcing vs.  Outsourcing

/ 
- I‘ ' 
, .  Client

ii‘ , /-I
- 
H i ‘I
I,  “I H
l“ l. ‘ L} rile I I 
I V ‘ ‘ii
«A ‘ '...
The Problem with Social Media Data

rs Tremendous amount of unstructured text

6 Investing in software doesn’t always make...
Why Crowdsourcing? 

- Division of labour

- Socializing social media insights

- Overcoming the context gap

5»3>‘”". '7....
Our Experiment

- Telecommunications brand monitor
F 30,000 tweets

 12 brands
 Theme and sentiment coding

- Developed in...
Our Experiment

Contract Customer Service Technical Issues
5 T  4'
I ii’ I‘ it
L'. ',~r 5-, ’ E T» 1 ~9a‘i
Reception Re_co...
Our coders

In‘-‘n

o
2 In-house coders

 IV’

 

7 students 1 academic coordinator
0 O O O O C O C
WMM <9. ‘MII <9;

5 Cr...
Recruitment & Training

- Ads posted on Springtern,  oDesk, 
and Craiglist

   
  

czr
E

E’ - Applicants submitted a res...
The Results - Sentiment Coding

Average Pain/ vise Agreement
n=250

7496 7996
59% I I

All coders vs.  master Top 10 coder...
Sentiment Coding - An Example

,  " @providerB @providerA You guys are HORRIBLE
, ' with customer care.  I'm going with pr...
The Results — Theme Coding

Average Pain/ vise Agreement
n=250

67%
63%

56%

     

All coders vs.  master Top 10 coders ...
Theme Coding - An Example

. , Classly,  @providerA hang up on a costumer
because she gets pissed off at YOUR mistakes. 
D...
The Results - Finding the Right

Benchmark
67"/ 
63% °
53%
All coders vs.  Top 10 coders vs.  Top 5 coders vs. 
master mas...
The Results - Finding the Right
Benchmark

Calibrated Pairwise Agreement

70% 67% 74%

    

62% 63%

    

53%

    

All...
Performance by Topic

Calibrated Pain/ vise Agreement

68%
59%
51%

     

Poor customer service,  Technical/ Coverage Iss...
Performance By Recruitment Group

Average Pairwise Agreement (Sentiment)

80% 73% 80%

74%  E 73%
° 69% 68% ~ 70% 65%

58%...
Operational Learning

- $0.15 cents a code

- Focus the codeframe

- Allow for some collaboration as a means for validatio...
Implications For Future Coding

- Crowdsourcing for codeframe development

- Socializing insights

- A community of coders...
Questions? 

John Crockett
Vice President,  Digital Innovation and Data Management
john. crockett@environics. ca

 

ENVIR...
Upcoming SlideShare
Loading in …5
×

Crowdsourcing social media scoring

629 views

Published on

Crowdsourcing social media scoring

Published in: Marketing
  • Be the first to comment

  • Be the first to like this

Crowdsourcing social media scoring

  1. 1. Crowdsourcing Social Media Scoring Evaluating An Alternative Approach for Text Analytics
  2. 2. The First Crowdsourcing Project 1], . mmi nx ‘ ‘Hm. ,, _.ku. ».m H. n. :.. .1 1). my 1.: 1). »: Hum! m ‘mm: w, ,.. Mm, N A ‘m 4 -‘- K >~M. : in. «mm: <~. H7m. ! Ihrlklud n. n, l,. ,' H-vnul-ad (hum. -m II--LN---‘ II. «mm! I v~>z| v~' - r -I 4. lmzmh Lu, -x. .r~ I «L lmzlwx . ., ! r=, ;hm ~ Ingluh s. .,vI. m lnxinh lwzl-ch l'rv: i-'-h n. ,;1.‘1. Immun v _ ~ _ 1v. n.. m-. H-h<Iluv'. {v. .s‘. ..m», In r . m.n. ~ y-, . . ,.. ,w. , . 1‘n'l. un. n | >.n. »mn Imxuun r'>.1.. u.m Dhhmun | ,‘. ._“, ,V The Oxford T 7 % 7 ‘W’ English Dictionary wit ‘~. I‘ H‘H[‘ --. .x. ._. “. 1 . » m. .m- . ...4.s. . r'——Ba/ .nuk lN‘ | ‘[-tl xx . >M‘R1‘ ENVIRON 'Cs wdsourcing Social Media Scoring 2 RESEARCH
  3. 3. Agenda - Why Crowdsourcing? - Our Experiment - The Results - Implications for Social Media Intelligence _E4_N_A/ _|_| -'’VON ICS
  4. 4. Crowdsourcing vs. Outsourcing / - I‘ ' , . Client ii‘ , /-I - H i ‘I I, “I H l“ l. ‘ L} rile I I I V ‘ ‘ii «A ‘ ' Open Call ‘i Request I ‘ I'll‘ ; _ ’ for Service ”’ ’ for Proposal ‘ ' _; iii . {V | I = E 3;‘ I“/ " INC WW7 72‘) = rlxI=1Ia:1=(r'1=i i7Inw1oil: mI_IIoiln1gi étoiomll mteidlhi auoinnigi 4!
  5. 5. The Problem with Social Media Data rs Tremendous amount of unstructured text 6 Investing in software doesn’t always make sense Value of multiple (human) perspectives :5‘) l_E! ‘.Y‘YI. :k= {.". |l’“'. l:J §1.. [.]n. [gi 5', = rlxI=1Ia:1=(r'1=i
  6. 6. Why Crowdsourcing? - Division of labour - Socializing social media insights - Overcoming the context gap 5»3>‘”". '7.“7‘>"":7!‘* iflroxvivtoikioiuiroflnlgl éioxofiall mteioillai €Y9ml‘lIl! l 3% §, ¥:LIA1:(C’I: i
  7. 7. Our Experiment - Telecommunications brand monitor F 30,000 tweets 12 brands Theme and sentiment coding - Developed instruction kit/ tutorial with example codes - Success? = Eix”V/ ~'v”{j'*"'(7:‘-. ’ ~7Inwo1l: roiI. IIoiln12l ~? tmi’l: .Il mtadltai ~?1omI1Il: I T = rlxI=1Ia:1=(r'1=i
  8. 8. Our Experiment Contract Customer Service Technical Issues 5 T 4' I ii’ I‘ it L'. ',~r 5-, ’ E T» 1 ~9a‘i Reception Re_cos Requests » i_"7.s_, ~, A —-, i'" A Promotions NewsiAlerts Complaints _‘ Y‘. - 1 ' . v — §: §’! ;:; :‘(IC: :fi: i‘pIl(: :;] lfmxvivtoilemuiultnigl ~? ‘l9I9fi, :.l| mteioihai sftomflntgl it I
  9. 9. Our coders In‘-‘n o 2 In-house coders IV’ 7 students 1 academic coordinator 0 O O O O C O C WMM <9. ‘MII <9; 5 Craigslisters 3 Offshore Contractors O“'HuMBi-: R L‘I't1IgSiI5t ENVIRON ICS Crowdsourcing Social Media Scoring 9 RESEARCH
  10. 10. Recruitment & Training - Ads posted on Springtern, oDesk, and Craiglist czr E E’ - Applicants submitted a resume - Project team responded with information package ENVIRON ICS Crowdsourcing Social II. /Iedia Scoring 10 RESEARCH
  11. 11. The Results - Sentiment Coding Average Pain/ vise Agreement n=250 7496 7996 59% I I All coders vs. master Top 10 coders vs. Top 5 coders vs. master master ENVIRONICS RESEARCH
  12. 12. Sentiment Coding - An Example , " @providerB @providerA You guys are HORRIBLE , ' with customer care. I'm going with providerC '« If only all cell phone providers were this accommodating! http: //t. co/ t0Wu6w5e F! ?*‘rv'v: —‘{: i'*"'(: :’~’ Iimwtoitzioiuiuflnlgi ~? YoxofI= _iI mtexoiftai §YHOll'lIl: l ‘P; :1_i=1_wA1:(r‘1=i
  13. 13. The Results — Theme Coding Average Pain/ vise Agreement n=250 67% 63% 56% All coders vs. master Top 10 coders vs. Top 5 coders vs. master master ENVIRONICS RESEARCH
  14. 14. Theme Coding - An Example . , Classly, @providerA hang up on a costumer because she gets pissed off at YOUR mistakes. Done with your phone company. No one use it. , , Worst customer service ever @providerB @providerC hardware sux and services are crap. Fix it or we're gone! FRVFV/ .'~: —‘{: i'*"'(: :’~‘ ‘: IlOViVlI}: {O1I, IlQidIllgI -? 't9m‘§: _|l mteuilhi £1;-io1I'| n1gi ‘L4! :1_i=1_wA1:(r‘1=i
  15. 15. The Results - Finding the Right Benchmark 67"/ 63% ° 53% All coders vs. Top 10 coders vs. Top 5 coders vs. master master master "I"/ ::‘~. ‘ lfuoivivrilaoiuitoflnigl §‘luofl: ,lI lVIt= u‘It: I §Yo-lolflllgl ‘I-, '»
  16. 16. The Results - Finding the Right Benchmark Calibrated Pairwise Agreement 70% 67% 74% 62% 63% 53% All coders vs. Top 10 coders vs. Top 5 coders vs. master master master ENVIRONICS Crowdsourcing Social Media Scoring 16 RESEARCH
  17. 17. Performance by Topic Calibrated Pain/ vise Agreement 68% 59% 51% Poor customer service, Technical/ Coverage Issues Promotions & Special leaving carrier, generally (n=29) Features (n=50) unhappy (n=29) ENVIRONICS RESEARCH
  18. 18. Performance By Recruitment Group Average Pairwise Agreement (Sentiment) 80% 73% 80% 74% E 73% ° 69% 68% ~ 70% 65% 58% 58% 52% 45% 26% 26% 5 8 3 1 6 4 7 2 1 2 3 5 4 3 2 1 0 HUMBER F craigshst ’=3*‘"”—'7~“‘: ‘*""‘7*" «fmv. vzoir. ~I. I-Int: -ui ex-umii mtminai awn-igi 1:1 HA‘ {K 1:!
  19. 19. Operational Learning - $0.15 cents a code - Focus the codeframe - Allow for some collaboration as a means for validation - Does the business action change despite coding conflicts? = Eiv*‘N/ "I”{j'*"I(T: ‘-. ‘ ‘-"IlOlVi'lOfI-: lO1lllQi'IILgl -fmonzll mt= ujlI= i -fitoiioimrgi ‘Isl = rlxI=1Ia:1=(r'1=i
  20. 20. Implications For Future Coding - Crowdsourcing for codeframe development - Socializing insights - A community of coders? r: i:‘I*‘Nf'v: —‘{: i""I(: :’. ’ Iilmwoil-éiolliuflugl ~? Yoioft: _Il Iviteioirni ~f1ouiI'Inigi aw :1_i=1_m1:(r‘1=i
  21. 21. Questions? John Crockett Vice President, Digital Innovation and Data Management john. crockett@environics. ca ENVIRONICS

×