SlideShare a Scribd company logo
1 of 13
Download to read offline
According to the majority rule, with three raters, 100% of the
comments have at least 66.7% agreement (2/3). Our data
shows that when the same comments are evaluated by five
raters, 40.1% (n=117) of the comments have at least 66.7%
agreement. With seven raters, 43.5% (n=127) have at least
66.7% agreement, and for nine raters, the value is 44.9%
(n=131).
0
10
20
30
40
50
60
70
80
90
100
Three Five Seven Nine
Percentage
At least 66.7% agreement
Thanks!
Dr. Joni Salminen
jsalminen@hbku.edu.qa
Qatar Computing Research
Institute

More Related Content

More from Joni Salminen

Determining Online Brand Reputation with Machine Learning from Social Media M...
Determining Online Brand Reputation with Machine Learning from Social Media M...Determining Online Brand Reputation with Machine Learning from Social Media M...
Determining Online Brand Reputation with Machine Learning from Social Media M...Joni Salminen
 
Is More Better?: Impact of Multiple Photos on Perception of Persona Profiles
Is More Better?: Impact of Multiple Photos on Perception of Persona ProfilesIs More Better?: Impact of Multiple Photos on Perception of Persona Profiles
Is More Better?: Impact of Multiple Photos on Perception of Persona ProfilesJoni Salminen
 
Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for...
Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for...Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for...
Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for...Joni Salminen
 
OSS-EBM: Open Source Software Entrepreneurial Business Modelling
OSS-EBM: Open Source Software Entrepreneurial Business ModellingOSS-EBM: Open Source Software Entrepreneurial Business Modelling
OSS-EBM: Open Source Software Entrepreneurial Business ModellingJoni Salminen
 
Gender effect on e-commerce sales of experience gifts: Preliminary empirical ...
Gender effect on e-commerce sales of experience gifts: Preliminary empirical ...Gender effect on e-commerce sales of experience gifts: Preliminary empirical ...
Gender effect on e-commerce sales of experience gifts: Preliminary empirical ...Joni Salminen
 
Tips for Scale Development: Evaluating Automatic Personas
Tips for Scale Development: Evaluating Automatic PersonasTips for Scale Development: Evaluating Automatic Personas
Tips for Scale Development: Evaluating Automatic PersonasJoni Salminen
 
Big Data, Small Personas: Research Agenda for Automatic Persona Generation
Big Data, Small Personas: Research Agenda for Automatic Persona GenerationBig Data, Small Personas: Research Agenda for Automatic Persona Generation
Big Data, Small Personas: Research Agenda for Automatic Persona GenerationJoni Salminen
 
Why do startups avoid difficult problems?
Why do startups avoid difficult problems?Why do startups avoid difficult problems?
Why do startups avoid difficult problems?Joni Salminen
 
Social Espionage: Drawing Benefit from Competitors’ Social Media Presence
Social Espionage: Drawing Benefit from Competitors’ Social Media PresenceSocial Espionage: Drawing Benefit from Competitors’ Social Media Presence
Social Espionage: Drawing Benefit from Competitors’ Social Media PresenceJoni Salminen
 
Strategic Digital Marketing (Digital Marketing '15 @ Oulu University)
Strategic Digital Marketing (Digital Marketing '15 @ Oulu University)Strategic Digital Marketing (Digital Marketing '15 @ Oulu University)
Strategic Digital Marketing (Digital Marketing '15 @ Oulu University)Joni Salminen
 
Social Media Marketing (Digital Marketing '15 @ Oulu University)
Social Media Marketing (Digital Marketing '15 @ Oulu University)Social Media Marketing (Digital Marketing '15 @ Oulu University)
Social Media Marketing (Digital Marketing '15 @ Oulu University)Joni Salminen
 
Search Engine Marketing (Digital Marketing '15 @ Oulu University)
Search Engine Marketing (Digital Marketing '15 @ Oulu University)Search Engine Marketing (Digital Marketing '15 @ Oulu University)
Search Engine Marketing (Digital Marketing '15 @ Oulu University)Joni Salminen
 
Introduction to Digital Marketing (Digital Marketing '15 @ Oulu University)
Introduction to Digital Marketing (Digital Marketing '15 @ Oulu University)Introduction to Digital Marketing (Digital Marketing '15 @ Oulu University)
Introduction to Digital Marketing (Digital Marketing '15 @ Oulu University)Joni Salminen
 
Web Analytics (Digital Marketing '15 @ Oulu University)
Web Analytics (Digital Marketing '15 @ Oulu University)Web Analytics (Digital Marketing '15 @ Oulu University)
Web Analytics (Digital Marketing '15 @ Oulu University)Joni Salminen
 
Generating Cultural Personas From Social Data - A Perspective of Middle Easte...
Generating Cultural Personas From Social Data - A Perspective of Middle Easte...Generating Cultural Personas From Social Data - A Perspective of Middle Easte...
Generating Cultural Personas From Social Data - A Perspective of Middle Easte...Joni Salminen
 
Who does what in marketing? Toward an understanding of marketer–machine inter...
Who does what in marketing? Toward an understanding of marketer–machine inter...Who does what in marketing? Toward an understanding of marketer–machine inter...
Who does what in marketing? Toward an understanding of marketer–machine inter...Joni Salminen
 
Social media marketing for researchers
Social media marketing for researchersSocial media marketing for researchers
Social media marketing for researchersJoni Salminen
 
Text analysis: An introduction to methods and techniques
Text analysis: An introduction to methods and techniquesText analysis: An introduction to methods and techniques
Text analysis: An introduction to methods and techniquesJoni Salminen
 
How has the internet transformed marketing
How has the internet transformed marketingHow has the internet transformed marketing
How has the internet transformed marketingJoni Salminen
 
Konversio-optimointi (2016)
Konversio-optimointi (2016)Konversio-optimointi (2016)
Konversio-optimointi (2016)Joni Salminen
 

More from Joni Salminen (20)

Determining Online Brand Reputation with Machine Learning from Social Media M...
Determining Online Brand Reputation with Machine Learning from Social Media M...Determining Online Brand Reputation with Machine Learning from Social Media M...
Determining Online Brand Reputation with Machine Learning from Social Media M...
 
Is More Better?: Impact of Multiple Photos on Perception of Persona Profiles
Is More Better?: Impact of Multiple Photos on Perception of Persona ProfilesIs More Better?: Impact of Multiple Photos on Perception of Persona Profiles
Is More Better?: Impact of Multiple Photos on Perception of Persona Profiles
 
Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for...
Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for...Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for...
Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for...
 
OSS-EBM: Open Source Software Entrepreneurial Business Modelling
OSS-EBM: Open Source Software Entrepreneurial Business ModellingOSS-EBM: Open Source Software Entrepreneurial Business Modelling
OSS-EBM: Open Source Software Entrepreneurial Business Modelling
 
Gender effect on e-commerce sales of experience gifts: Preliminary empirical ...
Gender effect on e-commerce sales of experience gifts: Preliminary empirical ...Gender effect on e-commerce sales of experience gifts: Preliminary empirical ...
Gender effect on e-commerce sales of experience gifts: Preliminary empirical ...
 
Tips for Scale Development: Evaluating Automatic Personas
Tips for Scale Development: Evaluating Automatic PersonasTips for Scale Development: Evaluating Automatic Personas
Tips for Scale Development: Evaluating Automatic Personas
 
Big Data, Small Personas: Research Agenda for Automatic Persona Generation
Big Data, Small Personas: Research Agenda for Automatic Persona GenerationBig Data, Small Personas: Research Agenda for Automatic Persona Generation
Big Data, Small Personas: Research Agenda for Automatic Persona Generation
 
Why do startups avoid difficult problems?
Why do startups avoid difficult problems?Why do startups avoid difficult problems?
Why do startups avoid difficult problems?
 
Social Espionage: Drawing Benefit from Competitors’ Social Media Presence
Social Espionage: Drawing Benefit from Competitors’ Social Media PresenceSocial Espionage: Drawing Benefit from Competitors’ Social Media Presence
Social Espionage: Drawing Benefit from Competitors’ Social Media Presence
 
Strategic Digital Marketing (Digital Marketing '15 @ Oulu University)
Strategic Digital Marketing (Digital Marketing '15 @ Oulu University)Strategic Digital Marketing (Digital Marketing '15 @ Oulu University)
Strategic Digital Marketing (Digital Marketing '15 @ Oulu University)
 
Social Media Marketing (Digital Marketing '15 @ Oulu University)
Social Media Marketing (Digital Marketing '15 @ Oulu University)Social Media Marketing (Digital Marketing '15 @ Oulu University)
Social Media Marketing (Digital Marketing '15 @ Oulu University)
 
Search Engine Marketing (Digital Marketing '15 @ Oulu University)
Search Engine Marketing (Digital Marketing '15 @ Oulu University)Search Engine Marketing (Digital Marketing '15 @ Oulu University)
Search Engine Marketing (Digital Marketing '15 @ Oulu University)
 
Introduction to Digital Marketing (Digital Marketing '15 @ Oulu University)
Introduction to Digital Marketing (Digital Marketing '15 @ Oulu University)Introduction to Digital Marketing (Digital Marketing '15 @ Oulu University)
Introduction to Digital Marketing (Digital Marketing '15 @ Oulu University)
 
Web Analytics (Digital Marketing '15 @ Oulu University)
Web Analytics (Digital Marketing '15 @ Oulu University)Web Analytics (Digital Marketing '15 @ Oulu University)
Web Analytics (Digital Marketing '15 @ Oulu University)
 
Generating Cultural Personas From Social Data - A Perspective of Middle Easte...
Generating Cultural Personas From Social Data - A Perspective of Middle Easte...Generating Cultural Personas From Social Data - A Perspective of Middle Easte...
Generating Cultural Personas From Social Data - A Perspective of Middle Easte...
 
Who does what in marketing? Toward an understanding of marketer–machine inter...
Who does what in marketing? Toward an understanding of marketer–machine inter...Who does what in marketing? Toward an understanding of marketer–machine inter...
Who does what in marketing? Toward an understanding of marketer–machine inter...
 
Social media marketing for researchers
Social media marketing for researchersSocial media marketing for researchers
Social media marketing for researchers
 
Text analysis: An introduction to methods and techniques
Text analysis: An introduction to methods and techniquesText analysis: An introduction to methods and techniques
Text analysis: An introduction to methods and techniques
 
How has the internet transformed marketing
How has the internet transformed marketingHow has the internet transformed marketing
How has the internet transformed marketing
 
Konversio-optimointi (2016)
Konversio-optimointi (2016)Konversio-optimointi (2016)
Konversio-optimointi (2016)
 

Recently uploaded

FREE NURSING BUNDLE FOR NURSES.PDF by na
FREE NURSING BUNDLE FOR NURSES.PDF by naFREE NURSING BUNDLE FOR NURSES.PDF by na
FREE NURSING BUNDLE FOR NURSES.PDF by naJASISJULIANOELYNV
 
Pests of jatropha_Bionomics_identification_Dr.UPR.pdf
Pests of jatropha_Bionomics_identification_Dr.UPR.pdfPests of jatropha_Bionomics_identification_Dr.UPR.pdf
Pests of jatropha_Bionomics_identification_Dr.UPR.pdfPirithiRaju
 
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptxSOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptxkessiyaTpeter
 
RESPIRATORY ADAPTATIONS TO HYPOXIA IN HUMNAS.pptx
RESPIRATORY ADAPTATIONS TO HYPOXIA IN HUMNAS.pptxRESPIRATORY ADAPTATIONS TO HYPOXIA IN HUMNAS.pptx
RESPIRATORY ADAPTATIONS TO HYPOXIA IN HUMNAS.pptxFarihaAbdulRasheed
 
Is RISC-V ready for HPC workload? Maybe?
Is RISC-V ready for HPC workload? Maybe?Is RISC-V ready for HPC workload? Maybe?
Is RISC-V ready for HPC workload? Maybe?Patrick Diehl
 
Transposable elements in prokaryotes.ppt
Transposable elements in prokaryotes.pptTransposable elements in prokaryotes.ppt
Transposable elements in prokaryotes.pptArshadWarsi13
 
Artificial Intelligence In Microbiology by Dr. Prince C P
Artificial Intelligence In Microbiology by Dr. Prince C PArtificial Intelligence In Microbiology by Dr. Prince C P
Artificial Intelligence In Microbiology by Dr. Prince C PPRINCE C P
 
Forest laws, Indian forest laws, why they are important
Forest laws, Indian forest laws, why they are importantForest laws, Indian forest laws, why they are important
Forest laws, Indian forest laws, why they are importantadityabhardwaj282
 
Pests of castor_Binomics_Identification_Dr.UPR.pdf
Pests of castor_Binomics_Identification_Dr.UPR.pdfPests of castor_Binomics_Identification_Dr.UPR.pdf
Pests of castor_Binomics_Identification_Dr.UPR.pdfPirithiRaju
 
GenBio2 - Lesson 1 - Introduction to Genetics.pptx
GenBio2 - Lesson 1 - Introduction to Genetics.pptxGenBio2 - Lesson 1 - Introduction to Genetics.pptx
GenBio2 - Lesson 1 - Introduction to Genetics.pptxBerniceCayabyab1
 
Solution chemistry, Moral and Normal solutions
Solution chemistry, Moral and Normal solutionsSolution chemistry, Moral and Normal solutions
Solution chemistry, Moral and Normal solutionsHajira Mahmood
 
Call Girls In Nihal Vihar Delhi ❤️8860477959 Looking Escorts In 24/7 Delhi NCR
Call Girls In Nihal Vihar Delhi ❤️8860477959 Looking Escorts In 24/7 Delhi NCRCall Girls In Nihal Vihar Delhi ❤️8860477959 Looking Escorts In 24/7 Delhi NCR
Call Girls In Nihal Vihar Delhi ❤️8860477959 Looking Escorts In 24/7 Delhi NCRlizamodels9
 
Bentham & Hooker's Classification. along with the merits and demerits of the ...
Bentham & Hooker's Classification. along with the merits and demerits of the ...Bentham & Hooker's Classification. along with the merits and demerits of the ...
Bentham & Hooker's Classification. along with the merits and demerits of the ...Nistarini College, Purulia (W.B) India
 
Speech, hearing, noise, intelligibility.pptx
Speech, hearing, noise, intelligibility.pptxSpeech, hearing, noise, intelligibility.pptx
Speech, hearing, noise, intelligibility.pptxpriyankatabhane
 
Harmful and Useful Microorganisms Presentation
Harmful and Useful Microorganisms PresentationHarmful and Useful Microorganisms Presentation
Harmful and Useful Microorganisms Presentationtahreemzahra82
 
BREEDING FOR RESISTANCE TO BIOTIC STRESS.pptx
BREEDING FOR RESISTANCE TO BIOTIC STRESS.pptxBREEDING FOR RESISTANCE TO BIOTIC STRESS.pptx
BREEDING FOR RESISTANCE TO BIOTIC STRESS.pptxPABOLU TEJASREE
 
TOPIC 8 Temperature and Heat.pdf physics
TOPIC 8 Temperature and Heat.pdf physicsTOPIC 8 Temperature and Heat.pdf physics
TOPIC 8 Temperature and Heat.pdf physicsssuserddc89b
 
Call Us ≽ 9953322196 ≼ Call Girls In Lajpat Nagar (Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Lajpat Nagar (Delhi) |Call Us ≽ 9953322196 ≼ Call Girls In Lajpat Nagar (Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Lajpat Nagar (Delhi) |aasikanpl
 

Recently uploaded (20)

FREE NURSING BUNDLE FOR NURSES.PDF by na
FREE NURSING BUNDLE FOR NURSES.PDF by naFREE NURSING BUNDLE FOR NURSES.PDF by na
FREE NURSING BUNDLE FOR NURSES.PDF by na
 
Pests of jatropha_Bionomics_identification_Dr.UPR.pdf
Pests of jatropha_Bionomics_identification_Dr.UPR.pdfPests of jatropha_Bionomics_identification_Dr.UPR.pdf
Pests of jatropha_Bionomics_identification_Dr.UPR.pdf
 
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptxSOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
SOLUBLE PATTERN RECOGNITION RECEPTORS.pptx
 
RESPIRATORY ADAPTATIONS TO HYPOXIA IN HUMNAS.pptx
RESPIRATORY ADAPTATIONS TO HYPOXIA IN HUMNAS.pptxRESPIRATORY ADAPTATIONS TO HYPOXIA IN HUMNAS.pptx
RESPIRATORY ADAPTATIONS TO HYPOXIA IN HUMNAS.pptx
 
Is RISC-V ready for HPC workload? Maybe?
Is RISC-V ready for HPC workload? Maybe?Is RISC-V ready for HPC workload? Maybe?
Is RISC-V ready for HPC workload? Maybe?
 
Transposable elements in prokaryotes.ppt
Transposable elements in prokaryotes.pptTransposable elements in prokaryotes.ppt
Transposable elements in prokaryotes.ppt
 
Artificial Intelligence In Microbiology by Dr. Prince C P
Artificial Intelligence In Microbiology by Dr. Prince C PArtificial Intelligence In Microbiology by Dr. Prince C P
Artificial Intelligence In Microbiology by Dr. Prince C P
 
Forest laws, Indian forest laws, why they are important
Forest laws, Indian forest laws, why they are importantForest laws, Indian forest laws, why they are important
Forest laws, Indian forest laws, why they are important
 
Pests of castor_Binomics_Identification_Dr.UPR.pdf
Pests of castor_Binomics_Identification_Dr.UPR.pdfPests of castor_Binomics_Identification_Dr.UPR.pdf
Pests of castor_Binomics_Identification_Dr.UPR.pdf
 
GenBio2 - Lesson 1 - Introduction to Genetics.pptx
GenBio2 - Lesson 1 - Introduction to Genetics.pptxGenBio2 - Lesson 1 - Introduction to Genetics.pptx
GenBio2 - Lesson 1 - Introduction to Genetics.pptx
 
Solution chemistry, Moral and Normal solutions
Solution chemistry, Moral and Normal solutionsSolution chemistry, Moral and Normal solutions
Solution chemistry, Moral and Normal solutions
 
Call Girls In Nihal Vihar Delhi ❤️8860477959 Looking Escorts In 24/7 Delhi NCR
Call Girls In Nihal Vihar Delhi ❤️8860477959 Looking Escorts In 24/7 Delhi NCRCall Girls In Nihal Vihar Delhi ❤️8860477959 Looking Escorts In 24/7 Delhi NCR
Call Girls In Nihal Vihar Delhi ❤️8860477959 Looking Escorts In 24/7 Delhi NCR
 
Bentham & Hooker's Classification. along with the merits and demerits of the ...
Bentham & Hooker's Classification. along with the merits and demerits of the ...Bentham & Hooker's Classification. along with the merits and demerits of the ...
Bentham & Hooker's Classification. along with the merits and demerits of the ...
 
Speech, hearing, noise, intelligibility.pptx
Speech, hearing, noise, intelligibility.pptxSpeech, hearing, noise, intelligibility.pptx
Speech, hearing, noise, intelligibility.pptx
 
Engler and Prantl system of classification in plant taxonomy
Engler and Prantl system of classification in plant taxonomyEngler and Prantl system of classification in plant taxonomy
Engler and Prantl system of classification in plant taxonomy
 
Hot Sexy call girls in Moti Nagar,🔝 9953056974 🔝 escort Service
Hot Sexy call girls in  Moti Nagar,🔝 9953056974 🔝 escort ServiceHot Sexy call girls in  Moti Nagar,🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Moti Nagar,🔝 9953056974 🔝 escort Service
 
Harmful and Useful Microorganisms Presentation
Harmful and Useful Microorganisms PresentationHarmful and Useful Microorganisms Presentation
Harmful and Useful Microorganisms Presentation
 
BREEDING FOR RESISTANCE TO BIOTIC STRESS.pptx
BREEDING FOR RESISTANCE TO BIOTIC STRESS.pptxBREEDING FOR RESISTANCE TO BIOTIC STRESS.pptx
BREEDING FOR RESISTANCE TO BIOTIC STRESS.pptx
 
TOPIC 8 Temperature and Heat.pdf physics
TOPIC 8 Temperature and Heat.pdf physicsTOPIC 8 Temperature and Heat.pdf physics
TOPIC 8 Temperature and Heat.pdf physics
 
Call Us ≽ 9953322196 ≼ Call Girls In Lajpat Nagar (Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Lajpat Nagar (Delhi) |Call Us ≽ 9953322196 ≼ Call Girls In Lajpat Nagar (Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Lajpat Nagar (Delhi) |
 

Problem of majority voting

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9. According to the majority rule, with three raters, 100% of the comments have at least 66.7% agreement (2/3). Our data shows that when the same comments are evaluated by five raters, 40.1% (n=117) of the comments have at least 66.7% agreement. With seven raters, 43.5% (n=127) have at least 66.7% agreement, and for nine raters, the value is 44.9% (n=131). 0 10 20 30 40 50 60 70 80 90 100 Three Five Seven Nine Percentage At least 66.7% agreement
  • 10.
  • 11.
  • 12.