Your SlideShare is downloading. ×
When the Conclusion Isn’t Supported by the Methodology (and why RateStars is utter nonsense)
When the Conclusion Isn’t Supported by the Methodology (and why RateStars is utter nonsense)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

When the Conclusion Isn’t Supported by the Methodology (and why RateStars is utter nonsense)

117

Published on

Sure, we all know to question whether a conclusion is supported by the data. But what about the data collection methodology itself? We sometimes have cases where the data and the conclusion seem to …

Sure, we all know to question whether a conclusion is supported by the data. But what about the data collection methodology itself? We sometimes have cases where the data and the conclusion seem to match, but a closer look and you realize the methodology was fundamentally unable to support the conclusion. Today’s case in point: RateStars.

Published in: Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
117
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. When the Conclusion Isn’t Supported by the Methodology (and Why RateStars is Utter Nonsense)Sure, we all know to question whether a conclusion is supported by the data. But what about thedata collection methodology itself? We sometimes have cases where the data and the conclusionseem to match, but a closer look and you realize the methodology was fundamentally unable tosupport the conclusion. Today’s case in point: RateStars.RateStars is a new website that promises to identify “The Top 100 in any Industry.” Sounds cool,right? Being on that list would be a sign of prestige. If I was looking for a consultant in a specificfield, I could now find the top 100 to choose from, right?Not so fast.First, let’s step back. If I wanted to create a list of the top 100 market researchers, how would Iaccomplish this?  A survey of peers?  Nominations by an esteemed council of recognized experts?  A calculation based on number of magazine/journal articles published in the past 5 years with points for public speaking, books published, and awards?Any of those methods would result in a list that could be credibly called “the Top 100 marketresearchers.”So, how does RateStars do it? Well, I looked. And I know this may be shocking, but they didn’tuse any of the methods I thought of.Here is how RateStars works:  First, the candidates have to allow it to access their LinkedIn account (or install the Facebook app), which I did temporarily only so I could peek under the hood.  The candidate nominates themselves to be on the list by creating a profile and selecting what category they want to be listed in.  The candidate then asks their friends and connections to rate them using a scale of not just stars, not just smileys, but stars AND smileys (yes, really). 1
  • 2.  Reviewers can rate anonymously but must sign in for the review to be published. Reviewers must also offer at least 20 words about the candidate.  The candidates are now listed in the top 100 of their category, in order of number of ratings.So unless more than 100 people ask to be listed, a candidate is guaranteed a place on the list.And because there are many, many related categories, a candidate could place themselves onmany lists—becoming a bigger fish in a smaller pond in order to gain a more desirable position.Within Market Research, a niche to begin with, there are over 40 subcategories.So what would a reader conclude? Joe Schmo is on the Top 100 list of “Market Research ProjectDirectors”? And is in the top 10 of “Project Coordinators”? Really?Don’t Trust the Methodology? Don’t Trust the ResultsSo to be precise, RateStars does not identify the top 100 in any industry. It identifies the top 100people who want to be listed and who took the time to campaign with their friends andconnections and who have friends willing to connect to applications and who have friends whobuy in to this methodology (or are too nice to say no). It is just another online popularity contest.There is nothing wrong with a popularity contest. But to claim it is anything more is just silly.[Apologies to the friends who have asked me to rate them on RateStars; I don’t mean to be harsh,just honest. Frankly, everyone who has asked me is a great researcher; your bios alreadydemonstrate this.][UPDATE: Just heard from a friend who was phished by RateStars; his LinkedIn account washijacked to send out ratings requests without his knowledge. Since I signed in to the site to peekat the details, I am hoping I did not inadvertently trigger the same treatment!]Author Bio:For more expert tips on market research subscribe to Research Rockstars free newsletterat http://www.researchrockstar.com.Have a market research question? Kathryn Korostoff is president of Research Rockstar, acompany that delivers online and in-person training to busy professionals seeking marketresearch excellence. Kathryn is a market research professional with a special interest in howorganizations acquire, manage, and apply market research. Over the past 20 years, she haspersonally directed more than 600 primary market research projects and published over 100bylined articles in trade magazines. http://twitter.com/ResearchRocks 2

×