Looking at the world through
NLP-colored glasses
Dr. John Feland, CEO, Argus Insights
Sentiment Analysis Symposium
Research Workshop 8 Nov 2011
Tweet it! @argusinsights

If you can not measure it, you can
not improve it.
-Lord Kelvin
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 2

Always Check your math
-my First Grade Teacher

Trust, but Verify
- Russian Proverb
(Доверяй, но проверяй)

Most Corporate Clients Treat Sentiment as a Black Box
Raw musings of Numerical Market
Magic
consumers Sentiment
Sentiment
Analysis
• Corporate Decision Makers blinded by precision
• Using these numbers to drive performance
• What is the impact of getting it wrong?
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 5

How we checked the math
Unnamed
sentiment
provider
• Over 1900 user reviews taken from the Tablet Market
• Sources include most of the major online retailers
• Leveraged best-in-class sentiment provider
– Scored entire verbatim using tuned sentiment model
– Usage is outside of vendor’s recommended use
• Surprised at the results
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 6

Initial compared sentiment scores directly
0.6
Actual Distribution SV Distribution
Percentage Distribution
0.5
0.4
0.3
0.2
0.1
0
-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1
Consumer Sentiment
• All scales normalized (linearly) to -1 to +1
• SV Sentiment Score within Review Range 7% of the time
• SV Sentiment within correct Positive, Neutral or Negative
territory 51% of the time
• SV results are biased towards the negative for Tablets
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 7

Scaled so that SV distro matches User distro
0.6
Actual Distribution SV Distribution
P{ercentage of Distribution
0.5
0.4
0.3
0.2
0.1
0
-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1
Consumer Sentiment
• Trimmed and scaled SV results to match actual
distribution
• Signficant performance improvements
– Correct Bin: 33% -> almost 5X improvement
– Correct Sign: 74% -> almost 50% improvement
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 8

Comparing actual to scored by product
Reviews Raw SV Score Corrected SV Score
1 ASUS Eee Slate RIM PlayBook RIM PlayBook
2 Iconia Andr Tablet iPad 2 iPad 2
3 Motorola Xoom Iconia Andr Tablet Iconia Andr Tablet
4 iPad 2 Samsung Galaxy Tab 7 Motorola Xoom
5 RIM PlayBook Motorola Xoom Samsung Galaxy Tab 7
6 Samsung Galaxy Tab 7 ASUS Eee Slate ASUS Eee Slate
7 Iconia Win Tablet Iconia Win Tablet Iconia Win Tablet
8 HP TouchPad HP TouchPad HP TouchPad
• Calculated sentiment could lead to the wrong
competitive understanding
• Selecting the wrong target can be disastrous for the
enterprise
• Responsibility to our clients demand transparency
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 9

New sentiment models - where the math checks out
• Argus Insights uses customer reported sentiment
• Proven predictor of market performance
• Full transparency to clients, enabling them to trust and verify
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 10

With great power comes
great responsibility
-Voltaire
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 11

Questions?