Data Discovery and BI - Is there Really a Difference?
IIML Talk_23012016
1. A talk delivered at
IIM Lucknow
24/01/2016
EXTRACT. LOAD. VISUALIZE
Becoming a data ninja
2.
3.
4. • INTRODUCTION
• WEB CRAWLING – THE DESIGN
• DIY – WEB CRAWLING
• TEXT ANALYTICS– THE DESIGN
• CASE STUDY
• DIY – TEXT ANALYTICS
• SOME UNSOLICITED ADVICE? – BUILDING YOUR FIRM
WE WILL TALK OF….
5. Who are we ?
We are a data science company, founded in 2009– with special interest in making the
world an intelligent place to live in.
We identify data and bring it to light, making it visible, cohesive, comparable and easy to
understand so that it really does support YOU in making the right decisions.
Who am I ?
I am a Practice Lead at JSM for Natural Language Processing & Machine Learning. I have
architected multiple solutions in the area of text analytics for multiple industries like finance,
healthcare, food & beverages & hospitality.
6. AREAS WE WORK ON
PHARMA
Sales Pitch Analysis
RETAIL
Predictive + IoT
FINANCE
Competitive
Intelligence
F&B
Customer Insights
MR
Scoping and Product
Evaluation
SaaS
NLP, ML, Text
8. TOOLS WE PLAY WITH
OPEN SOURCE
Inexpensive
DATABASES
Fast & Scalable
INSIGHTS
Python, R
TECHNIQUES
Latest yet tested
VISUALIZATIONS
D3, GCharts, Tableau
MANAGEMENT
Basecamp
12. • HTML pages are like textbooks – content, titles, subtitles, paragraphs and so on
•Javascript adds interactivity to the HTML pages
HTML PAGES
13. A crawler is a program that visits Web sites and reads their pages and other information in order to create
entries for a search engine index.
The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot."
OVERVIEW– WEB CRAWLING
17. • BEST of the lot
• Gives great flexibility – just click and extract
• Most of the sites are compatible
• Easy CSV/Google Docs Export
• Provides APIs for regular data updates
• Low training time
INTRODUCTION
18. USE CASES
You want to monitor feedback
http://www.consumercomplaints.in/snapdeal-com-b100038
22. • Works where Import.io fails
• Bit buggy, but does a god job of providing flexible choices of data extraction
• Most of the sites are compatible
• Easy CSV Export
• NO APIs for regular data updates
• Moderate learning curve
INTRODUCTION
24. • Don’t scrape too fast or you will get banned
• Respect robots.txt
• Extract only what you need
• Don’t overload their servers
• Don’t take data what’s not yours – only the data in public domain
PRECAUTIONS
26. As a brand owner with significant investments in social media, the
usual questions you might have in mind…
• Is the brand exuding same attributes I intended it to be?
• Is my internet presence helping me ?
• Can I measure my ROI for the money I spent?
• What are the measurable metrics for effective social media
management?
• When can I exploit emerging trends for my brand?
• How can I understand my customers better?
26
WHAT WOULD YOU WANT?
27. 27
WHAT WOULD YOU WANT TO TARGET?
TRANSACTIONAL
CONVERSATIONS
Users talk about
current, events,
share cat videos
and engage in
trivial gossip
INFORMATIONAL
CONVERSTIONS
Users engage with
the brand to air
appreciation or
complaints.
29. For a brand, say X, there would be thousands of conversations on blogs,
websites and social media revolving around X.
Several hundred blog posts are published that contain the keywords “X” This
company needs to know:
- How many of these posts are relevant and actually expressing opinions
about X?
- How many relevant posts are negative, and how many are positive?
- What particular aspects and features of X are being praised or criticized?
- For all of the above, what is the trend for the past few time periods?
- How is my brand perceived among people demographics?
- How is my brand faring against my competitors?
29
QUESTIONS TO ANSWER
30. BRIEF TERMINOLOGY
You build an algorithm, machine learns patterns, machine predicts, rinse & repeat.
MACHINE LEARNING
TEXT ANALYTICS
Analyzing unstructured text, assign structure, load into a BI/program to visualize
32. PROBLEM STATEMENT
The client had thousands of customer reviews which they wanted to analyse - to
understand customer feedback and identify improvement opportunities.
The broad questions we focused on;
What did they say about the
restaurant?
Keywords & topics of discussion across
the comments
What elements of the restaurant would
they want improved? – service, staff
behaviour, ambience etc.
When did the customer visit the store?
How is client’s traffic distributed over
time?
Ticket sizes across multiple customer
dimensions – age, gender, ratings,
location, time of visit etc.
Overall customer sentiments & views
about UCH
PRIMARY FOCUS AREAS SECONDARY FOCUS AREAS
34. APPROACH
Extract data
and validate
Corpus from
social media
Tokenise and
remove stop
words
Initiate ML models ,
NER , parsers &
topic algorithms
Initiate detection rules
for topics, keywords,
gender, sentiment and
multi-word concept
detection
Final Output
PRE - PROCESSING PARSING &
ANALYSIS
OUTPUT
Part of
Speech (POS)
Tagger
35. Author Value Type Sentiment
Duncan Riley Samsung Galaxy S6 Entity Positive
Duncan Riley Apple Entity Negative
Duncan Riley LoopPay Entity Neutral
Duncan Riley mobile payments Keyword Positive
Duncan Riley point of sales Keyword Positive
Structuring data from free flowing text is easy to use by existing reporting and business intelligence software.
Insights from the final reports can now be used for decision-making by the PR firm and their client
Actual blog post parsed through our
SmartText Engine
56. • Purse strings WILL be tightened
• Fewer ‘unicorn-level’ valuations
• No investment without revenue or clients
• MVP with traction – a must have
• Acquire or be acquired – or be a acquisition target
• Needs > Wants – solve a problem, but scope it out
•Bootstrapped startups will win, not the valuation-hungry
PREDICTIONS
57. Don't sell what you do - disguise it
Ex. Create marketing strategy, not analytics/Tableau
Nugget 1
58. A product that promotes laziness or transfers laziness from
seller to buyer will sell
Nugget 2
59. Is it something that Google/FB provides or can do? Even if it's
partial, danger is real.
Nugget 3
60. The product should be IP-able, scalable and monetize-able -
atleast 2/3 must be met
Nugget 4
