From Mining Raw Data to Story Visualization

12/12/2018 1Demetris Trihinas
trihinas.d@unic.ac.cy
1Tutorial | TechCamp Cyprus
Department of
Computer Science
Storytelling through Data
From Mining Raw Data to Story
Visualization
Demetris Trihinas
Department of Computer Science
University of Nicosia
Cyprus

Department of
Computer Science
Full-Time Faculty Member
“Developing scalable and self-adaptive tools for data management,
exploration and visualization”
@dtrihinas
http://dtrihinas.info
https://ailab.unic.ac.cy/

Department of
Computer Science
State | Unemployment
------------------------------
NY | 1.72
CA | 2.43
DC | 3.54
…
Raw bits n’ bytes
Structured data
Knowledge
Story
Today’s Talk

Department of
Computer Science
Data Collection
“Taping” into data sources

Department of
Computer Science
Data Collection
• The world’s data sources (e.g., social media, news
outlets) often permit –restricted– access to their data.
• Web Crawling: methodically scrape website content
• Application Programmable Interfaces (APIs)
• “ASK for permission and GET access to resource(s)”
• So… turn the “tap” of a data source (coding task) and store
the data somewhere (data warehousing).

Department of
Computer Science
Web Crawling

Department of
Computer Science
Data Collection via API
Data
Collection
GET access to tweets
You can have 1% for free
with this access token.
The tweet sink
Data
Warehouse
GET tweets
from @dtrihinas
or with #data_miningAlso, ask for
#cyprus and #cyprus

Department of
Computer Science
Twitter Search
Behind the “scenes” is the Twitter API

Department of
Computer Science

Department of
Computer Science
Data Overview
• Trawling through a couple of articles manually is easy.
• But… what about thousands of news articles from
multiple news outlets?
Humans are slow, Computers are fast!
• Get the data, store it and then mine it!

Department of
Computer Science
Big Data refers to datasets that are too large or
complex for traditional data-processing application
software to adequately deal with.

Department of
Computer Science
Big Data is… a Volume Problem

Department of
Computer Science
The Internet’s Digital Footprint

Department of
Computer Science
Sensory Data
Boeing 787 generates
40TB of data per hour
in flight.
Google’s self-driving car
generates 1GB of data
per minute.

Department of
Computer Science
The Internet of Things
21 Billion devices by 2020 accounting for 12% of the digital universe.

Department of
Computer Science
That’s a lot of data!

Department of
Computer Science
Big Data is… a Velocity Problem

Department of
Computer Science
Batch Data
• Assumes that the data is available when and if we want it
(e.g., reading and parsing data from a file or database)
• The application knows the dataset in advance and controls the
input rate of the data.
Count events by color
fetch data
<red, 3>
<yellow, 1>
<blue, 2>
<green, 2>
Application
Database

Department of
Computer Science
• Unbounded Data -> the volume of the data is overwhelming
• Conceptually infinite sequence of data items
• Push Model -> data arrives at high velocity and different rates
• Potentially multiple sources pushing data to the application at
different rates (data distribution changes over time)
Data Streams
Application
src1
src2
src3
0
2
4
input rate
t

Department of
Computer Science
US Presidential Elections 2016
Happiness Anger
Clinton
Trump
Per minute Emotions During First Debate
200K
tweets/min
https://qz.com/810092

Department of
Computer Science
Big Data is… a Value Problem

Department of
Computer Science
Data Mining
From bits and bytes to knowledge

Department of
Computer Science
Data Warehousing
• Data warehousing provides data storage and
management capabilities.
• Memory and storage has
never been cheaper.
1MB today is 10 times
cheaper than 5 years
ago!

Department of
Computer Science
Marketing Mantra
• Collect whatever data you can, whenever and
wherever possible.
• The expectation is that collected data will have value
either for the purpose collected or for a purpose not
yet envisioned.

Department of
Computer Science
Data Mining
• Data is useless unless you can convert it to structured
information and ultimately into knowledge.
• So… data mining provides you with the intelligence to
convert data into knowledge.

Department of
Computer Science
Confluence of Multiple Disciplines

Department of
Computer Science
We are drowning in data
but starved for knowledge…
John Naisbitt, 1982

Department of
Computer Science
What is NOT Data Mining
• Any question you can ask and get an –immediate and
concrete– answer from a database.
• How many sofas models does IKEA currently have in stock?
• How many sofas did IKEA sell in Sweden last month?
• Which IKEA customers bought a sofa worth more than 500
euros this year?

Department of
Computer Science
Data Mining Techniques
• Classification
• Clustering
• Pattern Discovery
• Associations
• Regression
• Outlier Detection

Department of
Computer Science
Classification
• Develop models (or functions) that feature the ability
to distinguish and describe a collection of various
attribute into classes.
• “Give a label to your data!”
• Should the IKEA sofa model S be added to this month’s
discount items (yes, no)?

Department of
Computer Science
Predicting Person’s Credit Worthiness
Attribute
Values
Classes
{Yes, No}

Department of
Computer Science
Google News
Classify
by type
Classify
by country

Department of
Computer Science
Clustering
• Develop models to group data together based on their
similarity or dissimilarity to data in other groups.
• Group IKEA customers based on how much disposable
income they have, or how often they tend to shop at a
particular IKEA branch.
• Similar to classification but with unknown classes.

Department of
Computer Science
Customer Demographics
Customers of this group usually buy sofa S so let’s send to
customer X an email with a discount for S.

Department of
Computer Science
Google News
Similar
articles
clustered
together

Department of
Computer Science
Google News
Article
Clustering
based on
similarity
Cluster
Classification
automated
label
generation

Department of
Computer Science
Pattern Discovery
• One of the most basic techniques in data mining is learning
to recognize patterns in the data.
• This is usually a recognition of some aberration in your data
happening at regular intervals, or an ebb and flow of a
certain variable over time.
• Sales of a certain product seem to spike just before the
holidays, or notice that warmer weather drives more
people to your website.

Department of
Computer Science
IKEA Sofa Sales Forecast
???

Department of
Computer Science
Association
• Association is related to tracking patterns, but is more
specific to dependently linked attributes.
• Model developed to look for specific events or
attributes that are highly correlated with another event
or attribute.
• When your customers buy a specific item, they also
often buy a second, related item.

Department of
Computer Science
People Also…

Department of
Computer Science

Department of
Computer Science
Outlier Detection
• Particular data points do not comply with general
behavior (pattern) of the rest of the data.
• We call them outliers.
• Credit card fraud from
irregular buying patterns
• Patient health from
irregular symptoms

Department of
Computer Science
Regression
• Used primarily as a form of modeling to identify the
likelihood of a certain variable, given the presence of
other variables.
• Project a certain price, based on other factors like
availability, consumer demand, and competition.
• How much should we sell the new IKEA sofa?

Department of
Computer Science
House Price Projection

Department of
Computer Science
Beware…
Data mining is NOT about fitting the model to the answer
YOU want!

Department of
Computer Science
Correlation
• Correlation is a statistical technique that tells us how
strongly pairs of variables are related.
• But… correlation does not tell us the why and how
behind the relationship.
• So… correlation just says that a relationship exists.

Department of
Computer Science
Ice-Cream and Sunglass Sales
As the sales of ice creams is increasing so do
the sales of sunglasses.

Department of
Computer Science
Causation
• Causation denotes that any change in the value of one
variable will cause a change in the value of another
variable.
• This means that one variable makes other to happen.

Department of
Computer Science
Exercise and Calories
• When a person is exercising then the amount of
calories burned increases every minute.
• The former (exercise) is causing the latter (calories
burned) to happen.

Department of
Computer Science
Ice-Cream and Homicides in New York
• A study in the 90’s showed that ice-cream sales are the
cause of homicides in New York.
• As the sales of ice-cream rise and fall, so do the
number of homicides -> correlation.
• But… does the consumption of ice-cream actually
cause the death of people in NY?
https://www.nytimes.com/2009/06/19/nyregion/19murder.html

Department of
Computer Science
Correlation Does NOT Imply Causation
• No… the two things are correlated.
• But this does NOT mean one causes other.
Correlation is something which
we think, when we can’t see
under the covers.
So the less the information we
have the more we are forced
to observe correlations.

Department of
Computer Science
Confidence Intervals
• How many football games do US citizens got to?
• To get an -exact- answer (100% correct), you must ask
everyone in the US (>350M people) -> Not practical!
• Use a random sample, meaning ask (much) less people
-> but we won’t be 100% correct.

Department of
Computer Science
• What we try to achieve: Get an interval that we are
confident that the actual answer lies within.
“I am 95% confident that the number of football games
people in the U.S. go to lies between 10 and 12”
• So basically, CIs describe the level of uncertainty
associated with a sample estimation.

Department of
Computer Science
Random Sample Selection
• Random… means random!
• You cannot just select 1000 people from one city, the
sample wont represent the whole US.
• You cannot just send FB messages to 1000 random
people, you will get a representation of US FB users,
and of course not all of the US citizens use FB.

Department of
Computer Science
Random Sample Distribution
• Without going into a lot of statistics, a perfectly
random sample distribution should look like this:
Assuming that you
actually selected a
random sample

Department of
Computer Science
Random Sample Distribution
• Without going into a lot of statistics, a perfectly
random sample distribution should look like this:
95%

Department of
Computer Science
• Random sample: 1000 US citizens
• Avg is 11 games and SD is 5 games.
• Let’s say we want a 95% confidence interval.
95%
11
With some statistics
we get an interval of
1 game for 95% CI.
We are 95% confident
that the average US
citizen watches between
10-12 games a year.

Department of
Computer Science
Data Visualization
Visually communicate analysis results

Department of
Computer Science
A picture is worth a 1000 words...
Chinese proverb

Department of
Computer Science
Unemployment Data in the US

Department of
Computer Science
Seismic Activity in California

Department of
Computer Science
Why Visualize Your Results?
Easier to interpret large
volumes of data because
the human eye can
immediately focus on
the main information.

Department of
Computer Science

Department of
Computer Science
Interactiveness
Focus even more on information that we care about and we can
perform “real-time” queries on the data.

Department of
Computer Science
Big Data Challenges
The human eye cannot find anymore the information that we
care about…

Department of
Computer Science
Big Data Challenges
Data navigation through interactiveness either does not work
or is not “real-time” anymore…

Department of
Computer Science
Data Science Process
Data
Warehousing
Data
Collection
Data
Mining
Data
Visualization
Insights StoryStruct
Info
Raw
Data

Department of
Computer Science
Data Science Process
Data
Warehousing
Data
Collection
Data
Mining
Data
Visualization
Insights Story
Struct
Info
Raw
Data
Data
Preprocessing
Preprocessed
Info

Department of
Computer Science
Data Preprocessing
• Data mining, especially on big data, is a -compute and
time- expensive process.
• Data Preprocessing can significantly increase
performance if performed before mining.
• Data Cleaning
• Data Reduction
• Data Transformation
Preprocessing can even take around
60% of your effort but totally worth it!

Department of
Computer Science
That’s a lot of data, but…
how much is actually useful!

Department of
Computer Science
Data Cleaning
• You would assume that data stored in a database is
ready for analysis, but… “dirty data”.
• Removing duplicate, erroneous or NA data.
• Statistically imputing missing data.
id name age score
1000
1001
Anna
John
42
fifty
84.7
89.5
age MUST be a number
id name age score
1000
1001
1002
Anna
John
Mat
42
50
29
84.7
89.5
Mat was sick on test day but is C-
average student so lets assume he
would have scored a 72.0

Department of
Computer Science
Data Transformation
• Reshape, sort and combine data to suitable format(s)
for analysis.
id name age score
1000
1001
1002
Anna
John
Mat
42
50
29
84.7
89.7
72.0
id name Eats Breakfast
1000
1001
1002
Anna
John
Mat
Yes
yes
no
id name age score
1001
1000
1002
John
Anna
Mat
50
42
29
90
85
72
Breakfast
1
1
0 Sort
by
score

Department of
Computer Science
Data Reduction
• Perform filtering on the data that is not needed for the
analysis to consume less resources and time.
• Analysis will be performed on US citizens so remove others.
• Use only a sample of the data to get an approximate, but
quick, answer
• Create random sample of 1K rows instead of 1M rows.
• Reduce the dimensionality of the problem
• The field age is not relevant to analysis.

Department of
Computer Science
Kepler.gl
Interactive Maps

Department of
Computer Science
Kepler.gl
Dimensionality reduction through “layering”.

Department of
Computer Science
Kepler.gl
Filter data through “real-time” queries.

Department of
Computer Science
Data Visualization
Putting everything together!

Department of
Computer Science

Department of
Computer Science
Storytelling through Data
From Mining Raw Data to Story
Visualization
Demetris Trihinas
Department of Computer Science
Cyprus

From Mining Raw Data to Story Visualization

More Related Content

What's hot

Similar to From Mining Raw Data to Story Visualization

More from Demetris Trihinas

Recently uploaded

From Mining Raw Data to Story Visualization