A Brief Presentation on Data Mining
Jason Rodrigues
Data Mining
• Introduction
• What is Data Mining?
• Challenges and Trends
• The origin of DM
• Data Mining Tasks
• Types of Data
• Dat...
What is Data Mining?
Definition:

Data mining is the process of extracting patterns from
data.- Wikipedia

Process of se...
What is Data Mining? (Other Definitions)

Extracting or “mining” knowledge from large amounts of
data

Data -driven disc...
What is Data Mining?
What is Data Mining?
– Certain names are more
prevalent in certain US
locations (O’Brien, O’Rurke,
O’...
What is Data Mining?
Moore's Law:

The law is named after Intel co-founder Gordon E. Moore, who described the
trend in hi...
Moore's Law:
What is Data Mining?
What is Data Mining? History

Data Analysis?

Data Warehouse?

Data Mining and Statistics
− Standard Deviation

Data M...
Data Mining Challenges and Trends

Commercial Viewpoint
Data Mining Challenges and Trends

Scientific Viewpoint
Data Mining Challenges and Trends
 Computationally expensive to investigate all
possibilities
 Dealing with noise/missin...
Predictive AnalysisPredictive Analysis
Presentation Exploration Discovery
Passive
Interactive
Proactive
Role of Software
B...
Data Mining Tasks

Prediction Methods
− Use some variables to predict unknown or
future values of other variables.

Desc...
Data Mining Tasks

Classification [Predictive]

Clustering [Descriptive]

Association Rule Discovery [Descriptive]

Se...
Data Mining Tasks
 Concept/Class description: Characterization
and discrimination
 Generalize, summarize, and contrast d...
Data Mining Tasks
 Classification and Prediction
 Finding models (functions) that describe and
distinguish classes or co...
Data Mining Tasks

Cluster analysis
− Class label is unknown: Group data to form
new classes,

Example: cluster houses t...
Data Mining Tasks
 Outlier analysis
 Outlier: a data object that does not comply with the
general behavior of the data
...
Takeaways
What is Data Mining?
Challenges and Trends
The origin of DM
Data Mining Tasks
What is Data?

Collection of data objects and
their attributes

An attribute is a property or
characteristic of an objec...
Attribute Values

Attribute values are numbers or symbols assigned to an
attribute

Distinction between attributes and a...
Types of Attribute

There are different types of attributes
– Nominal
 Examples: ID numbers, eye color, zip codes
– Ordi...
Properties of Attribute Values

The type of an attribute depends on which of
the following properties it possesses:
− Dis...
Attribute
Type
Description Examples Operations
Nominal The values of a nominal attribute
are just different names, i.e.,
n...
Discreet and Continuous Attributes

Discrete Attribute
− Has only a finite or countably infinite set of values
− Examples...
Attribute Values

Attribute values are numbers or symbols assigned to an
attribute

Distinction between attributes and a...
Types of Record Sets
Record
− Data Matrix
− Document Data
− Transaction Data
Graph
− World Wide Web
− Molecular Structures...
Record Sets

Data that consists of a collection of
records, each of which consists of a fixed
set of attributes
Tid Refun...
Data Matrics

If data objects have the same fixed set of numeric
attributes, then the data objects can be thought of as
p...
Document Data

Each document becomes a `term' vector,
− each term is a component (attribute) of the
vector,
− the value o...
Transaction Data

A special type of record data, where
− each record (transaction) involves a set of
items.
− For example...
Graph Data

A Generic Graph
5
2
1
2
5
Chemical Data
 Benzene Molecule: C6H6
Ordered Data

Sequences of transactions
An element of
the sequence
Items/Events
Ordered Data

Genomic Sequences
GGTTCCGCCTTCAGCCCCGCGCC
CGCAGGGCCCGCCCCGCGCCGTC
GAGAAGGGCCCGCCTGGCGGGCG
GGGGGAGGCGGGGCCGC...
Ordered Data

Spacio Temporal Data
Average Monthly
Temperature of
land and ocean
Data Quality

What kinds of data quality problems?

How can we detect problems with the
data?

What can we do about the...
Noise

Noise refers to modification of original
values
− Examples: distortion of a person’s voice when
talking on a poor ...
Outliers

Outliers are data objects with
characteristics that are considerably
different than most of the other data
obje...
Missing Values

Reasons for missing values
− Information is not collected
(e.g., people decline to give their age and
wei...
Duplicate Data

Data set may include data objects that are
duplicates, or almost duplicates of one
another
− Major issue ...
Six Rules of Data Quality
1. Data that is not used cannot be correct for very long
2. Data Quality in an information syste...
Takeaways
What is Data Mining?
Challenges and Trends
The origin of DM
Data Mining Tasks
Data Types
Data Quality
Its all about data mining
Upcoming SlideShare
Loading in...5
×

Its all about data mining

451

Published on

Its all about data mining, a brief introduction to the world of data mining.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
451
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Its all about data mining

  1. 1. A Brief Presentation on Data Mining Jason Rodrigues Data Mining
  2. 2. • Introduction • What is Data Mining? • Challenges and Trends • The origin of DM • Data Mining Tasks • Types of Data • Data Quality • Takeaways Agenda
  3. 3. What is Data Mining? Definition:  Data mining is the process of extracting patterns from data.- Wikipedia  Process of semi-automatically analyzing large databases to find patterns that are:  valid: hold on new data with some certainty  novel: non-obvious to the system  useful: should be possible to act on the item  understandable: humans should be able to interpret the pattern Other Definitions:  Non-trivial extraction of implicit, previously unknown and potentially useful information from data  Exploration & analysis, by automatic or semi-automatic means, of large quantities of data in order to discover meaningful patterns
  4. 4. What is Data Mining? (Other Definitions)  Extracting or “mining” knowledge from large amounts of data  Data -driven discovery and modeling of hidden patterns (we never new existed) in large volumes of data  Extraction of implicit, previously unknown and unexpected, potentially extremely useful information from data
  5. 5. What is Data Mining? What is Data Mining? – Certain names are more prevalent in certain US locations (O’Brien, O’Rurke, O’Reilly… in Boston area) – Group together similar documents returned by search engine according to their context (e.g. Amazon rainforest, Amazon.com,) What is not Data Mining? – Look up phone number in phone directory – Query a Web search engine for information about “Amazon”
  6. 6. What is Data Mining? Moore's Law:  The law is named after Intel co-founder Gordon E. Moore, who described the trend in his 1965 paper. The paper noted that number of components in integrated circuits had doubled every year from the invention of the integrated circuit in 1958 until 1965 and predicted that the trend would continue "for at least ten years"  The number of transistors that can be placed inexpensively on an integrated circuit has doubled approximately every two years. The trend has continued for more than half a century and is not expected to stop until 2015 or later.
  7. 7. Moore's Law: What is Data Mining?
  8. 8. What is Data Mining? History  Data Analysis?  Data Warehouse?  Data Mining and Statistics − Standard Deviation  Data Mining and AI and Machine Learning − Identify Possible Heart Attack cases 1980's1980's 1990's1990's 2000's2000's 2010's2010's
  9. 9. Data Mining Challenges and Trends  Commercial Viewpoint
  10. 10. Data Mining Challenges and Trends  Scientific Viewpoint
  11. 11. Data Mining Challenges and Trends  Computationally expensive to investigate all possibilities  Dealing with noise/missing information and errors in data  Choosing appropriate attributes/input representation  Finding the minimal attribute space  Finding adequate evaluation function(s)  Extracting meaningful information  Not overfitting
  12. 12. Predictive AnalysisPredictive Analysis Presentation Exploration Discovery Passive Interactive Proactive Role of Software Business Insight Canned reporting Ad-hoc reporting Online Analytical Processing Data mining Data Mining Challenges and Trends
  13. 13. Data Mining Tasks  Prediction Methods − Use some variables to predict unknown or future values of other variables.  Description Methods − Find human-interpretable patterns that describe the data.
  14. 14. Data Mining Tasks  Classification [Predictive]  Clustering [Descriptive]  Association Rule Discovery [Descriptive]  Sequential Pattern Discovery [Descriptive]  Regression [Predictive]  Deviation Detection [Predictive]
  15. 15. Data Mining Tasks  Concept/Class description: Characterization and discrimination  Generalize, summarize, and contrast data characteristics, e.g., dry vs. wet regions  Association (correlation and causality)  Multi-dimensional or single-dimensional association age(X, “20-29”) ^ income(X, “60-90K”)  buys(X, “TV”)
  16. 16. Data Mining Tasks  Classification and Prediction  Finding models (functions) that describe and distinguish classes or concepts for future prediction  Example: classify countries based on climate, or classify cars based on gas mileage  Presentation:  If-THEN rules, decision-tree, classification rule, neural network  Prediction: Predict some unknown or missing numerical values
  17. 17. Data Mining Tasks  Cluster analysis − Class label is unknown: Group data to form new classes,  Example: cluster houses to find distribution patterns − Clustering based on the principle: maximizing the intra-class similarity and minimizing the interclass similarity
  18. 18. Data Mining Tasks  Outlier analysis  Outlier: a data object that does not comply with the general behavior of the data  Mostly considered as noise or exception, but is quite useful in fraud detection, rare events analysis  Trend and evolution analysis  Trend and deviation: regression analysis  Sequential pattern mining, periodicity analysis
  19. 19. Takeaways What is Data Mining? Challenges and Trends The origin of DM Data Mining Tasks
  20. 20. What is Data?  Collection of data objects and their attributes  An attribute is a property or characteristic of an object − Examples: eye color of a person, temperature, etc. − Attribute is also known as variable, field, characteristic, or feature  A collection of attributes describe an object − Object is also known as record, point, case, sample, entity, or instance Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes 10 Attributes Objects
  21. 21. Attribute Values  Attribute values are numbers or symbols assigned to an attribute  Distinction between attributes and attribute values − Same attribute can be mapped to different attribute values  Example: height can be measured in feet or meters − Different attributes can be mapped to the same set of values  Example: Attribute values for ID and age are integers  But properties of attribute values can be different − ID has no limit but age has a maximum and minimum value
  22. 22. Types of Attribute  There are different types of attributes – Nominal  Examples: ID numbers, eye color, zip codes – Ordinal  Examples: rankings (e.g., taste of potato chips on a scale from 1-10), grades, height in {tall, medium, short} – Interval  Examples: calendar dates, temperatures in Celsius or Fahrenheit. – Ratio  Examples: temperature in Kelvin, length, time, counts
  23. 23. Properties of Attribute Values  The type of an attribute depends on which of the following properties it possesses: − Distinctness: = ≠ − Order: < > − Addition: + - − Multiplication: * / − Nominal attribute: distinctness − Ordinal attribute: distinctness & order − Interval attribute: distinctness, order & addition − Ratio attribute: all 4 properties
  24. 24. Attribute Type Description Examples Operations Nominal The values of a nominal attribute are just different names, i.e., nominal attributes provide only enough information to distinguish one object from another. (=, ≠) zip codes, employee ID numbers, eye color, sex: {male, female} mode, entropy, contingency correlation, χ2 test Ordinal The values of an ordinal attribute provide enough information to order objects. (<, >) hardness of minerals, {good, better, best}, grades, street numbers median, percentiles, rank correlation, run tests, sign tests Interval For interval attributes, the differences between values are meaningful, i.e., a unit of measurement exists. (+, - ) calendar dates, temperature in Celsius or Fahrenheit mean, standard deviation, Pearson's correlation, t and F tests Ratio For ratio variables, both differences and ratios are meaningful. (*, /) temperature in Kelvin, monetary quantities, counts, age, mass, length, electrical current geometric mean, harmonic mean, percent variation
  25. 25. Discreet and Continuous Attributes  Discrete Attribute − Has only a finite or countably infinite set of values − Examples: zip codes, counts, or the set of words in a collection of documents − Often represented as integer variables. − Note: binary attributes are a special case of discrete attributes  Continuous Attribute − Has real numbers as attribute values − Examples: temperature, height, or weight. − Practically, real values can only be measured and represented using a finite number of digits. − Continuous attributes are typically represented as floating-point variables.
  26. 26. Attribute Values  Attribute values are numbers or symbols assigned to an attribute  Distinction between attributes and attribute values − Same attribute can be mapped to different attribute values  Example: height can be measured in feet or meters − Different attributes can be mapped to the same set of values  Example: Attribute values for ID and age are integers  But properties of attribute values can be different − ID has no limit but age has a maximum and minimum value
  27. 27. Types of Record Sets Record − Data Matrix − Document Data − Transaction Data Graph − World Wide Web − Molecular Structures Ordered − Spatial Data − Temporal Data − Sequential Data − Genetic Sequence Data
  28. 28. Record Sets  Data that consists of a collection of records, each of which consists of a fixed set of attributes Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes 10
  29. 29. Data Matrics  If data objects have the same fixed set of numeric attributes, then the data objects can be thought of as points in a multi-dimensional space, where each dimension represents a distinct attribute  Such data set can be represented by an m by n matrix, where there are m rows, one for each object, and n columns, one for each attribute
  30. 30. Document Data  Each document becomes a `term' vector, − each term is a component (attribute) of the vector, − the value of each component is the number of times the corresponding term occurs in the document.
  31. 31. Transaction Data  A special type of record data, where − each record (transaction) involves a set of items. − For example, consider a grocery store. The set of products purchased by a customer during one shopping trip constitute a transaction, while the individual products that were purchased are the items. TID Items 1 Bread, Coke, Milk 2 Beer, Bread 3 Beer, Coke, Diaper, Milk 4 Beer, Bread, Diaper, Milk 5 Coke, Diaper, Milk
  32. 32. Graph Data  A Generic Graph 5 2 1 2 5
  33. 33. Chemical Data  Benzene Molecule: C6H6
  34. 34. Ordered Data  Sequences of transactions An element of the sequence Items/Events
  35. 35. Ordered Data  Genomic Sequences GGTTCCGCCTTCAGCCCCGCGCC CGCAGGGCCCGCCCCGCGCCGTC GAGAAGGGCCCGCCTGGCGGGCG GGGGGAGGCGGGGCCGCCCGAGC CCAACCGAGTCCGACCAGGTGCC CCCTCTGCTCGGCCTAGACCTGA GCTCATTAGGCGGCAGCGGACAG GCCAAGTAGAACACGCGAAGCGC TGGGCTGCCTGCTGCGACCAGGG
  36. 36. Ordered Data  Spacio Temporal Data Average Monthly Temperature of land and ocean
  37. 37. Data Quality  What kinds of data quality problems?  How can we detect problems with the data?  What can we do about these problems?  Examples of data quality problems: − Noise and outliers − missing values − duplicate data
  38. 38. Noise  Noise refers to modification of original values − Examples: distortion of a person’s voice when talking on a poor phone and “snow” on television screen Two Sine Waves Two Sine Waves + Noise
  39. 39. Outliers  Outliers are data objects with characteristics that are considerably different than most of the other data objects in the data set
  40. 40. Missing Values  Reasons for missing values − Information is not collected (e.g., people decline to give their age and weight) − Attributes may not be applicable to all cases (e.g., annual income is not applicable to children)  Handling missing values − Eliminate Data Objects − Estimate Missing Values − Ignore the Missing Value During Analysis − Replace with all possible values (weighted by their probabilities)
  41. 41. Duplicate Data  Data set may include data objects that are duplicates, or almost duplicates of one another − Major issue when merging data from heterogeous sources  Examples: − Same person with multiple email addresses  Data cleaning − Process of dealing with duplicate data issues
  42. 42. Six Rules of Data Quality 1. Data that is not used cannot be correct for very long 2. Data Quality in an information system is a function of its use, not its collection 3.Data quality will ultimately be no better than its most stringent use 4. Data quality problems tend to become worse with the age of the system 5. Less likely it is that some data element will change, more traumatic it will be when it finally does change. 6. Information overload affects data quality
  43. 43. Takeaways What is Data Mining? Challenges and Trends The origin of DM Data Mining Tasks Data Types Data Quality

×