This course is all about the data mining techniques and how we mine the data and get optimize results.This course is all about the data mining techniques and how we mine the data and get optimize results.This course is all about the data mining techniques and how we mine the data and get optimize results.This course is all about the data mining techniques and how we mine the data and get optimize results.This course is all about the data mining techniques and how we mine the data and get optimize results.This course is all about the data mining techniques and how we mine the data and get optimize results.This course is all about the data mining techniques and how we mine the data and get optimize results
Data preprocessing techniques
See my Paris applied psychology conference paper here
https://www.slideshare.net/jasonrodrigues/paris-conference-on-applied-psychology
or
https://prezi.com/view/KBP8JnekVH9LkLOiKY3w/
Data preprocessing techniques
See my Paris applied psychology conference paper here
https://www.slideshare.net/jasonrodrigues/paris-conference-on-applied-psychology
or
https://prezi.com/view/KBP8JnekVH9LkLOiKY3w/
Data preprocessing techniques are applied before mining. These can improve the overall quality of the patterns mined and the time required for the actual mining.
Some important data preprocessing that must be needed before applying the data mining algorithm to any data sets are completely described in these slides.
This presentation gives the idea about Data Preprocessing in the field of Data Mining. Images, examples and other things are adopted from "Data Mining Concepts and Techniques by Jiawei Han, Micheline Kamber and Jian Pei "
I've shown you in this ppt, the difference between Data and Big Data. How Big Data is generated, Opportunities with Big Data, Problem occurred in Big Data, solution of that problem, Big Data tools, What is Data Science & how it's related with the Big Data, Data Scientist vs Data Analyst. At last, one Real-life scenario where Big data, data scientists, and data analysts work together.
This presentation briefly discusses the following topics:
Classification of Data
What is Structured Data?
What is Unstructured Data?
What is Semistructured Data?
Structured vs Unstructured Data: 5 Key Differences
Data preprocessing techniques are applied before mining. These can improve the overall quality of the patterns mined and the time required for the actual mining.
Some important data preprocessing that must be needed before applying the data mining algorithm to any data sets are completely described in these slides.
This presentation gives the idea about Data Preprocessing in the field of Data Mining. Images, examples and other things are adopted from "Data Mining Concepts and Techniques by Jiawei Han, Micheline Kamber and Jian Pei "
I've shown you in this ppt, the difference between Data and Big Data. How Big Data is generated, Opportunities with Big Data, Problem occurred in Big Data, solution of that problem, Big Data tools, What is Data Science & how it's related with the Big Data, Data Scientist vs Data Analyst. At last, one Real-life scenario where Big data, data scientists, and data analysts work together.
This presentation briefly discusses the following topics:
Classification of Data
What is Structured Data?
What is Unstructured Data?
What is Semistructured Data?
Structured vs Unstructured Data: 5 Key Differences
Date: September 11, 2017
Course: UiS DAT630 - Web Search and Data Mining (fall 2017) (https://github.com/kbalog/uis-dat630-fall2017)
Presentation based on resources from the 2016 edition of the course (https://github.com/kbalog/uis-dat630-fall2016) and the resources shared by the authors of the book used through the course (https://www-users.cs.umn.edu/~kumar001/dmbook/index.php).
Please cite, link to or credit this presentation when using it or part of it in your work.
Data Mining DataLecture Notes for Chapter 2IntroducOllieShoresna
Data Mining: Data
Lecture Notes for Chapter 2
Introduction to Data Mining
by
Tan, Steinbach, Kumar
What is Data?Collection of data objects and their attributes
An attribute is a property or characteristic of an objectExamples: eye color of a person, temperature, etc.Attribute is also known as variable, field, characteristic, or featureA collection of attributes describe an objectObject is also known as record, point, case, sample, entity, or instance
Attributes
Objects
Attribute ValuesAttribute values are numbers or symbols assigned to an attribute
Distinction between attributes and attribute valuesSame attribute can be mapped to different attribute values Example: height can be measured in feet or meters
Different attributes can be mapped to the same set of values Example: Attribute values for ID and age are integers But properties of attribute values can be different
ID has no limit but age has a maximum and minimum value
Types of Attributes There are different types of attributesNominalExamples: ID numbers, eye color, zip codesOrdinalExamples: rankings (e.g., taste of potato chips on a scale from 1-10), grades, height in {tall, medium, short}IntervalExamples: calendar dates, temperatures in Celsius or Fahrenheit.RatioExamples: temperature in Kelvin, length, time, counts
Properties of Attribute Values The type of an attribute depends on which of the following properties it possesses:Distinctness: = Order: < > Addition: + - Multiplication: * /
Nominal attribute: distinctnessOrdinal attribute: distinctness & orderInterval attribute: distinctness, order & additionRatio attribute: all 4 properties
Attribute Type
Description
Examples
Operations
Nominal
The values of a nominal attribute are just different names, i.e., nominal attributes provide only enough information to distinguish one object from another. (=, )
zip codes, employee ID numbers, eye color, sex: {male, female}
mode, entropy, contingency correlation, 2 test
Ordinal
The values of an ordinal attribute provide enough information to order objects. (<, >)
hardness of minerals, {good, better, best},
grades, street numbers
median, percentiles, rank correlation, run tests, sign tests
Interval
For interval attributes, the differences between values are meaningful, i.e., a unit of measurement exists.
(+, - )
calendar dates, temperature in Celsius or Fahrenheit
mean, standard deviation, Pearson's correlation, t and F tests
Ratio
For ratio variables, both differences and ratios are meaningful. (*, /)
temperature in Kelvin, monetary quantities, counts, age, mass, length, electrical current
geometric mean, harmonic mean, percent variation
Attribute Level
Transformation
Comments
Nominal
Any permutation of values
If all employee ID numbers were reassigned, would it make any difference?
Ordinal
An order preserving change of values, i.e.,
new_value = f(old_value)
where f is a monotonic function.
An attribut ...
Data Mining DataLecture Notes for Chapter 2Introduc.docxwhittemorelucilla
Data Mining: Data
Lecture Notes for Chapter 2
Introduction to Data Mining
by
Tan, Steinbach, Kumar
What is Data?Collection of data objects and their attributes
An attribute is a property or characteristic of an objectExamples: eye color of a person, temperature, etc.Attribute is also known as variable, field, characteristic, or featureA collection of attributes describe an objectObject is also known as record, point, case, sample, entity, or instance
Attributes
Objects
Attribute ValuesAttribute values are numbers or symbols assigned to an attribute
Distinction between attributes and attribute valuesSame attribute can be mapped to different attribute values Example: height can be measured in feet or meters
Different attributes can be mapped to the same set of values Example: Attribute values for ID and age are integers But properties of attribute values can be different
ID has no limit but age has a maximum and minimum value
Measurement of Length The way you measure an attribute is somewhat may not match the attributes properties.
Types of Attributes There are different types of attributesNominalExamples: ID numbers, eye color, zip codesOrdinalExamples: rankings (e.g., taste of potato chips on a scale from 1-10), grades, height in {tall, medium, short}IntervalExamples: calendar dates, temperatures in Celsius or Fahrenheit.RatioExamples: temperature in Kelvin, length, time, counts
Properties of Attribute Values The type of an attribute depends on which of the following properties it possesses:Distinctness: = Order: < > Addition: + - Multiplication: * /
Nominal attribute: distinctnessOrdinal attribute: distinctness & orderInterval attribute: distinctness, order & additionRatio attribute: all 4 properties
Attribute Type
Description
Examples
Operations
Nominal
The values of a nominal attribute are just different names, i.e., nominal attributes provide only enough information to distinguish one object from another. (=, )
zip codes, employee ID numbers, eye color, sex: {male, female}
mode, entropy, contingency correlation, 2 test
Ordinal
The values of an ordinal attribute provide enough information to order objects. (<, >)
hardness of minerals, {good, better, best},
grades, street numbers
median, percentiles, rank correlation, run tests, sign tests
Interval
For interval attributes, the differences between values are meaningful, i.e., a unit of measurement exists.
(+, - )
calendar dates, temperature in Celsius or Fahrenheit
mean, standard deviation, Pearson's correlation, t and F tests
Ratio
For ratio variables, both differences and ratios are meaningful. (*, /)
temperature in Kelvin, monetary quantities, counts, age, mass, length, electrical current
geometric mean, harmonic mean, percent variation
Attribute Level
Transformation
Comments
Nominal
Any permutation of values
If all employee ID numbers were reassigned, would it make any difference?
Ordinal
An .
This is good description of student learning at beginner level.we have made it through the UNITY three D. Also using Android studio for this app.small student can learn through this app with ease.
Software Engineering Economics Life Cycle.Sulman Ahmed
Software Engineering Economics Life Cycle.
Software Engineering Economics Life Cycle.
Software Engineering Economics Life Cycle.
Software Engineering Economics Life Cycle.
Software Engineering Economics Life Cycle.
This slide is all about the Data mining techniques.This slide is all about the Data mining techniques.This slide is all about the Data mining techniques.This slide is all about the Data mining techniques;This slide is all about the Data mining techniques;This slide is all about the Data mining techniques.This slide is all about the Data mining techniques.This slide is all about the Data mining techniques
This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.This slide is about Data mining rules.
This slide is about all necessary information about the rules of data mining...
This slide is about all necessary information about the rules of data mining...
This slide is about all necessary information about the rules of data mining...
This slide is about all necessary information about the rules of data mining...
This slide is about all necessary information about the rules of data mining...
This slide is about all necessary information about the rules of data mining...
This course is all about the data mining that how we get the optimized results. it included with all types and how we use these techniques.This course is all about the data mining that how we get the optimized results. it included with all types and how we use these techniques.This course is all about the data mining that how we get the optimized results. it included with all types and how we use these techniques.This course is all about the data mining that how we get the optimized results. it included with all types and how we use these techniques.This course is all about the data mining that how we get the optimized results. it included with all types and how we use these techniques
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Embracing GenAI - A Strategic ImperativePeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
• The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
• The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate “any matter” at “any time” under House Rule X.
• The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
Francesca Gottschalk - How can education support child empowerment.pptxEduSkills OECD
Francesca Gottschalk from the OECD’s Centre for Educational Research and Innovation presents at the Ask an Expert Webinar: How can education support child empowerment?
The Roman Empire A Historical Colossus.pdfkaushalkr1407
The Roman Empire, a vast and enduring power, stands as one of history's most remarkable civilizations, leaving an indelible imprint on the world. It emerged from the Roman Republic, transitioning into an imperial powerhouse under the leadership of Augustus Caesar in 27 BCE. This transformation marked the beginning of an era defined by unprecedented territorial expansion, architectural marvels, and profound cultural influence.
The empire's roots lie in the city of Rome, founded, according to legend, by Romulus in 753 BCE. Over centuries, Rome evolved from a small settlement to a formidable republic, characterized by a complex political system with elected officials and checks on power. However, internal strife, class conflicts, and military ambitions paved the way for the end of the Republic. Julius Caesar’s dictatorship and subsequent assassination in 44 BCE created a power vacuum, leading to a civil war. Octavian, later Augustus, emerged victorious, heralding the Roman Empire’s birth.
Under Augustus, the empire experienced the Pax Romana, a 200-year period of relative peace and stability. Augustus reformed the military, established efficient administrative systems, and initiated grand construction projects. The empire's borders expanded, encompassing territories from Britain to Egypt and from Spain to the Euphrates. Roman legions, renowned for their discipline and engineering prowess, secured and maintained these vast territories, building roads, fortifications, and cities that facilitated control and integration.
The Roman Empire’s society was hierarchical, with a rigid class system. At the top were the patricians, wealthy elites who held significant political power. Below them were the plebeians, free citizens with limited political influence, and the vast numbers of slaves who formed the backbone of the economy. The family unit was central, governed by the paterfamilias, the male head who held absolute authority.
Culturally, the Romans were eclectic, absorbing and adapting elements from the civilizations they encountered, particularly the Greeks. Roman art, literature, and philosophy reflected this synthesis, creating a rich cultural tapestry. Latin, the Roman language, became the lingua franca of the Western world, influencing numerous modern languages.
Roman architecture and engineering achievements were monumental. They perfected the arch, vault, and dome, constructing enduring structures like the Colosseum, Pantheon, and aqueducts. These engineering marvels not only showcased Roman ingenuity but also served practical purposes, from public entertainment to water supply.
Introduction to AI for Nonprofits with Tapp NetworkTechSoup
Dive into the world of AI! Experts Jon Hill and Tareq Monaur will guide you through AI's role in enhancing nonprofit websites and basic marketing strategies, making it easy to understand and apply.
2024.06.01 Introducing a competency framework for languag learning materials ...Sandy Millin
http://sandymillin.wordpress.com/iateflwebinar2024
Published classroom materials form the basis of syllabuses, drive teacher professional development, and have a potentially huge influence on learners, teachers and education systems. All teachers also create their own materials, whether a few sentences on a blackboard, a highly-structured fully-realised online course, or anything in between. Despite this, the knowledge and skills needed to create effective language learning materials are rarely part of teacher training, and are mostly learnt by trial and error.
Knowledge and skills frameworks, generally called competency frameworks, for ELT teachers, trainers and managers have existed for a few years now. However, until I created one for my MA dissertation, there wasn’t one drawing together what we need to know and do to be able to effectively produce language learning materials.
This webinar will introduce you to my framework, highlighting the key competencies I identified from my research. It will also show how anybody involved in language teaching (any language, not just English!), teacher training, managing schools or developing language learning materials can benefit from using the framework.
2. What is Data?
• Collection of data objects and their
attributes
• An attribute is a property or
characteristic of an object
– Examples: eye color of a person,
temperature, etc.
– Attribute is also known as variable,
field, characteristic, or feature
• A collection of attributes describe an
object
– Object is also known as record, point,
case, sample, entity, or instance
Tid Refund Marital
Status
Taxable
Income Cheat
1 Yes Single 125K No
2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
5 No Divorced 95K Yes
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10 No Single 90K Yes
10
Attributes
Objects
3. Attribute Values
• Attribute values are numbers or symbols assigned to
an attribute
• Distinction between attributes and attribute values
– Same attribute can be mapped to different attribute values
• Example: height can be measured in feet or meters
– Different attributes can be mapped to the same set of
values
• Example: Attribute values for ID and age are integers
• But properties of attribute values can be different
– ID has no limit but age has a maximum and minimum value
4. Measurement of Length
• The way you measure an attribute is somewhat may not match the attributes
properties.
1
2
3
5
5
7
8
15
10 4
A
B
C
D
E
5. Types of Attributes
• There are different types of attributes
– Nominal
• Examples: ID numbers, eye color, zip codes
– Ordinal
• Examples: rankings (e.g., taste of potato chips on a scale from 1-
10), grades, height in {tall, medium, short}
– Interval
• Examples: calendar dates, temperatures in Celsius or Fahrenheit.
– Ratio
• Examples: temperature in Kelvin, length, time, counts
6. Properties of Attribute Values
• The type of an attribute depends on which of the
following properties it possesses:
– Distinctness: =
– Order: < >
– Addition: + -
– Multiplication: * /
– Nominal attribute: distinctness
– Ordinal attribute: distinctness & order
– Interval attribute: distinctness, order & addition
– Ratio attribute: all 4 properties
7. Attribute Type Description Examples Operations
Nominal The values of a nominal attribute
are just different names, i.e.,
nominal attributes provide only
enough information to distinguish
one object from another. (=, )
zip codes, employee
ID numbers, eye
color, gender: {male,
female}
mode, entropy,
contingency
correlation, 2 test
Ordinal The values of an ordinal attribute
provide enough information to order
objects. (<, >)
hardness of minerals,
{good, better, best},
grades, street
numbers
median,
percentiles, rank
correlation, run
tests, sign tests
Interval For interval attributes, the
differences between values are
meaningful, i.e., a unit of
measurement exists.
(+, - )
calendar dates,
temperature in Celsius
or Fahrenheit
mean, standard
deviation,
Pearson's
correlation, t and F
tests
Ratio For ratio variables, both differences
and ratios are meaningful. (*, /)
temperature in Kelvin,
monetary quantities,
counts, age, mass,
length, electrical
current
geometric mean,
harmonic mean,
percent variation
8. Attribute
Level
Transformation Comments
Nominal Any permutation of values If all employee ID numbers
were reassigned, would it
make any difference?
Ordinal An order preserving change of values,
i.e.,
new_value = f(old_value)
where f is a monotonic function.
An attribute encompassing the
notion of good, better best can
be represented equally well by
the values {1, 2, 3} or by { 0.5,
1, 10}.
Interval new_value =a * old_value + b where a
and b are constants
Thus, the Fahrenheit and
Celsius temperature scales
differ in terms of where their
zero value is and the size of a
unit (degree).
Ratio new_value = a * old_value Length can be measured in
meters or feet.
9. Discrete and Continuous Attributes
• Discrete Attribute
– Has only a finite or countably infinite set of values
– Examples: zip codes, counts, or the set of words in a collection of documents
– Often represented as integer variables.
– Note: binary attributes are a special case of discrete attributes
• Continuous Attribute
– Has real numbers as attribute values
– Examples: temperature, height, or weight.
– Practically, real values can only be measured and represented using a finite number
of digits.
– Continuous attributes are typically represented as floating-point variables.
10. • Dimensionality
– The dimensionality of a data set is the number of
attributes that the objects in the data set possess.
– Data with a small number of dimensions tends to be
qualitatively different than moderate or high-
dimensional data.
– The difficulties associated with analyzing high-
dimensional data are sometimes referred to as the
curse of dimensionality.
– Because of this, an important motivation in
preprocessing the data is dimensionality reduction.
Important Characteristics of Structured Data
11. • Sparsity
– For some data sets, such as those with asymmetric
features, most attributes of an object have values of 0; in
many cases, fewer than 1% of the entries are non-zero.
– In a database, sparsity and density describe the number
of cells in a table that are empty (sparsity) and that
contain information (density), though sparse cells are not
always technically empty—they often contain a “0” digit.
– Sparsity is an advantage because usually only the non-
zero values need to be stored and manipulated.
Important Characteristics of Structured Data
12. • Resolution
– It is frequently possible to obtain data at different levels of resolution;
and often the properties of the data are different at different resolution.
– For instance, the surface of the Earth seems very uneven at a
resolution of a few meters, but is relatively smooth at a resolution of
tens of kilometers.
– The patterns in the data also depend on the level of resolution,
• if the resolution is too fine, a pattern may not be visible or may be buried in noise;
• if the resolution is too coarse, the pattern may disappear.
– For example, variations in atmospheric pressure on a scale of hours
reflect the movement of storms and other weather systems, whereas,
on a scale of months, such phenomena are not detectable.
Important Characteristics of Structured Data
13. • Record
– Data Matrix
– Document Data
– Transaction Data
• Graph
– World Wide Web
– Molecular Structures
• Ordered
– Spatial Data
– Temporal Data
– Sequential Data
– Genetic Sequence Data
Types of Data Sets
14. Record Data
• Data that consists of a collection of records, each of
which consists of a fixed set of attributes
Tid Refund Marital
Status
Taxable
Income Cheat
1 Yes Single 125K No
2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
5 No Divorced 95K Yes
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10 No Single 90K Yes
10
15. Data Matrix
• If data objects have the same fixed set of numeric attributes, then the data
objects can be thought of as points in a multi-dimensional space, where each
dimension represents a distinct attribute
• Such data set can be represented by an m by n matrix, where there are m
rows, one for each object, and n columns, one for each attribute
• A variation of record data, but because it consists of numeric attributes,
standard matrix operation can be applied to transform and manipulate the
data.
• Therefore, it is the standard format for most statistical data.
1.12.216.226.2512.65
1.22.715.225.2710.23
ThicknessLoadDistanceProjection
of y load
Projection
of x Load
1.12.216.226.2512.65
1.22.715.225.2710.23
ThicknessLoadDistanceProjection
of y load
Projection
of x Load
16. Document Data
• Each document becomes a `term' vector,
– each term is a component (attribute) of the vector,
– the value of each component is the number of times the
corresponding term occurs in the document.
Document 1
season
timeout
lost
wi
n
game
score
ball
pla
y
coach
team
Document 2
Document 3
3 0 5 0 2 6 0 2 0 2
0
0
7 0 2 1 0 0 3 0 0
1 0 0 1 2 2 0 3 0
17. Transaction Data
• A special type of record data, where
– each record (transaction) involves a set of items.
– For example, consider a grocery store. The set of products
purchased by a customer during one shopping trip constitute
a transaction, while the individual products that were
purchased are the items.
TID Items
1 Bread, Coke, Milk
2 Beer, Bread
3 Beer, Coke, Diaper, Milk
4 Beer, Bread, Diaper, Milk
5 Coke, Diaper, Milk
18. Graph Data
• Examples: Generic graph and HTML Links
5
2
1
2
5
<a href="papers/papers.html#bbbb">
Data Mining </a>
<li>
<a href="papers/papers.html#aaaa">
Graph Partitioning </a>
<li>
<a href="papers/papers.html#aaaa">
Parallel Solution of Sparse Linear System of Equations </a>
<li>
<a href="papers/papers.html#ffff">
N-Body Computation and Dense Linear System Solvers
23. Data Quality
• What kinds of data quality problems?
• How can we detect problems with the data?
• What can we do about these problems?
• Examples of data quality problems:
– Noise and outliers
– missing values
– duplicate data
24. Noise
• Noise refers to modification of original values
– Examples: distortion of a person’s voice when talking on a poor phone and “snow” on television
screen
• Techniques from signal/image processing are used to reduce noise.
Two Sine Waves Two Sine Waves + Noise
25. Outliers
• Outliers are data objects with characteristics that are
considerably different than most of the other data
objects in the data set; or
• Values of an attribute that are unusual with respect to
the typical values for that attribute
26. Missing Values
• Reasons for missing values
– Information is not collected
(e.g., people decline to give their age and weight)
– Attributes may not be applicable to all cases
(e.g., annual income is not applicable to children)
• Handling missing values
– Eliminate Data Objects
– Estimate Missing Values
– Ignore the Missing Value During Analysis
– Replace with all possible values (weighted by their
probabilities)
27. Duplicate Data
• Data set may include data objects that are duplicates, or
almost duplicates of one another
– Major issue when merging data from heterogeous sources
• Examples:
– Same person with multiple email addresses
• Data cleaning
– Process of dealing with duplicate data issues
29. Aggregation
• Combining two or more attributes (or objects) into a
single attribute (or object)
• Purpose
– Data reduction
• Reduce the number of attributes or objects
– Change of scale
• Cities aggregated into regions, states, countries, etc
– More “stable” data
• Aggregated data tends to have less variability
30. Aggregation
Standard Deviation of
Average Monthly
Precipitation
Standard Deviation of
Average Yearly Precipitation
Variation of Precipitation in Australia from 1982 to 1993
31. Sampling
• Sampling is the main technique employed for data
selection.
• It is often used for both the preliminary investigation of the
data and the final data analysis.
• Statisticians sample because obtaining the entire set of data
of interest is too expensive or time consuming.
• Sampling is used in data mining because processing the
entire set of data of interest is too expensive or time
consuming.
• In some cases, using a sampling algorithm can reduce the
data size to the point where a better, but more expensive
algorithm can be used.
32. Sampling …
• The key principle for effective sampling is the following:
– using a sample will work almost as well as using the entire
data sets, if the sample is representative
– A sample is representative if it has approximately the same
property (of interest) as the original set of data
• Sampling is a statistical process, the representativeness
of any particular sample will vary, and the best that we
can do is choose a sampling scheme that guarantees
a high probability of getting a representative sample.
33. Types of Sampling
• Simple Random Sampling
– There is an equal probability of selecting any particular item
1. Sampling without replacement
• As each item is selected, it is removed from the population
2. Sampling with replacement
• Objects are not removed from the population as they are selected for the sample.
– In sampling with replacement, the same object can be picked up more than once
• Stratified sampling
– Split the data into several partitions; then draw random
samples from each partition
35. • What sample size is necessary to get at least
one object from each of 10 groups.
Sample Size
36. • When dimensionality increases,
data becomes increasingly
sparse in the space that it
occupies
• Definitions of density and
distance between points, which
is critical for clustering and
outlier detection, become less
meaningful
• Randomly generate 500 points
• Compute difference between max and min
distance between any pair of points
Curse of Dimensionality
37. Dimensionality Reduction
• Techniques to reduce the dimensionality of a data set by
creating new attributes that are a combination of the old
attributes
• Purpose:
– Avoid curse of dimensionality
– Reduce amount of time and memory required by data mining
algorithms
– Lead to a more understandable model
– Allow data to be more easily visualized
– May help to eliminate irrelevant features or reduce noise
• Techniques
– Principle Component Analysis
– Singular Value Decomposition
– Others: supervised and non-linear techniques
38. Dimensionality Reduction: PCA
• Most common approach for dimensionality reduction
• PCA is a linear algebra technique for continuous
attributes that find new attributes (principal components)
that are linear combination of the original attributes
• Goal is to find a projection that captures the largest
amount of variation in data
39. Feature Subset Selection
• Another way to reduce dimensionality of data to use only a
subset of the features
• Redundant features
– duplicate much or all of the information contained in one or more
other attributes
– Example: purchase price of a product and the amount of sales tax
paid
• Irrelevant features
– contain no information that is useful for the data mining task at
hand
– Example: students' ID is often irrelevant to the task of predicting
students' GPA
40. Feature Subset Selection
• Techniques:
– Brute-force approch:
• Try all possible feature subsets as input to data mining algorithm
– Embedded approaches:
• Feature selection occurs naturally as part of the data mining
algorithm
– Filter approaches:
• Features are selected before data mining algorithm is run
– Wrapper approaches:
• Use the data mining algorithm as a black box to find best subset
of attributes
42. Feature Creation
• Create new attributes that can capture the important
information in a data set much more efficiently than the
original attributes
• Three general methodologies:
– Feature Extraction
• domain-specific
– Mapping Data to New Space
– Feature Construction
• combining features
43. Feature Extraction
• The creation of a new set of features from the original
raw data is known as feature extraction.
– For example: a set of photographs, where each photograph
is to be classified according to whether or not it contains
human face. The raw data is a set of pixels and is not
suitable for many types of classification algorithms.
– However, if the data is processed to provide higher level
features, such as the presence or absence of certain types of
edges and areas that are highly correlated with the presence
of human faces, then a much broader set of classification
techniques can be applied to this problem.
43
44. Two Sine Waves Two Sine Waves + Noise Frequency
Fourier transform
Wavelet transform
Mapping Data to a New Space