Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Machine Learning Feature Selection - Random Forest Rupak Roy
Insights about feature selection and the variable importance using gini/information gain and variance for regression.
Let me know if anything is required. Happy to help, Talk soon! #bobrupakroy
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Machine Learning Feature Selection - Random Forest Rupak Roy
Insights about feature selection and the variable importance using gini/information gain and variance for regression.
Let me know if anything is required. Happy to help, Talk soon! #bobrupakroy
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Data preprocessing techniques are applied before mining. These can improve the overall quality of the patterns mined and the time required for the actual mining.
Some important data preprocessing that must be needed before applying the data mining algorithm to any data sets are completely described in these slides.
Naive Bayes is a kind of classifier which uses the Bayes Theorem. It predicts membership probabilities for each class such as the probability that given record or data point belongs to a particular class.
Data preprocessing techniques are applied before mining. These can improve the overall quality of the patterns mined and the time required for the actual mining.
Some important data preprocessing that must be needed before applying the data mining algorithm to any data sets are completely described in these slides.
Naive Bayes is a kind of classifier which uses the Bayes Theorem. It predicts membership probabilities for each class such as the probability that given record or data point belongs to a particular class.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Simple math for anomaly detection toufic boubez - metafor software - monito...tboubez
This is my presentation at Monitorama PDX in Portland on May 5, 2014
Simple math to get some signal out of your noisy sea of data
You’ve instrumented your system and application to the hilt. You can now “measure all the things”. Your team has set up thousands of metrics collecting millions of data points a day. Now what?
Most IT ops teams only keep an eye on a small fraction of the metrics they collect because analyzing this mountain of data and extracting signal from the noise is not easy. The choice of what analytic method to use ranges from simple statistical analysis to sophisticated machine learning techniques. And one algorithm doesn’t fit all data.
Making sense of citizen science data: A review of methodsolivier gimenez
My talk at International Congress for Conservation Biology 2015, in Montpellier.
Data collected through citizen science programs allow addressing many important questions in conservation biology related, e.g., to the shift in species range, the ecology of infectious disease or the effects of habitat loss and fragmentation on biodiversity. However, citizen science data are subject to serious statistical challenges when it comes to their analysis and the reliable extraction of the information they contain, mainly due to sampling biases generated by variation in the observation process. Numerous methods have been proposed to address this issue that can be split into two main strategies: either a new approach is developed to deal with a specific problem or an existing approach is used pending some pre-treatment of the data or post-processing of the results. I review these various methods, trying to make the links between them and emphasizing their advantages and drawbacks with respect to the question. I illustrate my talk with case studies drawn for the research conducted in our group, mainly on large carnivores. Based on this review, I end up this contribution by recommendations on the use of existing methods and by suggesting perspectives on future developments.
A talk given by Eugene Dubossarsky on predictive analytics at the Big Data Analytics meetup in Sydney this month. The talk is available at http://www.youtube.com/watch?v=aG16YSFgtLY
Why are anomalies important? Because they tell us a different story from the norm. An anomaly or an event might signify a failing heart rate of a patient, a fraudulent credit card activity, or an early indication of a tsunami. As such, it is extremely important to detect anomalies or anomalous events.
In this talk, we will give an introduction to anomaly detection. Anomalies are rare events. As a result, standard accuracy measures do not apply. But then, how do we evaluate an Anomaly Detection (AD) method? If we want to compare two or more AD methods, what kind of simple tests can we do? What are the data repositories that are available for AD?
We will also discuss an ensemble method for AD. Constructing an AD ensemble is challenging because the class labels are not known. We will look at an unusual ally from psychometrics – Item Response Theory – to help us in this construction.
Pitfalls of multivariate pattern analysis(MVPA), fMRI Emily Yunha Shin
Two papers review:
* (Part of) A primer on pattern-based approaches to fMRI: principles, pitfalls, and perspectives
* The impact of study design on pattern estimation for single-trial multivariate pattern analysis
Presentation to the third LIS DREaM workshop, held at Edinburgh Napier university on Wednesday 25th April 2012.
More information about the event can be found at http://lisresearch.org/dream-project/dream-event-4-workshop-wednesday-25-april-2012/
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...tboubez
This is my presentation from LISA 2014 in Seattle on November 14, 2014.
Most IT Ops teams only keep an eye on a small fraction of the metrics they collect because analyzing this haystack of data and extracting signal from the noise is not easy and generates too many false positives.
In this talk I will show some of the types of anomalies commonly found in dynamic data center environments and discuss the top 5 things I learned while building algorithms to find them. You will see how various Gaussian based techniques work (and why they don’t!), and we will go into some non-parametric methods that you can use to great advantage.
The use of data and its modelling in science provides meaningful interpretation of real world problems. This presentation provides an easy to understand overview of data visualization and analytics , and snippets of data science applications using R - programming.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
DMTM Lecture 13 Representative based clusteringPier Luca Lanzi
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.
The simplified electron and muon model, Oscillating Spacetime: The Foundation...RitikBhardwaj56
Discover the Simplified Electron and Muon Model: A New Wave-Based Approach to Understanding Particles delves into a groundbreaking theory that presents electrons and muons as rotating soliton waves within oscillating spacetime. Geared towards students, researchers, and science buffs, this book breaks down complex ideas into simple explanations. It covers topics such as electron waves, temporal dynamics, and the implications of this model on particle physics. With clear illustrations and easy-to-follow explanations, readers will gain a new outlook on the universe's fundamental nature.
Thinking of getting a dog? Be aware that breeds like Pit Bulls, Rottweilers, and German Shepherds can be loyal and dangerous. Proper training and socialization are crucial to preventing aggressive behaviors. Ensure safety by understanding their needs and always supervising interactions. Stay safe, and enjoy your furry friends!
This slide is special for master students (MIBS & MIFB) in UUM. Also useful for readers who are interested in the topic of contemporary Islamic banking.
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...Dr. Vinod Kumar Kanvaria
Exploiting Artificial Intelligence for Empowering Researchers and Faculty,
International FDP on Fundamentals of Research in Social Sciences
at Integral University, Lucknow, 06.06.2024
By Dr. Vinod Kumar Kanvaria
A review of the growth of the Israel Genealogy Research Association Database Collection for the last 12 months. Our collection is now passed the 3 million mark and still growing. See which archives have contributed the most. See the different types of records we have, and which years have had records added. You can also see what we have for the future.
Safalta Digital marketing institute in Noida, provide complete applications that encompass a huge range of virtual advertising and marketing additives, which includes search engine optimization, virtual communication advertising, pay-per-click on marketing, content material advertising, internet analytics, and greater. These university courses are designed for students who possess a comprehensive understanding of virtual marketing strategies and attributes.Safalta Digital Marketing Institute in Noida is a first choice for young individuals or students who are looking to start their careers in the field of digital advertising. The institute gives specialized courses designed and certification.
for beginners, providing thorough training in areas such as SEO, digital communication marketing, and PPC training in Noida. After finishing the program, students receive the certifications recognised by top different universitie, setting a strong foundation for a successful career in digital marketing.
MATATAG CURRICULUM: ASSESSING THE READINESS OF ELEM. PUBLIC SCHOOL TEACHERS I...NelTorrente
In this research, it concludes that while the readiness of teachers in Caloocan City to implement the MATATAG Curriculum is generally positive, targeted efforts in professional development, resource distribution, support networks, and comprehensive preparation can address the existing gaps and ensure successful curriculum implementation.
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
• The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
• The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate “any matter” at “any time” under House Rule X.
• The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
it describes the bony anatomy including the femoral head , acetabulum, labrum . also discusses the capsule , ligaments . muscle that act on the hip joint and the range of motion are outlined. factors affecting hip joint stability and weight transmission through the joint are summarized.
Introduction to AI for Nonprofits with Tapp NetworkTechSoup
Dive into the world of AI! Experts Jon Hill and Tareq Monaur will guide you through AI's role in enhancing nonprofit websites and basic marketing strategies, making it easy to understand and apply.
5. Prof. Pier Luca Lanzi
Data Representation
• Data are typically abstracted as a matrix, with n rows and d
columns, given as
• Rows are called instances, examples, records, transactions,
objects, points, feature-vectors, etc.
• Columns are called attributes, properties, features, dimensions,
variables, fields, etc.
5
7. Prof. Pier Luca Lanzi
Instances, Attributes, Concepts
• Instances (observations, case)
§ The atomic elements of information from a dataset
§ Also known as records, prototypes, or examples
• Attributes (variable)
§ Measures aspects of an instance
§ Also known as features or variables
§ Each instance is composed of a certain number of attributes
• Concepts
§ Special content inside the data
§ Kind of things that can be learned
§ Intelligible and operational concept description
7
8. Prof. Pier Luca Lanzi
Two Versions of the Weather Data 8
……………
YesFalseNormalMildRainy
YesFalseHighHotOvercast
NoTrueHighHotSunny
NoFalseHighHotSunny
PlayWindyHumidityTemperatureOutlook
……………
YesFalse8075Rainy
YesFalse8683Overcast
NoTrue9080Sunny
NoFalse8585Sunny
PlayWindyHumidityTemperatureOutlook
10. Prof. Pier Luca Lanzi
Attributes
• Numeric Attributes
§Real-valued or integer-valued domain
§Interval-scaled when only differences are meaningful
(e.g., temperature)
§Ratio-scaled when differences and ratios are meaningful
(e.g., Age)
• Categorical Attributes
§Set-valued domain composed of a set of symbols
§Nominal when only equality is meaningful
(e.g., domain(Sex) = { M, F})
§Ordinal when both equality (are two values the same?) and
inequality (is one value less than another?) are meaningful
(e.g., domain(Education) = { High School, BS, MS, PhD})
10
11. Prof. Pier Luca Lanzi
Numerical Attributes
• Not only ordered but measured in fixed and equal units
• Examples
§Attribute “temperature” expressed in degrees
§Attribute “year”
• Characteristics
§Difference of two values makes sense
§Sum or product doesn’t make sense
§Zero point is not defined
• Sometimes they are divided into “discrete” and “continuous”
11
12. Prof. Pier Luca Lanzi
Nominal Attributes (or Categorical)
• Values are distinct symbols
• Values themselves serve only as labels or names
• Example
§Attribute “outlook” from weather data
§Values: “sunny”, “overcast”, and “rainy”
• Characteristics
§No relation is implied among nominal values
§No ordering
§No distance measure
§Only equality tests can be performed
12
13. Prof. Pier Luca Lanzi
Other Types of Attributes
• Ratio Attributes
§Numerical attributes for which the measurement scheme
defines a zero point (e.g., an attribute representing distance)
• Ordinal Attributes
§Categorical attributes with an imposed order on values
§No distance between values defined
§For instance, the attribute “temperature” in weather data
“hot” > “mild” > “cool”
13
14. Prof. Pier Luca Lanzi
Nominal or Ordinal?
• Attribute “age” nominal
§If age = young and astigmatic = no
and tear production rate = normal
then recommendation = soft
• Attribute “age” ordinal
(e.g. “young” < “pre-presbyopic” < “presbyopic”)
§If age≤pre-presbyopic and astigmatic = no
and tear production rate = normal
then recommendation = soft
14
15. Prof. Pier Luca Lanzi
Why Specifying Attribute Types?
• Some algorithms fit some specific data types best
• Express the best possible patterns into data
• Make the most adequate comparisons
• Example
§Outlook > “sunny” does not make sense, while
§Temperature > “cool” or
§Humidity > 70 does
• Additional uses of attribute type
§Check for valid values
§Deal with missing values, etc.
15
17. Prof. Pier Luca Lanzi
Why Missing Values Exist?
• Faulty equipment, incorrect measurements, missing cells in manual
data entry, censored/anonymous data
• Review scores for movies, books, etc.
• Very frequent in questionnaires for medical scenarios
• Censored/anonymous data
• In practice, a low rate of missing values may be suspicious
• Interview data (“Did you ever …”)
17
18. Prof. Pier Luca Lanzi
Missing Values
• Frequently indicated by out-of-range entries (e.g. max/min float),
Nan or special values (e.g., zero)
• Missing value may have significance in itself
§E.g. missing test in a medical examination
• Most schemes assume that is not the case
§“missing” may need to be coded as additional value
• Does absence of value have some significance?
§If it does, “missing” is a separate value
§If it does not, “missing” must be treated in a special way
18
19. Prof. Pier Luca Lanzi
What Types of Missing Values?
• Missing completely at random (MCAR)
§ The distribution of an example having a missing value for an attribute does not depend on
either the observed data or the missing data
§ Example: some survey questions contain a random sample of the whole questionnaire
• Missing at random (MAR)
§ The distribution of an example having a missing value for an attribute depends on the
observed data, but does not depend on the missing data
§ Missing at Random means the propensity for a data point to be missing is not related to the
missing data, but it is related to some of the observed data
§ Whether or not someone answered #13 on your survey has nothing to do with the missing
values, but it does have to do with the values of some other variable
§ For example, people who don’t declare their salary not because of the amount of it but just
because they don’t want to.
• Not missing at random (NMAR)
§ The distribution of an example having a missing depends on the missing values.
§ For example, respondents with high income less likely to report income
• Note that NMAR and MAR might be difficult to identify and often require domain knowledge
19
20. Prof. Pier Luca Lanzi
Dealing with Missing Values
• Use what you know
§ Why data is missing
§ Distribution of missing data
• Decide on the best strategy to yield the least biased estimates
§ Deletion Methods (listwise deletion, pairwise deletion)
§ Single Imputation Methods (mean/mode substitution, dummy variable
method, single regression)
§ Model-Based Methods (maximum Likelihood, multiple imputation
20
21. Prof. Pier Luca Lanzi
Strategies for missing values handling
• The handling of missing data depends on the type
• Discarding all the examples with a missing values
§ Simplest approach
§ Allows the use of unmodified data mining methods
§ Only practical if there are few examples with missing values. Otherwise, it
can introduce bias
• Fill in the missing value manually J
• Convert the missing values into a new value
§ Use a special value for it
§ Add an attribute that indicates if value is missing or not
§ Greatly increases the difficulty of the data mining process
• Imputation methods
§ Assign a value to the missing one, based on the rest of the dataset. Use
the unmodified data mining methods.
21
22. Prof. Pier Luca Lanzi
Listwise Deletion
(Complete Case Analysis)
• Only analyze cases with available data
on each variable
• Simple, but reduces the data
• Comparability across analyses
• Does not use all the information
• Estimates may be biased if data not
MCAR
22
23. Prof. Pier Luca Lanzi
Pairwise deletion
(Available Case Analysis)
• Analysis with all cases in which
the variables of interest are
present
• Advantage
§Keeps as many cases as
possible for each analysis
§Uses all information
possible with each analysis
• Disadvantage
§Can’t compare analyses
because sample different
each time
23
24. Prof. Pier Luca Lanzi
Imputation methods
• Extract a model from the dataset to perform the imputation
• Suitable for MCAR and, to a lesser extent, for MAR
• Not suitable for NMAR type of missing data
• For NMAR we need to go back to the source of the data to
obtain more information
• Survey of imputation methods available at
http://sci2s.ugr.es/MVDM/index.php
http://sci2s.ugr.es/MVDM/biblio.php
24
25. Prof. Pier Luca Lanzi
Single Imputation Methods
• Mean/mode substitution (most common value)
§ Replace missing value with sample mean or mode
§ Run analyses as if all complete cases
§ Advantages: Can use complete case analysis methods
§ Disadvantages: Reduces variability
• Dummy variable control
§ Create an indicator for missing value (1=value is missing for observation;
0=value is observed for observation)
§ Impute missing values to a constant (such as the mean)
§ Include missing indicator in the algorithm
§ Advantage: uses all available information about missing observation
§ Disadvantage: results in biased estimates, not theoretically driven
• Regression Imputation
§ Replaces missing values with predicted score from a regression equation.
25
26. Prof. Pier Luca Lanzi
Do Not Impute (DNI)
• Simply use the default policy of the data mining method
• Works only if the policy exists
26
28. Prof. Pier Luca Lanzi
Inaccurate Values
• Data has not been collected for mining it
• Errors and omissions that don’t affect original purpose of data
(e.g. age of customer)
• Typographical errors in nominal attributes,
thus values need to be checked for consistency
• Typographical and measurement errors in numeric attributes,
thus outliers need to be identified
• Errors may be deliberate (e.g. wrong zip codes)
28
30. Prof. Pier Luca Lanzi
The Geometrical View of the Data
• When the data matrix contains only numerical values
§Every row can be viewed as a point in a d-dimension space
§Every column as a point in a n-dimensional space
30
37. Prof. Pier Luca Lanzi
Data Format
• Most commercial tools have their own proprietary format
• Most tools import excel files and comma-separated value files
37
Year,Make,Model,Length
1997,Ford,E350,2.34
2000,Mercury,Cougar,2.38
Year;Make;Model;Length
1997;Ford;E350;2,34
2000;Mercury;Cougar;2,38
38. Prof. Pier Luca Lanzi
Attribute-Relation File Format
(ARFF)
38
%
% ARFF file for weather data with some numeric features
%
@relation weather
@attribute outlook {sunny, overcast, rainy}
@attribute temperature numeric
@attribute humidity numeric
@attribute windy {true, false}
@attribute play? {yes, no}
@data
sunny, 85, 85, false, no
sunny, 80, 90, true, no
overcast, 83, 86, false, yes
...
http://www.cs.waikato.ac.nz/~ml/weka/arff.html
39. Prof. Pier Luca Lanzi
Additional Attribute Types
• ARFF supports string attributes:
• Similar to nominal attributes but list of values
is not pre-specified
• ARFF also supports date attributes:
• Uses the ISO-8601 combined date
and time format yyyy-MM-dd-THH:mm:ss
39
@attribute description string
@attribute today date
40. Prof. Pier Luca Lanzi
Additional Attribute Types
• ARFF supports sparse data, for instance the following examples,
• Can also be represented as,
40
0, 26, 0, 0, 0 ,0, 63, 0, 0, 0, “class A”
0, 0, 0, 42, 0, 0, 0, 0, 0, 0, “class B”
{1 26, 6 63, 10 “class A”}
{3 42, 10 “class B”}
41. Prof. Pier Luca Lanzi
Missing Values in ARFF 41
@relation labor
@attribute 'duration' real
@attribute 'wage-increase-first-year' real
@attribute 'wage-increase-second-year' real
@attribute 'wage-increase-third-year' real
@attribute 'cost-of-living-adjustment' {'none','tcf','tc'}
@attribute 'working-hours' real
@attribute 'pension' {'none','ret_allw','empl_contr'}
@attribute 'standby-pay' real
@attribute 'shift-differential' real
@attribute 'education-allowance' {'yes','no'}
@attribute 'statutory-holidays' real
@attribute 'vacation' {'below_average','average','generous'}
@attribute 'longterm-disability-assistance' {'yes','no'}
@attribute 'contribution-to-dental-plan' {'none','half','full'}
@attribute 'bereavement-assistance' {'yes','no'}
@attribute 'contribution-to-health-plan' {'none','half','full'}
@attribute 'class' {'bad','good'}
@data
1,5,?, ?, ?,40, ?, ?,2, ?,11,'average', ?, ?,'yes',?,'good'
2,4.5,5.8, ?, ?,35,'ret_allw', ?, ?,'yes',11,'below_average', ?,'full', ?,'full','good'
?, ?, ?, ?, ?,38,'empl_contr', ?,5, ?,11,'generous','yes','half','yes','half','good'
3,3.7,4,5,'tc', ?, ?, ?, ?,'yes', ?, ?, ?, ?,'yes', ?,'good'
42. Prof. Pier Luca Lanzi
Attribute Types and Interpretation
• Interpretation of attribute types in ARFF depends
on the mining scheme
• Numeric attributes are interpreted as
§Ordinal scales if less-than and greater-than are used
§Ratio scales if distance calculations are performed
(normalization/standardization may be required)
• Instance-based schemes define distance between nominal values
(0 if values are equal, 1 otherwise)
• Integers in some given data file: nominal, ordinal, or ratio scale?
42
43. Prof. Pier Luca Lanzi
DSPL: Dataset Publishing Language
• Open format by Google available at
http://code.google.com/apis/publicdata/
• Use existing data: add an XML metadata file existing CSV
• Read by the Google Public Data Explorer, which includes
animated bar chart, motion chart, and map visualization
• Allow linking to concepts in other datasets
• Geo-enabled: allows adding latitude and longitude data to your
concept definitions
43
45. Prof. Pier Luca Lanzi
Predictive Model Markup Language
• XML-based markup language developed by the Data Mining
Group (DMG) to provide a way for applications to define models
related to predictive analytics and data mining
• The goal is to share models between applications
• Vendor-independent method of defining models
• Allow to exchange of models between applications.
• PMML Components: data dictionary, data transformations, model,
mining schema, targets, output
45
47. Prof. Pier Luca Lanzi
Publicly Available Datasets
• UCI repository
§http://archive.ics.uci.edu/ml/
§Probably the most famous collection of datasets
• Kaggle
§http://www.kaggle.com/
§It is not a static repository of datasets, but a site that manages
Data Mining competitions
§Example of the modern concept of crowdsourcing
• KDNuggets
§http://www.kdnuggets.com/datasets/
47