Downloaded 103 times
![ Most famous data mining tasks:
Classification [Predictive]
Prediction [Predictive]
Association Rules [Descriptive]
Clustering [Descriptive]
Outlier Analysis [Descriptive]
Data Mining Tasks](https://image.slidesharecdn.com/1introduction-to-data-miningchapter1-181005051512/75/Introduction-to-data-mining-chapter-1-42-2048.jpg)
Uploaded byMahmoud Alfarra
Introduction to-data-mining chapter 1
The document is an introduction to data mining, covering its definition, interdisciplinary nature, processes, tasks, challenges, and applications. It emphasizes the non-trivial process of discovering patterns in large datasets, and describes phases such as data cleaning, integration, and evaluation. Additionally, it outlines various data mining tasks including classification, prediction, association, clustering, and outlier analysis, along with key challenges faced in the field.
More Related Content
Similar to Introduction to-data-mining chapter 1
Introduction to-data-mining chapter 1
- 1. Introduction to DataMining Mahmoud Rafeek Alfarra http://mfarra.cst.ps University College of Science & Technology- Khan yonis Development of computer systems 2016 Chapter 1
- 2. Outline Definition ofData Mining Data Mining as an Interdisciplinary field Process of Data Mining Data Mining Tasks Challenges of Data Mining Data mining application examples Introduction to RapidMiner
- 3.
- 4.
- 5. Definition of DataMining "Computers have promised us a source of wisdom but delivered a flood of data." "It has been estimated that the amount of information in the world doubles every 20 months." The Explosive Growth of Data: from terabytes to petabytes We are drowning in data, but starving for knowledge!
- 6. Definition of DataMining Knowledge discovery in databases (data mining) is “The non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data”.
- 7. Definition of DataMining Pattern is an arrangement of repeated parts. In a data table, a pattern is defined as a set of rows that share the same values in two or more columns. Consider for example, the following table that contains data about objects; shape, color, and weight.
- 8. Definition of DataMining WeightColorShapeRow # 100RedBox1-> 200RedBox2-> 300RedBox3-> 400BlueBox4 400BlueCone5 In this table, we have 3 rows (row 1, 2 and 3) that share the same values in two columns (Shape and Color). From this table, we can observe the following patterns: Most Boxes are Red. We can represent Pattern as rule: If Shape = Box then Color = Red.
- 9. Definition of DataMining Valid: Discovered patterns should be true on new data with some degree of certainty. Generalize to the future (other data). Novel: Patterns must be novel (should not be previously known).
- 10. Definition of DataMining Actionable: patterns should potentially lead to some useful actions. Understandable: Patterns must be made understandable in order to facilitate a better understanding of the underlying data.
- 11. Definition of DataMining Example: Credit Risk A credit risk is the risk of default on a debt that may arise from a borrower failing to make required payments. In the first resort, the risk is that of the lender and includes lost principal and interest, disruption to cash flows, and increased collection costs.
- 12. Definition of DataMining Is it valid? The pattern has to be valid with respect to a certainty level (rule true for the 86%) Is it novel? The value k should be previously unknown or obvious Is it useful? The pattern should provide information useful to the bank for assessing credit risk Is it understandable?
- 13. Definition of DataMining Other definition of data mining: “Is the process of extracting knowledge hidden from large volumes of raw data. The knowledge must be new, not obvious, and must be able to use it”.
- 14. Definition of DataMining Many people treat data mining as a synonym for another popularly used term , knowledge Discovery in Databases, or KDD. Alternatively, other view data mining as simply an essential step in the process of knowledge discovery in databases.
- 15. Definition of DataMining What is Data Mining?What is not Data Mining? Certain names are more common in certain US locations (O’Brien, O’Rurke, O’Reilly … in Boston area) Look up phone number in phone directory Group together similar documents returned by search engine according to their context (e.g. Amazon rainforest, Amazon.com,) information about “Amazon” Query a Web search engine
- 16. Data Mining andBusiness Intelligence Increasing potential to support business decisions End User Business Analyst Data Analyst DBA Decision Making Data Presentation Visualization Techniques Data Mining Information Discovery Data Exploration Statistical Summary, Querying, and Reporting Data Preprocessing/Integration, Data Warehouses Data Sources Paper, Files, Web documents, Scientific experiments, Database Systems
- 18. Introduction to DataMining Mahmoud Rafeek Alfarra http://mfarra.cst.ps University College of Science & Technology- Khan yonis Development of computer systems 2016 Lecture 2
- 19. Outline Definition ofData Mining Data Mining as an Interdisciplinary field Process of Data Mining Data Mining Tasks Challenges of Data Mining Data mining application examples Introduction to RapidMiner
- 20. Outline Definition of DataMining Data Mining as an Interdisciplinary field Process of Data Mining Data Mining Tasks Challenges of Data Mining Data mining application examples Introduction to RapidMiner
- 21. Data Mining asan Interdisciplinary field “Data mining is an interdisciplinary field bringing together techniques from machine learning, pattern recognition, statistics, databases, and visualization to address the issue of information extraction from large data bases”.
- 22. Data Mining asan Interdisciplinary field Data Mining Database Technology Statistics Other Disciplines Artificial Intelligence Machine Learning Visualization
- 23. Data Mining asan Interdisciplinary field Data mining is differ than statistics in kind of data (not only numerical) , kinds of methods ( mostly use machine learning methods), more than one hypotheses, amount of data (statistics uses samples).
- 24. Data Mining asan Interdisciplinary field Data Mining uses methods from Machine Learning such as decision tree and neural nets. Machine Learning uses samples and Data Mining uses whole data. Data Mining can access data from database. Machine Learning some times used to replace human where Data Mining to help human.
- 25. Data Mining asan Interdisciplinary field Databases part of Data Mining that provide the fast and reliable access to data. Databases used for data operation (Storing and retrieving data), Data Mining for Decision making.
- 26. Data Mining asan Interdisciplinary field Search techniques , Knowledge representation, Knowledge acquisition, maintenance and application are other branches of Artificial Intelligence which are highly related with Data Mining.
- 27. Data Mining asan Interdisciplinary field Visualization is used to gain visual insights into the structure of the data. Visualization is in large quantities used as a pre- and post-processing tool for data mining.
- 28. Process of DataMining Data Mining is essentially a process of data drive extraction of not so obvious but useful information from large databases. The entire process is interactive and iterative.
- 29. Process of DataMining Data Cleaning Data Integration Databases Data Warehouse Task-relevant Data Selection Data Mining Pattern Evaluation.
- 30. Data cleaning Real-worlddata tends to be incomplete, noisy and inconsistent. incomplete: lacking attribute values, lacking certain attributes of interest. ◦ e.g., occupation=“ ” (missing data) noisy: containing noise, errors, or outliers ◦ e.g., Salary=“−10” (an error) inconsistent: containing difference in codes or names, ◦ e.g., Age=“42” Birthday=“03/07/1997”
- 31. Data Integration Dataintegration is the merging of data from multiple sources. These sources may include multiple databases, data cubes, or flat files.
- 32. Data Selection Wheredata relevant to the analysis task are retrieved from the database. Therefore, irrelevant, weakly relevant or redundant attributes may be detected and removed.
- 33. Data Transformation Wheredata are transformed or consolidated into forms appropriate for mining by performing: Summary or aggregation operation, for example: Daily sales may be aggregated to monthly sales or annual sales. Generalization, for example: City may be generalized to country or age may generalized to young, middle- age, senior.
- 34. Data Mining Anessential process where intelligent methods are applied on data to covert it to knowledge in for decision making. Wide range of methods can be used in data mining such neural nets, decision tree and Association.
- 35. Pattern evaluation Toidentify the truly interesting pattern based on some interestingness measures. A pattern consider interesting if it is: Valid Novel Actionable Understandable
- 36. Knowledge Representation Knowledgepresentation is the framework that converts a large amount of data into a particular data or procedure that human being can figure out based on an intention. In Knowledge representation visualization tools and knowledge representation techniques are used to present the mined knowledge to the user.
- 38. Introduction to DataMining Mahmoud Rafeek Alfarra http://mfarra.cst.ps University College of Science & Technology- Khan yonis Development of computer systems 2016 Lecture 3
- 39. Outline Definition ofData Mining Data Mining as an Interdisciplinary field Process of Data Mining Data Mining Tasks Challenges of Data Mining Data mining application examples Introduction to RapidMiner
- 40. Data Mining Tasks Data mining tasks are the kind of data patterns that can be mined. Data Mining functionalities are used to specify the kind of patterns to be found in the data mining tasks.
- 41. In generaldata mining tasks can be classified into two categories: Descriptive mining tasks characterize the general properties of the data. Predictive mining tasks perform inferences on the current data in order to make predictions. Data Mining Tasks
- 42. Most famousdata mining tasks: Classification [Predictive] Prediction [Predictive] Association Rules [Descriptive] Clustering [Descriptive] Outlier Analysis [Descriptive] Data Mining Tasks
- 43. Classification Classification isused for predictive mining tasks. The input data for predictive modeling consists of two types of variables: Explanatory variables, which define the essential properties of the data. Target variables , whose values are to be predicted. Classification is used to predicate the value of discrete target variable.
- 44.
- 45. Prediction Similar toclassification, except we are trying to predict the value of a variable (e.g. amount of purchase), rather than a class (e.g. purchaser or non-purchaser).
- 46. Association Association Rulesaims to find out the relationship among valuables in database, resulting in deferent types of rules. Seek to produce a set of rules describing the set of features that are strongly related to each others.
- 47. Association Gender Age SmokerLAD% RCA% F 52 Y 85 100 M 62 N 80 0 M 75 Y 70 80 M 73 Y 40 99 M 66 N 50 45 … … … … … LAD%－ The percentage of heat disease caused by left anterior descending coronary artery. RCA%－ The percentage of heat disease caused by right coronary artery. Original data from a research on heart disease
- 48. Association Medical Association Rules NO.Rule 1 Gender=M∩Age≥70∩Smoker=YRCA%≥50(40%,100%) 2 Gender=F∩Age＜70∩Smoker=YLAD%≥70(20%,100%) Rule 1 indicates：40% of the cases are male, over 70 years old and have the habit of smoking, the possibility of RCA%≥50% is 100% Rule 2 indicates：20% of the cases are female, under 70 years old and have the habit of smoking, the possibility of LAD%≥70% is 100%
- 49. Clustering Finds groupsof data pointes (clusters) so that data points that belong to one cluster are more similar to each other than to data points belonging to different cluster.
- 50. Clustering Document Clustering: Goal:To find groups of documents that are similar to each other based on the important terms appearing in them. Approach: To identify frequently occurring terms in each document. Form a similarity measure based on the frequencies of different terms. Use it to cluster. Gain: Information Retrieval can utilize the clusters to relate a new document or search term to clustered documents.
- 51. Outlier Analysis Discoversdata points that are significantly different than the rest of the data. Such points are known as anomalies or outliers.
- 52. Outline Definition of DataMining Data Mining as an Interdisciplinary field Process of Data Mining Data Mining Tasks Challenges of Data Mining Data mining application examples Introduction to RapidMiner
- 53. Challenges of DataMining Scalability: Scalable techniques are needed to handle the massive scale of data. Dimensionality: Many applications may involves a large number of dimensions (e.g. features or attributes of data)
- 54. Challenges of DataMining Heterogeneous and Complex Data: In recent years complicated data types such as graph-based, text-free and structured data types are introduced. Techniques developed for data mining must be able to handle the heterogeneity of the data.
- 55. Challenges of DataMining Data Quality: Many data sets are imperfect due to present of missing values and noise un the data. To handle the imperfection, robust data mining algorithms must be developed.
- 56. Challenges of DataMining Data Distribution: As the volume of data increases , it is no longer possible or safe to keep all the data in the same place. As a result, the need for distributed data mining techniques has increased over the years.
- 57. Challenges of DataMining Privacy Preservation: While privacy intends to prevent the disclosure of information, data mining attempts to revel interesting knowledge about data. As a result, there is growing interest in developing privacy- preserving data mining algorithms.
- 58. Outline Definition of DataMining Data Mining as an Interdisciplinary field Process of Data Mining Data Mining Tasks Challenges of Data Mining Data mining application examples Introduction to RapidMine
- 59. Data mining application Science astronomy,bioinformatics, drug discovery, … Business advertising, CRM (Customer Relationship management), investments, manufacturing, sports/entertainment, telecom, e- Commerce, targeted marketing, health care, … Web search engines, web mining,… Government law enforcement, profiling tax cheaters,