Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. What Is Data Mining Data mining is the process that uses a variety of data analysis tools to discover patterns and relationships, which are hidden among the vast amount of data. From these patterns and relationships, the company will be able to make valid predictions in the future trends. Data mining tools can answer business questions that traditionally were too time consuming to resolve. They find hidden patterns and predictive information that experts may miss because those lie outside their expectations. Data mining is also known as knowledge data discovery (KDD). Those seeking to make a distinction between the terms data mining and KDD generally use KDD to refer to the process of discovering useful knowledge from data, while using data mining to refer to the application of algorithms for extracting patterns from data. Here are some examples for data mining: ♦Classification: To which set of predefined categories does this case belong? In marketing, when planning a mail shot, the categories may simply be the people who will buy and the people who will not buy. ♦Association: Which things occur together? For example, looking at shopping baskets, people who buy beer tend also to buy nuts at the same time. ♦Sequence: It is essentially a time-ordered association, although the associated events may be spread for apart in time. For example, most people will buy insurance after the marriage. ♦Clustering: It is like classification except that the categories are not normally known beforehand. It shows the collection of shopping baskets and discovers clusters corresponding to health food buyers, convenience food buyers, luxury food buyers and so on. Data mining process 1. Develop an understanding of the application, relevant prior knowledge, and the end user’s goals. 2. Create a target data set to be used for discovery.
  2. 2. 3. Clean and preprocess data (including handling missing data fields, noise in the data, accounting for time series, and known changes). 4. Reduce the number of variables and find invariant representations of data if possible. 5. Choose the data mining task (classification, regression, clustering, etc.). 6. Choose the data mining algorithm. 7. Search for patterns of interest (this is the actual data mining). 8. Interpret the pattern mined. If necessary, iterate through any of steps 1 through 7. 9. Consolidate knowledge discovered and prepare a report. Figure 1: Steps in the Data Mining Process (1) (2) (3) (4) Understand Target Data Clean, Preprocess Reduce No. Application for Discovery Data of Variables (5) (6) (7) (8) Choose Data Choose Data Search for Interpret Classification Mining Patterns Pattern Task Algorithm Mined (9) Prepare Report Data mining and data warehousing The data to be mined is usually extracted from an enterprise data warehouse into data mining database. However a data warehouse is not a requirement for data mining. Setting up a large data warehouse for data mining can be time consuming and cost millions of dollars because it has to consolidates data from multiple sources, resolves data integrity problems, and loads the data into a query database. Therefore, companies would like to mine data from one or more operational or transactional databases which is easy to extract data into read only database.
  3. 3. Data mining and OLAP One of the most common questions from data processing professionals is about the difference between data mining and On-Line Analytical Processing (OLAP). They are different tools but they can complement each other. Traditional query and report tools describe what is in a database. But OLAP is used to provide analysis for certain data why it is true. OLAP can assists analysts generate a series of hypothetical patterns and relationships and then uses queries against the database to verify them or disprove them. For example, an analyst might want to determine the factors that lead to loan defaults. He or she might initially hypothesize that people with low incomes are bad credit risks and analyze the database with OLAP to verify or disprove this assumption. If this hypothesis is not true, then an analyst might look at high debt as the determinant of risk. If this hypothesis is not supported by the data, either, then an analyst can try debt and income together as the best predictor of bad credit risks. Data mining is different from OLAP because rather than verify hypothetical patterns, it uses the data itself to find hidden patterns. For example, suppose an analyst who wants to identify the risk factor for load default and use a data mining tool. The data mining tool might discover that people with high debt and low incomes are bad credit risks which can be found by using OLAP. But it also can discover a pattern that an analyst does not think to try, such as age that is also a determinant of risk. Furthermore, OLAP is also complementary in the early stages of the knowledge discovery process because it can help user to explore the data, for instance by focusing attention on important variables, identifying exception, or finding interactions. This is important because the better understanding of the data, the more effective the knowledge discovery process will be. Data mining and hardware trends A key enabler of data mining is the major progress in hardware price and performance. The drop in the price of computer disk storage in these few years has
  4. 4. radically changed the collecting and storing massive amounts of data. The price not only drops in disk storage, but also in CPU. Each generation of chips greatly increases the power of CPU, while allowing the drop on the cost. This is also reflected in the price of RAM. Today, most PCs have 64 megabytes or more of RAM, and workstations have 256 megabytes or more, while servers with gigabytes are normal. Due to the power of the individual CPU has greatly in creased, all servers can support multiple CPUs using symmetric multi-processing, which allows hundreds of CPUs to work on finding patterns in the data. Data mining applications Data mining is increasingly popular because of the substantial contribution it can make. It can be used to control costs as well as contribute to revenue increases. Many organizations are using data mining to help manage all phases of the customer life cycle, including acquiring new customers increasing revenue from existing customers, and retaining good customers. Data mining offers value across a broad spectrum of industries. Telecommunications and credit card companies are two of the leaders in applying data mining to detect fraudulent use of their services. Insurance companies and stock exchanges are also interested in applying this technology to reduce fraud. Medical services use data mining to predict the effectiveness of surgical procedures, medical tests or medications. Companies active in the financial markets use data mining to determine market and industry characteristics, as well as to predict individual company and stock performance. Retailers are making more use of data mining to decide which products to stock in particular store, as well as to assess the effectiveness of promotions and coupons. Pharmaceutical firms are mining large databases of chemical compounds and of genetic material to discover substances that might be candidates for the treatments of disease. Helpful Hint ♦ The relation between the people who have domain knowledge and those who are data mining experts is a critical success factor. A data mining expert without domain
  5. 5. knowledge or a business expert who knows nothing about models and data are both useless. The data mining expert and the people who understand the business must work closely together. ♦ Data mining cannot find patterns of interest unless the tool is told what to look for. For example, to find ways of improving mailing list impact, the search would be directed to customers who have bought large amounts previously or who have responded frequently to previous offers. ♦ Data mining can be useful in many areas, not just those in which it has previously been successful. However, cost/benefit analysis needs to be done. If the cost of mining is larger than the anticipated return, it should not be undertaken. ♦ The techniques being applied in data mining are generally not new. They have been used for many years but not in a business context. With the large amounts of data now available in data warehouses and with new user-friendly software and interfaces, they have become accessible. Data mining no longer needs to be an extremely complex process. ♦ Data mining can be undertaken with relatively small databases. Large databases can be sampled. It is possible to have too much data and lose sight of the information it contains. For example, using both age and date of birth results in both factors being judged equally relevant and each will be assigned a lower weight. Current Limits to Data Mining ♦ Dealing with very large (e.g., terabyte) databases ♦ Dealing with high dimensionality, which increases the size of the search space and may also create spurious patterns ♦ Over fitting available data ♦ Rapid changes in data (non-stationary) that make previously discovered patterns invalid ♦ Missing and noisy data
  6. 6. ♦ Reducing the emphasis on fully automated, rapid-response environments and increasing the human / computer interaction ♦ Lack of full understanding of the patterns observed ♦ Managing changes in data and knowledge as available information is updated ♦ Dealing with non-numerical data, such as objects, text, and multimedia, more and more of which is being stored in databases ♦ Lack of integration with other systems All of these problems are areas of current research, but they are not yet fully solved. Nonetheless, despite these difficulties, data mining offers an important approach to achieving value from the data warehouse for use in decision support. Hand, D., Mannila, H., & Smyth, P. (2001). Principles of Data mining. The MIT Press: Cambridge, Massachusetts.