data_analysis_and_mining.ppt

400 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
400
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

data_analysis_and_mining.ppt

  1. 1. Chapter 18: Data Analysis and Mining Kat Powell
  2. 2. Chapter 18: Data Analysis and Mining <ul><li>Decision Support Systems </li></ul><ul><li>Data Analysis and OLAP </li></ul><ul><li>Data Warehousing </li></ul><ul><li>Data Mining </li></ul>
  3. 3. Decision Support Systems <ul><li>Make business decisions, often based on data collected by on-line transaction-processing systems </li></ul><ul><ul><li>What items to stock? </li></ul></ul><ul><ul><li>To whom to send advertisements? </li></ul></ul><ul><li>Examples of data used for making decisions </li></ul><ul><ul><li>Retail sales transaction details </li></ul></ul><ul><ul><li>Customer profiles (income, age, gender, etc.) </li></ul></ul>
  4. 4. Decision-Support Systems Overview <ul><li>Data analysis tasks are simplified by specialized tools and SQL extensions </li></ul><ul><li>Data mining seeks to discover knowledge automatically in the form of statistical rules and patterns from large databases. </li></ul><ul><li>A data warehouse archives information gathered from multiple sources, and stores it under a unified schema, at a single site. </li></ul><ul><ul><li>Important for large businesses that generate data from multiple divisions, possibly at multiple sites </li></ul></ul>
  5. 5. Data Analysis and OLAP <ul><li>OnLine Analytical Processing (OLAP) </li></ul><ul><ul><li>Online interactive tools for analysis and summary of data that present it to humans in understandable format </li></ul></ul><ul><li>Multidimensional data – data modeled as dim ension attributes and mea sure attributes in table summary format </li></ul><ul><ul><li>Measure attributes measure some value and can be aggregated upon </li></ul></ul><ul><ul><li>Dimension attributes define the dimensions on which measure attributes (or aggregates thereof) are viewed </li></ul></ul>
  6. 6. Cross Tabulation (aka cross-tab) of sales by item-name and color <ul><ul><ul><ul><li>item_name, color, and size are the dim ension attributes of the sales relation </li></ul></ul></ul></ul><ul><ul><ul><ul><li>the number values in each cell represent m e asure attributes </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Extra column and row store the cell totals for each column and row </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Summary of sales </li></ul></ul></ul></ul>
  7. 7. Relational Representation of Cross-tabs <ul><li>Cross-tabs also can be represented as relations </li></ul><ul><ul><li>The value ' all' is used to represent an aggregate, that is, the set of all values for an attribute </li></ul></ul><ul><ul><li>The SQL:1999 standard actually uses ' null' values in place of ' all' despite confusion with regular ' null' values </li></ul></ul>
  8. 8. Data Cube (Sales relation) <ul><li>Multidimensional generalization of a cross-tab </li></ul><ul><li>Can have n dimensions; we see 3 here </li></ul><ul><li>Cross-tabs can be used as views on a data cube </li></ul><ul><li>All cells contain values...even the invisible (inner) cells </li></ul>
  9. 9. Online Analytical Processing <ul><ul><li>Pivoting : changing the dimensions used in a cross-tab </li></ul></ul><ul><ul><li>EX: An analyst may select a cross-tab on item_name and size OR on color and size </li></ul></ul><ul><ul><li>Slicing: creating a cross-tab for fixed values only </li></ul></ul><ul><ul><ul><li>Sometimes called dicing , particularly when values for multiple dimensions are fixed . </li></ul></ul></ul><ul><ul><li>EX: An analyst may wish to see cross-tab on item_name and color for a fixed value of size (for instance...large) , instead of the sum across all sizes. </li></ul></ul>
  10. 10. Online Analytical Processing <ul><ul><li>OLAP systems permit users to view data at any desired level of granularity </li></ul></ul><ul><ul><li>Rollup: moving from finer-granularity data to a coarser granularity </li></ul></ul><ul><ul><li>EX: Starting from the data cube on the sales table, got the cross-tab by rolling up on the attribute ' size' </li></ul></ul><ul><ul><li>Drill down: The opposite operation - that of moving from coarser-granularity data to finer-granularity data </li></ul></ul><ul><ul><li>EX: usually generated from the original data like drilling for fine, rare jewels </li></ul></ul>
  11. 11. Data Warehousing <ul><ul><li>Data sources often store only current data, not historical data </li></ul></ul><ul><ul><li>Corporate decision making requires a unified view of all organizational data, including historical data </li></ul></ul><ul><ul><li>A data warehouse is a repository (archive) of information gathered from multiple sources, stored under a unified schema, at a single site </li></ul></ul><ul><ul><ul><li>Greatly simplifies querying, permits study of historical trends </li></ul></ul></ul><ul><ul><ul><li>Shifts decision support query load away from transaction processing systems </li></ul></ul></ul>
  12. 12. Data Warehousing
  13. 13. Warehouse Design Issues <ul><ul><li>When and how to gather data </li></ul></ul><ul><ul><ul><li>Source driven architecture: data sources transmit new information to warehouse, either continuously or periodically (e.g. at night) </li></ul></ul></ul><ul><ul><ul><li>Destination driven architecture: warehouse periodically requests new information from data sources </li></ul></ul></ul><ul><ul><ul><li>Keeping warehouse exactly synchronized with data sources (e.g. using two-phase commit) is too expensive </li></ul></ul></ul><ul><ul><ul><ul><li>Usually OK to have slightly out-of-date data at warehouse </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Data/updates are periodically downloaded form online transaction processing (OLTP) systems. </li></ul></ul></ul></ul>
  14. 14. More Warehouse Design Issues <ul><ul><li>What schema to use </li></ul></ul><ul><ul><ul><li>Schema integration </li></ul></ul></ul><ul><ul><li>Data cleansing </li></ul></ul><ul><ul><ul><li>E.g. correct mistakes in addresses (misspellings, zip code errors, etc.) </li></ul></ul></ul><ul><ul><ul><li>Merge address lists from different sources and purge duplicates </li></ul></ul></ul><ul><ul><li>How to propagate updates </li></ul></ul><ul><ul><ul><li>Warehouse schema may be a (materialized) view of schema from data sources </li></ul></ul></ul>
  15. 15. Data Mining <ul><ul><li>The process of semi-automatically analyzing large databases to find useful patterns </li></ul></ul><ul><ul><li>Prediction based on past history </li></ul></ul><ul><ul><ul><li>Predict if a credit card applicant poses a good credit risk, based on some attributes (income, job type, age, ..) and past history </li></ul></ul></ul><ul><ul><ul><li>Predict if a pattern of phone calling card usage is likely to be fraudulent </li></ul></ul></ul>
  16. 16. Data Mining (Cont.) <ul><ul><li>Some other examples of prediction mechanisms: </li></ul></ul><ul><ul><ul><li>Classification </li></ul></ul></ul><ul><ul><ul><ul><li>Given a new item whose class is unknown, predict to which class it belongs </li></ul></ul></ul></ul><ul><ul><ul><li>Regression formulae </li></ul></ul></ul><ul><ul><ul><ul><li>Given a set of mappings for an unknown function, predict the function result for a new parameter value </li></ul></ul></ul></ul>
  17. 17. Data Mining (Cont.) <ul><ul><li>Descriptive Patterns </li></ul></ul><ul><ul><ul><li>Associations </li></ul></ul></ul><ul><ul><ul><ul><li>Find books that are often bought by “similar” customers. If a new such customer buys one such book, suggest the others too. </li></ul></ul></ul></ul><ul><ul><ul><li>Associations may be used as a first step in detecting causation </li></ul></ul></ul><ul><ul><ul><ul><li>E.g. association between exposure to chemical X and cancer </li></ul></ul></ul></ul><ul><ul><ul><li>Clusters </li></ul></ul></ul><ul><ul><ul><ul><li>E.g. typhoid cases were clustered in an area surrounding a contaminated well </li></ul></ul></ul></ul><ul><ul><ul><ul><li>remains important in detecting epidemics </li></ul></ul></ul></ul>
  18. 18. Other Types of Mining <ul><ul><li>Text mining : application of data mining to textual documents </li></ul></ul><ul><ul><ul><li>cluster Web pages to find related pages </li></ul></ul></ul><ul><ul><ul><li>cluster pages a user has visited to organize their visit history </li></ul></ul></ul><ul><ul><ul><li>classify Web pages automatically into a Web directory </li></ul></ul></ul>
  19. 19. Other Types of Mining <ul><ul><li>Data visualization systems help users examine large volumes of data and detect patterns visually </li></ul></ul><ul><ul><ul><li>Can visually encode large amounts of information on a single screen </li></ul></ul></ul><ul><ul><ul><li>Humans are very good a detecting visual patterns </li></ul></ul></ul>
  20. 20. The END

×