The document discusses web mining including extracting information from web documents and services. It describes challenges of web mining like noisy and dynamic web data. It then outlines the data flow system including developing dimension tables for date, time of day, visitor, object, referrer, status, browser, and operating system. Finally, it specifies reports on statistics of visits, most/least popular pages, visitor locations, frequent visitors, top referrers/keywords, and most used browsers and operating systems.

1. WEB MINING
Submitted by:
Dheeraj Kashnyal
dheerajkashnyal55@gmail.com
ETL Design &
Report Specifications

2. Introduction
• Web mining is the use of techniques to automatically discover and extract
information from Web documents and services.
• Various kinds of information extracted via Web Mining:
• Web activity, from server logs and Web browser activity tracking.
• Web graph, from links between pages, people and other data.
• Web content, for the data found on Web pages and inside of documents.
• The project is based on extracting values from
web pages and other documents found on the
web.
• This presentation covers the ETL design and
Report Specification portion.
Web Mining

3. Challenges
• The Web is noisy. A Web page typically contains a mixture of many kinds of
information, e.g., main contents, advertisements, navigation panels,
copyright notices, etc.
• The Web is dynamic. Information on the Web changes constantly. Keeping
up with the changes and monitoring the changes are important issues.
• Much of the Web information is redundant. The same piece of information
or its variants may appear in many pages.
• Information/data of almost all types exist on the Web, e.g., structured tables,
texts, multimedia data, etc.
• Much of the Web information is semi-structured due to the nested structure
of HTML code.
Web Mining

4. Data Flow of the System
Web Mining

5. WM_FACT
Datekey
TODkey
Visitorkey
Referrerkey
Statuskey
Objectkey
Browserkey
OSKey
Timestamp of Request
GMT_Diff
TimeViewed
BytesTransferred
DATE_DIM_TB
Datekey
Date
DayOfWeek
DayOfWeekNumber
WeekNumber
Week
MonthDay
MonthNumber
Month
Quarter
Year
BROWSER_DIM_TB
Browser key
Browser Type
Browser Name
OS_DIM_TB
OS_key
OS Name
OS Type
STATUS_DIM_TB
Status_key
Status Code
StatusDescription
StatusType
REFERRER_DIM_TB
Referrer_key
ReferringURL
ReferringSite
Keyword
OBJECT_DIM_TB
Object_key
URL
FileName
FileType
ObjectType
Object_size
Content Page
PageName
PageType
VISITOR_DIM_TB
Visitor_key
VisitorFlag
IPAddress
DomainName
CountryCode
Country
User_Name
TOD_DIM_TB
TODkey
TOD Lower
TOD Higher
Period of Day
Developing Data Model

6. ETL DESIGN
 Given Source to Target(Dimension) Data Mapping
 Given Data Sources are files
 Mapping of Dimension Table
• DIMENSION TABLES
• Date Dimension.
• TOD Dimension
• Visitor Dimension
• Object Dimension
• Referrer Dimension
• Status Dimension
• Browser Dimension
• Operating System Dimension
• FACT TABLE
• Click Stream Fact
Web Mining

7. Cont.……
• FACT TABLE
• Click Stream Fact
• The time in seconds the visitor has viewed a page on a
particular date and time is stored in the click stream fact as a
measure.
• The bytes transferred to the user machine from the web
server are stored as a measure.
• The referrer key points to referred dimension, which
provides information about the referrer of the page.
• Rest are the Foreign Keys of the respective dimensions .
Web Mining

8. Report Specifications
• Statistics of Visits
• The measures reported are:
• No. of visitors during the day
• No. of content pages access by all visitors
• No. of objects accessed by all visitors
• Total size of the data that is being delivered
• Most popular (most accessed) pages
• The report should show the following measures
• No. of visitors during the day accessing this content page
• No. of hits during the day for this content page
• The report should show only the Top 5 pages accessed
based on the No. of hits
Web Mining

9. Report Specifications
• Least popular (least accessed) pages
• The report should show the following measures
• No. of visitors during the day accessing this content page
• No. of hits during the day for this content page
• The report should show only the Top 5 pages where the no.
of hits are the lowest
• Location of the visitors
• The report should show the Date of the visit
• The report should list the location of the Visitor
• Country Code
• Country Name
Web Mining

10. • Most frequent visitors
• The report should list the Visitor’s details
• IP Address
• Domain name of the visitor’s IP Address
• The report should show the following measures
• No. of hits during the date range for this content page
• Total size of content delivered
• The report should show only the Top 5 visitors accessed
based on the No. of hits.
• Top Referrers and Keywords
• The report should show the Referrer Domain and the
keyword used
• The report should show the no. of hits during the date range
and the time period
Web Mining
Report Specifications

11. Report Specifications
• Most used Browsers and Operating systems
• The report should retrieve data for the date range between
from and to date.
• The report should show the no. of hits during the date range
and the time period.
Web Mining

12. Web Mining