Introduction to Data Mining forWeb ApplicationsPaul-Alexandru Chirita, Ph.D.

About MeEducation:Ph.D., Information Retrieval & Data Mining, Univ. of Hannover, GermanyB.Sc., Ecole Polytechnique, Paris, France + “Politehnica” Univ. Bucharest, CS Dept.Roughly 8 yrs. in IT, out of which 7 in IR & DMNow in Adobe Romania (L3S, Yahoo!, Schlumberger and others in the past)

Web MiningThe application of Data Mining algorithms to discover patterns in the Web.Three dimensions:Usage MiningAnalyzes various access logs in order to provide input to Business DecisionsBy far the most used, with the highest ROIContent MiningAnalyzes Web page content in order to extract useful information (e.g., keywords, topic, content type, sentiment, etc.)Structure MiningAlso known as “Link Analysis”Investigates the hyperlink structure of the Web to improve current algorithms

AgendaClient side toolsGoogle AnalyticsOmnitureServer side toolsAW-StatsWebalizer / AWF-FullAdvanced analytics

AgendaClient side toolsGoogle AnalyticsOmnitureServer side toolsAW-StatsWebalizer / AWF-FullAdvanced analytics

Client side toolsPurpose:Return basic information about traffic on your Web Site, SEOMost of them are also (partly) integrated with Monetization Tools (e.g., AdWords)Pros:Hosted by third party sites, zero or minimal cost for youEasy to implement and integrate, no maintenanceCons:The client side tracking code will eat some of your bandwidth (~200-600 ms. additional response time)If your traffic increases “too much” you have to pay

Client-side tools: Google AnalyticsFree, and well-engineered!Shows statistics about:Basic stuff: Visits, Pages, etc.Visitor profiles: Browser, OS, Language/LocaleVisitor loyalty: How many times did each visitor return to your site, When was the last time they did it, For how longTrends: Is your traffic & popularity growing or decreasingTraffic sources: Entry/Exit pages, Referring sites & search enginesSome customization planned for the near-term futureGood for personal or small scale siteshttps://www.google.com/analytics

Client-side tools: Google Analytics [2]

Omniture: Site CatalystLow price per thousand of entries, but may become costly if you have a lot of traffic (millions of visits per day) or if you have many dozens of sensorsSame statistics as Google Analytics, but you can drill down very deep:Statistics per hour of day, per file type (html, cfm, etc.), per action type (download, view page, etc.)Visitor segmentation down to the level of cityPurchases, Promotions, and Many metrics for e-commerce (e.g., how many products added to the cart have actually been checked out)Most importantly, you can define ANY metric you want! (e.g., how many people click on my survey link, how many of them fill it in, etc.)www.omniture.com

Omniture: Site Catalyst [2]

AgendaClient side toolsGoogle AnalyticsOmnitureServer side toolsAW-StatsWebalizer / AWF-FullAdvanced analytics

Server side toolsPurpose:Return basic information about traffic on your Web SiteSimilar to the client-side tools, but currently more focused on Reliability & Application ImprovementsPros:Most importantly, zero bandwidth overhead for your app (Every ms counts!)Show a lot of developer specific information (errors, visitor browsers/OS, etc.)Very easy to installCons:Usually open source, but hard to extend with your own metrics

FREE Server side toolsSimilar statistics as with the Client Side tools, but…Less business specific information (do not include Visitor Loyalty, Trends, etc.)More developer specific data (errors & error types, HTTP status codes, etc.)Good for medium and large scale siteshttp://awstats.sourceforge.net/http://www.stedee.id.au/awffull/

Server side tools: AW Stats

Server side tools: Webalizer / AWF-Full

Paid Server side toolsOvercome most limitations of the free toolsLog everything into text files (see next Section)Provide some sort of SQL-like query language which helps you define any type of query you wantRun reports much fasterThe most expensive of them all, meant for professional usehttp://www.splunk.com/

AgendaClient side toolsGoogle AnalyticsOmnitureServer side toolsAW-StatsWebalizer / AWF-FullAdvanced analytics

How is this done in the heavy weight category ;-)Multiple log files, one per each functionality checkedAs simple as possible (see next slide for an example)The main guideline is to be able to parse any log file and generate statistics using only the command lineExample: Tab separated

Sample logDate & Time IP (hashed) User ID (hashed) Query ParametersSep 28 06:49:42 Ea9hjnc4ufTfU anonymous spell checker :0:10:en_US:en_US:0:0Sep 28 06:49:42 8NCTsHqR366 anonymous javascript :0:10:fr_FR:fr_FR:0:1Sep 28 06:49:42 K4nD5xy/R5fw anonymous text :0:10:en_US:en_US:0:1Sep 28 06:49:43 lRqBaIaUWxna yxDkhBEqC6xxR8z= module :0:10:en_US:en_US:0:0Sep 28 06:49:44 jMjJpy6bHAdb hPFLKaMNeShD0= delete spread :0:10:en_US:en_US:0:0Sep 28 06:49:44 r3xgRLagX1cQ6 anonymous _x :0:10:ru_RU:ru_RU:0:0Sep 28 06:49:45 b2DLBl3VTT67Q anonymous anti a :0:10:de_DE:de_DE:0:0Sep 28 06:49:45 KaKiB2ITEdPeM VcLic9CIy4QxVtJQ= create a star :0:10:en_US:en_US:0:0

What can be done using this dataYou can basically measure everything ;-)Plus you can enable loads of new features:Personalization for search, sold/promoted products, etc.Browsing recommendationsImprove site organization (make popular pages more accessible, promote some other pages and track their traffic increase, etc.)Search suggestionsAdvertising (keyword selection, etc.)

Personalized search and promotionsShow different results/ads to different users

Browsing recommendations

Search suggestions