This document summarizes Martin Klein's dissertation defense on using the web infrastructure for real-time recovery of missing web pages. The defense agenda includes discussing lexical signatures for web pages, techniques for estimating document frequency values, correlations between term counts and document frequencies, using web page titles and link neighborhoods for rediscovery, and evaluating tags. The motivation is that a significant percentage of web pages become inaccessible over time, so the research aims to re-discover missing pages using content and link-based methods from the live web infrastructure.

1. Using the Web Infrastructure for Real Time Recovery of Missing Web Pages Dissertation Defense Martin Klein mklein@cs.odu.edu Old Dominion University Norfolk, VA 07/18/2011 Committee: Dr. Michael L. Nelson (Advisor) Dr. Yaohang Li Dr. Michele C. Weigle Dr. Mohammad Zubair Dr. Robert Sanderson Dr. Herbert Van de Sompel

2. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 2 Motivation Background

3. The Problem 3

5. URIs inaccessible in CS papers: 23%-53%[Lawrence2001]

6. Inaccessible web pages: 67% after 4 years [Koehler2002]

7. Inaccessible objects in DLs: 3%[Nelson2002]

8. URIs inaccessible in high IF journals: 3.8% after 3 months; 13% after 27 months [Dellavalle2003]

9. URIs inaccessible in D-Lib Magazine: ~30%[McCown2005]

10. URIs inaccessible (and not archived) in scholarly articles: ~25%[Sanderson2011]4

12. Has anybody crawled and indexed it?

13. Do Google, Yahoo!, Bing have a copy of the page?

14. Has the page been archived by a web archive?

15. Information retrieval techniques needed to (re-)discover content5

17. Requires knowledge about content

18. Problem with homographs (jaguar, present, lead, M/mobile, etc)

19. Problem with very frequent terms/names (Michael Nelson, Eric Miller, etc)

20. Web archives

21. Helps for apple pie recipe but not for web page of transferred faculty, e.g.6

22. Content Similarity JCDL 2005 http://www.jcdl2005.org/ July 2005 http://www.jcdl2005.org/ Today 7

23. Content Similarity Hypertext 2006 http://www.ht06.org/ August 2006 http://www.ht06.org/ Today 8

24. Content Similarity PSP 2003 http://www.pspcentral.org/events/annual_meeting_2003.html http://www.pspcentral.org/events/archive/annual_meeting_2003.html August 2003 Today 9

25. Content Similarity ECDL 1999 http://www.informatik.uni-trier.de/~ley/ db/conf/ercimdl/ercimdl99.html http://www-rocq.inria.fr/EuroDL99/ October 1999 Today 10

26. Content Similarity Greynet 1999 http://www.konbib.nl/infolev/greynet/2.5.htm 1999 Today ? ? 11

27. Research Questions (1) The Problem Based on the WI, can we use content- and link structure based methods to (re-)discover missing web pages in real time? Investigated Methods: Lexical signatures Titles Tags Link neighborhood lexical signatures 12

28. Research Questions (2) The Problem What are the optimal characteristics of these methods (age, length, etc) with respect to retrieval performance? Can we improve the performance by consolidating two or more methods? Can we have a real-world implementation and evaluation of the above? 13

29. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 14 Motivation Background

30. Memento, Web Infrastructure (WI) 15

31. Lexical Signatures (LSs) First introduced by Phelps and Wilensky[Phelps2000] Small set of terms capturing “aboutness” of a document, “lightweight” metadata Resource Abstract 10,000 terms 200 terms 16

33. Term frequency (TF):

34. “How often does this word appear in this document?”

35. Inverse document frequency (IDF):

36. “In how many documents does this word appear?”17

37. Lexical Signatures -- Examples 18

38. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 19 A Comparison of Techniques for Estimating IDF Values to Generate LexicalSignatures for the Web(WIDM 2008) Motivation Background

39. Accurate IDF Values for LSs Screen scraping the Google web interface 20

40. The Dataset Local universe consisting of copies of URIs from the Internet Archivebetween 1996 and 2007 21

42. Google N-GramsNote: N-Grams provide term count (TC) and not DF values – ask me for details 22

43. LSs Example Based on all 3 methods URL: http://www.perfect10wines.com Year: 2007 Union: 12 unique terms 23

46. Comparing LSs Top 5, 10 and 15terms LC – local universe SC – screen scraping NG – N-Grams 25

48. Compared to the Google N-Gram baseline

49. Screen scraping method seems preferable

50. Similarity scores are slightly higher

51. Feasible in real time!!!Contribution: Established well performing IDF estimation technique. 26

52. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 27 Revisiting Lexical Signatures to (Re-)Discover Web Pages(ECDL 2008) Motivation Background

54. Conduct overlap analysis

55. Neither Phelps and Wilensky nor Park et al.[Park2004] did that

56. Park et al. just re-confirmed their findings after 6 months28

57. LSs Over Time - Example 10-term LSs generated for http://www.perfect10wines.com 29

58. LS Overlap Analysis Rooted: overlap between the LS of the year of the first observation in the IA and all LSs of the consecutive years that URI has been observed Sliding: overlap between two LSs of consecutive years starting with the first year and ending with the last 30

60. Highest overlap in the first 1-2 years after creation of the LS

61. Rarely peaks after that – once terms are gone do not return31

62. Evolution of LSs over Time Sliding Results: Overlap increases over time Seem to reach steady state around 2003 32

64. Identify URI in result set

65. For each URI it is possible that:URI is returned as the top ranked result URI is ranked somewhere between 2 and 10 URI is ranked somewhere between 11 and 100 URI is ranked somewhere beyond rank 100 considered as not returned 33

67. 5-, 6- and 7-term LSs seem best

68. Top mean rank (MR) value with 5 terms

69. Most top ranked with 7 terms

70. Binary pattern: either in top 10 or undiscovered

71. 8 terms and beyond do not show improvement34

72. Performance of LSs wrt Length nDCG for LSs consisting of 2-15 terms (mean over all years) 35

73. Performance of LSs over Time nDCG for LSs consisting of 2, 5, 7 and 10 terms 36

75. Rooted: quickly after generation

76. Sliding: seem to stabilize

77. LSs older than 5 years perform poorly

78. 5-, 6- and 7-term LSs seem to perform best

79. 7 – most top ranked

80. 5 – lowest mean rank

81. 2..4 as well as 8+ term LSs are insufficient Contribution: Determined age and length limits for LSs. 37

82. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 38 Evaluating Methods to Rediscover Missing Web Pages from the Web Infrastructure(JCDL 2010) Motivation Background

83. 59 copies The Problem The Problem Internet Archive - Wayback Machine www.aircharter-international.com http://web.archive.org/web/*/http://www.aircharter-international.com Lexical Signature (TF/IDF) Charter Aircraft Cargo Passenger Jet Air Enquiry Title ACMI, Private Jet Charter, Private Jet Lease, Charter Flight Service: Air Charter International 39

84. The Problem The Problem www.aircharter-international.com Lexical Signature (TF/IDF) Charter Aircraft Cargo Passenger Jet Air Enquiry 40

85. The Problem The Problem www.aircharter-international.com Title ACMI, Private Jet Charter, Private Jet Lease, Charter Flight Service: Air Charter International 41

86. The Idea Contributions Compare performance of two automated methods to rediscover web pages Lexical signatures (LSs) Titles Evaluate performance of combination of methods and suggest workflow for real time web page rediscovery 42

88. Binary retrieval pattern, URI either within top 10 or undiscovered43

90. Google and Yahoo! return more URIs for non-quoted titles

91. Same binary retrieval pattern44

92. Combination of Methods Combination of Methods Top Results for Combination of Methods 45

95. Preferred primary method

96. 5-term LSs secondary method

97. Results in 75% top ranked URIsContributions: Provided evidence for suitability of titles and introduced web page discovery framework. 46

98. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 47 Is This a Good Title?(Hypertext 2010) Motivation Background

99. ??? The Problem The Problem http://www.drbartell.com/ Lexical Signature (TF/IDF) Plastic Surgeon Reconstructive Dr Bartell Symbol University 48

100. The Problem The Problem http://www.drbartell.com/ Title Thomas Bartell MD Board-Certified - Cosmetic Plastic Reconstructive Surgery 49

101. The Problem The Problem www.reagan.navy.mil Lexical Signature (TF/IDF) Ronald USS MCSN Torrey Naval Sea Commanding 50

102. The Problem The Problem www.reagan.navy.mil ??? Title Home Page Is This a Good Title? 51

103. The Idea Contributions Display title evolution over time Compare to content evolution “Normalize” time as fixed size windows Provide prediction model for title’s retrieval potential 52

105. Binary retrieval pattern, URI either within top 10 or undiscovered53

106. Title Evolution – Example I Title Evolution - Example I www.sun.com/solutions 1998-01-27 Sun Software Products Selector Guides - Solutions Tree 1999-02-20 Sun Software Solutions 2002-02-01 Sun Microsystems Products 2002-06-01 Sun Microsystems - Business & Industry Solutions 2003-08-01 Sun Microsystems - Industry & Infrastructure Solutions Sun Solutions 2004-02-02 Sun Microsystems – Solutions 2004-06-10 Gateway Page - Sun Solutions 2006-01-09 Sun Microsystems Solutions & Services 2007-01-03 Services & Solutions 2007-02-07 Sun Services & Solutions 2008-01-19 Sun Solutions 54

107. Title Evolution – Example II Title Evolution - Example II www.datacity.com/mainf.html 2002-10-16 computer company in Manassas Virginia sells Custom Built Computers with Removable Hard Drives Kits and Iomega 2GB Jaz Drives (jazz drives) October 2002 DataCity 800-326-5051 toll free 2006-03-14 Est1989 Computer company in Stafford Virginia sells Custom Built Secure Computers with DoD 5200.1-R Approved Removable Hard Drives, Hard Drive Kits and Iomega 2GB Jaz Drives (jazz drives), introduces the IllumiNite; lighted keyboard DataCity 800-326-5051 Service Disabled Veteran Owned Business SDVOB 2000-06-19 DataCityof Manassas Park Main Page 2000-10-12 DataCityof Manassas Park sells Custom Built Computers & Removable Hard Drives 2001-08-21 DataCitya computer company in Manassas Park sells Custom Built Computers & Removable Hard Drives 55

109. Extract available titles of past 14 years

110. Compute normalized Levenshtein edit distance between titles of copies and baseline (today)(0 = identical;1 = completely dissimilar)56

112. Decay from 2005 on (with fewer copies available)

113. 4 year old title:40% chance to be unchanged57

115. X: avg edit distance of corresponding titles

116. overlap indicated by:green: <10red: >90

117. Semi-transparent: total amount of points plotted58

119. Number of nouns, articles etc.

120. Amount of title terms, characters [Ntoulas2006]

121. Observation of re-occurring terms in poorly performing titles - “Stop Titles”home, index, home page, welcome, untitled document The performance of any given title can be predicted as insufficient if it consists to 75% or more of a “Stop Title”! 59

123. Not all titles equally good

124. If the majority of title terms are Stop Titles its quality can be predicted poorContribution: Quantified title evolution and introduced stop titles. 60

125. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 61 Motivation Background Find, New, Copy, Web, Page - Tagging for the (Re-)Discovery of Web Pages(TPDL 2011)

127. Titles BUT What if no archived/cached copy can be found? 62

128. The Solution? The Problem Conferences Digitallibraries Conference Library Jcdl2005 63

130. Test combination of methods to improve retrieval performance

131. Investigate “descriptive” power of tags64

133. Same four retrieval cases introduced earlier

134. nDCG w/ binary relevance scoring

135. Mean Average Precision65

136. The Experiment The Problem Combining methods 66

138. ~50% of tags do not occur in page [Bischoff2008]

139. “Secret”:

140. ~50% of tags do not occur in current version of page

141. ergo: How about previous versions?67

143. 66.3% of our tags do not occur in page

144. 4.9% of tags occur in previous version of page Ghost Tags

145. represent a previous version better than the current one

146. What kind of tags are these?

147. Important to the document, to the Delicious user?68

148. Ghost Tags The Problem Document importance: TF rank User importance: Delicious rank Normalized rank: 0 - top 1 - bottom 69

150. Combining tags with titles and LSs gains URIs

151. Ghost Tags exist!

152. 1/3 of them are important to the page and userContributions: Added tags to web page discovery framework and introduced notion of Ghost Tags. 70

153. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 71 Motivation Background Rediscovering Missing Web Pages Using Link Neighborhood Lexical Signatures(JCDL 2011)

155. Titles BUT What if no archived/cached copy can be found? Plan A: Tags 72

156. Plan B The Problem Link neighborhood Lexical Signatures (LNLSs) is about Computer Dominion Norfolk Monarch extract 73

158. Length

159. Number of backlinks

160. Backlink levels

161. Radius of terms on backlink page74

162. The Radius on a Backlink Page The Problem Entire page Paragraph Anchor text 75

164. IDF values from Yahoo!

165. 1..7 and 10 termsQuery Yahoo! API Compute “goodness” (nDCG) 76

166. The Results The Problem 1st and 2nd level level-radius-rank better 77

167. The Results – Radius The Problem All Radii level-radius-rank 78

168. The Results – Backlink Rank The Problem Ranks 10 100 1000 level-radius-rank 79

169. The Results – In Numbers The Problem GOOD 1-anchor-1000 WINNER 1-anchor-10 80

171. Parsed from top 10backlink pages

172. Include firstbacklink level only

173. Consider anchor text onlyContributions: Added LNLS to web page discovery framework. 81

174. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 82 Motivation Background Synchronicity – Automatically Rediscover Missing Web Pages in Real Time (JCDL 2011)

175. Synchronicity Concluding Remarks Firefox add-on Triggers on 404 error Rediscover page via: Memento Title Lexical signature Tags Link neighborhood lexical signature URI modification http://bit.ly/no-more-404 83

176. Contributions Concluding Remarks Introduce reliable real-time approach to estimate IDF values Workflow for generation of well performing lexical signatures Performance evaluation of web page titles Investigation of tags for web page discovery Analysis of link neighborhood lexical signatures and their optimal parameter Introduce Synchronicity implementing the entire framework 84

177. Concluding Remarks 85

178. Next Stop… New Mexico Concluding Remarks 86

179. List of my Relevant Publications Concluding Remarks M.Klein, M.L.Nelson, “A Comparison of Techniques for Estimating IDF Values to Generate Lexical Signatures for the Web“, WIDM 2008, pp. 39-46 M.Klein, M.L.Nelson, “Revisiting Lexical Signatures to (Re-)Discover Web Pages”, ECDL 2008, pp. 371-382 M.Klein, M.L.Nelson, “Correlation of Term Count and Document Frequency for Google N-Grams“, ECIR 2009, pp. 620-627 M.Klein, M.L.Nelson, “Inter-Search Engine Lexical Signature Performance“, JCDL 2009, pp. 413-414 M.Klein, M.L.Nelson, “Investigating the Change of Web Pages Titles Over Time“, InDP 2009 M.Klein, J.Shipman, M.L.Nelson, “Is This a Good Title”, Hypertext 2010, pp. 3-12 M.Klein, M.L.Nelson, “Evaluating Methods to Rediscover Missing Web Pages from the Web Infrastructure”, JCDL 2010, pp. 59-68 M.Klein, J.Ware, M.L.Nelson, “Rediscovering Missing Web Pages Using Link Neighborhood Lexical Signatures”, JCDL 2011 M.Klein, M.Aly, M.L.Nelson, “Synchronicity - Automatically Rediscover Missing Web Pages in Real Time”, JCDL 2011 M.Klein, M.L.Nelson, “Find, New, Copy, Web, Page – Tagging for the (Re-)Discovery of Web Pages”, TPDL 2011 to appear 87

180. References Concluding Remarks Bischoff2008 K.Bischoff, C.Firan, W.Nejdl, R.Paiu, “Can All Tags Be Used for Search?” In: Proceedings of CIKM '08, pp.193-202, 2008 Dellavalle2003 R.P.Dellavalle, E.J.Hester, L.F.Heilig, A.L.Drake, J.W.Kuntzman, M.Graber, L.M.Schilling, “Information Science: Going, Going, Gone: Lost Internet References”, Science 302(5646), pp.787-788, 2003 Jones1973 K.Spärck Jones, “Index Term Weighting”, Information Storage and Retrieval, pp. 619-633, 1973 Kahle1997 B.Kahle, “Preserving the Internet”, Scientific American 276, pp.82-83, 1997 Koehler2002 W.C.Koehler, “Web Page Change and Persistence - A Four-Year Longitudinal Study”, JASIST 53(2), pp.162-171, 2002 Lawrence2001 S.Lawrence, D.M.Pennock, G.W.Flake, R.Krovetz, F.M.Coetzee, E.Glover, F.A.Nielsen, A.Kruger, C.L.Giles, “Persistence of Web References in Scientic Research”, Computer 34(2), pp.26-31, 2001 McCown2005 F.McCown, S.Chan, M.L.Nelson, J.Bollen, “The Availability and Persistence of Web References in D-Lib Magazine”, Proceedings of IWAW '05, 2005 Nelson2002 M.L.Nelson, B.D.Allen, “Object Persistence and Availability in Digital Libraries”, D-Lib Magazine 8(1), 2002 Ntoulas2006 A. Ntoulas, M.Najork, M.Manasse, D.Fetterly, “Detecting Spam Web Pages Through Content Analysis”, Proceedings of WWW ’06, pp 83-92, 2006 Park2004 S.T.Park, D.M.Pennock, C.L.Giles, R.Krovetz, “Analysis of Lexical Signatures for Improving Information Persistence on the World Wide Web”, TOIS 22(4), pp.540-572, 2004 Phelps2000 T.A.Phelps, R.Wilensky, “Robust Hyperlinks Cost Just Five Words Each”, technical report, UC Berkeley, 2000 Sanderson2011 R.Sanderson, M.Phillips, H.Van de Sompel, “Analyzing the Persistence of Referenced Web Resources with Memento”, Proceedings of OR '11, 2011 88

181. Using the Web Infrastructure for Real Time Recovery of Missing Web Pages Martin Klein mklein@cs.odu.edu http://www.cs.odu.edu/~mklein/

182. Backup Slides

184. Find more Stop Titles

185. Find more Ghost Tags

186. Identify “Stop Anchors”

187. Synchronicity 1.0

188. Web service

191. Shown, screen scraping works but

192. missing validation of baseline (Google N-Grams)

193. N-Grams seem suitable (recently created, based on web pages) but provide TC and not DF  what is their relationship?93

195. 95 Experiment Results Investigate correlation between TC and DF within “Web as Corpus” (WaC) Rank similarity of all terms

196. 96 Experiment Results Investigate correlation between TC and DF within “Web as Corpus” (WaC) Spearman’s ρ and Kendall τ

197. 97 Experiment Results Top 10 terms in decreasing order of their TF/IDF values taken from http://ecir09.irit.fr U = 14 ∩ = 6 Strong indicator that TC can be used to estimate DF for web pages! Google: screen scraping DF values from the Google web interface

198. 98 Experiment Results Show similarity between WaC based TC and Google N-Gram based TC TC frequencies N-Grams have a threshold of 200

199. Experiment Results Frequency of TC/DF Ratio Within the WaC Integer Values Two Decimals One Decimal 99

201. TC frequencies of WaC and Google N-Grams are very similiar

202. N-Grams are suitable for accurate IDF estimation for web pages Does not mean everything correlated to TC can be used as DF substitute! 100

203. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 101 Inter-Search Engine Lexical Signature Performance (JCDL 2009) Motivation Background

204. Inter-Search EngineLexical Signature Performance http://en.wikipedia.org/wiki/Elephant Elephant Tusks Trunk African Loxodonta Elephant, Asian, African Species, Trunk Elephant, African, Tusks Asian, Trunk

205. 103

206. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 104 Motivation Background Synchronicity – Automatically Rediscover Missing Web Pages in Real Time (JCDL 2011)

208. Events reveal underlying pattern, framework bigger than any of the synchronous systems

209. Carl Gustav Jung (1875-1961)

210. “meaningful coincidence”

211. Deschamps – de Fontgibu plumpudding examplepicture from http://www.crystalinks.com/jung.html 105

212. Synchro…What? Repo Man (1984) http://www.imdb.com/title/tt0087995/ http://www.youtube.com/watch?v=X4HQyqc-aVU 106

213. Agenda LSs for Web Pages DF Estimation Techniques TC-DF Correlation Web Page Titles Synchronicity Link Neighborhood LSs Book of the Dead Web Page Tags 107 Motivation Background (Not yet published)

215. 233 URIs returning status 404

216. Mechanical Turk to determine “aboutness”

217. Guess from URI string

218. Mementos for 161 URIs

219. Apply lexical signatures and title108

220. 5-term LSs Titles 109 Experiment Results Dice Similarity Coefficient of Top 100 Results D = 0 0.0 < D ≤ 0.3 0.3 < D ≤ 0.6 0.6 < D ≤ 1.0

221. 5-term LSs Titles 110 Experiment Results Jaro Distance of Top 100 Results J = 0 0.0 < J ≤ 0.3 0.3 < J ≤ 0.6 0.6 < J ≤ 1.0

223. Top 10 only

224. Relevant

225. Somewhat relevant

226. Not relevant

227. Broken URI

228. nDCG of top 10 results111

229. 5-term LSs Titles 112 Experiment Results Relevance of Top 10 Results

230. 113 Experiment Results nDCG of Top 10 Results