Download to read offline
More Related Content
![Aja group presentation_4[1]](https://cdn.slidesharecdn.com/ss_thumbnails/ajagrouppresentation41-100912033003-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Link Persistence, Website Persistence
![Building Web Archiving Collaborations to Save [More of] the Web](https://cdn.slidesharecdn.com/ss_thumbnails/resaw2015perricci-150610172008-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
![METRO Conference 2014: How collaboration can save [more of] the web: recent p...](https://cdn.slidesharecdn.com/ss_thumbnails/metroslides-20140110-perricci-140116121524-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
More from nullhandle
Link Persistence, Website Persistence
- 1. Link Persistence, Website Persistence NicholasTaylor @nullhandle May 28, 2013 ―Forward‖ by Flickr user Hitchster under CC BY 2.0
- 2. why preserve theweb?
- 3. broken links ―404‖ byFlickr user adactio under CC BY 2.0
- 4. 44 days (Kahle,1997)
- 5. 75 days (Kahle,2001)
- 6. 100 days (Kahle,2003)
- 7. variable (Sanderson, Phillips,and Van de Sompel, 2011) • literature review of 17 studies • research focused on scholarly citations • decay rates of 39-82% • over periods of 1-13 years
- 8. ―Digital documents lastforever—or five years, whichever comes first.‖ (Jeff Rothenberg, 1997) ―Out of books sprout... plants‖ by DeviantArt user quinn.anya under CC BY-SA 2.0
- 9. LINK CHECKING The Artand Science of ―http Blue Background‖ by DeviantArt user SoulArt2012 under CC BY-NC-ND 3.0
- 10. http response codes •404: ―Not Found‖ • 200: ―OK‖ • 301: ―Moved Permanently‖ • 500: ―Internal Server Error‖
- 11. automated link checker ―LaMachine @ Yokohama‖ by Flickr user chidorian under CC BY-SA 2.0
- 12. what link checkingtells us ―200 ok‖ by Flickr user reidab under CC BY-NC-SA 2.0
- 13. possible scenarios • linkworks; same website • link works; different website – website may or may not still exist • link doesn’t work; website still exists • link doesn’t work; website no longer exists
- 14. link works; samewebsite http://www.fair.org/ (2002) http://www.fair.org/ (2013)
- 15. link works; differentwebsite… http://www.fb.com/ (2002) http://www.fb.com/ (2013)
- 16. …but website stillexists http://www.fb.org/ (2013)
- 17. link doesn’t work… http://www.state.mo.us/(2002) http://www.state.mo.us/ (2013)
- 18. …but website stillexists http://www.sos.mo.gov/ (2013)
- 19. link doesn’t work; websiteno longer exists
- 20. assumptions • link works;same website • link works; different website – website may or may not still exist • link doesn’t work; website still exists • link doesn’t work; website no longer exists
- 21. research questions • howmuch are we overestimating website persistence? – some working links point to different websites • how much are we underestimating website persistence? – websites may still exist even though links don’t work or do work but point to different websites
- 22. WEB ARCHIVES A StudyUsing
- 23. Library of Congress U.S.Election 2002 Web Archive
- 24. preparing the listof links • exclude links corresponding to electoral candidate websites • 1,071 links – state government – political parties – advocacy organizations – major newspapers – political blogs
- 25. methodology automated • run Heritrixagainst links, ignoring robots.txt • log http response codes • log redirects manual • manually check each link • same website behind working link? • does website still exist?
- 26. methodology automated • run Heritrixagainst links, ignoring robots.txt • log http response codes • log redirects manual • manually check each link • same website behind working link? • does website still exist?
- 27.
- 28. same website? 83% 9% 8% working link;same site non-working link working link; different site
- 29. non-working link; website stillexists? 91% 7% 2% 8% working still exists doesn't exist
- 30. website still exists? 94% 6% stillexists doesn't exist
- 31. summary of results •how much are we overestimating website persistence? – 8% of working links point to different websites • how much are we underestimating website persistence? – 82% of websites associated with non-working links still exist – 48% of websites whose links now point to different websites still exist
- 32. what does itmean? • websites are (much more) persistent than links • websites are surprisingly durable? ―Golden Spider Silk‖ by Flickr user amandabhslater under CC BY-SA 2.0
- 33. WEBSITE CHECKING? Beyond LinkChecking, ―Check‖ by Flickr user ex.libris under CC BY-NC-ND 2.0
- 34. building a websitechecker 1. check whether link still works 2. check whether link still corresponds to website 3. check whether website still exists
- 35. ―Most web archivingproblems are problems of scale.‖ (Kris Carpenter Negulescu, 2012) ―chutes and ladders‖ by Flickr user reallyboring under CC BY-NC-SA 2.0
- 36. building a websitechecker 1. check whether link still works 2. check whether link still corresponds to website 3. check whether website still exists
- 37. Heritrix compares checksums ―Fingerprint‖by Flickr user CPOA under CC BY-ND 2.0
- 38. …but checksums arelimited ―Hashing Emily‖ by Flickr user wlef70 under CC BY-NC-SA 3.0
- 39. visual analysis ofpage changes Pehlivan, Ben-Saad, and Gançarski: ―Vi-DIFF: Understanding Web Pages Changes‖
- 40. building a websitechecker 1. check whether link still works 2. check whether link still corresponds to website 3. check whether website still exists
- 41. lexical signature ofarchived page Ware, Klein, and Nelson: ―An Evaluation of Link Neighborhood Lexical Signatures to Rediscover Missing Web Pages‖
- 42. find archived pagesw/ Memento • http protocol enhancement • enables discovery of archived resources in distributed web archives
- 43. lexical signatures ofbacklink pages
- 44. ―The future isalready here; it’s just not very evenly distributed.‖ (William Gibson, 1999) ―Time Travel‖ by Flickr user xcalibr under CC BY-NC-ND 2.0
- 45. Nicholas Taylor @nullhandle ―Thank You‖by Flickr user muffintinmom under CC BY 2.0
Editor's Notes
- #3 We’ve been losing the web for as long as it’s existed; the first webpage, created by Tim Berners-Lee, exists as only a copy recreated a year after the original.http://www.w3.org/History/19921103-hypertext/hypertext/WWW/TheProject.html
- #4 Mainstream recognition of the once-esoteric “page not found” http response code reflects the popular perception of the ephemerality of the web
- #5 I started looking into the literature on link persistence in preparation for writing a blog post for the Library of Congress’ digital preservation blog, the Signal. Brewster Kahle, founder of the Internet Archive, has offered various numbers for the average lifespan of a webpage over the years. As someone trying to archive the entire public web, he seemed like someone who would know.
- #8 A meta-study of 17 other studies of link persistence suggested that links decay…but at all sorts of different frequencies.
- #9 The ambiguity about the ephemerality of web content is reminiscent of Rothenberg’s famous quote about the persistence of digital documents in general.
- #10 Now let’s take a look at the simplest automated approach to checking the persistence of links.
- #11 When the client’s browser requests the resource at a particular URL, the web server first sends an http response code, indicating the disposition of the resource at the requested URL. These are some common response codes.
- #12 An automated link checker, also known as a “spider” or “robot”, works by requesting a series of links and recording the response codes.https://secure.flickr.com/photos/chidorian/3461667159/
- #13 Response codes are limited, however; they can tell us about the disposition of content at the specified URL, but they can’t tell us what the content at the specified URL is.
- #14 Considering a link and a corresponding website over time, there are a number of possible scenarios when we go back to check on the persistence of both.
- #15 The most straight-forward case is where the link, the website, and their correspondence all persist.
- #16 Sometimes, however, the link still works, but it points to a different website.
- #17 That website may still exist at another URL.
- #18 Alternatively, maybe the link doesn’t work.
- #19 But the website that the link previously corresponded to may still exist at another URL.
- #20 Lastly, sometimes both the link doesn’t work and the website doesn’t exist.
- #21 These examples illustrate that link persistence and website persistence are two different things and that using the former as a proxy for the latter misses some of the possible scenarios.
- #22 Considering those scenarios, conflating link persistence with website persistence will result in systematic mis-measurements of website persistence. How significant are these mis-measurements?
- #23 Measuring website persistence requires knowing about the state of websites in the past, a perfect use case for web archives. I decided to do a study based on the web archives I was most familiar with, those of the Library of Congress.http://lcweb2.loc.gov/diglib/lcwa/html/lcwa-home.html
- #24 The U.S. Election 2002 Web Archive is one of their earliest web archive collections. The Library of Congress has archived U.S. national election websites every two years since 2000.
- #25 There were many more links in the collection than were utilized in this study. Links corresponding to electoral candidate websites were excluded given that they were universally short-lived and would skew the results.
- #26 The study consisted of two stages. First, we ran Heritrix against the prepared list of links and logged http response codes and redirects.
- #27 In the second stage, we manually visited each link and noted whether it was the same website as we had previously archived. If it was a different website or if the link didn’t work, we attempted to locate the new location of the previously-archived website using a search engine.
- #28 The link checker found that 91% of the links ultimately returned a “200” response code. The remaining 9% ultimately returned either “4xx” or “5xx” series response codes.
- #29 Bringing in the data on whether the working links still corresponded to the same websites, the percentage of working links that still correspond to the same site drops to 83%. Now, 8% of all the links are working links pointing to different sites.
- #30 Diving in on the non-working links, roughly 77% of the previously-archived websites still exist, even though the previously-archived links no longer point to them.
- #31 In aggregate, the percentage of websites that still exist after 10 years is 3% higher (94%) than link checking would’ve suggested (91%).This isn’t at all to say that web archiving isn’t important – if I included the candidate websites, the pie chart would suddenly show that less than half of the websites still existed. Also, for example, the White House website has existed for these last ten years, but specific content on the website has invariably disappeared.
- #32 The results suggest that we may be marginally overestimating website persistence by conflating working links with website persistence but greatly underestimating website persistence by conflating non-working links with websites that have disappeared.
- #33 The key caveat for these results is that I excluded from the study over 1,000 URLs in the web archive collection that all would have likely been both non-working links and websites that no longer existed. The remaining set of URLs represented those about which I more reasonably supposed there was a more typical probability that they would either persist or disappear.
- #34 We’re able to effectively perform link checking with current technologies. Can we come up with a better approach to checking the persistence of websites? Better understanding website persistence would facilitate better capacity planning (e.g., by reducing storage requirements for near-duplicate resources), inform capture frequency scheduling, and increase confidence that captured links corresponded to desired websites.
- #35 A website checker would need to be able to check links, too, but that functionality is already covered. What are the prospects for tools that could check link and website correspondence and check whether a website still exists?
- #36 In theory, these two latter tasks aren’t that difficult; it’s just that they need to be automated in order to be scalable.
- #37 Let’s look first at possible tools for checking link and website correspondence.
- #38 Heritrix already has the ability to compare the checksums of a resource at a particular URL over successive visits. This allows for an “absolute” assessment of sameness.
- #39 However, even the smallest change is enough to produce a checksum mis-match. We need a tool that can assess the magnitude or importance of the difference between successive versions, not just the fact of a difference.
- #40 The Vi-DIFF algorithm evaluates both the structure of a webpage and its segmented visual appearance to assess the magnitude of change. As a follow-on to a link checker, the algorithm could be calibrated to indicate whether it was the same site as previously visited or an entirely new one.
- #41 Now let’s look first at possible tools for checking website persistence, irrespective of link persistence.
- #42 The lexical signature is a set of keywords that are sufficiently descriptive and unique to be used in a search engine to dereference the page.
- #43 If the URL no longer works but exists in an archive, the lexical signature can be derived from the archived page and used to locate the new URL.
- #44 If the URL itself isn’t archived, the lexical signature can be derived from backlinks.
- #45 These tools exist but are not yet in wide use in the web archiving community. Wider utilization of these tools would allow us to better assess website persistence and the discrepancy with link persistence.