Web archiving challenges and opportunities

528 views
420 views

Published on

This is my presentation for job interview as web archiving engineer at Stanford university libraries on Oct 25.

Published in: Software, Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
528
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • The notable exceptions of Japanese  {Bengali, Vietnamese} and German Portuguese
  • Web archiving challenges and opportunities

    1. 1. WEB ARCHIVING CHALLENGES & OPPORTUNITIES PRESENTATIONFOR WEBARCHIVINGENGINEERINGPOSITION Ahmed AlSum PhD Candidate Old Dominion University
    2. 2. Outline • Engineering Experience • IBM • Old Dominion University • Internet Archive • Web Archiving Challenges & Opportunities • Selection • Harvesting • Storage • Access • Community • Conclusions
    3. 3. Cairo, Egypt 2006 - 2009
    4. 4. CCSP Project • An internal IBM support portal that provides client-facing audiences a by-client, holistic view of client situations • Technologies: WebSphere Portal, DB2, deployed on zLinux machines
    5. 5. Responsibilities • Software Engineer • Enterprise Applications with J2EE platform technologies for frontend (Servlets, JSP, Portlet APIs), and backend tasks based on EJB • Front-end components based on Web 20 technologies (AJAX based on dojo 1.0, and Java Script) • Lotus Sametime (Plugins and Bot development) • Software engineer team leader • Support project quality activities • Lead code review and static analysis activities
    6. 6. Responsibilities • Administrator • Deploying Portal solutions on WebSphere Portal • WebSphere Portal Administration for standalone and clustered environment • Administration on Linux and Windows OS • DB2 server administration for single instance and multiple instances with HADR support • Customer support team lead • Leading customer support activities
    7. 7. Certifications
    8. 8. Sharing IBM Internal Solutions with Broader Community
    9. 9. Norfolk, VA USA 2009 - 2013
    10. 10. Memento • Memento is an HTTP extension to integrate the Past and the Current Web I Jacobs and N Walsh Architecture of the world wide web Technical report, W3C, 2004 http://wwww3org/TR/webarch/ Now T1 T2 T3
    11. 11. Memento • Developer and administrator for Memento aggregator and proxies
    12. 12. Memento Clients • Memento currently is I-D draft, it is promoted to move to RFC soon.
    13. 13. San Francisco, CA USA 2012
    14. 14. WAT Extraction • Web Archive Transformation (WAT) is a specification for structuring metadata generated by Web crawls • Technologies:
    15. 15. WEB ARCHIVING Challenges and Opportunities
    16. 16. Web Archive Life Cycle Hockx-Yu, H, 2011 The Past Issue of the Web In Proceedings of 3rd International Conference on Web Science pp 1–8
    17. 17. Selection • Decide what to capture Everything, any domain National domains Delegate selection to partners Users’ favorites • We studied what is already captured
    18. 18. How Much Of The Web Is Archived? S. G. Ainsworth, A. AlSum, H. SalahEldeen, M. C. Weigle, and M. L. Nelson In Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, JCDL '11, Ottawa, Canada 2011 See also: http://arxiv.org/abs/1212.6177
    19. 19. Archive categories We have 3 categories of archives • Internet Archive (classic interface) • Search engine • Other archives Selection U K U S Public Archives, ca. Late 2010 / Early 2011
    20. 20. 1000 URIs Ordered by First Observation Date Selection See also: http://ws-dl.blogspot.com/2011/06/2011-06-23-how-much-of-web-is-archived.html
    21. 21. Memento Distribution, ordered by the first observation date
    22. 22. How Much of the Web is Archived? It Depends on Which Web… Selection Including SE cache Excluding SE Cache 90% 79% 97% 68% 88% 19% 35% 16% Changes since 2011: no more free SE APIs; greatly reduced IA quarantine period; 15 public web archives 2013 95% 92% 23% 26%
    23. 23. Profiling Web Archive Coverage For Top-level Domain And Content Language A. AlSum, M. C. Weigle, M. L. Nelson, and H. Van de Sompel In Proceedings of the 17th International Conference on Theory and Practice of Digital Libraries, TPDL 2013, 2013 See also: http://arxiv.org/abs/1309.4008
    24. 24. Where is it archived? Selection IA Internet Archive CAN Library and Archives Canada PO Portuguese Web Archive CZ Archive of the Czech Web LoC Library of Congress BL British Library CAT Web Archive of Catalonia TW National Taiwan University IC Icelandic Web Archive UK UK National Library CR Croatian Web Archive AIT Archive It
    25. 25. Language Coverage Selection IA Internet Archive CAN Library and Archives Canada PO Portuguese Web Archive CZ Archive of the Czech Web LoC Library of Congress BL British Library CAT Web Archive of Catalonia TW National Taiwan University IC Icelandic Web Archive UK UK National Library CR Croatian Web Archive AIT Archive It
    26. 26. Growth Rate Selection IA Internet Archive CAN Library and Archives Canada PO Portuguese Web Archive CZ Archive of the Czech Web LoC Library of Congress BL British Library CAT Web Archive of Catalonia TW National Taiwan University IC Icelandic Web Archive UK UK National Library CR Croatian Web Archive AIT Archive It Borrowed Portuguese material from IA Stopped archiving since 2008 Steady growth Stopped getting new URIs, but still crawling
    27. 27. Selection Research Output • Some portions of the web are not well archived such as India and Africa. • Profiling helping us in Memento query routing. • IIPC proposal with Herbert Van de Sompel (LANL) and David Rosenthal (SUL). Selection
    28. 28. Selection at SUL • Focus on the missing parts of the Web • Twitter - Crowdsource: • UK Web archive: Twittervana • Internet Memory: Collect URIs from twitter APIs • VA Tech: CTRNET project • Stanford Community • World News collection: 10 news website from each county • Tools: Selection
    29. 29. Web Archive Life Cycle Hockx-Yu, H, 2011 The Past Issue of the Web In Proceedings of 3rd International Conference on Web Science pp 1–8
    30. 30. Harvesting • Services • Archive-It • WAS @ CDLib • Dedicated servers • New tools See also: http://ws-dl.blogspot.com/2013/07/2013-07-10-warcreate-and-wail-warc.html
    31. 31. Special Harvesting Techniques • Borrow old materials from other web archives • Ex Stanford WebBase Project* • 260 TB • 7 Billion webpages Harvesting *http://www-diglib.stanford.edu/~testbed/doc2/WebBase/
    32. 32. Special Harvesting Techniques • Social Media • Focus on shared resources in the social media Harvesting Hany M SalahEldeen, Michael L Nelson, Losing My Revolution: How Many Resources Shared on Social Media Have Been Lost?, Proceedings of TPDL 2012 http://ws-dl.blogspot.com/2012/02/2012-02-11-losing-my-revolution-year.html
    33. 33. Special Harvesting Techniques • SiteStory - Transactional Archive Harvesting Justin F Brunelle, Michael L Nelson, Lyudmila Balakireva, Robert Sanderson, Herbert Van de Sompel, Evaluating the SiteStory Transactional Web Archive With the ApacheBench Tool, Proceedings of TPDL 2013 Sitestory: http://mementoweb.github.io/SiteStory/
    34. 34. Harvesting • Challenges • Ajax and Web 2.0/3.0 • Streaming Media • URI challenges • Mobile Harvesting http://blog.dshr.org/2012/05/harvesting-and-preserving-future-web.html http://netpreserve.org/sites/default/files/resources/OverviewFutureWebWorkshop.pdf
    35. 35. Web Archive Life Cycle Hockx-Yu, H, 2011 The Past Issue of the Web In Proceedings of 3rd International Conference on Web Science pp 1–8
    36. 36. Storage (Format) • Flat files: • WARC files (ISO standard) • No-SQL db: • Hbase at Internet memory* • Storage at SUL: • We need to use both Storage *Philippe Rigaux, Understanding HBase— The data model, IM technology blog http://internetmemoryorg/en/indexphp/synapse/understanding_the_hbase_data_model/
    37. 37. Storage (Infrastructure) • Wrong solution could be a disaster Storage
    38. 38. Web Archive Life Cycle Hockx-Yu, H, 2011 The Past Issue of the Web In Proceedings of 3rd International Conference on Web Science pp 1–8
    39. 39. Accessing Web Archive URI-Based WayBack Machine • Textbox to enter the requested URI • BubbleMap to show you the available mementos
    40. 40. Accessing Web Archive Full-text search • Challenges: Temporal Page Rank, Rank per site or memento, Date filtering
    41. 41. Accessing Web Archive • Thumbnail View • Trade-off between building the thumbnail in real time or pre-building Also, trade-off between representing the thumbnail by URI or by embedded binary data Can we build partial thumbnail map?
    42. 42. Accessing Web Archive • Title View • Trade-off between, extracting all the titles and keeping it as a metadata about the memento and extracting the title from the HTML content on the real time Implemented using Simile: http://www.simile-widgets.org/timeline/
    43. 43. Accessing Web Archive • Wayback Machine API • XML interface for the list of available Mementos
    44. 44. Accessing Web Archive • Web Page Snapshot Replay • URI rewriting, javascript, a nd embedded resources
    45. 45. Accessing Web Archive • Page Completeness Degree • The completeness degree could be calculated on the real time by using the preserved HTTP status for the embedded resources See also: http://arxiv.org/abs/1309.5503
    46. 46. Accessing Web Archive • Reconstructing web site • Current approach is using the web archive public interface.
    47. 47. Accessing Web Archive • Wayback Annotator • Create collections • Select and save relevant content to their collections • Annotate & mark important parts of archived web pages • Share their work and collaborate on archived content use http://netpreserve.org/sites/default/files/resources/Predstavitev_07.pdf http://netpreserve.org/sites/default/files/resources/Wayback_annotator_06.pdf
    48. 48. Accessing Web Archive Collection-Based • In addition to browsing the collection, you can browse the URIs in this collection • Research questions: Collection overview
    49. 49. Accessing Web Archive • Collection visualization • Term frequency algorithms should be normalized to take the mementos density in consideration http://ws-dl.blogspot.com/2012/08/2012-08-10-ms-thesis-visualizing.html
    50. 50. Accessing Web Archive • Web Archive analytics See also: http://ilpubs.stanford.edu:8090/1037/1/arcspread.pdf • ArcSpread took a query from the user, extracted related information and displayed the results in spread sheet style.
    51. 51. Who And What Links To The Internet Archive Y. Alnoamany, A. AlSum, M. C. Weigle, M. L. Nelson In Proceedings of 17th International Conference on Theory and Practice of Digital Libraries, TPDL 2013, 2013 (Best Student Paper) See also: http://arxiv.org/abs/1309.4016
    52. 52. Serving Robots! • Log files analysis using Apache Pig • Access to IA wayback machine as Robots outnumber Humans • 10:1 in terms of sessions, • 5:4 in terms of raw HTTP accesses • 4:1 in terms of megabytes transferred Access Sessions 10 1 HTTP accesses 5 4 MB Transferred 4 1
    53. 53. Where do Wayback Machine Users Come From? Website Percentage Description en.wikipedia.org 12.9% Wikipedia archive.org 11.9% IA Home Page reddit.com 10.2% Social News Web Site google.TLD 9.9% Search Engine info-poland.buffalo.edu 1.5% Polish Studies de.wikipedia.org 1.4% Wikipedia cracked.com 1.2% Humor Site snopes.com 1.1% Urban Legends Reference Pages facebook.com 0.9% Social Media crochetpatterncentral.com 0.9% Crocheting Hobbies Access
    54. 54. Most Languages Self-Link Access
    55. 55. ArcLink: Optimization Techniques To Build And Retrieve The Temporal Web Graph A. AlSum, M. L. Nelson IIPC GA 2013, Ljubljana, Slovenia In Proceedings of the 13th international ACM/IEEE joint conference on Digital libraries, JCDL '13, 2013 See also: http://arxiv.org/abs/1305.5959
    56. 56. Easy Solved Questions Q: What are the available mementos for vancouver2010.com? Access
    57. 57. Solved Questions, but hard Q: What are the HTML titles for vancouver2010com through time? A Page scraping for all mementos Access
    58. 58. Impossible Questions Q What are the anchor-text that pointed to www.vancouver2010.com through time? Access … <a href=www.vancouver2010.com > Vancouver Olympics </a> …. … <a href=www.vancouver2010.com > Winter Olympics </a> … … <a href=www.vancouver2010.com > Vancouver 2010 </a> …
    59. 59. ArcLink Access Google code: https://code.google.com/p/arcsys/
    60. 60. Impossible Questions • Q What are the anchor-text that pointed to www.vancouver2010.com through time? Access
    61. 61. Thumbnail Summarization Techniques For Web Archives A. AlSum, and M. L. Nelson Submitted for publication.
    62. 62. Thumbnails Access Internet Archive UK Web archive
    63. 63. Thumbnail Creation Challenges • Scalability in Time • IA may need 361 years to create thumbnail per each memento using one hundred machine • Scalability in Space • IA will need 355 TB to store 1 thumbnail per each memento • Page quality Access
    64. 64. How many thumbnails do we need? Access www.unfi.com on the live Web
    65. 65. How many thumbnails do we need? Access www.unfi.com on the live Web
    66. 66. 40 Thumbnails are good. Access
    67. 67. Same technique applied to apple.com Access
    68. 68. From 8000 Mementos to 69 Thumbnails. Access
    69. 69. iTunes cover application Access
    70. 70. Community • I suggest to be a member in IIPC • Join the open Wayback Machine team • Join the Winter Olympics 2014 collaborative project, even as an observer
    71. 71. Community • Web Archiving Workshops WAC 2011, Ottawa, Canada WAC 2012, Stanford, CA, USA WADL 2013, Indianapolis, IN, USATempWeb 2013, Rio de Janeiro, Brazil
    72. 72. Tools to SUL Web Archive • Selection • Harvest • Analysis • Access
    73. 73. Conclusions • Be Selective: Cover missing parts of the Web • Be Older: Include WebBase • Be Smart: Innovative services • Be Helpful: Researcher Framework/Dataset • Be Active: Participate in the WA communities • Make a difference aalsum@cs.odu.edu @aalsum
    74. 74. BACKUP
    75. 75. What is missing? IA Internet Archive CAN Library and Archives Canada PO Portuguese Web Archive CZ Archive of the Czech Web LoC Library of Congress BL British Library CAT Web Archive of Catalonia TW National Taiwan University IC Icelandic Web Archive UK UK National Library CR Croatian Web Archive AIT Archive It
    76. 76. Thumbnail Features SimHash DOM tree Embedded resources Datetime
    77. 77. Clustering technique

    ×