Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MementoMap Framework for Flexible and Adaptive Web Archive Profiling


Published on

In this work we propose MementoMap, a flexible and adaptive framework to efficiently summarize holdings of a web archive. We described a simple, yet extensible, file format suitable for MementoMap. We used the complete index of the comprising 5B mementos (archived web pages/files) to understand the nature and shape of its holdings. We generated MementoMaps with varying amount of detail from its HTML pages that have an HTTP status code of 200 OK. Additionally, we designed a single-pass, memory-efficient, and parallelization-friendly algorithm to compact a large MementoMap into a small one and an in-file binary search method for efficient lookup. We analyzed more than three years of MemGator (a Memento aggregator) logs to understand the response behavior of 14 public web archives. We evaluated MementoMaps by measuring their Accuracy using 3.3M unique URIs from MemGator logs. We found that a MementoMap of less than 1.5% Relative Cost (as compared to the comprehensive listing of all the unique original URIs) can correctly identify the presence or absence of 60% of the lookup URIs in the corresponding archive while maintaining 100% Recall (i.e., zero false negatives).

Published in: Internet
  • Be the first to comment

  • Be the first to like this

MementoMap Framework for Flexible and Adaptive Web Archive Profiling

  1. 1. MementoMap Framework for Flexible and Adaptive Web Archive Profiling Sawood Alam, Michele C. Weigle, and Michael L. Nelson Old Dominion University, Norfolk, VA, USA Fernando Melo, Daniel Bicho, and Daniel Gomes FCT:, Lisbon, Portugal @ibnesayeed @WebSciDL @PT_WebArchive Supported by NSF Grant IIS-1526700 JCDL '19, June 4, 2019, Fort Worth, Urbana-Champaign, Illinois
  2. 2. @ibnesayeed 2 $ memgator -a archives.json -f cdxj > | grep -v "^!" | cut -d '/' -f 3 | sort | uniq -c | sort -nr 198014 13548 1191 1044 953 525 225 221 23 $ memgator -a archives.json -f cdxj > | grep -v "^!" | cut -d '/' -f 3 | sort | uniq -c | sort -nr 410 2 2 1 Cross-archive Memento Lookup With MemGator
  3. 3. @ibnesayeed Memento Aggregator 3
  4. 4. @ibnesayeed Memento Aggregator 4
  5. 5. @ibnesayeed Memento Aggregator 5
  6. 6. @ibnesayeed Memento Aggregator 6
  7. 7. @ibnesayeed Memento Aggregator 7
  8. 8. @ibnesayeed Memento Aggregator 8
  9. 9. @ibnesayeed Broadcasting is Evil 9 From: Michael Nelson [] Sent: Wednesday, December 02, 2015 12:33 PM To: Jones, Gina Cc: Rourke, Patrick; Grotke, Abigail Subject: Re: WebSciDL Hi Gina, I'll investigate. memgator is software that one my students wrote, but I suspect the traffic you're seeing is b/c it is deployed in can you share the IP addr from where you're seeing the traffic? I presume the requests are for Memento TimeMaps? It should not being actually scraping HTML pages. regards, Michael On Wed, 2 Dec 2015, Jones, Gina wrote: > Hi Michael, we have a slight configuration issue with the current OW > set up for our webarchives. I think, from looking at the logs, that > "MemGator:1.0-rc3 <@WebSciDL>" is really causing some issues on our wayback. > Do you know who is running this scraper? Itʼs not part of memento is it? > > Gina Jones > Web Archiving Team > Library of Congress From: Ilya Kreymer <> Date: Wed, 2 Dec 2015 10:33:56 -0800 Subject: high traffic on oldweb! To: Herbert Van de Sompel <>, Sawood Alam <> Hi Herbert, Sawood, Herbert: Perhaps you are lucky that I am not using the LANL aggregator, as the traffic has gotten really high, and also I was asked to remove an archive due to the traffic it was causing temporarily.. I am thinking that ability to remove source archives quickly is an important aspect of an aggregator. Sawood: Hopefully yours will support something like this so I don't need to restart the container to change the archivelist ;) Ilya Broadcasting is wasteful, both clients & archives suffer!
  10. 10. @ibnesayeed Memento Lookup Routing 10 Let’s fix the broadcasting issue with a more informed routing.
  11. 11. @ibnesayeed MemGator Log Responses from Various Archives 11 93% of the requests made from MemGator to upstream archives were wasteful.
  12. 12. @ibnesayeed What is Archived in What is Accessed from MemGator? 12 Blind spot of a content-based profile Blind spot of a usage-based profile
  13. 13. @ibnesayeed If Only Archives Could Tell When to Ask Them ● Websites advertise their holdings using sitemap.xml, why can’t archives? ○ Archives have billions or even hundreds of billions URI-Ms ○ Such exhaustive lists would go stale very quickly ● How about robots.txt? ○ It is compact, but is exclusion format, it does not tell what the site has ○ It assumes a single domain, patterns are for paths (not the domain name) ● How about combining the two ideas? ○ Introducing MementoMap! 13
  14. 14. @ibnesayeed A MementoMap Example 14 !context [""] !id {uri: ""} !fields {keys: ["surt"], values: ["frequency"]} !meta {type: "MementoMap", name: "A Test Web Archive", year: 1996} !meta {updated_at: "2018-09-03T13:27:52Z"} * 54321/20000 com,* 10000+ org,arxiv)/ 100 org,arxiv)/* 2500~/900 org,arxiv)/pdf/* 0 uk,co,bbc)/images/* 300+/20-
  15. 15. @ibnesayeed SURTs Representation with Wildcard 15 Original SURTs did not have wildcards. In practice the common “http://(” prefix is removed.
  16. 16. @ibnesayeed Index Statistics 16 The Internet archive is about 150 times bigger than
  17. 17. @ibnesayeed Top TLDs 17 was created to archive sites of interest of Portuguese people. Over time web archives collect many things they didn’t intend to and miss a lot they would have liked to archive.
  18. 18. @ibnesayeed Who Would have Thought has 10K+ .онлайн Sites? 18 “.онлайн” (encoded as “xn--80asehdb”) is an IDN gTLD which means “.online”
  19. 19. @ibnesayeed Distribution of URI-Ms over URI-Rs in 19 70% mementos belong to only 30% URI-Rs.
  20. 20. @ibnesayeed URI-M vs. URI-R Summary of 20
  21. 21. @ibnesayeed Last two years are still in embargo period. Yearly URI-Rs, URI-Ms, and Status Codes in 21 Early years of data came from various other archives.
  22. 22. @ibnesayeed Cumulative Growth of URI-Ms and URI-Rs in 22 50% mementos were captured in the last two active years alone.
  23. 23. @ibnesayeed Most Archived URI-Rs in 23 Arquivo is obsessed with transparent single pixel images and corner graphics.
  24. 24. @ibnesayeed Unique Items With Exact Host and Path Depths 24 Where do we draw the line? 5+ or 10+ deep?
  25. 25. @ibnesayeed HxPx Host and Path Depth Statistics of 25
  26. 26. @ibnesayeed Shape of HxPx Key Tree of 26
  27. 27. @ibnesayeed Global HxPx Reduction Rate 27
  28. 28. @ibnesayeed Incremental Children Reduction Rate 28
  29. 29. @ibnesayeed Processed Lines vs. Compacted MementoMap Growth 29 com,example)/a/1/x com,example)/a/2 com,example)/a/3 com,example)/b/1 com,example)/b/2 com,example)/c/1 com,example)/a/* com,example)/b/1 com,example)/b/2 com,example)/c/1 com,example)/*
  30. 30. @ibnesayeed MementoMap Generation, Compaction, and Lookup 30 1.5% Relative Cost yields 60% Accuracy. can save 60% waisted traffic by publishing 119MB summary file!
  31. 31. @ibnesayeed Dissemination and Discovery Methods 31 GET /.well-known/mementomap HTTP/1.1 Host: Link: <>; rel="mementomap" <link href="" rel="mementomap"> Well-known URI Link Header Link HTML Element
  32. 32. @ibnesayeed Future Work ● Generate MementoMap on the whole index, not a sample ● Generate blacklists by processing access logs ● Incorporate MementoMap in replay systems ● Encourage archives and aggregators to adopt it 32
  33. 33. @ibnesayeed Conclusions ● Described MementoMap - a flexible and efficient archive profiling framework ● Analyzed complete index of to understand nature of web archives ● Evaluated MementoMap against’s index ● Save 60% of the wasted MemGator traffic with 1.5% cost (a 119 MB file) ● Proposed “mementomap” as a well-known URI suffix as well as a link relation for dissemination of MementoMap ● Implemented a single-pass, memory-efficient, and parallelization-friendly MementoMap generation/compaction algorithm ● Open-sourced the implementation ○ 33