SlideShare a Scribd company logo
1 of 14
Download to read offline
“ArchioNet”
Israeli Internet Domain
Archive
Agenda

o NLI Digital Library Infrastructure
o “ArchioNet” Project Scope
o Technical Issues
o The Project in Numbers
o Legislation
o What’s Next
NLI Digital Library Infrastructure
“ArchioNet” Project scope
•
•

Why do we need this project ?
What do we harvest?

•
•

•

Phase A : “Way back machine” in NLI Only , “Archionet” Only.
Phase B : Over the Web , Cross Reference Discovery.

When we started?

•
•
•

Phase b : Hebrew characters sites

How to enable accessibility:

•
•
•

Phase A : .IL web site

Phase A : 2 full crawl annually started September 2013
Phase B : additional 4 subject based crawl annually.

Where to execute the harvest ?

•
•

Phase A : NLI with Internet Archive.
Phase B : NLI Infrastructure
Technical Issues

•
•
•

Which Crawler ( version ) to use ?
Cataloguing and Search tool
What to harvest ?

•
•
•
•
•
•
•

Seeds is needed

Depth of a site
Robots.txt
The Deep Web

How to store and preserve a WARC file
Virus Detection
System Architecture
The Project in Numbers

• ~220K web sits
• 0.5 Giga byte/Site
• ~100 Tera / Harvest
• Avg page lifetime ~ 100 days
• 2 Full Harvest - Annually
Legislation

• Can NLI Harvest
• Where is it accessible ?
• Intellectual Properties
• What can/should we block ?
Thank You

Back
F1 hadar miller__israeli_internet_archive-nli
F1 hadar miller__israeli_internet_archive-nli
F1 hadar miller__israeli_internet_archive-nli
F1 hadar miller__israeli_internet_archive-nli
F1 hadar miller__israeli_internet_archive-nli

More Related Content

More from evaminerva

G12 susan hazan_roundtableopenaccesjewish
G12 susan hazan_roundtableopenaccesjewishG12 susan hazan_roundtableopenaccesjewish
G12 susan hazan_roundtableopenaccesjewishevaminerva
 
G11 alex valdman_yerushaproject
G11 alex valdman_yerushaprojectG11 alex valdman_yerushaproject
G11 alex valdman_yerushaprojectevaminerva
 
G11 alex valdman_yerushaproject
G11 alex valdman_yerushaprojectG11 alex valdman_yerushaproject
G11 alex valdman_yerushaprojectevaminerva
 
G10 ronit gadish_alexandervainer_hebrewterminology
G10 ronit gadish_alexandervainer_hebrewterminologyG10 ronit gadish_alexandervainer_hebrewterminology
G10 ronit gadish_alexandervainer_hebrewterminologyevaminerva
 
G10 ronit gadish_alexandervainer_hebrewterminology
G10 ronit gadish_alexandervainer_hebrewterminologyG10 ronit gadish_alexandervainer_hebrewterminology
G10 ronit gadish_alexandervainer_hebrewterminologyevaminerva
 
G8 seroussi sprinzak_mappingjewishculture
G8 seroussi sprinzak_mappingjewishcultureG8 seroussi sprinzak_mappingjewishculture
G8 seroussi sprinzak_mappingjewishcultureevaminerva
 
G8 seroussi sprinzak_mappingjewishculture
G8 seroussi sprinzak_mappingjewishcultureG8 seroussi sprinzak_mappingjewishculture
G8 seroussi sprinzak_mappingjewishcultureevaminerva
 
G7 menahem katz_hillelgershuni_textualvariants
G7 menahem katz_hillelgershuni_textualvariantsG7 menahem katz_hillelgershuni_textualvariants
G7 menahem katz_hillelgershuni_textualvariantsevaminerva
 
G7 menahem katz_hillelgershuni_textualvariants
G7 menahem katz_hillelgershuni_textualvariantsG7 menahem katz_hillelgershuni_textualvariants
G7 menahem katz_hillelgershuni_textualvariantsevaminerva
 
G6 jonathan bendovsqe_minerva 2016
G6 jonathan bendovsqe_minerva 2016G6 jonathan bendovsqe_minerva 2016
G6 jonathan bendovsqe_minerva 2016evaminerva
 
G5 orit rosengarten_leonlevy_dl_deadseascrolls
G5 orit rosengarten_leonlevy_dl_deadseascrollsG5 orit rosengarten_leonlevy_dl_deadseascrolls
G5 orit rosengarten_leonlevy_dl_deadseascrollsevaminerva
 
G5 orit rosengarten_leonlevy_dl_deadseascrolls
G5 orit rosengarten_leonlevy_dl_deadseascrollsG5 orit rosengarten_leonlevy_dl_deadseascrolls
G5 orit rosengarten_leonlevy_dl_deadseascrollsevaminerva
 
G3 stoeck and_hayim_lapin_nextgenerationculturalheritage
G3 stoeck and_hayim_lapin_nextgenerationculturalheritageG3 stoeck and_hayim_lapin_nextgenerationculturalheritage
G3 stoeck and_hayim_lapin_nextgenerationculturalheritageevaminerva
 
G3 stoeck and_hayim_lapin_nextgenerationculturalheritage
G3 stoeck and_hayim_lapin_nextgenerationculturalheritageG3 stoeck and_hayim_lapin_nextgenerationculturalheritage
G3 stoeck and_hayim_lapin_nextgenerationculturalheritageevaminerva
 
G2 michale satlow_inscriptionsisraelpalestine
G2 michale satlow_inscriptionsisraelpalestineG2 michale satlow_inscriptionsisraelpalestine
G2 michale satlow_inscriptionsisraelpalestineevaminerva
 
G2 michale satlow_inscriptionsisraelpalestine
G2 michale satlow_inscriptionsisraelpalestineG2 michale satlow_inscriptionsisraelpalestine
G2 michale satlow_inscriptionsisraelpalestineevaminerva
 
F3 sigal arieerez_reconnectingpast_evaminerva2016
F3 sigal arieerez_reconnectingpast_evaminerva2016F3 sigal arieerez_reconnectingpast_evaminerva2016
F3 sigal arieerez_reconnectingpast_evaminerva2016evaminerva
 
F3 sigal arieerez_reconnectingpast_evaminerva2016
F3 sigal arieerez_reconnectingpast_evaminerva2016F3 sigal arieerez_reconnectingpast_evaminerva2016
F3 sigal arieerez_reconnectingpast_evaminerva2016evaminerva
 
F2 kepa rodriguez_ehri_integration_retrieva_minerva_2016
F2 kepa rodriguez_ehri_integration_retrieva_minerva_2016F2 kepa rodriguez_ehri_integration_retrieva_minerva_2016
F2 kepa rodriguez_ehri_integration_retrieva_minerva_2016evaminerva
 
F1 haim gertner_ehri_and_yadvasheminternationalcooperatio
F1 haim gertner_ehri_and_yadvasheminternationalcooperatioF1 haim gertner_ehri_and_yadvasheminternationalcooperatio
F1 haim gertner_ehri_and_yadvasheminternationalcooperatioevaminerva
 

More from evaminerva (20)

G12 susan hazan_roundtableopenaccesjewish
G12 susan hazan_roundtableopenaccesjewishG12 susan hazan_roundtableopenaccesjewish
G12 susan hazan_roundtableopenaccesjewish
 
G11 alex valdman_yerushaproject
G11 alex valdman_yerushaprojectG11 alex valdman_yerushaproject
G11 alex valdman_yerushaproject
 
G11 alex valdman_yerushaproject
G11 alex valdman_yerushaprojectG11 alex valdman_yerushaproject
G11 alex valdman_yerushaproject
 
G10 ronit gadish_alexandervainer_hebrewterminology
G10 ronit gadish_alexandervainer_hebrewterminologyG10 ronit gadish_alexandervainer_hebrewterminology
G10 ronit gadish_alexandervainer_hebrewterminology
 
G10 ronit gadish_alexandervainer_hebrewterminology
G10 ronit gadish_alexandervainer_hebrewterminologyG10 ronit gadish_alexandervainer_hebrewterminology
G10 ronit gadish_alexandervainer_hebrewterminology
 
G8 seroussi sprinzak_mappingjewishculture
G8 seroussi sprinzak_mappingjewishcultureG8 seroussi sprinzak_mappingjewishculture
G8 seroussi sprinzak_mappingjewishculture
 
G8 seroussi sprinzak_mappingjewishculture
G8 seroussi sprinzak_mappingjewishcultureG8 seroussi sprinzak_mappingjewishculture
G8 seroussi sprinzak_mappingjewishculture
 
G7 menahem katz_hillelgershuni_textualvariants
G7 menahem katz_hillelgershuni_textualvariantsG7 menahem katz_hillelgershuni_textualvariants
G7 menahem katz_hillelgershuni_textualvariants
 
G7 menahem katz_hillelgershuni_textualvariants
G7 menahem katz_hillelgershuni_textualvariantsG7 menahem katz_hillelgershuni_textualvariants
G7 menahem katz_hillelgershuni_textualvariants
 
G6 jonathan bendovsqe_minerva 2016
G6 jonathan bendovsqe_minerva 2016G6 jonathan bendovsqe_minerva 2016
G6 jonathan bendovsqe_minerva 2016
 
G5 orit rosengarten_leonlevy_dl_deadseascrolls
G5 orit rosengarten_leonlevy_dl_deadseascrollsG5 orit rosengarten_leonlevy_dl_deadseascrolls
G5 orit rosengarten_leonlevy_dl_deadseascrolls
 
G5 orit rosengarten_leonlevy_dl_deadseascrolls
G5 orit rosengarten_leonlevy_dl_deadseascrollsG5 orit rosengarten_leonlevy_dl_deadseascrolls
G5 orit rosengarten_leonlevy_dl_deadseascrolls
 
G3 stoeck and_hayim_lapin_nextgenerationculturalheritage
G3 stoeck and_hayim_lapin_nextgenerationculturalheritageG3 stoeck and_hayim_lapin_nextgenerationculturalheritage
G3 stoeck and_hayim_lapin_nextgenerationculturalheritage
 
G3 stoeck and_hayim_lapin_nextgenerationculturalheritage
G3 stoeck and_hayim_lapin_nextgenerationculturalheritageG3 stoeck and_hayim_lapin_nextgenerationculturalheritage
G3 stoeck and_hayim_lapin_nextgenerationculturalheritage
 
G2 michale satlow_inscriptionsisraelpalestine
G2 michale satlow_inscriptionsisraelpalestineG2 michale satlow_inscriptionsisraelpalestine
G2 michale satlow_inscriptionsisraelpalestine
 
G2 michale satlow_inscriptionsisraelpalestine
G2 michale satlow_inscriptionsisraelpalestineG2 michale satlow_inscriptionsisraelpalestine
G2 michale satlow_inscriptionsisraelpalestine
 
F3 sigal arieerez_reconnectingpast_evaminerva2016
F3 sigal arieerez_reconnectingpast_evaminerva2016F3 sigal arieerez_reconnectingpast_evaminerva2016
F3 sigal arieerez_reconnectingpast_evaminerva2016
 
F3 sigal arieerez_reconnectingpast_evaminerva2016
F3 sigal arieerez_reconnectingpast_evaminerva2016F3 sigal arieerez_reconnectingpast_evaminerva2016
F3 sigal arieerez_reconnectingpast_evaminerva2016
 
F2 kepa rodriguez_ehri_integration_retrieva_minerva_2016
F2 kepa rodriguez_ehri_integration_retrieva_minerva_2016F2 kepa rodriguez_ehri_integration_retrieva_minerva_2016
F2 kepa rodriguez_ehri_integration_retrieva_minerva_2016
 
F1 haim gertner_ehri_and_yadvasheminternationalcooperatio
F1 haim gertner_ehri_and_yadvasheminternationalcooperatioF1 haim gertner_ehri_and_yadvasheminternationalcooperatio
F1 haim gertner_ehri_and_yadvasheminternationalcooperatio
 

Recently uploaded

4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptxmary850239
 
Mythology Quiz-4th April 2024, Quiz Club NITW
Mythology Quiz-4th April 2024, Quiz Club NITWMythology Quiz-4th April 2024, Quiz Club NITW
Mythology Quiz-4th April 2024, Quiz Club NITWQuiz Club NITW
 
Q-Factor General Quiz-7th April 2024, Quiz Club NITW
Q-Factor General Quiz-7th April 2024, Quiz Club NITWQ-Factor General Quiz-7th April 2024, Quiz Club NITW
Q-Factor General Quiz-7th April 2024, Quiz Club NITWQuiz Club NITW
 
How to Fix XML SyntaxError in Odoo the 17
How to Fix XML SyntaxError in Odoo the 17How to Fix XML SyntaxError in Odoo the 17
How to Fix XML SyntaxError in Odoo the 17Celine George
 
4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptx4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptxmary850239
 
CHEST Proprioceptive neuromuscular facilitation.pptx
CHEST Proprioceptive neuromuscular facilitation.pptxCHEST Proprioceptive neuromuscular facilitation.pptx
CHEST Proprioceptive neuromuscular facilitation.pptxAneriPatwari
 
Daily Lesson Plan in Mathematics Quarter 4
Daily Lesson Plan in Mathematics Quarter 4Daily Lesson Plan in Mathematics Quarter 4
Daily Lesson Plan in Mathematics Quarter 4JOYLYNSAMANIEGO
 
CLASSIFICATION OF ANTI - CANCER DRUGS.pptx
CLASSIFICATION OF ANTI - CANCER DRUGS.pptxCLASSIFICATION OF ANTI - CANCER DRUGS.pptx
CLASSIFICATION OF ANTI - CANCER DRUGS.pptxAnupam32727
 
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnvESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnvRicaMaeCastro1
 
Grade Three -ELLNA-REVIEWER-ENGLISH.pptx
Grade Three -ELLNA-REVIEWER-ENGLISH.pptxGrade Three -ELLNA-REVIEWER-ENGLISH.pptx
Grade Three -ELLNA-REVIEWER-ENGLISH.pptxkarenfajardo43
 
Unraveling Hypertext_ Analyzing Postmodern Elements in Literature.pptx
Unraveling Hypertext_ Analyzing  Postmodern Elements in  Literature.pptxUnraveling Hypertext_ Analyzing  Postmodern Elements in  Literature.pptx
Unraveling Hypertext_ Analyzing Postmodern Elements in Literature.pptxDhatriParmar
 
How to Manage Buy 3 Get 1 Free in Odoo 17
How to Manage Buy 3 Get 1 Free in Odoo 17How to Manage Buy 3 Get 1 Free in Odoo 17
How to Manage Buy 3 Get 1 Free in Odoo 17Celine George
 
Oppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and FilmOppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and FilmStan Meyer
 
Indexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdfIndexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdfChristalin Nelson
 
ARTERIAL BLOOD GAS ANALYSIS........pptx
ARTERIAL BLOOD  GAS ANALYSIS........pptxARTERIAL BLOOD  GAS ANALYSIS........pptx
ARTERIAL BLOOD GAS ANALYSIS........pptxAneriPatwari
 
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...Nguyen Thanh Tu Collection
 
Expanded definition: technical and operational
Expanded definition: technical and operationalExpanded definition: technical and operational
Expanded definition: technical and operationalssuser3e220a
 

Recently uploaded (20)

Paradigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTAParadigm shift in nursing research by RS MEHTA
Paradigm shift in nursing research by RS MEHTA
 
4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx
 
Mythology Quiz-4th April 2024, Quiz Club NITW
Mythology Quiz-4th April 2024, Quiz Club NITWMythology Quiz-4th April 2024, Quiz Club NITW
Mythology Quiz-4th April 2024, Quiz Club NITW
 
Q-Factor General Quiz-7th April 2024, Quiz Club NITW
Q-Factor General Quiz-7th April 2024, Quiz Club NITWQ-Factor General Quiz-7th April 2024, Quiz Club NITW
Q-Factor General Quiz-7th April 2024, Quiz Club NITW
 
How to Fix XML SyntaxError in Odoo the 17
How to Fix XML SyntaxError in Odoo the 17How to Fix XML SyntaxError in Odoo the 17
How to Fix XML SyntaxError in Odoo the 17
 
4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptx4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptx
 
CHEST Proprioceptive neuromuscular facilitation.pptx
CHEST Proprioceptive neuromuscular facilitation.pptxCHEST Proprioceptive neuromuscular facilitation.pptx
CHEST Proprioceptive neuromuscular facilitation.pptx
 
Mattingly "AI & Prompt Design: Large Language Models"
Mattingly "AI & Prompt Design: Large Language Models"Mattingly "AI & Prompt Design: Large Language Models"
Mattingly "AI & Prompt Design: Large Language Models"
 
Daily Lesson Plan in Mathematics Quarter 4
Daily Lesson Plan in Mathematics Quarter 4Daily Lesson Plan in Mathematics Quarter 4
Daily Lesson Plan in Mathematics Quarter 4
 
CLASSIFICATION OF ANTI - CANCER DRUGS.pptx
CLASSIFICATION OF ANTI - CANCER DRUGS.pptxCLASSIFICATION OF ANTI - CANCER DRUGS.pptx
CLASSIFICATION OF ANTI - CANCER DRUGS.pptx
 
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnvESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
 
Grade Three -ELLNA-REVIEWER-ENGLISH.pptx
Grade Three -ELLNA-REVIEWER-ENGLISH.pptxGrade Three -ELLNA-REVIEWER-ENGLISH.pptx
Grade Three -ELLNA-REVIEWER-ENGLISH.pptx
 
Unraveling Hypertext_ Analyzing Postmodern Elements in Literature.pptx
Unraveling Hypertext_ Analyzing  Postmodern Elements in  Literature.pptxUnraveling Hypertext_ Analyzing  Postmodern Elements in  Literature.pptx
Unraveling Hypertext_ Analyzing Postmodern Elements in Literature.pptx
 
How to Manage Buy 3 Get 1 Free in Odoo 17
How to Manage Buy 3 Get 1 Free in Odoo 17How to Manage Buy 3 Get 1 Free in Odoo 17
How to Manage Buy 3 Get 1 Free in Odoo 17
 
Oppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and FilmOppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and Film
 
Indexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdfIndexing Structures in Database Management system.pdf
Indexing Structures in Database Management system.pdf
 
ARTERIAL BLOOD GAS ANALYSIS........pptx
ARTERIAL BLOOD  GAS ANALYSIS........pptxARTERIAL BLOOD  GAS ANALYSIS........pptx
ARTERIAL BLOOD GAS ANALYSIS........pptx
 
prashanth updated resume 2024 for Teaching Profession
prashanth updated resume 2024 for Teaching Professionprashanth updated resume 2024 for Teaching Profession
prashanth updated resume 2024 for Teaching Profession
 
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
 
Expanded definition: technical and operational
Expanded definition: technical and operationalExpanded definition: technical and operational
Expanded definition: technical and operational
 

F1 hadar miller__israeli_internet_archive-nli

  • 2. Agenda o NLI Digital Library Infrastructure o “ArchioNet” Project Scope o Technical Issues o The Project in Numbers o Legislation o What’s Next
  • 3. NLI Digital Library Infrastructure
  • 4. “ArchioNet” Project scope • • Why do we need this project ? What do we harvest? • • • Phase A : “Way back machine” in NLI Only , “Archionet” Only. Phase B : Over the Web , Cross Reference Discovery. When we started? • • • Phase b : Hebrew characters sites How to enable accessibility: • • • Phase A : .IL web site Phase A : 2 full crawl annually started September 2013 Phase B : additional 4 subject based crawl annually. Where to execute the harvest ? • • Phase A : NLI with Internet Archive. Phase B : NLI Infrastructure
  • 5. Technical Issues • • • Which Crawler ( version ) to use ? Cataloguing and Search tool What to harvest ? • • • • • • • Seeds is needed Depth of a site Robots.txt The Deep Web How to store and preserve a WARC file Virus Detection System Architecture
  • 6. The Project in Numbers • ~220K web sits • 0.5 Giga byte/Site • ~100 Tera / Harvest • Avg page lifetime ~ 100 days • 2 Full Harvest - Annually
  • 7. Legislation • Can NLI Harvest • Where is it accessible ? • Intellectual Properties • What can/should we block ?