Ariadne Harvesting

  • 1,234 views
Uploaded on

Harvesting metadata in repository networks, including ARIADNE, GLOBE, MACE and MELT

Harvesting metadata in repository networks, including ARIADNE, GLOBE, MACE and MELT

More in: Education , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,234
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
16
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb HARVESTING METADATA IN REPOSITORY NETWORKS Bram Vandeputte K.U.Leuven ARIADNE 1 1
  • 2. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb WHO AM I • Working since 2006 with Erik Duval • ARIADNE harvester, OAI-PMH target support, validation service, etc. • International projects : MACE, MELT, Globe, Ariadne, Aspect, iCoper,... • Other interests : Information visualisation, Interoperability, TEL 2 2
  • 3. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb Overview • OAI-PMH • Validation Service • Harvester Infrastructure • Projects • MACE • MELT • GLOBE • OAI-PMH Target 3 3
  • 4. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb OAI-PMH • Server - Client model • Pull mechanism • options : • selective harvesting • incremental harvesting 4 4
  • 5. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb OAI-PMH • Verbs : Identify, ListRecords, GetRecord • Parameters : • baseUrl • from & until date • metadataPrefix • sets 5 5
  • 6. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb Validation Service • Interoperability : Application Profile (AP) • Manual check : very time consuming • Need a tool for enforcing an AP => validation scheme • A set of validation rules • Reusable 6 6
  • 7. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb Validation Service LOM loose MELTv1.0/ MACEv1.0 minimal melt.xsd lomloose.xsd mace.xsd classification vocab service MELTv1.0/full vcard validator Legend empty attribute uses fields extends MACEv1.0 validationScheme conditional LRE taxon field fields check numerical vcard validator validation component 7 7
  • 8. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb 8 8
  • 9. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb The ARIADNE Harvester Infrastructure 8 8
  • 10. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester harvester log 8 8
  • 11. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository SPI SQI harvester log validation service 8 8
  • 12. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository SPI SQI External Repository OAI OAI-PMH LOM LOM LOM harvester log validation service 8 8
  • 13. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository OAI-PMH SPI SQI External Repository LOM LOM LOM OAI OAI-PMH LOM LOM LOM harvester log validation service 8 8
  • 14. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository OAI-PMH SPI SQI External Repository LOM LOM OAI OAI-PMH LOM LOM LOM LOM harvester log validation service 8 8
  • 15. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository OAI-PMH SPI SQI External Repository LOM LOM OAI OAI-PMH LOM LOM LOM LOM LOM harvester log validation service 8 8
  • 16. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository OAI-PMH SPI SQI External Repository LOM LOM OAI OAI-PMH LOM LOM LOM LOM Validation Msg harvester log validation service 8 8
  • 17. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository OAI-PMH SPI SQI External Repository LOM LOM OAI OAI-PMH Validation LOM Msg LOM LOM LOM harvester log validation service 8 8
  • 18. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository OAI-PMH SPI SQI External Repository LOM LOM OAI OAI-PMH LOM LOM LOM LOM Validation Msg harvester log validation service 8 8
  • 19. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository OAI-PMH SPI SQI External Repository LOM LOM OAI LOM OAI-PMH LOM LOM LOM Validation Msg harvester log validation service 8 8
  • 20. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb ARIADNE Harvester ARIADNE Repository OAI-PMH SPI SQI External Repository LOM LOM LOM OAI 2 6 LOM OAI-PMH 1 Validation LOM Msg LOM LOM LOM 4 5 3 Validation Validation LOM Msg Msg harvester log validation service 9 9
  • 21. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb Harvesting in networks • MACE • MELT • GLOBE 10 10
  • 22. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb MACE • Metadata for Architectural Contents in Europe • www.mace-project.eu • Harvesting, Searching, Enriching, (geo-) Tagging, attention metadata 11 11
  • 23. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb MACE Uses Lucene for index search WINDS Validation OAI-PMH service SPI Harvested metadata OAI-PMH SPI IRB ARIADNE Harvester OAI-PMH MACE metadata SPI OAI-PMH DYNAMO Enriched metadata CuminCAD 12 12
  • 24. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb MACE 13 13
  • 25. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb MELT • MELT : a Metadata Ecology for Learning and Teaching • Ministries of education across europe • enriching, translating, better quality metadata • practical test for scalability of harvester framework 14 14
  • 26. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb MELT Multilingual Vocabulary Thesaurus Registry Validation BMBWK EUN MEC RVP Enrichment service Toolkit EUN AMG OAI-PMH SQI ARIADNE C-H FWU MESC TLF Harvester EUN MDS SPI CFL INDIRE MSU UL-FRI ARIADNE OAI Harvester SQI (EUN) ARIADNE METADATASTORE EDUCATIO KUL NCTE XTEC 15 15
  • 27. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb 16 16
  • 28. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb GLOBE • Global Learning Objects Brokered Exchange • Worldwide network of networks of repositories • => Both SQI and OAI-PMH are used on various levels in the network 17 17
  • 29. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb GLOBE BMBWK EUN MEC RVP SQI UDDI OAI OAI-PMH Registry SQI ARIADNE C-H FWU MESC TLF SQI Client Harvester OAI CFL INDIRE MSU UL-FRI MELT SQI SQI Federated EDUCATIO KUL NCTE XTEC GLOBE Search SPI OAI OAI-PMH ARIADNE SQI Harvester WINDS DYNAMO ARIADNE Harvester MACE IRB CuminCAD OAI OAI OAI SQI SQI SQI SQI SQI SQI SQI LACLO NIME EUN EdNA CGIAR Pubelo Merlot OAI OAI SQI SQI SQI SQI SQI SQI ISKME KERIS III COSL LORNET Ariadne 18 18
  • 30. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb 19 19
  • 31. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb OAI-PMH target 20 20
  • 32. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb OAI-PMH Target Software 2. Mapping Process Repository metadata Repository LOM metadata 1. Get metadata - identifier - lom.general.identifier out of database - title - lom.general.title - url - lom.general.description - project description - lom.technical.location - ... - ... 3. Copy results in OAI-PMH service Repository OAI-PMH result DYNAMO LOM 4. Serve results ARIADNE DYNAMO LOM metadata LOM Harvester DYNAMO metadata Repository LOM metadata metadata 21 21
  • 33. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb • Questions ? 22 22
  • 34. ARIADNE K.U.Leuven www.ariadne-eu.org www.cs.kuleuven.be/~hmdb References • SPI : http://ariadne.cs.kuleuven.be/lomi/index.php/SimplePublishingInterface • SQI : http://ariadne.cs.kuleuven.be/lomi/index.php/LorInteroperability • IEEE LOM : http://ltsc.ieee.org/wg12/ • OAI-PMH : http://www.openarchives.org/ • OAI-PMH target wiki : http://ariadne.cs.kuleuven.be/lomi/index.php/Setting_Up_OAI-PMH 23 23