This talk describes an Infovore, a tool that uses the Map/Reduce approach to clean up, filter and combine RDF data sets to deliver purpose-built data sets for practical consumers of linked data

1. Infovore, an Open-Source Map/Reduce
Framework For Processing Graph
Data
Paul Houle
Ontology2

5. 2+ billion facts, 20+ gb!

6. the data your project needs

7. Why handle complete data sets?
Quality Perimeter
Infovore

8. RDF Tools vs.Invalid Triples
Image cc-by from arj03

9. Scaling Limits of Triple Stores
CPU Main Memory
CPU
CPU
CPU
CPU
CPU
Random-access bottleneck
Hard Drive or Flash Storage

10. Map/Reduce conserves memory!
Image cc-by-sa from Anua22a

11. Partitioning Data
md5(“http://dbpedia.org/resource/Tree”) =
b78f8f508982ceb4e8dd3510fac75f62
331 332330 333 334 335… …

12. If you really try it…
331
332330
333
334 335
… …

13. Preprocessing Freebase
• Expand prefixes
• Remove
• fbase:type.type.instance
• fbase:type.type.expected_by
• rdfs:type w/ fbase:* subject
• Reverse
• Fbase:type.permission.controls
• Fbase:dataworld_gardening_hint.replaced_by
• Rewrite
• Fbase:type.object.type to rdfs:type

14. Parallel Super Eyeball

15. sort | uniq
:Surgeon a :Occupation .
:Surgeon rdfs:label “Surgeon” @en.
:Surgeon :mustHave :Md.
:Tree a :Plant .
:Tree rfs:label “Tree” @en .
:Tree :has :Leaves .
:Victory a :AbstractConcept .
:Vectory rdfs:label “Victory” .
:Victory :emotialTone :Positive .

16. Huge scalability…
:Tree
:Victory
:Surgeon
Main memory

17. Pig, Hadoop and All That…
Source: http://www.dbis.informatik.hu-berlin.de/forschung/projekte/query-optimization-in-rdf-databases.html

18. Monitoring for Quality Control
Operational Statistics(rdf)
Preprocess Partition Clean Sort Classify Filter

19. :basekb

20. Parallel Loading into Triple Stores
331 332330 333 334 335… …
Openlink Virtuoso
4x Speedup

21. :basekb lite
:Freebase
:Chosenfacts
:Rulebox
:Chosentopics

22. rdf diff

23. See for yourself
https://github.com/paulhoule/infovore/wiki