Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hobbit presentation at Apache Big Data Europe 2016

528 views

Published on

Hobbit presentation at Apache Big Data Europe 2016.

This work was supported by grants from the EU H2020 Framework Programme provided for the project HOBBIT (GA no. 688227).

Published in: Science
  • Be the first to comment

  • Be the first to like this

Hobbit presentation at Apache Big Data Europe 2016

  1. 1. Unified Benchmarking of Big Data Platforms The HOBBIT Platform Axel-Cyrille Ngonga Ngomo Horizon 2020 GA No 688227 01/12/2016–30/11/2018 Apache Big Data Sevilla, Spain November 15, 2016 Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 1 / 44
  2. 2. Summary Rationale A community-driven unified benchmarking platform for the community Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 2 / 44
  3. 3. Summary Rationale A community-driven unified benchmarking platform for the community Focus on Big (Linked) Data Provide benchmarks and baselines Provide reference implementation of KPIs Extensible and referenceable Result analysis Open-Source http://project-hobbit.eu @hobbit_project Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 2 / 44
  4. 4. A Lot of Data 1 1http://www.ibmbigdatahub.com/infographic/four-vs-big-data Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 3 / 44
  5. 5. A Lot of Tools 2 2https://cloudramblings.me/ Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 4 / 44
  6. 6. A Lot ... of Tools Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 5 / 44
  7. 7. Questions Developers: How good is my tool? Vendors: Who is my tool good for? Users: Which tool(s) should I use for my application? Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 6 / 44
  8. 8. Many Questions Where are the current bottlenecks? Which steps of the data lifecycle are critical? Which solutions are available? Which key performance indicators are relevant? How well do or should tools perform? How do existing solutions perform w.r.t. relevant indicators? Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 7 / 44
  9. 9. A Lot of Views 4 4https://steemit.com/philosophy/@l0k1/ subjectivity-and-truth-how-blockchains-model-consensus-buildingNgonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 8 / 44
  10. 10. Solution Benchmark Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 9 / 44
  11. 11. Solution Benchmark Components Dataset(s), e.g., Twitter stream, sensor data Task(s), e.g., entity recognition, storage Performance indicators, e.g., precision, recall, queries per second Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 9 / 44
  12. 12. Solution Benchmark TPCH-H (3,000 GB Results): −5.6 × 106 QphH between 2014 and 20165 QALD: ≈ 5% increase in Micro F-Measure ACE2004: ≈ 6% increase in Micro F-measure 5http://www.tpc.org/tpch/results/tpch_perf_results.asp Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 10 / 44
  13. 13. Challenges Dataset and KPI Mismatch Year ACE Wiki AQUAINT MSNBC IITB Meij AIDA/CoNLL N3 collection KORE50 Wiki-Disamb30 Wiki-Annot30 SpotlightCorpus SemEval-2013task12 SemEval-2007task7 SemEval-2007task17 Senseval-3 NIF-basedcorpus Microposts2014 Softwareavailable? Webserviceavailable? Cucerzan 2007 Wikipedia 2008 * Miner Illinois Wikifier 2011 * Spotlight 2011 AIDA 2011 ** TagMe 2 2012 Dexter 2013 KEA 2013 WAT 2013 AGDISTIS 2014 Babelfy 2014 NERD-ML 2014 BAT- 2013 * Framework NERD 2014 Framework GERBIL 2014 * Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 11 / 44
  14. 14. Challenges Unclear KPI Semantics Example Federated queries in distributed storage solutions Which time do we measure? First or last result? With or without network delay? Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 12 / 44
  15. 15. Challenges Unclear KPI Semantics Example Entity recognition and linking When is an annotation correct? Weak or strong annotation? Semantically equivalent or exact URI? Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 13 / 44
  16. 16. Solution! Unified Benchmarking Framework Rationale Provide all benchmark components in one package Include reference datasets and baselines Devise standardized tasks and reference KPI implementations Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 14 / 44
  17. 17. Solution! Unified Benchmarking Framework Rationale Provide all benchmark components in one package Include reference datasets and baselines Devise standardized tasks and reference KPI implementations Benchmark Core Web service calls Dataset Wrapper Web service calls Interface View Annotator Wrapper Interface View Open Datasets Configuration (Model) ... Benchmark Core Your Annotator Your Dataset DataHub.io GERBIL Core Controller Persistent Experiment Database (Model) Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 14 / 44
  18. 18. GERBIL HOBBIT v0.1 Features Unified benchmarking platform for NER/NEL 18 reference annotation systems 32 reference datasets Reference implementations of KPIs Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 15 / 44
  19. 19. GERBIL HOBBIT v0.1 Features Unified benchmarking platform for NER/NEL 18 reference annotation systems 32 reference datasets Reference implementations of KPIs Advantages Benchmarking ≈ 30× faster Archiving of results Citeable URIs Additional analysis http://gerbil.aksw.org http://github.org/aksw/gerbil Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 15 / 44
  20. 20. GERBIL HOBBIT v0.1 Features Unified benchmarking platform for NER/NEL 18 reference annotation systems 32 reference datasets Reference implementations of KPIs Advantages Benchmarking ≈ 30× faster Archiving of results Citeable URIs Additional analysis Availability Open-source project Local deployment Online instance Feedback for developers and users http://gerbil.aksw.org http://github.org/aksw/gerbil Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 15 / 44
  21. 21. GERBIL HOBBIT v0.1 Annotator Tasks NIF-based Annotators 2519 Babelfy 958 DBpedia Spotlight 922 TagMe 2 811 WAT 787 Kea 763 Wikipedia Miner 714 NERD-ML 639 Dexter 587 AGDISTIS 443 Entityclassifier.eu NER 410 FOX 352 Cetus 1 Overall 24.3K exps 50+ papers http://gerbil.aksw.org http://github.org/aksw/gerbil Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 16 / 44
  22. 22. HOBBIT Rationale Rationale A community-driven unified benchmarking platform for the community Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 17 / 44
  23. 23. HOBBIT Rationale Rationale A community-driven unified benchmarking platform for the community Build upon 24.3K GERBIL experiments Experiments focus on Big Linked Data Designed to accomodate all Big Data Cover all steps of the Big (Linked) Data lifecycle Open benchmarks based on industrial data and use cases Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 17 / 44
  24. 24. HOBBIT Aims 1 Gather real requirements Performance indicators Performance thresholds 2 Develop benchmarks based on real data 3 Provide universal benchmarking platform Standardized hardware Comparable results 4 Periodic benchmarking challenges 5 Periodic reporting 6 Found independent Hobbit association Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 18 / 44
  25. 25. HOBBIT Overview Data Collection Industry data Measure Collection Benchmark Creation Benchmark 1 KPIs Tasks KPIs Tasks KPIs Tasks KPIs Tasks KPIs Tasks KPIs Tasks Benchmark 2 Benchmark n HOBBIT Platform Solution 1 Solution k Solution 2 Challenges Reports Participants/Community Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 19 / 44
  26. 26. Survey Questions Questions In what areas are organizations active? What do people expect from benchmarks? How are benchmarks being used? Profile Count Solution providers 56 Technology users 67 Scientific community 65 Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 20 / 44
  27. 27. Survey Can your solution be benchmarked? Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 21 / 44
  28. 28. Survey Do you benchmark your solution? Own datasets and settings in many cases Own implementations of measures Results not comparable Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 22 / 44
  29. 29. Survey Application Areas http://big-data-europe.eu Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 23 / 44
  30. 30. HOBBIT Platform Features Uses established deployment technologies (Docker) Decoupled components Benchmark and systems can be written in different languages Uses scalable message queues for communication Open-source implementation Supports distributed benchmarks and systems Online instance on server cluster Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 24 / 44
  31. 31. HOBBIT Platform Features Features Unified benchmarking platform for Big Data 20+ reference annotation systems 40+ reference datasets Reference implementations of KPIs Advantages Benchmarks derived from real industrial data and use cases Scalable size of benchmarks Archiving of results Citeable URIs Result analysis Availability Open-source project Local deployment Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 25 / 44
  32. 32. HOBBIT Platform Architecture Platform Controller Data Generator Task Generator Data Generator Data Generator Task Generator Task Generator Front End Benchmarked System data flow creates component Storage Analysis Benchmark Controller Evaluation Module Eval. Storage Logging Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 26 / 44
  33. 33. HOBBIT Platform Benchmark Initialization Platform Controller Data Generator Task Generator Data Generator Data Generator Task Generator Task Generator Benchmarked System data flow creates component Storage Benchmark Controller Eval. Storage Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 27 / 44
  34. 34. HOBBIT Platform Benchmark Execution Platform Controller data flow creates component Storage Data Generator Task Generator Data Generator Data Generator Task Generator Task Generator Benchmarked System Benchmark Controller Eval. Storage ex:Entity rdf:type ex:Class ... Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 28 / 44
  35. 35. HOBBIT Platform Benchmark Execution Platform Controller data flow creates component Storage Data Generator Task Generator Data Generator Data Generator Task Generator Task Generator Benchmarked System Benchmark Controller Eval. Storage v ex:Entity ... SELECT ?v WHERE { ?v a ex:Class } Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 29 / 44
  36. 36. HOBBIT Platform Benchmark Execution Platform Controller data flow creates component Storage Data Generator Task Generator Data Generator Data Generator Task Generator Task Generator Benchmarked System Benchmark Controller Eval. Storage X v ex:Entity ... Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 30 / 44
  37. 37. HOBBIT Platform Benchmark Evaluation data flow creates component Platform Controller Storage Benchmark Controller Evaluation Module Eval. Storage precision=... recall=... F1-score=... precision=... recall=... F1-score=... benchmark parameters: ... v ex:Entity ... v ex:Entity ... Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 31 / 44
  38. 38. HOBBIT Platform Benchmarks Streaming and static deterministic benchmarks Realistic benchmarks Controlled volume and velocity Generation and Acquisition Conversion of XML into RDF Entity recognition and linking Relation extraction Analysis and Processing Link Discovery Machine Learning Supervised and unsupervised Storage and Curation Triple stores Versioning Incl. updates Visualization and Services Question Answering Faceted Browsing Usage-based benchmarks Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 32 / 44
  39. 39. Datasets TWIG Goal: Simulate real Twitter Firehose Relies on 476 million tweets as training data Mimicking algorithm based on Distribution of character frequencies Distribution of transportation frequency Network topology Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 33 / 44
  40. 40. Datasets LinkedConnections Goal: Simulate real transport network Real transportation data from Belgium for training Mimicking algorithm based on Observed correlation between population density and transportation Distribution of transportation frequency Network topology Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 34 / 44
  41. 41. Datasets Printing Machinery Goal: Simulate events from printing machinery Mimicking algorithm using event correlations and distributions Changing plate Double sheet Early sheet Finish job Misaligned sheet Missing sheet Operation partially completed Performance Printing interval Produktion Good Sheet Side guide warning Start job Washing blanket Washing impression cylinder Washing ink rollers with washing ink fountain roller with washing plates Mai 01 00:00 Mai 01 06:00 Mai 01 12:00 Mai 01 18:00 Mai 02 00:00 Time Events Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 35 / 44
  42. 42. Datasets Weidmüller Goal: Simulate events from injection molding machinery Mimicking algorithm using event correlations and distributions Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 36 / 44
  43. 43. Datasets Semantic Publishing Goal: Simulate data from the BBC Generator based on manually configurable set of correlations Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 37 / 44
  44. 44. HOBBIT Runs Triple Stores 1 4 16 1 10 100 1000 QmpH Updates virtuoso blazegraph fuseki SPARQL worker updates Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 38 / 44
  45. 45. HOBBIT Runs Runtimes 10× more effort for reduction of error rate by 30% Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 39 / 44
  46. 46. HOBBIT Runs A2KB System AIDA/CoNLL-Comp. IITB KORE50 MSNBC Microp.2014-Train N3-Reuters-128 AIDA 0.668 0.141 0.625 0.622 0.363 0.391 Babelfy 0.448 0.129 0.564 0.423 0.311 0.289 DBpedia Spotlight 0.545 0.262 0.341 0.457 0.448 0.320 FOX 0.512 0.100 0.268 0.127 0.309 0.518 FREME NER 0.358 0.074 0.160 0.208 0.254 0.263 WAT 0.673 0.137 0.543 0.631 0.403 0.480 xLisa 0.363 0.233 0.352 0.365 0.322 0.274 Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 40 / 44
  47. 47. Summary Rationale A community-driven unified benchmarking platform for the community Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 41 / 44
  48. 48. Summary Rationale A community-driven unified benchmarking platform for the community Provide benchmarks and baselines Provide reference implementation of KPIs Extensible and referenceable Open-Source @hobbit_project Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 41 / 44
  49. 49. Join HOBBIT α completed Join the HOBBIT community Provide KPIs Provide datasets Join the platform development Follow us on Twitter https://project-hobbit.eu Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 42 / 44
  50. 50. Thank You Axel Ngonga AKSW Research Group Institute for Applied Informatics ngonga@informatik.uni-leipzig.de Michael Röder AKSW Research Group Institute for Applied Informatics roeder@informatik.uni-leipzig.de Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 43 / 44
  51. 51. Acknowledgment This work was supported by grants from the EU H2020 Framework Programme provided for the project HOBBIT (GA no. 688227). Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 44 / 44

×