Download as PDF, PPTX
Uploaded byFabio Simeoni
Grade@cnr
Grade is a tool for sustainably managing linked data repositories. It allows users to generate, update, and disseminate linked data in three phases - ingesting source data, transforming it in a staging area, and publishing it. Key features include storage abstraction, interactive dissemination through SPARQL query testing and result visualization, and staged transformations through defined transformation tasks. The tool aims to make currently manual and batch processes more automated, interactive, and production-ready for linked data managers.
Grade@cnr
- 1. Hello Grade Simeoni, Baldassarre SUSTAINABLELINKED DATA
- 2. we like inin FAO. FISHERIES LINKED OPEN DATA (F) LOD
- 3. we use itfor FACP FISHERIES AND AQUACULTURE COUNTRY PROFILES
- 4. we found ithard to manage data integration issues, in LOD context.
- 5.
- 6.
- 7.
- 8. the full story context,details, demo.
- 9. why we didit relation to past work implications for the future what we delivered how we worked together our toolchain our stack all about Grade, in a black-box. context
- 10. a pinch oftheory key features data integration activities now put ‘Semantics’ in the black box. under the hood
- 11. features in context realdata simulated scenarios demonstration a day in the life of a data manager.
- 12.
- 13. mod-A architecture. no monoliths,specialised components
- 14.
- 15. repositories. “one thing” isdisseminate data. you put data there. act as a producer. I find it there. act as a consumer. we’ve exchanged it data hubs
- 16. not all alike holddifferent data, hold it differently. eg. master data in relational databases. RTMS eg. cross-master “relations” in triple stores. flod
- 17. all managed. with appropriatetools. Grade is a tool to manage Linked Data repositories.
- 18. flod generate it from externalsources. three macro requirements. update it in place when sources change. disseminate it via FACP-facing services. facp clients
- 19.
- 20. the extraordinary LinkedData Manager. all we could throw at requirements. doubles as javadev. has sysadmin skills. oversees operations. a rare find. (an inefficient employee anyway.)
- 21. make interactive what’snow batch make online what’s now offline. make automated what’s now manual. make production-ready what’s now not. a Web App for Data Managers. With Grade, we turn things around. to generate, update, and disseminate Flod.
- 22. A Tale ofTwo Managers. Linked Data Manager designs ontologies. defines maintenance strategy. defines services. implements strategy. General Data Manager explores and validates data. monitors services. can implement strategy. can define services.
- 23. hold on, developtools for a use case? isn’t it a disproportionate use of force ? no, if we reuse them.
- 24. reuse beyond FACP. Gradepromotes Flod can be entrusted with more use cases. Grade complements RTMS two-pronged data management strategy.
- 25.
- 26. reuse beyond Flod. requirementsrecur across similar repositories. Grade promotes a LD Network with our external partners in Fishery.
- 27.
- 28. the dev patternbehind Grade. a viable approach to architecture.
- 29. SOFTWAREUSE CASEs MOD-A drives/ validates make it general goes on to enhance reuse to solve other seed
- 30. architecture “on demand” we“tunnel” its investment through seeds. - driven development reconciles short-term need with longer-term benefit. seed
- 31. but, there isa catch. try build reusable software in timescales appropriate to seeds. it’s challenging.
- 32. our challenges. very sustainedpace. fully distributed team. busy schedule for some members. no CI services. a fairly new stack.
- 33. coping measures. result focus. crabwalking progressiverefinement feature weighting tight feedback cycles. reactive steering min. blocking dependencies tools for max visibility
- 34. Trello boards. activities: maydo, doing, done, won’t do. actions: discuss, report, notify, alert. include, attach, or link: know-hows, how-tos, docs.
- 35. Continuous Deployment real-time progressvisibility, off premises. github drone.io d4science AWS +NO-IP sources CI BUILDS dependencies (nexus) deployment (micro)
- 36. our software stack. fuseki JAVA8 lombok jersey/WELD/jackson plenty of innovation for us. BACKEND POLYMER / ANGULAR DI Google’s dart frontend
- 37. why the frontend. incrementalinnovation. familiarity safety platform tooling control productivity momentum
- 38.
- 39.
- 40. first, layman’s concepts. LOGICALphysical
- 41. LOGICAL physical subject OBJECT PREDICATE ‘SCHEMAS’ TRIPLESnamedgraphs QUERIESupdates endpoints STORES
- 42. different storage arrangements. mono-pointMULTI-point MULTI-STORE small scale, simplest.medium scale, simple.large scale or linked.
- 43. Grade can workwith any model. adapts to infrastructure, doesn’t prescribe it. at any time. change to migrate, scale out, network with partners. with any provider. read/write access via HTTP-based standards.
- 44. Grade has Datasets. logicalasset partitions. Datasets map to Endpoints endpoints have addresses. Endpoints have Graphs. can inspect, add, edit, remove, copy/move directly in Grade.
- 45. this is storageabstraction.
- 46.
- 47.
- 48. Query Playground. color highlighting ! keywordcompletion ! prefix completion ! graph name completion ! query templates ! address abstraction DYNAMIC UI !
- 49. OMNI format paste ineditorFOLLOW YOUR NOSE breadcrumb
- 50. paste in BROWSER URI-BASEDNEGOTIATIoN REAl-TIME API
- 51.
- 52. Grade can exposedata on demand. no predefined, hardcoded dissemination services. without Java code well within reach of (Linked) Data Managers. with perks monitoring, testing, and documentation also covered.
- 53.
- 54.
- 55. generate and updatein 2-phases. sources ingest & TRIPLIFY “raw” first, ingest into staging area in raw form.then, transform in public form for production. transForM “public” STAGING (triage) production
- 56. the notion ofstaging area. a secondary asset that serves as a “dropbox”, a hole for pigeons. raw as in uncooked. no LD design, ontology-less triples. easily produced… blindfold conversions from any XML, JSON, CSV… yet “semantically edible”. can cook it with SPARQL-driven recipes.
- 57. publication tasks create oroverwrite whole graphs. TRANSFORMATION query staging source graphs production target graph
- 58. update tasks add orremove a “delta” of triples to/from a graph. TRANSFORMATION query staging Transform DIFFERENCE QUERY delta production
- 59. we keep alltasks in a catalogue. a third LD asset just for “recipe definitions”. tasks catalogue
- 60.
- 61.
- 62.
- 63. real test executionshappen “on deck”. CATALOGUE AT HAND current or past executions results, as in playground
- 64. putting it alltogether… production DECK STAGING CATALOGUE
- 65. hold on, howdoes data get into Grade? who populates the staging area? ? ingest & TRIPLIFY “raw” sources
- 66. drop-in servicepublishers itto staging. REST dropin service +Java API. PUSH admins it.upload drop(“…/codes.xml”).with(xml).in(prod).as(“…/codes.xml”)
- 67. admin-units countries vessels direct or indirect,codelist or mappings. Flod has about sources.20 gear eez-flagstate-eploitation asfis-worms RFB species-distributions EEZ asfis eez-country-sovereignity eez-fsa_intersection fsa-hierarchies EXPLOITATION-‐STATUS marine-regions
- 68. the project withinthe project.
- 69.
- 70. publish service forASFIS species list. acquire list, transform it, expose it through service. ! publish service for ASFIS - WORMS species mapping. acquire mapping, transform it, expose it through service. coming on:
- 71.
- 72. phase : delivery producethe solution (3m) GRADE 1.0-ea 1 phase : post-delivery disseminate, document, and deploy the solution (2m) GRADE 1.0 2 where we are. phase : next-gen GRADE 2.0 3