Grade is a tool for sustainably managing linked data repositories. It allows users to generate, update, and disseminate linked data in three phases - ingesting source data, transforming it in a staging area, and publishing it. Key features include storage abstraction, interactive dissemination through SPARQL query testing and result visualization, and staged transformations through defined transformation tasks. The tool aims to make currently manual and batch processes more automated, interactive, and production-ready for linked data managers.

1. Hello Grade
Simeoni, Baldassarre
SUSTAINABLE LINKED DATA

2. we like in in FAO.
FISHERIES LINKED OPEN DATA
(F) LOD

3. we use it for FACP
FISHERIES AND AQUACULTURE COUNTRY PROFILES

4. we found it hard to manage
data integration issues, in LOD context.

5. linked data, sustainably.

6. Grade

7. “we”

8. the full story
context, details, demo.

9. why we did it
relation to past work
implications for the future
what we delivered
how we worked together
our toolchain
our stack
all about Grade, in a black-box.
context

10. a pinch of theory
key features
data integration activities
now put ‘Semantics’ in the black box.
under the hood

11. features in context
real data
simulated scenarios
demonstration
a day in the life of a data manager.

12. context

13. mod-A architecture.
no monoliths, specialised components

14. functionally small.
services, libraries, apps.

15. repositories.
“one thing” is disseminate data.
you put data there.
act as a producer.
I find it there.
act as a consumer.
we’ve
exchanged it
data
hubs

16. not all alike
hold different data, hold it differently.
eg. master data
in relational databases.
RTMS
eg. cross-master “relations”
in triple stores.
flod

17. all managed.
with appropriate tools.
Grade is a tool to manage
Linked Data repositories.

18. flod
generate it
from external sources.
three macro requirements.
update it in place
when sources change.
disseminate it
via FACP-facing services.
facp
clients

19. so far “unsustainably” met.

20. the extraordinary Linked Data Manager.
all we could throw at requirements.
doubles as javadev.
has sysadmin skills.
oversees operations.
a rare find.
(an inefficient employee anyway.)

21. make interactive what’s now batch
make online what’s now offline.
make automated what’s now manual.
make production-ready what’s now not.
a Web App for Data Managers.
With Grade, we turn things around.
to generate, update, and disseminate Flod.

22. A Tale of Two Managers.
Linked Data Manager
designs ontologies.
defines maintenance strategy.
defines services.
implements strategy.
General Data Manager
explores and validates data.
monitors services.
can implement strategy.
can define services.

23. hold on, develop tools for a use case?
isn’t it a disproportionate use of force ?
no, if we reuse them.

24. reuse beyond FACP.
Grade promotes Flod
can be entrusted with more use cases.
Grade complements RTMS
two-pronged data management strategy.

25. RTMS
local
boundary
(fips, fao)
LOCAL
clients
Flod
external
clients
external
SOURCES
gems
agrovoc
IRD

26. reuse beyond Flod.
requirements recur across similar repositories.
Grade promotes a LD Network
with our external partners in Fishery.

27. Flod
worms
EEZ
WIOFISH
STATBASE

28. the dev pattern behind Grade.
a viable approach to architecture.

29. SOFTWAREUSE CASEs MOD-A
drives / validates
make it general
goes on to enhance
reuse to solve other
seed

30. architecture “on demand”
we “tunnel” its investment through seeds.
- driven development
reconciles short-term need with longer-term benefit.
seed

31. but, there is a catch.
try build reusable software in
timescales appropriate to seeds.
it’s challenging.

32. our challenges.
very sustained pace.
fully distributed team.
busy schedule for some members.
no CI services.
a fairly new stack.

33. coping measures.
result focus.
crabwalking
progressive refinement
feature weighting
tight feedback
cycles.
reactive steering
min. blocking dependencies
tools for max visibility

34. Trello boards.
activities: may do, doing, done, won’t do.
actions: discuss, report, notify, alert.
include, attach, or link: know-hows, how-tos, docs.

35. Continuous Deployment
real-time progress visibility, off premises.
github drone.io d4science AWS
+NO-IP
sources CI BUILDS dependencies
(nexus)
deployment
(micro)

36. our software stack.
fuseki
JAVA 8
lombok
jersey/WELD/jackson
plenty of innovation for us.
BACKEND
POLYMER / ANGULAR DI
Google’s dart
frontend

37. why the frontend.
incremental innovation.
familiarity
safety
platform
tooling
control
productivity
momentum

38. under the hood

39. CLIENTS
storage
abstraction
infrastructure
interactive
dissemination
staged
transformation
sources
three Big Features.

40. first, layman’s concepts.
LOGICAL physical

41. LOGICAL physical
subject OBJECT
PREDICATE
‘SCHEMAS’
TRIPLESnamed graphs
QUERIESupdates
endpoints
STORES

42. different storage arrangements.
mono-point MULTI-point MULTI-STORE
small scale, simplest.medium scale, simple.large scale or linked.

43. Grade can work with any model.
adapts to infrastructure, doesn’t prescribe it.
at any time.
change to migrate, scale out, network with partners.
with any provider.
read/write access via HTTP-based standards.

44. Grade has Datasets.
logical asset partitions.
Datasets map to Endpoints
endpoints have addresses.
Endpoints have Graphs.
can inspect, add, edit, remove, copy/move directly in Grade.

45. this is storage abstraction.

46. next, interactive dissemination

47. DOCUMENTATION
target
narrow
scope
SPARQL
testing

48. Query Playground.
color highlighting
!
keyword completion
!
prefix completion
!
graph name completion
!
query templates
!
address abstraction
DYNAMIC UI !

49. OMNI format
paste in editorFOLLOW YOUR NOSE
breadcrumb

50. paste in BROWSER
URI-BASED NEGOTIATIoN
REAl-TIME API

51. on demand

52. Grade can expose data on demand.
no predefined, hardcoded dissemination services.
without Java code
well within reach of (Linked) Data Managers.
with perks
monitoring, testing, and documentation also covered.

53. this is interactive
dissemination.

54. finally, staged transformations
sources

55. generate and update in 2-phases.
sources
ingest
&
TRIPLIFY “raw”
first, ingest into staging area in raw form.then, transform in public form for production.
transForM
“public”
STAGING
(triage)
production

56. the notion of staging area.
a secondary asset that serves as a “dropbox”,
a hole for pigeons.
raw as in uncooked.
no LD design, ontology-less triples.
easily produced…
blindfold conversions from any XML, JSON, CSV…
yet “semantically edible”.
can cook it with SPARQL-driven recipes.

57. publication tasks
create or overwrite whole graphs.
TRANSFORMATION
query
staging
source
graphs
production
target
graph

58. update tasks
add or remove a “delta” of triples to/from a graph.
TRANSFORMATION
query
staging
Transform
DIFFERENCE
QUERY
delta
production

59. we keep all tasks in a catalogue.
a third LD asset just for “recipe definitions”.
tasks
catalogue

60. testing
doc.
I/O
transform

61. Task Playground.

62. stoppable
remote LOGS
partial resultS
FINAL resultS

63. real test executions happen “on deck”.
CATALOGUE AT HAND
current or past
executions
results, as
in playground

64. putting it all together…
production
DECK
STAGING
CATALOGUE

65. hold on, how does data get into Grade?
who populates the staging area?
?
ingest
&
TRIPLIFY “raw”
sources

66. drop-in servicepublishers it to staging.
REST dropin service +Java API.
PUSH
admins it.upload
drop(“…/codes.xml”).with(xml).in(prod).as(“…/codes.xml”)

67. admin-units
countries
vessels
direct or indirect, codelist or mappings.
Flod has about sources.20
gear eez-flagstate-eploitation
asfis-worms
RFB
species-distributions
EEZ
asfis
eez-country-sovereignity
eez-fsa_intersection
fsa-hierarchies
EXPLOITATION-­‐STATUS
marine-regions

68. the project within the project.

69. demo

70. publish service for ASFIS species list.
acquire list, transform it, expose it through service.
!
publish service for ASFIS - WORMS species mapping.
acquire mapping, transform it, expose it through service.
coming on:

71. feedback

72. phase : delivery
produce the solution (3m)
GRADE 1.0-ea
1
phase : post-delivery
disseminate, document, and deploy the solution (2m)
GRADE 1.0
2
where we are.
phase : next-gen
GRADE 2.0
3