This document discusses building materialized views of linked data systems using microservices. It outlines challenges with current architectures and proposes a new architecture using microservices to build materialized views that map closely to different query profiles. The views would be updated via publish APIs and distributed to read APIs. Join operations could be handled during writes, reads, or in the views. Other considerations include tracking ontology changes and maintaining a single source of truth.

1. Building materialised views of linked
data systems using microservices
Augustine Kwanashie

2. Outline
o Introduction
o Current architecture and challenges
o Building materialised views
o Other things to consider

3. Publish Distribute
Publish and distribute content metadata

4. 1
I’ll try to create like
Beckham 01-07-2018:19:01:01
urn:cps:1289394
3
2
about
English Football Team
http://wikidata.org/123
locator
Kieran
Trippier
label
Metadata on Tagging

5. Top 10 Articles
About “English Football Team”
Ordered by date published

6. Simplified Architecture
Write API
Triplestore
Read API
DistributionSystems
EditorialSystems

7. SPARQL endpoints
Performance and
Data Integrity
Flexibility and Pace
of Innovation
Custom APIs

8. Projected performance by 2019
60%
99 percentile response
time
100%
data volume

9. So what do we know about the API
requests?

10. We can group API requests by their
query profiles

11. Query by identifier
CONSTRUCT {
. . .
} WHERE {
<urn:01> a core:Atricle .
. . .
}

12. Query with filters
CONSTRUCT {
. . .
} WHERE {
?id property1 <urn:01> .
?id property2 "value2" .
. . .
}

13. Multi-hop query
CONSTRUCT {
. . .
} WHERE {
?id1 <urn:property1> ?id2
?id2 <urn:property2> ?id3
?id3 <urn:property3> "value3" .
}

14. We can group API requests by their
volume and performance
requirements

15. Low volume and
performance
requirements
High volume and
performance
requirements
More complex
queries
Mostly simple
queries

16. Build views that map closely to
query profiles

17. Target architecture
Event
Store
WriteAPI
Publish API
Publish API
Publish API
ReadAPI
Distribute
Query by ID
Multi-hop

18. The publish pipeline
λ
View DBIngest
View API
WriteAPI
Read API
Data
Input Queue
ID: 838394
Operation: Create
Timestamp: 1540906781999

19. Send to DLQ is errors persist
λ
View DBIngest
WriteAPI
Read API
Input Queue
Dead Letter Queue

20. Notify clients of a new ingest
λ
View DBIngest
View API
λ
SNS
Notifier

21. Verify ingest is successful
λ
View DBIngest
View API
λ
Verifier
Read API
Dead Letter Queue

22. The distribution pipeline
Read API
Triplestore
Router
View API
View DB 1
ReadAPIs
View API
View DB 2

23. Route traffic based on profile and format
If request matches {
format: "ld+json"
query: "?id=<GUID>"
}
Then route to View 1

24. Failover to the Triplestore
Read API
Triplestore
Router
View API
View DB 1
ReadAPIs

25. Split traffic between Views
Read API
Triplestore
Router
View API
View DB 1
ReadAPIs
60% of traffic
40% of traffic

26. What about JOINS?

27. {
”@id": "urn:article:01",
"about": [
"urn:tag:01",
"urn:tag:02",
…
]
}
{
”@id": "urn:tag:01",
"label": "Nigeria",
”@type": "Place"
}

28. {
”@id": "urn:article:01",
"about": [
{
”@id": "urn:tag:01",
"label": "Nigeria",
”@type": "Place"
},
…
]
}

29. Previously…
Write APIs
Triplestore
Read APIs
PUT <urn:article:01>
PUT <urn:tag:01>
combined data

30. Join on Writes
Publish API
ReadAPI
Distribute
PUT <urn:article:01>
PUT <urn:tag:01>
custom view for
combined data

31. Join on Reads
Publish API
Publish API
ReadAPI
Distribute
PUT <urn:article:01>
PUT <urn:tag:01>
combined data

32. Other things to consider

33. Tracking Ontology Changes
biz:Company
rdf:type owl:Class ;
rdfs:comment "A company featured in BBC news"^^xsd:string ;
rdfs:isDefinedBy <http://www.bbc.co.uk/ontologies/…> ;
rdfs:label "A company featured in BBC news"^^xsd:string ;
rdfs:subClassOf core:Organisation .

34. Tracking Ontology Changes
<http://www.bbc.co.uk/things/01#id> a biz:Company ;
core:label "Amazon Inc. " .
<http://www.bbc.co.uk/things/01#id> a core:Organisation .
Generated implicit triples

35. Single source of truth
Publish API
Triplestore
Ingest Script
Query all IDs
ID: 838394
Operation: Create
Timestamp: 1540906781999

36. Summary:
Using multiple data sources that match
specific query types is feasible and
beneficial

37. Thank You