Introductory training course on the Linked Data registry and its application to Environmental reference data.

1. Registry
Environment registry service
Introductory webinar
1

2. Registry
Outline
• reference data and role of the registry
• environment registry service
– demonstration
– information model
– services
– status
• processes and usage
– data flows – preparation and access
– governance
– implications
• Q & A
2

3. Registry
Goals of the webinar
• understand the registry service
– what it is
– what it’s for
• stimulate thinking on
– how to make use of it
– processes and governance
• feedback
– future training needs (technical details course)
– service requirements
3

4. Registry
Motivation
Multiple forces
– deliver government policy
• ITC Strategy
• Digital Policy
• Open Data
• Open Standards
– INSPIRE
– Defra move to shared services (ICT Futures,
Knowledge Strategy and Open Data Strategy)
"Government IT must be open - open to the people
and organisations that use our services and open
to any provider, regardless of their size.”
4

5. Registry
Motivation
• key to making data
sharable, reusable, open is
accessible reference data
Picture CC-BY-2.0 © FutUndBeidl@flickr.com
5

6. Registry
EG13987 P816L H 0.x u35 2012
Reference data
Grafham water phosphate level High 0.x mg/l 2012
• standardized terms to identify things in data
• normally represented by coded identifiers
• key to meaning of the dataentities, places, objects
substances, determinands
units
classifications, codes
assessment methodology
sampling methodology
6

7. Registry
Reference data identifiers
• to enable data reuse and access we need
reference data that is:
– independent of a particular data system
• global identifiers
– stable
• persistent identifiers
– interpretable
• identifiers you can look up (“resolvable”)
7

8. Registry
Reference data identifiers
Open standards process
• persistent resolvable
identifiers challenge
• adopt HTTP 1.1 and URLs for this
– board resolution 24 Sep 2013
• obvious fit
– global (DNS)
– resolvable (http)
8

9. Registry
Reference data identifiers
• => requirement to create and maintain URIs
for identifiers in reference data
• challenges
– how to share an authoritative namespace?
– what do they deference to?
– managing authoritative lists of terms?
9

10. Registry
UKGovLD Registry
Registry – tool to manage URIs for reference data
Services:
– manage controlled lists of identifiers as URIs
– store core data so the identifiers resolve
– namespace management
– other: validation, discovery, version and history
Design and open source implementation
Defra instance at:
http://environment.data.gov.uk/registry/
10

11. Registry
Picture CC-BY-2.0 © Annie Roi @flickr.com
11

12. Registry
Demo
• front page – list of all registers in registry
12

13. Registry
Demo
• filter – just registers from Environment Agency
13

14. Registry
Demo
• collection of code lists on some theme
14

15. Registry
Demo
• a flat code list
15

16. Registry
Demo
• an individual code in the code list
16

17. Registry
Demo
• a hierarchical code list
17

18. Registry
Demo
• a hierarchical code list
18

19. Registry
Machine accessible
• each collection of entries or individual entries
has a URI
• access through a browser you see a web page
• a machine can request specific format like
JSON directly (content negotiation)
19

20. Registry
Picture CC-BY-2.0 © Annie Roi @flickr.com
20

21. Registry
Information model
Register - a controlled list of items
21

22. Registry
Information model
• Registers can contain registers or simple items
root register
top level of definitions
a themed collection
http://environment.data.gov.uk/registry/def/catchment-planning/RiverBasinDistrict/
http://environment.data.gov.uk/registry/def/catchment-planning/
http://environment.data.gov.uk/registry/def/
http://environment.data.gov.uk/registry/
a code list
codes
• http://environment.data.gov.uk/registry/def/catchment-planning/RiverBasinDistrict/UK01
• http://environment.data.gov.uk/registry/def/catchment-planning/RiverBasinDistrict/UK02
• ...
22

23. Registry
Information model
Register - a controlled list of items
– manager and governance policy
– lifecycle and status for items in the register
23

24. Registry
Information model
Register item
– a definition of something
• concept, organization, geographic area, substance ...
– represented as a set of property values
– type and label required
– values can be simple (strings, numbers) or URIs
http://environment.data.gov.uk/registry/def/catchment-planning/RiverBasinDistrict/UK05
type http://location.data.gov.uk/def/am/wfd/RiverBasinDistrict
label Anglian
notation UK05
24

25. Registry
Information model
• Register item
– no constraints on what properties you use to
describe the item (“schema-less”)
– open ended, can add richer descriptions later
– gives a lot of flexibility for what you can register
– tame this by adopting a few standard patterns
• but can add more as needs change
25

26. Registry
Information model
Standard patterns
• collections of codes
– SKOS (Simple Knowledge Organization Scheme)
– Concepts grouped in to ConceptSchemes
• organizations
– ORG (Organization Ontology)
26

27. Registry
Aside on RDF and linked data
• there’s a standard for how to represent such
descriptions of things identified by URI
– Resource Description Framework (RDF)
• the registry design and implementation is built
on this standards stack
• the standard patterns are RDF vocabularies
• but don’t need to use RDF in your information
systems in order to use the registry
27

28. Registry
Information model
Linking
– value of a property can be a URI
• any URI – same register, other register, external
– use for hierarchical structure within a register
• concept schemes with broader/narrower links
• organizational structure with sub-organizational links
28

29. Registry
Information model
Hierarchy within a register
Anglia
...
Central
Eastern
Northern
Midlands
hasSubOrganization
topConceptOf
ea-areas
29

30. Registry
Information model
Linking
– value of a property can be a URI
• any URI – same register, other register, external
– use for hierarchical structure within a register
• concept schemes with broader/narrower links
• organizational structure with sub-organizational links
– use for cross matching between code lists
• exact match within MMO experimental data
30

31. Registry
Information model
Cross-links between registers
Addresses
...
Admin areas
MMO/data-theme
Address
...
Administrative
Units
MMO/topic-category
exactMatch
31

32. Registry
Information model
Linking
– value of a property can be a URI
• any URI – same register, other register, external
– use for hierarchical structure within a register
• concept schemes with broader/narrower links
• organizational structure with sub-organizational links
– use for cross matching between code lists
• exact match within MMO experimental data
– use to relate registered URIs with other URIs
• same as links from organizations to organogram IDs
32

33. Registry
Information model
External items
– can register URIs outside the registry service
– register holds
• definitive list of items
• metadata about the registered items (status etc)
• copy of minimal description information
– up to the URI owner to maintain the URI
Managed items
– URI is within the register namespace
– maintenance done within the registry service
33

34. Registry
Information model
Navigation browse hierarchy
of registers
text search for
register or item
filter set of registers
34

35. Registry
Information model
Filters
– category
• classification of subject that the register is about (e.g. Water)
– entity
• the type of thing in the register (e.g. Regions and Habitats)
– owner
• the organization which owns and manages the register
Extensible
– this is just metadata associated with the registers
– could extend the category schemes or add others
35

36. Registry
Information model
• Summary
– Register
• controlled list
• arranged in a hierarchy like folders in a file system
• can be annotated and classified to help with navigation
– Item
• entry in register
• can be external to the registry or internal (managed)
• has a URI and extensible set of descriptive properties
• optional standard patterns for the descriptions
• properties enable links between items
36

37. Registry
Picture CC-BY-2.0 © Annie Roi @flickr.com
37

38. Registry
Registry services
• Outline of services
– manage controlled lists of identifiers as URIs
– serve data for the registers and managed items
– namespace management
– validation
– discovery
– version and history management
38

39. Registry
Registry services
An API for everything
– REST API for each of the services
– user interface is layered on top
– can build external tools which provide other
interfaces but work via the API
– interface itself is template-driven and easy to
modify
39

40. Registry
Registry services
• Outline of services
– manage controlled lists of identifiers as URIs
• create register
• register item(s)
• update items
• change status
40

41. Registry
Registry services
• Outline of services
– manage controlled lists of identifiers as URIs
– serve data for the registers and managed items
• return as RDF, JSON, CSV (TBD)
• view as web page in browser
• control over how lists (registers) are returned
– see metadata as well as the items
– filter by status
– page through long lists
41

42. Registry
Registry services
• Outline of services
– manage controlled lists of identifiers as URIs
– serve data for the registers and managed items
– namespace management
• requests to parts of URI space can be forwarded to
other services
• some support for federation
42

43. Registry
Registry services
• Outline of services
– manage controlled lists of identifiers as URIs
– serve data for the registers and managed items
– namespace management
– validation
• test of a set of URIs are valid
43

44. Registry
Registry services
• Outline of services
– manage controlled lists of identifiers as URIs
– serve data for the registers and managed items
– namespace management
– validation
– discovery
• supports text search
• user interface navigation support
44

45. Registry
Registry services
• Outline of services
– manage controlled lists of identifiers as URIs
– serve data for the registers and managed items
– namespace management
– validation
– discovery
– version and history management
• stores history of item versions
• versioned URIs
• see item or register at point in time
45

46. Registry
Registry services
• Outline of services
– manage controlled lists of identifiers as URIs
– serve data for the registers and managed items
– namespace management
– validation
– discovery
– version and history management
46

47. Registry
Status
• open design and open source implementation
– managed by UKGovLD
• proof of concept deployment
– proved principle, and running stably for 8 months
• pilot deployment for environment.data.gov.uk
– supported for one year
– alpha but robust so far
– security model for update based on OpenID
– intention to continue service if pilot successful
– may require enhancements e.g. server replication
47

48. Registry
Picture CC-BY-2.0 © Annie Roi @flickr.com
48

49. Registry
Processes and usage
• how do you use the registry?
• how do you get data into it?
• what data should go into it?
• how will that be managed and governed?
49

50. Registry
Using the reference data
Manual consultation
– e.g. developer looking up meaning of term in data
Registry
service
lookup URI via
browser
50

51. Registry
Using the reference data
Use code list in IT application
– e.g. data entry dialog, export mapping ...
Registry
service
IT
Application
RDF
[CSV]
JSON
local
format
export code
list via web API
or manual
download
support
other
formats?
use
directly
map to
application
specific format
51

52. Registry
Using the reference data
• Publish data using the references
– use the URIs instead of free text or opaque codes
– can use URIs in CSV or JSON, doesn’t have to be
RDF linked data
52

53. Registry
Using the reference data
Site Det Measurement
1 A 0.1
1 B 50
2 A 0.5
2 B 10
Site Determinad Value
A X 100
B X 10
C X 20
D X 100
Site Det Measurement
1 http://… 10
1 http://… 50
2 http://… 50
2 http://… 10
Site Determinad Value
A http://… 100
B http://… 10
C http://… 20
D http://… 100
Site Det Measurement
1 http://… 10
1 http://… 50
2 http://… 50
2 http://… 10
A http://… 100
B http://… 10
C http://… 20
D http://… 100
53

54. Registry
SPARQL
Publishing reference data
Registry
service
manual
registration
and update
Existing
local
code lists
ETL JSON
custom
extraction
registry-util
converter
[CSV] RDF
data preparation
service takes
simple CSV inputdirect federation
to RDF source
Code list server
proxy requests
54

55. Registry
Publishing reference data
• which data?
– reference data that enables data reuse or sharing
– between organizations or as part of open data
– connective reference data
– judgement required
55

56. Registry
Governance
Local Experimental Final
Network
Standard
Business owners
Data board
Registry administration
Organization SRO Organization SRO
publishers
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
• ----
56

57. Registry
Processes and usage
• Registry is just tool
– manage a set of global, persistent identifiers
– to enable data to be reused and integrated
• whether across organizations or as open data
– but up to you
• which reference data should be managed this way
• how the maintenance process should work in practice
• when to map data to the shared identifiers
– it’s there to reduce the cost of such management
• to gain the benefits of reusable data
• not to add additional processes for the sake of it
57

58. Registry
Next steps
Technical “how to” training planned
Coverage:
• preparing data for registration
• registration and managing entries
• accessing data
Suitable for
• potential registry administrators or publishers
58

59. Registry
Links
• Design and API details
https://github.com/UKGovLD/ukl-registry-poc/wiki
• Alpha site
http://environment.data.gov.uk/registry
59

60. Registry
Picture CC-BY-2.0 © Annie Roi @flickr.com
60

61. Registry
Spares
61

62. Registry
reg:Register
rdfs:label [1..*]
dct:description [1..*]
reg:owner [1] (foaf:Agent)
reg:manager [1] (foaf:Agent)
dct:license [0..*]
reg:containedItemClass [0..*]
reg:operatingLanguage [0..*]
reg:governancePolicy [0..*] (rdfs:Resource)
reg:validationQuery [0..*]
dct:modified [0..1] (inferred)
void:uriLookupEndpoint [0..*]
void:uriSpace [0..1]
void:exampleResource [0..*]
void:openSearchDescription [0..*]
reg:subregister
reg:RegisterItem
rdfs:label [1..*]
dct:description [0..*]
dct:dateSubmitted [1] (automatic)
dct:dateAccepted [0..1]
dct:modified [0..1] (inferred)
reg:itemClass [1..*]
reg:submitter [1] (foaf:Agent)
dct:license [0..*]
reg:status [1..*]
reg:category [0..*] (skos:Concept)
reg:notation [0..1]
reg:alias [0..*]
reg:hasView [0..*]
reg:representationOf [0..*]
reg:register
reg:predecessor
version:Version
owl:versionInfo [1]
time:Inteval
version:interval
dct:replaces
dct:replacedBy
version:VersionedThing
version:currentVersion
reg:Status
reg:statusNotAccepted
reg:statusSubmitted
reg:statusInvalid
reg:statusAccepted
reg:statusValid
reg:statusExperimental
reg:statusStable
reg:statusDeprecated
reg:statusSuperseded
reg:statusRetired
dct:isVersionOf
reg:status
reg:EntityReference
reg:entity [1]
reg:sourceGraph [0..1]
reg:definition
Void:Dataset
ldp:Container
ldp:membershipPredicate
reg:inverseMembershipPredicate
Full information model
62

63. Registry
Status lifecycle
63

64. Registry
Convenient views
• full RegisterItem/Register structure complex
• versioning makes that a lot worse
//registry
Register
VersionedThing
//registry:1
Register
Version
//registry/_reg
RegisterItem
VersionedThing
//registry/_reg:1
RegisterItem
Version
//registry/reg
Register
VersionedThing
//registry/reg/_foo
RegisterItem
VersionedThing
//registry/reg/foo
(entity)//registry/reg:1
Register
Version
//registry/reg/_foo:1
RegisterItem
Version
dct:versionOf
dct:versionOf dct:versionOf
dct:versionOf
reg:register
reg:register
reg:definition
reg:definition
//registry/_reg:2
RegisterItem
Version
//registry:2
Register
Version
//registry/reg:2
Register
Version
//registry/reg/_foo:2
RegisterItem
Version
64

65. Registry
Conceptual architecture
router
renderer
request
processor
user
credentials
roles and
bindings
auth
registry
core
logic
Registry
RDF store
text index
style and
templates
external
UI
admin
UI
log
audit
trail
storeAPI
nginx
proxy
conf
API
65

66. Registry
Aside on details
• internally a register item is more complex
– metadata about status, when submitted etc
– the description of the thing “entity”
Register
RegisterItem
label
description
status
submitter
item class
date submitted etc
...
register
register
entitydefinition
EntityReference
entitydefinition
EntityReference
RegisterItem
label
description
status
submitter
item class
date submitted etc
66

67. Registry
Structure – information model
• managed entity
– URL in registry namespace
– registry holds master copy of the entity data
Register http://.../def/catchment-planning/RiverBasinDistrict/
Register Item http://.../def/catchment-planning/RiverBasinDistrict/_UK05
Entity http://.../def/catchment-planning/RiverBasinDistrict/UK05
67

68. Registry
Structure – information model
• referenced entity
– URL external to registry (well, register)
– registry holds minimal copy of data
Register http://.../def/catchment-planning/RiverBasinDistrict/
Register Item http://.../def/catchment-planning/RiverBasinDistrict/_UK05
Entity http://agency.gov.uk/RDB/Anglia
68

69. Registry
Information model
complicated by:
– item v. entity
– versioning
reg:Register entity
reg:register
reg:definition
reg:entity
reg:RegisterItem reg:EntityReferencereg:RegisterItem reg:EntityReference
69

70. Registry
Information model
• default linked data view of Register is simplified
• configurable
– alternative membership property or inverse property
– so can make a register look like a skos:Collection,
skos:ConceptScheme, owl:Ontology ...
– also acts as a LDP container
• but can request full view (?_view=withMetadata)
reg:Register entity
reg:register
reg:definition
reg:entity
reg:RegisterItem reg:EntityReferencereg:RegisterItem reg:EntityReference
induced membership
relation default is
rdfs:member
container view
full view
70

71. Registry Federation, delegation and namespaces
reg:Delegated
reg:delegationTarget
reg:NamespaceForward
reg:forwardingCode [0..1]
reg:DelegatedRegister
reg:enumerationSubject [0..1]
reg:enumerationPredicate [0..1]
reg:enumerationObject [0..1]
reg:Register
reg:FederatedRegister
reg:forwardingCode [0..1]
71

72. Registry Federation, delegation and namespaces
Case 1: External entities
– identifier published in different namespace
– want to include it in authoritative list
Solution:
– just register as a referenced entity
– already seen this
– authoritative because it’s on the list
– can record properties of the entity, and maintain
history
– no namespace management involved
72

73. Registry
Referenced entities
/local
/id
/local-authority
Registry External service
e.g. opencommunities.org
Hosted by LA directly
73

74. Registry
Case 2: Namespace allocation
– want someone else to serve part of the registry
namespace
– might be a single item or a complete register sub tree
– e.g. allocating namespace in location.data.gov.uk for
serving INSPIRE spatial object identifiers
Solution:
– reg:NamespaceForward
– can be a redirect (30X) or proxy (200)
– no constraints on whether target acts like a Registry
– target ought to serve linked data with URIs in the right
namespace, but not required
Federation, delegation and namespaces
74

75. Registry
Namespace forward
/local
/id
/local-authority
Registry External web site
could be anything
75

76. Registry Federation, delegation and namespaces
Case 3: Federated register
– want someone else to run part of the registry
infrastructure but act like one big registry
– integrated search, validation etc
Solution:
– reg:FederatedRegister
– can be a redirect (30X) or proxy (200)
– target endpoint must comply with Registry API at
least for search, validation and entity lookup
76

77. Registry
Federated register
/local
/id
/local-authority
Registry
Federated registry
/local-authority
/id
77

78. Registry Federation, delegation and namespaces
Case 4: Delegating a register
– some one else to serve the list of contents of the
register
– but they only have triple store, not full registry
implementation
Solution:
– solution eg:DelegatedRegister
– specify SPARQL endpoint and triple
pattern to enumerate members
reg:DelegatedRegister
reg:delegationTarget [1]
reg:enumerationSubject [0..1]
reg:enumerationPredicate [0..1]
reg:enumerationObject [0..1]
78

79. Registry
Delegated register
/local
/id
/local-authority
Registry
External SPARQL service
79

80. Registry
Security model
• authentication
– OpenID (e.g. Google, Google profile)
• authorization
– permissions
• Register, Update, StatusUpdate, Force, Grant, GrantAdmin
• inherit down the tree
• e.g.: Register,Update:/example/local
– can grant to known user or anyone authenticated
– bundled into roles
• Maintainer – Update, Grant
• Manager – Register, StatusUpdate, Update, Grant
• Authorized – Register, Update, StatusUpdate - for experimental
areas
• Administrator - anything
80