1. Roberto García and Rosa Gil
HCI & Data Integration Research Group
Universitat de Lleida, Spain
Social Media Copyright Management
using Semantic Web and Blockchain
21st International Conference on Information Integration and Web-based Applications & Services (iiWAS2019)
December 2-4 2019, Munich, Germany

2. 2
Motivating Scenario
• Reuse of User Generated Video (UGV) from Social Media for
journalistic purposes
• H2020 project InVID, In Video Veritas – Verification of Social
Media Video Content for the News Industry
https://www.invid-project.eu
1. Discovering social media about current events
2. Video verification to avoid fake news
3. Request reuse, check licensing, negotiate terms, sign agreements,… even
economic compensation
• WP4. Rights Management – Led by Universitat de Lleida
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19

3. 3
Objectives
• Sophisticated models for copyright information:
• Rights status
• Reuse terms
• Negotiation
• Copyright agreements
• Trust and confidence on rights statements
• Potentially legally binding (time stamp, signatures, tamper proof,…)
• Proposed approach:
• Semantic Web: rich information modelling and reasoning
• Blockchain: immutable and accountable information storage
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19

4. 4
Semantic Web – Copyright Ontology
• Copyright knowledge representation
• Copyright Ontology: based on fundamental ontological distinction:
• Abstract: intangible
• Process: happens,
temporal stages
(action, event,…)
• Object: can be defined
independent of time
(includes digital objects)
Victor Hugo’s
Les Misérables
Abstract
Objects
Processes
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19

5. 5
Copyright Ontology
• Also capture the dynamic
parts of the copyright value chain
• Actions performed
by value chain participants
• Plus consumer actions:
• Buy, Attend, Access,
Play, Tune,…
• Plus licensing actions:
• Agree/Disagree
• Transfer, Attribute,…
Victor Hugo’s
Les Misérables
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19
Creator Actor Producer Broadcaster User
Motion PictureScript
Adaptation Performance
manifest perform record
Communication
broadcastadapt
Literary Work
tune

6. 6
• Model full details of an action,
its dimensions like as a verb in
a sentence (roles):
• who performs it, what is
manipulated, when, where…
Copyright Ontology
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19
Who?
What?
When? Where?
Role Kind Main Role Description
who schema:agent
The direct performer or driver of the
action (animate or inanimate)
schema:participant
Other co-agents that participated in
the action indirectly, for instance a
recipient
what schema:object
The object upon which the action is
carried out
schema:result The result produced in the action
where schema:location Where an action takes place
schema:fromLocation
The original location of the object or
the agent before the action
schema:toLocation
The final location of the object or the
agent after the action
when schema:startTime
When the action started or the time it
is expected to start
schema:endTime
When the action finished or the time
it is expected to end
pointInTime
The point in time when the action
happens
duration
The amount of time the action
requires to complete
with schema:instrument
The object that helps the agent
perform the action
why aim The reason or objective of the action
how manner The way the action is carried out
if condition
Something that must hold or happen
before the action starts
then consequence
Something that must hold or happen
after the action is completed

7. 7
Architecture
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19
Rights
Management
Services
Rights
Database
InVID Rights
Management
Rights Data
Distributed
Ledger
Identify
Provider
Semantic
Repository
Users & Orgs
DB
Rights Management App
(for Journalists and Content Owners)
uPort
(self-sovereign id)
Identify
Attestation
Semantic Copyright
Management

8. 8
Check UGV Rights Status
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19
https://rights.invid.udl.cat

9. 9
Show Reuse Conditions
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19

10. 10
Request Reuse
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19
Current video, plus
all future YouTube videos by content owner or
in any social network linked to InVID profile

11. 11
Accept or Negotiate
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19

12. 12Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19
License Reasoning
• Streamline licensing
• License to organisation or everyone
• License future videos
• Semantic representation
of agreements
• Semantic queries to check
previous agreements
• Including territories, timeframes
or revocations
InVID Rights
Management
Rights
Database
JSON-LD
Semantic
Repository
Semantic
Copyright
Management

13. 13
• JSON-LD serialisation of a
Reuse Agreement:
• Grants any member of the Daily
Planet permission to republish a
YouTube video whose owner is the
Google user
Store Agreement
{
"@context": {
"@vocab": "http://invid.udl.cat/ontology/",
"cro": "http://rhizomik.net/ontologies/copyrightonto.owl#",
"schema": "http://schema.org/"
},
"@id": "…/reuseAgreements/1", "@type": "cro:Agree",
"cro:when": "2019-02-16T15:15:00Z",
"cro:who": [
{
"@id": ”…/inVIDUsers/1", "@type": "schema:Person",
"schema:name": "Clark Kent",
"schema:email": "journalist@invid-project.eu",
"schema:memberOf": {
"@id": "…/organizations/1",
"@type": "schema:Organization",
"schema:name": "Daily Planet"
}
},
{
"@id": "…/contentOwners/1", "@type": "schema:Person",
"username": "user”, "schema:email": "user@gmail.com",
"schema:name": "Google User"
} ],
"cro:what": {
"@id": "…/reuseTerms/1", "@type": "cro:MakeAvailable",
"schema:startTime": "2019-03-01T10:44:00Z",
"schema:endTime": "2019-05-01T10:44:00Z",
"cro:who": { "@id": "…/organizations/1" },
"cro:what": {
"@id": "…/youTubeVideos/_5l7vn1QdKM", "@type": "YouTubeVideo",
"user": {
"@id": "…/youTubeChannels/MyChannel", "@type": "YouTubeChannel",
"contactURL": "http://www.youtube.com/channel/MyChannel/about",
"contentOwner": { "@id": "…/contentOwners/1" }
}
}
}
}
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19

14. 14
License Reasoning
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19

15. 15
• SPARQL standard for semantic
queries
• Check intended reuse against
existing agreements
• Encapsulate complexities minimising
implementation cost
• Flexibility and scalability
• Example:
• Active agreements, not disagreed,
with Make Available term
• what: restricted to the YouTube
video _5l7vn1QdKM
• startTime: 2019-11-15
• who: is InVIDUser 2, Organization 1
or any organization InVIDUser 2 is a
member of
• where: Spain or a region Spain is
contained in
SPARQL for License Reasoning
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19
PREFIX …
SELECT DISTINCT ?isAuthorized ?why
WHERE {
?agree rdf:type cro:Agree ;
cro:what ?term ; cro:when ?agreeDate
FILTER ( xsd:dateTime( ?agreeDate) <= now() )
OPTIONAL {
?disagree rdf:type cro:Disagree ;
cro:what ?term ; cro:when ?disagreeDate
FILTER ( xsd:dateTime( ?disagreeDate) <= now() )
}
BIND((bound(?agree) && (!bound(?disagree))) AS ?isAuthorized)
BIND(if(bound( ?disagree), ?disagree, ?agree) AS ?why)
?term rdf:type cro:MakeAvailable .
?term cro:what <…/youTubeVideos/_5l7vn1QdKM> .
?term schema:startTime ?start FILTER ("2019-11-15" >= ?start)
…
{
{ ?term cro:who <…/inVIDUsers/2>}
UNION
{ ?term cro:who <…/organizations/1>}
UNION
{ ?term cro:who ?organization .
<…/inVIDUsers/2> schema:memberOf ?organization }
}
{
{ ?term cro:where "Spain"}
UNION
{ ?term cro:where ?regionName .
?region rdfs:label ?regionName .
?country rdfs:label "Spain" .
?country (schema:containedInPlace)+ ?region
}
}
}

16. 16
Trustful Agreements
• Use Ethereum Smart Contracts
• Blockchain as a global shared computer
• Immutable transactions (executed in all nodes)
• Encode rules guaranteed to execute
• Smart contract keeps track of semantic agreements
• Participants digitally sign negotiation steps, last by both (agreement)
• Identity management using uPort mobile app
• Self-Sovereign Identities (e.g. email attestations)
• Transaction signing: scan QR code
• Optional: remuneration using cryptocurrency wallet
https://www.uport.me
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19
Rights
Database
InVID Rights
Management
Distributed
Ledger
Agreement
Time Stamping
Accountability
Auditability
Tamper Proof
Identify
Attestations

17. 17
Reuse Request
address journalist
address contentOwner
int stepstCount
bool revoked
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19
Ethereum Blockchain Implementation
smart contract
Reuse Request
address journalist
address contentOwner
int stepstCount
bool revoked
Reuse Request
address journalist
address contentOwner
int stepsCount
bool revoked
Negotiation Step
bool signedJournalist
uint64 journalistTimestamp
bool signedOwner
uint64 ownerTimestamp
string hash
Negotiation Step
bool signedJournalist
uint64 journalistTimestamp
bool signedOwner
uint64 ownerTimestamp
string hash
https://rights.invid.udl.cat/blockchain/guide
Reuse Terms
(JSON-LD)
Reuse Terms
(JSON-LD)

18. 18
Future Work
• Smart contract for copyright registration
• Link to social media content through trusted oracles (off-chain data)
• Upload authorship evidences to facilitate disputes resolution
• Work in progress: copyrightly.io
• Smart contract for usage tracking
• Register copyright uses
• Decentralized tracking against registered agreements
• On-chain escrow service for crypto payments
• Smart contracts for dispute resolution
• Legal experts participation through incentive and reputation mechanisms
• Connect to existing projects like kleros.io
• Courts as last resort using evidences collected on-chain
Social Media Copyright Management using Semantic Web and Blockchain - iiWAS’19