This paper proposes a reputation framework to identify community leaders in ontology engineering projects. The framework uses sensors to assign reputation scores based on objective and subjective characteristics of community leaders, such as community activity, quality of interactions, and engagement of others. An experiment applying this framework in an ontology engineering course found considerable overlap between the scores it produced and a survey identifying project drivers, encouraging further investigation. The paper concludes the framework provides a means to identify community leaders but notes limitations, such as interactions outside collaborative platforms, that require additional experiments across different ontology projects.

1. Using a Reputation Framework to
Identify Community Leaders in
Ontology Engineering
(short paper)
Christophe Debruyne and Niels Nijs
Vrije Universiteit Brussel STARLab
11-09-2013 @ ODBASE 2013
vrijdag 13 september 13

2. Introduction
‣ Ontology Engineering
‣ ... is a social process
‣ ... is far from trivial → requires appropriate methods & tools
‣ Workflow, roles, and responsibilities
‣ Community Leaders
‣ Members in the stakeholder group that drive the ontology project
‣ Problem: the identification of community leaders in ontology
engineering for the automatic assignment of responsibilities.
2
vrijdag 13 september 13

3. Introduction
‣ Trust and Reputation Systems
‣ Are used to
‣ ... increase the reliability and trust between agents
‣ ... improve contribution quality
‣ ... build or increase co-operation
‣ ...
‣ Have been applied to grant rights/privileges to certain users
‣ Are reputation frameworks suitable for identifying community
leaders in an ontology-engineering project?
‣ Approach
‣ Identify the characteristics of a community leader
‣ Propose framework and “sensors” assigning scores
‣ Apply the framework in an ontology engineering experiment
‣ Compare the output with results from survey
3
vrijdag 13 september 13

4. Towards a reputation framework
‣ Characteristics of a community leader:
(C1) Energy, passionate persistence & optimism
(C2) Goal-Driven
(C3) Build Trust
(C4) Willing to take risks
(C5) Pull and communicate with others
(C6) Work systematically
(C7a) Share knowledge, power and credit
(C7b) Work interdependently
(C8) Understand others
‣ Sensors for ...
(A1) Community activity
(A2) “Quality” of interactions
(A3) Engage others
(A4) Quality of results (annotations, for instance)
(A5) Cross-community activity
‣ Objective vs. Subjective Sensors (!)
4
A1 A2 A3 A4 A5
C1 X X X
C2 X X X
C3 X
C4
C5 X X X X
C6 X X
C7a
C7b X
C8 X
“Coverage”
vrijdag 13 september 13

5. Towards a reputation framework
‣ We define A as the set of all human agents
‣ We define P as the set of all platforms
‣ Reputation results R is defined as [0; 100] ∪ {⊘}
‣ Platform configurations ⟨p,wp,S⟩
‣ p a platform in P
‣ wp the weight of the platform
‣ S a set of sensor configurations ⟨s,ws⟩
‣ s a reputation sensor and ws the weight of the sensor
‣ Compute the reputation scores of a user for a particular platform
5
platform p1platform p1platform p1
s1 s2 s3
a1
a2
a3
...
50 25 75
60 80 ⊘
⊘ ⊘ ⊘
platform p1
result
a1
a2
a3
...
50
70
⊘
vrijdag 13 september 13

6. ‣ For every user, we compute the result for every platform
‣ We remove users with no results
‣ We remove platforms with at most 1 result
‣ Compute z-scores for each platform and rescale
‣ distance between given value and mean in number of st.devs
‣ Overall score is the weighted average of all rescaled scores
Towards a reputation framework
6
p1 p2 p3 p4
a1
a2
a3
a4
60 80 90 ⊘
50 40 ⊘ ⊘
⊘ ⊘ ⊘ ⊘
80 ⊘
p1 p2
a1
a2
a4
60 80
50 40
80 ⊘
p1 p2
a1
a2
a4
-0,27 1
-1,07 -1
1,34 ⊘
p1 p2
a1
a2
a4
45,5 33,33
32,2 66,67
72,3 ⊘
z-score rescale
vrijdag 13 september 13

7. Experiment
‣ Experiment: large ontology engineering project
‣ 36 students in the MSc in Computer Science program
‣ Held in the context on a course on ontology engineering
‣ First develop own information system
‣ Then develop ontologies to
‣ ... enable semantic interoperability between the systems
‣ ... annotate an existing system
‣ Survey
‣ On voluntary basis, give at most three names of those considered to
have driven the project
‣ Analyze “overlap” between responses and scores obtained by the
reputation framework
7
vrijdag 13 september 13

8. Experiment
8
vrijdag 13 september 13

9. ‣ 17 of the 36
participants
participated
in the survey
‣ Considerable
overlap
encourages
further
investigation
Data and Results
9
vrijdag 13 september 13

10. Limitations and Conclusion
‣ Limitations
‣ Interactions outside of the collaborative platform
‣ Additional experiments (planned for March 2013)
‣ Types of ontology-engineering projects and communities
‣ Conclusions
‣ Proposed a means for identifying community leaders in ontology
engineering using a reputation framework
‣ Objective and subjective reputation sensors
‣ Applied the reputation framework in an ontology-engineering
project and validated the results using a survey
10
vrijdag 13 september 13

11. Fin. Questions?
vrijdag 13 september 13