ISWC 2019 Spotlight paper

1. Agenda
• What is microbenchmark?
• Why microbenchmarks for QA?
• QaldGenDataset for microbenchmarking
• QaldGen microbenchmarking framework
11/7/2019 QaldGen 2

2. What is Microbnechmark?
• Specialized
• Focused
• Use-case specific
• Useful for componenet-level testing
• E.g, single relation QA benchmark
11/7/2019 QaldGen 3

3. Why Microbnechmarks for QA?
• Multiple QA-related factors are influecing the F scores
• Fine-grained testing of the systems
• Pinpoint more detailed limitations
• Some systems are designed for special purpose or use-case
• Avoids comparing apple with oranges
11/7/2019 QaldGen 4

4. Effect of Question Type
0.3
0.35
0.4
0.45
0.5
0.55
0.6Avg.Fscore When In which Who What Which Where Give me How many
Easy Difficult
5
From QALD6 results

5. Effect of #Triple Patterns
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Avg.FScore
#TP = 1 #TP = 2 #TP = 3
#Triple Patterns has inverse relationship with the F score, i.e., the
more the Triple patterns the harder the question to answer
6

6. Comparison on SPARQL query forms
SPARQL ASK queries are much difficult to
answer as compared to SPARQL SELECT
0.3
0.32
0.34
0.36
0.38
0.4
0.42
0.44
0.46
Avg.FScore
SELECT ASK
7

7. Effect of #Answer
#Answers have direct relationship with the F score, i.e.,
the more answer the easiest the question to answer
0.35
0.37
0.39
0.41
0.43
0.45
0.47
0.49
0.51
Avg.FScore
#A = 1 #A = 2 #A = 3
8

8. Effect of Answer Types
When answer is type of Date then it is more easy to answer
and when answer is type of string it is difficult to answer
0.35
0.37
0.39
0.41
0.43
0.45
0.47
0.49
0.51
Avg.FScore
String Boolean
Resource Date
9

9. Effect of Aggregates functions
If the SPARQL query corresponding to a question has
aggregate functions then it is difficult to answer
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Avg.FScore
Aggregates = true
10

10. Why Micobenchmarking for QA?
• QALD-6 Winner: CANALI QA System
• But, for questions starting “Give me” UTQA is winner
• NLIWOD SPARQL Query Builder has F-score
0.48 -was reported as baseline over LC-QuAD
• But, SINA on same dataset for two triples patterns in
SPARQL queries has F-Score 0.80
• But, SINA reports F-score 0.0 for SPARQL queries with
four triples
11/7/2019 QaldGen
Definition of “baseline” is subjected
to type of the questions, and specific
features of the questions
11

11. What is QaldGen?
• Micorbenchmarks selector framework
• For QA over Linked Data
• Personalized benchmarking
• Reusing LC-Quad and QALD
• Based on graph clustering
11/7/2019 QaldGen 12

12. QaldGen Architecture
11/7/2019 QaldGen 13
QaldGenQaldGenData
Questions
Selection
Vectors
Representation
Clusters
Formation
Final
Questions
Selection
QaldGen
RDF
Benchmark
Contribution 1 Contribution 2

13. QaldGenData
• RDF Dataset for QA over linked data
• NER related testing
• Relation linking testing
• QA systems testing
• Constructed from LC-Quad and QALD-9
• 408 questions from QALD-9
• 5000 questions from LC-Quad
• Annotated each question with 51 features
11/7/2019 QaldGen 14

14. QaldGenData: Annotations
11/7/2019 QaldGen 15
QaldGen
L(Q)
Answer
Entities,
Relations,
Classes
Question
Total:
Entities,
Relations,
Classes
Entity1,
Relation1,
Class1
L(Entity1,2,3),
L(Relation1,2,3),
L (Class1,2,3)
Avg. words:
Entities,
Relations,
Classes
POS
tags
IN,JJ,JJR,JJS,NN,
NNS,NNP,NNPS
,PRP,RB,RBR,RB
S,VB,VBD,VBG,
VBN,VBP,VBZ,
WDT,WP,SYM
Origin
QALD9 or
LC-QuAD typeBoolean? Number? SPARQL
Text
Tota
l TPs
Total
BGPs
SPARQL
feature
s
Ans
wers
Result
size
Proj.
Vars
Text

15. QaldGen: Micorobenchark Selection
11/7/2019 QaldGen 16
QaldGenQaldGenData
Questions
Selection
Vectors
Representation
Clusters
Formation
Final
Questions
Selection
QaldGen
RDF
Benchmark

16. QaldGen: Micorobenchark Selection
11/7/2019 QaldGen 17
QaldGenQaldGenData
Questions
Selection
Vectors
Representation
Clusters
Formation
Final
Questions
Selection
QaldGen
RDF
Benchmark

17. Step1: Questions Selection
• From QaldGenData
• Using a single SPARQL query
• Select desired features as projections
• Options for personalization
• Use-case specific benchmarking
• Using SPARQL constructs, e.g., Filters
11/7/2019 QaldGen 18

18. Step2: Vectors Representation
19
Question: " What is the time zone of SaltLake City?
Golden Answer:
SELECT DISTINCT ?uri
WHERE
{
<http://dbpedia.org/resource/Salt_Lake_City>
<http://dbpedia.org/ontology/timeZone> ?uri
}
Annotated features as projections
Question Related:
• Total Words: 9
• Total Entities: 1
• Total Relations: 1
• Total Classes: 0
• Avg. Entities Words: 1
Answer Related
• #Triple patterns: 1
• #Results: 1
• #BGPs: 1
• #Projection Vars: 1
Feature Vector
Normalized Feature Vector
11/7/2019 QaldGen
9 1 1 0 1 1 1 1 1
0.3 0.14 0.2 0 0.36 0.33 0.06 0.5 0.33

19. Step3: Clusters Formation
• We do clustering to get sufficient diversity in the selected benchmark
• Draw vectors on multi-dementional space and get clusters
• QaldGen supports
• Kmeans++
• FEASIBLE
• Agglomerative
• Random
• FEASIBLE-Exemplars
• DBSCAN+Kmeans++
• Others clustering techniques can easily be integrated
11/7/2019 QaldGen 20

20. Step3: Clusters Formation
21
Plot feature vectors in a multidimensional space
Q F1 F2
Q1 0.2 0.2
Q2 0.5 0.3
Q3 0.8 0.3
Q4 0.9 0.1
Q5 0.5 0.5
Q6 0.2 0.7
Q7 0.1 0.8
Q8 0.13 0.65
Q9 0.9 0.5
Q10 0.1 0.5
Suppose we need a benchmark of 3 questions
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
11/7/2019 QaldGen

21. 22
Calculate average point
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
Avg.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
11/7/2019 QaldGen
Step3: Clusters Formation

22. Step3: Clusters Formation
23
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Select point of minimum Euclidean distance to avg. point
*Red is our first exemplar
11/7/2019 QaldGen

23. Step3: Clusters Formation
24
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Select point that is farthest to exemplars
11/7/2019 QaldGen

24. Step3: Clusters Formation
25
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
11/7/2019 QaldGen

25. Step3: Clusters Formation
26
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Select point that is farthest to exemplars
11/7/2019 QaldGen

26. Step3: Clusters Formation
27
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
11/7/2019 QaldGen

27. Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Step3: Clusters Formation
28
Calculate distance from Q1 to each exemplars
11/7/2019 QaldGen

28. Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Step3: Clusters Formation
29
Assign Q1 to the minimum distance exemplar
11/7/2019 QaldGen

29. Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Step3: Clusters Formation
30
Repeat the process for Q2
11/7/2019 QaldGen

30. Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Step3: Clusters Formation
31
Repeat the process for Q3
11/7/2019 QaldGen

31. Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Step3: Clusters Formation
32
Repeat the process for Q6
11/7/2019 QaldGen

32. Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Step3: Clusters Formation
33
Repeat the process for Q8
11/7/2019 QaldGen

33. Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Step3: Clusters Formation
34
Repeat the process for Q9
11/7/2019 QaldGen

34. Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Step3: Clusters Formation
35
Repeat the process for Q10
11/7/2019 QaldGen

35. Step4: Final Questions Selection
36
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
Avg.
Avg.
Avg.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Calculate Average across each cluster
11/7/2019 QaldGen

36. Step4: Final Questions Selection
37
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
Avg.
Avg.
Avg.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Calculate distance of each point in cluster to the average
11/7/2019 QaldGen

37. Step4: Final Questions Selection
38
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
Avg.
Avg.
Avg.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Select minimum distance question as the final benchmark
query from that cluster
Purple, i.e., Q2 is the final selected question from yellow cluster
11/7/2019 QaldGen

38. Step4: Final Questions Selection
39
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
Avg.
Avg.
Avg.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Select minimum distance query as the final benchmark
query from that cluster
Purple, i.e., Q3 is the final selected question from green cluster
11/7/2019 QaldGen

39. Step4: Final Questions Selection
40
Q1
Q2 Q3
Q4
Q5
Q6
Q7
Q8
Q9Q10
Avg.
Avg.
Avg.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Select minimum distance query as the final benchmark
query from that cluster
Purple, i.e., Q8 is the final selected question from brown cluster
Our benchmark queries are Q2, Q3, and Q8
11/7/2019 QaldGen

40. Evaluation Setup
• GERBIL and Frankenstein frameworks are used to provide common
experiment settings
• Personalised 200 question sample is generated using QaldGen for
evaluating end to end QA system
• Personalised 100 question samples are generated using QaldGen for
evaluating NED and Relation linking tools
• Precision, Recall, F-score are measured to report final performances.
11/7/2019 QaldGen 41

41. Results
11/7/2019 QaldGen 42
Systems Dataset Task P R F
Baseline(gAns
wer2)
QALD-9 QA system 0.29 0.33 .030
gAnswer2 QaldGen QA system 0.24 0.24 0.24
QUEPY QaldGen QA system 0.27 0.27 0.27
Baseline
(TagMe)
LC-QuAD NED 0.69 0.67 0.68
TagMe QaldGen NED 0.44 0.40 0.41
DBpedia
Spotlight
QaldGen NED 0.37 0.38 0.36
Baseline
(RNLIWOD)
LC-QuAD RL 0.25 0.22 0.23
RNLIWOD QaldGen RL 0.23 0.19 0.20
EARL QaldGen RL 0.38 0.39 0.37

42. QaldGen Demo (http://qaldgen.aksw.org/)
11/7/2019 QaldGen 43

43. Conclusions
• The definition of a “baseline” is subjected to the type of questions
considered
• Overall baseline on a dataset may not be a baseline for particular type of
questions
• Reason: A QA system or its component for modular framework may not be
implemented to target some type of questions
• Microbenchmarking helps to understand exact pitfalls of a system
• For future
• Integrate LCQuad 2.0
• Google QA dataset
11/7/2019 QaldGen 44

44. Thanks
• QaldGen is open source https://github.com/dice-group/qald-generator
• ISWC 2019 Demo available from http://qaldgen.aksw.org/
11/7/2019 QaldGen 45