This document is a presentation about completing a PhD and doing high quality research. It discusses the scientific method and different methodologies used in research, including qualitative and quantitative methods. It provides advice to students such as reading widely, discussing your research, and documenting your experiments. The presentation aims to inspire students and teach good lessons about the PhD process and research.

1. Valentina Tamma
University of Liverpool
The tao of knowledge:
the journey vs the goal
Picture by J. A. Alba, on Pixabay.com

2. V.TammaEKAW 2018 PhD Symposium
The brief
“A 45 min talk about completing a PhD
and doing high quality research that
would inspire and teach good lessons to
the students.”
!2
Picture by R. Higgins, on Pixabay.com

3. V.TammaEKAW 2018 PhD Symposium
The brief
“A 45 min talk about completing a PhD
and doing high quality research that
would inspire and teach good lessons to
the students.”
!3
Picture by R. Higgins, on Pixabay.com

4. V.TammaEKAW 2018 PhD Symposium
The brief
“A 45 min talk about completing a PhD
and doing high quality research that
would inspire and teach good lessons
to the students.”
!4
Picture by R. Higgins, on Pixabay.com

5. V.TammaEKAW 2018 PhD Symposium
The brief
“A 45 min talk about completing a PhD
and doing high quality research that
would inspire and teach good lessons
to the students.”
!5
Picture by R. Higgins, on Pixabay.com

6. V.TammaEKAW 2018 PhD Symposium
Pearls of wisdom: PhD Comics
!6
Piled Higher and Deeper by Jorge Cham www.phdcomics.com
title: "Frozen" - originally published 5/7/2014

7. V.TammaEKAW 2018 PhD Symposium
This talk
!7
Disney
Disney

8. V.TammaEKAW 2018 PhD Symposium
This talk
!8
Disney
Disney

9. V.TammaEKAW 2018 PhD Symposium
An apprenticeship in knowledge creation
• A PhD is an (individual) research
project involving advanced
scholarship, that makes an original
contribution to knowledge
• PhD:
• Philosophiae: from the Greek, meaning “love of knowledge”,
“pursuit of wisdom”, “systematics investigation”
• Doctor: from the classical Latin “Teacher” (to show, teach,
cause to know)
!9
Picture by fancycrave, on Pixabay.com

10. V.TammaEKAW 2018 PhD Symposium
Scientific method
“Research is systematic investigation to establish the facts.”
!10
Creswell, J. W.: Educational Research, 2008
• A research methodology defines what the activity of
research is, how to proceed, how to measure progress,
and what constitutes success.
• but the methodology depends on the scientific field of enquiry;
• Formal sciences: mathematics, logic, statistics, …
• Natural sciences: chemistry, biology, physics, …
• Social Sciences: psychology, linguistics, anthropology, …

11. V.TammaEKAW 2018 PhD Symposium
• Some argue that CS, and its
disciplines are not a science
• “CS it is the science of information processes
and their interactions with the world.” (P. Denning
2005).
!11
CS
AIKA &
KM
Is this science?

12. V.TammaEKAW 2018 PhD Symposium
• Some argue that CS, and its
disciplines are not a science
• “But the claim that artificial objects […] do not
lend themselves to natural-science methods of
research is fallacious. An artificial object is as fully
bound by the laws of nature as any natural
object. […] Scientific laws limit the set of possible
objects, natural or artificial.” (H. Simon 1993)
!12
CS
AIKA &
KM
Is this science?

13. V.TammaEKAW 2018 PhD Symposium
Where do KA and KM fit?
!13
requires rigorous
argumentsdemands
repeatable
experiments
takes theorems
as processes
asks whether an
object works
reliablyArea dependent
requirements

14. V.TammaEKAW 2018 PhD Symposium
When science blends with art
• “Science in the making”: The processes
by which scientific facts are proposed,
argued, & accepted;
• A new model appears as art whilst it is in the making;
• It becomes a “fact” only after it gains consensus.
• It is a messy, political, human process, fraught with emotion and
occasional polemics. (Denning 1995)`
• After sufficient time and validation, a
model becomes part of the scientific
body of knowledge.
!14
Picture by E. Riva, on Pixabay.com

15. V.TammaEKAW 2018 PhD Symposium !15
Adapted from G Dogdig-Crnkovic
State research
problem
Review existing
theories and
observations
Formulate
hypothesis
Deduce
consequence
and make
predictions
Evaluate the
hypothesis
Hypothesis must be redefined
Theory
confirmed or
proposed
Consistency
achieved
Selection
amongst
competing
theories
Hypothesis must be adjusted
Adapted by G. Dogdig-Crnkovic
Scientific
method - the
process

16. V.TammaEKAW 2018 PhD Symposium
The research problem
• The objective of the investigation.
• It identifies a problem / difficulty that needs solving;
• There must be some value attached to it and the
beneficiaries could be clearly identified;
• There might be alternative means to reach the
objectives;
• It should be feasible, not too generic or narrow
focussed;
• It should have some level of novelty;
• Can often be subdivided in and bounded by a
number of sub-questions.
!16
Picture by G. Altmann, on Pixabay.com

17. V.TammaEKAW 2018 PhD Symposium
In defence of good hypotheses
• Stating your hypotheses clearly is half
of the job done:
• Often there are many hypotheses
• that might be decomposed in a set of subsidiary hypotheses;
• Ambiguous hypotheses cause major
misunderstandings in the reader (reviewer!)
• Vague hypotheses lead to poor methodological
consequences:
• Inconclusive evidence;
• Research direction lacks focus.
!17
Picture by qimono, on Pixabay.com

18. V.TammaEKAW 2018 PhD Symposium
The hypothesis
• A conjectural answer to a
research question
• it is framed and scoped within the
context of existing knowledge;
• it is clearly formulated, with a
measurable / verifiable objective;
• should be refutable
• Popper's test for what constitutes science
!18
Which one?
1. Our alignment approach is better than
the ones presented in the state of the
art.
2. Our alignment approach performs
better than current systems on X
ontologies in the benchmark / on
ontologies with expressivity of type Y.
3. Our alignment approach improves
precision and/or recall wrt all of the
systems included in the evaluation
challenge, for the track Z.

19. V.TammaEKAW 2018 PhD Symposium
The hypothesis
• A conjectural answer to a
research question
• it is driven by a scientific problem;
• it states the why, how and possibly
the who;
• it is framed by a task;
• which is particularly important when some
artefact is produced as part of the research;
!19
Which one?
1. We designed an ontology that
effectively models domain X.
2. We designed an ontology that
models task Y in domain X, and
aims to answer the following
competency questions.
3. We developed a new ontology editor
and users like it;
4. Our ontology editor facilitates the
editing of large ontologies by
domain experts;

20. V.TammaEKAW 2018 PhD Symposium
Dimensions of investigation
• Properties can be investigated across different levels:
• properties of a technique vs those of its parameters,
• inherent properties of a task vs a complete system;
• relationship between tasks, parameters, systems;
• Different dimensions for the comparison:
• scientific;
• engineering;
• cognitive science;
• Different means of investigation:
• Theoretical;
• Experimental;
!20
Picture by G. Altmann, on Pixabay.com

21. V.TammaEKAW 2018 PhD Symposium !21
Scientific Engineering Cognitive
Behaviour: The effect or result of the
method. The absolute or comparative
assessment wrt an external "gold
standard"
Dependability: The reliability, security
and safeness of the system
implementing the method
External: The model exhibits the
appropriate external behaviours (similar
to scientific behaviour, but the baseline is
different)Coverage: The range of application of
the method. It identifies a set of
situations to which its application is
relevant
Usability: The ease of use of the system
from the perspective of the end user
Internal: The model works in the same
way as the phenomenon / observation
that it models
Efficiency: The resources consumed by
the method. The resources measured are
usually time or space.
Maintainability: The ability of the
system to evolve in order to meet
changes in the user's requirements.
Adaptability: The model accounts for a
wide range of occurring behaviours
Scalability: The potential for the system
to continue to work within realistic
resource limits on the most complex
examples
Evolvability: The model truthfully
represents the evolution or learning of the
ability it models
Cost: Resources (developer time,
money…) needed to build and/or
maintain the system
Fitness: The extent to which the system
adheres to the user’s requirements
Adapted from A. Bundy: The need for hypotheses in informatics
Dimensions of
investigation

22. V.TammaEKAW 2018 PhD Symposium
Predictions
• From hypotheses we can derive
predictions:
• Hypothesis: theory explaining why a
phenomenon occurs
• testable hypotheses
• Prediction: using the hypothesis, scientists
calculate the measurable data points they
believe will result in a given experiment
• often involves different properties of the model being
developed
!22
Picture by G. Altmann, on Pixabay.com

23. V.TammaEKAW 2018 PhD Symposium
Evaluation
• Demonstrate / show that the
hypothesis holds:
• Theoretical proof
• checks properties such as correctness, completeness,
termination and complexity
• Experimental evaluation:
• Quantitative research method
• Qualitative research method
• Can sometimes be used together to evaluate
different aspects of a proposed method
!23
Picture by G. Altmann, on Pixabay.com

24. V.TammaEKAW 2018 PhD Symposium
Inclusive models of scientific research
• Scientific research is concerned with
stating knowledge claims
• These claims need to be evaluated and
validated in some way
• Depending on the specific area of interest
(and on the aspect we are evaluating)
given research methods are employed,
e.g.:
• Case study, Experiment, Survey, Proof
!24

25. V.TammaEKAW 2018 PhD Symposium
Qualitative and quantitative methods
• Quantitative:
• Methods associated with
measurements (on numeric
scales)
• Prevalent in natural sciences
• Used to test hypotheses or create
a set of observations for inductive
reasoning
• Accuracy and repeatability are
imperative
• Qualitative
• Methods involving case studies
and surveys
• Prevalent in social sciences
• Used to generate comprehensive
description of processes,
mechanisms, or settings
• Characterise participant
perspectives and experiences
!25

26. V.TammaEKAW 2018 PhD Symposium
Mixing the two
!26
Approach: Inductive
Goal: Depth, local meaning,
generate hypotheses
Setting: Natural
Sampling: Purposeful
Data: Words, Images; Narrow but rich
Data analysis: Iterative interpretation
Values: Personal involvement and partiality
(subjectivity, reflexivity)
Approach: Deductive
Goal: Breadth, generalisation,
test hypotheses
Setting: Experimental
Sampling: Probabilistic
Data: Numbers; Shallow but broad
Data analysis: Statistical tests, models
Values: Detachment and impartiality
(objectivity)
MIXED
Adapted from B. Young and D. Hren: Introduction to qualitative research methods
Qualitatitve Quantitatitve

27. V.TammaEKAW 2018 PhD Symposium
Qualitative studies
• Tell the reader about the design being used
• the use of qualitative research and the intent behind it
• But also involves discussing:
• the sample for the study,
• the data collection process,
• the recording procedure;
• Allows inductive and deductive data analysis;
!27

28. V.TammaEKAW 2018 PhD Symposium
A use case: ontology engineering
• Anecdotal evidence tells us that robust
evaluation and validation methods are still
not widespread in ontology engineering
• ISWC 2017 resources track (76 submissions)
• JWS special issue on ontology engineering (41
submissions)
• Often characterised by misunderstandings between reviewers and
authors opinions on what constitutes a suitable evaluation
!28

29. V.TammaEKAW 2018 PhD Symposium
The artefact and the process
• Often these papers describe an
ontology and the process used to
construct it
• There are a number of quantitative metrics to
evaluate an ontology:
• Accuracy, Completeness, Conciseness, Consistency,
Computational efficiency, Adaptability, Clarity…
• Less defined methods are used for the
construction process:
• but field studies, use cases, interviews, observations, longitudinal
studies can help to add further detail
!29

30. V.TammaEKAW 2018 PhD Symposium
ISWC 2017, Resources track
!30
Criteria How to
Is the ontology logically correct? Proofs (reasoners integrated in ontology editors), quantitative,
Is the chosen design suitable for the intended
purpose? Qualitative
Is the chosen design of high quality? (e.g., no
hacks and workarounds, no redundancy)
Quantitative
Have other resources been reused? E.g., upper
level ontologies, design patterns
Qualitative (reflective)
Is the documentation of good quality? Are the
core ideas of the ontology described?
Quantitative & Qualitative

31. V.TammaEKAW 2018 PhD Symposium
Word of caution
• Designing qualitative experiments
requires care:
• resource intensive
• significant sample of domain experts / users
• to support stratification
• control groups
• ordering of questions and groups answering
questions is important
• types and wording of questions, and question format
• support for different types of research (basic,
applied and summative)
!31
Picture by Clker-Free-Vector-Images, on Pixabay.com

32. V.TammaEKAW 2018 PhD Symposium
Some advice
• Read, read, read… and read
• Read anything that captures your imagination
• Read with questions in mind:
• “How can I use this?”
• “Does this really do what the authors claim?”
• “Do I understand the results in the paper?”
• Talk about your research
• To your supervisor(s), to your colleagues, to
students in other departments
• It will help you hone and shape the arguments
!32
Picture by Free-Photos, on Pixabay.com

33. V.TammaEKAW 2018 PhD Symposium
Some advice
• Divide your time between activities
• Proving your hypotheses, writing about your
research, etc
• Document your experiments:
• Make an experimental plan, describe in detail
materials, methods and participants
• There is always light at the end of the
tunnel!
!33
Picture by Free-Photos, on Pixabay.com

34. V.TammaEKAW 2018 PhD Symposium
Conclusion: part 1
• KA & KM is a science:
• And borrows methodological aspects from other disciplines;
• And all these are needed:
• The synergy gives further strengths and novel insights
• because they complement each other’s limitations.
• We should exploit these synergies in our
research by using the appropriate research
methodologies
• we should become familiar with other research methods and
be prepared to use and adapt evaluation methods from other
disciplines
!34

35. V.TammaEKAW 2018 PhD Symposium
Conclusion: part 2
• There is value in well designed
qualitative methodologies
• They might be necessary to evaluate some
unique aspects of our research
• And there is an opportunity for
creating new methods by creatively
combining quantitative and qualitative
• whilst striving for the rigour and precision of
these methods
!35
Picture by mohamed Hassan , on Pixabay.com

36. V.TammaEKAW 2018 PhD Symposium
See you at the next…
• Conference…
• EKAW, K-Cap, ISWC, ESWC, WWW, ECAI
IJCAI, AAAI
• … or Journal
• Journal of Web Semantics, Semantic Web
Journal, Transactions on Knowledge and Data
Engineering, Data Semantics, Applied Ontology,
Knowledge Engineering Review, Artificial
Intelligence Journal, Journal of AI Research
!36
Disney

37. V.TammaEKAW 2018 PhD Symposium
Some useful resources
• H. Akkermans and J. Gordijn1
: Ontology Engineering, Scientific
Method, and the Research Agenda. In Proceedings of EKAW
2006
• A. Bernstein and N. Noy: Is This Really Science? The Semantic
Webber’s Guide to Evaluating Research Contributions. Technical
report:
https://www.merlin.uzh.ch/publication/show/9417
• A. Bundy: The need for hypotheses in informatics. Technical report:
http://www.inf.ed.ac.uk/teaching/courses/irm/notes/
hypotheses.html
• D. Chapman (Ed): How to do Research At the MIT AI Lab.
Technical report:
https://dspace.mit.edu/handle/1721.1/41487
• P.R. Cohen: Empirical methods for Artificial Intelligence. 1995
• J.W. Creswell. Educational Research: Planning, Conducting, and
Evaluating Quantitative and Qualitative Research. 2008
• P. Denning: Is Computer Science Science. Comms of the ACM,
Vol. 48, No. 4, 2005
• G. Dodig-Crnkovic: Scientific Methods in Computer Science. In
Proceedings of the Conference for the Promotion of Research in IT
at New Universities and at University Colleges in Sweden, 2002
• C. M. Judd, E.R. Smith, L.H. Kidder: Research methods in social
relations, 1986
• B. Latour. Science in action. 1988
• K. Popper: The Logic of Scientific Discovery, 1934
• H. Simon: Artificial Intelligence: An empirical Science. In Artificial
Intelligence Journal, vol 77, issue 1
• V. Tamma and F. Lecue: ISWC 2017 Resources Track: Instructions
for Authors and Reviewers. Technical report: https://goo.gl/
426CEv2
!37