This document discusses the maturity of ontology development as an engineering discipline. It analyzes ontology development using the Capability Maturity Model (CMM), which assesses the formality and optimization of software development processes. While ontology development has established technologies and methods, it still relies heavily on ad hoc practices. The document argues ontology development would benefit from increased understanding of development processes, tool support for those processes, and the ability to quantitatively measure and manage processes. This would help move ontology development from a reliance on individual expertise to more mature and repeatable engineering practices.

1. The state of the
Nation for Ontology
Development
Robert Stevens
School of Computer Science
University of Manchester
Oxford Road
Manchester
United Kingdom
M13 9PL
Robert.Stevens@Manchester.ac.uk
http://www.cs.manchester.ac.uk/our-research/

2. What’s the state of ontology
development?
 Ontologies are fairly well established as supports to
information systems
 We have KR languages like OWL that are widely
used
 We have at least one well established OWL API
 We have varieties of tools that are research
outcomes
 We have lots of opinions on development processes
 How mature are we as an engineering discipline
and what does it say about where to go next?

3. What is maturity?
 The term "maturity" relates to the degree of formality
and optimization of processes, from ad hoc
practices, to formally defined steps, to managed
result metrics, to active optimization of the
processes
 http://en.wikipedia.org/wiki/Capability_Maturity_Mo
del

4. What is maturity?
 Immaturity: Ad hoc, firefighting, improvisation, lack
of rigorous process management; lack of objective
measures of product quality
 Maturity: Organisation wide processes for
managing software development and processes;
processes are defined and accord with how work is
done; there is communication and training;
processes are monitored, analysed and updated in
response
 True maturity is when we no longer have heroics
and value judgement and work from an evidence
base

5. Moving from oral to fixed
history
Oral
history
Fixed
history

6. The Capability Maturity Model
 The Capability Maturity Model (CMM – now CMMI) is
used to assess the capabilities of institutions that
deliver (or not) software products
 Developed by the Software Engineering Institute
 Can we assess the maturity of ontology
development as a discipline rather than as
institutions?

7. A spectrum of maturity
Ad hoc
and
heroics
Repeatable
processes
Defined
processes
and
training
Quantitatively
managed
Optimised
processes
1
2
3
4
5

8. Bits of the CMM
 Maturity levels – From ad hoc to metric, analysis and
managed change
 Key process areas – A set of related activities that
when performed together achieve a goal
 Goals – the features of a key process area that must
exist for that area to have been implemented in an
effective and persistent manner
 Common features – Commitment and ability to
perform, activities performed, measurement, analysis
and verification
 Key practices – the infrastructure and practices that
contribute to the implementation of the area of
activity

9. Software maturity levels and their
key process areas
Initial
• None
Repeatable
•Requirements
Management
•Software
Project
Planning
•Software
Project
Tracking and
Oversight
•Software
Subcontract
Management
•Software
Quality
Assurance
•Software
Configuration
Management
Defined
•Organization
Process Focus
•Organization
Process
Definition
•Training
Program
•Integrated
Software
Management
•Software
Product
Engineering
•Intergroup
Coordination
•Peer Reviews
Managed
•Quantitative
Process
Management
•Software
Quality
Management
Optimizing
•Defect
Prevention
•Technology
Change
Management
•Process
Change
Management

10. A level 3 key process area
Software
Product
Engineering
Analysis
Design
Coding
Testing
Documentation

11. Metrics for key process areas
Requirements
Management
KPA
Requirement
status
Requirements
management
Requirements
stability
has metrics for

12. Can We Apply this to a Discipline?
 The CMM looks at maturity in an organisation
 It looks at the processes and their management (in its
broadest sense)
 A key is to have common practices in that organisation
 There has to be something to manage
 For a discipline we don’t need one common practice
to be in place
 …, but we do need common, replicable practices to
exist
 The question is, are we capable of being mature?

13. What maturity doesn’t mean
 Everyone in the discipline does the same things
the same way
 We don’t all need to use the same methods
 But we do need to make those methods
repeatable
 This can be done for as many styles of ontology
development as we see fit
 Being level one doesn’t mean a decent
ontology cannot be made

14. How to make a set of ontology
development CMM process areas
Initial
• None
Repeatable
•Requirements
Management
•Ontology
Project
Planning
•Ontology
Project
Tracking and
Oversight
•Ontology
Subcontract
Management
•Ontology
Quality
Assurance
•Ontology
Configuration
Management
Defined
•Organization
Process Focus
•Organization
Process
Definition
•Training
Program
•Integrated
Ontology
Management
•Ontology
Product
Engineering
•Intergroup
Coordination
•Peer Reviews
Managed
•Quantitative
Process
Management
•Ontology
Quality
Management
Optimizing
•Defect
Prevention
•Technology
Change
Management
•Process
Change
Well, it's not really that easy Management

15. A level 3 key process area for
ontology development
Ontology
Product
Engineering
Analysis
Design
Coding
Testing
Documentation

16. A collection of ontology
development practices
Knowledge
gathering and
requirements
• Specification
• Knowledge
Acquisition
• Elicitation
Modelling and
implementation
• Conceptualis.
• Formalization
• Enrichment
• Update
• Repair
• Reuse
• Integration
Testing
• Evaluation
• Quality
Assurance
• Assessment
• Verification
Development
process
management
• Configuration
Management
control
• Versioning
• Documentation
Do we know how to do these, let
alone manage them?

17. How mature is the ontology
development discipline?
 Still too much ad hoc and heroics in the discipline
 Can still produce good ontologies, but…
 We have parts of the infrastructure (technology
readiness level?)
 We have some of the defined processes
 Not much in the way of metrics
 We’ve come a long way, but…

18. What should happen next?
 We can transfer a lot from software engineering
 But we need to know more about what needs to be
done and how it is done – and then make the tools
to support it
 Then we can manage it
 Increased sociotechnical understanding of the
development processes for ontologies
 Tool support for those processes
 The ability to measure more and to depend on
value judgements less

19. Acknowledgements
 Nico Matentzoglu did the pictures
 Misconceptions of the capability Maturity Model,
Karl E. Wiegers at www.processimpact.com
 Capability Maturity Model for Software, Version 1.1.
Mark C. Paulk, Bill. Curtis, Mary Beth Chrissis, Charles
V. Weber
 Towards a glossary of activities in the ontology
engineering field MC Suárez-Figueroa, A Gómez-
Pérez