Your SlideShare is downloading. ×
0
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Experimenting with eXtreme Design (EKAW2010)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Experimenting with eXtreme Design (EKAW2010)

369

Published on

Presentation of the research paper "Experimenting with eXtreme Design" at EKAW 2010.

Presentation of the research paper "Experimenting with eXtreme Design" at EKAW 2010.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
369
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Experimenting with eXtreme Design Eva Blomqvist, Valentina Presutti, Enrico Daga, and Aldo Gangemi STLab, ISTC-CNR, Rome, Italy EKAW 2010 Lisbon (Portugal) – 2010-10-12
  • 2. Outline <ul><li>Background on Content ODPs </li></ul><ul><li>Problem – Research Questions </li></ul><ul><li>XD Tools and XD methodology </li></ul><ul><li>Experiment setup </li></ul><ul><li>Results of analysis </li></ul><ul><ul><li>Confirming previous results </li></ul></ul><ul><ul><li>XD Tools </li></ul></ul><ul><ul><li>XD Methodology </li></ul></ul><ul><li>Conclusions and future work </li></ul>
  • 3. Background - ODPs <ul><li>It is difficult to construct good quality ontologies! </li></ul><ul><ul><li>Lack of guidelines </li></ul></ul><ul><ul><li>Best-practices exist but how can we communicate them? </li></ul></ul><ul><li>Classes of problems in ontology design that can be solved by applying common solutions =&gt; Ontology Design Patterns (ODPs) </li></ul><ul><li>Content ODPs (CPs) </li></ul><ul><ul><li>small ontologies with explicit documentation of design rationales </li></ul></ul><ul><ul><li>modeling “good practices” </li></ul></ul><ul><li>An ontology can be seen as a… </li></ul><ul><ul><li>Composition of CPs + </li></ul></ul><ul><ul><li>Dependencies + </li></ul></ul><ul><ul><li>Expansion </li></ul></ul>
  • 4. Background – ODPs (cont.) <ul><li>A Content ODP (CP) is always associated with requirements, usually expressed using Competency Questions (CQs) </li></ul><ul><li>Example: InformationRealization What are the physical realizations of this information object? What information objects are realized by this physical object? </li></ul>
  • 5. Background – ODPs (cont.) <ul><li>Collected at http://www.ontologydesignpatterns.org </li></ul><ul><li>Currently: Logical-, Architectural-, Content-, Re-engineering-, Alignment-, and Lexico-syntactic ODPs </li></ul>
  • 6. Problem <ul><li>Overall question: What are the benefits of ODPs in ontology engineering? (Initial focus on Content ODPs) </li></ul><ul><li>Study conducted in 2008-09 (reported at K-CAP’09) </li></ul><ul><ul><li>Are Content ODPs perceived as useful? </li></ul></ul><ul><ul><ul><li>Yes! </li></ul></ul></ul><ul><ul><li>Are the ontologies constructed using Content ODPs ‘better’, in some modelling quality sense? </li></ul></ul><ul><ul><ul><li>Coverage of problem decreased (slower?) but major improvement in usability aspects, and fewer common mistakes </li></ul></ul></ul><ul><ul><li>Are the tasks given to the participants solved faster when using Content ODPs? </li></ul></ul><ul><ul><ul><li>Not really, rather slower (too little experience?) </li></ul></ul></ul><ul><ul><li>How do participants use the Content ODPs provided, and what support would be beneficial? </li></ul></ul><ul><ul><ul><li>How to find and select ODPs? How to reuse them? Tools? </li></ul></ul></ul>
  • 7. Problem (cont.) <ul><li>Since the previous study, we have developed: </li></ul><ul><ul><li>The eXtreme Design (XD) methodology </li></ul></ul><ul><ul><li>XD Tools </li></ul></ul><ul><li>Summary of research questions: </li></ul><ul><ul><li>Can we confirm the results from the previous study (Questions 1-4)? + How is modularity affected? </li></ul></ul><ul><ul><li>Does XD Tools support the process of reusing CPs? </li></ul></ul><ul><ul><li>Does the XD methodology support the process of reusing CPs, and does it affect any of the aspects from the previous study (e.g., time, quality)? </li></ul></ul>
  • 8. Related Work <ul><li>Ontology Engineering methodologies </li></ul><ul><ul><li>Classical methodologies are similar to software engineering methodologies </li></ul></ul><ul><ul><li>Few focus on collaboration, and very few are pattern-based – (Clark &amp; Porter, 1997) and (Maas &amp; Janzen, 2009) </li></ul></ul><ul><ul><li>Most are not evaluated, just motivated through… </li></ul></ul><ul><ul><ul><li>Example use case </li></ul></ul></ul><ul><ul><ul><li>Theoretical comparison </li></ul></ul></ul><ul><li>Pattern-based tools </li></ul><ul><ul><li>Mainly for Logical ODPs (e.g. support from OPPL, and templates and wizards in Protégé) </li></ul></ul><ul><ul><li>Tool evaluation mostly as benchmarking/comparison </li></ul></ul>
  • 9. XD Tools <ul><li>Plugin for Eclipse-based OE environments (TopBraid Composer, NeOn Toolkit, etc.) </li></ul><ul><li>Main parts </li></ul><ul><ul><li>Registry browser </li></ul></ul><ul><ul><li>XD Selector </li></ul></ul><ul><ul><li>Specialization wizard </li></ul></ul><ul><ul><li>Annotation dialogue </li></ul></ul><ul><ul><li>XD Analyzer </li></ul></ul>
  • 10. XD Tools Browse, search, and get Content ODPs Analyze your ontology against good practices and patterns Specialize, compose, annotate ODPs and ontologies
  • 11. XD Methodology <ul><li>An agile approach to ontology engineering </li></ul><ul><li>Two sets </li></ul><ul><ul><li>The problem space, i.e., the actual modeling issues (local problems) </li></ul></ul><ul><ul><li>The solution space, i.e., reusable modeling solutions </li></ul></ul><ul><li>Test-driven: unit tests on modules – integration tests on overall ontology </li></ul><ul><li>Task-focused: solve only and exactly the requirements at hand </li></ul><ul><li>Divide-and-conquer (design pairs) – collaboration and integration are crucial aspects </li></ul>
  • 12. XD Methodology (cont.) <ul><li>… </li></ul><ul><li>Step 5 - Select a coherent set of CQs, treating one modelling issue, for development iteration. </li></ul><ul><li>Step 6 - Match the CQs to CPs, candidate CPs for reuse are identified. </li></ul><ul><li>Step 7 - Select CPs to use: best fit to local problem without unnecessary overhead. </li></ul><ul><li>Step 8 - Reuse (import, specialize) and integrate (compose, extend) selected CPs. </li></ul><ul><li>Step 9 - Test (unit tests, e.g., through SPARQL queries) and fix. Then start the next iteration. </li></ul><ul><li>… </li></ul>
  • 13. Experiment Setup <ul><li>2 sessions, 35 participants in total (inexperienced, mainly master and PhD students, and junior researchers) </li></ul><ul><li>Controlled settings: same tools, same training and instructions, same tasks, same patterns </li></ul><ul><li>Procedure: </li></ul><ul><li>One difference: Session 2 – XD Tools more mature </li></ul>
  • 14. Experiment Setup (cont.) <ul><li>Task 1: </li></ul><ul><li>Context The national association for promotion of theater in Italy wants to set up a web-based system for keeping track of details about theater productions and the actors at different theaters. In order to support reasoning about the productions, the system should be based on an ontology. … </li></ul><ul><li>Story: theater productions During each year a number of theatre festivals are held in cities around Italy. In January 2007 a festival called “Roma Loves Shakespeare” took place in Rome. Two different productions of “The Merchant of Venice” participated, one from a theatre in Pisa and the other from a theatre institute in Venice, featuring an ensemble of university art students. Other plays were Othello and a Midsummer Night’s Dream … </li></ul><ul><li>Competency questions (CQs) and contextual statements of theater production </li></ul><ul><ul><li>When did a certain theatre festival take place? </li></ul></ul><ul><ul><li>Where did a certain festival take place? </li></ul></ul><ul><ul><li>What plays could be seen during a certain theatre festival? </li></ul></ul><ul><ul><li>In what city is a certain theatre located? </li></ul></ul><ul><ul><li>In what country is a certain city located? </li></ul></ul><ul><ul><li>What play is the basis of a certain production? </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><ul><li>Contextual statements: </li></ul></ul><ul><ul><li>A production has exactly one premier. </li></ul></ul>
  • 15. Evaluation of Results <ul><li>Analysis of questionnaire responses </li></ul><ul><li>Evaluation of the constructed ontologies (Mainly functional and usability evaluations – see K-CAP’09 paper) </li></ul><ul><ul><li>Coverage of problem: terminological coverage and task coverage </li></ul></ul><ul><ul><li>Usability: e.g., presence of labels, comments, inverse relations, disjointness, level of axiomatization </li></ul></ul><ul><ul><li>Modelling mistakes: incomplete solutions, solutions with shortcomings </li></ul></ul><ul><ul><li>Pattern usage: implicit and explicit </li></ul></ul>
  • 16. Results – Confirming previous conclusions? <ul><li>Are CPs perceived as useful by the participants? </li></ul><ul><ul><li>Confirmed – Increase for second session: Due to XD Tools? </li></ul></ul><ul><li>Are the ontologies constructed using CPs ‘better’, in some modelling quality sense? </li></ul><ul><ul><li>Coverage: Reduction of terminological coverage is no longer detected – Due to XD Tools? </li></ul></ul><ul><ul><li>Usability: Confirmed – Most prominent improvement! </li></ul></ul><ul><li>Are the tasks solved faster when using CPs? </li></ul><ul><ul><li>With tool support: no longer slower! </li></ul></ul><ul><li>What common modelling ‘mistakes’ can be identified, when not using patterns and when using CPs? </li></ul><ul><ul><li>Decrease in occurrence of most frequent mistakes confirmed (44% average decrease) - Same types of mistakes </li></ul></ul>
  • 17. Results - Modularity <ul><li>Do CPs increase the modularity of ontologies? </li></ul><ul><ul><li>Task 1: no ontologies are modularized </li></ul></ul><ul><ul><li>Task 2: the ontologies contain on average 7.5 modules </li></ul></ul><ul><ul><li>Conclusion: Since the participants choose to reuse the CPs as OWL-modules , rather than ideas for solutions, this inherently introduces modularity </li></ul></ul>
  • 18. Results – XD Tools <ul><li>How well does XD Tools support… …finding CPs to reuse? …specializing CPs? </li></ul><ul><li>Does XD Tools introduce overhead? </li></ul>
  • 19. Results – XD Tools <ul><li>How well does XD Tools support… …finding CPs to reuse? …specializing CPs? </li></ul>
  • 20. Results – XD Tools <ul><li>How well does XD Tools support… …finding CPs to reuse? …specializing CPs? </li></ul><ul><li>Does XD Tools introduce overhead? </li></ul>
  • 21. Results – XD methodology <ul><li>Is the XD methodology perceived as useful by the participants? </li></ul><ul><li>Is the XD methodology a ‘natural’ way to work with CPs? </li></ul><ul><li>Are the ontologies constructed using the XD methodology ‘better’, in some modelling quality sense, than the ontologies constructed ‘only’ using CPs? </li></ul><ul><li>Are the tasks solved faster when using the XD methodology, compared to ‘only’ using CPs? </li></ul><ul><li>What common modelling ‘mistakes’ can be identified and are they different from the ones noted when ‘only’ using CPs? </li></ul>
  • 22. Results – XD methodology <ul><li>Is the XD methodology perceived as useful by the participants? </li></ul><ul><ul><li>94% claimed to follow the XD methodology closely </li></ul></ul>
  • 23. Results – XD methodology <ul><li>Is the XD methodology perceived as useful by the participants? </li></ul><ul><li>Is the XD methodology a ‘natural’ way to work with CPs? </li></ul>
  • 24. Results – XD methodology <ul><li>Is the XD methodology perceived as useful by the participants? </li></ul><ul><li>Is the XD methodology a ‘natural’ way to work with CPs? </li></ul><ul><li>Are the ontologies constructed using the XD methodology ‘better’, in some modelling quality sense, than the ontologies constructed ‘only’ using CPs? </li></ul><ul><ul><li>Small increase from Task 2 to Task 3 (could be due to maturation) </li></ul></ul><ul><li>Are the tasks solved faster when using the XD methodology, compared to ‘only’ using CPs? </li></ul><ul><ul><li>Small increase – however, mistakes seem to be found more easily, hence some parts seem to be done faster! </li></ul></ul>
  • 25. Results – XD methodology <ul><li>… </li></ul><ul><li>What common modelling ‘mistakes’ can be identified and are they different from the ones noted when ‘only’ using CPs? </li></ul><ul><ul><li>Continued decrease in occurrence of most frequent mistakes from Task 2 to 3 (15% average decrease) </li></ul></ul><ul><ul><li>Two types of errors decrease significantly more than the others: </li></ul></ul><ul><ul><ul><li>N-ary relations – decrease by 64% </li></ul></ul></ul><ul><ul><ul><li>Missing datatype properties – decrease by 46% </li></ul></ul></ul><ul><ul><li>Conclusion: XD is test-driven – structured unit testing leads to finding certain types of mistakes </li></ul></ul>
  • 26. Future Work <ul><li>Currently ongoing </li></ul><ul><ul><li>Development of the integration and collaboration part of the XD methodology </li></ul></ul><ul><ul><li>Explorative experiments – 1 session performed </li></ul></ul><ul><ul><ul><li>From experimenting on the iteration of the design pairs to experimenting on the complete development team </li></ul></ul></ul><ul><ul><ul><li>Things to be considered </li></ul></ul></ul><ul><ul><ul><ul><li>Collaboration between pairs – integration pair? </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Integration testing and refactoring </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Information feedback loop </li></ul></ul></ul></ul><ul><ul><ul><ul><li>… </li></ul></ul></ul></ul><ul><li>Planned experiment </li></ul><ul><ul><li>Comparative study of XD and other methodologies </li></ul></ul>
  • 27. Future Work (cont.) <ul><li>Additional experimental work </li></ul><ul><ul><li>Other domains and tasks </li></ul></ul><ul><ul><li>New target groups – experts </li></ul></ul><ul><ul><li>Other types of patterns </li></ul></ul><ul><ul><ul><li>Presentation and representation issues </li></ul></ul></ul><ul><ul><li>Analysing future ontologies (when patterns are more wide-spread) </li></ul></ul><ul><li>XD Tools </li></ul><ul><ul><li>Shortcuts for expert users to reduce overhead </li></ul></ul><ul><ul><li>Improve search functionalities </li></ul></ul><ul><li>XD methodology </li></ul><ul><ul><li>Elaborate collaboration and integration aspects </li></ul></ul>
  • 28. Conclusions <ul><li>Confirmed conclusions </li></ul><ul><ul><li>Content ODPs are perceived as useful </li></ul></ul><ul><ul><li>They improve mainly the usability aspects of the ontologies, but also reduces the occurrence of a number of common ‘mistakes’ </li></ul></ul><ul><ul><li>Development is neither faster nor slower </li></ul></ul><ul><li>XD Tools </li></ul><ul><ul><li>Supports the use of CPs – no longer slower </li></ul></ul><ul><ul><li>Perceived as useful, although adds some overhead </li></ul></ul><ul><li>XD methodology </li></ul><ul><ul><li>Perceived as useful, and it is a natural way of working with CPs (descriptive and pragmatic) </li></ul></ul><ul><ul><li>Main advantage seems to be testing – drastically decreases some common ‘mistakes’ </li></ul></ul>
  • 29. Thank you! Questions?

×