Research Design for the                   Evaluation of the Strongly Sustainable                         Business Model On...
Research Design for the Evaluation of the StronglySustainable Business Model OntologyTable of ContentsList of Figures     ...
5.    Evaluation Research Design – Overview                                            68      5.1    Introduction        ...
List of FiguresFigure 1: Information Systems Research Framework                                         16Figure 2: Resear...
List of TablesTable 1: Guidelines for Undertaking High Quality Design Science Research                      14Table 2: Sco...
Table 33: Metrics Whose Values Are Being Gathered by E1b B-Labs Impact Assessment           Survey v3 Comparative Analysis...
AbbreviationsAI         Artificial IntelligenceBMO        Osterwalder Business Model Ontology (Osterwalder, 2004)MIS      ...
Research Design for the Evaluation of the StronglySustainable Business Model Ontology1. Introduction1.1 Research PurposeTh...
The purpose of this research3 is:    To explore, using design4 and systems methodologies, whether a pragmatic descriptive ...
•    The objectives evaluation stage    •    The process to be used to undertake the evaluation of the designed artefact  ...
1.4 Document StructureI will begin by presenting an updated overview of my overall research design, based onadditional ins...
2. Research Design – Overview2.1 IntroductionThis section presents an updated overview of my overall research design, base...
2.3 The Design of the SSBMO Research Project        As a result of newness of this approach to research Al-Debei noted in ...
Application of Guidelines to SSBMO Research  Guideline                     Description                                    ...
ontology domain. However, I have also chosen to include significant systems thinking elementsin my research design.The inc...
2.5 The Framework for Conducting Design Science Research         In the same article in which Hevner et. al. provide guide...
Al-Debei, summarizing the advice from several scholars, notes that such foundational theories           are normally origi...
Using Figure 2, the conceptual sequence of my research project is as follows:   1. The problem is understood (P1) and the ...
2.8 Process of Inquiry for Conducting the SSBMO Research          Other recent ontology development research projects have...
The process enquiry consists of four related “swim lanes” of activities. These swim lanescorrespond to the conceptual sequ...
13.Based on the learning from evaluation activities E2b and E3 the fourth and final version                          of th...
The details behind these choices of activities were introduced in Version 4.1 of my ResearchProposal (August 8, 2011), and...
2.9.2 Evaluation as Part of Build ActivitiesHevner et. al. recommends that during the build activities “the artefact itsel...
Firstly, considering Baskerville et. al.’s innovative and recent work which proposed amethodology which integrates Soft Sy...
3. Evaluation in Ontology, Design Science and Systems Research3.1 IntroductionThe proximity of Design Science and Ontology...
3. Use this review to attempt to improve the quality and rigor of my own research design        and hence the utility of t...
In providing additional substance to this definition Cleven et al observe that:   •   Evaluation may be “quantitative or q...
3.4 Purpose / Objective of Evaluation3.4.1 OverviewDespite the concerns described above, overall the literature on the pur...
Going further than Osterwalder, Hevner et. al also describe how the output of evaluation is used:“the evaluation of the ar...
evaluation” (Hevner & Chatterjee, 2010, p.109). Hevner & Chatterjee go on to describe threepurposes of evaluation ((pp.110...
•   Negative feedback is analysed to identify changes to the design which would improve its           utility30.Clearly as...
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Research design for Evaluation of Strongly Sustainability Business Model Ontology
Upcoming SlideShare
Loading in …5
×

Research design for Evaluation of Strongly Sustainability Business Model Ontology

1,899 views

Published on

This document summarizes my overall research design for the strongly sustainable business model ontology (chapter 1) and then provides the detailed research design for the evaluation phase of my design science research in Environmental Studies (chapter 2-10)

For more details about the background on Strongly Sustainable Business Models please see http://slab.ocad.ca/SSBMs_Defining_the_Field and http://www.EdwardJames.biz/Research.

Published in: Business, Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,899
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
54
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Research design for Evaluation of Strongly Sustainability Business Model Ontology

  1. 1. Research Design for the Evaluation of the Strongly Sustainable Business Model Ontology Prepared Prior to Commencing Field Work for a Master Thesis at PUBLIC VERSION Toronto  Antony Upward  Ontario  Student # 211135423  Canada  email: aupward@yorku.ca / +1 416 576 2542 MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 1 Version 2.1
  2. 2. Research Design for the Evaluation of the StronglySustainable Business Model OntologyTable of ContentsList of Figures 4List of Tables 5Abbreviations 71. Introduction 8 1.1 Research Purpose 8 1.2 Document Purpose 9 1.3 Commentary and Possible Contribution 10 1.4 Document Structure 112. Research Design – Overview 12 2.1 Introduction 12 2.2 Design Science – A Definition 12 2.3 The Design of the SSBMO Research Project 13 2.4 The Scope of the SSBMO Research Project 15 2.5 The Framework for Conducting Design Science Research 16 2.6 The Framework for Conducting the SSBMO Research 17 2.7 The Research Cycle for Conducting Design Science Research 18 2.8 Process of Inquiry for Conducting the SSBMO Research 19 2.9 Summarizing the SSBMO Build Research Activities and Outputs (D1-4) 21 2.10 Summarizing the SSBMO Evaluate Research Activities and Outputs (E1-3) 243. Evaluation in Ontology, Design Science and Systems Research 25 3.1 Introduction 25 3.2 Reviewing the Literature 26 3.3 Evaluation: A Definition 26 3.4 Purpose / Objective of Evaluation 28 3.5 Process of Evaluation 32 3.6 Research Outputs Requiring Evaluation 39 3.7 Evaluation Metrics 39 3.8 Techniques for Capturing Values of the Metrics 514. Choosing the Approach for Evaluation of the SSBMO 65 4.1 The Current State of the Art 65 4.2 Overall Evaluation Research Design Goal 65 4.3 Detailed Evaluation Research Design Goals 66 4.4 Constraints on the Choices of Evaluation Research Design 66 4.5 Process for Creating Evaluation Research Design 67MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 2 Version 2.1
  3. 3. 5. Evaluation Research Design – Overview 68 5.1 Introduction 68 5.2 Evaluation Design Framework 68 5.3 Unit of Analysis 70 5.4 Evaluation Process Context 70 5.5 Research Outputs to Be Evaluated 71 5.6 Sources and Validity of Comparator Knowledge 72 5.7 Chosen Metrics 79 5.8 Chosen Techniques for Gathering Valid Values of the Metrics 82 5.9 Evaluation Stage Activity Groups – the Overall Evaluation Process 86 5.10 Chosen General Techniques Increasingly Likelihood of Collecting Valid Metric Values 94 5.11 Summary of Evaluation Research Design 966. Evaluation Activity Group 1 – Comparative Analysis (E1) 97 6.1 Introduction to Comparative Analysis 97 6.2 Comparative Analysis Using the CATWOE (K1) Knowledge Source (E1a) 97 6.3 Comparative Analysis Using the B-Lab Impact Assessment v3 (K2) Knowledge Source (E1b) 100 6.4 Comparative Analysis of The Timberland Company (K3) Knowledge Source (E1c) 104 6.5 Updating Ontology Design Based on Evaluation Results 1097. Evaluation Activity Group 2 – Third Party Review (E2) 110 7.1 Gathering Informal Feedback (E2a) 110 7.2 Introduction to Formal Expert Interviews (E2b) 112 7.3 Execution Protocol for Formal Expert Interviews (E2b-1 thru 7) 1168. Evaluation Activity Group 3 – Case Studies (E3) 126 8.1 Introduction to Case Studies 126 8.2 Execution Protocol for Case Studies (E3) 130 8.3 Additional Objectives of Case Study Work 1409. Updating Ontology Design Based on Evaluation Results (D4) 14110. Concluding on Overall Research Results 14111. Bibliography 14212. Appendix E2: Third Party Reviewer Names (CONFIDENTIAL) 145 12.1 Informal Third Party Review (E2a) 145 12.2 Formal Third Party Review (E2b) 14613. Appendix E3: Case Study Names (CONFIDENTIAL) 14714. Appendix: Human Participants Research Protocol and Risk Assessment 148 14.1 Ethics Approval 148 14.2 Informed Consent Form 149MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 3 Version 2.1
  4. 4. List of FiguresFigure 1: Information Systems Research Framework 16Figure 2: Research Framework for the SSBMO Research Project 17Figure 3: Design Science Research Cycle 18Figure 4: Overall Process of Inquiry for the SSBMO 19Figure 5: Generic Steps for Evaluating a Designed Artefact 34Figure 6: Soft Design Methodology (SDM) 37Figure 7: Number of the 83 Metrics Mentioned By How Many of the 17 Sources 42Figure 8: Number of the 17 Sources Mentioning How Many of the 83 Metrics 43Figure 9: Prescription vs. Description in the Build and Use of the SSBMO 74Figure 10: Relationship of Knowledge Sources for the Build and Evaluation of the SSBMO (K0 thru K6) 79Figure 11: Summary of Knowledge Sources, Techniques and Metrics used in Comparative Analysis Evaluation Activity Group (E1) 89Figure 12: Summary of Knowledge Sources, Techniques and Metrics used in 3rd Party Review Evaluation Activity Group (E2) 91Figure 13: Summary of Knowledge Sources, Techniques and Metrics used in Case Study Evaluation Activity Group (E3) 93MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 4 Version 2.1
  5. 5. List of TablesTable 1: Guidelines for Undertaking High Quality Design Science Research 14Table 2: Scope of the SSBMO Research Project 15Table 3: Summary of the SSBMO Build Research Activities and Outputs (D1-4) 21Table 4: Recommended Evaluation Steps and the Research Stage When They Should be Undertaken 33Table 5: Framework for Designing Design Science Research 36Table 6: Types of Evaluation in Soft Design Science 38Table 7: Types and Counts of Metrics Literature Consulted 41Table 8: Metrics Designed to Give Feedback on the Context of an Evaluation 44Table 9: Metrics Designed to Give Generic Feedback on the Utility of the Artefact Being Evaluated 45Table 10: Metrics Designed to Give Feedback on the Completeness Aspect of the Utility of the Artefact Being Evaluated 46Table 11: Metrics Designed to Give Feedback on the Quality Aspect of the Utility of the Artefact Being Evaluated 47Table 12: Metrics Designed to Give Feedback on the Beauty Aspect of the Utility of the Artefact Being Evaluated 48Table 13: Definitions of Evaluation Metrics 50Table 14: Characteristics of Evaluation Techniques and Metric Validity 52Table 15: Summary of Evaluation Techniques 56Table 16: Expectation / Desirability Matrix 58Table 17: Summary of Artificial Intelligence Ontology Evaluation Techniques / Metrics 59Table 18: Evaluation Technique Groupings 60Table 19: Techniques Employed to Evaluate BMO 61Table 20: Techniques and Metrics Employed to Evaluation Innovation Ontology 63Table 21: Techniques Employed to Gather Valid Values of Certain Metrics 64Table 22: SSBMO Evaluation Design Framework 69Table 23: SSBMO Expectation / Desirability Matrix 71Table 24: Source of Comparator Knowledge for the Evaluation of the SSBMO 78Table 25: Evaluation Metrics for each SSBMO Research Objective and Research Output Component 80Table 26: Definition of Selected Metrics for Evaluation of SSBMO Utility 82Table 27: Evaluation Techniques, Required Comparator Knowledge Sources and Design Artefacts for each SSBMO Research Output Component 85Table 28: SSBMO Evaluation Activity Groups 87Table 29: Identification of Risks and Application of Mitigation Techniques in All Evaluation Activities 95Table 30: Summary of Evaluation Research Design 96Table 31: Metrics Whose Values Are Being Gathered by E1a CATWOE Comparative Analysis Evaluation Activity 98Table 32: Evaluation Activity Sub-Group E1a – Questions and Metrics 100MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 5 Version 2.1
  6. 6. Table 33: Metrics Whose Values Are Being Gathered by E1b B-Labs Impact Assessment Survey v3 Comparative Analysis Evaluation Activity 102Table 34: Scoring Scheme for Comparison of B-Labs Impact Assessment Survey (K2), to Osterwalder’s BMO (K0-PF) and the SSBMO (K0-SS) 103Table 35: Scheme for Interpreting Values of Metrics from Scores in Evaluation Activity Sub- Group E1b 104Table 36: Metrics Whose Values Are Being Gathered by E1c The Timberland Company Comparative Analysis Evaluation Activity 106Table 37: Metric Values to Result from Comparing CATWOE (K1) Knowledge Source and The Timberland Company Business Model Described Using SSBMO (K3-BM) 107Table 38: Metric Values to Result from Comparing Public Knowledge of The Timberland Company (K3) and the SSBMO constructs and model (K0-SS, K0-PF) 108 rdTable 39: Metrics Whose Values Are Being Gathered by E2a Informal 3 Party Expert Review Evaluation Activity 111Table 40: Comparison Knowledge Source and Design Artefact to be used in Evaluation Activity E2b Formal Expert Interviews 114Table 41: Metrics Whose Values Are Being Gathered by E2b Formal Expert Interviews Evaluation Activity (Overall) 116Table 42: Items to Validate Demographics of Interviewees (E2b) 119Table 43: Metrics Whose Values Are Being Gathered by E2b Formal Expert Interviews Evaluation Activity (Fit of K4 to SSBMO) 120Table 44: Items to Assess Fit of Expert Knowledge of Business Models (K4) to SSBMO (K0-PF, K0-SS) 121Table 45: Metrics Whose Values Are Being Gathered by E2b Formal Expert Interviews Evaluation Activity (Fit of K5 to SSBMO Example Instantiation K4-BM) 122Table 46: Assess Expert Knowledge of Operating Firms (K5) to SSBMO Example Instantiation (K3-BM) 125Table 47: Items to Close Expert Feedback Interviews 125Table 48: Metrics Whose Values Are Being Gathered by E3 Case Studies Evaluation Activity 130Table 49: Metric Values Resulting from Comparing CATWOE (K1) Knowledge Source and Case Business Model Described Using SSBMO (K6-BM1 thru 3) 132Table 50: Items to Gather Demographics of Case Firm Employees (E3) 135Table 51: Items to Gather Case Information (K6-E1 thru 3) 136Table 52: Metric Values Resulting from Comparing Case Employee Knowledge Source (K6- E1 thru 3) and Case Business Model Described Using SSBMO (K6-BM1 thru 3) 137Table 53: Items to Assess Fit of Case Firm Employee Knowledge (K6-E1 thru 3) to Specific Example of SSBMO Instantiation (K6-BM1 thru 3) 139Table 54: Items to Close Case Firm Employee Feedback Interviews 140Table 55: Informal Third Party Reviewer List and Demographics (Confidential) 145Table 56: Formal Third Party Reviewer List and Demographics (Confidential) 146Table 57: Case Firm List and Employee Demographics (Confidential) 147MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 6 Version 2.1
  7. 7. AbbreviationsAI Artificial IntelligenceBMO Osterwalder Business Model Ontology (Osterwalder, 2004)MIS Management Information SystemsSDM Soft Design MethdologySSBMO Strongly Sustainable Business Model Ontology. The output from this design science research projectSSM Soft Systems Methodology or Soft Systems MethodMES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 7 Version 2.1
  8. 8. Research Design for the Evaluation of the StronglySustainable Business Model Ontology1. Introduction1.1 Research PurposeThe working title1 of this thesis is An Outline of a Normative Ontology for Strongly Sustainable Business Models: An Exploration of a Proposition Using a Design Science Approach including a Comparative Case Study of Firms Seeking to Improve Their SustainabilityThe business problem this design science research project is looking to solve is: Increasing the quality (reliability, consistency, effectiveness) of strongly sustainable business models and the efficiency of business model designers who create them.2This project is seeking to solve a problem in the environment at large by creating somethinguseful. Hence this is an applied research project.1 A possible improved title is: Towards an Ontology for Strongly Sustainable Business Models: A Design ScienceExploration2 Osterwalder noted in a recent presentation that “in entrepreneurship [unlike in car design] we still rely on real-lifecrash tests [through the creation of news firms with new business models] which leads to costly failures” Thismeans the sustainability of those businesses is low (even in conventional profit-first terms), hence risk is high forbusiness model designers and the stakeholders of the firms instantiating those business models. Overall the designprocess is inefficient and ineffective in the use of existing knowledge of how to design better business models andthe communication of the design. As a result the failure rate of new businesses is high (Osterwalder, 2011b,slide 19[minute 3.00-3.30]).Aligned with this, Bullinger, in her review of the design science literature, states “the value of an information systemdesign theory [and implicitly instantiations of artefacts using that theory] lies in the reduction of uncertainty bylimiting the system features and development activities to a manageable set. Thus, reliability of development aswell as likelihood of success could be increased […]” (Bullinger, 2008, p.222).This also aligned well with my own experience from consulting projects over the past 20 years.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 8 Version 2.1
  9. 9. The purpose of this research3 is: To explore, using design4 and systems methodologies, whether a pragmatic descriptive tool can be built to improve the application of the science of design5 to strongly sustainable business models.Hence, my overall research question is: Is it possible to design a useful normative ontology of a business model that can be used to describe a firm’s strongly sustainable business model design?As examples, this includes answering questions such as: • How difficult is it for a business model designer to describe a strongly sustainable business model using Osterwalder’s BMO? • What is a list of constructs and relationships required to describe strongly sustainable business models that are missing, incomplete and surplus in Osterwalder’s BMO? • Is it easier, for business model designers who, based on their world-views, have different conceptions of success (from maximizing short term monetary profit to balancing achieving defined environmental, social and economic objectives), to use the SSBMO to describe a their chosen business’s business model?See sections 6 thru 8 for more questions this research will attempt to answer.1.2 Document PurposeThis document builds on the Research proposals submitted earlier6. Its primary function is todescribe in detail the research design I propose to execute during the evaluation phase of thisdesign science research project.My extensive review of the relevant literature strongly suggests that the ability of a researcher toundertake the evaluation stage of a design science research project rigorously and achieve a highquality of results is strongly dependent on the level of preparation of the researcher. i.e. does theresearcher consciously understand and can justify, based on legitimate precedent: • Their world-view / biases, • The objectives of the research3 Version 4.1. of my research proposal suggested the purpose of this research was two fold: 1. How a normative business model ontology can describe instances of firms’ strongly sustainable business models, and 2. What are the perceptions and reactions of managers, in firms’ attempting various levels of attempt to improve their sustainability, to the validity and utility of the ontologyIs this new statement of purpose an improvement?4 i.e. Design as a scientific research method5 i.e. The (art, craft and) science of how to do (good) design6 Version 3.2 of my Research Proposal was submitted following my MES II-III exam (June 8, 2011) and Version 4.1was submitted following extensive discussions with David Johnston (August 8, 2011) and was subsequentlyreviewed by Martin Bunch and Rod MacRea (September 2011)MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 9 Version 2.1
  10. 10. • The objectives evaluation stage • The process to be used to undertake the evaluation of the designed artefact • The rationale for this process • How the results of the evaluation will be used to meet the objectives of the research1.3 Commentary and Possible ContributionBefore describing the structure of this document I find it necessary to comment on both the onprocess of creating this document and the nature of this document.Uncovering and analysing the literature has turned out to be a time consuming process since therelevant literature comes from multiple fields. Further it has been come clear the literature onhow to best undertake design science evaluations is far from comprehensive and far fromprescriptive (and perhaps can never be).The work on this documented started in September 2011 and consumed the majority of my timeuntil mid/late December 2011.As this work proceeded I made three observations: 1. I had not encountered in the works reviewed as complete an analysis of the literature related to ontology engineering, design science and systems thinking evaluation theory and practice. 2. My overall research design appears to be novel in several respects. My research design is based on my understanding of the nature of the research topic (sustainability of human organizations) and hence the attributes of the research methodology required to generate legitimate knowledge. This has led me to integrate ontology engineering, design science and systems thinking into a single research design7. 3. It is intellectually challenging (and hence enjoyable) to construct a rigorous and high quality evaluation process.This leads me to ask: Is there a contribution to the design science field based which can bederived from my: 1. Analysis and integration of the ontology engineering, design science and soft systems methodology (SSM) literature on evaluation (section 3 of this document – ~40 pages, plus several analysis working documents and spreadsheets)? 2. The process by which I constructed the design of the evaluation stage of this project? (section 4 and portions of section 5 - ~10 pages) 3. The design of the research for undertaking the evaluation stage of this project? (sections 5 thru 8 - ~60 pages)Feedback on the potential for a publishable contribution is requested.7 See the most recent version of the presentation “Design science, systems thinking and the creation of ontologies” ,for details. These details will be included in the final thesis. Presentation included with this document.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 10 Version 2.1
  11. 11. 1.4 Document StructureI will begin by presenting an updated overview of my overall research design, based onadditional insights gained since August 2011 (see Section 2 – Research Design – Overview).Section 2 will be used, along with my recent presentation “Design science, systems thinking andthe creation of ontologies” (included with this document) to prepare the final research designsection of my thesis.Next I will review the literature related to the process, metrics and techniques for the evaluationdesign science research outputs (such as ontologies) (Section 3 – Evaluation in Ontology DesignScience Research). This then serves as a basis for the rest of the document, as follows: • Section 4 – Choosing the Approach for Evaluation of the SSBMO describes how I chose (how I designed) the evaluation approach based on the literature reviewed in section 3. • Section 5 thru 8 describes the details of the evaluation research design as follows: – Section 5 provides an overview, including identifying the unit of analysis, describing the overall process and discussing issues of research quality. – Section 6 thru 8 describe the three evaluation activities selected: comparative analysis, third-party review and case studies. • Section 9 and 10 conclude by providing a link to the research tasks which remain once the evaluation is completed.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 11 Version 2.1
  12. 12. 2. Research Design – Overview2.1 IntroductionThis section presents an updated overview of my overall research design, based on additionalinsights gained since August 2011. This material will be used, along with the earlier documents,to prepare the final research design section of my thesis.2.2 Design Science – A DefinitionUsing design as a formal framework for conducting research is relatively new, and its use withinthe social sciences and management sciences, such as information systems, is probably less than25 years old8.Hevner et. al. published what is now considered to be a seminal article9 about design scienceresearch in the information systems field in MIS Quarterly in 2004. Based on their summary ofthe field these authors present the following definitions: Design science […] creates and evaluates […] artefacts intended to solve indentified organizational problems (Hevner et al., 2004, p.77) Design is both a process (set of activities) and a product (artefact) – a verb and a noun. It describes the world as acted upon (processes) and the world as sensed (artefacts). This Platonic view of design supports a problem solving paradigm that continuously shifts perspective between design processes and designed artefacts for the same complex problem. The design process is a sequence of expert activities that produces an innovative product (i.e., the designed artefact). The evaluation of the artefact then provides feedback information and a better understanding of the problem in order to improve both the quality of the product and the design process. This build-and-evaluate loop is typically iterated a number of times before the final design artefact is generated. During this creative process, the design-science researcher must be cognizant of evolving both the design process and the design artefact as part of the research. (Hevner et al., 2004, p.78)Hevner et. al. were writing the above from the perspective of the MIS management science sub-discipline. Writing from the Innovation management science sub-discipline, Bullinger states, inher 2008 design science PhD10 that developed an ontology for management of the innovationprocess in small and medium businesses, that: “Design science researches strive to solve problems by an action-oriented approach, in order to find an viable artefact”, i.e. a solution to a problem (Bullinger, 2008, p.216)8 Final thesis will include appropriate citations for this statement.9 Google Scholar reports 2612 citations as of November 24, 201110 Bullinger’s PhD thesis has been published as a monograph without change. See bibliographic entry.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 12 Version 2.1
  13. 13. 2.3 The Design of the SSBMO Research Project As a result of newness of this approach to research Al-Debei noted in his 2010 design science PhD11, which developed an ontology for designing innovative mobile data services, that “the scheme to construct design artefacts in information systems design-science research is still very broad” (Al-debei, 2010, p.35). From a disciplinary perspective, following Bullinger, Al-Debei and Osterwalder (Osterwalder, 2004), I consider ontology, such as the SSBMO, to be primarily an Information Systems artefact. Citing others, Hevner et. al. describe the creation of an artefact to solve a problem in a specific context as an experiment “posing a question to nature”… “existing knowledge is used where appropriate; however, often the requisite knowledge is nonexistent”, hence “reliance of creativity and trial-and-error search are characteristic of such research efforts”, i.e. abduction (Hevner et al., 2004, p.81). This makes it challenging to design this type of research in detail before undertaking the research activities, but no less important than as for natural science. Much can be learned from the comparison of planned activities vs. activities deemed required by the researcher in the moment. To provide assistance to researchers Hevner et. al. have proposed guidelines for designing information systems design science research projects (Hevner et al., 2004, pp.82-90). Researchers such as Al-Debei and Bullinger, used these guidelines to ensure the appropriateness of their overall approach of their work (Al-debei, 2010, p.42; SeeBullinger, 2008, p.225 & p.232)12. The following table presents Hevner et. al.’s guidelines (first two columns) and describes how my research design is applying each (third column). Application of Guidelines to SSBMO Research Guideline Description Design1. Design as Design-science research must produce an Resulting artefacts are the ontology for stronglyan Artefact innovative purposeful (viable) artefact in sustainable business model design. The ontology the form of a construct, a model, a consists of constructs related in a model. method, and/or an instantiation. Instantiations of the ontology are created to aid in the evaluation of the validity and utility of the constructs and model.2. Problem The objective of design-science research The business problem addressed by the solution isRelevance is to develop an innovative purposeful increasing the quality (reliability, consistency, artefact for a specified problem domain effectiveness) of strongly sustainable business models and the efficiency of business model designers who create them. This has the additional benefit of reducing the risks to business model designers and users. 11 Al-Debei lists 13 works published or pending based on his PhD, including scholarly journal articles, conference papers and book chapters. 12 Osterwalder was working on his PhD before these guidelines had been published. MES Thesis: Strongly Evaluation Research Design December 19, 2011 Sustainable Business Model Ontology 13 Version 2.1
  14. 14. Application of Guidelines to SSBMO Research Guideline Description Design3. Design Because the artefact is purposeful, it must Evaluation takes place using a set of metricsEvaluation yield utility (including quality, and gathered via a number of techniques. See sections 3 efficacy) for the specified problem, i.e. a thru 8 of this document. design artefact must be rigorously demonstrated via well-executed evaluation methods.4. Research Since the artefact must be innovative, Contributions are expected fromContributions novelty is crucial (solving a heretofore 1. The capture of novel key concepts and their unsolved problem, or solving a known relationships that organization’s should consider problem in a more effective or efficient when attempting to be strongly sustainable manner), effective design-science 2. A novel tool which practitioners can use to more research must provide clear and verifiable efficiently and effectively design organizations contributions in the areas of the design strongly sustainable business models. artefact, design foundations, and/or design methodologies.5. Research The artefact itself must be rigorously Guidelines from the field of Design and InformationRigor defined, formally represented, coherent, Sciences are followed during the build and and internally consistent. Design-science evaluation of the ontology, e.g. the Ontology is research relies upon the application of formally represented using the Entity Relationship rigorous methods in terms of construction Modelling formalism. and evaluation of the artefact.6. Design as a The process by which it is created, and Specific guidelines for ontology design and theSearch often the artefact itself, incorporates or general science of design are applied for the firstProcess enables a search process whereby a time to the domain of strongly sustainable business problem space is constructed and a models. mechanism posed or enacted to find an effective solution. The search for an effective artefact requires utilizing available means to reach desired ends and satisfy laws in the environment.7. Design-science research must be Results of research are presented to industrialCommunicati presented effectively both to technology- partners as well as to the research community.on of oriented as well as management-oriented This is helped by one of the overarching purposesResearch audiences. of an ontology – establishing a shared language to support understanding and problem solving Table 1: Guidelines for Undertaking High Quality Design Science Research Derived from (Hevner et al., 2004, pp.82-90) 2.3.1 Inclusion of Systems Thinking in the Research Design The basis for the design of this research project is the research framework (process, methods, techniques, tools) adapted from design science to the strongly sustainable business model MES Thesis: Strongly Evaluation Research Design December 19, 2011 Sustainable Business Model Ontology 14 Version 2.1
  15. 15. ontology domain. However, I have also chosen to include significant systems thinking elementsin my research design.The inclusion of systems thinking elements is an extension to the “pure” design science ontologybuild/evaluate approach adopted by Osterwalder, Bullinger and Al-Debei. This is requiredbecause of: 1. The inherently holonic nature of the domain a strongly sustainable business model is attempting to describe and 2. Because I believe that you can’t effectively research systems using linear non-systemic methods. 132.4 The Scope of the SSBMO Research ProjectThe following table has been adapted from the important14 article by March and Smith (March &Smith, 1995, p.255) and updated to include an additional design output commonly accepted bydesign science researchers (e.g. Vaishnavi & Kuechler, 2009, p.6). Design Science Research Activities D. E. T. J. Build Evaluate Theorize Justify Output (Develop / (Validate) Elements Design) 1. ConstructsDesign Science Research 2. Models Output 3. Instantiations 4. Method 15 5. Better Theories Table 2: Scope of the SSBMO Research ProjectThe areas of the table shaded in green is the scope of my thesis (boxes D1-4 and boxes E1-3),based on what might be practically accomplished within the scope of a masters thesis, given I amanchoring my ontology upon Osterwalder’s existing ontology.13 See Version 4.1 of my research proposal and the recent presentation “Design science, systems thinking and thecreation of ontologies”. These two working documents will be used to justify and describe my integration ofsystems thinking elements into my research design. This will be included in my thesis.14 Google Scholar report 1122 citations as of November 24, 201115 As noted in sub-section 1.3, working on this document, reviewing the design science, ontology engineering andsystems literature has highlighted that the approach documented here for ontology evaluation may have some novelaspects, and may also hence be a contribution.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 15 Version 2.1
  16. 16. 2.5 The Framework for Conducting Design Science Research In the same article in which Hevner et. al. provide guidelines for designing information systems design science research, the authors also provide a process perspective on the information systems research. They refer to this process perspective as the “Information Systems Research Framework”, and use to illustrate how information systems research may include both the descriptive and design science approaches.Environment Relevance Research Rigor Knowledge BasePeople Foundations (What)• Roles 2a Develop / Build • Theories• Capabilities • Theories • Frameworks• Characteristics • Artefacts • Instruments • ConstructsOrganizations • Models• Strategies assess • Methods Iterative refine• Structure & Culture 1b. Applicable • Instantiations• Processes 1a. Problem Design Knowledge • Etc. ProcessTechnology Methodologies (How)• Infrastructure 2b. Justify / Evaluate • Data Analysis Techniques• Applications • Analytical • Formalisms• Communications • Case Study • Measures Architecture • Experimental • Validation Criteria• Development Capabilities • Field Study • Etc. • Simulation 3a. Application 3b. Additions to to solve problem knowledge base Figure 1: Information Systems Research Framework (Hevner et al., 2004, Figure 2 p.80) The sequence in which the research process unfolds is 1a. defining the problem, 1b. determine the applicable knowledge, and then iteratively (possibly in one project, or over multiple projects over time), 2a. develop / build theories or artefacts, and 2b. justify the theories or evaluate the artefacts. This is then followed by 3a. application of the research output to solve problems in the environment and/or 3b. additions to the knowledge base. Hevner et. al. note that the knowledge base from which applicable knowledge is drawn comes from many “reference disciplines” which “provide foundational theories, frameworks, instruments, constructs, models, methods, and instantiations used in the develop/build phase of a research study, and methodologies to provide guidelines in the justify/evaluate phase”. They go on to state that “rigor is achieved by appropriately applying existing foundations and methodologies” to each research acitvity (Hevner et al., 2004, p.76). MES Thesis: Strongly Evaluation Research Design December 19, 2011 Sustainable Business Model Ontology 16 Version 2.1
  17. 17. Al-Debei, summarizing the advice from several scholars, notes that such foundational theories are normally originated outside the information systems field [and that] such theories could be useful as they may suggest helpful approaches to information design problems” (Al-debei, 2010, p.35). 2.6 The Framework for Conducting the SSBMO Research To help set the overall framework for my research design, I have followed Bullinger’s lead (Bullinger, 2008, p.231), and adapted Hevner et. al.’s framework to this research project.Environment Relevance Research Rigor Knowledge BasePeople Philosophical• Executives, Entrepreneurs, • Critical pragmatism D. Build Investors, Business • Strongly Epistemological Architects, Consultants Sustainable • Systems Business Model • InformationOrganizations Ontology artefact: • Design• Strategy, operations and innovation planning and 1. Constructs Disciplinary Frames* 2. Model decision making groups 3. Method • Natural science 4. Instantiation P2. Applicable • Ecological: sociology,Technology P1. Problem economics & management Knowledge• Communication support • Organization (Innovation, assess Iterative refine• Generative (Abduction) Strategy, OM/IS) support Quality (reliability, Design consistency, Methods• Evaluative (Decision effectiveness) Process (D1-4) • Data collection, analysis Making) support and efficiency of creation of strongly E. Evaluate design and evaluation sustainable E1: Comparative techniques business models E2: Third-Party Tools / Techniques / E3: Case Study Formalisms • Literature Review • Entity Relationship Modelling • Interviews C1. Application C2. Additions to to solve problem knowledge base Figure 2: Research Framework for the SSBMO Research Project MES Thesis: Strongly Evaluation Research Design December 19, 2011 Sustainable Business Model Ontology 17 Version 2.1
  18. 18. Using Figure 2, the conceptual sequence of my research project is as follows: 1. The problem is understood (P1) and the applicable knowledge identified (P2) (These activities will be collectively referred to as “Preparation”). 2. Iteratively Build (D1-4) and Evaluate (E1-3) the SSBMO artefact. On occasion this will include iterations of preparation activities. 3. Communication the results of the research (C1, C2) (These activities will be collectively referred to as “Communication”).2.7 The Research Cycle for Conducting Design Science ResearchKuechler and Vaishnavi provide the following “research cycle” which suggests how thisframework can be turned into a process for inquiry (aka a task oriented project plan), showingprototypical, but more specific tasks for the researcher to undertake (Kuechler & Vaishnavi,2008, p.493) Figure 3: Design Science Research Cycle (Kuechler & Vaishnavi, 2008)The process steps (activities) in the research cycle aligns well with the overall research processdescribed by Hevner et. al. in their framework. Specifically, the Prepare (P) activities maps to“awareness of problem”, the Build” (D) maps to “suggestion and development, Evaluate (E)maps to “evaluation, and Communicate (C) maps to “conclusion”.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 18 Version 2.1
  19. 19. 2.8 Process of Inquiry for Conducting the SSBMO Research Other recent ontology development research projects have successfully used this “research cycle” to develop their own research project plans (Al-debei, 2010, p.21). Other researchers have used research cycles from the ontology engineering field which are highly similar (Bullinger, 2008,"1.1 Ontology Engineering" pp.199-215, p.207, p.232)16. Inspired by other researchers, and using both Hevner et. al.’s research framework and Kuechler and Vaishnavi’s research cycle, I developed my overall process of inquiry / project plan, integrating design and systems thinking17, for this research project. See figure below. P pP. r a Literature Review e r - e B D1: First D2: Second D3: Third Iteration of Build D4: Forth u Iteration of Iteration of Iteration of D. i Build 1 Build 2 3 Build 4 l d E1: Comparative E2b: 3rd Party E Analysis Review: Expert v a Interviews E. l u a E2a: 3rd Party E3: Formal t e Review: Informal Evaluation: Events Case Study Design Working Papers #1..n C Write-up Lit. Review, Finalize o m Design, E1, E2, E3 & Write-up: m Original Revised Research Design Design & u Proposal Proposal Case Study C. n i c Research Logs and Reflection Diary / Logs a t Possible Articles for Publication & e Other Communication 20 31 20 27 28 16 30 12 11 12 ay ay y ov b ly Fe l Ju M M Ju N Today: Develop Detailed Evaluation Research Design Figure 4: Overall Process of Inquiry for the SSBMO 16 Osterwalder had completed his PhD before this research cycle had been developed. 17 This aspect was introduced in Version 4.1 of my Research Proposal (August 8, 2011), and has been subsequently elaborated in my recent presentation “Design science, systems thinking and the creation of ontologies”. These materials will be included in the final methodology section of my thesis. MES Thesis: Strongly Evaluation Research Design December 19, 2011 Sustainable Business Model Ontology 19 Version 2.1
  20. 20. The process enquiry consists of four related “swim lanes” of activities. These swim lanescorrespond to the conceptual sequence of activities from the research framework Prepare (P),Build (D), Evaluation (E) and Communicate (C).The details and justification for the activities in the in the Prepare, Build and Communicate swimlanes will be described in the final thesis document. Describing and justify the details of theactivities in the Evaluation swim lane is the purpose of this document.Within this swim lane structure the practical sequence of my research project is as follows: 1. The original project proposal is written and approved (MES II-III exam May 27, 2011) 2. The templates to capture my research logs, reflections, and research diary are established and start to be used. 3. To understand the problem and the applicable knowledge, the “key theoretical frames” (K0), which will be required to build and evaluate the SSBMO the literature review work is started. 4. Using the initial output from the literature review a first version of the SSBMO artefact (constructs and model) is built (D1). The build uses an iterative systems thinking approach examining the function, structure, process and context of business models18. 5. Details of the SSBMO artefact are captured in the initial version of the “Design Working Papers”. 6. Based on learning from the first version of the SSBMO further literature review work is undertaken and a second version of the SSBMO artefact is built (D2) and described in the Design Working Papers. Again an iterative systems thinking approach is applied to the build activity. 7. Based on learning from the second version of the SSBMO further literature review work is undertaken and an initial third version of the SSBMO artefact is built (D3 is started) and described in the Design Working Papers. Again an iterative systems thinking approach is applied to the build activity. 8. Based on the accumulated learnings a revised project proposal is prepared, reviewed and approved (August 8, 2011). 9. The detailed research design of the evaluation activities is determined and documented (this document). 10.The Comparative Analysis and Informal Third Party Review Evaluation activities are undertaken (E1, E2a) and written up. 11.Based on the learning from evaluation activities E1 and E2a as well as the learning from the initial work on the third version of the SSBMO and the third version of the SSBMO artefact is finalized (D3 is completed) and described in the Design Working Papers. 12.Using the completed third version of the SSBMO the Formal Third Party Review and Case Study evaluation activities are undertaken (E2b and E3) and written up.18 See Version 4.1 of my Research Proposal for details.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 20 Version 2.1
  21. 21. 13.Based on the learning from evaluation activities E2b and E3 the fourth and final version of the SSBMO is built (D4) and described in the final project write-up. 14.The final write up of all aspects of the project is then undertaken (the thesis document), and submitted for review and approval. 15.Activities to communicate the results of the thesis work to practitioners and academics are then undertaken. 2.9 Summarizing the SSBMO Build Research Activities and Outputs (D1-4) The following table summarizes all the above and describes the goals, outcomes and metrics for the build activities of this project. This table is complemented by Table 30, sub-section 5.11 which summarizes the evaluation research design, the development of which is the topic of the remainder of this document. D. Build Research Activity Output Elements Goals Outcomes Methods 1. Constructs Identify the Ontology will contain D1-D4:Literature Review using relevant issues descriptions of the entities, Osterwalder’s PhD as a anchor for strongly and the contextual systems / along with numerous other sustainable groupings important to sources of key disciplinary business models describing strongly knowledge sustainable business models D1-D4: Secondary Data 2. Models Describe the Ontology will contain Gathering:Research Output “logic” of a descriptions of the strongly relationships between the • Knowledge gained from sustainable entities, systems and groups Course Work firm’s business and hence the “logic” of the • Informal discussions model business model with former colleagues and students 3. Instantiations With limited The ontology (the constructs explanation, and the model of their •Attending relevant have a manager relationships) will be practitioner events in a firm be able expressed diagrammatically D1-D4: “Science” of Design19 to understand and be presentable to D1-D4: Systems thinking the ontology managers techniques20 Table 3: Summary of the SSBMO Build Research Activities and Outputs (D1-4) 19 i.e. There is a body of knowledge about what constitutes the good design of some thing (physical or conceptual), and the processes, tools, techniques used to create such a design. This is the “science” of design. Contrast this to the overall methodological approach for this thesis using the processes, tools and techniques of design to undertake scientific research (design science). 20 See section 2.9.3 below for introduction to these techniques. More detail will be provided in final thesis. MES Thesis: Strongly Evaluation Research Design December 19, 2011 Sustainable Business Model Ontology 21 Version 2.1
  22. 22. The details behind these choices of activities were introduced in Version 4.1 of my ResearchProposal (August 8, 2011), and have subsequently been elaborated in my recent presentation“Design science, systems thinking and the creation of ontologies”. This detail will be includedin the final methodology section of my thesis.However, in reviewing the literature on design science artefact evaluation a number of pointsrelevant to both evaluation, but applicable to the build activities emerged. These points aredescribed in the following two sub-sections.2.9.1 Evaluation as Part of Preparation ActivitiesCiting March and Smith’s 1995 work (1995, pp.260-261), Osterwalder (2004, pp.127-129)suggests that the first level of evaluation is the strength of the researchers argument highlightingthe similarities and differences between previous work, i.e. the key theoretical frames (K0), andthe ontology’s design (constructs, models, instantiations) – i.e. evaluation actually starts duringprepare / literature review.Brank in his 2005 review of ontology evaluation methods from an ontology engineeringperspective suggest there are four broad categories of evaluation methods, of which the thirdaligns with March and Smiths recommendation “comparing the ontology to some authoritativedata source” (Brank, Grobelnik, & Mladenić, 2005).This was also the approach explicitly taken by Al-Debei. During the initial build of his ontologyhe “evaluated against the existing body of business model literature and his ontology DesignQuality Evaluation Framework” (Al-debei, 2010, p.21).Hence, as shown in the SSBMO process of inquiry (sub-section 2.8, Figure 4), the extantliterature will be used throughout all four iterations of the build activity, to continuously evaluatethe SSBMO against “authoritative data sources”, such as the BMO and applicable literaturedrawn from .natural and social sciences21, 22.The importance to evaluation of existing research supports one of the primary reasons forselecting Osterwalder’s BMO as the basis for the design of the SSBMO. The BMO hasconsiderable strength because of the evidence to support the BMO’s validity, and the apparentlack of any directly comparable work 23.21 Collectively I am referring to this body of applicability literature as the key theoretical frames for the SSBMO,and labelling this as K0.22 This body of literature is summarized in my recent presentation “Design science, systems thinking and thecreation of ontologies” and will be described in detail in my thesis literature review.)23 Both Al-Debei and Bullinger review a range of business model (and related, e.g. enterprise) ontologies (Al-debei,2010, pp.67-97; Bullinger, 2008, pp.133-171); Further with over 170,000 copies of the popular work derived fromhis PhD sold, a iPad app and a busy speaking and consulting business, Osterwalder’s work has considerablepractitioner validation (Osterwalder, Pigneur, & Clark, 2009; Osterwalder, 2011a; Smith, Osterwalder, BusinessModel Foundary, & Hortis - Le Studio, 2011)MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 22 Version 2.1
  23. 23. 2.9.2 Evaluation as Part of Build ActivitiesHevner et. al. recommends that during the build activities “the artefact itself must be rigorouslydefined [and] formally represented” (Hevner et al., 2004, p.81,table 1 p.82).Based on his review of the literature of ontology engineering and design science Al-Debeiidentifies six criteria for evaluating design artefacts (Al-debei, 2010, pp.43-46). The firstcriteria, clarity, matches Hevner et. al.’s recommendation, and hence must be undertaken duringbuild activities.Al-Debei, summarizing the literature on clarity, states that An ontology needs to successfully and objectively communicate the intended meaning of defined terms. Defined terms are concepts describing the domain, which will most likely be nouns (i.e. objects [constructs]), or verbs (i.e. relationships [a model]). Creating a list of these terms is important, as well as documenting their definitions in natural language [impacts human understanding / communication]. (Al-debei, 2010, p.43)Al-Debei goes on to provide a summary of items which lead, if they occur in the designedontology, to a reduction of clarity: 1. Construct overload: two or more ontological constructs map to one modelling (i.e. grammatical) construct. 2. Construct redundancy: two or more modelling constructs map to one ontological construct. 3. Construct excess: an existing modelling construct does not map to any existing ontological construct. 4. Construct deficit: an existing ontological construct does not map to any existing modelling construct. (Al-debei, 2010, p.44)My selection of the Entity Relationship Modelling formalism, coupled with my extensivedocumentation of the SSBMO in the Design Working Papers during the build activities will helpto meet these recommendations and maximize the clarity of the ontology24.2.9.3 Systems Thinking Evaluation Techniques Relevant to the Build ActivitiesWhile I will leave the description of the systems thinking elements of the build activities untilmy final thesis, I do want to describe some elements of systems thinking, related to evaluation,which need to be considered during the build activities25. The systems thinking elements of theevaluation activities are described in the sections below.24 See Working Paper #1 along with SSBMO summary entity relationship diagram v1.021 (July 17, 2011) forexamples of this documentation. A summary of Working Paper #1 and the summary entity relationship diagram areincluded in this Prezi Presentation. This presentation has been successfully used in evaluation activity E2a InformalThird Party Review and a number of other occasions when I have needed to communicate the content of theSSBMO’s design.25 A familiarity with Soft Systems Methdology is assumed in this document. An overview to SSM will be providedin the thesis. ( For a good summary see Jackson, 2000, pp.246-270)MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 23 Version 2.1
  24. 24. Firstly, considering Baskerville et. al.’s innovative and recent work which proposed amethodology which integrates Soft Systems Methodology (SSM) and Design Science(Baskerville, Pries-Heje, & Venable, 2009). These authors suggest how design scienceevaluation activities are highly similar to SSM comparison activities and serve the same purpose:determining whether the designed artefact (which in the case of SSM are root definitions and amodel) are fit for purpose. "The search for the design solutions [SSM stage 3 root definitions and stage 4 modelling, design science build activities] and the evaluation of the design solution are activities that take place in the abstract world of design thinking. Artefact construction and its evaluation are activities that take place in the real world of the social systems into which the artifact becomes situated.”This suggests that during build activities SSM techniques used to test completeness andadequacy of root definitions and models would be useful. One of the key techniques of SSM inthis regard is the application of CATWOE framework26.Specifically, I will use CATWOE to help determine whether each build iteration27 is complete(i.e. during build activities D1, D2, D3 & D4).Secondly, considering Ledington and Ledington’s 1999 paper (Ledington & Ledington, 1999).These authors offer a number of critiques and one suggestion for a techniques to improve theoutcomes of the SSM comparison processes (, p.336). To be effective in a design sciencecontext these techniques would need to be employed during the build phase.Ledington and Ledington’s insight is that it is natural for designers to (need to) have highexpectations that the artefact they are building will be highly desirable and highly important tosolving the problem at hand. Similarly, it is natural for users to (want to) have high expectationsthat the artefact that is being built will be highly desirable and highly important to solving theirproblem. This mental frame of reference is (implicitly) shared / co-created by both the designersand users. It acts as a context (a bias) to all subsequent evaluation activities. Thus, these authorssuggest, it is critical in the build activities to surface this context to help the designers to consideralternatives and the users to specify their evaluation criteria (Ledington & Ledington, 1999,p.336). This technique is discussed in more detail in sub-section 3.8.2.10 Summarizing the SSBMO Evaluate Research Activities and Outputs (E1-3)The explanation and justification for the four activities identified in the “Evaluate” swim lane ofthe process of inquiry is the topic of the rest of this document.A summary table for the evaluation activities, equivalent to Table 3, sub-section 2.9 whichsummarizes the build activity, can be found in Table 30, sub-section 5.11.26 The CATWOE framework consists of technique and knowledge. The technique aids the researching inconfirming that root definitions and models include all relevant: Customers, Actors, Transformation Processes,Weltanschauung (World-Views), Owners and Environmental Constraints.27 See Research Proposal v4.1 p.12MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 24 Version 2.1
  25. 25. 3. Evaluation in Ontology, Design Science and Systems Research3.1 IntroductionThe proximity of Design Science and Ontology Engineering is shown by the common notion ofconceptualization, construct and model. “The main difference is the inclusion of solutions[italics in the original] in Design Science which is not relevant to modeling an ontology”(Bullinger, 2008, p.221, footnote #245). Further, both approaches explicitly require theevaluation of an artefact.The proximity of Design Science and Soft Systems research is shown in three key ways: 1. The common desire to produce an improvement in the world. Design Science attempts this via the build/evaluation of an artefact. Soft Systems attempts this via planned action based on understanding a problem, modelling the problem and potential solutions, and comparison of these with the real world problem at hand (Baskerville et al., 2009). 2. The high degree of alignment of the notions of constructs, models and instantiations in Design Science with the notions of root definitions, models and solution construction in Soft Systems research (Baskerville et al., 2009). 3. The common idea of evaluation to test utility of the of constructs, models and instantiations in design artefacts and the comparison of root definitions and model to the “real-world” problem, and the subsequent comparison of the implemented solution with reduction in the “real-world” problem (Baskerville et al., 2009).These proximities strongly support the use of these three approaches, design science, ontologyengineering and soft systems, in my research28.Although there are clear commonalities between Design Science and Ontology Engineering, andbetween Design Science and Soft Systems research, it is only recently that the literature in thesethree traditions has started to be integrated. However, no work was located which attempted tointegrate Ontology Engineering with Soft Systems methods.I hope, by reviewing the body of work from these three previously largely isolated fields, to offerthe following contributions: 1. Offer a (the?) first attempt at a comprehensive review of the literature in these three diverse fields on their approach to evaluation to provide better guidance to other researchers in designing high quality and rigorous systems oriented design research focused on ontology evaluation (see section 3). 2. Offer a (the?) first version, based on this literature review, of a process to help myself and other researchers choose their evaluation research design (see section 4)28 The explanation and justification for this assertion as far as the evaluation activities in my research is included inthis document. An introductory explanation and justification of the application of these approaches to the prepare,build and evaluate activities has been outlined the presentation “design science, systems thinking and ontologies”and will be fully documented in my thesis.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 25 Version 2.1
  26. 26. 3. Use this review to attempt to improve the quality and rigor of my own research design and hence the utility of the SSBMO artefact over comparable artefacts produced by other researchers (see sections 5 thru 8).3.2 Reviewing the LiteratureWith these objectives in mind this section reviews the literature related to the evaluation of: • Ontologies (from Ontology Engineering), • Design science artefacts (from Design Science) • Ontologies that have been produced through design science research, and • Root definitions and models (From Soft Systems Methodology Research).The next section (section 4) describes how I choose (how I designed) the evaluation approach forthe SSBMO based on this review of the literature. Subsequent sections (5 thru 8) provide thedetails of the chosen evaluation research design for the SSBMO.This section is organized as follows: 1. Definitions of and broad perspectives on evaluation are reviewed 2. Views on the purposes of the evaluation process are presented 3. Views on the processes which can be used to evaluate artefacts are discussed 4. The research outputs (artefacts) which require evaluation are described 5. The metrics which have been used to measure artefacts are explained, and finally 6. The techniques (methods) used to capture values for the metrics are considered29.3.3 Evaluation: A DefinitionCleven et. al. (Cleven et al., 2009), propose a “general framework” to the design of theevaluation of design science artefacts. However, they helpfully begin with a review ofantecedents. This begins with a historical review of the role of evaluation within every-day lifeand more specifically within design science research.Cleven et. al. note that it is hard to define evaluation. However, synthesizing the definitions ofevaluation presented, leads me to the following working definition: Evaluation is the process of determining the worth, merit, significance and opportunities for the improvement of artefacts through the objective and systematic collection of information. Evaluations are the outcomes of that process.29 It is only through the capturing of values for the metrics that enable the researcher (and others) to judge whetheror not an artefact has met its design goals or not.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 26 Version 2.1
  27. 27. In providing additional substance to this definition Cleven et al observe that: • Evaluation may be “quantitative or qualitative, or a mix of these” and that evaluation “is strongly although not always sharply distinct from explanation”. • Accomplishing evaluation is hard work due to challenges with objectivity, comparability and tractability and the complexity inherent in resolving these challenges in a cost and time effective manner. • These complexities require an “adequate framework […to] support a structure proceeding of the evaluation intentions”. • A variety of different evaluation techniques have been developed from a range of disciplines: Information Systems, Business Administration, Sociology, Computer Science. • Many researchers have “deplored the absence of appropriate evaluation methods”, and that there has been “uncontrolled growth of new methods developed by eager practitioners and researchers” and that many of these “are lacking a theoretical foundation and an empirical validation of their utility”. This is particularly observed for artefacts, such as ontologies, created within the Information Systems field.Hevner & Chatterjee ((2010, pp.109-111) align with my working definition of evaluation andthese observations on the process of evaluation, characterizing evaluation as a “rather difficultand complex” process, refering to the “art” of evaluation (p.111).As noted earlier, Baskerville et. al. have proposed an integration of Soft Systems Methodologyand Design Science (Baskerville et al., 2009). In this paper these authors note how designscience evaluation activities are highly similar to SSM comparison activities serving the samepurpose, and this also observation fits well with my working definition above.Within this idea that soft systems comparison and design science evaluation are the same,Ledington and Ledington’s 1999 paper introduce a number of useful observations andinnovations related to the problems of comparison, which will be discussed in detail below. Fornow I note that Ledington and Ledington align with the design science researchers’ concernsnoted above about the poor state of knowledge and practice about evaluation. They state“comparison is problematic both in theory and practice” and go on to call some existing softsystems researchers comparison practices “nonsense”, adding “clearly, knowledge aboutcomparison is inadequate and creates practical difficulties in both transferring the approach toothers” (Ledington & Ledington, 1999).MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 27 Version 2.1
  28. 28. 3.4 Purpose / Objective of Evaluation3.4.1 OverviewDespite the concerns described above, overall the literature on the purpose of evaluation appearsclear and consistent, albeit recent. Summarizing this literature it would seem that: The purpose of evaluation within design science research is to provide feedback on the utility of the designed artefact which is then used to prepare a final revised artefact.Utility is a very broad concept. The research output whose utility should be assessed and themetrics that might measure utility will be discussed in detail in the sub-sections 3.4 and 3.5, afterdiscussing the process of evaluation in sub-section 3.3. However, in summary the literaturesuggests that utility includes three basic ideas: 1. Completeness 2. Quality 3. Beauty3.4.2 Supporting Summary of the LiteratureThe following summary of the literature is provided to support the above definition of thepurpose of evaluation in design science and ontology engineering research, and the definition ofcomparison in soft systems research.The Design Science and Ontology Engineering PerspectivesIn his 2004 PhD Osterwalder observed that “of all the authors that presented different businessmodel frameworks only [one] has written about some kind of evaluation having [been] applied”and this was informal, via the use of the framework in consulting work. “None of the authorshas set up any hypothesis and tested them in a field setting” (Osterwalder, 2004, p.142).Osterwalder suggests that the purpose of evaluation is to compare the designed artefact with the“initial goals of the research” (i.e. the problem which the design is trying to solve) (Osterwalder,2004, p.127).Writing at the same time as Osterwalder, Hevner et. al. (2004, p.82, table 1 p.82) agree withOsterwalders position, stating that since the designed artefact is purposeful, it must yield utility(including quality, and efficacy) for the specified problem, i.e. a design artefact must berigorously demonstrated via well-executed evaluation methods. Evaluation is a crucial component of the research process. The business environment establishes the requirements upon which the evaluation of the artefact is based. A design artefact is complete and effective when it satisfies the requirements and constraints of the problem it was meant to solve (Hevner et al., 2004, p.85).MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 28 Version 2.1
  29. 29. Going further than Osterwalder, Hevner et. al also describe how the output of evaluation is used:“the evaluation of the artefact then provides feedback information and a better understanding ofthe problem in order to improve both the quality of the product and the design process” (Hevneret al., 2004, p.78). i.e. the purpose of evaluation is the “identification of weaknesses in the […]artefact and the need to refine and reassess” (Hevner et al., 2004, p.80).In 2005 Ontology engineering researcher Brank et. al. concurred with this perspective:“Ontology evaluation is the problem of assessing a given ontology from the point of view of aparticular criterion of application, typically in order to determine which of several ontologies [i.e.competing design choices] would best suit a particular purpose” (Brank et al., 2005).Writing four years after Osterwalder and Hevner, Bullinger (2008, p.211) observed that “atpresent, the field of ontology evaluation is only emerging”. She goes on to cite Hevner et. al.(2004) noting that “it is typical for the field that the establishment of a working solution as wellas the characterization of the environments in which this artefact works are of primary concern,even if it cannot (yet) be explained completely why the solution works” (Bullinger, 2008, p.216).Summarizing Osterwalder and March & Smith ((March & Smith, 1995; Osterwalder, 2004)Bullinger states the “objective of the evaluation process is to determine the degree of correctnessof the ontology” (Bullinger, 2008, p.213) and that “evaluation means determining whether thedesign artefacts produced are effective, i.e. achieve their purpose, provide value, and / or productadverse or unwanted side-effects” (p.220) and that artefacts “should be evaluated before (re)useby one or more applications and users” (p.211).Concluding on her review of the design science literature Bullinger states that “objective of theevaluation process is to determine the degree of correctness of the ontology” (p.213) and goes tosay that the ultimate evaluation of a designed artefact is its viability and utility in solving theproblem which the designed artefact was intended to solve (p.216).More recently still, Cleven et. al. (Cleven et al., 2009), propose a framework for design scienceartefact evaluation. This framework is helpful in providing a more granular view of the purposeof evaluation. In this framework they suggest that there are four “closely interweaved”“functions” of evaluation: 1. Acquisition of Knowledge to help put “management decisions on a rational basis”. 2. Gaining Control over the utility of the designed artefact. 3. Development, based on the learning from the knowledge and control functions, enables the improvement of the artefact through dialog between designer and users. 4. Legitimization of the artefact based on either the process of its design (e.g. ex-anti traceability of antecedents and processes of construction), or the fulfilment of suitable metrics (e.g. ex-post).A year later, citing a range of antecedents, Al-Debei states in his PhD thesis that “it is importantthat ontologies are of a good quality, in order that they serve their intended purposes and beshared as well as reused [in] different applications” (Al-debei, 2010, p.17).Finally, in a recent book chapter focused specifically on evaluation of MIS artefacts in designscience, Hevner & Chatterjee state “the designed […] artefact is a social-technical entity thatexists within an environment (business and/or social) which lays out the requirements for itsMES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 29 Version 2.1
  30. 30. evaluation” (Hevner & Chatterjee, 2010, p.109). Hevner & Chatterjee go on to describe threepurposes of evaluation ((pp.110-111): 1. Promotional – provide evidence to uses of the utility of an artefact to increase the artefact’s subsequent use. 2. Scholarly – to uncover the (disciplinary) principles related to the artefact and its use. 3. Practical – to provide evidence to designers of the efficacy or efficiency of the content of, and methods or techniques used to build and evaluate the artefact.The Soft Systems PerspectiveAs noted earlier, Baskerville et. al. have proposed an integration of Soft Systems Methodologyand Design Science (Baskerville et al., 2009). In this paper these authors note how designscience evaluation activities are highly similar to soft systems methodology comparisonactivities serving the same purpose, determining whether the designed artefact (which in the caseof SSM are root definitions and a model) is fit for purpose.Overall the purpose of these comparison activities, which occurs in two places in the softsystems method (see Figure 6, next sub-section for details), is to ensure the people trying to solvethe problem reflexively / dialectically reflect on as many aspects as possible of their problem,their understandings of it, their proposed solutions, and the fit of the real-world to those solutions(Jackson, 2000, p.254; Ledington & Ledington, 1999).The intent, through this systemic approach to comparison, is to increase the likelihood, throughdialog and single and double loop learning, of uncovering solutions which dissolve the problemfor as many of the participants (stakeholders) as possible. SSM solutions ought therefore to besignificantly better aligned with more of the participants’ world views and hence needs. Suchresults are believed by many to create solutions superior to those arrived at through imposition ornegotiation.3.4.3 Implications for the SSBMOConsidering the literature above from a practical perspective, there are implications for theanalysis of any interviews used to gather feedback within a design science research project.Specifically, the idea that the purpose of evaluation is feedback to improve the designed artefactcalls into question both the need for and purpose of a “code book, a typical qualitative researchtechnique, as well as the nature of the “coding” process required following any interviews usedto gather feedback.In descriptive science research, such as those taking a grounded theory approach, the code bookand the coding process are critical in uncovering the evidence gathered from interviews to justifya theory, and to provide a documented chain of evidence from the evidence to the theory.However, in design science it appears that the use of the evidence gathered during interviews isdifferent. Instead of codes the researcher is looking for feedback (positive and negative) on allaspects of the utility of the designed artefact. Each element of feedback is then used in thesubsequent iteration of the build activities, specifically: • Positive feedback is recorded to support the existing design choicesMES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 30 Version 2.1
  31. 31. • Negative feedback is analysed to identify changes to the design which would improve its utility30.Clearly as the number of elements of feedback increases from multiple interviews the need forsome kind of grouping to aid the researcher in the analysis of negative feedback may be required.While these groups may superficially appear to fulfill a similar purpose to descriptive sciencecodes, they do not.However as with descriptive science interviews it is important for the researcher to undertakecross-case / meta-analysis. Not all interviewees will provide the same feedback. Some of thefeedback will likely be contradictory, based on the interviewees different world-views. Suchfeedback is a rich source of insight on artefact utility for the researcher. The researcher will needto synthesize and justify appropriate responses in the revised design to such contradictoryfeedback.In synthesising contradictory feedback the researcher needs to be aware that the underlyingworld-view differences may only be resolved through the researcher undergoing “double-loop”learning. This means in the subsequent iteration of the build activity, the researcher must notonly consider revising existing constructs and relationships in the ontology (“single-loop”learning), but consider revising assumptions made in the selection of these elements andchoosing wholly new assumptions and elements (“double-loop” learning).This is explored in detail in sub-sections 3.8 and applied in the evaluation activities which useinterviews for gathering feedback (Section 7, Third Party Review – E2, and section 8, CaseStudies – E3)3.4.4 Assumptions for the SSBMOArtificial intelligence (AI) ontology engineering makes a distinction between processes ofevaluation, validation, verification, assessment and testing (Bullinger, 2008, p.211 footnote#240).El Debei suggests there are two types evaluation activities: verification and validation: 1. Verification mainly refers to technical activities that ensure the syntactic correctness and cleanness of an ontology 2. Validation refers to semantic correctness; that is the process of ensuring that an ontology corresponds to the phenomenon that it is supposed to represent. (Al-debei, 2010, p.61)The differences are, according to these authors, because for AI and highly formalized(computerized) ontologies many of these processes can be undertaken by a computer.In this project, and this document, I will take these processes as synonyms of evaluation.30 See the next sub-sections for a discussion of the process of how the feedback is judged positive or negative.MES Thesis: Strongly Evaluation Research Design December 19, 2011Sustainable Business Model Ontology 31 Version 2.1

×