1. 2nd Business and System Conference 2013
Riga (Latvia), November 2013
Patterns in Business Analysis and
Enterprise Modeling:
How to evaluate their value?
Kurt Sandkuhl
The University of Rostock, Germany
Overview
Motivation
Background
• Patterns for Capturing Organizational Knowledge
• Evaluating the Business Value of IT
Evaluation Strategy for Patterns
Pattern Examples
• Information Demand Patterns
• Task Patterns
Pattern Use in Business Analysis
Summary and Future Work
12.05.2011 © 2011 The UNIVERSITY OF ROSTOCK | FACULTY FOR COMPUTER SCIENCE AND ELECTRICAL ENGINEERING
2
1

2. Motivation
Business analysis and solution design projects
• are highly specialized for the specific enterprise under consideration, i.e.
project is unique
• different enterprises in the same area often show similarities
• potential to reuse at least some of the knowledge from earlier business
analysis projects in later projects
Different techniques for supporting reuse in analysis and solution design
• high abstraction level: reference models/frameworks (COBIT, TOGAF, ITIL)
• Lower abstraction level: patterns
• Methods: means of capturing proven knowledge as support for action
(ARIS, OneMethodology, 4EM)
How to evaluated the “value” of patterns for business analysis process and the
solution developed?
Overview
Motivation
Background
• Patterns for Capturing Organizational Knowledge
• Evaluating the Business Value of IT
Evaluation Strategy for Patterns
Pattern Examples
• Information Demand Patterns
• Task Patterns
Pattern Use in Business Analysis
Summary and Future Work
12.05.2011 © 2011 The UNIVERSITY OF ROSTOCK | FACULTY FOR COMPUTER SCIENCE AND ELECTRICAL ENGINEERING
4
2

3. Patterns
Patterns in Computer Science and Enterprise Modeling:
• Initial definition:
Christopher Alexander: “Each pattern describes a problem which occurs over
and over again in our environment, and then describes the core of the
solution to that problem, in such a way that you can use this solution a million
times over, without ever doing it the same way twice”
•
Pattern type examples:
– Analysis Patterns (Fowler)
– Design Patterns (Gamma et al.)
– Architecture Patterns (Buschmann et al.)
– Workflow Patterns (van der Aalst et al.)
– Knowledge Patterns (Clark et al.)
– and many more …
5
Organizational Knowledge Patterns
An organizational knowledge pattern is a formalization of knowledge for a recurring
organizational task abstracting from organization-specific aspects, which is of value for
an organizational actor and an asset for an organization.
Characteristics
• OKP address recurring organizational tasks and at the same time abstracting from
a specific organization. They represent organizational knowledge, not individual
knowledge
• OKP are expressed in a formalized way, which requires a formal language or at
least a structured representation
• OKP are an asset of the organization, i.e. are not only a resource as such but
capture knowledge about the resource’s use
• An OKP is of value for an organizational actor in its original form and / or its
adaptation for a specific organization.
3

4. Business Value of IT
Kinds of approaches for evaluating the business value of IT
• Process-oriented approaches
• Perceived value approaches
• Project-focused approaches
• Multi-perspective approaches based on performance indicators
Process Oriented approaches (Mooney et al., 1995)
Focus: value created by process improvements in different dimensions
Potential IT Business Value Metrics (Mooney et al. 1995)
4

5. Perceived Values – DeLone and McLean (1992 + 2006)
Information can be captured at different levels
• the technical level,
• the semantic level and
• the effectiveness level
Identification of six dimensions of IS success:
Project-based Approach - Parker (1998)
•
•
•
•
Identification of stakeholders and risk factors
Financial and strategic objectives of the business; add the technical dimensions of a project
Definition of evaluation scales
Negotiation of the importance of each evaluation criterion via a weighting factor
(CSC 1997)
5

6. Multi-perspective approach - Zee (2002)
A ‘dashboard’ of relevant performance indicators for IT
Three levels
 Business Value of IT (business management level)
 Effectiveness of IT (IT management level)
 Effectiveness and Efficiency of IT supply (IT supply
management level)
BVIT is measurable on three dimensions
 Financial performance
 Business performance
 Strategic performance
Overview
Motivation
Background
• Patterns for Capturing Organizational Knowledge
• Evaluating the Business Value of IT
Evaluation Strategy for Patterns
Pattern Examples
• Information Demand Patterns
• Task Patterns
Pattern Use in Business Analysis
Summary and Future Work
12.05.2011 © 2011 The UNIVERSITY OF ROSTOCK | FACULTY FOR COMPUTER SCIENCE AND ELECTRICAL ENGINEERING
12
6

7. Evaluation Strategy for Patterns: Origin
Evaluation strategy: integrating two different lines of work
1.
pattern use as part of solutions
– evaluation of the solution, not the elicitation process of the pattern
type and the actual pattern
– patterns were evaluated from the economic perspective using the
balanced scorecard approach
2.
pattern type elicitation and actual pattern development
– evaluation strategies from naturalistic inquiry
• alternating phases of evaluation in theory and practice
• Internal and external validation
13
Evaluation Strategy: Characteristics
Four evaluation stages
• Assessment of the patterns by experts from the field
• Evaluation by the pattern users: perception of the pattern utility
• Case studies. Observation of real-world cases by researchers
• Measurement using scorecards: long-term way of assessing
Four perspectives
• the pattern concept from the perspective of an individual
• the pattern concept from the perspective of an organization
• the actual pattern from the perspective of an individual
• the actual pattern from the perspective of an organization
14
7

8. Overview
Motivation
Background
• Patterns for Capturing Organizational Knowledge
• Evaluating the Business Value of IT
Evaluation Strategy for Patterns
Pattern Examples
• Information Demand Patterns
• Task Patterns
Pattern Use in Business Analysis
Summary and Future Work
12.05.2011 © 2011 The UNIVERSITY OF ROSTOCK | FACULTY FOR COMPUTER SCIENCE AND ELECTRICAL ENGINEERING
15
Evaluation Strategy: Status for Example Patterns
Pattern:
Individual
perspective
Pattern:
organizational
perspective
Concept:
Individual
perspective
Concept:
Organizational
perspective
Assessment:
Expert Reviews
IDP:
[25] and [26]
IDP: [24] for
collaborative
engineering
TP: [32]
IDP:
[25] and [26]
IDP: [27] for change
management
TP: [30]
Perception:
User Evaluation
IDP: subject of section 4.2
Observation:
Case Studies
IDP: [24] and [27]
Measurement:
Scorecard
IDP: Not performed yet
TP: summarized in section 5.2
IDP: Information Demand Patterns
IDP: [28
TP: [31]
TP: TaskPatterns
16
8

9. Information Demand Patterns
Information Demand Patterns (Excerpt)
• Change Administrator
• Process Verifier
• Material Specification Responsible
• Responsible for Quote Preparation
• Responsible for Proposal Writing
• Branding Responsible
18
Example: Proposal Writing Responsible
Context
• Research and development projects at universities or research
institutes to a large extent are based on project funding. Preparing
successful proposals for funding is a task that requires experience
and accurate information about the published call, funding conditions,
economic parameters, etc., which usually originate from different
information sources and actors. The pattern describes the information
demand typically experienced by the role responsible for preparing a
proposal for funding.
• The pattern is supposed to be useful for any type of organization
aiming at submitting proposals for project funding to funding agencies.
19
9

10. Problem The pattern addresses the general problem of submitting proposals of
unnecessary low quality, which basically is wasting resources at the
preparing organization or reducing the probability and volume of grants. This
includes the following problems, which were often observed when preparing
proposals for funding:
• Submitted proposals fail to meet the formal requirements published in the
call, which usually leads to a rejection. Examples are missing
attachments, missing signature, wrong number or different kind of
partners.
• Submitted proposals contain an invalid or incorrect budget. Typical
mistakes are to include non-eligible costs, use the wrong cost rates or
don’t include applicable surcharges
• Proposals ask for funding of activities, which already are being funded in
other projects of the same fund giver
• Proposals are submitted to late, either due to deadline changes or
misunderstandings
20
Tasks and Responsibility
The tasks of the role responsible for a proposal include
• The preparation of the a competitive, complete and high quality
technical content of the proposal
• The preparation of an accurate, complete and consistent formal part
of the proposal
• Partner management, i.e. coordination of all contributions from
participating partners
• On-time submission of the proposal including all attachments, forms
and signatures
21
10

11. Information Demand
The information demand of the role responsible for a proposal consists of:
• The work program or policy documents guiding the funding from the fund giver
• General conditions from the fund giver stating eligibility of costs, preconditions to
apply for funding or evaluation procedure
• The complete call information, including supplementary material like the guide for
proposers, schedule of information meetings, forms to be submitted, etc.
• Economic parameters from the own organization, including personnel cost rates,
surcharges, administrative overheads, investment and facility costs to be
included, VAT rates, etc.
• Earlier funded projects at the fund giver in the subject field of the call under
consideration
• Plans of the competitors regarding submissions in the same call to the fund giver
22
Quality Criteria
General
importance
Work Program
Call Content
Accurate
high
high
as early as
possible
complete
high
high
decisive
decisive
high
decisive
General Conditions
high
high
high
Economic Parameter
high
decisive
Earlier Funded Projects
high
Nice to
have
Nice to
have
Nice to
have
Nice to
have
Nice to
have
Nice to
have
Competitor plans
Nice to
have
high
high
Nice to
have
11

12. Effects
Economic
effect
Time
efficiency
Quality
effect
Motivation
Learning
Experience
Customer
Work Program
low
Very high
Very high
moderate
moderate
moderate
Call Content
low
Very high
Very high
moderate
moderate
moderate
General
Conditions
moderate
Very high
moderate
low
low
moderate
Economic
Parameter
Very high
low
Very high
low
low
low
Earlier Funded
Projects
low
moderate
Very high
moderate
moderate
low
Competitor plans
low
moderate
moderate
low
moderate
low
24
Perception – Organisational Perspective
•
•
Method:
– Questionnaire, 5 point Likert scale
– Demand pattern ”proposal responsible”
Participants
– 4 different organisations from 3 different countries
– 2 research institutes, 2 universities
– Vice-director / Assoc. Dean for Research
– 25 – 100 proposals / year
– 20+ employees involved
25
12

13. Selected Results - Organisational Perspective
•
All respondents agree
– Very much / much with the tasks / responsibilities
– Very much / much with the quality criteria
– that the described effects are valuable / very valuable
(exception: timeline, motivation)
– that all elements of the pattern are needed
– Could imagine to use the demand pattern for
• training of new employees
• Training of current employees
• Basis or best practice description
• Improvement of best practices
26
Perception – Individual Perspective
•
•
Method:
– Questionnaire, 5 point Likert scale
– Demand pattern ”proposal responsible”
Participants
– 3 PhD students (less than 2 proposals)
– 4 researchers (4-6 proposals)
– 1 Head of research group (10-12 proposals)
– 2 senior researchers (more than 20 proposals)
– from
• 4 different organisations from 3 different countries
• 2 research institutes, 2 universities
27
13

14. Selected Results – Individual Perspective
•
The respondents agree
– All agree very much / much with the tasks / responsibilities
– 70% find that the information demand, quality criteria, effects and
task & responsibility descriptions are valuable / very valuable
– 90% find that all elements of the pattern are needed
– Difficult to estimate how much effort the overall pattern or the
different parts of the information demand pattern would save
– 3 propose the same additional part: ”background information from
program manager at the fund giver”
28
Task
Pattern
Example
30
14

15. Balanced Scorecard
Approach: Balanced Scorecard (Kaplan & Norton):
A measurement-based strategic management system, which provides a method of
aligning business activities to the strategy, and monitoring performance of strategic
goals over time.
Implementation of a scorecard in each MAPPER use case and on project level
Financial
Customer
Internal Business
Process
Learning and
Growth
31
Evaluation Results – Use and Quality of Task Patterns
•
•
•
•
Number of Best Practices
– number of best practice descriptions increased significantly
– C1: 7; C2: 5; C3: 6; C4: 9
Quality of Best Practices
– average level of detail of the best practice descriptions improved
– intensity of use of best practices at significantly C2 and C4,
– accuracy of the best practice descriptions
– training new employees within C4
Updating best practice descriptions
– The time needed to update best practice descriptions decreased
– at C3 22 days to 14 days; at C4 6 to 4 months; stable at C1 and C2
Conclusion: clearly positive evaluation of task patterns
32
15

16. Evaluation Results (2)
•
•
Quality of Product Documentation
– Number of design rules:
C3: from 95 to 160; C2 from 25 to 27; C4: no changes
– refinement level improved at C4 from 3 refinements to 4;
Cycle Time, Time to market
– indicators provided by C4
• material specification task: from 4 months to 3 months
(reduced by 25%)
– indicators provided by C1
• average length of the target setting process: from 14 to 12
months
• average time for assessing solutions for a customer problem:
reduced by 50%.
33
Overview
Motivation
Background
• Patterns for Capturing Organizational Knowledge
• Evaluating the Business Value of IT
Evaluation Strategy for Patterns
Pattern Examples
• Information Demand Patterns
• Task Patterns
Pattern Use in Business Analysis
Summary and Future Work
12.05.2011 © 2011 The UNIVERSITY OF ROSTOCK | FACULTY FOR COMPUTER SCIENCE AND ELECTRICAL ENGINEERING
34
16

17. Pattern Use for Business Analysis
Knowledge areas according to BABOK:
• Business Analysis Planning and Monitoring
• Enterprise Analysis
• Elicitation
• Requirements Analysis
• Solution Assessment and Validation
• Requirements Management and Communication
35
Pattern Use for Business Analysis (2)
IDP and TP do not offer support:
• Business Analysis Planning and Monitoring
• Enterprise Analysis
• Elicitation
• Requirements Analysis
• Solution Assessment and Validation
• Requirements Management and Communication
Patterns do not offer method support, but provide solution elements!
36
17

18. Pattern Use for Business Analysis (3)
Knowledge areas with potential pattern use:
• Enterprise Analysis
– identify the potential source for shortcomings and
operational problems in an enterprise
• Elicitation
– patterns propose a solution which has to be configured or
instantiated for the company under consideration
• Solution Assessment and Validation
– knowledge patterns can be used as a best practice reference
37
Summary & Future Work
The intention of our work is to contribute to evaluating the value of
patterns and improve patterns for use in enterprise modeling and
business analysis
Contributions so far
• (1) a proposal for how to evaluate the maturity and value of patterns
• (2) two actual pattern types and results regarding their evaluation
• (3) a discussion of pattern use in the context of business analysis
Future work
• Continue evaluation of more pattern types
• Further investigate the use in business analysis
18

19. Thank you for your attention!
Questions?
39
19