The purpose of Open Surveys is to help understand and improve the effectiveness of an organisational change or some aspect of organisational performance based on respondent comments. Open Surveys and their analysis are based on, and developed from, respondents’ thoughts and feelings expressed in their own words. • In an Open Survey respondents answer a few questions in detail by expressing themselves in their own words. Responses are grouped into categories, and from these categories broader patterns are built showing how comments are linked. Such patterns are, or can be fitted together, into a model covering and representing the collective views of respondents.

1. Open Surveys and their analysis
Open Surveys and their analysis
1. Open Surveys
The purpose of Open Surveys is to help understand and improve the effectiveness of an
organisational change or some aspect of organisational performance based on respondent
comments.
Open Surveys and their analysis are based on, and developed from, respondents’ thoughts
and feelings expressed in their own words.
Open Surveys are very different to what are termed Statistical Surveys for, in a sense, they
work in reverse. That is to say:

In an Open Survey respondents answer a few questions in detail by expressing
themselves in their own words. Responses are grouped into categories, and from these
categories broader patterns are built showing how comments are linked. Such patterns
are, or can be fitted together, into a model covering and representing the collective
views of respondents. In comparison;

Statistical Surveys use a “forced” response approach to questions that are structured
around a pre-agreed model. These models have factors (categories) that are used as the
basis to develop questions around an area of interest. Respondents have little to no
opportunity to explain their answers. The answers provide a means of statistically
(quantitatively) confirming responses against a survey’s underlying model’s factors
(categories).
As Statistical Survey’s generally do not explore questions in detail or depth, this can affect
how valid they are where issues are more complex and need respondents to explain reasons
for believing something, as is available through Open Surveys.
Open Surveys are also termed Open Question Surveys and use respondent comments that
are sometimes described as Narrative data; Text data; or Qualitative data.
2. Advantages and disadvantages of Open Surveys
Advantages:
 Allows respondents to answer in their own words.
 Allows for "richness" or "depth" of explanation in responses.
 Can identify and explore issues not currently fully recognised or understood.
 Can identify options for further action.
Disadvantages:
 Limited by the writing skills of respondents.
 Analysis involves a more complex interpretation.
 Stronger role of interpretation in analysing responses.
David Alman
October 2012 Version 3
Page 1

2. Open Surveys and their analysis
3. Designing Open Survey Questions
Questions used in Open Surveys usually use a word such as how, what, when, where, and
why, to allow respondents to express their thoughts and feelings such as their attitudes,
opinions, understanding, likes, dislikes, suggestions, and ideas.
Open (sometimes referred to as unstructured) questions are ones in which possible answers
are not suggested in the design of the survey questions, and where respondents answer in
the way they see the world.
Open ended questions are framed to encourage self explanation of thoughts and feelings in
a sentence, paragraph, or more.
Open ended questions include some direction on which the feedback is sought, such as:
 Please tell us what changes you would like to see?
 Please tell us what in your view we do well?
 Where could we improve?
 Why do you use our services?
4. Analysing Open Survey Responses
4.1 Open Survey and Analysis Framework
The purpose of an Open Survey design and analysis of responses is to provide a framework
that:
a) Accurately and as thoroughly as possible captures the intent of respondent comments;
and
b) Analyses these comments so that they construct, step by step, an understanding of
issues raised in a way that is both meaningful and relevant: That “fit and work” [1].
An explanation of how respondent comments are analysed, developed, and tested is
provided in 4.2 Steps in Analysing Comments.
4.2 Steps in Analysing Comments
Step 1. Read through all comments to get a feeling for the responses, and themes that
emerge from this overview.
Step 2 Develop Categories
2.1 Create categories. Identify categories (sometimes referred to as labels) from the
different themes that emerge from comments. When a response is allocated to a category
this is based on what is actually said and also any underlying meaning that might be
recognised.
2.2 Test to improve Category Analysis. Test to see if there are alternative categories that
have a better fit.
David Alman
October 2012 Version 3
Page 2

3. Open Surveys and their analysis
2.3 Decide if there are sub categories. Think about what the categories are about. Once
comments have been categorised look again at the responses in each category to identify
what is being expressed by different viewpoints. Are these consistent with the category,
and could they also identify sub categories that are of value in the analysis?
2.4 Quantify Categories. Assign comments to at least one category (or sub category) and
group them so the quantity of responses in each category can be more easily counted.
Refer to Diagram 1. Developing Categories from Respondent Comments.
Respondent Comments
ID
All Survey Comments
54
My supervisor is not
supportive and
criticises me in front of
others.
There is a lot of rework
causing delays.
Teams and Divisions
are unwilling to work
together.
Work processes are
inefficient and
bureaucratic.
There are conflicting
work objectives and
targets.
Work output is more
important than people.
84
165
248
362
420
Category: Role performance
ID
Comments
84
Total
220
There is a lot of rework causing delays
Category: Organisational objectives
ID
Comments
362
Total
63
There are conflicting objectives and
targets
Category: Work Relations
ID
Comments
165
Total
116
Teams and Divisions are
unwilling to work together
Category: Work processes
ID
Total
432
Comments
248
Work processes are
inefficient and
bureaucratic
Total
260
Diagram 1. Developing Categories from Respondent Comments
Step 3 Link Categories to develop Patterns and Model(s)
Once comments have been studied and categorised (and sub categorised), the next step is
to see how these categories link to form patterns and models:
 Do some categories link in some way, and how do unrelated others link to form
patterns?
 Do these patterns, together, form to represent one or more models?
In developing patterns check if there are exceptions to the rule that require the patterns to
be broadened to include other categories, or changed into other patterns, or cast doubt
over a pattern as a “rule”.
Refer to Diagram 2. Developing Patterns from Categories.
David Alman
October 2012 Version 3
Page 3

4. Open Surveys and their analysis
Pattern of Cultural Issues
Pattern of Performance Issues
Category: Organisational
objectives
Category: Work Processes
Category: Organisational
values
Category: Work Relations
Category: Job relationships
Category: Job Performance
Diagram 2. Developing Patterns from Categories
In developing models these may be either:
a) Drawn from the patterns of analysis;
b) Brought to the analysis from a recognised published model that links to categories
and their patterns. For example a recognised model relevant to the type of survey
carried out such as a Hospitality, Health Service, Service Delivery, Management,
Environmental Management, or Community Model.
Either approach to developing a model offers advantages, such as an internally developed
model can assist in gaining commitment from those involved in its development whereas
the use of an external model offers validity through its credibility as a recognised
benchmark.
Whichever approach is used the selected model should be as inclusive as possible of all
respondent comments.
Refer to Diagram 3. Organisational Alignment Model based on fit to Patterns.
David Alman
October 2012 Version 3
Page 4

5. Open Surveys and their analysis
Organisational
Objectives
Organisational
Values
Organisational
Performance
Organisational
Culture
Work
processes
Work
relations
Job
performance
Job
relationships
Diagram 3. Organisational Alignment Model based on fit to Patterns
Step 4. Write up the analysis: Once all comments are categorised and analysed, and the
patterns they form identified along with any model that fits those patterns, then the next
step is to write up in a summary.
The analysis can include:
 An explanation of a model and the patterns that pull the analysis together.
 Categories and sub categories identified (at the very least the more significant
categories).
 The number of comments covered by a category. This can be in either broad
quantitative terms covering more than one category, or specific to each category.
 An explanation about the categories and sub categories along with supporting (non
identifiable) quotations drawn from respondent comments.
 Recommendations based on the analysis. Recommendations for improvements can
come directly from comments where there are several similar responses or from a
single, different, comment.
The survey design and analysis should demonstrate validity:
 Face Validity:
o How survey findings make sense in terms of credibility, relevance, and usefulness
to respondents and those who decide to use survey recommendations;
o Use respondents own words in providing credibility to survey recommendations.
David Alman
October 2012 Version 3
Page 5

6. Open Surveys and their analysis

Construct Validity: The extent to which the survey design and analysis minimise error
and misinterpretation and is consistent in “fitting” with respondent comments. That is,
where comments fit with constructs (or concepts) in terms of developing criteria;
patterns; and models.
Notation.
[1] In references below, see Concept Mapping as an alternative Approach for the Analysis of
Open-ended Survey Responses, the term Concept Mapping is used. The term Construct
Mapping is used here instead of the term Concept Mapping so that there can be a better
and more logical alignment to Construct Validity. In this respect an Open Survey analysis
could be viewed as a form of Construct Mapping in that comments are organised (i.e.
constructed) first into categories; these categories are then organised (constructed) into
patterns; and where these patterns may then be organised (constructed) into a model.
When constructing such a map the outcomes (whether this relates to the construction of
categories, patterns, or a model) should readily, not forcibly, “fit” with respondent
comments and also “work” in the sense that categories, patterns, and any model selected
can be seen to provide meaningful and relevant conclusions. “Fit” in this sense relates to
Construct Validity, and “work” to Face Validity.
References include:
Outline explanations of Open Survey Analysis Frameworks
Analysing open-ended questions. Website:
http://intelligentmeasurement.wordpress.com/2007/12/18/analyzing-open-endedquestions/ Downloaded 8/8/12
Analysing Qualitative Data. Website: http://learningstore.uwex.edu/assets/pdfs/g365812.pdf Downloaded 8/8/12
A Brief Guide to the Analysis of Open-Ended Survey Questions. Website:
http://cms.cerritos.edu/uploads/ResearchandPlanning/Brief_Guide_to_OpenEnded_Survey_Questions.pdf Downloaded 8/8/12
Concept Mapping Explained
Concept Mapping as an alternative Approach for the Analysis of Open-ended Survey
Responses Website:
http://www.socialresearchmethods.net/research/Concept%20Mapping%20as%20an%20Alt
ernative%20Approach%20for%20the%20Analysis%20of%20OpenEnded%20Survey%20Responses.pdf Downloaded 8/8/12
Comprehensive References on Open Survey Design and Analysis
Qualitative Data Analysis. Ian Dey. Website:
http://www.drapuig.info/files/Qualitative_data_analysis.pdf Downloaded 8/8/12
Patton, M.Q. (1990). Qualitative evaluation and research methods (2nd ed). Newbury Park,
California: SAGE Publications Inc.
Organisational Alignment Model in Diagram 3 adapted from Figure 3.9 in:
David Alman
October 2012 Version 3
Page 6

7. Open Surveys and their analysis
Addison, R., Haig. C., & Kearney. L. (2009). Performance architecture: The art and science of
improving organizations. San Francisco, California: Pfeiffer.
Key words
Open surveys; Open survey analysis; Construct Mapping; Concept Mapping
David Alman
October 2012 Version 3
Page 7