This document discusses two new methodological innovations - SenseMaker and structural topic modelling - that could help bridge the gap between grassroots experiences and development research. It reviews literature on bottom-up development and traditional qualitative techniques. The document then outlines how SenseMaker and structural topic modelling were used in a research project in Kyrgyzstan to derive a development agenda from participant voices. Insights from each method are analyzed and their complementarity discussed.

1. Narrating Development:
Exploring the utility of
SenseMaker® and Structural
Topic Modelling for ‘bottom-up’
development research
Dissertation submitted in part-fulfilment of the
Masters Course in Global Governance and Ethics,
UCL, September 2016.
Candidate LMPX8

2. Contents
Abstract 2
Introduction 3
Literature Review 9
Bottom-Up Development . . . . . . . . . . . . . . . . . 9
Traditional Techniques . . . . . . . . . . . . . . . . . . 11
SenseMaker®, Cynefin and Knowledge . . . . . . . . . 13
Methodology 18
National Consultations . . . . . . . . . . . . . . . . . . 19
SenseMaker . . . . . . . . . . . . . . . . . . . . . . . . 22
Structural Topic Modelling . . . . . . . . . . . . . . . . 27
Data Analysis 34
Economic Growth and Employment . . . . . . . . . . . 34
State Administration, Governance, Democracy . . . . . 37
Ethnicity, Difference and Social Cohesion . . . . . . . . 40
Discussion 46
Conclusion 51
Bibliography 52
1

3. Appendices 57
Abstract
Many NGOs have long eschewed the suggestion that they are
“representatives” of the global poor, preferring to construe them-
selves in a weaker sense as “echo chambers” for marginalised
voices. This seems a pertinent metaphor, given that echoes in-
variably sound more distant and hollow than they did at the
source. This is by no means a criticism, so much as it is a frank
acknowledgement of the fact that inevitably, when trying to
echo the voices of millions of down the corridors of power, some
things will be lost in translation. The need to bridge this gap
in both policy and research is well-understood by academics,
and undertaken every day by development practitioners. Recent
innovations in machine learning and complexity theory have pro-
duced two novel methodological tools, which potentially promise
to slash the practical costs of field research, whilst simultane-
ously providing a fundamentally richer form of data and a radical
approach to how it is modelled. This paper seeks to assess the
fruits of - and explore the synergy between - Cognitive Edge’s
SenseMaker® framework and (Roberts M. E., et al., 2014)’s
structural topic model, by comparing it to a suitably analogous
research project, with the same objective but using a conventional
mixed-methodology.1
1
Full replication materials are available on the author’s Github page, or
alternatively at Dropbox.
2

4. Introduction
Development research places on its devotees a complicated set
of demands. On one hand, it asks us to explain a host of
economic, sociological, infrastructural and political obstacles
facing individuals, and how they interact with one another as
part of a dynamic self-reinforcing system, usually at country-
level. This is a natural playground for quantitative methods and
its adherents, where scale necessitates a degree of abstraction
from the lives of those development is supposed to benefit. On
the other hand, the last two decades in development research
have involved a great deal of theoretical soul-searching - and has
produced some hard truths. To many, it has become apparent
that painting an accurate picture with a broad statistical brush,
or in a more general sense, fabricating development policy on
behalf of the global poor in absentia, is at best inherently self-
defeating, and at worst economically devastating and socially
regressive.
The challenge thus becomes, how does one design research and
policy on a scale such that meaningful, sustainable difference can
be made, but ensure that it remains the legitimate possession
of its intended beneficiaries? This paper takes the position that
human experience, is invariably a structured form of data, and
it is often not the model or the choice of data, so much as how
we conceptualise it that alters what we observe. That structure,
in the three cases this paper addresses, might be a formal, con-
3

5. ceptual one built into the data by coder; a looser, sense-making
structure half-consciously generated in the same moment as the
data itself; or a probabilistic, quantitative structure approxi-
mated by an algorithm. Thus, it might be possible to bridge the
gap between grassroots experience and development research not
by tweaking our models and searching for ever more accurate
specification, but by tweaking the data itself - or at least our
perception knowledge exists in that data.
To proceed in this endeavour, the second section of this paper
describes two new methodological innovations from divergent
disciplines and underlines their particular salience for develop-
ment research. The third, outlines how each has specifically
been implemented as a means of deriving a truly participant-
driven development agenda for the future of Kyrgyzstan. The
fourth analyses the insights of each design, with an emphasis
on understanding the comparative merits of each, and the fifth
presents a discussion how they complement each other. But first
we conclude the present section with two recent examples of how
the disconnect between people and policy can serve to thwart
well-meaning development efforts.
Among its many controversies, much in the development commu-
nity has been made of a particular rider to the US Dodd-Frank
Wall Street Reform Act. To curtail the violence associated with
the extraction and sale of conflict minerals, Section 1502 of the
act requires publically traded companies to report to the Secu-
rities and Exchanges Commission whether or not they source
4

6. conflict minerals from the DRC or its neighbours.
The fear, even before implementation, was that of a de facto
boycott of Congolese minerals owing to market uncertainty over
regulation and the practical impossibility of tracing the origin
of particular minerals such as gold (Seay, 2012). Put in more
substantive terms, despite the fervent protests of advocates and
academics, the bill was passed without amendment and 5-12
million miners and their dependents have been significantly ad-
versely affected since the legislation was introduced. Furthermore,
little has been achieved in the way of reducing violence, because
in contrast to popular narratives, minerals in fact play a rel-
atively small role in driving conflict in the DRC (Autesserre,
2012). The failure of this initiative is not beyond the scope
of quantitatively-minded technicians to understand. When a
treatment succeeds in having the desired impact on a given re-
sponse variable, but fails to achieve or subverts, the substantive
objective of the project, we might construe this one of two ways.
Firstly, that our response variable (volume of trade in conflict
minerals) was all along a poor proxy for the inverse of what we
were attempting to achieve (the well-being of individuals in the
DRC). Thus the problem is one of model specification, one that
could be fixed through the correct operationalisation of more
appropriate variables. Or alternatively, one might argue that a
relationship does exist between trade in minerals and well-being,
but that it is only significant and negative in the presence of
some other condition(s) or variables; for the sake of argument,
5

7. say, labour market flexibility, a skilled workforce, and growth of
other sectors of the economy.
Thus, researchers and policymakers in the global north have
a relative simple means of understanding where this initiative
took a wrong turn. But the point here, is not so much that
development is somehow “un-model-able”, so much that in light
of devastating consequences of Dodd-Frank in Eastern and Cen-
tral Africa, reducing the needs of some individuals to a handful
of easily operationalised variables, will often involve misspec-
ifying the “real” interests of those individuals if they are not
sufficiently involved in the process. In development, where a
single misguided intervention can undo years of painstakingly
earned marginal gains, there is little room for failing then run-
ning the experiment again with an updated model. Moreover,
all social scientists understand that even the best formal models
are merely simplifications, or approximations of a much more
complex system, that a certain amount of unexplained variance
will always exist, that the model will never generalise to every
scenario (Schwartz-Shea & Yanow, 2012). Because empirical
development research often takes place in the field, it may often
be ethically difficult to justify an idealised research design, even
when practically possible:
“It is often hard to convince a government to random-
ize a development program, which can be antithetical
to its aims; for example, a program that aims to
6

8. reduce poverty will (hopefully) focus on poor peo-
ple, not some random sub-sample of people, some
of whom are poor and some not. It can be hard to
justify to the public why some people in obvious need
are deliberately denied access to the program (to
form the control group) in favor of some who don’t
need it.” (Ravallion, 2008, p. 9)
The previous example was simple at least in so far as it easy to
retrospectively point out what went wrong, at least in a prac-
tical sense. Things become slightly messier when one tenet of
development comes to contradict another. In March 2011, in
tandem with three Ghanaian NGOs, Oxfam published a report
titled “Achieving a Shared Goal: Free Universal Health Care in
Ghana”. Having criticised healthcare provision in Ghana as “un-
fair and inefficient” and asserting that Ghana’s National Health
Insurance Authority had overstated the number of individuals
covered by the national health insurance scheme by 44%, the
report generated a sizeable controversy in the development com-
munity (Rubenstein, 2014). Though the NHIA did eventually
adjust the methods by which it calculated coverage, the scathing
tone in which some perceived the report to have put down a
“home-grown African initiative”, and the way in which major
institutions discursively sidelined the three African NGOs that
had co-commissioned what came to be known as the “Oxfam
critique”, raised serious questions about the relationship between
development NGOs from the global North and the countries and
7

9. NGOs with whom they are meant to be “partners”.
In such an instance, how does an NGO balance the need to
criticise and thus prompt improvement, but also be reflexively
conscious of the privileged position from which one speaks? As a
positive, the concerns of the report resulted in change in policy
that (let’s assume) is for the better of Ghanaian healthcare. But
in doing so, the relationship between NGOs in the developed and
developing world was dealt significant damage, a relationship
that in the minds of many functions (in theory) as a transmis-
sion belt, conveying ideas, beliefs and outlooks from the most
marginalised parts of the world to the tables of power (Clark,
1992). Moreover, this incident might have been avoided if either
camp could claim to have the voices of uninsured Ghanaians rein-
forcing their position; whilst the NHIA or Oxfam’s three partners
in Ghana might well have provided a better approximation of a
“voice of the unrepresented”, or at least a genuinely Ghanaian
one, this is not equivalent to “bottom-up” development:
“NGOs have often created their own abstract con-
stituencies; are socialized in the value systems and
thought patterns of the global elite; and project their
own construct of the issues purported to be those
of the poor while they consciously or unconsciously
protect their own interests and those of their kind. It
is not a question of Northern versus Southern NGOs,
as is often portrayed; it is the poor versus both.”
8

10. (Nyamugasira, 1998, p. 300)
NGOs thus find themselves in an awkward position - how to
amplify these voices, without appropriating them.
Literature Review
Bottom-Up Development
As much as the author hopes these examples have been as in-
structive as they are illustrative, it is necessary to position them
within a broader set of scholastic trends spanning a multitude of
disciplines. The preceding cases provide an illustrative example
of how a disconnect between those in charge of development and
those who would supposedly benefit from it can led to drastically
adverse outcomes. This broad, generalised notion appears in
plethora of more concrete, formalised notions across social science
and political theory, and has been brought to prominence by
more than a few scholars of various traditions and philosophical
persuasions. In development scholarship, this is manifested most
visibly in the seminal contributions of William Easterly, but his
central theme - that top-down, Western-derived approaches to
the developing world are fundamentally flawed, underpins various
critiques (Easterly, 2006).
Poststructuralists have long emphasised discourses such as “white
saviour” and “white man’s burden” that “reinscribe colonial
9

11. narratives of Africa’s diverse peoples as passive and helpless”
(Bell, 2013, p. 3). From a more materialist orientation, many
economists espouse theories that highlight the relationships of
power and dependency that are reinforced by the international
aid regime (Easterly, 2006). Others, that the neoliberal eco-
nomic medicines that worked for MEDCs do not necessarily work
when exported to radically different societies (Stiglitz, 2001),
others still, that developed nations never used these medicines
to develop at all - rather these infant economies were nurtured
under protectionism, and global free markets were only intro-
duced to cement their dominance once mature (Chang, 2003).
Development as concept, owes much of its intellectual heritage
to the work of Edward Said (via Amartya Sen), who originally
crystallised Western condescension to the developing world as
“orientalism” (Said, 1978).
Taking Sen’s lead, sustainable development rests on the premise
that development is only meaningful, and only sustainable, if
those who would benefit from it are themselves the architects of
that development. Indeed, this is where development was once
originally differentiated from modernisation, as an internally
driven, self-realising project, rather than striving to achieve some
idealised Western notion of an advanced society (Sen, 1999).
This is not so much a research puzzle that is to be finally “solved”
so much as it is a heuristic that all professionals working in
development must use to guide their efforts, so to avoid a lengthy
digression, suffice to say in the words of Michael Edwards:
10

12. “Agendas for advocacy should grow out of action and
practical development experience, not from the minds
of thinkers in the North, however brilliant these may
be.” (Edwards, 1993, p. 173)
Thus we return to the central tension between avoiding develop-
ment as designed and prescribed by researchers in a laboratory,
and being able to mobilise and effectively implement resources
overwhelming situated in the Global North.
Traditional Techniques
In social science, when we want to understand how people con-
ceptualise their own experiences in a relative degree of depth, we
often turn to qualitative methods. Particularly with regards an
exploratory research puzzle, attempts to identify new phenom-
ena are constrained in supervised learning by the necessity of
identifying and operationalising (potentially) relevant variables
in advance. In a semi- or unstructured interview setting, space
remains for unanticipated data to arise from the unorchestrated
behaviour of the respondent. In this sense, people’s experiences
are a form of unstructured data, thus it is not immediately
amenable to analysis using supervised learning methods.
If we wish to quantify it of course, arduous, painstaking hand-
coding by skilled coders can be used to convert unstructured
information into structured data. Although a popular and often
fruitful methodology, this too has many drawbacks. Firstly, even
11

13. the most ardent devotees of quantitative methods accept that
beyond a certain granular fineness, human experience cannot be
further homogenised into numerical variables without the loss
of significant data. An open, inductively-minded coding scheme,
more concerned with the discovery of new phenomena than
accurate categorisation and inter-coder reliability can mitigate
this challenge, but the burden still rests on the coder to sift the
potentially relevant data from mere “information”, and there
is no determinant way of knowing what, if anything has been
missed. (Schreier, 2012)
Secondly, coding can be an incredibly costly practice. In labora-
tory settings, with a relatively small sample size and adequate
manpower this is a manageable cost. But when attempting cross-
national or country-level evaluations, and particularly in devel-
opment (where research is often field-based), language barriers,
time, access or funding constraints can render such a methodology
off-limits to all but the most resource-blessed undertakings.
Finally, no survey framework is ever created in a vacuum. Every
researcher, drawing on their own research question and pre-
existing assumptions about the social world, will shepherd their
participants towards certain responses. Again, a well-designed,
well executed design will mitigate these issues and still be able
make strong, valid knowledge claims, but matter how open-
ended the framework a degree of path dependency is always
guaranteed. (Halperin & Heath, 2012) Moreover, researcher bias
is often introduced not at the design stage, but by the researcher
12

14. themselves during the interview itself in interaction with the
participant.
SenseMaker®, Cynefin and Knowledge
The aforementioned pitfalls of quantitative and qualitative tech-
niques are not novel, on the contrary they will be familiar to
anyone with a basic understanding of research design. Nonethe-
less, it helpful to restate them in light of the new techniques
that will be extrapolated here, which exhibit some of the better
(and worse) qualities of both. SenseMaker®, is a bespoke data
collection framework developed by Cognitive Edge, that aims to
capture human experience on a large scale in its rawest and un-
curated form, through “micronarratives” - short stories given by
participants in response to a single, maximally-open prompting
question.
Before outlining the methodology itself, it is worth taking a
moment to consider the theoretical premises on which it rests,
and why they are particularly salient for development research.
SenseMaker® draws on Dave Snowden’s Cynefin framework,
developed in complexity theory and knowledge management as
a radical challenge to deterministic and rational choice-oriented
understandings of how change occurs a complex system (Snowden
& Boone, 2007). A full discussion of the Cynefin framework
is beyond the scope of this paper2
, but for our purposes, it
2
For a succinct but enlightening exposition of the Cynefin framework,
see (Snowden & Boone, 2007).
13

15. assumes that a “complex system” - such as, in the present case,
a developing society with a history of inter-ethnic conflict - is
one in which:
1) Multiple “right” answers/courses of action may exist, but
cannot be reached deductively.
2) The whole system is more than the sum of its parts (limit-
ing one’s ability to understand it by analysing individual
components).
3) Constant flux removes order/predictability.
4) A realm of “unknown unknowns” (in brief, not only do we
not know the causal mechanisms, we do not even know the
features and phenomena for which we are searching).
5) It can only be understood in retrospect.
Without overstating the case, there is probably much in this line
of thinking will resonate with development scholars who have
spent much time wrestling with dilemmas such as: why law-like
generalisations as to what “causes” development are so hard
to sustain? Why successful development interventions prove
difficult to scale up or stretch to different contexts, and why
country-level development successes can so often be explained
but not replicated? (Ravallion, 2008)
The objective of Cynefin, and its workhorse toolkit SenseMaker®,
is not to provide definite answers to any of these questions. In
fact, it supposes that by the time you’ve reached a scientifically
formal answer, the question will have changed anyway. Instead
14

16. the idea is to “make sense” of a given situation, rapidly, in
real time, so that in lieu of formal model of what is occurring
and a definitive recipe of how respond, one might develop some
general, scenario-specific heuristics for responding dynamically
to a changing situation.
In practical terms, SenseMaker® is a software package, gener-
ally accessed on a tablet or laptop and implemented online for
real-time data collection. Respondents are asked a prompting
question, in the case of the dataset that will be explored later:
“Think about a recent experience of yours, or some-
one you know, that made you feel hopeful or upset
about the future of your country or community. What
happened?”
Of course no researcher-initiated prompt will ever be entirely
value-free, but it is evident from this example how SenseMaker®
seeks to minimise unconscious cues in comparison with conven-
tional methods. By the same token, with so little information
as to the researcher’s interests at hand, one would assume that
performance on the part of the participant is also minimised.
But going beyond these common methodological obstacles found
across social science, collecting data in this way possesses two
further merits with respect to how knowledge is stored and
communicated in complex systems.
Knowledge management over the past decade has increasingly
come round to the notion that knowledge is not something that
15

17. exists in a static, independent form, “but an ephemeral, active
process of relating” (Stacey, 2001). As such, attempts to di-
rect respondents towards “useful” knowledge through structured
survey or interview frameworks are flawed, in that knowledge
artifacts are not simply apples to picked by researchers - “we can
always know more than we can tell, and we will always tell more
than we can write down” (Snowden D. , 2002, p. 111). Thus,
distilling them down numerical scores on a predetermined scale,
or recoding them in accordance with an academically derived
coding scheme, not only reduces data, but fundamentally distorts
its ontological status.
Recent implementations of SenseMaker® in political science take
this quite seriously. (Lynam & Fletcher, 2015) adopt this position
when assuming there considerably is more to know about people’s
attitudes towards climate change than they actually express when
asked directly. Development researchers who care about the gulf
between understandings of the challenges that the global poor
face in their own terms, and in the terms of policymakers and
academics, should take note of this case especially, as (Lynam
& Fletcher, 2015) analyse a comparable case of “power-people”
dissonance, in this instance on attitudes to climate change.
Secondly, earlier in this paper, it was suggested that human
experience, self-described, was a form of unstructured data. In
the sense of formal scientific method this is true, but in a cogni-
tive, epistemological sense narrative is in fact a highly structured
practice by which humans store, make sense of and convey knowl-
16

18. edge (Lynam & Fletcher, 2015). In contrast to quantitative text
analysis which generally posits a “bag of words” assumption,
under which the semantic structure of text is stripped out by the
model, “[n]arrative is a vital human activity which structures
experience and gives it meaning”; it “is a way of knowing” that
“assist[s] humans to make life experiences meaningful”. (Kramp,
2004, pp. 104, 107) In bridging the gap described at the start of
this essay between the outlooks and mindsets of researchers and
policymakers and people in developing communities, the utility of
narrative experience captured by SenseMaker® for development
research is well worth exploring.
Having supplied a short story in their own terms, respondents
then complete a short series of interactive “signifers”, (this phase
is predetermined by the researcher) which commonly come in
three forms:
1. Placing a marker on triads or dyads, providing ternary
data or a polarity measure of where the story places on
two/three research relevant dimensions.
2. Answering a series of multiple choice questions with regards
the story itself. (E.g. “In this story who had the power to
influence events?”)
3. Answering a series of multiple choice questions about
the respondent. (Often common demographic informa-
tion, e.g. age, gender, ethnicity). As is apparent, by pre-
specifying a set of responses, this second phase represents a
17

19. departure from the participant-led orientation of the narra-
tives. Having the participant self-signify their stories in the
second instance preserves the independence of the stories
from the research design, whilst allowing the researcher to
flexibly manage the trade-off between having respondents
self-signify and getting to information they suspect will be
interesting for the research.
But perhaps the principal opportunity offered to development
researchers by SenseMaker® is the massive reduction in prac-
tical expense. By eliminating the resource-intensive procedure
of collecting and hand-coding data, SenseMaker® introduces a
mixed/hybrid method of field research that is vastly more feasible
in practical terms - especially when collected digitally . Not only
does this drastically lower the barriers of entry for research, but
by the time collection and coding would otherwise have been
completed and any meaningful insights have been produced -
following previous theoretical assertions - in a complex system
the situation on the ground will likely have changed entirely
(Snowden & Boone, 2007).
Methodology
In order to investigate the utility of novel methodological tech-
niques for development research, this paper seeks to compare
the fruits of three approaches. Firstly, the findings of a conven-
18

20. tional mixed-methods investigation, the Post-2015 Development
Agenda National Consultations in the Kyrgyz Republic, as a refer-
ence framework. Secondly, analysis of the self-signified responses
of the participants in a SenseMaker® mass capture performed
by UNDP Kyrgyzstan. This consists primarily in descriptive
statistics and standard regression techniques. Finally, the re-
sults of the application to the micronarratives themselves of a
recent innovation in quantitative text analysis, structural topic
modelling (Roberts, et al., 2014).
National Consultations
The Post-2015 Development Agenda National Consultations in
the Kyrgyz Republic (henceforth the National Consultation) be-
gan in late 2012 as a UN-led initiative that sought to compose a
broad picture of perspectives on development from all sections of
Kyrgyz society, gravitating around the question “What Future
Do You Want for Kyrgyzstan?”. The consultative process con-
sisted of a diverse range of methodologies of a mostly qualitative
colouring, namely: surveys, focused group discussions, social
media discussions and voting. 1685 respondents representing
a range of traditional and non-traditional voices were engaged
with the object of identifying three important themes they see
as “important for the country”. 48% of respondents were women,
split evenly across urban and rural areas, and traditional and
non-traditional sectors of society. 6% of respondents were under
19

21. 18, and 44 children who participated in focused group discussions
were from disadvantaged or remote areas of the country. More
than half the respondents had a graduate or postgraduate edu-
cation, roughly reflectively of a society (41.1%) where the state
has long put great emphasis on education (UN in the Kyrgyz
Republic, 2013).
The National Consultation findings represents an ideal reference
document for comparison of traditional methodologies against
the new, for three reasons. Firstly, the sample size is comparable
to that of the SenseMaker® capture3
. Secondly, the objectives of
both research initiatives are broadly similar. Both are exploratory
research designs that aim to uncover what development challenges
are meaningful to different people in Kyrgyzstan, and what a
people-centric development agenda ought to look like4
. Consider
the SenseMaker® prompting question in comparison with the
National Consultation “main question”:
“Think about a recent experience of yours, or some-
one you know, that made you feel hopeful or upset
about the future of your country or community. What
happened?”
“What Future Do You Want for Kyrgyzstan?”
Finally, the research design for the National Consultation is very
3
For a succinct but enlightening exposition of the Cynefin framework,
see (Snowden & Boone, 2007).
4
National Consultation: n = 1,685; SenseMaker®: n = 1002
20

22. much in the spirit of the development issues discussed in the pre-
vious section of this paper. A range of qualitative methodologies
were employed in order get a broad sweep of beliefs and perspec-
tives from all corners of Kyrgyz society, expressed in their own
terms, in order to inductively derive a development agenda from
society itself. The published report details its findings as succinct,
prose discussions of the results of the combined methodologies on
eleven global development “themes” in order of their prominence
in the responses, rather providing a statistical breakdown of
each method. These eleven global themes are the only part of
the design derived “from above”, rather than the participants
themselves, and serve to make the finding congruent with the
global Post-2015 framework and the UN’s broader sustainable
development agenda, and are intended to organise the findings,
rather than guide their substantive content (UN Development
Group, 2012). Reporting the findings in this way is in keeping
with the goal of an agenda that is participant-centric, in so far
as it does not homogenise diverse responses down to researcher-
specified numerical variables or concepts in a coding scheme -
the object is to paint a picture, not discover a distribution.
As the goal of this paper is to compare the relative benefits of
different methodological approaches for bottom-up development,
that the National Consultation is so different in the technical
implementation of its design, but so similar in its underlying
methodological ideals, it makes for as close-to-optimal a reference
framework as one could hope for. By choosing this research,
21

23. the aim was to approximate something akin to a most-similar-
systems design (MSSD) - in which the contrasting methodologies
themselves were the systems.
SenseMaker
Because of the way it is inextricably related to the substantive
premise of this essay, the theoretical underpinnings of Sense-
Maker® have largely been extrapolated in the previous section.
Nevertheless, it would be prudent to outline the details of this par-
ticular implementation. Starting in 2015, UNDP’s Regional Hub
for Europe and the CIS spearheaded a collaboration between five
national UNDP offices (Kyrgyzstan, Moldova, Serbia, Tajikistan
and Yemen), Unicef Kyrgyzstan, Cognitive Edge and University
College London. Over several months, data was collected using
SenseMaker® from between 500 and 1400 individuals in each
country5
. Each UNDP office developed their own framework of
signifiers under the guidance of Cognitive Edge, reflecting their
unique research interests. In Kyrgyzstan, UNDP collected 1002
micronarratives from participants broadly stratified by age, gen-
der, ethnicity/language, education, across urban and rural areas
of each of the Republic’s seven oblasts (regions). The signifiers
consisted of nine triads, one dyad, four “stones”6
, six multiple-
5
It should be noted that UNDP Kyrgyzstan had a more specific interest
relating to the consequences of perceived Islamisation and ethnic difference
for development in Kyrgyzstan. Whilst this will have had a negligible
bearing on narratives themselves, given the prompting question, it will have
guided the development of the signifiers.
6
A “stone” involves placing a marker in two-dimensional space, where
the x axis represents one polarity measure (harmony/dissonance), whilst the
22

24. choice questions about the narratives, and seven multiple choice
questions about the narrators themselves (largely demographic
information).
To make for a meaningful comparison of SenseMaker® to the
National Consultation and to the final method, STM, the main
variables of interest was taken from the self-signification phase,
specifically a question that asks “This Story Involves.?” with 12
multiple-choice non-exclusive responses:
1 Security Democracy
2 Education Human Rights
3 Politics Conflict
4 Violence Int. Relations
5 Business/Trade Cooperation
6 Justice Other
Whilst these responses are not perfectly analogous to the themes
of the National Consultation, the question itself captures the gen-
eral notion of “what this story contains, that can be understood
through the lens of development”. Moreover, both the question
and responses speak to the idea of a topic for the story, a merit
that will become self-evident later. These 12 respondent-signified
responses were regressed over the demographic information in
the same dataset using standard OLS regression, namely: age,
gender, region, language/ethnicity7
, and a binary variable denot-
ing whether the participant came from an urban or rural area.
y axis represents another (society/individual). Both measures must relate
semantically to the marker stone, which will having a signifying question
(“How the government sees events?”), but need not relate to each other.
7
The exact question (in English) was “What is your native lan-
guage/ethnicity?”.
23

25. An interaction effect was included to observe whether living in
urban or rural area of each region yielded anything interesting,
but in most models this simply reduced the sample size in each
factor level too much to retain any useful statistical power. Ide-
ally, it might have been interesting to analyse “story involves”
in relation to some of the relevant dimensions from the triads
(e.g. “In this story the government was. stable/fair/accessible),
but the prevalence of missing values8
in the triads put additional
strain on an already small sample.
The strength of SenseMaker® as a commercial software package,
lies more in its incredibly powerful toolbox for data collection,
rather as an analytical instrument. This author did not possess
the software itself, just the dataset, two sets of findings produced
with the software performed by Cognitive Edge and UNDP Kyr-
gyzstan respectively, and the generous intellectual contributions
of Cognitive Edge staff. The package includes some accessible,
intuitive devices to “make sense” of what is happening in the
data, probably adequate for the purposes of the user who is not
trained in statistical methods, but is perhaps a little weak if
one is trying make robust, insightful scientific knowledge claims.
(Lynam & Fletcher, 2015) came to the same conclusion, and
their work saw fit to introduce their own methods. The largest
limitation is the inability to analyse the narratives themselves,
rather than the just the signifiers. As the rawest, unconditioned
8
The triads were stored as continuous data, but users could alternatively
specify N/A.
24

26. aspect of the participants’ responses, it was essential to find
a technique that could systematically analyse the narratives,
preferably with respect to their distribution over the signifiers -
which is where our third methodology is introduced.
25

27. SecurityEducationPoliticsViolenceBusiness/TradeJustice
female177(0.15)125(0.11)82(0.07)47(0.04)40(0.03)91(0.08)
male186(0.18)96(0.09)74(0.07)48(0.05)39(0.04)84(0.08)
urban.area125(0.15)80(0.1)78(0.09)44(0.05)37(0.04)80(0.1)
rural.area238(0.18)141(0.11)78(0.06)51(0.04)42(0.03)95(0.07)
Batken77(0.21)29(0.08)19(0.05)10(0.03)17(0.05)15(0.04)
Chui69(0.18)43(0.11)30(0.08)16(0.04)10(0.03)31(0.08)
Issykul63(0.13)31(0.06)58(0.12)25(0.05)28(0.06)46(0.1)
Jalalabad31(0.14)33(0.15)11(0.05)3(0.01)6(0.03)17(0.07)
Naryn0(0)0(0)0(0)0(0)0(0)0(0)
Osh95(0.19)69(0.14)24(0.05)31(0.06)11(0.02)47(0.09)
Talas28(0.14)16(0.08)14(0.07)10(0.05)7(0.04)19(0.1)
3632211569579175
DemocracyHumanRightsConflictInt.RelationsCooperationOther
female44(0.04)202(0.18)82(0.07)40(0.03)21(0.02)196(0.17)
male41(0.04)163(0.16)72(0.07)36(0.03)22(0.02)169(0.16)
urban.area41(0.05)113(0.14)58(0.07)29(0.03)31(0.04)121(0.14)
rural.area44(0.03)252(0.19)96(0.07)47(0.04)12(0.01)244(0.18)
Batken20(0.05)53(0.14)37(0.1)18(0.05)5(0.01)75(0.2)
Chui11(0.03)58(0.15)26(0.07)9(0.02)8(0.02)79(0.2)
Issykul30(0.06)56(0.12)25(0.05)24(0.05)24(0.05)71(0.15)
Jalalabad4(0.02)49(0.22)12(0.05)11(0.05)0(0)50(0.22)
Naryn0(0)0(0)0(0)0(0)0(0)1(1)
Osh11(0.02)111(0.22)42(0.08)6(0.01)3(0.01)53(0.11)
Talas9(0.04)38(0.19)12(0.06)8(0.04)3(0.02)36(0.18)
853651547643365
Table1:DescriptiveStatisticsfor’StoryInvolves’
26

28. Structural Topic Modelling
The purpose of handcoding a body of text, is to construct a
researcher-defined structure over what is otherwise unstructured
data, so that massive volumes of information can be distilled
down to scale that is digestible for human comprehension. How
this data is then “digested” will vary across quantitative and
qualitative traditions and by the question at hand, but the goal
is always to sift sense from noise by imposing structure.
In selecting certain bits of information to be coded whilst dis-
missing others, the coder is subjectively classifying what is “data”
and what is mere “information”, in terms of a specific research
question. Thus ontologically, data never exists independently
in its own right, only ever in relation the question at hand, in
other words it is created by the coder rather than simply located
(Schwartz-Shea & Yanow, 2012). Qualitative methodologies come
with a range of heuristics and norms for dealing this subjectivity,
tending towards either controlling it out of the design, or ac-
knowledging its inevitability and be as open and reflexive about
it as possible, depending one’s philosophical position - but either
way subjectivity it remains.
Self-signification with SenseMaker®, understood as “self-coding”
is no less subjective, but this is less of problem when the par-
ticipant’s own understandings and experiences are what is sub-
stantively interesting to us, rather than some objective truth
contained in within them. Bias is still introduced however, vis-à-
27

29. vis the framework of signifiers designed a priori by the researcher.
This ceases to be a problem when it comes to the narratives,
where the researchers only input is the design and delivery of
the prompting question. But now we have come full circle - in
narrative form, with not a researcher in sight and data white
as snow, but there is no intelligible structure to it by which we
might understand several hundred, or several thousand at once.
Fortunately, analysis of unstructured (in the quantitative sense)
data has evolved rapidly over the past decade. Probabilistic
topic modelling has emerged as a popular collection of methods
for inferring the “hidden” thematic structure of a corpus of
documents, without requiring any prior information besides the
documents themselves. The most ubiquitous of these, Latent
Dirichlet Allocation (LDA), formally assumes that 1) a pre-
specified number of topics (K) exist in the corpus, and that a
topic (βk) is formally defined as a distribution over the fixed
vocabulary (of the corpus). LDA assumes that documents in
the corpus are generated via the following process (Blei, Ng, &
Jordan, 2003):
1. Choose the length of the document N Poisson(ξ)
2. Choose the proprotions of the document in each topic
θ Dirichlet(α)
3. Then, for each of the N words in a document (d).
a. Choose a topic (k) multinomial(θ)
28

30. b. Choose a word (w) p(wn|k, β), a multinomial
probability conditioned on the topic.
Thus the task at hand is to reverse engineer this process, such
that the hidden structure of the corpus - supposedly produced
by the imagined “generative” process above - is revealed. The
topics, are thus understood as latent variables, that existed a
priori and simply need to be discovered.
Computing the posterior distribution of the latent variables
presents us with an intractable integration problem, because
the topic distribution (θ) and the parameters (β) are coupled
(Dickey, 1983). To deal with this, LDA uses a Variational
Expectation-Maximization algorithm to optimize a lower bound
on the marginal log likelihood (the probability of the data given
(α) and (β)) (Blei, Ng, & Jordan, 2003).
In other words, Variational-EM is an iterative process, alternating
between using topical content to update the topical composition
of the documents, and using the topical makeup of the documents
to update the content of the topics. The algorithm continues
until the relative change in the lower bound with each EM step
has become so small as to reach a pre-defined level of tolerance,
and we can be sufficiently certain of having approximated the
distribution of the latent variables (topics) accurately (Roberts,
Stewart, & Tingley, 2016).
To make a useful comparison to the insights of SenseMaker® and
the National Consultation, a topic model capable of analysing the
29

31. relationships between the topics and the signifiers was necessary.
The National Consultation went to great pains to gather data
from a descriptively diverse cross-section of Kyrgyz society, whilst
SenseMaker®’s principal analytic components revolve around
unsupervised comparison of the story-specific and respondent-
specific variables. Structural Topic Modelling (STM), is a recent
variation on the standard LDA algorithm which includes a modi-
fication that allows the topic proportions to vary over observed
metadata (Roberts M. E., et al., 2014).
Topic modelling is a remarkably automated method for deriving
structure in textual data, but is not entirely without priors.
Chiefly the number of topics must be pre-specified by the user,
a choice for which there is no hard and fast rule, but there
are a number of heuristics. The stm package for R includes a
function searchK() which will recursively run stm models with
different values for the number of topics (K), and provide some
diagnostics with which come to a decision about optimal (K).
In a general sense, the optimal (K) is that where the topics are
“best defined”. Formally, (Roberts M. E., et al., 2014) have taken
this to mean exist in two measures, firstly, of semantic coherence,
meaning that a high-probability words tend to co-occur within
documents, and exclusivity, in so far as high-probability words
in a given topic are unlikely to occur in the top words of other
topics. These criteria are broadly analogous to (Gerring, 2001)’s
“consistency” and “differentiation” for assessing the validity of
concepts in empirical social science, and also speak to the more
30

32. general notion of external and internal validity.
As an implementation of LDA, STM attempts to infer the unob-
served structure of the data, assuming it was “generated” from
a random Dirichlet distribution. One problem that arises from
this, is that the distribution, having (K) dimensions (topics), will
therefore possess a multitude of local modes, rather than being
globally convex. As such, the EM algorithm may become stuck in
one of these local modes, thus the solution of LDA is sensitive to
the starting values of the random allocation (Roberts, Stewart, &
Tingley, 2016). The stm package offers to approaches to dealing
with this. The first, is simply to run the model recursively, and
see how stable the posterior distribution is over different values
of initialisation, from which we can derive of probabilistic level
of certainty about having accurately estimated it.
The second, is by using an experimental implementation of
spectral learning to place the starting values in an optimal region
of the parameter space at a very low computational cost (Arora, et
al., 2013). The collapsed Gibbs sampler in LDA means LDA itself
can be used to perform the same task, but because LDA is itself
multimodal, whereas spectral algorithms lead to a deterministic
solution, using the former, the model only needs to run once.
Spectral algorithms can in fact be be used as an alternative to
the LDA algorithm for fitting the model itself, but has been
shown to yield poor performance on corpuses smaller than 13,000
documents (Roberts, Stewart, & Tingley, 2016). Using LDA
to fit the STM but a spectral algorithm to choose the starting
31

33. values, is a hybrid solution offered by the stm package that neatly
balanced computational costs against the need to optimise the
lower bound, and was used to identify the a range of potential
optimal values for (K) and to fit the final stm models themselves.
However, though these metrics have held-up well in comparison
to human evaluation of subjects (Chang, Boyd-Graber, Wang,
Gerrish, & Blei, 2009), they should not entirely replace the
researcher’s judgement (Roberts M. E., et al., 2014). In this
implementation, searchK() was run on with a range of $(K =
5,.,50), and the resulting models were narrowed down according
to two metrics. The first, was the residuals of each model, which
(Taddy, 2012) has demonstrated are (σ2
= 1) approaching “true”
(K). The second, as recommended by (Roberts M. E., et al.,
2014) was to plot a coherence-exclusivity frontier, and choose
the outermost values of (K). Having identified values of 21,
23, 24, 25 and 26, the top ten most representative stories, and
top seven most probable words from each topic, for each model
were examined by hand. In addition, a score between one and
seven was given to each topic in each model, corresponding
the number of most probable words that could be intuitively
“chained” together as being semantically related. The mean of
these scores for each model was observed, but not taken as strictly
prescriptive - this exercise was designed simply to structure the
process of manual examination mentally, not dictate it.
The stm package contains some downstream functions for plotting
and visualising the data, but as a package in relative infancy these
32

34. did not always prove particularly practical. Topic prevalence
was initially analysed using plot.STM, to garner some general
insights, but with relatively weak confidence intervals and in-
sensitivity to the optional ridge prior, after a point it became
more expedient to extract the topic proportions matrix from the
model output, and implement some of R’s more familiar instru-
ments. Model subset selection was implemented to whittle down
a messy predictor space and improve model performance. OLS
regression of six topics produced useful insights and were retained
(girls/marriage, youth, uzbekistan/citizenship, election/voting,
elections/politics, transport/infrastructure). Though a greater
range of relationships were modelled and thus more interesting
phenomena located, these models continued to prove insensi-
tive to ridge regression and lasso, thus ordinary least squares
remained preferred.
33

35. Data Analysis
0.00 0.02 0.04 0.06 0.08 0.10 0.12
Top Topics
Expected Topic Proportions
Topic 19: uzbekistan/citizenship
Topic 26: school
Topic 14: elections/politics
Topic 10: utilities
Topic 13: tajikistan/women
Topic 16: university/corruption
Topic 23: agriculture/trade
Topic 2: <Non discernable>
Topic 5: trade/kazakhstan
Topic 11: finance/agriculture
Topic 21: transport/infrastructure
Topic 1: kyrgyzstan/emigration
Topic 8: law & order
Topic 3: nation/peace
Topic 18: family/relations
Topic 22: girls/marriage
Topic 7: youth
Topic 20: education
Topic 17: rural
Topic 24: jobs/russia
Topic 4: election/voting
Topic 25: new facilities
Topic 9: city/infrastructure
Topic 12: family/marriage
Topic 6: water
Topic 15: dependents
Because it was designed to be so comprehensive in scope and
stakeholders addressed, and because makes intuitive to use the
“reference” methodology as a control/baseline, the next section
will proceed by outlining the contributions of the National Con-
sultation in terms of three of it “global themes” for development,
then demonstrating what added value SenseMaker® and STM
can provide, if any. The first two themes were chosen on merit
of being the most prominent themes in the National Consulta-
tion, and the third because it was the only one derived from the
participants themselves.
Economic Growth and Employment
Across the National Consultation, economic growth and employ-
ment opportunities were the peoples’ most commonly discussed
34

36. and cited feature important to “the future they want”. Women
from rural areas cited this more commonly than any other demo-
graphic (52%). Reasons this theme was cited tended to emphasise
negative social factors that affect the entire country, “resulting in
a widespread social mistrust and a sense of social insecurity” (UN
in the Kyrgyz Republic, 2013, p. 6). Among these, was a need
for development that would “raise all boats”, a need to improve
living standards, to address state inefficiency and the corruption
pervades the delivery of basic services. People also expressed
concern for the high rate of foreign debt and the depressive
effect it had on individual economic wellbeing, that access to
socio-economic opportunities were not equally distributed among
citizens, a that there was a need for to attract foreign investment
and cooperate with regional partners.
When asked what ought to be done, people specified “progressive
and necessary” reforms to address corruption, improve the hu-
man resource capacity of the civil service, develop “high ethical
and moral qualities amongst the people” and strengthen inter-
national cooperation. Asked to identify who was most capable
of action in this regard, respondents identified the legislative
and executive authorities, the general populace, businesses, and
the international community. Most said of themselves there was
nothing they could do personally, though a few said they could
work to contribute.
“Taken together a picture begins to emerge that
35

37. strongly links improvements in good governance and
mechanisms for empowerment with macro and mi-
croeconomic improvements that would be equitably
allocated throughout society of Kyrgyzstan.” (UN in
the Kyrgyz Republic, 2013)
Interestingly, the variable best approximated to this theme,
“Story involves - business/trade” appeared in 7.91% of all stories,
surpassing only “cooperation” as a prevalent theme. Correspond-
ing with only 79 and 43 respectively this probably explains why
these were also the only two topics not to demonstrate any sta-
tistically significant relationships with the demographic variables
whatsoever. Initially this extreme differential between the two
methods raised an important possibility - one might speculate
that the nature of research setting was conditioning responses.
In other words, in the context of a focus-group, formally organ-
ised by the government and participating NGOs, respondents
felt more inclined to speak about development as a macro-level
phenomenon, as national economic progress, in other words, in
the terms of those facilitating the discussion.
Whilst an important possibility, STM analysis suggested
a more nuanced picture. The topics labelled “dependents”
and “jobs/russia” were the 1st and 7th most prevalent topics
respectively. Analysis of the 10 most representative stories
in dependents showed that this topic often related difficulties
involving providing for dependent family members to job
36

38. opportunities and job insecurity. Jobs/Russia, as one might
expect, generally covers economic migration, remittances, and
lack of domestic opportunity as a push factor thereof. “new
facilities” and “city/infrastructure” also placed 4th and 7th,
both of which routinely referenced a need or instance of new
investment in local communities (the latter emphasising roads
& waste disposal). Thus the underrepresentation of economic
issues in the self-signification of the narratives perhaps reflects
response set bias - whilst people are concerned about the
economy and jobs, business/trade fails to operationalise this
sentiment very well.
Breakdown of topic prevalence by gender also revealed that
“utilities”, “finance/agriculture”9
and “trade/kazakhstan” were
overwhelming discussed by men, whilst the most prevalent topic
overall, dependents, was also the topic most heavily weighted
towards women.
State Administration, Governance, Democ-
racy
The second most prominent theme in the National Consultation,
and possibly the one most inclined to overlap with others, was
the administration of the state - specifically the flaws therein.
Most respondents made reference to the “poor performance of
9
Close inspection revealed that finance/agriculture was not so much a
topic where finance and agriculture were discussed in relation to one another,
so much this topic was capturing money and jobs in urban areas, and money
and farming in rural ones.
37

39. the government”, to corruption, and the state’s role in a “need
to develop the economy”. There was also an abstract sentiment
that “the system should be more socially oriented or people
oriented.” Some suggested greater regulation of state salaries
and the improvement of international relations should also be
a priority of the state. When asked what ought to be done,
people expressed a desire for a “change the political sphere
especially in the area of delegation of authority”. Mostly, this
manifested in the oft repeated desire for economic development
and to fight corruption, but also in the vaguer assertion that the
“human qualities” of state officials should be improved, and the
professionalism of the civil service increased.
No one variable in the SenseMaker® data related perfectly ap-
proximated the role of state, which is maybe unsurprising given
the multiplicity of roles the state plays in society. In fact, it might
be fair to say that all twelve Story Involves variables cover some
facet of state administration. Nevertheless, it may be instructive
to look at those stories that the respondents signified involving
“democracy” and “justice”. In Osh, the oblast (region/province)
with largest Uzbek population and the epicentre of the riots in
2010 that resulted in 400 deaths and the displacement of 400,000
predominantly Uzbek citizens, respondents were more likely to
tell stories that involved justice (8.2%) whilst being less likely
to signify democracy (-5.9%), both significant at the 10% level
(Wachtel, 2013). Participants in Jalalabad, the second focal
point of hostilities in 2010, were equally less likely (8.2%) to tell
38

40. stories that involved justice. Issykul, the most north-westerly
oblast and the most distant from the violence, reported stories
involving justice 61.1% more often. This finding congruent with
recent research that suggests that “poor governance”, “abuse of
executive power”, and “open displays of defiance by southern of-
ficials”, was a significant causative factor in the violence (Collins,
2011, pp. 159, 162). The object of the instigators (widely and
credibly reported to be composed at least in part by members of
the Kyrgyz military and police in the south), is cited as being to
make the provisional government look weak in the run up to the
election. To the extent that most Uzbek’s now “view the new
government as too inept to protect them, or worse, as complicit
in the slaughter”, this seems to have worked, and the self-signified
responses support this finding (Collins, 2011, p. 160).
Estimate Std. Error t value Pr(>|t|)
DQ1.Age40-44 -0.14 0.06 -2.35 0.02
DQ1.Ageunder 17 -0.11 0.06 -1.88 0.06
DQ5.Liveurban area 0.83 0.38 2.21 0.03
DQ6.Where.do.you.live.Issykul -0.66 0.38 -1.74 0.08
DQ6.Where.do.you.live.Jalalabad 0.08 0.05 1.74 0.08
DQ6.Where.do.you.live.Osh 0.08 0.04 1.92 0.06
DQ6.Where.do.you.live.Talas 0.09 0.05 1.90 0.06
DQ7.LanguageUighur 0.28 0.16 1.76 0.08
Table 2: Stories involving Justice
Because corruption and state administration is largely an is-
sue that crosscuts other substantive themes like education and
healthcare, discussion of corruption was distributed throughout
several STM topics. Re-running STM with a content covariate
for gender, revealed significant differences in how men and women
39

41. discussed their interactions with the state. “polic[e]” was the
most probable word in the topic labelled “law and order” as dis-
cussed by men, but was only 19th among women. On inspection
of the most representative stories, men were more inclined to
discuss instances of violence and corruption by state officials,
whereas women focused more often on the apparent absence or
impotence of the state in improving their lives. Both genders
were inclined to discuss theft in equal measure, but having ob-
served the prevalence of the word “cattl[e]” of men’s stories but
not women’s, exploration of topics containing this word using
KWIC demonstrated that theft was just one aspect of a broad
range of phenomena relating to economic loss discussed by men,
property disputes (land and livestock) being chief among them.
The same treatment of the word “rude” (20th most probable
among women) revealed unpleasant interactions with state offi-
cials of many different types (doctors, police, customs officers) -
though in fairness “rude” often related to stories in which private
citizens were the antagonists.
Ethnicity, Difference and Social Cohesion
As composite issue area, the insights of the National Consulta-
tion are drawn from three related themes: Peace and Stability;
and Values10
. With respect to the former, most respondents ex-
pressed a belief that there exists a “need to improve interethnic
10
The theme “Inequalities” might have suited this issue area well also,
but it composed a very small section of the report and would not have been
a particularly fruitful inclusion.
40

42. relations”, promote tolerance and “understanding of the fact that
we are one people.” There was an awareness of the interaction
between peace/stability and improving the quality of education,
the economy and unemployment. Interestingly, peace and sta-
bility was the theme which people were most inclined to see
themselves as capable of making a contribution to change, and
were positive about the notion of the general population taking
“more active civic positions”.
“Values” was pertinent to integrate with this issue area partly
because of its inextricable relationship with culture and difference,
and partly because it was a special theme, unique in that it was
the only one derived from the participants during the consultation
process rather than pre-specified. Participants directly intimated
that culture should be promoted in order to “ensure the future
they have is one of economic stability and peaceful harmony”,
and that this required a balance between the traditional and
“what is emerging in the culture of technology”. Values and
culture were put forwards as important themes in their own
right because there is “an observable decline in the cultural
and spiritual values of the people in general”. Some responses
indicated that this might be done by bringing “people of other
countries in like volunteers and other international workers”.
Others, emphasises the need to engage people, particularly youth,
in the formulation of development policy, highlighting a belief
that this would increase volunteerism an improve the “passive”
relationship between citizens and the state.
41

43. Only 95 respondents signified that their story involved violence,
a variable that one would assume represents a strong inverse
of peace - whilst this is substantively positive, it does make it
difficult to make statistical inferences about who is telling such
stories. Respondents aged 30-34 (-.11.0%.05), 35-39 (-.10.2%.05),
40-44 (-8.8%.1), 50-54 (-8.3%.1), and under 17 (-9.8%) were
less inclined to speak about violence, prompting one to wonder
if these stories are being told overwhelming by young people
between 17 and 29. The infrequency of the stories containing
violence but this beyond the scope of regression, but a descriptive
summary indicates that nearly half (45) of these 95 stories fell
into this age bracket. Finally, when the STM topic girls/marriage
(14) was regressed over story signification, it was disheartening
to find it positively correlated with stories about violence, but
negatively so with stories about politics, suggesting perhaps that
domestic violence is in many cases an issue raised but not one
seen as inherently “political” or “part of politics”.
0 1
0 0
17-19 68 14
20-24 99 13
25-29 113 18
30-34 84 4
35-39 150 9
40-44 71 5
45-49 65 7
50-54 63 5
55-59 50 9
60 and over 54 6
under 17 87 5
Table 3: Stories involing violence by Age
42

44. Further comparison by region and ethnicity was difficult due to
sample size, particularly with regards to ethnicity, where the
majority of stories were given by Kyrgyz and Russian respondents.
Nonetheless stories involving conflict were 9.3% more prevalent
in Talas, which borders both Uzbekistan and Kazakhstan, and
relationships with other countries was a less prevalent topic in the
capital province Chui (also on the Kazakh border), perhaps as
a consequence of being predominantly Kyrgyz and Russian and
comparatively ethnically homogenous (United Nations, 2013).
The most interesting theme revealed by STM was the topical
overlap of themes that revolved around economic necessities and
ethnic difference. Over 16% of narratives collected in Osh made
reference to topic 6 “water”, and analysis of the most representa-
tive stories revealed several instances where water was difficult to
access because control of (limited) supply was physically located
on the Tajik side of the border. This topic, along with utilities
(10), finance/agriculture (11), trade/kyrgyzstan (5) were signifi-
cantly more prevalent among men, but stories about water told
by women in rural areas often centred on the difficulty faced by
women and children in carrying clean water long distances.
These stories also tended to score highly with tajikistan (13),
which unsurprisingly was unsurprisingly spoken about the most
in Batken, with which Tajikistan shares a border, but curiously
topic prevalence was almost as high in the country’s northern
provinces Talas, Chui and Issykul. It was hoped that this might
explained by analysing topical content by regional, but in fact
43

45. the word probabilities remained remarkably consistent. Topic
prevalence though was furthered differentiated over the urban
and rural population. Residents in rural Chui spoke a great
deal about tajikistan and trade/kazakhstan, whilst urban Chui
comparatively negligent but was highly inclined to talk about
uzbekistan/citizenship (19). Given that stories self-signified as be-
ing about relations with other countries were fewer in urban Chui
at a statistically significant level, perhaps further investigation
of the significance of Uzbekistan to urban residents in the capital
province would be insightful. Finally, respondents who self-
identified as Tajik, told stories in which uzbekistan/citizenship
was 8% more prevalent.
−0.10 −0.05 0.00 0.05 0.10
Difference in topic proportions: urban vs. rural
Rural ... Urban
kyrgyzstan/emigration
<Non discernable>
nation/peace
election/voting
trade/kazakhstan
water
youth
law & order
city/infrastructure
utilities
finance/agriculture
family/marriage
tajikistan/women
elections/politics
dependents
university/corruption
rural
family/relations
uzbekistan/citizenship
education
transport/infrastructure
girls/marriage
agriculture/trade
jobs/russia
new facilities
school
44

46. −0.2 −0.1 0.0 0.1 0.2
Ethnicity/International Relations by region
Topic Proportions
Chui
Chui
Chui
Naryn
Naryn
Naryn
Osh
Osh
Osh
Jalalabad
Jalalabad
Jalalabad
Batken
Batken
Batken
Talas
Talas
Talas
−0.2 −0.1 0.0 0.1 0.2
Ethnicity/International Relations by region (urban areas only)
Topic Proportions
Chui
Chui
Chui
Osh
Osh
Osh
Jalalabad
Jalalabad
Jalalabad
Issykul
Issykul
Issykul
Talas
Talas
45

47. −0.2 −0.1 0.0 0.1 0.2
Ethnicity/International Relations by region (rural areas only)
Topic Proportions
Chui
Chui
Chui
Naryn
Naryn
Naryn
Osh
Osh
Osh
Jalalabad
Jalalabad
Jalalabad
Batken
Batken
Batken
Talas
Talas
Talas
Discussion
It should be noted at the outset, that for the various regres-
sion techniques implemented to model the relationships between
SenseMaker® signification and demographics, and STM topics
and demographics, the adjusted-R2 was consistently and ex-
tremely low. A mean of 0.03393367 was recorded for the 12
models regressing each level of the “Story Involves” variable over
demographic information. At first this seemed curious, given
many healthy estimates, standard errors and p-values, but on
reflection given the unique way in which the data was collected,
this is perhaps to be expected. Were I to walk out into the street
now, and ask a passer-by to “tell me story” rather than ask her
directly what the local council ought to do about traffic in the
town centre, her response will likely be conditioned by whether
46

48. or not her bus was time this morning, what she’d done that
weekend, or whether or not she’d had a cooked breakfast, just
as much as her income bracket, ethnicity or education.
SenseMaker® asks the researcher to abandon the idea of shepherd-
ing participants towards some purportedly useful, well-defined
response variable, and make their own one up - so that it exhibits
a high variance is perhaps unsurprising. Because we would expect
the simple everyday phenomena described above - so stochastic
and uninterpretable that statisticians would generally classify
it as noise - to have an abnormally significant effect on the re-
sponses, it is reasonable to assume that a model which explains
most of the variance in undirected human conversation is slightly
overreaching. This is not to say the modelling micronarratives
is useless, a statistically significant relationship between two
variables is still useful for an explanatory research question, even
if a predictive model would be wildly inaccurate. But is does
become difficult to reliably claim that no important predictors
have been omitted.
The greatest benefit of the exploring the narratives was perhaps
the ability to observe social life often in a more granular, anec-
dotal form. Often the National Consultation and SenseMaker®
would point to the same phenomena, but express them in very
different ways. For instance, to say that the “human qualities of
state officials” could be improved is one thing, but to hear, as
one respondent indicated:
47

49. “I work as a taxi driver. In the evening we with the
friend went taksovat. And suddenly we were stopped
by traffic police. Because of not wearing a seat belt,
they have been fined. My friend began to interfere
with them. Traffic police got angry and drove his
car. Said that he there paid the penalty. We thought
because of our nationality we have done so.”
represents a much more intimate, telling reproduction of the same
thing. In this form, we learn how racism transmitted through
a power relationship manifested in a negative experience, how
the law is used as a pretext for discrimination. The ability the
observe the causal progression of events - encounter, resistance,
confrontation - is a level of depth generally garnered only through
resource-intensive, qualitative methods such as process tracing or
ethnography. In an essay seemingly less intended for a machine
learning audience in the Harvard Business Review, (Blei, 2012)
asks the reader to “imagine searching and exploring documents
based on the themes that run through them. We might”zoom
in" and “zoom out” to find specific or broader themes; we might
look at how those themes changed through time or how they are
connected to each other“. The utility of this to the development
researcher will naturally depend on the question at hand, but the
capability to use quantitative text analysis to impose structure
on data, so that one can rapidly navigate to this level of depth
represents a significant innovation.
48

50. Though the narratives made for an excellent resource, self-
signification with SenseMaker® proved quite a delicate tool.
Significant relationships between the signifiers were often hard
to come by, and interesting ones even more so. Partly this is a
reflection of sample size, and missingness in the triads that made
them unworkable, but it also became apparent that their utility
is incredibly sensitive to the framework respondents are presented
with. When the content of the narratives is unobserved, as it is
for the researcher designing the framework, specifying questions
that make coherent sense in every context whilst keeping their
investigative teeth sharp becomes a powerful trade-off. In this
respect, the National Consultation and STM performed much
better because both methods inductively derive the structure of
the data from the data itself. One might say that SenseMaker®
permits respondents to define the distribution of the data, in
that they choose their responses, but how it is hierarchical or-
ganised remains the realm of researcher - a major caveat if one
is trying to claim “participant-driven” development research.
Perhaps allowing textual self-signification guided by a question,
would hand some liberty back to participants, though this would
likely increase the time-costs of data collection - SenseMaker®’s
flagship strength.
A final observation, raises a fascinating dilemma about the theo-
retical compatibility of topic modelling and the conceptual basis
of sustainable development. STM works by trying to separate
topics out, maximising their exclusivity. But when the content
49

51. of topics was investigated it became increasingly clear that even
seemingly unrelated topics such as water and tajikistan were in
fact inextricably related to due to unanticipated phenomena, in
this case water supplies spanning national borders. This presents
us with a conundrum. On the one hand we have found a sub-
stantively interesting phenomenon. On the other, the decision to
use an algorithm to separate these topics is undermined by the
simple fact, that they are not separate, they are fundamentally
interrelated. How does one decide whether the topic labelled
“university/corruption” is poorly defined, or the prevalence of
stories describing bribes in the education system make it a sensi-
ble, coherent whole? It might suit our intuition better that water
and infrastructure are “one issue” and it is thus “semantically
coherent” that they subsumed under the same topic by STM,
but if the same thing occurred for water and tajikistan we might
be inclined to say these are “two related issues”, but separate.
We might be right in a sense, if foreign affairs is also related
tajikistan but not to water, but this simply brings us round to
the same problem - no one topic is an island unto itself.
From this perhaps we should take two things. Firstly, that sepa-
rating out data in maximally-exclusive topics is antithetical to an
understanding of development which by contrast, emphasises the
complex and dynamic interplay of different development themes.
But secondly, and on a brighter note, it was the failure of the
STM model to generate these “exclusive” topics, that brought
the interesting phenomenon in the data to our attention. By
50

52. relying on structure in the data that is mathematical rather than
semantic in nature, our assumptions to the semantic relationships
between topics are more likely to be challenged. In this respect,
a flawed STM - one that refuses to conform to our preconceived
notions of exclusivity and semantic coherence - might be as useful
to us as a successful one.
Conclusion
As a tool of data collection, SenseMaker® represents a promising
asset for development research. Whilst the self-signification is
element is perhaps stronger as a concept at this stage than it is in
implementation, the micronarratives are a rich and flexible data
source that promises to make the admirable notion of “bottom-
up” development research into a feasible reality. The utility of
narrative data in itself will depend on the research question at
hand, but the reduction of time, access and financial barriers
are difficult to ignore under any circumstances. In many ways
STM provides the analytical depth that SenseMaker® lacks on
its own, though the focus on topic prevalence over topic content
(to a great extent a computational necessity) slightly diminishes
its inferential power.
51

53. Bibliography
Arora, S., Ge, R., Halpern, Y., Mimno, D., Moitra, A., Sontag,
D., & Wu, Y. (2013). A Practical Algorithm for Topic Modeling
with Provable Guarantees. Proceedings of The 30th International
Conference on Machine Learning, (pp. 280-288).
Autesserre, S. (2012). Dangerous tales: dominant narratives on
the Congo and their unintended consequences. African Affairs,
202-22.
Bell, K. E. (2013). Raising Africa?: Celebrity and the Rhetoric
of the White Saviour. Journal of Multidisciplinary International
Studies, 10(1).
Blei, D. (2012). Probabilistic Topic Models. Journal of the
Association of Computational Machinery, 55(4), 77-84.
Blei, D., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet
Allocation. Journal of Machine Learning Research, 3, 993-1022.
Chang, H.-J. (2003). Kicking Away the Ladder: Infant Indus-
try Promotion in Historical Perspective. Oxford Development
Studies, 31(1), 21-32.
Chang, J., Boyd-Graber, J., Wang, C., Gerrish, S., & Blei, D.
E. (2009). Reading Tea Leaves: How Humans Interpret Topic
Models. Advances in Neural Information Processing Systems,
(pp. 288-296).
Clark, J. (1992). Policy influence, lobbying and advocacy. In
M. Edwards, & D. Hulme, Making a Difference: NGOs and
52

54. Development in a Changing World. London: Earthscan.
Collins, K. (2011). Kyrgyzstan’s Latest Revolution. Journal of
Democracy, 22(3), 150-164.
Dickey, J. (1983). Multiple hypergeometric functions: Probabilis-
tic interpretations and statistical uses. Journal of the American
Statistical Association, 78, 628-637.
Easterly, W. (2006). The White Man’s Burden: Why the West’s
Efforts to Aid the Rest Have Done So Much Ill and So Little
Good. New York: Basic Books.
Easterly, W. (2013). The tyranny of experts : economists, dic-
tators, and the forgotten rights of the poor. Oxford: Oxford
University Press.
Edwards, M. (1993). ‘Does the doormat influence the boot?’:
Critical Thoughts on UK NGOs and International Advocacy.
Development in Practice, 3(3), 163-175.
Fukuda-Parr, S. (2004). Millenium Development Goals: Why
They Matter. Global Governance, 10(4), 395-402.
Gerring, J. (2001). Social Science Methodology: A Unified
Framework. Cambridge: Cambridge University Press.
Halperin, S., & Heath, O. (2012). Political Research: Methods
and Practical Skills. Oxford: Oxford University Press.
Kramp, K. M. (2004). Exploring life and experience through
narrative inquiry. In K. deMarrais, & S. Lapan, Foundations for
research: methods of inquiry in education and the social sciences
53

55. (pp. 103-12). Mahwah, New Jersey: Lawrence Erlbaum.
Lynam, T., & Fletcher, C. (2015). Sensemaking: A Complexity
Perspective. Ecology and Society, 20(1), 65-79.
Muchina, O. (1995, October-December). Qualitative indicators
of transformational development. Together, 3-5.
Nyamugasira, W. (1998). NGOs and Advocacy: How Well Are
The Poor Represented? Development in Practice, 8(3), 297-308.
Pitkin, H. (1972). The Concept of Representation. London:
Cambridge University Press.
Ravallion, M. (2008). Doing Impact Evaluation No. 11: Evalua-
tion in the Practice of Development. World Bank.
Rehfeld, A. (2006). Towards a General Theory of Political
Representation. Journal of Politics, 68(1), 1-21.
Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-
Luis, J., Gadarian, S. K., . . . Rand, D. G. (2014). Structural
Topic Models for Open-Ended Survey Responses. American
Journal of Political Science, 58(4), 1064-1082.
Roberts, M., Stewart, B. E., & Tingley, D. (2016). Navigating
the Local Modes of Big Data: The Case of Topic Models. In
Data Analytics in Social Science, Government, and Industry.
New York: Cambridge University Press.
Rubenstein, J. (2014). The Misuse of Power, Not Bad Represen-
tation: Why It’s Beside The Point That Nobody Elected Oxfam?
The Journal of Political Philosophy, 22(2), 204-230.
54

56. Said, E. (1978). Orientalism. London: Routledge and Kegan
Paul.
Schreier, M. (2012). Will Qualitative Content Analysis Work For
Me? Decision Aids. In Qualitative Content Analysis In Practice
(pp. 37-57). London: Sage.
Schwartz-Shea, P., & Yanow, D. (2012). Interpretive Research
Design: Concepts and Processes. New York: Routledge.
Seay, L. (2012). What’s Wrong with Dodd-Frank 1502? Conflict
Minerals, Civilian Livelihoods, and the Unintended Consequences
of Western Advocacy (Working Paper). Centre for Global Devel-
opment.
Sen, A. (1999). Development as Freedom. Oxford: Oxford
University Press.
Snowden, D. (2002). Complex Acts of Knowing: Paradox and
Descriptive Self-Awareness. Journal of Knowledge Management,
6(2), 100-111.
Snowden, D., & Boone, M. (2007). A Leader’s Framework for
Decision Making. Harvard Business Review, 85(11), 68-76.
Stacey, R. D. (2001). Complex responsive processes in organiza-
tions: Learning and knowledge creation. New York: Routledge.
Stiglitz, J. (2001). Financial Markets and Development. Oxford
Review of Economic Policy, 5(4), 55-68.
Taddy, M. A. (2012). On Estimation and Selection for Topic
Models.
55

57. UN Development Group. (2012). Post-2015 Development
Agenda: Guidelines for Country Dialogues. Retrieved from
beyond2015.org: http://www.beyond2015.org/sites/default/
files/Post%202015%20Guidelines%20ENG1.pdf
UN in the Kyrgyz Republic. (2013, October 2). Findings of
the Post-2015 Development Agenda National Consultations in
the Kyrgyz Republic. Retrieved from UN in the Kyrgyz Repub-
lic: http://www.un.org.kg/index2.php?option=com_resource&
task=show_file&id=23133
United Nations. (2013). Third Report on Progress Towards
Achieveing the Millenium Developmetnt Goals. Bishkek: United
Nations in the Kyrgyz Republic.
Wachtel, A. (2013). Kyrgyzstan between democratization and
ethnic intolerance. Nationalities Papers: The Journal of Nation-
alism and Ethnicity, 41(3), 971-986.
Wagner, W., & Hayes, N. (2005). Everyday discourse and com-
mon sense. The theory of social representations. New York:
Palgrave, New York.
56

58. Appendices
Estimate Std. Error t value Pr(>|t|)
DQ1.Age20-24 -0.27 0.07 -3.72 0.00
DQ1.Age25-29 -0.16 0.07 -2.33 0.02
DQ1.Ageunder 17 -0.22 0.07 -2.98 0.00
DQ2.Gendermale 0.06 0.03 1.80 0.07
DQ6.Where.do.you.live.Talas -0.13 0.06 -2.05 0.04
Table 4: Stories involving Security
Estimate Std. Error t value Pr(>|t|)
DQ2.Gendermale -0.05 0.03 -1.90 0.06
DQ6.Where.do.you.live.Jalalabad 0.18 0.05 3.49 0.00
DQ6.Where.do.you.live.Osh 0.10 0.05 2.10 0.04
Table 5: Stories involving Education
Estimate Std. Error t value Pr(>|t|)
DQ1.Age30-34 -0.11 0.05 -2.37 0.02
DQ1.Age35-39 -0.10 0.04 -2.50 0.01
DQ1.Age40-44 -0.09 0.05 -1.87 0.06
DQ1.Age50-54 -0.08 0.05 -1.72 0.09
DQ1.Ageunder 17 -0.10 0.05 -2.16 0.03
DQ6.Where.do.you.live.Osh 0.06 0.03 1.93 0.05
Table 6: Stories involving Violence
−0.10 −0.05 0.00 0.05 0.10
Difference in topic proportions by Gender
Female ... Male
kyrgyzstan/emigration
<Non discernable>
nation/peace
election/voting
trade/kazakhstan
water
youth
law & order
city/infrastructure
utilities
finance/agriculture
family/marriage
tajikistan/women
elections/politics
dependents
university/corruption
rural
family/relations
uzbekistan/citizenship
education
transport/infrastructure
girls/marriage
agriculture/trade
jobs/russia
new facilities
school
57

59. Estimate Std. Error t value Pr(>|t|)
DQ1.Age40-44 -0.14 0.06 -2.35 0.02
DQ1.Ageunder 17 -0.11 0.06 -1.88 0.06
DQ5.Liveurban area 0.83 0.38 2.21 0.03
DQ6.Where.do.you.live.Issykul -0.66 0.38 -1.74 0.08
DQ6.Where.do.you.live.Jalalabad 0.08 0.05 1.74 0.08
DQ6.Where.do.you.live.Osh 0.08 0.04 1.92 0.06
DQ6.Where.do.you.live.Talas 0.09 0.05 1.90 0.06
DQ7.LanguageUighur 0.28 0.16 1.76 0.08
Table 7: Stories involving Justice
Estimate Std. Error t value Pr(>|t|)
DQ1.Ageunder 17 -0.24 0.07 -3.24 0.00
DQ6.Where.do.you.live.Jalalabad 0.22 0.06 3.69 0.00
DQ6.Where.do.you.live.Osh 0.20 0.05 3.72 0.00
DQ6.Where.do.you.live.Talas 0.13 0.06 2.06 0.04
DQ7.LanguageTatar 0.35 0.19 1.86 0.06
Table 8: Stories involving Human Rights
−0.10 −0.05 0.00 0.05 0.10
Difference in topic proportions: urban vs. rural
Rural ... Urban
kyrgyzstan/emigration
<Non discernable>
nation/peace
election/voting
trade/kazakhstan
water
youth
law & order
city/infrastructure
utilities
finance/agriculture
family/marriage
tajikistan/women
elections/politics
dependents
university/corruption
rural
family/relations
uzbekistan/citizenship
education
transport/infrastructure
girls/marriage
agriculture/trade
jobs/russia
new facilities
school
58

60. Estimate Std. Error t value Pr(>|t|)
DQ6.Where.do.you.live.Talas -0.10 0.05 -2.15 0.03
DQ7.LanguageUighur 0.30 0.15 1.95 0.05
Table 9: Stories involving Conflict
Estimate Std. Error t value Pr(>|t|)
DQ1.Age45-49 0.08 0.04 1.94 0.05
DQ6.Where.do.you.live.Chui -0.06 0.03 -1.69 0.09
DQ7.LanguageTajik 0.20 0.12 1.70 0.09
Table 10: Stories involving Int. Relations
−0.05 0.00 0.05 0.10 0.15 0.20
Infrastructural Concerns by Region
Topic Proportions
Chui
Chui
Chui
Naryn
Osh
Osh
Osh
Jalalabad
Jalalabad
Jalalabad
Batken
Batken
Batken
Issykul
Issykul
Issykul
Talas
Talas
Talas
59

61. Estimate Std. Error t value Pr(>|t|)
DQ6.Where.do.you.live.Jalalabad 0.14 0.06 2.28 0.02
DQ6.Where.do.you.live.Osh -0.13 0.05 -2.45 0.01
Table 11: Stories involving Other
−0.05 0.00 0.05 0.10 0.15 0.20
Ethnicity/International Relations by region
Topic Proportions
Chui
Chui
Chui
Naryn
Osh
Osh
Osh
Jalalabad
Jalalabad
Jalalabad
Batken
Batken
Batken
Issykul
Issykul
Issykul
Talas
Talas
Talas
−0.10 −0.05 0.00 0.05 0.10
Difference in topic proportions: urban vs. rural
Rural ... Urban
kyrgyzstan/emigration
<Non discernable>
nation/peace
election/voting
trade/kazakhstan
water
youth
law & order
city/infrastructure
utilities
finance/agriculture
family/marriage
tajikistan/women
elections/politics
dependents
university/corruption
rural
family/relations
uzbekistan/citizenship
education
transport/infrastructure
girls/marriage
agriculture/trade
jobs/russia
new facilities
school
60

62. −0.2 −0.1 0.0 0.1 0.2
Ethnicity/International Relations by region
Topic Proportions
Chui
Chui
Chui
Naryn
Naryn
Naryn
Osh
Osh
Osh
Jalalabad
Jalalabad
Jalalabad
Batken
Batken
Batken
Talas
Talas
Talas
−0.2 −0.1 0.0 0.1 0.2
Ethnicity/International Relations by region (urban areas only)
Topic Proportions
Chui
Chui
Chui
Osh
Osh
Osh
Jalalabad
Jalalabad
Jalalabad
Issykul
Issykul
Issykul
Talas
Talas
61

63. −0.2 −0.1 0.0 0.1 0.2
Ethnicity/International Relations by region (rural areas only)
Topic Proportions
Chui
Chui
Chui
Naryn
Naryn
Naryn
Osh
Osh
Osh
Jalalabad
Jalalabad
Jalalabad
Batken
Batken
Batken
Talas
Talas
Talas
62