-Masters in Sociology-Market research and for profit gov’t program evaluations-Cancer Care Ontario since 2006, Cancer Services and Policy Research Unit-Material presented here relates back to my work at Cancer Care Ontario-Novel data collection method that we thought may of interest
-Simple enough question, right?-Picture a diagram in your head-No right answer!
Do any of these look like what you pictured?Is there anything here that you wouldn’t consider a diagram?These five pictures have all been called a “diagram” in publications
Only two of these we’re calling a diagram for this discussion
This is what I’m referring to when I say “diagram”…something like these.
I don’t want to dwell on this aspect too much. But historically in some disciplines there has been a lot of discussion on the different types of diagrams and categorizing themBroad and inclusive framework so we can have that interdisciplinary dialogue and learn from each otherAnderson (1997) stated more generally: "diagrams are pictorial, yet abstract, representations of information, and maps, line graphs, bar charts, engineering blueprints, and architects' sketches are all examples of diagrams, whereas photographs and video are not". On the other hand Lowe (1993) defined diagrams as specifically "abstract graphic portrayals of the subject matter they represent".
-Diagrams sit on a visual–verbal spectrum -pictures, diagrams, lists-While there may be overlap, we discuss these as ideal types.Muriah Umoquit, Peggy Tso, TundeVarga-Atkins, Mark O'Brien, Johannes Wheeldon. Diagrammatic elicitation: defining the use of diagrams in data collection, International Journal of Qualitative Methods.
I don’t want to dwell on this slide too muchSummarizes the differences and similarities between three categories of graphic communications used as qualitative data collection approaches: drawings, diagrams and tables & lists. I just want to emphasize that it borrows the best of both sides: simplifies complex ideas, can have some abstraction, normally some meaning to spatial arrangements, combines verbal and visualThink of your own work ----diagrams are often used in results, when you’re trying to explain complex ideas…when a table or text just won’t work.So that was your whirlwind introduction to diagrams in general
Mark Dobrow (PhD) and Paul Ritvo (PhD) are based at Cancer Care Ontario, Louise Lemieux-Charles (PhD) and Walter Wodchis (PhD) are based at the Department of Health Policy, Management and Evaluation at the University of Toronto and David Urbach (MD, MSc) is based at Toronto Western Hospital, University Health Network. Dr Dobrow has expertise in conducting case studies and qualitative data analysis and his research interests relate to governance and accountability in health policy contexts. Dr Lemieux-Charles’ research interests relate to performance management and organizational learning with recent experience studying the relationships of mid-level performance indicators to managing healthcare organizational performance at the strategic level. DrRitvois a behavioural scientist with expertise related to the behavioural component of clinical decision-making and in assessing care quality from a quality of life perspective. DrUrbachis a health services researcher and surgeon who studies the optimization of clinical decision-making, with a focus on new surgical procedures and health technologies. DrWodchishas expertise in health economics and finance with research interests in the impact of financing arrangements on the quality of care. Further background information on each applicant’s relevant expertise and experience are described in the Summary of Progress section
-Conducted the document analysis looking at lot of the formal contracts-Interviews: working on interview guide-Who do we ask about clinical accountability?-One problem was who we were asking -------64 interviews: oncologists, clinical leaders,nurses, therapists, regional vice presidents -each would have a different view of this -sensitive ---therapist or nurse talking about a doctor, payment and incentives -Physician accountability relationships = complex, how do you begin to answer this? List of potential prompts hugs
Collected three diagrams from each participants and had a verbatim transcript
-With out compromising our data ----so keeping in mind it was not randomized-mini study within a study-Looking at these two types of data collection methods –head-to-head to find out which one was more efficient and which one was more effective at collecting the data-Cited four times but accessed about 4 thousand times
-Again, keeping in mind that we didn’t randomize the application 1.-editing a diagram had less questions2.-seemed to enjoy this more ---less noted discomfort and 3.said they likes it4. tie. We collected data from everyone. Those that didn’t make a diagram made a list (remember that visual/verbal spectrum)5. Took less time to draw one then to look at our diagrams and edit ----yes, there were two diagrams but given position they normally related to one or the other (macro or mirco)-
-What about effectiveness?-We looked at number of relationships, number of links (more links then relationship =hieratical/complex structure)-Tie!-Depends what you’re looking for
-No clear winner-Do what validation of earlier stages and comments on specific things? Or is your research project more exploratory and you’re looking for an original visuals
-Double screening, two researchers reviewed titles and abstracts-Used Reference Manager -Almost three thousand references identified, after the removal of duplicates
-Plays well with other methods-Focus groups….on paper, with sticky notes, local material (stones, string, etc.)-One-on-one interview-Distributing to a whole class room and they do the exercise in mass-Colleagues who are working on applying this method electronically…CCO: Disease Pathway Mangement
(not in systematic review but good example of how adaptable using diagrams is and how it can be used with a vast amount of different methods)Stephen Sillett, Jennifer Jimenez, Muriah Umoquit. (2011). “Socio-Drama Topography (SDT) as a Participatory Method with Youth in South Africa” in Creative Arts in Research for Community and Cultural Change, Cheryl L McLean, Robert Kelly, eds. DetseligTemeron Press.
-diabetic patients knowledge before and after patient education-asked to draw out their knowledge around the term “sugar”-use arrows, circle concepts and connect((Mind Map))
-conceptual representations of the flu-Qualitative analysis of this –count how many cross-linkis there are, how many levels of the hierarchy, how many concepts((Concept Map))
-Second-year med student life-cycle diagram-doesn’t stand on it’s own-used to illustrate her pointsBF= boyfriendCW= early physician role models-Used as a recall aid or to help with the thinking process
Analysis is highly dependent on type of diagram-May have a mix of both quanlative and qualitativeStructural scoring: e.g. link between two concepts is 10 points, invalid link is 0 points, example is 1 point, invalid Relational: importance determined by the research or comparison to a diagram created by an expert
-Sorting into 12 categories-Do different groups have different thought processesCole: looking at mental models of students, differences of different disciplines and grade, can then align with their library thesauri
Karen Pollock who worked with Jenna Hartel from the Faculty of Information at UofT, they collected 137 iSquares from 6 graduate classes. “What is Information?” Identified 12 concepts.
Mike, human geography, UK White male University Professor researching condom use with married women in rural Africanvillages -Diagrams diffuse the tension (distances self….not their husband), overcome language and cultural barriers -use local materials: rocks, string, strips of paper, bottle caps-note that they put pebbles on each one ---rating of the different items on diagram -Same is true if groups are using large pieces of paper or a white board
Diagrammatic Elicitation:Using diagrams as a data collecton method
Diagrammatic Elicitation: USING DIAGRAMS AS A DATA COLLECTION METHOD Muriah Umoquit, MA Research Associate Clinical Decision Making and Health Care
DIAGRAMMATIC ELICITATION: OUTLINE Thinking about diagrams Participant vs research led diagrammatic elicitation Best practices and examples GOAL: Basic introduction and make this data collection method assessable! 2
A diagram is...while a concept mapis... Diagramsand a mind map is... Diagrams Network maps Mind maps Concept maps Geographical maps Organisational charts Tables Drawings
WHAT IS A DIAGRAM?Open-endedness Linear flow Visual Verbal Adapted and visualised from Banks. Visual Methods in Social Research. (2001) Umoquit et al. Diagrammatic elicitation: defining the use of diagrams in data collection (under review at the International Journal for Qualitative Methods)
WHAT IS A DIAGRAM? Drawings Diagrams Tables & Lists Purpose Captures essence Ability to simplify Highly structured ideas and (a salient feature) variables complex ideas Abstraction Primarily abstract Some level of Concrete abstraction Structure Undirected in representation More direct in terms of Most direct in terms of representation (internal) representation; structure inherent where pre-agreed notation is used Notation No pre-set or agreed May use pre-set or Uses pre-set or agreed conventions or notations conventions or notations agreed conventions or notations Spatial Spatial arrangements may Spatial arrangements of Spatial arrangements of signs or may not carry meaning carries meaning Meaning signs usually carries meaning Verbal/visual Visual signs dominate over Composite of both Verbal signs dominate over visual verbal spectrum visual and verbal Table adapted from Varga-Atkins and O’Brien (2009) and Engelhardt (2002) Umoquit et al. Diagrammatic elicitation: defining the use of diagrams in data collection (under review at the International Journal for Qualitative Methods)
EXAMPLE: DIAGRAMMING CLINICALACCOUNTABILITY Clinical Accountability and Value-Based Purchasing: Linking Funding to Quality of Care • Mark Dobrow, Louise Lemieux- Charles, Paul Ritvo, David Urbach, Walter Wodchis • Document analysis, interviews, focus groups
“We would like to examine how youperceive clinical accountabilityrelationships in the Ontario cancersystem. Where do you view yourself inrelation to the individuals, groups andorganizations that you are accountable toor are accountable to you?”
Participant-led diagrams Researcher-led diagrams• Participants need guidance • Accessible and understood and instruction by participants • Member validation of earlier• Original diagram created, great for exploratory means research stages • Increased focus on• Unbiased view: no prompts researcher identified issues 17• Valuable original visual created • Verbal comments including more detail, insights and examples
EXAMPLES FROM SYSTEMATIC REVIEW A multidisciplinary systematic review of the use of diagrams as a means of collecting data from research subjects • Variable terminology across and even within fields
SYSTEMATIC REVIEW 13 indexed sources (EMBASE; HealthSTAR; Medline; Cumulative Index to Nursing and Allied Health Literature; GEOBASE; InfoTrac Environmental Issues & Policy eCollection; ProQUEST Dissertations; Design and Applied Arts Index; Education Resources Information Center; International Bibliography of the Social Sciences; PsychINFO; Public Affairs Information Services; and Social Science) Citation Index) General search tool Google scholar, Google books Input from 35 experts English Used diagramming techniques in the data collection Were evaluative or reflective about the data collection process and/or analysis No publication date or publication type restrictions (dissertations, unpublished work) 19
EXAMPLE OF BEFORE AND AFTER Pre-Intervention Post-Intervention 23
EXAMPLES OF COMPARING BETWEEN GROUPS 5th Grade 8th Grade 11th Grade 24
EXAMPLE OF A LESS STRUCTURED DIAGRAM “It‟s just you are in such a weird bubble and it‟s such a weird culture. (…) something happens to you when you go through med school and it‟s just nice to have people outside of it that…can keep you grounded.” 25
WHAT IS „THE‟ DATA? = data = data e.g. Haidet et al. 2008 e.g. West et al. 2000 = data e.g. Jafri et al. 2008
ANALYSIS Counting Number of concepts, links, examples Structural scoring Weights assigned to hierarchical structures, links, etc. Relational scoring Reflects quality or importance of concepts e.g. compared to expert diagram 27
ANALYSIS Thematic and content analysis Often used with other data collection techniques Identify core themes Verification of other analysis A Visual Approach to the Perennial Question: "What is Information?" Presentation and Expert Panel at iConference, Toronto, Ontario, February 2012 29
PRESENTATION OF RESULTS Diagrams can be scanned, photographed or recreated by computer Recommend showing at least one original Variation in terminology between disciplines, relays to the reader the type and structure of diagram created or used 30
DIAGRAMMATIC ELICITATION OVERVIEW Decided on participant or researcher-led Decide on level of structure of diagram Decide on analysis Simple to adapt to topic, context Excellent addition to other data collection methods Readily accepted by participants from youth to elites 32
Diagrammatic Elicitation: USING DIAGRAMS AS A DATA COLLECTION METHOD Questions?