Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data Understandable


Published on

Presentation to two business schools on how to think about, design, execute, and narrate messy qualitative research and big data.

Published in: Education
  • Be the first to comment

Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data Understandable

  1. 1. 1MESSY RESEARCHHow to Make Qualitative DataQuantifiable and Make Messy DataUnderstandableDr. Gigi Johnson
  2. 2. Core • When to chose itQualitative • Major challenges in designIssues and analysis • How to tell messy stories for real decisions and action by business
  3. 3. 3Why • Make a decisionBusiness • Deploy resources as anResearch organization? • Convince others in organization • Rule out alternatives • Influence certain people in the company • Understand change ahead of company
  4. 4. 4How Much Do We Need?• How much data and what data?• From where?• What analysis do we need to do?• What narrative/presentation is enough? Often, in business research, we tend to focus on volumes . . . . . . missing focus on analysis and presentation for decision-making
  5. 5. 5What is Truth in Business?• What is enough information to make a decision—or real “Truth”?• How much information of what type is “enough”?• How messy will this be and still be “normal”?• Who can best find the data?• Which people know what part of the answers we need?• Is the core data we need reachable?• What is our role as researcher with perspective?
  6. 6. 64 Types of Qualitative Data Leech, N. L., & Onwuegbuzie, A. J. , 2005
  7. 7. 7Focus on Quantitative Issues Design characteristic• Validity • Sample sizes• External Validity: • Statistical significance generalize findings across • Sampling bias populations, tasks, and • Coding consistency environments (Campbell • Control groups & Stanley, 1966) • Pre- and post-testing• Internal Validity: Design rule out other factors • Instrument design (tested other than the validity) Independent Variable
  8. 8. 8Related Issues with Qualitative Issues Design characteristic• Trustworthiness • Triangulation • Code/Recoding• Truth Value/Credibility • Technique• Applicability/Fitness/ • Member Check (show analysis Transferability to participants) (Janesick,• Consistency/ 2000; Merriam, 1998) Dependability • Interview corroboration• Neutrality/ • Peer debriefing Confirmability • Auditability • Bracketing • Balance• (Guba 1981, Schmid, 1981) • (Lincoln & Guba, 1985)
  9. 9. 9WHAT IS QUALITATIVE?When should we use it?
  10. 10. 10Qualitative vs. QuantitativeQuantitative Qualitative•Helpful when “answering •Looking at single case orquestions of who, where, small number of caseshow many, how much, and •Looking at in-contextwhat is the relationship situation, framed by wordsbetween specific variables” and narratives(Adler, 1996, p. 5) •Looking for in-context•Striving for causation or for relationships andgeneralizing to larger connectionspopulations •Creating hypotheses or instruments for quantitative
  11. 11. 11Qualitative Can Enrich QuantitativeExamples:“Prebriefing” (Collins et al., 2006), checkingpotential quantitative survey participants forwillingness and suitabilityPilot study to assess the appropriateness of aninstrument like a questionnaire or surveyRuling out hypotheses
  12. 12. 12Challenge of Qualitative • Difficulty in capturing lived experiences via text • Creating a “bricolage” – an assemblage of representations that fit a complex situation (Denzin & Lincoln, 2005) Use of Qualitative Analytical tools helps connect this complex in-context environment into a way that others can understand.
  13. 13. 13DESIGNING THE RESEARCHGoalsNarrativesPopulations & SamplesDataInstruments
  14. 14. 14Qualitative Research:Collecting/Combining Narrative(s)
  15. 15. 15Populations and “Level”• Populations: Total target group • AMR: Group could be a regional or business population, or could be members at a level in the organization• Sample: Group in study
  16. 16. 16“Who” has the Data?• Thinking in terms of Five Forces • Vendors, Customers, Competitors• Reasons to share• “Knowing” • Belief, research, or connections • Expert does not mean “knows” real information• Similar question: Secondary Research and connecting Primary to it
  17. 17. 17Research Methods• Document Analysis• Focus Groups• Observations• Interviews• Shadowing• Participant Observation• Literature Review• Oral History/Ethnography• Social Network Analysis (SNA) • Quantifying/mapping context
  18. 18. 18Literature Review• Check out what research has been done on the research methods that you are considering, e.g., focus groups, narrative research, document analysis• Google Scholar: Good launching pad
  19. 19. 19Sampling Methods and Size• Quantitative: Concern with probabilities and similarities to overall population• Qualitative? • Snowball sampling: uses social networks and connections to identify unknown populations • Convenience sampling • Judgment Sample: based on framework of variables from researchers • Maximum Sampling, Extreme• How much is enough? • Saturation (repeated patterns) (Rubin & Rubin, 1995).
  20. 20. 20Instruments• Creating a Questionnaire• Focus Group – Outline, Objectives• Surveys – may be instruments already tested for validity• Interviews • Open Ended • Semi-Structured • Test and plan coding methods upfront; what will you input the answers into? • Grounded Theory: Grand Tour Question(s)
  21. 21. 21DOING THE DARNEDRESEARCHRecording and measurementTranscriptionField Notes
  22. 22. 22Recording as Strategy Methods Video Audio (including cell phone) Issues Affect on Outcomes: Performance Security/Storage Permission Transcription Glitches/Errors: Multiple devices
  23. 23. 23Transcription as Friend and Foe01: Exactly. And, as far as doing it, the other, I think the biggest obstacle, is training. Is getting=G Is opportunity.01 It is an opportunity. But . . .Group ((chuckles))01 . . . it is an obstacle as far as the [district is concerned.]=G =[It is hard to not say it.]=01 Because they will not give that time to really teach and train. Even, you know, Im gonna walk in as the . . . the real Luddite. And be able to walk out and feel like I can go out and use the equipment. Not just say it.11 Yeah.
  24. 24. 24Undercurrents from Field Notes Individual impressions Notes before and after sessions Bring your own biases, context, and observations to the table
  25. 25. 25ANALYZING THE DARNEDRESEARCHUsually NOT in qualitative business researchplans
  26. 26. 26Designing the Analysis• Not just casually connecting• Causality vs. Correlation• Two analysis directions • Old-fashioned and robust • Excel worksheets or written on documents • Hand coding and counting • Alternatives • Computer-assisted data qualitative data analysis software (CAQDAS)
  27. 27. 27Recursive Abstraction• Fancy phrase for summarizing, then summarizing the summaries • Usual accidental business research method• Helps to have consistent methods for summarizing between coders/team members, or a coding worksheet
  28. 28. 28Coding• Chunking text data, then adding a code • You can code and iteratively recode/emergent (Tesch, 1990).• Method: aimed to continue to narratively code while bridging to new ideas and surfacing new categories until you began to find pattern codes and themes (Miles & Huberman, 1994).
  29. 29. 29Key Phrase Frequency• Word counts are based on the belief that all people have distinctive vocabulary and word usage patterns.• “Linguistic fingerprints” (Pennebaker, Mehl, & Niederhoffer, 2003, p. 568).• Gives context to words like “many,” “frequently,” etc. terms are fundamentally quantitative.
  30. 30. 30KWICKeywords-in-context (KWIC; Fielding & Lee, 1998)•Data analysis method that reveals how respondents usewords in context•Compares words that appear before and after “key words”
  31. 31. 31Narrative Analysis (NA)• (Nearly) all qualitative research is filtered by contexts, beliefs, and methods of communication• NA evaluates patterns, threads, tensions, and themes within the transcripts and field notes (Clandinin & Connelly, 1994, 2000; Ryan & Bernard, 2000).• Can pull out portions of text where themes are mentioned (Ryan & Bernard, 2000)
  32. 32. 32Triangulation• Assesses the integrity of the inferences that one draws from more than one vantage point (Lincoln & Guba, 1985)• Use of multiple data sources, multiple researchers, perspectives, tools, and/or methods (Denzin, 1989; Schwandt, 2001)• Adds confirmability, dependability, and credibility to data collection
  33. 33. 33USING ANALYSIS TOOLSExamplesQuantitative Analysis (CAQDAS)TranscriptionData Visualization
  34. 34. 342 Reasons for Tools• Help the team gather, sort, visualize, and engage messy and abundant qualitative data• Explain and convince client of validity of research done • Ability to walk through analytical process and explain the patterns in the data
  35. 35. 35Example: Express Scribe
  36. 36. 36Example: ATLAS.ti
  37. 37. 37 CAQDAS • Tools for recording, storing, indexing, content searching, mapping/networking, and sorting data (Lewins & Silver, 2005; Morse & Richards, 2002)The University of Surrey’s CAQDAS Networking Project Reviews:
  38. 38. 38Another Example: Qualrus
  39. 39. 39DATA VISUALIZATION ANDINFOGRAPHICSFinding the StoryTelling the StoryPersuading Change with the Story
  40. 40. 40Using Data to Tell and Be the Story• Abundant Data (“Big Data”) from in-context data collection in our connected world• Social Network Analysis (SNA) – how we are all connected• Big company problem – large volumes of data to digest and act upon • “Investigating relationships” – not just for presentation, but for research teams to visualize emerging patterns
  41. 41. 41Concept Mapping:One Method of Data Visualization
  42. 42. 42 • Great list:Data • Piktochart – Transforms your information into memorable presentations. • - Create interactive charts and infographics.nInfographics for • Gephi – Like Photoshop for data. Graph visualization and manipulation software.Decisions • Tableau Public - Free data visualization software. • Free Vector Infographic Kit – Vector infographic elements from(others in MediaLoot.Appendix slides) • Weave – Web-based analysis and visualization environment. • iCharts – Charts made easy. • ChartsBin – A web-based data visualization tool. • GeoCommons – See your data on a map. • VIDI – A suite of powerful Drupal visualization modules. • Prefuse – Information visualization software. • StatSilk – Desktop and online software for mapping and visualization. • Gliffy – Online diagram and flowchart software. • Hohli – Online charts builder. • Many Eyes – Lets you upload data and create visualizations. • Google Chart Tools – Display live data on your site.
  43. 43. 43Questions? Dr. Gigi Johnson @maremel Maremel Institute
  44. 44. 44Playing • Wordle – http://www.wordle.comwith – fun tool to turn words fromWords documents into word maps • Tagxedo -- – similar to Wordle, Tagxedo lets you create word clouds and sculptures from URLs, Tweets, and other social media documents, as well as export them into a variety of formats.
  45. 45. 45 We can tinker with maps, both as pre-Playing made images as well as data-drivenwith Maps tools. •Web Resources Depot -- -- shares a variety of world map images for use •Free PSD Files -- -- This site has easy images for further editing for presentations •GunnMap -- -- creates world maps with your data •StatPlanet Map Maker -- -- also creates interactive maps
  46. 46. 46 • Several tools let you expand how you lay outPlaying concept maps and linked ideas: • FreeMind -- – I enjoywith this free tool. Graphically simple, it lets you play with a free tool for mind mapping that can be adaptedConcept into all sorts of other applications. • Webspiration – – IMaps miss its freemium mode; it now has a trial period and then costs $6/month. I found Inspiration and Webspiration wonderful for group presentations and immediate work. • MindManager -- – this concept management tool starts at $20/month for one and discounts for group collaboration. • MindNode -- – This tools for Mac computers comes at a moderate price -- $20 for the mac and $10 as an iOS Apps. • VUE by Tufts -- -- I really enjoy this “Visual Understanding Environment” tool, which combines concept maps with search and graphics. •
  47. 47. 47 • Prezi -- -- My recentPresentations undergraduate class spent half of their projects in Prezi, which has a zooming camera metaphor across a vast digital white board. They enjoyed putting in music, video, and other embedded content. I got a bit dizzy, but enjoyed the creativity. • Sliderocket -- -- Several of my students enjoyed using Sliderocket for class presentations. It gave them a robust and elegant toolset to work with. • Brainshark -- -- Friends who are professional business development executives heartily recommend Brainshark as a way to pre- package and present content at a distance.
  48. 48. 48 • Google Charts API - -- you can useGraphs Google Charts to create animations in charts, dashboards, and lots of other goodies • Gliffy -- -- I just found Gliffy, a great diverse creatorand of charts and graphs. Different versions of it work with different social workspace/sharing software: • Hohli -- – free online chart builderCharts • Creately -- -- (paid but cheap at $5/month/person) is a online tool to build charts, and collaborate around them • Many Eyes -- -- an experiment by IBM Research and the IBM Cognos software group let users create and evaluate data visualizations. • GGobi -- -- free data visualization tool for your datasets • Mondrian -- -- open source toolset for charting and graphing data plots and more complex graphs and data- driven visuals • OpenDX -- -- Older open source software, based on IBM’s visualization data explorer. • Spotfire -- – a whole visualization suite, free for individuals for the first year, then $99/year thereafter. • Visualizefree -- -- Sampler of more complex system; shows real-time images from the FAA of flights as a sample • Mycrocosm -- -- quirky tool to create displays of your own personal data that you can input by cell or email and track
  49. 49. 49 • Hans Rosling’s GapminderPlaying Foundation worked with Trendalyzer, which then was sold to Google in 2007, thenwith folded away when Google Labs.Motion • VIDI -- -- VIDI Data, run by the Jefferson Institute, provides aCharts visualization module for Drupal CMS to show motion charts, timelines, geodata, and comparative data. • TrendCompass -- -- lets you add your own data to their data visualization tool if you register • Eurostat Explorer -- o/ -- sample with EU data that can be played with using a motion graphic.
  50. 50. 50Playing • Tweakersoft’s Vector Designer -- ordesigner.html -- This $20 MacImages App helps users create simple vector designs. • GIMP -- -- For those who would want to tinker with Photoshop, but wince at the pricetag, GIMP (“GNU Image Manipulation Program”) is an open source alternative. • Inkscape -- -- open source vector graphics •
  51. 51. 51Playing There are lots of extensive tools to work with large public databases.with Data •Google Public dataResources -- -- From the creators of abundant and specialized search comes search just for public data sources • -- owse and data/channels -- Visualizing provides links to all sorts of sample and interesting data sets
  52. 52. 52 • KDnuggets News newsletter on Data MiningAdditiona and Knowledge Discoveryl Tools -- tion.html -- longer list of free and paid data visualization tools
  53. 53. 53Select References• Leech, N. L., & Onwuegbuzie, A. J. (2007). An array of qualitative data analysis tools: A call for qualitative data analysis triangulation. School Psychology Quarterly, 22, 557-584.• Lewins, A., & Silver, C. (2007). Using Software in Qualitative Research: A Step-by-Step Guide, Sage.• Lewins, A. (2008). CAQDAS: Computer Assisted Qualitative Data Analysis in (ed) N. Gilbert, Researching Social Life (ed.)(3rd ed). London: Sage.• Lewis, R.B., & Maas, S.M. (2007). QDA Miner 2.0: Mixed-Model Qualitative Data Analysis Software, Field Methods 19: 87-108• Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.• Silver, C., & Fielding, N. (2008). Using computer packages in qualitative research. In C. Willig & W. Stainton-Rogers (eds.), The Sage Handbook of Qualitative Research in Psychology. London: Sage.