Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Trend Spotting Workshop

168 views

Published on

Trend Spotting Workshop. A practical guide to making sense of large information sources. Workshop run with Gemma Long (QAA) at etc.venues Maple House, Birmingham, 23rd February 2017.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Trend Spotting Workshop

  1. 1. Trend Spotting: A practical training day to help you make sense of large information sources Marieke Guy and Gemma Long, QAA Maple House, Birmingham 23rd February 2017
  2. 2. Presentation 1: Introduction to the day
  3. 3. Gemma Your facilitators for the day…
  4. 4. Agenda for the day Time Session 10:00 – 10:30 Registration and coffee 10:30 – 10:40 Presentation: Introduction to day 10:40 – 11:00 Practical session: Introductions and aims for the day 11:00 – 11:20 Presentation: Introduction to qualitative (language-based) data & thematic analysis 11:20 – 11:50 Practical session: Sharing examples of research analysis 11:50 – 12:20 Presentation: Bringing together sources 12:20 – 13:00 Lunch
  5. 5. Agenda for the day Time Session 13:00 – 13:30 Practical session: Considering scenarios 13:30 – 14:00 Presentation: Carrying out data analysis 14:00-14:30 Practical session: Thematic analysis 14:30-14:45 Coffee break 14:45-15:15 Presentation: Outputs of language-based analysis 15:15-15:45 Practical session: Comparing good and bad report examples 15:45 -16:00 Presentation: Conclusions and resources list Feedback sheets 16:00 Finish
  6. 6. A practical training day to help you make sense of large information sources
  7. 7. • Help you better organise and make sense of large information sources • Help you carry out language-based research and development • Help you with market research • Help you with business enhancement • Help you with report planning • Help you write better, more engaging reports Aims for the day
  8. 8. “To understand is to perceive patterns.” Isaiah Berlin
  9. 9. Information sources
  10. 10. Key terms Term Definition Project The piece of work to be carried out. Research question Question that identifies what needs to be found out. Collecting and organising sources The process of pulling together information sources and organising them in some way. Coding (into nodes) The process of pulling out sections of text and allocating words or short phrases that give it meaning. Thematic analysis Form of analysis in that involves examining and recording themes. Report writing Writing up a report that pulls together all the outcomes from your analysis.
  11. 11. About QAA
  12. 12. • Founded in 1997 • Offices in England, Scotland & Wales • Higher Education Agency • http://www.qaa.ac.uk/ About QAA
  13. 13. Our mission is to safeguard standards and improve the quality of UK higher education, wherever it is delivered around the world
  14. 14. QAA strategic aims Aim 1: enhance the quality and secure the academic standards of UK higher education, wherever delivered, in order to maintain public confidence Aim 2: provide leadership, through knowledge and resources, in assuring and enhancing the quality of higher education within the UK and internationally Aim 3: extend and enhance the value and reach of QAA’s services, within and beyond UK higher education
  15. 15. QAA: future directions
  16. 16. QAA: future directions i) Secure our role as the UK’s agency of choice for higher education ii) Develop as a client-focused organisation iii) Be an agile organisation, able to adapt quickly iv) Offer innovative services, for different market sectors v) Deliver tangible benefits for the sector QAA’s Board has begun the process of developing our next strategy to 2020
  17. 17. • Non-subscriber events on:  Student Engagement  Managing Quality & Enhancing Quality  Professional, Statutory and Regulatory Body (PSRB)  Life after Degree Awarding Powers (DAP)  Transnational Education post EU  Degree Apprenticeships • Quality Enhancement Network (QEN):  Two events in April on the theme of Entrepreneurship and Apprenticeships (1 London, other tbc)  Two events in June on Postgraduate Research (1 Leeds, 1 London)  Quality Code consultation event in June Upcoming events
  18. 18. Practical 1: Introductions and aims for the day
  19. 19. • Introduce yourselves • Who are you? • Where do you work? • What do you do? • Share a reason why you are here • Share an aim for the day To the wider group
  20. 20. • One willing volunteer should empty their purse, wallet or bag on the table • Arrange and cluster the content into categories • Label each pile • Discuss In small groups
  21. 21. Presentation 2: Introduction to qualitative data & thematic analysis
  22. 22. By Mark Johnstone, FlowingData
  23. 23. What is qualitative data?
  24. 24. Qualitative data vs Quantitative data
  25. 25. • Information that is not in a numerical form i.e. language-based data, descriptive data… • Examples include: survey responses, diary accounts, open-ended questionnaires, unstructured interviews, unstructured observations, collections of reports • Often about interactions and relationships • Analysis of such data tends to be more difficult than looking at quantitative data (numbers) Qualitative data
  26. 26. • To identify themes and patterns and share in the form of reports • To answer particular questions (or theories) • To help inform decision making and business planning How is it used?
  27. 27. • Using data that you have access to as an organisation to help guide decisions that improve success • Informed because should be based on more than just numbers – contextualised and use staff intelligence • Important part of strategic planning • Important to have data that backs up the decisions that are being made Data-informed decision making
  28. 28. • Anything more than you can easily read during the work time available • Perhaps more than 20 pages? • It’s all about organisation and process • It’s also about reproducibility and reuse • Big data – volume, velocity, variety • Tools, tools, tools… What are large volumes??
  29. 29. • Why have you been asked to do this work? • Who is it for? Who will see it? Where will it go? • Is there an agenda behind it? Where are the sensitivities? • Who is leading on the work? What about sign off? • What will be the output? • What is the business enhancement purpose? • How will success be measured? Starting point
  30. 30. “If you do not know how to ask the right question, you discover nothing.” W. Edwards Deming
  31. 31. • What do you need to produce? • Who is it for? • Is it for internal or external viewing? • When should it be delivered? • How long should it be? • How can it be promoted? End point
  32. 32. The template is designed to help you when planning the production of a report or publication. It asks all the questions you need to have considered before starting work. It also ensures that you have thought through the sign off process and considered the resources you will need to publish the report/publication. The time spent planning at the beginning should reduce the time spent later on. Publication report template
  33. 33. Name/title of publication/report: Purpose/objectives for the publication Why is this report being commissioned? What does it hope to achieve? How will the report benefit the organisation? Is this report part of a contractual obligation? Is this report part of a series? Target audience(s) Who is this report being written for? Who else will find it interesting? Potential risks/issues for this publication Are there any controversial issues that are likely to be raised in this report? Are there any sensitive areas around this report’s subject? Report writer Who is responsible for writing this report? Budget code Project manager Name of individual who will manage the project overall. Which format will the report be published in? Where will it be published? What format? Word count/ no. of pages. Key terminology Use of terminology. Publication deadline Potential timing issues? Marketing lead times? Project approval by director Signed: Date:
  34. 34. Report writing Plan • Brainstorm • Outline Analyse data Compose • Introduction • Impact • Recommendations • Summary • Resources Edit
  35. 35. • Visual information management tools • Method of storing, organising. Prioritising, learning, reviewing and memorising information • Stimulates creativity • Helps you see bigger picture and detail • Many software tools: Mind maps
  36. 36. • What assumptions do you hold about the research question? • What values and life experiences may shape your interpretation of the data? Reflexivity exercise
  37. 37. Practical 2: Examples of your research so far
  38. 38. • Think about the ‘research’ you have worked on in the past that has looked at large information sources • Share one recent example with the group • Each write down the following on a post-it note: • One challenge of carrying out this work • One lesson learnt of carrying out this work • One success in carrying out this work • Share your post-its on the wall In small groups
  39. 39. Presentation 3: Bringing together data sources
  40. 40. • Interviews • Surveys • Consultations • Focus groups • Liaison reports – soft intelligence • Polls Collecting data sources
  41. 41. • Existing reports • Case studies • Magazines, journals, newspapers, books • Grey literature • Web pages • Databases Using existing data sources
  42. 42. • Importance of taking a critical approach to appraising sources • Who wrote it or said it? • Do they have an agenda? • Were they given a template or series of questions? • Questionnaire bias Importance of context
  43. 43. • It takes skill – attend a course ;-) • Think about what you want to find out • Keep it short • Think about your audience • Use simple words and avoid being too formal • Expand acronyms • Avoid leading questions • Balance not bias – Likert scale… • Avoid double negatives Writing a good questionnaire
  44. 44. • Use of more than one method of data collection or research • Bringing together quantitative (numbers) and qualitative (words) data • Most research ends up using some numbers Mixed methods research
  45. 45. Practical 3: Considering scenarios
  46. 46. • Look at the two scenarios allocated • Brainstorm a plan for the project manager • To think about: • Starting point – research question • Middle – how can they analyse this data? What do they need to keep in mind? • End point – what do they need to produce as an output? • What challenges can you identify? In small groups
  47. 47. Presentation 4: Carrying out data analysis
  48. 48. Two Approaches Deductive Research Inductive research Aimed at testing an idea Creating a new theory Similar to grounded theory (constructing a theory through analysis of data) Top down approach Bottom up approach Theory driven Data driven Have an idea in mind about what you want to say or show. Perhaps you have an agenda already? Am open to seeing what the data shows. Much more exploratory.
  49. 49. Theory Hypothesis Observation Data creation Confirmation Observation Pattern Tentative Hypothesis Theory Deductive vs Inductive
  50. 50. Themes Trends Patterns Codes Mapping
  51. 51. Analysing language-based data Extract themes Identify relationships Highlight differences Create generalisations Identify similarities From Helen Dixon, Education Consultant
  52. 52. • Common form of analysis in social science research • Involves examining and recording themes • Importance of organising data • Key element is ‘coding’ – recognising important moments in the data and highlighting them Thematic analysis familiarisation with data generating initial codes searching for themes among codes reviewing themes defining and naming themes producing the final report
  53. 53. • Occur numerous times across the data – but frequency not always related to importance • Researcher judgement is key tool • Try to avoid preconceptions • Semantic and latent themes – look beyond what people say – underlying ideas • Themes and codes are different Themes
  54. 54. • Trends are the general direction of travel: “our customers are starting to prefer…” • Patterns are series of data that repeats: “Time has shown that customers like x” • Trendlines • Upwards and downwards • Trend analysis Trends and patterns
  55. 55. • Actively look for patterns within your environment, industry and people’s behaviour • Look at how information is structured • Look for relationships between different pieces of information • Think about cause and effect relationships Actively looking for patterns
  56. 56. • Things that are similar • Things that are different • Things that are frequent • Things that are sequential or run in cycles • Things that are opposite • Things that are caused by one another • Things that are in relation to one another Recognising patterns
  57. 57. • Chronology • Key events • Settings • People • Places • Processes • Ideas Things to look at…
  58. 58. What is a code? “A word or short phrase that symbolically assigns a summative, salient, essence-capturing, and/or evocative attribute for a portion of language-based or visual data.” Saldaña, J (2009). The Coding Manual for Qualitative Researchers.
  59. 59. • Gathering all the information about a topic together for further exploration – you code into nodes • Nodes can be topics, people, places, sections of a report, positive feedback etc. • Coding is heuristic • Different projects require different approaches • Need for consistency across projects • Can be carried out in cycles Coding
  60. 60. • Automated semantic analysis • Good for very large volumes of data • Provides numerical indication of relevance • Highly inclusive and objective • Clustering algorithm • e.g. PTES work using Leximancer • Still requires researcher interpretation Concept mapping
  61. 61. • Think about the sentiments associated with a concept • Certain words connected with certain sentiments • Favourable or unfavourable? • e.g. difficult = unfavourable, enjoyable = favourable • How will you deal with this? Sentiment identification
  62. 62. • Call them what you like: log notes / self-memos / summaries / reflexivity journal / notes • Supports transparent process and team working • Keep notes on:  Project aims: goals, assumptions, concerns, research question  Sources: where they are from, what is their remit, how they are organised  Project progress: why you have chosen the codes and theme, insights etc.  Ideas for future analysis • Make sure notes written in logs are different from the data – different font etc. Research logs
  63. 63. • Kick off meeting where you define terms and nodes • Keep in contact – IM, Yammer, Skype etc. • Regular catch ups – daily stand ups? • Share what you have learnt – share early and often • Log your findings • Have a retrospective to share lessons learnt • The benefits of different perspectives Team working
  64. 64. • Nvivo – from QSR http://www.qsrinternational.com/nvivo-product • Atlas-ti – from Scientific Software development GMbH http://atlasti.com/ • MAXQDA – from VERBI http://www.maxqda.com/ • Leximancer http://info.leximancer.com/ • Excel – Part of MS Windows • Many tools out there – some open source e.g. RQDA • None analyse the data – just help organise!! Language-based analysis tools
  65. 65. • Collect together categories • Structure them in some way (perhaps using a mind map) • Think about headings and sub-headings • Start thinking about the conclusion – what have you discovered • Think about key findings • List your favourite quotations Before writing up
  66. 66. Practical 4: Thematic analysis and ‘coding’
  67. 67. • Look at the source material given • Decide on your coding approach • Start to code the text using the highlighter pens • Cut out the coded content and place into piles • Write a list of the codes you have identified on post-it notes and label your piles Individually
  68. 68. • Merge categories if appropriate • Staple the piles of categories • Arrange the categories on the table thinking about hierarchy and structure • Feed back to the wider group In small groups
  69. 69. Presentation 5: Outputs of language-based analysis
  70. 70. • Internal reports • Summaries, recommendations • External reports • End of year review • Grant applications/tenders • Consultation responses • Infographics Outputs
  71. 71. “The value of an idea lies in the using of it.” Thomas A. Edison
  72. 72. • If you ask for feedback you should act on it • Pick the areas you can respond to • Offer a strategy for dealing with them • Don’t ask if you don’t want to hear the answer • “You said – we did” campaign • #YouSaidWeDid • e.g. at universities based on NSS Feedback loop
  73. 73. • Reports look good with a few numbers in! • Think about key stats from your project:  How many data sources?  When were they collected?  How many participants?  What percentage of overall participants was this?  Answers to any yes/no questions? • Bar and pie charts • Graphs and sparklines • Tables Combining with numbers
  74. 74. • Placing data in a visual context • Helps users understand the significance of the data • Want users to think about substance rather than methodology • Use the art of comparison: time-series, ranking, ratios, deviation, frequency, correlation, geographical location • Dangers of spurious accuracy – avoid 34.567%, use about a third • Think about story telling approaches Data visualisation with numbers
  75. 75. • Think about story telling approaches • Word tags, bubble clouds, tree maps • Word counts • Venn diagrams • Cluster analysis • Using quotes • Using photos and icons Data visualisation with words https://www.behance.net/gallery/7526739/Nineteen-Qualitative- Data-Visualization https://infogr.am/
  76. 76. • Infographics Side by side https://visage.co/turn-qualitative-data-visual-storytelling-content/
  77. 77. • Systematic approach to presenting clear and user-focussed information • Focus is on layout • Information is chunked up into digestible chunks • Uses banners, tables, headings, margins etc. Information mapping
  78. 78. • Answers the brief (or the research question) • Consider the audience • Well structured and coherent • Offers clear examples • Starts with an executive summary • Ends with a sound conclusion • Make appropriate recommendations • Gives appropriate references Writing a good report
  79. 79. • Honesty • Agendas – being upfront about them • Biases, values and judgements of researchers • The messiness of reality • Timing – familiarise yourself with the data • Volume of data and work • Prioritise the practical • Conflicts of editing Challenges
  80. 80. Practical 5: Comparing good and bad reports
  81. 81. • Look at the selection of reports • What works? • What doesn’t work? • List the possible faults of a report • Think about the type of reports you create and how they could be improved • Feed back to the group In small groups
  82. 82. Conclusions & Feedback
  83. 83. • What have you learnt today? • Did we cover all areas you wanted to hear about? • What are your next steps? Please fill in the feedback form! To the wider group
  84. 84. Resources
  85. 85. • Thematic analysis: http://designresearchtechniques.com/casestudies/thematic-analysis/ • Saldaña, J (2009). The Coding Manual for Qualitative Researchers. • HEA PTES survey responses https://www.heacademy.ac.uk/resource/their-own-words • Data visualisation beyond numbers: https://www.techchange.org/2015/05/27/data-visualization-beyond-numbers-tools-for- qualitative-data-visualization/ • Visualising data: http://www.visualisingdata.com/ • Thematic coding – video with Graham Gibbs https://www.youtube.com/watch?v=B_YXR9kp1_o • Better Evaluation – thematic coding: http://betterevaluation.org/en/evaluation-options/thematiccoding • Wikihow – Write a great report: http://www.wikihow.com/Write-a-Great-Report Useful resources
  86. 86. • All images from:  Pixabay – CC0 - pixabay.com/  or author’s own  Or url given Credits
  87. 87. qaa.ac.uk enquiries@qaa.ac.uk +44 (0) 1452 557050 © The Quality Assurance Agency for Higher Education 2017 Registered charity numbers: 1062746 and SC037786 Thank you

×