Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to analyse questionnaire data: an advanced session

130 views

Published on

How to analyse questionnaire data: an advanced session

Published in: Education
  • Login to see the comments

  • Be the first to like this

How to analyse questionnaire data: an advanced session

  1. 1. How to analyse questionnaire data: an advanced session Prof Bart Rienties 13 July 2020 A special thanks to Dr Rachel Slater, Dr Christine Thomas & Dr Jenna Mittelmeier (University of Manchester)
  2. 2. Workshop objectives • By the end of this session you will familiar with: –how to analyse the questionnaire data using common psychometric and linguistic techniques. –Concepts introduced in this session are computing of constructs, factor analysis, missing values, reliability, and validity, –Reporting on these questionnaire results and advanced techniques (e.g., ANOVA, correlations, regressions, SEM) –Triangulation of quantitative data with qualitative data.
  3. 3. What is a questionnaire • A research tool for data collection • Usually a set of structured questions for which answers can be coded and analysed quantitatively • Can also include open questions • Can be self-administered or through interview • On-line, postal, telephone, face-to-face • Can also be used for qualitative analysis using semi- structured questions (face-to-face or by telephone)
  4. 4. Questionnaire design in the survey process • Research aim and research questions • Identify the population and sample • Decide how to collect replies • Design your questionnaire • Run a pilot survey • Carry out main survey • Analyse the data • Report findings and dissemination
  5. 5. Questionnaire design in the context of the survey process Research aim & questions Research population & sample Survey type & method for collecting replies Think about analysis Questionnaire design Pilot (always !) Run survey Analyse data Report findings
  6. 6. Why use a questionnaire? Strengths and limitations?
  7. 7. Strengths • Can be quick and relatively simple way to collect data • Insightful when large number of participants involved • Reach respondents in widely dispersed locations • Can be relatively low cost • Standardised and structured questions • Analysis can be straight-forward and responses pre-coded
  8. 8. Strengths • Can cover activities and behaviour, knowledge, attitudes, preferences • Use to describe, compare or explain • Effective for collecting quantitative data – information that can be counted or measured • Low pressure for respondents • Lack of interviewer bias (possibility of ‘ghost interviewer’ effect)
  9. 9. Limitations • Biases –non response, self selection, questionnaire fatigue, acquiescence, extreme response styles • Quality of data? Confidence in results? Reliability? • Unsuitable for some people –e.g. poor literacy, visually impaired, young children, not online • Question wording can have major effect on answers • Misunderstandings cannot be corrected • Can be difficult to account for cultural and language differences
  10. 10. Limitations • No opportunities to probe and develop answers – breadth vs depth • No control over the context and order questions are answered in postal surveys • No check on incomplete responses • Design issues with moving through online surveys • Seeks information only by asking, can we trust what people say? e.g. issues with over-reporting
  11. 11. Questionnaire/Survey design: follow the pros There is already a lot known about your research question – Better to “re-use” and replicate (in order to compare) than to create yourself. Find the instrument at: • http://inn.theorizeit.org/ • www.scholar.google.com • www.eric.ed.org – A good validated questionnaire enhances generalisation • Possibility to compare and contrast your findings to others. • Identify different/similar trends, thereby increasing relevance of your research • Easier to publish (if you want to) in good, ISI- ranked journals • Possibilities to get more citations, as others can use and refer to your research
  12. 12. Search for your instrument: http://inn.theorizeit.org/
  13. 13. Case study 1: Self-determination Theory
  14. 14. Example of Academic Motivation Scale Academic Motivation Scale developed by Vallerand et al 1992 • 28 items • Likert Response scale 1-7 • Questions randomised • Questions/constructs developed based upon theoretical concepts Self- Determination Theory • 3 specific constructs for intrinsic motivation Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senécal, C., & Vallières, E. F. (1992). The academic motivation scale: A measure of intrinsic, extrinsic, and amotivation in education. Educational and Psychological Measurement, 52, 1003–1017.
  15. 15. Case Study 2:What predicts (international) student progression? Input Process Output Learner characteristics (incl. prior education, gender, cultural background) Academic adjustment (incl. personal-emotional adjustment, attachment to institute) Social adjustment (incl. study support, satisfaction with social Environment, financial support) Family characteristics (incl. support, finance, child- care) Learning design (incl. assessment, learning materials, communication) Engagement with learning (incl. VLE engagement, attending sessions, submitting assignments, social media) Academic performance over time (incl. grades, credits, GPA) Degree outcomes (incl. Employment, migration) Interviews with 140+ UNISA students, 40+ stake holders, country reports Madge, C., Breines, M., Beatrice Dalu, M.T., Gunter, A., Mittelmeier, J., Prinsloo, P., & Raghuram, P. (2019). WhatsApp use among African international distance education (IDE) students: transferring, translating and transforming educational experiences. Learning, Media andTechnology, 44(3), 267-282. Raghuram, P., Breines, M. R., & Gunter,A. (2020). Beyond #FeesMustFall: International students, fees and everyday agency in the era of decolonisation. Geoforum. Roos Breines, M., Raghuram, P., & Gunter,A. (2019). Infrastructures of immobility: enabling international distance education students in Africa to not move. Mobilities, 1-16. doi: 10.1080/17450101.2019.1618565
  16. 16. Hypotheses ■ H1 Internationalisation at Home (IaH) students have higher academic adjustment scores relative to Internationalisation Abroad (IA), and Internationalisation at Distance (IaD) students. ■ H2 IaH students have higher social adjustment scores relative to IA, and IaD students. ■ H3 IaH students have higher personal-emotional adjustment scores relative to IA, and IaD students. ■ H4 IaH students have higher attachment scores relative to IA, and IaD students. ■ H5 Access to technology at home is positively related to academic adjustment ■ H6 Access to technology at home is positively related to social adjustment ■ H7 Access to technology at home is positively related to personal-emotional adjustment ■ H8 Access to technology at home is positively related to attachment at UNISA ■ H9 Being from South Africa and having access to technology has a positive impact on academic adjustment ■ H10 Academic adjustment is positively predicted by social adjustment, personal emotional adjustment, attachment, access to technology, and being from South Africa. Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12
  17. 17. SACQ Questionnaire ■ Student Adaptation to College Questionnaire • measures how well students manage the educational demands of the university experience. Academic Adjustment • measures how well students deal with interpersonal experiences at the university (e.g., making friends, joining groups) Social Adjustment • measures how well students maintain emotional equilibrium (particularly in the face of adjustment stressors), and indicates whether the student experiences general psychological distress or shows somatic symptoms of distress Personal Emotional Adjustment • assesses the degree of identification with and commitment towards the university Attachment Baker, R.W., and Siryk, B. (1999). SACQ Student Adaptation to College Questionnaire. Los Angeles: Western Psychological Services. Rienties, B., Beausaert, S., Grohnert,T., Niemantsverdriet, S., and Kommers, P. (2012). Understanding academic performance of international students:
  18. 18. Data collection ■ First, in our initial study (Mittelmeier et al. 2019) we sampled 2634 students from a first- year level course unit with undergraduate students studying for a Bachelor of Science degree in Mathematics and Programming in the College of Science, Engineering and Technology: 320 (11.77%) students (IaH = 270, IaD = 36) responded. ■ In the second phase, we broadened our sampling approach to additional STEM qualifications, whereby we specifically sampled IaD and IA students using MIS data. 5273 students in the selected programmes were invited to participate through an email sent to their university email address, which included a link to the online survey.Altogether, in the two phases 1295 students participated in this study, which is a large sample of participants with a very reasonable response rate of 16.38% (Nulty, 2008) Mittelmeier, J., Rogaten, J., Long, D., Sachikonye, M., Gunter, A., Prinsloo, P., et al. (2019). Understanding the adjustment of first-year distance education students in South Africa: Factors that impact students’ experiences. The International Review of Research in Open and Distributed Learning 20(3). doi: 10.19173/irrodl.v20i4.4101.
  19. 19. Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12 ■ Substantial differences in demographics and socio-economic conditions between three groups of students
  20. 20. Using Explorative and Confirmatory Factor Analysis Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senécal, C., & Vallières, E. F. (1992). The academic motivation scale: A measure of intrinsic, extrinsic, and amotivation in education. Educational and Psychological Measurement, 52, 1003–1017. • Factor analyses allow you to determine whether there is a particular structure in your data (which hopefully is in line with your theoretical model) • Identify which items stick together, and which are in other categories • AMS indicate that items fall neatly into expected categories
  21. 21. Checking for reliability using Cronbach Alphas • Cronbach alpha scores allow you to identify whether items within a construct are reliable (>.60) • Identify which constructs are not reliable (<.60) • Correlations between items might indicate why α <.60 • Pre-vs post-test (aka test vs re-test) good approach to test stability of construct
  22. 22. Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12 ■ No significant differences between the three internationalisation categories in terms of academic and social adjustment (H1 –H2). ■ Significant differences were found in terms of emotional adjustment, whereby South Africans living in SouthAfrica (IaH) indicated significantly lower personal-emotional adjustment scores relative to their peers. Similarly, significant differences were found in terms of attachment (-H4)
  23. 23. Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12 ■ Substantial and significant differences were found between the three internationalisation categories, whereby IaD students had significantly higher access to technology, all medium to large in effect size
  24. 24. Mittelmeier, J., Rienties, B., Rogaten, J., Gunter,A., Raghuram, P. (2019) Internationalisation at a Distance and at Home: Academic and Social Adjustment in a South African Distance Learning Context. International Journal of Intercultural Relations, 72, September 2019, 1-12 ■ The three categories of internationalisation had a significant impact on social adjustment, emotional adjustment, and attachment, but not academic adjustment. South Africans living in South Africa (IaH) had higher social adjustment relative to IaD (H2), but lower emotional adjustment and attachment relative to IaD (-H3, -H4). ■ Access to technology significantly predicted academic adjustment and emotional adjustment, indicating that students with better technology access also had better academic adjustment (H5) and emotional adjustment (H7). Age positively predicted all four SACQ scores, indicating that relatively older learners had better adjustment than younger learners. Gender and type of occupation (being a full time student, looking after the family, and other occupation (i.e., retired from paid work, unable to work due to long-term sickness, unemployed) had no significant impact on the SACQ scores. English as a first language also had no impact on SACQ scores, except for a negative impact on emotional adjustment, while working part-time had a positive impact on emotional adjustment. ■ The majority of black UNISA students felt better adjusted to the distance learning setting than others. Finally, relative to first-year students those who studied in second-year, third-year, and post-graduate level had significantly lower attachment towards UNISA. Furthermore, those at third-year and post-graduate level had significantly lower (self-reported) academic adjustment and personal-emotional adjustment relative to first-year students.
  25. 25. A potential model for predicting school dropout • Psychometric instruments can be used to understand why and how people behave, act, think, etc. • Build theoretical models how one construct links to another
  26. 26. Role of motivation on dropout vs persistent students • Can be used to compare basic descriptives • Test hypotheses
  27. 27. Case Study 2:What predicts (international) student progression? Input Process Output Learner characteristics (incl. prior education, gender, cultural background) Academic adjustment (incl. personal-emotional adjustment, attachment to institute) Social adjustment (incl. study support, satisfaction with social Environment, financial support) Family characteristics (incl. support, finance, child- care) Learning design (incl. assessment, learning materials, communication) Engagement with learning (incl. VLE engagement, attending sessions, submitting assignments, social media) Academic performance over time (incl. grades, credits, GPA) Degree outcomes (incl. Employment, migration) Interviews with 140+ UNISA students, 40+ stake holders, country reports
  28. 28. Thank you! Questions?
  29. 29. How to analyse questionnaire data: an advanced session Prof Bart Rienties A special thanks to Dr Rachel Slater, Dr Christine Thomas & Dr Jenna Mittelmeier

×