It’s time to ‘Face’ the truth. Is Facebook’s Survey Monkey a legitimate research and pedagogical tool?

1,579 views

Published on

Presentation by Stephen Hole in collaboration with Christopher House (University of Wales: Trinity Saint David, Swansea) and Gavin Bunting (University of Wales) at the Research-Teaching Practice in Wales Conference, 10th September 2013, at the University of Wales, Gregynog Hall. Slidecast edited by Professor Simon Haslett.

Published in: Education, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,579
On SlideShare
0
From Embeds
0
Number of Embeds
665
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

It’s time to ‘Face’ the truth. Is Facebook’s Survey Monkey a legitimate research and pedagogical tool?

  1. 1. It’s time to ‘Face’ the truth. Is Facebook’s Survey Monkey a legitimate research and pedagogical tool? Christopher House, Gavin Bunting and Stephen Hole . 1
  2. 2. Survey Monkey • Survey Monkey is: – – – – – Well known; Free for small surveys; Can niche non spatial data; Special interest groups; A growing mechanism of choice for undergraduate research. 2
  3. 3. Sources Surveyed • Faculty of Applied Design & Engineering: – PhD Students (some with 1,000 + replies); – MSc & BSc class exercises and dissertations; – HEI staff awareness survey. • School of Applied Computing: – Approximately 80 undergraduate projects and dissertations per year for the last 5 years; – 20% Dissertations; – 80% Projects; • Approximately 50% of students will use a survey to gather primary data; • Numbers are growing year on year. 3
  4. 4. Surveys: Setting the Scene (1) • Surveys are probably the most commonly used data gathering instrument world wide. • Students’ perceive survey based research: – To be straight-forward; – An easy means of gathering primary data. • Students' often fail to consider: – Response rate; – Reliability; – Validity. 4
  5. 5. Surveys: Setting the Scene (2) • They do not expect to encounter problems and often produce surveys that do not: – – – – – Take into account survey design; Ask the right questions; Gather appropriate samples; Provide data that can be meaningfully analysed; Take into account relevant statistical analysis techniques. 5
  6. 6. Surveys: Setting the Scene (3) • Badly designed surveys when administered: – Render the analysed results: • At best inconclusive; • At worst producing information that results in flawed decision making. • Due to the time frame of an undergraduate dissertation there is limited opportunity to redesign and administer a further study. 6
  7. 7. Survey Process Activities • The survey process typically includes the following activities: – – – – – – – – – – Setting specific, measurable objectives; Planning and scheduling the survey; Ensuring that appropriate resources are available; Designing the survey; Preparing the data collection instrument; Validating the instrument; Selecting the participants; Administering and scoring the instrument; Analysing the data; Reporting the results (Pfleeger and Kitchenam, 2001) 7
  8. 8. Student Survey Dilemma (1) • Not looking before they leap: – Data gathering equates to a questionnaire; – A questionnaire equates to Survey Monkey; – Sample population equates to Facebook, Twitter and other Social Media contacts; – Data Analysis equates to Bar Charts, Pie Charts, Line Graphs etc. 8
  9. 9. Student Survey Dilemma (2) • Not looking before they leap results in: – – – – – Poor quality data; Poor analysis; Weak conclusions; Inappropriate recommendations; Low grading. 9
  10. 10. Question & Data Types • Open Format; • Closed Format: – – – – – – – Dichotomous; Range; Radio Button; Rank Order Scaling; Multiple Choice; Likert Scale; Numeric. • Question type will determine data type: – Survey Monkey constrains the options and flexibility of survey design. 10
  11. 11. Statistical Methods • Parametric – Numeric data that is from a normal (bell shaped) distribution. • Non-parametric – Data is ordinal in nature and does not fit a normal distribution; – Data is reliant on some form of ranking and is not numeric in nature; – Likert Scales generate ordinal (non-parametric data). 11
  12. 12. Typical Statistical Measures • Parametric – – – – Chi Square; Linear Regression; ANOVA; T Test. • Non-parametric – Kendall’s Tau; – Mann-Whitney U; – Spearman’s Rank Correlation Coefficient; – Anderson-Darling; – Kolmogorov-Smimov. 12
  13. 13. Recognising Data Types • Students may not understand the difference between parametric and nonparametric data and as a result: – Use a parametric statistical test on ordinal data; – Draw inaccurate results from their analysis. 13
  14. 14. Student Checklist (1) • Survey Monkey is a legitimate research and pedagogical tool provided the following checklist is used: – Questionnaire must be a suitable data collection method; – Administration by use of a web interface will not skew the sample population; – Sample population must be identifiable: • Not an invite to all friends on Facebook, Twitter, etc; – Quality design principles are incorporated into the questionnaire construction; 14
  15. 15. Student Checklist (2) – Ethical approval has been granted; – No questionnaire to be administered without supervisors approval; – Questions meet the objectives of the research hypothesis; – Collected data can be suitably analysed; – Statistical measures used in the analysis match the type of data being collected: » Parametric for normal bell-shaped data; » Non-parametric for ordinal data. 15
  16. 16. Student Checklist (3) – Select appropriate statistical analysis software: • • • • • SPSS; Microsoft Excel; Weka Data Mining Tool; SQL Server Data Mining Tool; Etc. – Understand the output generated by the software with sufficient background reading. 16
  17. 17. P& K Enhanced Survey Guidelines • The survey process typically includes has the following activities: – Setting specific, measurable objectives: • Administration by use of a web interface will not skew the sample population. – Planning and scheduling the survey: • Consider innovative completion drivers. – Ensuring that appropriate resources are available: • Less significant in SM surveys. – Designing the survey: • Question type/data. – Preparing the data collection instrument: • Consider technical synchronisation of software packages. 17
  18. 18. P& K Enhanced Survey Guidelines – Validating the instrument: • Vary depending on level of academic qualification. – Selecting the participants: • Carefully following methodological practice. – Administering and scoring the instrument: • Controlled by SM. – – – – Analysing the data; Reporting the results; Ethical approval has been granted; Supervisory approval has been granted. 18
  19. 19. Case Study • Handout to be supplied. 19
  20. 20. Any Questions? 20

×