1. Asking the right questions: Collecting and validating data – refining, rephrasing, and refocusing
Dr. Corinne Haigh, The School of Education, Bishop’s University, chaigh@ubishops.ca
PROCEDE 2014: People, Passion, Proof – The Data Difference
PROCEDE 2014
1
2. Session Outline
What makes a good research question?
Where do I begin?
What is the danger of a single story?
How can I be sure that the measurements I take match the question I am trying to answer?
How do we define what we wish to measure?
What are reliability and validity?
What are some ethical issues to consider?
PROCEDE 2014
2
3. Pollev.com/chaighbu
From any browser
(code or keyword) <your response>
From a text message
Participating with Poll Everywhere: How to vote via the web or text messaging
22333
10. Reflection
What types of questions was I able to ask using the polling software?
What type of information was I able to gather?
How could I analyze this information to answer a specific question?
How might I present it to others?
Take a moment to reflect on these questions individually.
PROCEDE 2014
10
11.
12.
13. Reflection
What types of questions was I able to ask using the polling software?
What type of information was I able to gather?
How could I analyze this information to answer a specific question?
How might I present it to others?
Take a moment to reflect on these questions individually.
PROCEDE 2014
13
14. Now, think of a recent “problem” you have encountered in your professional life…
Briefly describe it.
How did/might you approach it?
What did/do you hope to learn as a result?
Take a few minutes to reflect on these questions individually, and then compare notes with a colleague
PROCEDE 2014
14
15. The danger of a single story
Asking the right questions involves considering multiple perspectives
We will now watch one video that illustrates the danger of being exposed to only a single story
And I’ve included a link to another for you to watch at another time…
PROCEDE 2014
15
16. Chimamanda Adichie
Novelist Chimamanda Adichie tells the story of how she found her authentic cultural voice -- and warns that if we hear only a single story about another person or country, we risk a critical misunderstanding.
The Danger of a Single Story
PROCEDE 2014
16
17. Ben Goldacre
Doctor and epidemiologist Ben Goldacre shows us, at high speed, the ways evidence can be distorted, from the blindingly obvious nutrition claims to the very subtle tricks of the pharmaceutical industry.
Battling Bad Science
PROCEDE 2014
17
18. Checkpoint
Take 2-3 minutes and write about an educational example of the danger of a single story
Try to write for the full time
You will then discuss your writing with a partner (or in a trio) and then with the whole group
PROCEDE 2014
18
19. Asking a question often starts from necessity
How do I know when…?
Think up some endings to my sentence starter.
PROCEDE 2014
19
20. The Assessment for Learning (AL) Cycle
1. What is known
2. What might be happening and why
3. What you need to find out
4. What you think is happening
5. What you say
6. What you need to do
7. Did what you chose to do work?
Gathering referral information
Generating hypotheses
Checking your hypotheses by collecting information
Arriving at tentative judgement
Reporting findings
Planning instruction
Evaluating success
Further information needed?
20
PROCEDE 2014
21. Refining Questions
Initial questions may be vague, but vague questions can never be answered
Refine questions by:
Consulting past research in the area
Speaking with colleagues
Applying theory
Replicating someone else’s study
Clarifying contradictory findings
PROCEDE 2014
21
22. What is a variable?
A variable is something that can be changed, such as a characteristic or value
We are generally looking at whether change in one variable results in change in another variable
Can you generate some examples of variables from your practice? What kind of relationships between these variables might be interesting to investigate?
PROCEDE 2014
22
23. Learning environment
Academic
Physical
Emotional
Social
Initial Concern
Gaps in learning; opportunity to learn; history of successes/failures; teachers; past and present programs
Vision; hearing; health history
Reading profiles; academic profile; experiences; cognitive development; aptitude (IQ); Language profile
Family; self concept; motivation; security
Culture; friends; maturity
23
PROCEDE 2014
24. Operationally defining a variable
Specifies exactly how to measure and/or manipulate the variables in a study
Go back to one of the examples you generated for the previous question – how would you define it?
Remember, it must be measureable (think about setting SMART goals for IEPs)
PROCEDE 2014
24
25. Types of variables
Independent vs. dependent
Independent: precedes, influences, or predicts the dependent variable (e.g., teaching method)
Dependent: affected or predicted by the independent variable (e.g., student engagement)
Extraneous and confounding
Extraneous: source of error affecting the results (e.g., time of day)
Confounding: varies systematically with the independent variable (e.g., the teacher)
PROCEDE 2014
25
26. 26
How can we make sense of a complex process? We make inferences…
1.
2.
3.
PROCEDE 2014
27. 27
BALLOON
- Recognize the picture as balloons
- Select the appropriate word
- Produce the appropriate articulation
What mental activity needs to occur to identify the picture and say what it is?
How is making an inference about learning different than blowing up a balloon?
- we cant see the “mental activity”
-Therefore, we must make inferences about the mental activity from observable behaviours.
PROCEDE 2014
28. Qualitative research
A means for exploring and understanding the meaning that individuals or groups ascribe to a social or human problem
Emphasizes natural settings, understanding, verbal narratives, and flexible designs
28
PROCEDE 2014
29. Qualitative techniques and data
Data: Verbal descriptions, field notes, observations, documents, photographs, people’s own words, narrative
Techniques: observation, participant observation, open-ended interviewing, review of documents and artifacts
PROCEDE 2014
29
30. Quantitative research
A means for testing objective theories by examining the relationship among variables
Emphasizes numbers, measurements, deductive logic, control, and experiments
30
PROCEDE 2014
32. How do I know which to use?
It is critical to match the reason for the research with the appropriate method
Determined by the purpose and the research question
Using each method will be a challenge
Either can be used well, or poorly
Depth of understanding is important, regardless of the design
Leads to greater impact
PROCEDE 2014
32
33. Checkpoint
Think about some of the potential research questions you generated earlier – what sort of approach would be best suited to answering them?
Consider the data you’d want to collect (or existing data you’d like to access) and the techniques you could use to do so
PROCEDE 2014
33
34. Reliability
Test-retest reliability:
Agreement between scores on the same test, given to the same people, at different times
Inter-rater reliability
Agreement among different raters or observers
Inter-item reliability
Are all of the items on the test equally good at assessing the same underlying ability?
34
PROCEDE 2014
35. Validity
Content validity
The range and relevance of the items
Can be very difficult to measure for some more complex aspects of learning
Construct validity
Concerns the underlying theoretical construct: is the test measuring the target skill or behaviour
Test this by comparing scores in two different measures – should be highly related if they’re measuring the same construct
35
PROCEDE 2014
37. Some broad ethical questions to consider:
Does answering this question benefit the individuals being studied? Will the answer be meaningful for others?
Are participants being marginalized or disempowered?
Are participants being deceived about the intent of the study?
Are participants at more than minimal risk?
Minimal risk: “Research in which the probability and magnitude of possible harms implied by participation in the research is no greater than those encountered by participants in those aspects of their everyday life that relate to the research.” (TCPS 2, Chapter 2)
PROCEDE 2014
37
38. Some broad ethical questions to consider:
Did all participants benefit from the “treatment”?
Is there a possibility that participants will disclose harmful and/or intimate information? What will be done with this information?
How will you protect the anonymity of participants?
PROCEDE 2014
38
39. Checkpoint
Describe an ethical question or concern you’ve had when trying to collect data to answer a specific question (or speculate about a potential ethical issue that may arise in future).
How did/could you address it?
First reflect on this individually and then have a discussion at your table.
PROCEDE 2014
39
40. Some final points for reflection…
Sometimes coming up with a good research problem or question is the hardest part
The approach you take to answering your question will depend on:
The problem you’re trying to address
Your worldview or assumptions about research
The specific inquiry strategies you wish to use
Your personal experience
There are multiple ways to approach data collection and analysis and no one is better than the other as long as you aware of their strengths and limitations
PROCEDE 2014
40