Pick a Card
Discovering Student’s Perceptions
Andrew Csizmadia
Wednesday 17th April 2013
Purpose
• Discover trainee
ICT Computer
Science teachers’
perception of
Computational
Thinking
Relevance
“We need to bring
computational thinking
into our schools.”
(DfE, 2013a)
“Computational thinking is a fundamental
skill for everyone, not just for computer
scientists. To reading, writing and
arithmetic, we should add computational
thinking to every child’s analytical ability.“
(Jeannette Wing, 2006)
“…change the world through
computational thinking.”
(DfE, 2013b)
Tools to Use
What is Card Sorting?
• Qualitative participatory
design technique, used to
explore how participants
group items together into
categories and relate concepts
to one another (Martin &
Hanington, 2012)
• Tool to determine
participant’s mental model of
grouping items (Ross, 2011)
• Generative method (Nielsen,
2004) using open card sort
• Evaluative method (Ross,
2011) using closed card sort
• Generate a category tree or
folksonomy
Approach – Card Sorting
Advantages Disadvantages
Simple Content-Centric technique
Cheap Results may vary
Quick to Execute Analysis – time consuming
Established May capture “surface”
characteristics only
Involves users
Provides a good foundation
(Source: Spencer & Warfel, 2004)
Design/Methodology/Approach
Card Selection
• 104 Computational Thinking
related cards identified
following desk based
literature review
Recommended Time
• Allow 30 minutes for each
multiple of 50 cards (Martin
& Hanington, 2012)
Recommended Sample Size
• Between 15 (Nielsen, 2004)
and 30 (Tullis & Wood, 2004)
provides a correlation
between 0.90 and 0.95
respectively
Design/Methodology/Approach
Card Selection
• 104 Computational Thinking
related cards identified
following desk based
literature review
Recommended Time
• Allow 30 minutes for each
multiple of 50 cards (Martin
& Hanington, 2012)
Recommended Sample Size
• Between 15 (Nielsen, 2004)
and 30 (Tullis & Wood, 2004)
provides a correlation
between 0.90 and 0.95
respectively
Design/Methodology/Approach
1 2 3
Analysis of Data
Process
Know
Your Data
Focus the
Analysis
Categorize
Information
Identify
Patterns
Interpretation
(Source: Taylor-Powell & Renner, 2010)
• Identify
themes
• Organize
them into
coherent
emergent
categories
• Larger
categories
• Relative
importance
• Relationships
Analysis of Data
K-Mean Analysis
How often a card is placed in a category
Single-linked dendrogram
Analysis of Data
Average linkage dendrogram
Complete linkage dendrogram
Dissimilar matrix
Analysis of Data
Two-dimensional multidimensional scaling analysis
Co-occurrence matrix
Originality/Value
The tool provides an original approach to capturing trainee teachers’
perceptions of Computational Thinking.
Questions
References
Berg, E., A. (1948) A Simple Objective Technique for Measuring Flexibility in Thinking. The Journal of General
Psychology. 39(52) pp.15-22.
Coxon, A. and MacMillan, P. (1999) Sorting Data: Collection and Analysis. Thousand Oaks, CA: Sage Publications.
Department for Education (2013a) Computer Science to be included in the EBacc [Online]. Available at:
http://www.education.gov.uk/inthenews/inthenews/a00221085/ebacccompsci (Accessed: 15 April 2013).
Department for Education (2013b) Computing Programmes of study for key stages 1-4 [Online]. Available at:
http://media.education.gov.uk/assets/files/pdf/c/computing%2004-02-13_001.pdf (Accessed: 15 April 2013).
Martin, B. and Hanington, B. (2012) 100 ways to Research Complex Problems, Develop Innovative Ideas and
Design Effective Solutions. Beverley, MA: Rockport Publications, pp.26-27.
Nawaz, A. (2012). A Comparison of Card-sorting Analysis Methods. The 10th Asia Pacific Conference on Computer
Human Interaction (APCHI2012).
Nielsen, J. (2004) Card Sorting How Many Users to Test? [Online]. Available at:
http://www.nngroup.com/articles/card-sorting-how-many-users-to-test/ (Accessed: 15 April 2013).
Nielsen, J. (1995) Usability Testing for the 1995 Sun Microsystems' Website. [Online]. Available at:
http://www.nngroup.com/articles/card-sorting-how-many-users-to-test/ (Accessed: 15 April 2013).
Ross, J. (2011) Comparing User Research Methods for Information Architecture. [Online]. Available at:
http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf (Accessed: 15 April 2013).
Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media.
Spencer, D. and Warfel, T. (2004) Card sorting: A Definitive Guide. [Online]. Available at:
http://boxesandarrows.com/card-sorting-a-definitive-guide/ (Accessed: 15 April 2013).
Taylor-Powell, E. and Renner, M. (2010) Analyzing Qualitative Data. Madison: University of Winconsin. [Online].
Available at: http://learningstore.uwex.edu/assets/pdfs/g3658-12.pdf (Accessed: 15 April 2013).
Tullis, T. and Wood, L. (2004) How many users are enough for a card-sorting study? In Proceedings UPA'2004,
Minneapolis, MN. Available at: http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf
(Accessed: 15 April 2013).
Wing, J.,M. (2006) Computation Thinking. Communication of the ACM. 49(3) pp. 33-35 [Online]. Available at:
http://www.cs.cmu.edu/afs/cs/usr/wing/www/publications/Wing06.pdf (Accessed: 15 April 2013).
References
Fincher, S. and Teneberg, J. (2005) Making sense of card sorting data. Expert Systems, 22(32), pp.89-93.
Hannah, S. (2005) Sorting Out Card Sorting: Comparing Methods for Information Architects, Usability Specialists
and Other Practitioners. Available at:
https://scholarsbank.uoregon.edu/xmlui/bitstream/handle/1794/7818/2005-hannah.pdf?sequence=1 (Accessed:
15 April 2013).
Packer, T., L., Boshoff, K. and DeJonge, D. (2008) Development of the Activity Card Sort — Australia. Australian
Occupational Therapy Journal, 55(3), pp.199-206.
Rugg, G. and McGeorge, P. (1997) The sorting techniques: a tutorial paper on card sorts, picture sorts and item
sorts. Expert Systems, 14(2).
Santos, G., J. (2006) Card sort technique as a qualitative substitute for quantitative exploratory factor analysis.
Corporate Communications: An International Journal, 11(3), pp.288 – 302.
Tetting, D. (1988) Q-Sort Update. Western Journal of Nursing Research, 10(6), pp.757-765.
Tullis, T. and Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing and Presenting Usability
Metrics. New York: Elsevier: Morgan Kaufmann.

Presentation pick a card - newman 17-04-13 - final

  • 1.
    Pick a Card DiscoveringStudent’s Perceptions Andrew Csizmadia Wednesday 17th April 2013
  • 2.
    Purpose • Discover trainee ICTComputer Science teachers’ perception of Computational Thinking
  • 3.
    Relevance “We need tobring computational thinking into our schools.” (DfE, 2013a) “Computational thinking is a fundamental skill for everyone, not just for computer scientists. To reading, writing and arithmetic, we should add computational thinking to every child’s analytical ability.“ (Jeannette Wing, 2006) “…change the world through computational thinking.” (DfE, 2013b)
  • 4.
  • 5.
    What is CardSorting? • Qualitative participatory design technique, used to explore how participants group items together into categories and relate concepts to one another (Martin & Hanington, 2012) • Tool to determine participant’s mental model of grouping items (Ross, 2011) • Generative method (Nielsen, 2004) using open card sort • Evaluative method (Ross, 2011) using closed card sort • Generate a category tree or folksonomy
  • 6.
    Approach – CardSorting Advantages Disadvantages Simple Content-Centric technique Cheap Results may vary Quick to Execute Analysis – time consuming Established May capture “surface” characteristics only Involves users Provides a good foundation (Source: Spencer & Warfel, 2004)
  • 7.
    Design/Methodology/Approach Card Selection • 104Computational Thinking related cards identified following desk based literature review Recommended Time • Allow 30 minutes for each multiple of 50 cards (Martin & Hanington, 2012) Recommended Sample Size • Between 15 (Nielsen, 2004) and 30 (Tullis & Wood, 2004) provides a correlation between 0.90 and 0.95 respectively
  • 8.
    Design/Methodology/Approach Card Selection • 104Computational Thinking related cards identified following desk based literature review Recommended Time • Allow 30 minutes for each multiple of 50 cards (Martin & Hanington, 2012) Recommended Sample Size • Between 15 (Nielsen, 2004) and 30 (Tullis & Wood, 2004) provides a correlation between 0.90 and 0.95 respectively
  • 9.
  • 10.
    Analysis of Data Process Know YourData Focus the Analysis Categorize Information Identify Patterns Interpretation (Source: Taylor-Powell & Renner, 2010) • Identify themes • Organize them into coherent emergent categories • Larger categories • Relative importance • Relationships
  • 11.
    Analysis of Data K-MeanAnalysis How often a card is placed in a category Single-linked dendrogram
  • 12.
    Analysis of Data Averagelinkage dendrogram Complete linkage dendrogram Dissimilar matrix
  • 13.
    Analysis of Data Two-dimensionalmultidimensional scaling analysis Co-occurrence matrix
  • 14.
    Originality/Value The tool providesan original approach to capturing trainee teachers’ perceptions of Computational Thinking.
  • 15.
  • 16.
    References Berg, E., A.(1948) A Simple Objective Technique for Measuring Flexibility in Thinking. The Journal of General Psychology. 39(52) pp.15-22. Coxon, A. and MacMillan, P. (1999) Sorting Data: Collection and Analysis. Thousand Oaks, CA: Sage Publications. Department for Education (2013a) Computer Science to be included in the EBacc [Online]. Available at: http://www.education.gov.uk/inthenews/inthenews/a00221085/ebacccompsci (Accessed: 15 April 2013). Department for Education (2013b) Computing Programmes of study for key stages 1-4 [Online]. Available at: http://media.education.gov.uk/assets/files/pdf/c/computing%2004-02-13_001.pdf (Accessed: 15 April 2013). Martin, B. and Hanington, B. (2012) 100 ways to Research Complex Problems, Develop Innovative Ideas and Design Effective Solutions. Beverley, MA: Rockport Publications, pp.26-27. Nawaz, A. (2012). A Comparison of Card-sorting Analysis Methods. The 10th Asia Pacific Conference on Computer Human Interaction (APCHI2012). Nielsen, J. (2004) Card Sorting How Many Users to Test? [Online]. Available at: http://www.nngroup.com/articles/card-sorting-how-many-users-to-test/ (Accessed: 15 April 2013). Nielsen, J. (1995) Usability Testing for the 1995 Sun Microsystems' Website. [Online]. Available at: http://www.nngroup.com/articles/card-sorting-how-many-users-to-test/ (Accessed: 15 April 2013). Ross, J. (2011) Comparing User Research Methods for Information Architecture. [Online]. Available at: http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf (Accessed: 15 April 2013). Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media. Spencer, D. and Warfel, T. (2004) Card sorting: A Definitive Guide. [Online]. Available at: http://boxesandarrows.com/card-sorting-a-definitive-guide/ (Accessed: 15 April 2013). Taylor-Powell, E. and Renner, M. (2010) Analyzing Qualitative Data. Madison: University of Winconsin. [Online]. Available at: http://learningstore.uwex.edu/assets/pdfs/g3658-12.pdf (Accessed: 15 April 2013). Tullis, T. and Wood, L. (2004) How many users are enough for a card-sorting study? In Proceedings UPA'2004, Minneapolis, MN. Available at: http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf (Accessed: 15 April 2013). Wing, J.,M. (2006) Computation Thinking. Communication of the ACM. 49(3) pp. 33-35 [Online]. Available at: http://www.cs.cmu.edu/afs/cs/usr/wing/www/publications/Wing06.pdf (Accessed: 15 April 2013).
  • 17.
    References Fincher, S. andTeneberg, J. (2005) Making sense of card sorting data. Expert Systems, 22(32), pp.89-93. Hannah, S. (2005) Sorting Out Card Sorting: Comparing Methods for Information Architects, Usability Specialists and Other Practitioners. Available at: https://scholarsbank.uoregon.edu/xmlui/bitstream/handle/1794/7818/2005-hannah.pdf?sequence=1 (Accessed: 15 April 2013). Packer, T., L., Boshoff, K. and DeJonge, D. (2008) Development of the Activity Card Sort — Australia. Australian Occupational Therapy Journal, 55(3), pp.199-206. Rugg, G. and McGeorge, P. (1997) The sorting techniques: a tutorial paper on card sorts, picture sorts and item sorts. Expert Systems, 14(2). Santos, G., J. (2006) Card sort technique as a qualitative substitute for quantitative exploratory factor analysis. Corporate Communications: An International Journal, 11(3), pp.288 – 302. Tetting, D. (1988) Q-Sort Update. Western Journal of Nursing Research, 10(6), pp.757-765. Tullis, T. and Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing and Presenting Usability Metrics. New York: Elsevier: Morgan Kaufmann.

Editor's Notes

  • #6 folksonomy is a system of classification derived from the practice and method of collaboratively creating and managing tags to annotate and categorizecontent this practice is also known as collaborative tagging,[3]social classification, social indexing, and social tagging. Folksonomy, a term coined by Thomas Vander Wal,
  • #7 AdvantagesSimple – Card sorts are easy for the organizer and the participants. Cheap – Typically the cost is a stack of 3×5 index cards, sticky notes, a pen or printing labels, and your time. Quick to execute – You can perform several sorts in a short period of time, which provides you with a significant amount of data.Established – The technique has been used for over 10 years, by many designers. Involves users – Because the information structure suggested by a card sort is based on real user input, not the gut feeling or strong opinions of a designer, information architect, or key stakeholder, it should be easier to use. Provides a good foundation – It’s not a silver bullet, but it does provide a good foundation for the structure of a site or product.DisadvantagesDoes not consider users’ tasks – Card sorting is an inherently content-centric technique. If used without considering users’ tasks, it may lead to an information structure that is not usable when users are attempting real tasks. An information needs analysis or task analysis is necessary to ensure that the content being sorted meets user needs and that the resulting information structure allows users to achieve tasks.Results may vary –The card sort may provide fairly consistent results between participants, or may vary widely. Analysis can be time consuming – The sorting is quick, but the analysis of the data can be difficult and time consuming, particularly if there is little consistency between participants.May capture “surface” characteristics only – Participants may not consider what the content is about or how they would use it to complete a task and may just sort it by surface characteristics such as document types.