3. Relevance
“We need to bring
computational thinking
into our schools.”
(DfE, 2013a)
“Computational thinking is a fundamental
skill for everyone, not just for computer
scientists. To reading, writing and
arithmetic, we should add computational
thinking to every child’s analytical ability.“
(Jeannette Wing, 2006)
“…change the world through
computational thinking.”
(DfE, 2013b)
5. What is Card Sorting?
• Qualitative participatory
design technique, used to
explore how participants
group items together into
categories and relate concepts
to one another (Martin &
Hanington, 2012)
• Tool to determine
participant’s mental model of
grouping items (Ross, 2011)
• Generative method (Nielsen,
2004) using open card sort
• Evaluative method (Ross,
2011) using closed card sort
• Generate a category tree or
folksonomy
6. Approach – Card Sorting
Advantages Disadvantages
Simple Content-Centric technique
Cheap Results may vary
Quick to Execute Analysis – time consuming
Established May capture “surface”
characteristics only
Involves users
Provides a good foundation
(Source: Spencer & Warfel, 2004)
7. Design/Methodology/Approach
Card Selection
• 104 Computational Thinking
related cards identified
following desk based
literature review
Recommended Time
• Allow 30 minutes for each
multiple of 50 cards (Martin
& Hanington, 2012)
Recommended Sample Size
• Between 15 (Nielsen, 2004)
and 30 (Tullis & Wood, 2004)
provides a correlation
between 0.90 and 0.95
respectively
8. Design/Methodology/Approach
Card Selection
• 104 Computational Thinking
related cards identified
following desk based
literature review
Recommended Time
• Allow 30 minutes for each
multiple of 50 cards (Martin
& Hanington, 2012)
Recommended Sample Size
• Between 15 (Nielsen, 2004)
and 30 (Tullis & Wood, 2004)
provides a correlation
between 0.90 and 0.95
respectively
10. Analysis of Data
Process
Know
Your Data
Focus the
Analysis
Categorize
Information
Identify
Patterns
Interpretation
(Source: Taylor-Powell & Renner, 2010)
• Identify
themes
• Organize
them into
coherent
emergent
categories
• Larger
categories
• Relative
importance
• Relationships
11. Analysis of Data
K-Mean Analysis
How often a card is placed in a category
Single-linked dendrogram
16. References
Berg, E., A. (1948) A Simple Objective Technique for Measuring Flexibility in Thinking. The Journal of General
Psychology. 39(52) pp.15-22.
Coxon, A. and MacMillan, P. (1999) Sorting Data: Collection and Analysis. Thousand Oaks, CA: Sage Publications.
Department for Education (2013a) Computer Science to be included in the EBacc [Online]. Available at:
http://www.education.gov.uk/inthenews/inthenews/a00221085/ebacccompsci (Accessed: 15 April 2013).
Department for Education (2013b) Computing Programmes of study for key stages 1-4 [Online]. Available at:
http://media.education.gov.uk/assets/files/pdf/c/computing%2004-02-13_001.pdf (Accessed: 15 April 2013).
Martin, B. and Hanington, B. (2012) 100 ways to Research Complex Problems, Develop Innovative Ideas and
Design Effective Solutions. Beverley, MA: Rockport Publications, pp.26-27.
Nawaz, A. (2012). A Comparison of Card-sorting Analysis Methods. The 10th Asia Pacific Conference on Computer
Human Interaction (APCHI2012).
Nielsen, J. (2004) Card Sorting How Many Users to Test? [Online]. Available at:
http://www.nngroup.com/articles/card-sorting-how-many-users-to-test/ (Accessed: 15 April 2013).
Nielsen, J. (1995) Usability Testing for the 1995 Sun Microsystems' Website. [Online]. Available at:
http://www.nngroup.com/articles/card-sorting-how-many-users-to-test/ (Accessed: 15 April 2013).
Ross, J. (2011) Comparing User Research Methods for Information Architecture. [Online]. Available at:
http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf (Accessed: 15 April 2013).
Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media.
Spencer, D. and Warfel, T. (2004) Card sorting: A Definitive Guide. [Online]. Available at:
http://boxesandarrows.com/card-sorting-a-definitive-guide/ (Accessed: 15 April 2013).
Taylor-Powell, E. and Renner, M. (2010) Analyzing Qualitative Data. Madison: University of Winconsin. [Online].
Available at: http://learningstore.uwex.edu/assets/pdfs/g3658-12.pdf (Accessed: 15 April 2013).
Tullis, T. and Wood, L. (2004) How many users are enough for a card-sorting study? In Proceedings UPA'2004,
Minneapolis, MN. Available at: http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf
(Accessed: 15 April 2013).
Wing, J.,M. (2006) Computation Thinking. Communication of the ACM. 49(3) pp. 33-35 [Online]. Available at:
http://www.cs.cmu.edu/afs/cs/usr/wing/www/publications/Wing06.pdf (Accessed: 15 April 2013).
17. References
Fincher, S. and Teneberg, J. (2005) Making sense of card sorting data. Expert Systems, 22(32), pp.89-93.
Hannah, S. (2005) Sorting Out Card Sorting: Comparing Methods for Information Architects, Usability Specialists
and Other Practitioners. Available at:
https://scholarsbank.uoregon.edu/xmlui/bitstream/handle/1794/7818/2005-hannah.pdf?sequence=1 (Accessed:
15 April 2013).
Packer, T., L., Boshoff, K. and DeJonge, D. (2008) Development of the Activity Card Sort — Australia. Australian
Occupational Therapy Journal, 55(3), pp.199-206.
Rugg, G. and McGeorge, P. (1997) The sorting techniques: a tutorial paper on card sorts, picture sorts and item
sorts. Expert Systems, 14(2).
Santos, G., J. (2006) Card sort technique as a qualitative substitute for quantitative exploratory factor analysis.
Corporate Communications: An International Journal, 11(3), pp.288 – 302.
Tetting, D. (1988) Q-Sort Update. Western Journal of Nursing Research, 10(6), pp.757-765.
Tullis, T. and Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing and Presenting Usability
Metrics. New York: Elsevier: Morgan Kaufmann.
Editor's Notes
folksonomy is a system of classification derived from the practice and method of collaboratively creating and managing tags to annotate and categorizecontent this practice is also known as collaborative tagging,[3]social classification, social indexing, and social tagging. Folksonomy, a term coined by Thomas Vander Wal,
AdvantagesSimple – Card sorts are easy for the organizer and the participants. Cheap – Typically the cost is a stack of 3×5 index cards, sticky notes, a pen or printing labels, and your time. Quick to execute – You can perform several sorts in a short period of time, which provides you with a significant amount of data.Established – The technique has been used for over 10 years, by many designers. Involves users – Because the information structure suggested by a card sort is based on real user input, not the gut feeling or strong opinions of a designer, information architect, or key stakeholder, it should be easier to use. Provides a good foundation – It’s not a silver bullet, but it does provide a good foundation for the structure of a site or product.DisadvantagesDoes not consider users’ tasks – Card sorting is an inherently content-centric technique. If used without considering users’ tasks, it may lead to an information structure that is not usable when users are attempting real tasks. An information needs analysis or task analysis is necessary to ensure that the content being sorted meets user needs and that the resulting information structure allows users to achieve tasks.Results may vary –The card sort may provide fairly consistent results between participants, or may vary widely. Analysis can be time consuming – The sorting is quick, but the analysis of the data can be difficult and time consuming, particularly if there is little consistency between participants.May capture “surface” characteristics only – Participants may not consider what the content is about or how they would use it to complete a task and may just sort it by surface characteristics such as document types.
