Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1/35
SATToSE 17, Madrid
Can we Measure Computational Thinking with
Tools? Present and Future of Dr. Scratch
Jes´us Moreno-...
2/35
SATToSE 17, Madrid
(cc) 2017 Jes´us Moreno Le´on, Gregorio Robles and Marcos Rom´an Gonz´alez
Some rights reserved. T...
3/35
SATToSE 17, Madrid
What is Dr. Scratch?
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Sc...
4/35
SATToSE 17, Madrid
Why automatic analysis? (Teacher perspective)
Enjoying while marking students’ projects
J. Moreno-...
5/35
SATToSE 17, Madrid
Why automatic analysis? (Learner perspective)
Analyzing a Python program with Pylint
J. Moreno-Le´...
6/35
SATToSE 17, Madrid
Remixing other researchers’ ideas
Analysis of Scratch projects
Scrape: visual representation of th...
7/35
SATToSE 17, Madrid
Assessment of CT development
CT dimension Basic Intermediate Proficient
Logical Thinking if if else...
8/35
SATToSE 17, Madrid
Assessment of CT development: Logical Thinking
Different levels of development of logical thinking:...
9/35
SATToSE 17, Madrid
Assessment of CT development: Data Representation
Different levels of development of data represent...
10/35
SATToSE 17, Madrid
Code smells (I)
Errors or bad programming habits
Dead code
Attribute initialization
Default names...
11/35
SATToSE 17, Madrid
Code smells (II)
Bad/default naming of sprites
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez P...
12/35
SATToSE 17, Madrid
Code smells (and III)
Example of repeated code Solution to avoid repeated code
Blocks should be c...
13/35
SATToSE 17, Madrid
Goals
Review the current state of the validation
process of the tool
Show examples to illustrate ...
14/35
SATToSE 17, Madrid
Validation Process: Ecological Validity (I)
Ecological validity
Are young learners able to analyz...
15/35
SATToSE 17, Madrid
Validation Process: Ecological Validity (II)
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Pre...
16/35
SATToSE 17, Madrid
Validation Process: Ecological Validity (and III)
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´ale...
17/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (I)
Convergent validity
Comparison of the evaluations pro...
18/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (II)
Comparison with human experts - Dr. Scratch contest ...
19/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (III)
Scatter plot for experts evaluation (x-axis) and Dr...
20/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (IV)
Scatter plot for Dr. Scratch assessment (x-axis) and...
21/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (V)
Scatter plot for Dr. Scratch assessment (x-axis) and ...
22/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (VI)
One of the CT-test items
J. Moreno-Le´on, G. Robles ...
23/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (and VII)
Scatter plot for Dr. Scratch assessment (x-axis...
24/35
SATToSE 17, Madrid
Validation Process: Discriminant Validity (I)
Discriminant validity
Projects shared in the Scratc...
25/35
SATToSE 17, Madrid
Validation Process: Discriminant Validity (II)
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez P...
26/35
SATToSE 17, Madrid
Validation Process: Discriminant Validity (and III)
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´a...
27/35
SATToSE 17, Madrid
Validation Process: Face Validity (I)
Face validity
Educators won’t use the tool unless they feel...
28/35
SATToSE 17, Madrid
Validation Process: Face Validity (and II)
The CT score provided by Dr. Scratch is accurate
J. Mo...
29/35
SATToSE 17, Madrid
Validation Process: Factorial Validity
Factorial validity
Study potential relationships between t...
30/35
SATToSE 17, Madrid
Limitations
Teachers should not rely exclusively on Dr. Scratch
Fundamental CT skills not assesse...
31/35
SATToSE 17, Madrid
Future Work
1 User accounts
2 Teacher dashboard
3 Organization dashboard
Background picture: Simo...
32/35
SATToSE 17, Madrid
Links with Software Evolution (I)
Tools like Dr. Scratch offer the possibility of
easily tracking ...
33/35
SATToSE 17, Madrid
Links with Software Evolution (II)
Examining the Relationship between Socialization and Improved ...
34/35
SATToSE 17, Madrid
Links with Software Evolution (and III)
Software clones in scratch projects: on the presence of c...
35/35
SATToSE 17, Madrid
Learn more
Dr. Scratch references
Moreno, J., & Robles, G. (2014). Automatic detection of bad pro...
36/35
SATToSE 17, Madrid
Present and Future of Dr. Scratch
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez
jesus.moreno@p...
Upcoming SlideShare
Loading in …5
×

Can we Measure Computational Thinking with Tools? Present and Future of Dr. Scratch

605 views

Published on

Slides for the presentation at the Seminar Series on Advanced Techniques & Tools for Software Evolution (SATToSE), which took place in Madrid (Spain) on 7–9 June 2017.

Published in: Education
  • Be the first to comment

Can we Measure Computational Thinking with Tools? Present and Future of Dr. Scratch

  1. 1. 1/35 SATToSE 17, Madrid Can we Measure Computational Thinking with Tools? Present and Future of Dr. Scratch Jes´us Moreno-Le´on, Gregorio Robles, Marcos Rom´an-Gonz´alez jesus.moreno@programamos.es, grex@gsyc.urjc.es, mroman@edu.uned.es GSyC/Libresoft, Universidad Rey Juan Carlos SATToSE, Madrid, June 7th, 2017 Jes´us Moreno-Le´on Present and Future of Dr. Scratch
  2. 2. 2/35 SATToSE 17, Madrid (cc) 2017 Jes´us Moreno Le´on, Gregorio Robles and Marcos Rom´an Gonz´alez Some rights reserved. This work licensed under Creative Commons Attribution-ShareAlike License. To view a copy of full license, see http://creativecommons.org/licenses/by-sa/3.0/ or write to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. Some of the figures have been taken from the Internet Source, and author and licence if known, is specified. For those images, fair use applies. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  3. 3. 3/35 SATToSE 17, Madrid What is Dr. Scratch? J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  4. 4. 4/35 SATToSE 17, Madrid Why automatic analysis? (Teacher perspective) Enjoying while marking students’ projects J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  5. 5. 5/35 SATToSE 17, Madrid Why automatic analysis? (Learner perspective) Analyzing a Python program with Pylint J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  6. 6. 6/35 SATToSE 17, Madrid Remixing other researchers’ ideas Analysis of Scratch projects Scrape: visual representation of the blocks used (and not used). Hairball: static analyzer of Scratch projects to detect errors. Brennan & Resnick: New frameworks for studying and assessing the development of CT. Seiter & Foreman: Progression of Early CT Model. Wilson, Hainey & Connolly: Evaluation of games to gauge understanding of programming concepts. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  7. 7. 7/35 SATToSE 17, Madrid Assessment of CT development CT dimension Basic Intermediate Proficient Logical Thinking if if else logic operations Data representation modifiers of object properties variables lists User interactivity green flag keyboard, mouse, ask and wait webcam, input sound Control flow sequence of blocks repeat, forever repeat until Abstraction and problem decomposi- tion more than one script use of custom blocks use of ’clones’ (in- stances of sprites) Parallelism two scripts on green flag two scripts on key pressed or sprite clicked two scripts on receive message, video/audio input, backdrop change Synchronization wait message broad- cast, stop all, stop program wait until, when backdrop changes to, broadcast and wait Table: Level of development for each CT dimension J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  8. 8. 8/35 SATToSE 17, Madrid Assessment of CT development: Logical Thinking Different levels of development of logical thinking: basic (top), developing (center) and proficient (bottom). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  9. 9. 9/35 SATToSE 17, Madrid Assessment of CT development: Data Representation Different levels of development of data representation: basic (top), developing (center) and proficient (bottom). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  10. 10. 10/35 SATToSE 17, Madrid Code smells (I) Errors or bad programming habits Dead code Attribute initialization Default names Repeated scripts J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  11. 11. 11/35 SATToSE 17, Madrid Code smells (II) Bad/default naming of sprites J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  12. 12. 12/35 SATToSE 17, Madrid Code smells (and III) Example of repeated code Solution to avoid repeated code Blocks should be created to avoid repetition of code J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  13. 13. 13/35 SATToSE 17, Madrid Goals Review the current state of the validation process of the tool Show examples to illustrate the kind of software evolution investigations that could be performed using the tool Background picture: rebel-performance.com J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  14. 14. 14/35 SATToSE 17, Madrid Validation Process: Ecological Validity (I) Ecological validity Are young learners able to analyze their projects and independently learn from the tips that the tool provides? Workshops with over 100 students (10 to 14 years) in 8 schools J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  15. 15. 15/35 SATToSE 17, Madrid Validation Process: Ecological Validity (II) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  16. 16. 16/35 SATToSE 17, Madrid Validation Process: Ecological Validity (and III) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  17. 17. 17/35 SATToSE 17, Madrid Validation Process: Convergent Validity (I) Convergent validity Comparison of the evaluations provided by Dr. Scratch with other measurements of similar constructs (Human) expert evaluators Software engineering complexity metrics CT-test Background picture: Joanna Bourne J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  18. 18. 18/35 SATToSE 17, Madrid Validation Process: Convergent Validity (II) Comparison with human experts - Dr. Scratch contest award ceremony at Google Campus, Madrid (Spain) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  19. 19. 19/35 SATToSE 17, Madrid Validation Process: Convergent Validity (III) Scatter plot for experts evaluation (x-axis) and Dr. Scratch assessment (y-axis). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  20. 20. 20/35 SATToSE 17, Madrid Validation Process: Convergent Validity (IV) Scatter plot for Dr. Scratch assessment (x-axis) and Cyclomatic Complexity (y-axis). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  21. 21. 21/35 SATToSE 17, Madrid Validation Process: Convergent Validity (V) Scatter plot for Dr. Scratch assessment (x-axis) and Halstead’s vocabulary (y-axis). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  22. 22. 22/35 SATToSE 17, Madrid Validation Process: Convergent Validity (VI) One of the CT-test items J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  23. 23. 23/35 SATToSE 17, Madrid Validation Process: Convergent Validity (and VII) Scatter plot for Dr. Scratch assessment (x-axis) and CT-test (y-axis). J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  24. 24. 24/35 SATToSE 17, Madrid Validation Process: Discriminant Validity (I) Discriminant validity Projects shared in the Scratch repository are categorized under one or more project types: games, animations, music, art and stories. Is Dr. Scratch able to detect differences in the CT dimensions developed when programming different types of Scratch projects? Background picture: Bodie Pyndus J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  25. 25. 25/35 SATToSE 17, Madrid Validation Process: Discriminant Validity (II) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  26. 26. 26/35 SATToSE 17, Madrid Validation Process: Discriminant Validity (and III) J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  27. 27. 27/35 SATToSE 17, Madrid Validation Process: Face Validity (I) Face validity Educators won’t use the tool unless they feel that Dr. Scratch measures what it promises (i.e. assessing CT) Over 400 teachers who participate in a 40-hours Scratch coding training course have been surveyed Background picture: janmennens J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  28. 28. 28/35 SATToSE 17, Madrid Validation Process: Face Validity (and II) The CT score provided by Dr. Scratch is accurate J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  29. 29. 29/35 SATToSE 17, Madrid Validation Process: Factorial Validity Factorial validity Study potential relationships between the CT dimensions assessed by Dr. Scratch Simplification of the feedback report? Background picture: jennyrotten J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  30. 30. 30/35 SATToSE 17, Madrid Limitations Teachers should not rely exclusively on Dr. Scratch Fundamental CT skills not assessed: debugging and remixing. Functionality or creativity not evaluated. Portfolio analysis would be more accurate. Background picture: Robert Couse-Baker J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  31. 31. 31/35 SATToSE 17, Madrid Future Work 1 User accounts 2 Teacher dashboard 3 Organization dashboard Background picture: Simon Cunningham J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  32. 32. 32/35 SATToSE 17, Madrid Links with Software Evolution (I) Tools like Dr. Scratch offer the possibility of easily tracking learners progression and projects evolution, both in terms of software complexity and presence of bad smells J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  33. 33. 33/35 SATToSE 17, Madrid Links with Software Evolution (II) Examining the Relationship between Socialization and Improved Software Development Skills in the Scratch Code Learning Environment. Moreno-Le´on J., Robles G. & Rom´an-Gonz´alez M. Journal of Universal Computer Science 22 (12), pages 1533-1557, 2016. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  34. 34. 34/35 SATToSE 17, Madrid Links with Software Evolution (and III) Software clones in scratch projects: on the presence of copy-and-paste in computational thinking learning. Robles, G., Moreno-Le´on, J., Aivaloglou, E., & Hermans, F. In 2017 IEEE 11th International Workshop on Software Clones (IWSC), pages 1-7, 2017. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  35. 35. 35/35 SATToSE 17, Madrid Learn more Dr. Scratch references Moreno, J., & Robles, G. (2014). Automatic detection of bad programming habits in scratch: A preliminary study. In Frontiers in Education Conference (FIE), 2014 IEEE (pp. 1-4). IEEE. Moreno-Le´on, J., Robles, G, & Roman-Gonz´alez, M. (2015). Dr. Scratch: Automatic Analysis of Scratch Projects to Assess and Foster Computational Thinking. RED. Revista de Educaci´on a Distancia, 15(46). Moreno-Le´on, J., Robles, G, & Roman-Gonz´alez, M. (2016). Comparing computational thinking development assessment scores with software complexity metrics. In Global Engineering Education Conference (EDUCON), 2016 IEEE (pp. 1040-1045). IEEE. Moreno-Le´on, J., Roman-Gonz´alez, M., Harteveld, C. & Robles, G. (2017). On the Automatic Assessment of Computational Thinking Skills: A Comparison with Human Experts. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. (pp. 2788-2795). ACM. J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch
  36. 36. 36/35 SATToSE 17, Madrid Present and Future of Dr. Scratch J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez jesus.moreno@programamos.es, grex@gsyc.urjc.es, mroman@edu.uned.es GSyC/Libresoft, Universidad Rey Juan Carlos SATToSE, Madrid, June 7th, 2017 J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

×