Slides for the presentation at the Seminar Series on Advanced Techniques & Tools for Software Evolution (SATToSE), which took place in Madrid (Spain) on 7–9 June 2017.

1. 1/35
SATToSE 17, Madrid
Can we Measure Computational Thinking with
Tools? Present and Future of Dr. Scratch
Jes´us Moreno-Le´on, Gregorio Robles,
Marcos Rom´an-Gonz´alez
jesus.moreno@programamos.es, grex@gsyc.urjc.es, mroman@edu.uned.es
GSyC/Libresoft, Universidad Rey Juan Carlos
SATToSE, Madrid, June 7th, 2017
Jes´us Moreno-Le´on Present and Future of Dr. Scratch

2. 2/35
SATToSE 17, Madrid
(cc) 2017 Jes´us Moreno Le´on, Gregorio Robles and Marcos Rom´an Gonz´alez
Some rights reserved. This work licensed under Creative Commons
Attribution-ShareAlike License. To view a copy of full license, see
http://creativecommons.org/licenses/by-sa/3.0/ or write to
Creative Commons, 559 Nathan Abbott Way, Stanford,
California 94305, USA.
Some of the figures have been taken from the Internet
Source, and author and licence if known, is specified.
For those images, fair use applies.
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

3. 3/35
SATToSE 17, Madrid
What is Dr. Scratch?
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

4. 4/35
SATToSE 17, Madrid
Why automatic analysis? (Teacher perspective)
Enjoying while marking students’ projects
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

5. 5/35
SATToSE 17, Madrid
Why automatic analysis? (Learner perspective)
Analyzing a Python program with Pylint
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

6. 6/35
SATToSE 17, Madrid
Remixing other researchers’ ideas
Analysis of Scratch projects
Scrape: visual representation of the blocks used (and not
used).
Hairball: static analyzer of Scratch projects to detect errors.
Brennan & Resnick: New frameworks for studying and
assessing the development of CT.
Seiter & Foreman: Progression of Early CT Model.
Wilson, Hainey & Connolly: Evaluation of games to gauge
understanding of programming concepts.
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

7. 7/35
SATToSE 17, Madrid
Assessment of CT development
CT dimension Basic Intermediate Proficient
Logical Thinking if if else logic operations
Data representation modifiers of object
properties
variables lists
User interactivity green flag keyboard, mouse, ask
and wait
webcam, input sound
Control flow sequence of blocks repeat, forever repeat until
Abstraction and
problem decomposi-
tion
more than one script use of custom blocks use of ’clones’ (in-
stances of sprites)
Parallelism two scripts on green
flag
two scripts on key
pressed or sprite
clicked
two scripts on receive
message, video/audio
input, backdrop
change
Synchronization wait message broad-
cast, stop all, stop
program
wait until, when
backdrop changes to,
broadcast and wait
Table: Level of development for each CT dimension
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

8. 8/35
SATToSE 17, Madrid
Assessment of CT development: Logical Thinking
Different levels of development of logical thinking: basic (top),
developing (center) and proficient (bottom).
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

9. 9/35
SATToSE 17, Madrid
Assessment of CT development: Data Representation
Different levels of development of data representation: basic (top),
developing (center) and proficient (bottom).
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

10. 10/35
SATToSE 17, Madrid
Code smells (I)
Errors or bad programming habits
Dead code
Attribute initialization
Default names
Repeated scripts
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

11. 11/35
SATToSE 17, Madrid
Code smells (II)
Bad/default naming of sprites
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

12. 12/35
SATToSE 17, Madrid
Code smells (and III)
Example of repeated code Solution to avoid repeated code
Blocks should be created to
avoid repetition of code
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

13. 13/35
SATToSE 17, Madrid
Goals
Review the current state of the validation
process of the tool
Show examples to illustrate the kind of
software evolution investigations that could
be performed using the tool
Background picture: rebel-performance.com
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

14. 14/35
SATToSE 17, Madrid
Validation Process: Ecological Validity (I)
Ecological validity
Are young learners able to analyze their projects
and independently learn from the tips that the
tool provides?
Workshops with over 100 students (10 to 14
years) in 8 schools
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

15. 15/35
SATToSE 17, Madrid
Validation Process: Ecological Validity (II)
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

16. 16/35
SATToSE 17, Madrid
Validation Process: Ecological Validity (and III)
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

17. 17/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (I)
Convergent validity
Comparison of the evaluations provided by Dr.
Scratch with other measurements of similar
constructs
(Human) expert evaluators
Software engineering complexity metrics
CT-test
Background picture: Joanna Bourne
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

18. 18/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (II)
Comparison with human experts - Dr. Scratch contest award
ceremony at Google Campus, Madrid (Spain)
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

19. 19/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (III)
Scatter plot for experts evaluation (x-axis) and Dr. Scratch
assessment (y-axis).
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

20. 20/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (IV)
Scatter plot for Dr. Scratch assessment (x-axis) and Cyclomatic
Complexity (y-axis).
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

21. 21/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (V)
Scatter plot for Dr. Scratch assessment (x-axis) and Halstead’s
vocabulary (y-axis).
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

22. 22/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (VI)
One of the CT-test items
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

23. 23/35
SATToSE 17, Madrid
Validation Process: Convergent Validity (and VII)
Scatter plot for Dr. Scratch assessment (x-axis) and CT-test
(y-axis).
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

24. 24/35
SATToSE 17, Madrid
Validation Process: Discriminant Validity (I)
Discriminant validity
Projects shared in the Scratch repository are
categorized under one or more project types:
games, animations, music, art and stories.
Is Dr. Scratch able to detect differences in the
CT dimensions developed when programming
different types of Scratch projects?
Background picture: Bodie Pyndus
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

25. 25/35
SATToSE 17, Madrid
Validation Process: Discriminant Validity (II)
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

26. 26/35
SATToSE 17, Madrid
Validation Process: Discriminant Validity (and III)
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

27. 27/35
SATToSE 17, Madrid
Validation Process: Face Validity (I)
Face validity
Educators won’t use the tool unless they feel
that Dr. Scratch measures what it promises (i.e.
assessing CT)
Over 400 teachers who participate in a 40-hours
Scratch coding training course have been
surveyed
Background picture: janmennens
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

28. 28/35
SATToSE 17, Madrid
Validation Process: Face Validity (and II)
The CT score provided by Dr. Scratch is accurate
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

29. 29/35
SATToSE 17, Madrid
Validation Process: Factorial Validity
Factorial validity
Study potential relationships between the CT
dimensions assessed by Dr. Scratch
Simplification of the feedback report?
Background picture: jennyrotten
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

30. 30/35
SATToSE 17, Madrid
Limitations
Teachers should not rely exclusively on Dr. Scratch
Fundamental CT skills not assessed: debugging
and remixing.
Functionality or creativity not evaluated.
Portfolio analysis would be more accurate.
Background picture: Robert Couse-Baker
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

31. 31/35
SATToSE 17, Madrid
Future Work
1 User accounts
2 Teacher dashboard
3 Organization dashboard
Background picture: Simon Cunningham
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

32. 32/35
SATToSE 17, Madrid
Links with Software Evolution (I)
Tools like Dr. Scratch offer the possibility of
easily tracking learners progression and
projects evolution, both in terms of software
complexity and presence of bad smells
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

33. 33/35
SATToSE 17, Madrid
Links with Software Evolution (II)
Examining the Relationship between Socialization and Improved Software
Development Skills in the Scratch Code Learning Environment. Moreno-Le´on
J., Robles G. & Rom´an-Gonz´alez M. Journal of Universal Computer Science 22
(12), pages 1533-1557, 2016.
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

34. 34/35
SATToSE 17, Madrid
Links with Software Evolution (and III)
Software clones in scratch projects: on the presence of copy-and-paste in
computational thinking learning. Robles, G., Moreno-Le´on, J., Aivaloglou, E.,
& Hermans, F. In 2017 IEEE 11th International Workshop on Software Clones
(IWSC), pages 1-7, 2017.
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

35. 35/35
SATToSE 17, Madrid
Learn more
Dr. Scratch references
Moreno, J., & Robles, G. (2014). Automatic detection of bad programming
habits in scratch: A preliminary study. In Frontiers in Education Conference
(FIE), 2014 IEEE (pp. 1-4). IEEE.
Moreno-Le´on, J., Robles, G, & Roman-Gonz´alez, M. (2015). Dr. Scratch:
Automatic Analysis of Scratch Projects to Assess and Foster Computational
Thinking. RED. Revista de Educaci´on a Distancia, 15(46).
Moreno-Le´on, J., Robles, G, & Roman-Gonz´alez, M. (2016). Comparing
computational thinking development assessment scores with software complexity
metrics. In Global Engineering Education Conference (EDUCON), 2016 IEEE
(pp. 1040-1045). IEEE.
Moreno-Le´on, J., Roman-Gonz´alez, M., Harteveld, C. & Robles, G. (2017). On
the Automatic Assessment of Computational Thinking Skills: A Comparison
with Human Experts. In Proceedings of the 2017 CHI Conference Extended
Abstracts on Human Factors in Computing Systems. (pp. 2788-2795). ACM.
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch

36. 36/35
SATToSE 17, Madrid
Present and Future of Dr. Scratch
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez
jesus.moreno@programamos.es, grex@gsyc.urjc.es, mroman@edu.uned.es
GSyC/Libresoft, Universidad Rey Juan Carlos
SATToSE, Madrid, June 7th, 2017
J. Moreno-Le´on, G. Robles & M. Rom´an-Gonz´alez Present and Future of Dr. Scratch