TEEM'16 - Track 1 Computational thinking in pre-university education Authors: Ernest Redondo, Joaquim Regot, David Fonseca, Francesc Valls and Lluís Giménez https://youtu.be/A4sAuTuIs7Y

1. What is the question?
IMPACT OF QUESTION LENGTH AND ILLUSTRATION SUPPORT ON
THE SUCCESS AND SKIP RATES OF PRE-UNIVERSITY MOCK ONLINE
TESTS

2. Computational thinking beyond CS
Students of technical drawing are required to possess a computational thinking mindset (Wing,
2006)
◦ Require advanced spatial reasoning
◦ Solve complex problems
◦ Use abstraction and decomposition
Which piece fits perfectly the one on the left?

3. Online quizzes as educational tools
Quizzes are standardized tests with a restricted set of possible answers, allowing easier (and
often automated) grading.
In the European Higher Education Area, the Bologna process emphasizes the importance of self-
learning.
How can online quizzes be useful in education, beyond grading?
◦ Students can self-assess their learning process anytime, anywhere and any number of times
◦ Receive immediate feedback on their performance

4. Case of study
This paper analyzes over 20,000 responses of the preparatory online quiz for the Spanish
University Admission Test (Selectividad) of the technical drawing subject.
The Selectividad is closely related to other admission tests to colleges and universities, such as
SAT in the US, A Level in the UK, Abitur in Germany or Baccalauréat in France.
With the objective of improving future iterations of the test, the authors wanted to research the
variables that:
◦ Increased the number of successful attempts
◦ Lowered the number of questions skipped

5. Experimental data
The analyzed data included the following variables on the results of the online mock test:
◦ Year collected (2009 or 2015)
◦ Intake (June or September)
◦ Summary of the number of responses passed, failed or disregarded (per question)
In addition, more data was derived from the content of the question:
◦ Whether the question was only textual or included an illustration
◦ Length of the question (including all responses), as a proxy of reading comprehension difficulty
◦ Knowledge subdomain inside the subject of technical drawing

6. Results: Influence of image support
To research the influence of image support, a
series of density graphs were produced
showing the pass, fail and skip rates when the
question was text only or had the support of
an illustration in two the years studied:
◦ The skip rate was much lower when the question
had image support
◦ The pass ratio was higher when a figure was
present
◦ When overlaying the same graph for both years
the results were very similar regardless of the 6
years elapsed

7. Results: Performance across years
To reveal the changes across the 6-year span, a
dot plot was used, depicting the pass, fail and
skip rates of each question, reflecting changes
in educational methodology or student profile
between those years:
◦ On 2009 the questions were more successfully
answered and less likely to be skipped than 6
years later
◦ Questions with image support showed higher
success rates overall, and their skip ratio was
much lower (and in many cases close to zero)

8. Results: Performance between intakes
The relationship between the results in both
intakes was examined in detail in a graph that
combined a violin plot with a box plot:
◦ On both years the fail rate was higher in the
September intake (the students who take the
September exam are the ones who did not pass
the June exam)
◦ The skip rates and pass rates of both years were
reversed, suggesting that September students
were more inclined to answer on 2015 than on
2009 but, were less successful in their attempts

9. Results: Influence of question length
To measure reading comprehension, the
studied ratios were plotted against question
length as scatterplots, weighted by number of
responses:
◦ The success ratio slightly decreased from 2009
to 2015, but the skip ratio was almost the same
on both years
◦ The success ratio greatly increased when a
picture was included
◦ The students tended to skip longer questions
more often
◦ The questions with illustrations were rarely
skipped

10. Conclusions on the results
Questions with image support received more successful attempts (i.e. students selected the
correct answer) while at the same time significantly reduced the percentage of questions
skipped, compared to the questions without images.
Students were more confident about their knowledge when dealing questions with image
support (were skipped less often), and therefore these questions were:
◦ Less difficult
◦ Easier to understand
◦ More capable of motivating the students
As the length of the question increased (as a proxy of reading comprehension difficulty), the
success rate decreased while the skip rate increased.

11. Conclusions on quiz design
The findings suggest that students struggled to fully understand the longer questions or failed to
read all the answers of the multiple choice test. The fact that longer questions were skipped
more often supports this hypothesis, because students might abandon the question if they
perceived that it was too long either because they felt they would leave less time to answer the
shorter questions, or because they had trouble understanding what they were asked about.
Identifying the causes of these results should allow the authors to either include more questions
with image support or rephrase the non-graphic questions to be shorter or more easily
comprehended.

12. Conclusions on student profile
Although the questions in 2009 and 2015 were the same, the analysis of the variation across the
6 years between studies showed a reduction in the success ratio while the skip ratio stayed
relatively the same.
The authors suggest that while the students have quickly adopted new mobile and
communication technologies, the curriculum they are taught and the way they are tested has
not adapted yet, and this disconnect manifests in lower success rates.

13. Future work
While current results have provided insight and further research questions using just statistical
graphical methods, future studies will use statistical hypothesis testing to provide more robust
conclusions while at the same time providing a measure of the influence of each factor using
multiple regression analysis.
To measure question complexity more accurately, the semantic analysis of the content of the
question will also be explored in future studies.

14. Thank You!
Research supported by the Non-Oriented Fundamental Research Project EDU2012-37247/EDUC
of the VI National Plan for Scientific Research, Development and Technological Innovation 2008-2011, Government of Spain,
titled “E-learning 3.0 in the teaching of architecture. Case studies of educational research for the foreseeable future”
David Fonseca
Architecture School – La Salle Universitat Ramon Llull
fonsi@salle.url.edu
Ernest Redondo, Joaquim Regot, Francesc Valls, Lluís Giménez
Barcelona School of Architecture – BarcelonaTech
{ernesto.redondo, joquim.regot, francesc.valls, lluis.gimenez}@upc.edu