The document analyzes whether the First LEGO League (FLL) robotics competition rubric evaluates 21st century skills. Through an expert review of the FLL rubric and literature on 21st century skills frameworks, the study found that the rubric adequately assesses communication, problem solving, critical thinking, and developing quality products through its evaluation of projects, robot design, and teamwork. Areas like ICT literacy and social/cultural skills were also found to be evaluated to a lesser extent. The researchers concluded that the FLL rubric provides an effective means of reinforcing and measuring key 21st century skills.

1. Are 21st Century Skills Evaluated in Robotics
Competitions?
The Case of First LEGO League Competition
Mireia Usart, Despoina Schina, Vanessa Esteve-Gonzalez & Mercè Gisbert
mireia.usart@urv.cat
Applied Research Group in
Education and Technology
Marie Skłodowska
Curie No. 713679

2. 1. Introduction
2. Our Study
3. Methodology
4. Results
5. Discussion
6. Conclusions
Contents

3. 1. Introduction
The Voogt and Pareja Roblin (2012) meta-analysis of 6 frameworks
• P21 (2007)
• EnGauge 21st century skills
• Assessment and teaching of 21st century skills (ATCS),
• (NETS/ISTE) framework
• Competences for new millennium
• learners by Organization for Economic Cooperation
• and Development (OECD, 2005),
• Center for Social and Economic Research (Gordon et al., 2009).
Romero, M., Usart, M. & Ott, M., (2015)

4. 1. Introduction
Educational Robotics
Robotic Competitions
First Lego League
STEAM (ER) COMPETITION
—> Develop a Scientific project & product
Primary and secondary school students’ engagement with ER
has the potential to transmit key skills for future citizens and
equip them with 21st century skills
(Eguchi, 2013; Afari and Khine, 2017).

5. 1. Introduction
Evaluates
?
• Digital-Age literacy
• Inventive thinking
• Effective communication
• High Productivity
Littterature review:
no research has been
published in the
topic of skills’
evaluation in ER.

6. 2. Our Study
! Main Aim
Explore whether 21st century skills are assessed in the context of the
FLL competition.
! Research Question
Which are the 21st Century Skills assessed using the FLL Rubric?

7. 3. Methodology
! Instruments:
Mentioned in all
Frameworks
Mentioned in most
Frameworks
• Communication
• Collaboration
• ICT literacy
• Social and/or cultural
skills
• Creativity
• Critical Thinking
• Problem solving
• Develop quality
products/productivity
21st Century Skills (8 top list)
Skill areas and skills in each of the 3 Categories:
Core Values (an evaluation based on how the team members work
together to accomplish a given task.)
Project (teams carry out a research, regarding a real-world and
present their research and solutions.)
Robot Design (design & present innovative solution)
FLL Evaluation Rubrics

8. 3. Methodology
! Context & Sample:
Participants:
! 4 experts from Spain and Greece
! 25-52 years, females, ET experts
! Teachers, judges and researchers
Cross 21st century skills - Rubric
! Based on the definitions
! Qualitative study
Expert Evaluation

9. 4. Results
RQ: Can 21st Century Skills be evaluated using the FLL Rubric?
0
3
6
9
12
15
18
Research
Innovative
solution
Presentation
M
echanicalDesign
Program
m
ingStrategy
&
Innovation
Inspiration
Team
w
orkG
racious
Professionalism
®
Number of 21st-century skills
Project
Robot Design
Core Values
Skill areas / Categories

10. 4. Results
RQ: Which are the 21st Century Skills evaluated using the FLL Rubric?
21st-century skills distribution in FLL rubric
Communication
Collaboration
Social /Cultural Skills
ICT litteracy
Creativity
Critical Thinking
Problem solving
Develop quality products
0 4 7 11 14
10
14
4
9
6
10
8
14

11. 5. Discussion
RQ: Which are the 21st Century Skills evaluated using the FLL Rubric?
21st century skills
FLL Categories and skill areas
most evaluated
Communication Project category, and
Presentation skill area in particular
evaluate this skill.
Core Values category (scoring
higher in our study), with
Inspiration and Teamwork to be
the skill areas most closely
related to this 21st century skill.
Problem
Solving
Programming and Strategy and
Innovation skill areas (from Robot
Design category) are the areas that
are most closely related to problem
solving.
Develop Quality
Products
Both Robot Design and Project
(Innovative Solution and Inspiration
skill areas) focus on planning and
managing useful robots for real needs
that are tested during the competition.
Social/Cultural
skills
As in the above-mentioned
Communication skill, Social/Cultural
skills are mainly measured in Project
and Core Values categories.
• Digital-Age literacy (“Multicultural literacy”)
• Inventive thinking
• Effective communication
• High Productivity
x
v
v
v

12. The 21st century skills are
assessed in the FLL competition
through the evaluation rubric for
each category:
Robot Design
Core Values
Project
6. Conclusions
RQ: Which are the 21st Century Skills evaluated using the FLL Rubric?
This exploratory study is a first step in the research of how 21st century skills
are reinforced and can be evaluated in the context of ER.
Further research will include the integration and the evaluation of 21st
century skills in teacher training for ER.
With the evaluation
rubric giving particular
emphasis on
evaluation of:
+Communication
Problem Solving
-ICT Literacy
CriticalThinking
The FLL rubric is a 21st
century skills evaluation
instrument.
FLL coaches and experts
should take into account
when delivering ER training
sessions.

13. References
Alimisis, D.: Educational robotics: open questions and new challenges. Themes Sci. Technol. Educ. 6(1), 63–71 (2013).
Chalmers, C.: Learning with FIRST LEGO League. In: Society for Information Technology and Teacher Education (SITE)
Conference, pp. 5118–5124. Association for the Advancement of Computing in Education (AACE), New Orleans,
(2013).
Chen X.: How does participation in FIRST LEGO League robotics competition impact children’s problem-solving process?.
In: Lepuschitz W., Merdan M., Koppensteiner G., Balogh R., Obdržálek D. (eds) Robotics in Education. RiE 2018.
Advances in Intelligent Systems and Computing, vol 829. Springer, Cham (2018).
Eguchi, A.: RoboCupJunior for promoting STEM education, 21st century skills, and technological advancement through
robotics competition. Rob. Auton. Syst. 75, 692–699 (2015).
Kickmeier-Rust, M. D., Dietrich, A., 2012. A domain model for smart 21st century skills training in gamebased virtual worlds.
In ICALT 2012 Proceedings. Rome, pp. 680–681.Kaloti-Hallak, F., Armoni, M., Ben-Ari, M.: Students’ attitudes and
motivation during robotics activities. In: Workshop in Primary and Secondary Computing Education, pp. 102–110
(2015).
Romero, M., Usart, M., Ott, M., 2015. Can Serious Games Contribute to Developing and Sustaining 21st Century Skills?
Games and Culture, pp. 148-177.
Voogt, J., Pareja Roblin, N., 2012. A comparative analysis of international frameworks for 21st century competences:
Implications for national curriculum policies, Journal of Curriculum Studies, 44, pp. 299 321.

14. Applied Research Group in
Education and Technology
Marie Skłodowska
Curie No. 713679
Questions?
Thanks!
Mireia Usart, Despoina Schina, Vanessa Esteve-Gonzalez & Mercè Gisbert
mireia.usart@urv.cat