M. Agape, „Otto Academy - Educational Mentor PoV”, IMPACT EdTech Remote Schooling, Online, 26.04.2021.

1. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871275.
Neither the European Commission (EC) nor any person acting on behalf of the Commission is responsible for how the following information is used. The
views expressed in this document are the sole responsibility of the authors and do not necessarily reflect the views of the EC.
Otto Academy
Educational Mentor PoV

2. Summary
• Activities
• Results

3. Coaching sessions
- Communication channels
o ISDI hub
o Email
o WhatsApp
o Videoconference (Zoom, Google Meet)
- Meetings
o Weekly – each Monday morning
o When needed

4. Work (management related)
• Speak the same language
• KPIs vs OKRs
• Agree 2 KPIs & 2 deliverables of Educational Mentoring Plan
• Design pilot timeline
• Develop surveys related to customer satisfaction
• Solve problems (reschedule if necessary)

6. Work (related to online courses)
• Training related to:
o Key Competences for Lifelong Learning (EC)
o Standards Examples (e.g., CSTA K-12 CS, ISTE, ITEEA, NGSS)
o Skills (cognitive, personal, and social)
o Course design steps
o Bloom’s taxonomy of educational objectives
o Learning objectives formulation
o Lesson’s events
• Select standards criteria for online courses
• Develop surveys related to course quality and learner feedback
• Generate ideas for STEM lessons with Otto
• Review (together) online courses

7. Otto robots for my students

8. Hybrid class with Otto

9. Additional evaluation
instruments
Survey Purpose Target
Learner
Feedback
Gather feedback related to
courses from all learners.
Learners (students, teachers,
parents, hobbyists)
Course
Quality
Review quality of the online
courses according to standards.
Teachers
Customer
Satisfaction
Measure overall satisfaction of
the customers.
Customers (teachers, parents,
hobbyist, students over 18 y/o)

10. Learner feedback
• Learner profile
• Feedback related to courses

11. Learner profile (role, experience)
0
2
4
6
8
10
12
14
16
18
20
0
courses
1 – 2
courses
3 – 5
courses
6 – 10
courses
11+
courses
Previous learner experience with online courses
Student Teacher Hobbyist Parent

12. Learner country of residence

13. Age & gender of learners
0
5
10
15
20
7 - 10 y.o 11 - 13 y.o. 14 - 18 y.o. 19+ y.o.
Student learners
Boy Girl
0
2
4
6
8
18 – 25
y.o.
25 - 34
y.o.
35 – 44
y.o.
45 – 54
y.o.
55 – 65
y.o.
66+ y.o.
Non-student learners
Female Male

14. Feedback on course elements
1
1
1
0
2
1
1
0
2
2
0
0
4
4
6
7
10
6
21
21
20
18
20
19
42
43
40
42
37
43
0 10 20 30 40 50 60 70 80
Videos and / or audios
Text and graphic content
Quizzes (not graded questions)
Tests (graded questions)
Forum discussions
Course interface
Very dissatisfied Dissatisfied Neither satisfied nor dissatisfied Satisfied Very satisfied

15. Feedback on course elements (II)
Is there anything you would like to mention about any of these course elements?
• Parent (France): "Course in French."
• Student (Mexico): "Otto's Complete Design Process."
• Student (South Africa): "Great but the final test marking isn't too good."
• Teacher (Australia): "Information about all the updates was very useful."
• Teacher (Brazil): "I wish continuous efforts by translate all important instructors’ materials.
You are great team. Great job is in here."
• Teacher (USA): "For use with students the quizzes could be slightly more intense."

16. Statements about course
0
0
0
0
0
1
0
0
8
8
9
10
21
20
23
18
40
40
37
41
0 10 20 30 40 50 60 70 80
The course clearly explains what you were expected to
learn from the course (e.g., give learning objectives)
The information in the course is easily understandable.
The activities, and quizzes are related to the course
objectives.
The tests are related to the course objectives.
Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree

17. Course impact on learner KSAs
1
0
0
1
0
2
3
0
2
0
6
14
12
13
7
22
14
21
23
26
38
38
36
30
36
0 10 20 30 40 50 60 70 80
This course enhanced my knowledge in course topic.
This course helped me to develop critical thinking,
creative thinking, and problem solving.
This course helped me to develop practical skills in
robotics, electronics, programming, and 3D modeling.
This course improved my communication skills
(including English communication).
This course increased my interest in course topic.
Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree

18. Course impact on learner KSAs (II)
Is there anything you would like to mention about how the course contributes to the
development of knowledge, skills and attitudes?
• Student (Portugal): "Yes, I understand the robotic."
• Student (South Africa): "It's great, it changed my attitude towards robots, my programming
skills and knowledge."
• Teacher (Australia): "The videos were informative and help to locate Otto Blocky software.
I'm using to teach kids about programming Ottos."
• Teacher (Spain): "I am very happy because I am learning practical things very well that I was
totally unaware of."
• Teacher (Spain): "It is very suitable for people like me who start almost from level zero."

19. The difficulty level of course for learners
0 5 10 15 20 25 30 35 40
Build your own robot! (Mechanical Assembly)
Code your own robot (Programming)
Design your own robot (3D modeling)
Engineer your own robot (Electronics)
Too high Very high, but I could learn Appropriate level Too easy

20. How engaging was the course for learners
0 5 10 15 20 25
Build your own robot! (Mechanical Assembly)
Code your own robot (Programming)
Design your own robot (3D modeling)
Engineer your own robot (Electronics)
How engaging was the course for learners
Very engaging Somewhat engaging Neither engaging nor disengaging Not very engaging Not at all engaging

21. Best thing about course
• Student (Canada): "Easy."
• Teacher (Brazil): "Circuits."
• Hobbyist (Brazil): "Didactic."
• Students (Spain): "It's free."
• Hobbyist (Colombia): "Videos."
• Student (Portugal): "The quiz."
• Student (Germany): "The coding."
• Teacher (Perú): "Videos and images."
• Student (Spain): "Robot programming."
• Student (Mexico): "Practicing in
Tinkercad."
• Student (Mexico): "The construction of Otto."
• Student (Portugal): "Can do it at my own pace."
• Student (Portugal): "Learn to assemble Otto."
• Hobbyist (Spain): " Practice with the robot."
• Teacher (Brazil): "Otto and your actual ecosystem."
• Teacher (Israel): "Learning Otto way of thinking."
• Student (India): "That it was very easy to learn."
• Student (Mexico): "Designs that can be
implemented.“
• Students (Spain): "The idea of thinking how to
improve it."

22. Best thing about course (II)
• Student (South Africa): "Learning about Otto Blockly, open-source code and the easy design of an
Otto robot and how it is meant for all ages making learning fun which it really does."
• Teacher (Australia): "Clear and descriptive with good graphics and images and excellent 'How to
Videos'."
• Teacher (Portugal): "It is very clear, and the topics are well presented."
• Teacher (Spain): "From my initial experience (zero) it has been an excellent way to learn."
• Teacher (Spain): "I have loved this course! I had learned some things from the Tinkercad tutorials
but the videos you show are much more enlightening and very useful."
• Teacher (Spain): "It has been very constructive for me, and I am learning a lot."
• Teacher (Spain): "It is very suitable for people like me who start almost from level zero, and it helps
me learn what I need for my classes."
• Teacher (USA): "You include a lot of pictures. That is very helpful."

23. Worst thing about course
• Hobbyist (Spain): "Nothing, everything was well explained."
• Hobbyist (Brazil): "No legends to pdfs."
• Student (Mexico): "Internet."
• Student (Mexico): "It does not show how to perform an Otto from
scratch."
• Student (Portugal): "Too much text."
• Student (South Africa): "The final test and the final test marking."
• Student (Spain): "The long videos."
• Teacher (Australia): "Sometimes not knowing where I was especially
during question in the quizzes."
• Teacher (Israel): "Moving through the pages was too complicated."

24. Course quality
• Content
• Instructional design
• Students’ assessment
• Technology
• Course evaluation and support
Criteria based on the National Standards for Quality Online
Courses, version 2, 2011, iNACOL

25. Course content quality
1 – Very poor; 5 - Excellent
Criteria
Response
frequency Mean
1 2 3 4 5
The goals and objectives clearly state what the participants will know or
be able to do at the end of the course. The goals and objectives are
measurable in multiple ways. 1 4 2 4.14
The course content and assignments are of sufficient rigor, depth, and
breadth to teach the standards being addressed. 3 4 4.57
Information literacy and communication skills are incorporated and
taught as an integral part of the curriculum. 1 2 2 2 3.71
The course reflects multi - cultural education, and the content is
accurate, current, and free of bias or advertising. 2 5 4.71
Privacy policies are clearly stated. 1 6 4.71
Assessment and assignment answers and explanations are included. 2 5 4.71

26. Course instructional design quality
Criteria
Response frequency
Mean
1 2 3 4 5
Course design reflects a clear understanding of all students’ needs and
incorporates varied ways to learn and master the curriculum.
1 1 2 3 4.00
The course instruction includes activities that engage students in active
learning. 5 2 4.29
The course provides opportunities for students to engage in higher-order
thinking, critical reasoning activities and thinking in increasingly
complex ways. 3 4 4.57
Readability levels, written language assignments and mathematical
requirements are appropriate for the course content and grade - level
expectations. 2 2 3 4.14
The course design provides opportunities for appropriate instructor -
student interaction, including opportunities for timely and frequent
feedback about student progress. 1 3 3 4.29
Students have access to resources that enrich the course content. 1 2 4 4.43

27. Students’ assessment quality
Criteria
Response
frequency Mean
1 2 3 4 5
Student evaluation strategies are consistent with course goals and
objectives, are representative of the scope of the course and are clearly
stated.
2 3 2 4.00
The course structure includes adequate and appropriate methods and
procedures to assess students’ mastery of content. 1 2 4 4.43
Ongoing, varied, and frequent assessments are conducted throughout
the course to inform instruction. 1 1 2 3 4.00
Assessment strategies and tools make the student continuously aware of
his/ her progress in class and mastery of the content. 3 4 4.57
Grading rubrics are provided to the instructor and may be shared with
students. 1 1 3 2 3.71

28. Course technology
Criteria
Response
frequency Mean
1 2 3 4 5
Clear and consistent navigation is present throughout the course.
2 1 4 4.29
Rich media are provided in multiple formats for ease of use and access in
order to address diverse student needs. 2 4 1 3.86
All technology requirements (including hardware, browser, software,
etc...) are specified. 1 1 5 4.57
Prerequisite skills in the use of technology are identified.
3 4 4.57
The course uses content specific tools and software, appropriately.
1 2 4 4.43
Copyright and licensing status, including permission to share where
applicable, is clearly stated and easily found. 1 1 5 4.29

29. Course evaluation and support
Criteria
Response
frequency Mean
1 2 3 4 5
The course provider uses multiple ways of assessing course
effectiveness.
1 3 3 4.29
The course is evaluated using a continuous improvement cycle
for effectiveness and the findings used as a basis for
improvement. 3 4 4.57
The course is updated periodically to ensure that the content is
current.
3 4 4.57
The course provider offers technical support and course
management assistance to students, the course instructor, and
the school coordinator. 3 4 4.57

30. Course future work
• Align the course to educational standards
• Create lesson plans for teachers
• Take in account the ignored online course standard criteria
• Create different versions of courses targeted to specific age
categories and initial knowledge level
• Create STEM lessons based on Otto

31. Customer satisfaction
• Customer Effort Score (CES)
• Customer Satisfaction Score
(CSAT)
• Net Promoter Score (NPS)
Category Resp.
Hobbyist
9
Parent
4
Student
(over 19 y/o) 8
Teacher
18
Total
39

32. Customer Effort Score
Overall, how satisfied are you with the support received from Otto DIY?
(1 - very dissatisfied, 5 - very satisfied)
Response frequency
Total
responses
Positive
responses
(4 & 5)
Mean Median
1 2 3 4 5
0 0 5 10 24 39 34 4.49 5
𝑪𝑬𝑺 =
34
39
= 0.8718 = 𝟖𝟕. 𝟏𝟖 %
𝑪𝑬𝑺𝒕𝒆𝒂𝒄𝒉𝒆𝒓𝒔 =
16
18
= 0.8889 = 𝟖𝟖. 𝟖𝟗 %

33. Customer Satisfaction Score
How would you rate your overall satisfaction with our course?
(1 - poor, 5 - excellent)
Response frequency
Total
responses
Positive
responses
(4 & 5)
Mean Median
1 2 3 4 5
0 0 3 12 24 39 36 4.54 5
𝑪𝑺𝑨𝑻 =
36
39
= 0.9231 = 𝟗𝟐. 𝟑𝟏 %
𝑪𝑺𝑨𝑻𝒕𝒆𝒂𝒄𝒉𝒆𝒓𝒔 =
16
18
= 0.8889 = 𝟖𝟖. 𝟖𝟗 %

34. Net Promoter Score
On a scale of 0 to 10, how much would you recommend our course to
other (student, teacher, parent, hobbyist...)?
(0 – not at all likely; 10 – extremely likely).
Response frequency Total
responses
Promoters Passives Detractors
0 1 2 3 4 5 6 7 8 9 10 (9 - 10) (7 - 8) (0 - 6)
0 0 0 0 0 1 3 1 7 7 20 39 27 8 4
𝑵𝑷𝑺 =
27 − 4
39
=
23
39
= 0.5897 = 𝟓𝟖. 𝟗𝟕 %
𝑵𝑷𝑺𝒕𝒆𝒂𝒄𝒉𝒆𝒓𝒔 =
13 − 1
18
=
12
18
= 0.6667 = 𝟔𝟔. 𝟔𝟕 %

35. Customer Satisfaction
- CS < 100% ☺
- Otto robot – all users love him
- Otto Academy – some course improvements
needed

36. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871275.
Neither the European Commission (EC) nor any person acting on behalf of the Commission is responsible for how the following information is used. The
views expressed in this document are the sole responsibility of the authors and do not necessarily reflect the views of the EC.
Thank you!
Mihai Agape, Otto DIY Educational Mentor, mihai.agape@gmail.com