2013 07 05 (uc3m) lasi emadrid jgzubia deusto learning analytics primeras experiencias weblab deusto
Upcoming SlideShare
Loading in...5
×
 

2013 07 05 (uc3m) lasi emadrid jgzubia deusto learning analytics primeras experiencias weblab deusto

on

  • 468 views

2013 07 05

2013 07 05
(uc3m)
lasi
emadrid
jgzubia
deusto
learning analytics primeras experiencias weblab deusto

Statistics

Views

Total Views
468
Slideshare-icon Views on SlideShare
440
Embed Views
28

Actions

Likes
0
Downloads
4
Comments
0

1 Embed 28

http://www.aristoscampusmundus.net 28

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    2013 07 05 (uc3m) lasi emadrid jgzubia deusto learning analytics primeras experiencias weblab deusto 2013 07 05 (uc3m) lasi emadrid jgzubia deusto learning analytics primeras experiencias weblab deusto Presentation Transcript

    • Learning analytics and remote  experimentation Universidad de Deusto (Bilbao, Spain) IQS ‐ Universidad Ramon Llull (Barcelona, Spain) Aristos Campus Mundus
    • Presentation • International Campus of Excellence: Aristos Campus Mundus • University of Deusto – DeustoTech : Pablo  Orduña, Aitor Almeida, Mari Luz Güenaga,  Susana Romero y Javier García Zubía  (zubia@deusto.es) • IQS – Ramón Llull University: Jordi Cuadros y  Lucinio González
    • Agenda • What is a remote laboratory? • Remote lab examples. Data mining. • Case of study: Analysis of opinion surveys of  users of WebLab‐Deusto • Case of study: Analysis of learning outcomes with VISIR • Conclusion and future work
    • What is a remote lab? • It allows the user to experiment out of the lab,  as being there. • Internet are the hands and the eyes of the  user: student, teacher, citizens. • The experience can be worse than in the lab,  but it can be similar, or even it can be better  because the user controls more the  experiment. But, what about the feeling and  the inmersion?
    • What is a remote lab? • Advantages: organizational, academical,…. • It is cheap because it can be shared among different institutions. • It is always available. • It is powerful because the teacher can add all that he wants • The teacher can track everything done by the user.  • Disadvantages.
    • Examples of remote labs • WebLab‐Deusto (Spain), VISIR (Sweden), iLAB (USA), labShare (Australia), etc. • WebLab‐Deusto offers experiments with basic analog circuits, basic digital circuits,  microcontrollers, robots, incubator,  Archimedes’ principle….. • Weblab‐Deusto also offers the system to schools, etc.
    • Examples of remote labs • WebLab‐Deusto Robot: • Move, upload, program…
    • Examples of remote labs • VISIR allows to mount circuits and analyze them
    • Data mining in WebLab‐Deusto
    • Data mining in WebLab‐Deusto • Tracking: who, when, where, how long, web  browser, IP, what device, experiment, state of the queue, Facebook…. And the most important • What commands were executed and with what results • The teacher can reproduce the session to see and  analyze the profile of the user: knowledge and  learning • Since 2009 more than 55 000 sessions.
    • Data mining in WebLab‐Deusto  • Is it useful? Is it usual?
    • Analysis of the opinion survey • At the end of the semester the students answer to a survey with 18 questions. • The first survey was designed on 2003 with 15  questions. • In the year 2008‐2009 the current survey was designed merging the previous survey with the results of the eMERGE project (Bordeaux)  and Ma&Nickerson.
    • Analysis of the opinion survey Q1. WebLab helps me with the subject, the concepts, the exercises, projects, etc Q2. Using the WebLab, I fell that it is real and it is not a simulation Q3. It is a good idea to extend this WebLab to all the students Q4. I have enjoyed using the WebLab Q5. WebLab is easy to use Q6. The quality of the WebCam is good Q7. The different I/O devices (switches, buttons, etc.) are easy to use Q8. I don’t have problems with the assigned time Q9. The I/O devices implemented are well selected Q10. Even being far from the WebLab, I have felt that I control it Q11. I would like to use the WebLab in others subjects Q12. I am satisfied with the WebLab Q16. The user’s manuals are good and clear. Q17. I have been motivated by the WebLab to learn more about the subject Q18. The WebLab is a high quality software (access, management, availability, etc.)
    • Analysis of the opinion survey • Is it correct?  • Is it useful and meaningful?  • Does it contain information? • For what is it being used by WebLab‐Deusto? • Learning analytics
    • Analysis of the opinion survey • X‐O: – Experience • The students use the weblab in a regular way in two different subjects since 2008 to 2012 – LP: Programmable logic (VHDL&CPLD) (123 surveys) Third year. – DE: Electronics design (VHDL&FPGA) (73 surveys) Fifth year. – Opinion survey • 18 questions, Likert pseudo‐scale with 5 levels
    • Analysis of the opinion survey • Fiability – Cronbach’s alfa • Construct Validity – Is there an internal consistency? • Item analysis – Homogeneity index
    • Analysis of the opinion survey • FIABILITY • Cronbach’s alfa • Result for the survey: 0.78 • It is a medium to high value
    • Analysis of the opinion survey • Construct validity • Designed substructure (by me) – UTILITY: Q01, Q03, Q11, Q12, Q17 – USABILITY: Q04, Q05, Q07, Q08, Q09, Q16, Q18 – INMERSION: Q02, Q06, Q10 – OTHERS: Q13, Q14, Q15
    • Analysis of the opinion survey • FIABILITY – UTILITY α = 0,74 – USABILITY α = 0,64 – INMERSION α = 0,40 – OTHERS α = 0,27 • Looking to the last two values it seems that these two factors do not exist
    • Analysis of the opinion survey Q02 Q06 Q10 Q04 Q05 Q07 Q08 Q09 Q16 Q18 Q01 Q03 Q11 Q12 Q17 Q13 Q14 Q15 Q02 1,00 Q06 0,20 1,00 Q10 0,30 0,07 1,00 Q04 0,25 ‐0,07 0,19 1,00 Q05 0,24 0,11 0,29 0,19 1,00 Q07 0,21 0,20 0,21 0,10 0,39 1,00 Q08 0,15 0,09 0,19 0,21 0,08 0,14 1,00 Q09 0,17 0,15 0,26 0,13 0,29 0,37 0,15 1,00 Q16 0,32 0,21 0,25 0,15 0,34 0,15 0,09 0,16 1,00 Q18 0,32 0,26 0,36 0,35 0,27 0,18 0,07 0,28 0,37 1,00 Q01 0,43 0,21 0,34 0,37 0,41 0,19 0,11 0,24 0,35 0,58 1,00 Q03 0,27 0,05 0,20 0,27 0,38 0,12 0,09 0,21 0,22 0,44 0,46 1,00 Q11 0,04 ‐0,03 0,31 0,37 0,16 0,19 0,05 0,13 0,20 0,36 0,36 0,28 1,00 Q12 0,25 0,17 0,46 0,34 0,39 0,37 0,15 0,33 0,29 0,59 0,61 0,44 0,43 1,00 Q17 0,25 0,00 0,12 0,50 0,07 0,06 0,14 0,07 0,22 0,31 0,38 0,18 0,41 0,27 1,00 Q13 0,05 ‐0,08 0,04 0,10 0,13 0,09 0,10 ‐0,01 ‐0,01 0,02 0,02 0,08 0,17 0,13 0,05 1,00 Q14 0,08 0,09 0,12 ‐0,07 0,12 0,03 0,06 0,19 ‐0,02 0,13 0,04 0,13 ‐0,01 0,08 ‐0,12 0,35 1,00 Q15 0,02 0,03 ‐0,01 ‐0,06 ‐0,07 ‐0,01 0,05 ‐0,09 ‐0,04 ‐0,14 ‐0,10 ‐0,10 ‐0,04 ‐0,18 0,01 ‐0,13 ‐0,09 1,00
    • Analysis of the opinion survey • Items‐factor correlations Q02 Q06 Q10 Q04 Q05 Q07 Q08 Q09 Q16 r_IN 0,71 0,68 0,64 0,17 0,31 0,31 0,2 0,28 0,38 r_US 0,42 0,23 0,44 0,56 0,63 0,57 0,47 0,58 0,56 r_UT 0,33 0,1 0,4 0,53 0,37 0,26 0,15 0,26 0,35 r_ZZ 0,08 0,02 0,09 0 0,13 0,07 0,11 0,08 ‐0 Q18 Q01 Q03 Q11 Q12 Q17 Q13 Q14 Q15 r_IN 0,46 0,47 0,25 0,15 0,42 0,17 ‐0 0,14 0,02 r_US 0,61 0,56 0,43 0,36 0,61 0,35 0,11 0,1 ‐0,1 r_UT 0,62 0,76 0,62 0,74 0,75 0,68 0,13 0,02 ‐0,1 r_ZZ 0,05 0 0,09 0,08 0,07 ‐0 0,78 0,79 0,16
    • Analysis of the opinion surveys • Factorial analysis.  Three factors Four factors Five factors Factor 1 Factor 2 Factor 3 Factor Q01 0,50 0,27 0,58 4 Q02 0,17 0,10 0,65 3 Q03 0,40 0,29 0,34 4 Q04 0,75 0,09 0,07 1 Q05 0,08 0,61 0,31 2 Q06 -0,28 0,11 0,65 Q07 -0,05 0,75 0,08 2 Q08 0,15 0,26 0,03 Q09 0,01 0,70 0,09 2 Q10 0,27 0,50 0,21 2 Q11 0,69 0,28 -0,05 1 Q12 0,46 0,57 0,32 4 Q16 0,15 0,13 0,64 3 Q17 0,74 -0,09 0,17 1 Q18 0,46 0,27 0,55 4
    • Analysis of the opinion surveys • Identified factors – Q04, Q11, Q17 ENJOYMENT – Q02, Q16 INMERSION? – Q01, Q03, Q12, Q18 CLIENT SATISFACTION – Q05, Q07, Q09, Q10 USABILITY • Excluded questions – Q06, Q08, Q13, Q14 and Q15
    • Analysis of the opinion surveys • Fiability of the survey excluding Q06, Q08,  Q13, Q14 y Q15: 0,83 • Fiability of the new identified model – ENJOYMENT α = 0,69 – ¿INMERSION? α = 0,49 – CLIENT SATISFACTION α = 0,81 – USABILITY α = 0,71
    • Analysis of the opinion surveys Q04 Q11 Q17 Q02 Q16 Q01 Q03 Q12 Q18 Q05 Q07 Q09 Q10 Q06 Q08 Q13 Q14 Q15 Q04 1,00 Q11 0,37 1,00 Q17 0,50 0,41 1,00 Q02 0,25 0,04 0,25 1,00 Q16 0,15 0,20 0,22 0,32 1,00 Q01 0,37 0,36 0,38 0,43 0,35 1,00 Q03 0,27 0,28 0,18 0,27 0,22 0,46 1,00 Q12 0,34 0,43 0,27 0,25 0,29 0,61 0,44 1,00 Q18 0,35 0,36 0,31 0,32 0,37 0,58 0,44 0,59 1,00 Q05 0,19 0,16 0,07 0,24 0,34 0,41 0,38 0,39 0,27 1,00 Q07 0,10 0,19 0,06 0,21 0,15 0,19 0,12 0,37 0,18 0,39 1,00 Q09 0,13 0,13 0,07 0,17 0,16 0,24 0,21 0,33 0,28 0,29 0,37 1,00 Q10 0,19 0,31 0,12 0,30 0,25 0,34 0,20 0,46 0,36 0,29 0,21 0,26 1,00 Q06 ‐0,07 ‐0,03 0,00 0,20 0,21 0,21 0,05 0,17 0,26 0,11 0,20 0,15 0,07 1,00 Q08 0,21 0,05 0,14 0,15 0,09 0,11 0,09 0,15 0,07 0,08 0,14 0,15 0,19 0,09 1,00 Q13 0,10 0,17 0,05 0,05 ‐0,01 0,02 0,08 0,13 0,02 0,13 0,09 ‐0,01 0,04 ‐0,08 0,10 1,00 Q14 ‐0,07 ‐0,01 ‐0,12 0,08 ‐0,02 0,04 0,13 0,08 0,13 0,12 0,03 0,19 0,12 0,09 0,06 0,35 1,00 Q15 ‐0,06 ‐0,04 0,01 0,02 ‐0,04 ‐0,10 ‐0,10 ‐0,18 ‐0,14 ‐0,07 ‐0,01 ‐0,09 ‐0,01 0,03 0,05 ‐0,13 ‐0,09 1,00
    • Analysis of the opinion surveys • items‐factor CORRELATIONS Q04 Q11 Q17 Q02 Q16 Q01 Q03 Q12 Q18 r_DI 0,78 0,77 0,82 0,23 0,24 0,47 0,31 0,44 0,43 r_IN 0,25 0,15 0,28 0,8 0,82 0,48 0,3 0,33 0,43 r_SC 0,42 0,45 0,36 0,4 0,39 0,83 0,72 0,83 0,81 r_US 0,22 0,29 0,12 0,33 0,33 0,43 0,33 0,56 0,4 Q05 Q07 Q09 Q10 Q06 Q08 Q13 Q14 Q15 r_DI 0,18 0,15 0,14 0,27 ‐0 0,16 0,13 ‐0,1 ‐0 r_IN 0,36 0,22 0,2 0,34 0,25 0,15 0,02 0,03 ‐0 r_SC 0,45 0,27 0,33 0,43 0,22 0,13 0,08 0,12 ‐0,2 r_US 0,72 0,71 0,7 0,64 0,19 0,2 0,09 0,16 ‐0,1
    • Analysis of the opinion surveys • Homogeneity index Q04 Q11 Q17 Q02 Q16 Q01 Q03 Q12 Q18 0,58 0,59 0,54 0,52 0,53 0,75 0,57 0,75 0,70 Q05 Q07 Q09 Q10 Q06 Q08 Q13 Q14 Q15 0,57 0,46 0,47 0,56 0,19 0,21 0,12 0,08 ‐0,1 Q1. WebLab helps me in the subject concept, exercises, projects, etc Q12. I am satisfied with the WebLab Q18. The WebLab is a high quality software (access, management, availability, etc.)
    • Analysis of the opinion surveys • Enjoyment: Q04, Q11, Q17 (196 surveys). 75%
    • Analysis of the opinion surveys • Inmersion?: Q02, Q16. 92%
    • Analysis of the opinion surveys • Client satisfaction: Q01, Q03, Q12, Q18. 97%
    • Analysis of the opinion surveys • Usability: Q05, Q07, Q09, Q10. 97%
    • Analysis of the opinion surveys • Does the subject/matter influence the results?
    • Analysis of the opinion surveys • Kolmogorov‐Smirnov test for the two subjects
    • Analysis of the opinion surveys • Mann‐Whitney test
    • Analysis of the opinion surveys • Does the year influence the results?
    • Analysis of the opinion surveys • Kruskal‐Wallis test
    • Analysis of the opinion surveys
    • Learning evaluation • O‐X‐O: Around 50 students of two different and  similar groups …. Analog electronics. 2012‐2013 – Pre‐test • 10 questions, multiple choice with 4 answers – Activity/Treatment: 2 weeks • 2 working sessions with VISIR in the classroom • 1 reviewing session – Post‐test • 10 questions, multiple choice with 4 answers • The same questions but in different order
    • Learning evaluation
    • Learning evaluation • Fiability – Cronbach’s alfa • Construct validity – Content – Construct, internal consistency? • Items analysis – Difficulty index – Homogeinity index
    • Learning evaluation • Cronbach’s alfa • Result for the survey: 0,52 • It is a low value, but the results of the next test are good. It means that the survey it is not very good, so it should be improved.
    • Learning evaluation: Questions 1‐5 P01 P02 P03 P04 P05 a 70 32 23 31 32 b 8 29 2 23 2 c 22 37 73 22 51 d 6 7 8 30 21 P01 P02 P03 P04 P05 a 34 12 12 12 13 b 6 12 0 6 1 c 9 25 38 9 25 d 5 5 4 27 15 P01 P02 P03 P04 P05 a 36 20 11 19 19 b 2 17 2 17 1 c 13 12 35 13 26 d 1 2 4 3 6 ALL THE TESTS PRE‐TEST POST‐TEST
    • Learning evaluation: Question 2 • In the circuit with two resistors of the same value, the voltage in each resistor is: A: it is equal, B: it is the half of E in each resistor C: serial or paralell D: it is null
    • Learning evaluation: Question 5 • Which of the following circuits measures the voltage in R1?
    • Learning evaluation: Questions 6‐10 P06 P07 P08 P09 P10 a 46 9 35 18 10 b 17 19 18 41 18 c 34 57 3 27 10 d 9 21 50 20 67 P06 P07 P08 P09 P10 a 17 7 15 9 6 b 10 8 7 15 10 c 21 20 3 18 6 d 6 19 29 12 31 P06 P07 P08 P09 P10 a 29 2 20 9 4 b 7 11 11 26 8 c 13 37 0 9 4 d 3 2 21 8 36 ALL THE TESTS PRE‐TEST POST‐TEST
    • Learning evaluation: Question 7 • The total value of the resistor is: A: close to 1 kohm B: close to 100 ohm C: close to 10 kohm D: power the circuit
    • Learning evaluation: Question 8 • Which of the following circuits measures the total resistance?
    • Learning evaluation • Difficulty index P01 P02 P03 P04 P05 0,61 0,28 0,64 0,21 0,28 P06 P07 P08 P09 P10 0,41 0,50 0,34 0,37 0,60
    • Learning evaluation • Difficulty index. The test is too difficult. • We should redesign it. Difficulty index Interpretation Recommended number of  items < 0,25 Very difficukt 10% (1) 1 [0,25;  0,45) Difficult 20% (2) 5 [0,45; 0,55) Normal 40% (4) 1 [0,55; 0,75] Easy 20% (2) 3 > 0,75 Very easy 10% (1) 0
    • Learning evaluation • Homogeneity index P01 P02 P03 P04 P05 0,43 0,38 0,24 0,50 0,38 P06 P07 P08 P09 P10 0,47 0,53 0,50 0,49 0,43
    • Learning evaluation • Question 3: Which of the following circuits measures the current in R1?
    • Learning evaluation
    • Learning evaluation
    • Learning evaluation
    • Learning evaluation • Total punctuation pre post 0246810
    • Learning evaluation • The difference between groups is statistically  significant: – Paired t‐test, p < 0,001 Paired t-test data: dfParelles[, 3] and dfParelles[, 2] t = 4.9189, df = 51, p-value = 9.474e-06 – Wilcoxon test (rangos con signo), p < 0,001 Wilcoxon signed rank test with continuity correction data: dfParelles[, 3] and dfParelles[, 2] V = 825, p-value = 1.886e-05
    • Conclusion • The survey designed to know the opinion of  the users is useful, but it is clearly and easily improvable. • The use of VISIR has a positive effect in the students’ learning with basic electronics.
    • Future work • Remote Labs + Learning Analytics • Improve and redesign the survey for the users. • Analyse the inmersion concept when using remote labs. • Extend the analysis of the VISIR effect in the students’ learning. • Better explotation of the available data: user profiles, use profiles, session profiles • Massive use of remote labs in MOOCs. • Data mining.
    • Who is copying?
    • Future work • New network at spanish level fostering  collaborations between groups working on LA  in Spain • First meeting at some point in autumn to be  established  • 100% Open (contact snola@deusto.es) • http://www.snola.deusto.es/
    • Learning analytics and remote  experimentation Universidad de Deusto (Bilbao, Spain) IQS ‐ Universidad Ramon Llull (Barcelona, Spain) Aristos Campus Mundus