Evaluation: Try-out


Published on

The final part of our evaluation was a try out with a nine year old child and one of the designers. The findings of this evalutions can be read in slideshare below. During the end presentation next week you can see a video of this try-out.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Evaluation: Try-out

  1. 1. Try-out (Notice: a video of the try-out will be presented next Tuesday.)In the last week before the presentation of our design, one of the designers tested the entire projectherself together with her 9 year old daughter (the pupil). This try-out was filmed and will bepresented during the end-presentation next Tuesday. The designer started with an explanation, inthe way it was written in the teacher manual. So the designer acted as teacher, and her daughter as apupil. Together they went through each step of the project. The designer observed the parts of theproject where pupils should work independently. The entire project took 3 hours, whereas the thirdstep, the actual design of the artwork, was executed a bit faster than should be because of thelimited time.This try-out gave insight of the practicality of the design as a whole, the clarity of the text and tasks,the time that is needed per step, and the expected educational value (will our product achieve itsgoals?).PracticalityThe scenario: In general the scenario was executed as expected, but not step 1. The informationfound on the Wikipanion app was limited, and the pupil was not sure whether it was about the rightlocation. Therefore she needed to go to the location itself to verify the information she had found,and to find out exactly about which part of the location it was. The other three steps were executedas expected, with the exception of sending email. The apps that were used all had email functions,and it appeared to be far more logical to email directly from the app (e.g. the notes app and themoodboard app). Therefore the email link in the instruction was not used. The final presentation wasskipped because of the time.Clarity/interpretation: for the greater part the text was clear. Most difficult was the Aurasmaexplanation in step 4b. What made it even more complicated, is the fact that switching to the step toread the instruction, caused that you had to start over in Aurasma. Also not completely clear was theway the examples in the introduction should be treated. From the teacher manual is seemed thatthey were shown and explained before the pupils started the instruction. But when the pupils weregoing through the first screen on the Ipad, the examples were presented again to be discussed in thegroup. This was a bit confusing. It also wasn’t very clear in the teacher manual.Another clarity problem was in the app used for step 3: 123D Sculpt. When saving the artwork, it wassaved in the app and not in the photo library. To achieve saving in the library, it was necessary toapply the following operation in the app 123D Sculpt: first put on transparent background, thenmake a photo, then go to gallery, at last click on the option send to Photo Library. The pupil did notfind this out by herself, so the designer had to help her do it.Step 2 was executed without any problem and completely independently.Execution of steps: step 1 was executed correctly, but the information found was superficial andlimited. It seemed not possible for the pupil to find background information like historical oranecdotic stories on the basis of the instruction. She was also helped to verify the info of Wikipanion.Without help she would probably tell a story that was not completely true. Also finding the exact
  2. 2. spot for the artwork on the map was not found without help. The story was written in the app“Notes”, but the pupil did not write the name of the street above it, as was stated in the instruction.The introduction before step 1 took 15 minutes and step 1 took 1 hour and 15 minutes.Step 2 was executed independently. The moodboard was filled with photos from Google images anda title. The pupil was capable of describing the mood she was trying to express. This step took 30minutes.Step 3 was executed with a little help of the designer. This was also because of the time. Withexclusion of the saving, she would have been able to find out the app herself. This step took 45minutes.Step 4 was executed with a lot of help. The text of the instruction was unclear to the pupil, and shecould not remember the actions she had to take in 4b. She was also confused about the icons andslides when she read the instruction. The designer had to give a hint that she could resize and rotatethe artwork after placing it. The last part, scanning the image to make the artwork appear, was not aproblem. The result however, according to the designer, was disappointing. Publishing the artworkon a A4 paper is far more dull than outside on the real location. This step took 15 minutes, but thepresentation part was not executed.EffectivenessExpected effectiveness: will our product reach its goals?Our main goal is citizenship. This goal was not reached. The information about the location was verysuperficial, and without help it could even have been incorrect. Engagement could have beenreached by the stories behind the location, but these stories were not revealed by the actions taken.This also caused that the pupil did not like this step: she did a lot of effort and the outcome wasuninteresting. Probably our target group pupils (12-15) are more skilled in finding relevantinformation, but the instruction does not support better or deeper investigation. The app Wikipanionwas too limited.The pupil did develop technical skills: the apps, saving pictures from google, copying text, savingobjects to the photo library, and artistic designing on the ipad. She also learned to read a map, butshe did need help for that. This pupil is at least 3 years younger than our intended target group, butshe is skilled above average with computers, and she is good at text interpretation. It is expected thatthe technical skills she learned could also be learned by first year secondary schools pupils, but thisdepends on their experience with tablets. Perhaps most of them already know all this: the startinglevel of the target group should be tested if this technical goal is considered to be important.The creative/art goal will probably be reached. The classical part of the lessons give room forexplanation and group discussion about art. The process of making art by finding a story, throughmaking a moodboard to creating an artwork were well understood by the 9 year old pupil. Thereforeit is to be expected that the target group will understand this as well.Design implications
  3. 3. What we will definitely change is step 4b. We decided that this should be done in pairs, so one pupilcan read the instruction and the other can execute this in Aurasma. This is logic, because testing ifothers can scan your artwork will also be done in pairs. We will also remove the mail link from theinstruction. We will also change the introduction text, probably in the teacher manual, but maybe inthe pupil instruction.Perhaps we will change the instruction of step 3, and give a hint how to save an artwork of 123DSculpt. It is not sure whether pupils who are older, and read some English, will be able to find it outtheir selves with peers. This specific aspect should be tested with the target group.We will not change the instruction of the first step. Although it is far more efficient to go to the reallocation to combine information of the internet with the real life, this is not possible yet. Internetconnection is needed outside, and schools have Wifi only tablets. In the future, this should bechanged, because it really improves the quality of our design. Also for the end result (scanning theartwork by other students) working outside on the real location is much more exciting.We will discuss the possibility to put in again the interview. We had removed it before, because thistakes a lot of time to film and edit. But we could skip the filming, and just add the interview to gaininformation. Another disadvantage was the fact that interviewing is scary at this age, but you couldallow interviewing someone pupils know well, or doing it in pairs.