Report	on	C‐SAP	(Higher	Education	Academy	Subject	Centre	for	Sociology,	Anthropology	and	Politics)	focus	group	and	user	t...
Manchester	Focus	Group	and	User	Testing	Report	Introduction	and	Summary	As part of the OER Phase 2 C-SAP Collections Proje...
Focus	Group	Searching	Online	Though individual social science sites were mentioned, Google emerged as the clearfavourite f...
‘Live’	vs	VLE	use	of	Digital	Resources	Two contexts for the use of digital resources were discussed; during lectures and a...
The incentives for contributing to such a site generated an interesting discussion about howsites become popular enough to...
Notes	From	User	Testing	Front	Page	       Who is the site for? Know it is social research methods but who is the site for...
Reviews	       In general participants didn’t ‘get’ that the reviews were a main part of the site       One user thought...
   Use different background colour.       Background colour described as ‘hospital green’Sign‐Up	       Needs better si...
Upcoming SlideShare
Loading in …5
×

User testing and focus group report at Manchester University (C-SAP collections project)

542 views
465 views

Published on

Focus group and user testing of the front-end website http://methods.hud.ac.uk/ at the University of Manchester on 27th July 2011. Part of the OER Phase 2 C-SAP Collections Project

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
542
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

User testing and focus group report at Manchester University (C-SAP collections project)

  1. 1.  Report on C‐SAP (Higher Education Academy Subject Centre for Sociology, Anthropology and Politics) focus group and user testing at the University of Manchester, July 2011 _________________________________________________________________________________    Isabelle Brent C‐SAP, August 2011      This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK: England & Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/    1 
  2. 2. Manchester Focus Group and User Testing Report Introduction and Summary As part of the OER Phase 2 C-SAP Collections Project a user-testing of the front-endwebsite took place at the University of Manchester on 27th July 2011 together with a mini-focus group.The user testing benefited from the wide range of backgrounds of participants and generateda number of specific suggestions that have been forwarded to the developer. The mainoverall finding was the lack of understanding about the purpose of the website in promotingresource reviews. When this aspect was explained there was widespread support for theidea but this is not currently communicated in the website. The Google custom search wasparticularly popular together with the links page. It was felt that the site looks too much like ablog and needs more design to highlight the contents of the website.The focus group provided more support for the increasing use of Google and YouTube inacademia. There was a general enthusiasm for using digital resources, particularly videos,both in lectures and on VLEs; reservations related to technical rather than intellectual issues.There was also a widespread acceptance of sharing teaching online and the use of CreativeCommons licenses.Participants Emily Bannister (Researcher Development team. User testing only)Andrew Gold (Learning Technologist. User testing only)Elisa Pieri (PhD researcher. User testing only)Keir Martin (Lecturer Social Anthropology)Mark Brown (Teaching Fellow Centre for Statistics and Survey research)Kamie Kitmitto (Landmap manager/MIMAS)Ian Fairweather (C-SAP)Graham Gibbs (C-SAP, Focus Group Chair)Anna Gruszczynska (C-SAP)Isabelle Brent (C-SAP) This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK: England & Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/    2 
  3. 3. Focus Group Searching Online Though individual social science sites were mentioned, Google emerged as the clearfavourite for online searches of academic materials. The attachment to Google is clearly notbecause of lack of awareness of other sources and participants expressed the view thatGoogle’s search capabilities have improved. Increasingly I’m using Google as the widest net to cast a search  Increasingly Google is actually easier The use of the word ‘increasingly’ suggests that the academics do not set out with theexpectation of using Google but that it is proving more effective than other strategies. Thegeneral faith in Google is expressed well by one participant: If I want to find a theatre ticket, you know, whatever I want to find I put it into   Google and I’ve found in terms of doing academic searches that pretty much   every time I’ve looked, if it’s there I’ll find it on the first two pages, whether it’s   a book about Robert Owen and socialist humanitarianism in the 1930s . . . I’ll   find it on Google. The only resources that Google was not the primary search tool for was searching forvideos. YouTube was the preferred site and was considered to be the video equivalent ofGoogle: Google is for searching and YouTube is for videos Participants were aware of other sites like Vimeo and particular sites hosted by institutionssuch as methods@manchester, however it was felt that YouTube was the first place to look.Using digital resources in teaching All the focus group participants were comfortable with incorporating digital resources intotheir teaching though it is clearly quite a new practice. The change to Blackboard 9 was citedas having made a big difference in teacher’s ability to integrate digital sources. As well asYouTube, ‘Box of Broadcast’ was mentioned as a useful resource because results can beembedded in a VLE: . . . it pulled up a Panorama programme and a couple of radio programmes   [on demography] that just gives you a link to embed within a virtual learning   environment, something like Blackboard and it’s great Other digital resources such as newspapers and websites were also mentioned. This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK: England & Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/    3 
  4. 4. ‘Live’ vs VLE use of Digital Resources Two contexts for the use of digital resources were discussed; during lectures and asadditional materials provided within Blackboard. Participants were aware that there was adefinite limit to the amount of time spent watching a video during a lecture. There wasgeneral agreement when one participant said: If you do too much video in the actual lecture students think it is a cop‐out During lectures videos are used to illustrate points and to point to further resources. Forexample one participant mentioned the use of a YouTube clip to illustrate a particularanthropological ritual which was particularly popular with students. As he commented, itshows that it is: real people and you can see that there are emotions attached to it  Another strategy is to show a clip and refer the students to Blackboard to watch the fullversion. Participants described collecting relevant resources to put on course pagesincluding videos, websites and newspapers. In addition to a concern with not wanting to playclips for too long during a lecture, the most common reason for focusing on the VLE was aconcern that technology would not work: I would be wary about having a lecture that relied on it, just because I don’t trust   any of the infra‐structure . . . Whereas I trust my brain and my mouth to get through   sixty minutes. This reservation was expressed at a number or points and suggests that more extensive useof digital resources is being held back by a basic concern for technical support.QualityAssessing the quality of digital resources, particularly videos was discussed and identified asa barrier to further use. This was particularly the case with YouTube: The problem is quality control and you end up watching a lot of crap made by   people who don’t really know what they are talking about . . .  Since the focus group took place after the user-testing session there was a consensus thatthe idea behind the review of resources of the C-SAP collections project was a valuable one: There is a lot of very bad stuff and the real appeal of what you were showing us   was that . . . somebody’s done the work a bit before Intute was referred to as being a valuable resource because the materials had been vetted.The labour-intensiveness of this process was acknowledged and different star rating reviewsystems were discussed such as Amazon and music review sites as a potential alternative. This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK: England & Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/    4 
  5. 5. The incentives for contributing to such a site generated an interesting discussion about howsites become popular enough to generate contributions: It depends on who I imagine the community of people to be who are clicking   ‘like’ or ‘dislike’ so that is a way of building a brand. And then I would be more   likely to do it myself if I thought it was part of that community as well, so I imagine   it is how you get to that critical mass where people imagine there are enough   like‐minded people involved.    Participants were interested in systems which marked out resources as being particularlyinteresting. The number of views on YouTube was one indicator that acted as an initial filterand the ‘likes/dislikes’ bar.The focus group described trust and quality in a particular way which relates to a filteringprocess. Participants reserved the judgment of quality for themselves and were unsure thatany indicator would of itself make them ‘trust’ a resource. However the issue is more how tohave a filtered short-list. This is particularly the case with teaching resources which, inaddition to general concerns with quality, have to fit into a particular curriculum and teachingapproach: The key thing with a video is that students can watch it in their own time about how   to do a method but you have to be very careful at the way that is taught and the way   the video matches your teaching.  The issue of ‘fit’ was identified as a problem with some digital teaching resources whichcannot be ‘chopped down to fit into a pre-existing course’.Attitudes to Openness All participants were comfortable with the notion of sharing resources and the concept ofCreative Commons licenses. Michael Wesch’s (Wesch 2008) work on YouTube was cited asa positive example within anthropology whose work has been viewed by millions online.The increasing use of videos and other digital resources available online has also affectedthe attitude of our focus group to making materials open themselves. All agreed that theywould be happy to see their materials freely available online given they were of sufficientquality. The potential barriers they identified related to institutional restrictions rather thanpersonal concerns over intellectual property. This suggests that the use of freely availabledigital resources (whether covered by CC licenses or not) provides a model of sharing thatencourages users of such materials to contribute their own content.The reservations expressed were mainly practical, such as wanting to veto poor qualitymaterials. One participant felt that it would be difficult if the university decided to suddenlymake everything CC-licensed because of concerns over copyright issues for images thatmight not have been attributed. The main obstacle to opening up more materials wasperceived to be institutional with some universities not allowing materials to be open. This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK: England & Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/    5 
  6. 6. Notes From User Testing Front Page  Who is the site for? Know it is social research methods but who is the site for? Is it for teachers or students?  Needs to have clear focus for teachers.  Should we be making it more attractive to students?  Say what the site does for users. Its benefits, resources, teaching materials.  Website not upfront about what it is about  Could identify type of user and route them to separate pages e.g. portals.  Looks like a trustworthy academic site  Front page undersells itself, too academic – need to put clear statement of benefits on front page  Make title say it is for teaching  Image looks like it may be about business/consultancy  ‘our recent review’ not necessary on front page.  ‘Most recent’ assumes you are a regular visitor. May not have regular visitors, - seems disproportionate.  Don’t want to have to scroll up and down  Needs RSS on home page  Same list in sidebar and main page not necessary  Sidebar contains things that say ‘active community’  Video of the week – only makes sense for regular visitors  Confusion over whether it is a blog or a website – ‘posts’ may need replacing  Should include more web2.0 features in addition to Google +1Videos  Main box on home page should show the core contents of the site – our pitch or USP  Video item, should open video directly. Can we embed them all?  At the end of long lists only a few items show and then there is ‘older posts’ at the bottom. This should say More… or some such  Video titles sometimes very broad. Needs abstract of video with the resource. (Can we pick up YouTube description??)  Arrows confusing – should be to play video.  Looks like a PowerPoint with sideways arrows – also makes ‘video of the week’ unclear  ‘Video of week’ wrong name  Wanted to be able to browse through video resources This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK: England & Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/    6 
  7. 7. Reviews  In general participants didn’t ‘get’ that the reviews were a main part of the site  One user thought they were critical appraisal of method itself rather than review of teaching resource  ‘Reviews’ is an odd title. (Use ‘Resource Reviews’?)  When they were explained they were popular - not too long and useful. Pedagogic content especially useful  ‘Write a reply’ is odd wording under a resource/video. Reword to ‘comment on resource’  Checklist for reviews needs to be more visible (add an icon link to this on login)  Reviews a little too specific? Need glossary or more general ones - perceived to be aimed at teachers of methods already knowing about the subject  Having search box above list of reviews in panel to the right makes it appear as if it is a search of reviewsResources  Categories for taxonomy should be more logically organised. Retain logical order (not alphabetic)  Resources. Say they will be relevant to you  Wants to find resources first and then link to its reviews  Don’t like 2-step process of getting to resources – intermediate stage looks like blog postGoogle Custom Search  Custom search – need to say clearly what it does  Is Intute in the custom search tool?  Thought custom search tool was for the site – needs new name.  Search – Make clear the scope of the custom search tool – e.g. that it searches outside this site. (Call it ‘UK resources search’?)  Custom search – make a big selling point for the site. Say it searches high quality sites.  Impressed by Custom Search Tool (is Jorum in the custom search)  Compared with Google, all stuff looks relevant  Problem of pop-up page from Custom Search – lose it when you go to another site.  Show sites we use in a pop-up list?  Is there a case for including American sites?Appearance  Likes the graphic. Arresting, combines things you might use This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK: England & Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/    7 
  8. 8.  Use different background colour.  Background colour described as ‘hospital green’Sign‐Up  Needs better sign up for contributors.  Tell people that e-mail address will not appear – to preserve confidentiality or don’t use e-mail.Links  Add ESDS to links page – ESDS International especially useful  Popular to have links in one placeEvents  Categorise results and avoid past eventsMisc  Andy Field has a very good website (linked to his books – add to our custom search)  Other ESRC funded projects will now run for 2-3 years and then will need to be added to this site.  There is a JISC funded project in Philosophy doing event listing. – involves the Institute of Philosophy and will finish in November (could use their way of getting events??)  Personalisation through registration suggestedReference Wesch, M. (2008). An anthropological introduction to YouTube, YouTube.   This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK: England & Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/    8 

×