Your SlideShare is downloading. ×
  • Like
Case Study
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply
Published

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
981
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
10
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. The Use of Synchronous Digital Video for Observation of, and Feedback on, Teaching Performance in the Learning and Skills Sector. A Case Study for the Standards Unit of the Department for Education and Skills: ITT Reform Pilots This case study narrative is an adjunct to the evaluation report Martin Dyke, Alan Harding, Sue Lajeunesse June 2006 Further Information: Dr Martin Dyke M.Dyke@soton.ac.uk School of Education Tel: 023 8059 5563 University of Southampton Fax: 0238059 3556 Highfield Southampton SO17
  • 2. Introduction The University of Southampton worked with Cricklade College, Education and Training Services of the British Army, the Isle of Wight College, and Southampton City College to deliver this project for the DfES Standards Unit. The project used video conferencing systems in partnership colleges and training organisations as a tool to support the remote observation of teaching performance in real time. The project was aimed at developing strategies to improve and support Initial Teacher Education, facilitate Human Resource Management and help support observation of teaching by subject specialists. The overarching aim was to: • To develop operational policy, procedures and make recommendations for the remote observation, analysis and feedback of teaching performance. The team worked closely with teachers to develop protocols that helped us utilise this technology in a supportive and developmental way. The online observation equipment was installed in partnership centres these included a work-based education and training establishment and General Further Education Colleges. This case-study broadly focuses on the partnership colleges for which this description is typical: ...a medium size general further education (FE) college …The college delivers a wide and predominantly vocational curriculum to students aged 16 to 18 and to adults. There were 1,050 students aged 16 to 18 and 4,200 aged over 19 enrolled at the time of inspection in term one, with final over 19 enrolment for the previous year being 9,854. The college delivers work-based learning in engineering, construction, hospitality and to a small number of learners in hair and beauty. The college also delivers technical certificates on behalf of other local work-based learning provider (OFSTED, 2005) However, it should be noted that the commentary is generic; it draws on the blended experience of using the technology across the range of partnership centres in which it was applied. The case-study therefore is not specific to the experience of one institution. The Technology The technology consisted of portable Video Conferencing Suites (CODECs) one in the college classroom and the other based at the University. These were run across the Internet Protocol over the JANET network. The portable video conferencing suite consisted of a flight box with a camera and CODEC, a screen, speakers and DVD recorder. A second camera was cabled directly to the CODEC. The equipment is pictured overleaf in Figure 1. M.Dyke Page 2 04/05/2010
  • 3. Figure 1 The main technical challenge found with the application of this technology was securing a connection across institutional information systems and fire-walls. The systems and technology were tested in the summer 2005 and test observations were completed successfully across Internet Protocol (IP). Technically, the process worked with high quality audio and video maintained for the duration of an observed lesson. Observers could watch the lesson, manipulate and switch between cameras, enabling them to survey the whole learning environment and communicate effectively with the class and observed tutor. On return from the summer vacation the systems no longer appeared to work as reliably. Educational organizations tend to complete work on information systems infrastructure and security in the summer vacation; these networks and firewalls, together with different proprietary standards, are said to be the bane of video conferencing (BECTA 2006). Establishing a connection and using the H.323 protocol to negotiate the firewall and network configurations have proved to be the greatest challenge: H.323 provides the call management element of a video conference. It enables session initiation, call setup, call closing, call routing and error handling. After encoding the audio and video data is transmitted using other standards such as H.263 (video) and G.711 (audio). (BECTA March 2006). These complex technological issues have impacted on this project. To date, twenty five opportunities for online observations of teaching have been created, of four have been unsuccessful. In each case there was a failure to establish an online connection. A process of trial and error enabled the project to find a combination of equipment and settings that worked. The most reliable connections were established when the organization’s firewall had been bypassed, a process that can meet a high level of M.Dyke Page 3 04/05/2010
  • 4. organizational resistance. Later in the project it was discovered that the proprietary version of H.323 on our CODEC was not compatible with the version on the University firewall. It proved essential that both the CODECS and the institutional firewalls were using compatible versions of H.323 protocol. Mike Harwood resolved the technical issues and his account is recorded below. Mike Harwood - University Data Networking Group To get around the problem of not knowing what was really going on behind the scenes and through the firewall, I began running test calls to external sites and capturing or logging the TCP/IP traffic for analysis from the firewall. This meant starting a capture or recording of all traffic flowing through the firewall from Martin’s video conference IP address to another external video conference IP address. I took a large number of packet captures and began analyzing them against some captures of successful connections. I built up a detailed understanding of all the steps taken by the video conferencing devices when they try to negotiate, connect, start to send and receive data, video and audio. I then began to see a similarity in the tests and a common failing point during the initial connection start-up. There are a large number of TCP and UDP network ports required to start a video conference call and they all need to be opened up in the university campus firewall if the traffic is to pass in and out. The firewall rules that are in place in our firewall are based on 2 main TCP ports, 1719 and 1720. If traffic is seen coming in on those ports it is assumed that this is video conferencing traffic. The 'language' for video conferencing traffic is specified by the H323 protocol and our campus firewall is aware of this protocol and can read into H323 video conferencing data traffic and look for any requests from the video conferencing devices asking to open further TCP or UPD ports (other than the already open 1719 and 1720). So we rely on the firewall to open these extra ports dynamically on request by inspecting the packets as they pass through. Unfortunately, it turns out that since the firewall upgrade the firewall’s protocol definition has changed and the firewall was no longer properly recognising these requests to open additional firewall ports. As a result the video conferencing calls would just hang whilst trying to connect and eventually time out and fail because they could get no response through the firewall. To properly confirm this assumption I set up an environment outside of the firewall on the outside of the campus network with Martin’s video conferencing device and was then able to call a number of external sites successfully. I then proceeded to call around and ask if any support groups knew of similar problems with video conferencing equipment and our type of firewall and eventually one support group confirmed they had seen similar problems. I then called our firewall support group and showed them our firewall logs and captures and they started to look into the problem. Eventually they suggested we tried installing an upgrade package, a patch, to the firewall as it was supposed to have some fixes and additions to the H323 protocol handling. Once this was installed I called around and did some further tests with Martin and some of the other members of staff, that had reported problems, and they said that the problems had now been resolved and their video-conferencing devices had started working properly again. M.Dyke Page 4 04/05/2010
  • 5. Planning Initial permissions were sought through a senior manager within the respective organisation. Typically, there was a three month lead in period before the observation hardware and associated equipment could be installed and the testing period undertaken. In that time, permissions were sought and granted, filtering from senior management level to staff on the ground. Broadly speaking, following initial acknowledgement from senior management, the communication between the project team and the college sub-divided into technical support and curriculum teams. Initially, the technicians liaised with the organisation’s firewall support company to open a number of specific ports for the hardware and were happy, following a testing phase, that ‘these holes’ would not compromise the organisation’s technical security. Once formal permission had been granted for the project, and technical issues addressed, meetings were held with various curriculum teams and participants. The project aims and processes were outlined, participants completed a questionnaire on their attitudes and preferences to online observation and were given an opportunity to ask questions and seek clarification. The observation process and protocol was developed from this consultation with participants; the intention was to apply the technology in a way in which the observed were comfortable. For example, the participants expressed a preference for the observation not to be video recorded. Accordingly, only live observations and feedback in real time with no digital recording we conducted. The project team also noted that video recording of classes was best avoided as it raised more complex legal and ethical considerations than live observation in real time Arrangement of the observations tended to be directly between the project team and the volunteering teachers. Sufficient notice was given of the online observation to allow the lecturers to seek permission of their students to be involved in the process. Regular updates regarding the progress of online observations was filtered back through the communication network by the project team; where appropriate and with permissions sought, the online observation was treated as an annual observation for the member of staff. Sue Lajeunesse – University Project Team “There is no doubt that the logistical arrangements and hardware installation were time consuming and involved a number of staff members to be present at both the host organisation and the test organisations.” In order to make this communication not surveillance technology you do need to work with the participants sensitively, use the applications in a way that puts them at ease In order to install the equipment successfully geographical location of the partnership centres was less significant than the quality of the personal relationships established with both teaching and technical staff. A project such as this presents complex organisational and technical challenges that require from, both sides, a degree of patience, goodwill, good humour and mutual support. Being able to communicate regularly and freely with familiar people and organisations proved invaluable. The technical issues were within the broad skill set of those involved; but can be said to have required a predisposition to learn how to apply these skills to new applications and contexts. Where individuals embraced the challenge of developing new skills the M.Dyke Page 5 04/05/2010
  • 6. project was successful, where individuals were less comfortable with experimenting with new applications for information technology progress was relatively slow. The technical staff in colleges and partnership centres had the skills to enable them to make the necessary adjustments to their networks and fire-wall infrastructures. The project success relied heavily upon technical staff that were able to commit to and support the developments. The Observation Within an observed classroom two cameras and one microphone were installed. The online observers were able to control both cameras and to use what is called ‘Far End Camera Control’. This enabled us, at the University, to switch between cameras to pan and zoom. These functions were found to be very powerful and key to providing valid observation judgements and feedback. Another important feature of this approach was that the online observers were always visible and able to talk to the class and teacher. Far End Camera Control and the visual presence of the online observer throughout were critical features of the observation protocol. A typical configuration is illustrated in Diagram 1 below. It was arranged, for many of the observations, for a trained observer to be present in the classroom to enable comparison of the descriptions and judgement of online observers with that of a face-to-face observer. Online Observation Configuration IP Online Network TV Observer(s) TV Teacher and Class Observer Remote control of pan and zoom of cameras Option to record Diagram One On the day of the planned observation, an online connection was established some ten to fifteen minutes before the lesson began and the observed teacher received a pre- observation briefing on the process. At the start of the lesson the online observers introduced themselves to the students and explained the purpose and process of the research. The ethical issues, such as extent of the Far End Camera Control and who would be watching, were made explicit and the participants given an opportunity to ask questions. Once the lesson started, the online observers muted their microphone but remained visible to the observed throughout. M.Dyke Page 6 04/05/2010
  • 7. Figure 2 For each quartile of the lesson the observers recorded descriptions of the teaching and learning and issues related to the online observation and provided an OfSTED / ALI grade. Each observer provided a judgment against the criteria and an overall professional judgment for the lesson. The observations covered a range of subject areas including Travel and Tourism, IT, Essential Skills and Teacher Training. The profession and modal grades given to the observations correlated with the OFSTED outcomes for the appropriate Inspection areas, where applicable. One out of Five observations was moderated for the project through simultaneous online and in-class observation. Figure 2 gives an observer eye view of the class from the camera at the front of the classroom, in this photo the observer is about to switch cameras and therefore has the camera icons on screen. Figure 3 overleaf provides an image of the equipment from the camera at the back of the classroom. Feedback Paul McKillop - Observed Online at Isle of Wight College I was given immediate, constructive criticism by Sue and she was able to indicate the areas of weakness and make suggestions on how the workload can be shifted between tutor and student using active learning. She promised to send some supportive texts on the subject, and did. These materials have been extremely useful on an ongoing basis. The opportunity for a new tutor to draw on such expertise is exceptional. At the end of each observation the observed teacher and the class were asked questions about the process and their response to online observation. Each teacher was given both written and oral feedback on her/his performance in confidence. The oral teacher feedback was provided either face to face or online, depending on the individual circumstances and context. Providing a confidential setting for feedback proved more difficult than expected, in some cases it was possible to use the same classroom used for observation, once the students had left, for feedback. However, it was difficult for the person delivering the feedback to judge the level of privacy, they needed to confirm the far end volume of their voice and ensure people were not entering the room, out of camera shot, during the post-observation debrief and discussion. In the right conditions observation feedback via video online was as successful as face-to-face for both parties. Quite often the room was booked for use by another class and alternative arrangements were necessary. In these circumstances M.Dyke Page 7 04/05/2010
  • 8. instant messenger software, such as MSN or Macromedia Breeze, was used for a web- cast conference. Figure 3 Overall findings and experiences Art the outset there was a belief that online observation as a second best alternative to face-to-face observation. It was considered an approach that might work in situations where moderation of teaching practice was required or where managing the supply of tutors and demand of students for observation was difficult; for example in the provision of subject specialist observers. By the end of the project it became apparent that it is a viable alternative means of observation. When blended with face-to-face observation it can enrich the observation and feedback process. Although the observer gets a different perspective when observing online it enables the observer to focus more on the micro level of interaction. Interestingly it was also found that the observation descriptions and grades of the online observer were the same as the classroom observer in 12 of 14 observed lessons and only one grade out on the other two. In this project we found it a valid and reliable means of observing lessons and making judgements about teaching performance. The online observations were most successful when the online observer knew the institution and the teacher personally, where they have observed the teacher face-to- face and established an effective working relationship. The only negative feedback about the process was received, on the one occasion, when teacher was not known to the online observers. Once the networking problems were overcome the equipment was relatively easy to setup and use. Observing a lesson and making notes while driving the cameras at a distance does take a little getting use to and requires intense concentration. It was found that observers had to actively and continuously scan the class from different angles in order to make informed judgements. From a health and safety perspective online observers found that more than three hours of watching a screen per day was the maximum they could reasonably manage. Online observation was more intense and tiring than initially expected. M.Dyke Page 8 04/05/2010
  • 9. In relation to the placement of cameras it was found that cameras positioned in the corners of classrooms were least intrusive this reduced the extent of the panning necessary to 90°. Further, when the camera was centrally located on a side wall of a classroom observers needed to pan 180°. With such a wide range of panning the students were more likely to be conscious of the camera movement. The ideal camera positions were in corners of room with one camera at the back of the room and one at the front. The monitor was usually placed at the front, with the observer on a screen facing the class. This did not prove to be a significant distraction but the project team’s view is that the position and level of visibility of the observers should be negotiated with the class. Observer presence was, for the team, a key dimension of the technology being used for communication not surveillance. A typical configuration is illustrated in Diagram One. For all observations the online observer could be seen and was able to communicate with both teacher and students. This was found to be an important element in the process of working with the teachers and not simply observing from afar. Individual observed teachers managed the visual and audio presence of the online observer in a way they found most comfortable. Sylvia Waghorn – Observed online at Southampton City College I found the experience of online observation fascinating and felt that the observation itself was no more intrusive than had a person been sitting in the room. The students reported that they soon became comfortable enough with the equipment that they ignored it. The Future? The use of online observation can provide a less obtrusive mechanism for the observation of teaching. It can help provide for moderation of teaching practice without the need for a number of observers being physically present in the classroom. It can also provide an alternative means to link up trainee teachers with subject specialist observers and advisors. Potentially it has significant implications as an efficient and effective tool to support the improvement of teaching and conduct of practitioner research. There is the scope to broaden the classroom teacher’s community of practice and to facilitate greater externality in quality assurance processes. Sally Sims – Participant at Cricklade College Personally, I am very keen see the wide-spread use of this technology. I believe it will extend more equality of opportunity to trainee PCET teachers to access mentoring support (through subject specialist observation feedback) where such specialists are geographically remote. The project team are keenly aware of the proliferation of surveillance cameras often manipulated over IP, and are seeking to create a framework for the developmental use of this technology in educational settings. The project team is keen to apply these applications as communication not surveillance technology; the project has worked with practitioners, teachers and students to establish acceptable protocols for the use of digital technologies in the observation of teaching performance. M.Dyke Page 9 04/05/2010
  • 10. Acknowledgements: This project has worked closely with members of the University of Southampton’s partnership of colleges including: • Cricklade College • Defence Educational and Training Services (Army) • Isle of Wight College • Southampton City College The project could not have been successful without the full commitment and involvement of each organisation’s students, teachers, technical staff and senior management teams. Particular thanks go to: John Baglow, Steve Bax, Sue Baxter, Paul Buckland, Dave Crome, Stephen Ellison, Eddy Elms, Elliston Ensor, Sue Flawn Paul Gallop and his IT team, Terry Gaskin, Mike Harwood, Richard Hill, Andy May, Paul Morgan, Paul McKillop, Dean Palmer, Mark Powell, Gillian Quinton, Jon Searle, Sally Sims, Margaret Smith, , Sylvia Waghorn, Adrian Ware, and Paul Wyatt. The research team would like to give their whole hearted thanks to the individuals who participated in this research, with have enjoyed working with you and look forward to working together in the future. References BECTA (2006) Tech News March 2006 http://www.becta.org.uk/page_documents/research/technews/mar06.pdf OFSTED (2005) Inspection Report for Southampton City College Available at: http://www.ofsted.gov.uk/reports/manreports/2466.htm (Accessed 20 June 2006). M.Dyke Page 10 04/05/2010
  • 11. The Use of Synchronous Digital Video for Observation of, and Feedback on, Teaching Performance in the Learning and Skills Sector. A presentation for the Standards Unit of the Department for Education and Skills: ITT Reform Pilots Martin Dyke Alan Harding Sue Lajeunesse July 2006 Centre of excellence for research and teaching P roject Aims Aim To develop operational policy and procedures and to make recommendations for the remote observation, analysis and feedback of teaching performance. University of School of Post Compulsory Southampton Education Education and Training Objectives  To develop operational policy and procedures and to make recommendations for the remote observation, analysis and feedback of teaching performance.  eva lua te the obs erva tion of tea ching performa nce a nd the experience a nd preferences of tea chers , obs ervers a nd other s ta keholders .  develop opera tiona l policy a nd procedures from the experience a nd eva lua tions of pa rticipa nts /s ta keholders  eva lua te the us e of dig ita l technolog y in the obs erva tion, a na lys is a nd feedba ck on tea ching performa nce  Department for Education and Skills  S ta nda rds Unit University of School of Post Compulsory Southampton Education Education and Training M.Dyke Page 11 04/05/2010
  • 12. Key F inding s : Cons ulta tion a nd Ques tionna ire  Teachers at ease with the ‘real time’ online observation  75% a g reed to pa rticipa te in project  Majority would also agree to process being recorded  70% a g ree to recording for modera tion purpos es  66% a g ree to recording for res ea rch purpos es  56% a g ree to recording for non-commercia l tea cher educa tion purposes  Majority ambivalent about process  P refer to know observer (33% prefer to 3% prefer not to)  P refer observer knows their colleg e (39% prefer to 3% prefer not)  P refer s ubject s pecia lis t (31% prefer to 3% prefer not)  Small group strongly opposed to process (22%)  Two thirds of which convinced they would not chang e their mind  Concerns express ed: ‘impers ona l’ ‘confidentia lity’ ‘intrus ive’ ‘dis ruptive’ ‘loss of a tmos phere’ ‘uncomforta ble with being seen on ca mera ’ University of School of Post Compulsory Southampton Education Education and Training Online Obs erva tion Config ura tion Online Observation Configuration IP Online TV Network IP Observer(s) Online TV Network TV Observer(s) TV Teacher and Class Teacher and Class Observer Remote control of pan and zoom Observer of cameras pan Remote control record and zoom of Option to of cameras Option to record University of School of Post Compulsory Southampton Education Education and Training Overview: Obs erva tion of Tea ching  Partnership colleges and training organisations  Live in real time  As Communication not Surveillance Media  Online Observer Present in Classroom  Far End Camera Control  Over Internet Protocol  Process developed in Consultation with teachers  Video Conference Technology University of School of Post Compulsory Southampton Education Education and Training M.Dyke Page 12 04/05/2010
  • 13. Methods  Action Research Model  Participatory Aspirations  Consultation and Questionnaire  118 pa rticipa nts to da te  Protocol built upon feedback from consultation  Direct Observation Data  25 Obs erva tions of tea ching to da te  Us ed s ta nda rd UK Qua lity As s ura nce criteria a nd cla s s ifica tions  User Experience Interviews and Questionnaire University of School of Post Compulsory Southampton Education Education and Training Key F inding s : Obs erva tion da ta  Four of Twenty-four observations failed  F irewa ll a nd Networking Config ura tion P roblems A Comparsion of Online Observer and In-class Observer Grades of Lessons  Twenty-one observations 4.5 fourteen combined online 4 3.5 Observation Grades observation with observer in 3 2.5 classroom. 2 1.5  S pea rma n R ho = 0.77 a t 0.01 1 0.5 s ig nifica nce level 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Observation Number  Online observer g ra ded Online Observer In-class Observer clos er to the mea n  Qua lita tive Des criptions of les s on s imila r University of School of Post Compulsory Southampton Education Education and Training Conditions in which tes ted  Observers shared cultural capital, close working relationship  Observers’ close relationship with organisations and observed  Adult classes in regular classrooms  Good quality audio and video maintained  Far End Camera Control  Online observer presence in classroom University of School of Post Compulsory Southampton Education Education and Training M.Dyke Page 13 04/05/2010
  • 14. Key F inding s : Obs erva tion da ta  Aperture and Context  Online Observer Presence  Communica tion Not S urveilla nce  Far End Camera Control Essential  Verification of Doubts  Online Obs erva tion Intens e  Cons cious delibera te a ctive s ca nning  Observed quickly forget cameras  Observed positive about experience  Obs erva tion etiquette University of School of Post Compulsory Southampton Education Education and Training Cha lleng es / Is s ues  Ethical  Organisational & technical  F irewa lls a nd Networks  Intensification of Work?  Surveillance & Power  Semiotics  Aperture  Ma cro a nd Micro  Need to Constantly Verify Doubt University of School of Post Compulsory Southampton Education Education and Training Next S teps  Video Rich Learning For further information contact: Environments  Synchronous High End Dr Martin Dyke Video-communication School of Education  Webcast development University of Southampton  New configurations for Southampton SO17 1BJ UK eLearning  Bringing together diverse communities of practice M.Dyke@soton.ac.uk 0044 23 8059 5563 University of School of Post Compulsory Southampton Education Education and Training M.Dyke Page 14 04/05/2010