Your SlideShare is downloading. ×
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply



Published on

Published in: Technology, Education

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. 2010 CRC PhD Student Conference Supporting multimodal media recommendation and annotation using social network analysis Adam Rae Supervisors Stefan Rüger, Suzanne Little, Roelof van Zwol Department The Knowledge Media Institute Status Full Time Probation Viva After Starting Date October 2007 Research Hypothesis By analysing and extracting information from the social graphs de- scribed by both explicit and implicit user interactions, like those found in online media sharing systems like Flickr1 , it is possible to augment existing non-social aware recommender systems and thereby significantly improve their performance. Large scale web based systems for sharing media continue to tackle the problem of helping their users find what they are looking for in a timely manner. To do this, lots of good quality metadata is required to sift through the data collection to pick out exactly those documents that match the information need of the user. In the case of finding images from the online photo sharing website Flickr, this could be from over 4 billion examples. How can we help both the system and the user in enriching the metadata of the media within the collection in order to improve the experience for the user and to reduce the burden on the underlying data handling system? Can modelling users, by themselves and within the context of the wider online community help? Can this modeling be used to improve recommender systems that improve the experience and reduce cognitive burden on users? Existing approaches tend to treat multimedia in the same way they have dealt with text documents in the past, specifically by treating the textual meta- data associated with an image as a text document, but this ignores the inherently different nature of the data the system is handling. Images are visual data, and while they can be described well by textual metadata, they cannot be described completely by it. Also, the user cannot be ignored in the retrieval process, and learning more about a user provides information to the system to tailor results to their specific requirements. Users interact online, and these interactions form a 1 Page 91 of 125
  • 2. 2010 CRC PhD Student Conference new type of data that has yet to be fully explored nor exploited when modelling users. The work presented here combines the mining of social graphs that occur in Flickr with visual content and metadata analysis to provide better personalised photo recommender mechanisms and the following experiment and its analysis are a major component in my overall thesis. Interaction Scenario In order to address this research question, multiple experiments have been car- ried out, one of which I present here: Envisage an incoming stream of photos made available to a user. In systems of a scale similar to Flickr, this could be thousands of im- ages per second. Can a system that uses cues from the social, visual and semantic aspects of these images perform better than one that uses the more traditional approach of using only semantic informa- tion, according to specifically defined objective metrics? How does performance vary between users? An experiment was carried out that mines data from the social communities in Flickr, from the visual content of images and from the text based metadata and uses a machines learning mechanism to merge these signals together to form a classifier that, given a candidate image and prospective viewing user, decides whether the user would label that image as a ‘Favourite’2 - see Figure 1. Related Work The significant influence that our peers can have on our behaviour online has been studied by researchers such as Lerman and Jones[3], and the particular interaction that occurs between users and visual media in particular in the work of Nov et al.[4]and Kern et al[2]. Their insights into the importance of understanding more about a user in order to best fulfil their information need supports the hypothesis that this kind of information can be usefully exploited to improve systems that try to match that need to a data set supported by social interaction. Here I extend their ideas by incorporating this valuable social data into a complementary multimodal framework that takes advantage of multiple types of data. The use of social interaction features in the work of Sigurbjörnsson and van Zwol[7] and Garg and Weber[1] inspired my more comprehensive feature set and its analysis. Their findings that aggregating data generated from online communities is valuable when suggesting tags is important and I believe also transfers to recommendation in general as well as to the specific task of recom- mending images. In fact, I demonstrated this in previous work on social media tag suggestion[6]. I use some of the human perception based visual features outlined in the work of San Pedro and Siersdorfer[5], as these have been shown to work well in similar experimental scenarios and cover a range of visual classes. I extend them further with a selection of other high performing visual features. 2A binary label Flickr users can use to annotate an image they like. Page 92 of 125
  • 3. 2010 CRC PhD Student Conference Incoming stream of previously unseen candidate images Textual Social Visual User information User information Feature Extraction User A User B Has tagged beaches before Member of urban animals group Trained Classifier Potential Favourite Images Potential Favourite Images for User B for User A Figure 1: Diagram of the image classification system used with Flickr data. Experimental Work 400 users of varying levels of social activity were selected from Flickr and their ‘Favourite’ labelled images collected. This resulted in a collection of hundreds of thousands of images. To train my classifier, these images were treated as positive examples of relevant images. I generated a variety of negative example sets to reflect realistic system scenarios. For all photo examples we extracted visual and semantic features, and social features that described the user, the owner of the photo, any connection between them as well as other behaviour metrics. We then tested our classifier using previously unseen examples and measured the performance of the system with a particular emphasis on the information retrieval metric of precision at 5 and 10 to reflect our envisaged use case scenario. Results An extract of the results from the experiment are shown in Table 1. They can be summarised thus: • It is possible to achieve high levels of precision in selecting our positive examples, especially by using social features. This performance increase is statistically significantly higher than the baseline Textual run. These social signals evidently play a significant rôle when a user labels an image a ‘Favourite’ and can be usefully exploited to help them. • The value of individual types of features is complex, but complementary. The combined systems tend to perform better than the individual ones. • It is far easier to classifier photos that are not ‘Favourites’ than those that are, as shown by the high negative values. This can be used to narrow down the search space for relevant images by removing those that are obviously not going to interest the user, thus reduing load on both the user and the system. Page 93 of 125
  • 4. 2010 CRC PhD Student Conference System Accuracy + Prec. + Rec. - Prec. - Rec. Textual 0.87 0.48 0.18 0.88 0.97 Visual 0.88 1.00 0.09 0.88 1.00 Social 0.92 0.80 0.56 0.94 0.98 Textual+Visual 0.88 0.62 0.27 0.90 0.97 Textual+Social 0.92 0.77 0.62 0.94 0.97 Visual+Social 0.93 0.89 0.56 0.94 0.99 Text+Vis.+Soc. 0.93 0.84 0.62 0.94 0.98 Table 1: Accuracy, precison and recall for various combinations of features using the experiments most realistic scenario data set. Photos labelled as ‘Favourites’ are positive examples, and those that are not are negative examples. Higher numbers are better. • As is typical in this style of information retrieval experiment, we can trade- off between precision and recall depending on our requirements. As we are interested in high precision in this particular experiment, we see that the combination of the Visual+Social and Text+Visual+Social runs give good precision without sacrificing too much recall. References [1] Nikhil Garg and Ingmar Weber. Personalized, interactive tag recommenda- tion for flickr. In Proceedings of the 2008 ACM Conference on Recommender Systems, pages 67–74, Lausanne, Switzerland, October 2008. ACM. [2] R. Kern, M. Granitzer, and V. Pammer. Extending folksonomies for image tagging. In Workshop on Image Analysis for Multimedia Interactive Services, 2008, pages 126–129, May 2008. [3] Kristina Lerman and Laurie Jones. Social browsing on flickr. In Proceedings of ICWSM, December 2007. [4] Oded Nov, Mor Naaman, and Chen Ye. What drives content tagging: the case of photos on flickr. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 1097–1100, Flo- rence, Italy, 2008. ACM. [5] Jose San Pedro and Stefan Siersdorfer. Ranking and classifying attractive- ness of photos in folksonomies. In WWW, Madrid, Spain, April 2009. [6] Adam Rae, Roelof van Zwol, and Börkur Sigurbjörnsson. Improving tag recommendation using social networks. In 9th International conference on Adaptivity, Personalization and Fusion of Heterogeneous Information, April 2010. [7] Roelof van Zwol. Flickr: Who is looking? In IEEE/WIC/ACM Inter- national Conference on Web Intelligence, pages 184–190, Washington, DC, USA, 2007. IEEE Computer Society. Page 94 of 125
  • 5. 2010 CRC PhD Student Conference The effect of Feedback on the Motivation of Software Engineers Rien Sach Supervisors Helen Sharp Marian Petre Department/Institute Computing Status Fulltime Probation viva After Starting date October 2009 Motivation is reported as having an effect on crucial aspects of software engineering such as productivity (Procaccino and Verner 2005), software quality (Boehm 1981), and a project’s overall success (Frangos 1997). Feedback is a key factor in the most commonly used theory in reports published on the motivation of software engineers (Hall et al. 2009), and it is important that we gain a greater understanding of the effect it has on the motivation of software engineers. My research is grounded in the question “What are the effects of feedback on the motivation of software engineers?”, and focuses on feedback conveyed in human interactions. I believe that before I can focus my question further I will need to begin some preliminary work to identify how feedback occurs, what types of feedback occur, and the possible impact of this feedback. Motivation can be understood in different ways. For example, as a manager you might consider motivation as something you must maintain in your employees to ensure they complete work for you as quickly as possible. As an employee you might consider motivation as the drive that keeps you focused on a task, or it might simply be what pushes you to get up in the morning and go to work. Herzberg (1987) describes motivation as “a function of growth from getting intrinsic rewards out of interesting and challenging work”. That’s quite a nice definition; and according to Herzberg motivation is intrinsic to one’s self. Ryan and Deci (2000) describe intrinsic motivation as “the doing of activity for its inherent satisfaction rather than for some separable consequence” (Page 60). Herzberg (1987) defines extrinsic factors as movement and distinguishes it from motivation, stating that “Movement is a function of fear of punishment or failure to get extrinsic rewards”. Ryan and Deci (2000) state that “Extrinsic motivation is a construct that pertains whenever an activity is done in order to attain some separable outcome”. There are 8 core motivational theories (Hall et al. 2009) and some of the theories focus on motivation as a “a sequence or process of related activities” (Hall et al. 2009) called process theories, while others focus on motivation “at a single point in time” (Couger and Zawacki 1980) called content theories. Page 95 of 125
  • 6. 2010 CRC PhD Student Conference As reported in a systematic literature review conducted by Beecham et al (2007), and their published review of the use of theory inside this review in 2009 (Hall et al 2009), the three most popular theories used in studies of motivation in Software Engineering were Hackman and Oldman’s Job Characteristics Theory (68%), Herzberg’s Motivational Hygiene Theory (41%), and Maslow’s Theory of Needs (21%)1. Hackman and Oldman’s Job Characteristics Theory focuses on the physical job, and suggests five characteristics (skill variety, task identity, task significance, autonomy, and feedback) that lead to three psychological states which in turn lead to higher internal motivation and higher quality work. Herzberg’s Hygiene Theory suggests that the only true motivation is intrinsic motivation, and this leads to job satisfaction, where extrinsic factors are only useful in avoiding job dissatisfaction. One of the five key job characteristics in Hackman and Oldman’s theory is feedback. Feedback is not explicitly mentioned in Herzberg’s Motivational Hygiene Theory, but he notes that it is a part of job enrichment, which he states is “key to designing work that motivates employees” (Herzberg 1987). However this is a managerial view point. Software Engineers are considered to be current practitioners working on active software projects within the industry. This includes programmers, analysts, testers, and designers who actively work and produce software for real projects in the real world. From a management perspective, gaining a greater understanding of what motives employees could prove invaluable in increasing productivity and software quality, and from an individual perspective the prospect of being given feedback that motivates you and makes your job more enjoyable and improves the quality of your work experience could lead to a more successful and enjoyable work life. My proposed research is divided into stages. In the first stage I plan to conduct interviews and diary studies to identify the types of feedback in software engineering and how feedback is experienced by software engineers. I then plan to conduct additional studies to identify what impact this feedback has on software engineers and how that impact is evident. Finally, I plan to observe software engineers at work to see feedback in context, and to compare those observations to the information gathered during the first two stages. At the end of my PhD I hope to accomplish research that leads to a greater understanding of what feedback is inside software engineering and how it is given or received. Subsequently I wish to gain an understanding of how this feedback alters the motivation of software engineers and how this manifests as something such as behaviour, productivity or attitude. 1 The percentages are a representative of how many of 92 papers the theories were found to be explicitly used in. There can be multiple theories used in any one paper, and the 92 papers were part of a systematic literature review conducted by Hall et al (2007) sampling over 500 players. Page 96 of 125
  • 7. 2010 CRC PhD Student Conference References B.W. Boehm, Software Engineering Economics, Prentice-Hall, 1981. COUGER, J. D. AND ZAWACKI, R. A. 1980. Motivating and Managing Computer Personnel. John Wiley & Sons. S.A. Frangos, “Motivated Humans for Reliable Software Products,” Microprocessors and Microsystems, vol. 21, no. 10, 1997, pp. 605–610. Frederick Herzberg, One More Time: How Do You Motivate Employees? (Harvard Business School Press, 1987). J. Procaccino and J.M. Verner, “What Do Software Practitioners Really Think about Project Success: An Exploratory Study,” J. Systems and Software, vol. 78, no. 2, 2005, pp. 194–203. Richard M. Ryan and Edward L. Deci, “Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions,” Contemporary Educational Psychology 25, no. 1 (January 2000): 54-67. Tracy Hall et al., “A systematic review of theory use in studies investigating the motivations of software engineers,” ACM Trans. Softw. Eng. Methodol. 18, no. 3 (2009): 1-29. Sarah Beecham et al., “Motivation in Software Engineering: A systematic literature review,” Information and Software Technology 50, no. 9-10 (August 2008): 860-878. Page 97 of 125