Effects of Developers’ Training on User-Developer Interactions in Information Systems Development

Effects of Developers’ Training on User-Developer Interactions in Information Systems Development Jennifer McCauley, Sharoda A. Paul, Andrey Soares April 20, 2006 IST541 – Qualitative Research in IST College of Information Sciences and Technology The Pennsylvania State University {jmccauley, spaul, asoares}@ist.psu.edu Abstract: The importance of user-developer interactions during the development of an information system has been a long-running theme in information systems research. This research seeks to highlight a gap in the current literature: the contribution of the developer’s formal educational background to the relationship between developers and users. Using an interpretivist epistemology, the researchers employed qualitative interviews to examine how far developers’ perception of the importance of interacting with the user was influenced by their formal education, or the lack thereof. Interviewing both formally and informally trained developers, eleven categories of interest were identified as pertinent to determining the developers’ beliefs about the importance of user interaction. Three of these categories were explored as promising for future research: academic background, work experience, and developer’s access to user knowledge. This research has implications for education of information systems developers as well as for industry interested in hiring software developers. Keywords: information systems development, user interaction, developers training 1 INTRODUCTION User participation and involvement has been a long running theme in information systems (IS) 1

research, specifically with respect to the development of information systems. Research on the correlation between user participation/involvement and success of information systems development (ISD) has produced contradictory results. While some researchers have found that greater user participation leads to more user satisfaction, there has been no agreement on whether greater user participation guarantees successful ISD. In examining the effect of user participation, much attention has been paid to the relationship between users and developers, especially on the subject of how they share and transfer knowledge about the system. However, there has been very little research into how the users’ or the developers’ experiences and training affect this relationship. In our research we are hoping to focus on the developer’s contribution to the user-developer relationship in ISD. By contributions we mean when, how and why they seek user interaction during the ISD process. In this respect, we are hoping to bring to light a little-studied aspect of developers’ contribution – how their formal training, or the lack thereof, affects their perception and commitment to involving the user in the development of an information system. 1.1 Relevance to IST The research explores how technical knowledge and training affects developers’ relationship with users. We hope to capture that information through exploring academic background, experience, information system definitions, and user role beliefs. Our approach focuses on all three constructs of the IST research philosophy – information, technology, and people. The ISD process, though concerned with the developed of technological systems, is a social process shaped by human actors. Developing an information system not only requires developers and users to have knowledge about the domains of design and use of the system, but it also requires effective information exchange between them. Our research attempts to shed light on the 2

dynamics of the developer’s role in user/developer interactions and points of influence for improving those interactions. These findings will also highlight areas for future educational opportunities and research collaborations between social and technical research within the realm of Information Sciences and Technology. The individual research interests of research team members cover a broad spectrum of issues related to information systems development. Other than interest in user involvement in ISD, we are interested in studying implementation of large-scale information systems, ontologies used to capture users knowledge in ISD, and re-skilling of IT professionals into the information sector. 1.2 Roadmap The following research proposal will clarify the contributions of the developer-user interactions within the area of Information System Development. Section two will highlight the current literature in the area of developer-user interaction. Section three will clarify the research questions pursued, identify the methodological approach taken, reasoning behind the methods employed, data collection implemented, issues that arose during the research process and roles of the researchers. Section four examines the data gathered though coding, analysis of that data and the findings. Section five summarizes the findings, implications and contributions the research adds to the developer-user research area. Section six summarizes the research in its entirety. 2 LITERATURE REVIEW User participation and involvement in the information systems development process has been a long running theme in information systems research. User participation refers to the activities performed by the user during systems development while user involvement is the importance and personal relevance of the system to the user (Barki & Hartwick, 1989), (Barki & Hartwick, 1994), (Hartwick & Barki, 1994). Grudin (1991) argued the importance of users’ participation in 3

design, and suggested a classification for the term user (i.e. end-user, customer, client, etc.). Even though users’ involvement and participation in the ISD process has been studied for over four decades, it continues to present challenges for future research. Much attention has been paid to issues related to users’ participation and involvement in the process of developing systems. Several researchers (Bailey & Pearson, 1983), (Doll & Torkzadeh, 1989), (Hwang & Thorn, 1999), (Robey & Farrow, 1982), (Tait & Vessey, 1988) have concluded that user participation significantly increases system success and end-user satisfaction. Other researchers, (Ives, 1984), (Barki & Hartwick, 1994), (Butler & Fitzgerald, 1997), (Cavaye, 1995), have found that user participation may not contribute much to improving the quality of the system being developed. Most literature in this area refers to the relationship between users and developers as user participation or user involvement. For the purposes of this research, we use the term interaction, referring to a broader idea that covers both participation and involvement issues (Grudin, 1991), (Rosove, 1969). Empirical studies have also failed to reach consensus on whether user involvement ensures success of the ISD process. Ives et. al. found no results supporting the correlation while Pettingell et al. found a fair to moderate correlation (Emam, 1996). Wong’s case study analysis of ETHICS versus a successful IS developed without structured theory found correlation between users and success of ISD’s (Wong & Tate, 1994). Iivari’s Case Study analysis (2004) suggested that users’ involvement could be influenced by the organization’s culture. User participation should focus on users’ knowledge to achieve more effective design (He, 2004), (Wilson ET. Al., 1997). Users accumulate rich application domain knowledge due to long period of exposure to their job context and “the knowledge perspective of user participation captures the quantity (how much) or the frequencies (how often) of various knowledge activities 4

user participants perform during an ISD process” (He, 2004, p. 2). While He’s research addresses the knowledge interaction between users and developers, our research will focus on the developers’ perception of when the users’ knowledge is needed during the ISD phases and how their formal training, or the lack thereof affects this perception. Kautz & Pries-Heje (1999) and Nosek (2001) highlight that little research has focused on education and training of systems development methodologies. Nosek suggests “education involves understanding abstract theory while training involves the accretion of skills necessary to perform a task” (p. 205). Roberts (2000) points out the importance of education for developers. However, he states that some capable developers have achieved huge accomplishment with no formal education. Roberts (2000) also suggests that “formal education in computing is neither a necessary nor a sufficient predictor of productivity in the software development industry” (p. 85). This issue with developers’ lack of training has been stressed by Rosove (1969), who acknowledges the need for improving developers training, and updating computer sciences programs. He argues that recent graduate students are weedy in software development techniques because computer science programs emphasize operating systems, programming languages, and some applications skills. Rosove also recommends that developers should be educated to work and to interact with users, as well as to design systems for them. However, training has little influence in adopting a system development methodology (Kautz & Pries-Heje, 1999). A recent study (Latzina, 2002) has shown that developers’ training in usability still poses an important issue on how developers involve users in ISD. 2.1 Preconceptions The preconceptions applied to this research are based on the authors’ professional and academic 5

experiences. In terms of academic experience, two researchers have a Computer Science background, and one researcher has a Management Information Systems background. Thus, we have all been educated and trained in software engineering, and have been exposed to the ideas and concepts related to ISD as part of our formal education. These preconceptions guided our research approach with respect to defining the research questions, sampling selection, elaborating the interview questions, performing the interviews, and analyzing the data. Based on our own academic backgrounds and the common ground we shared, we initially started with the assumption that developers who had courses in software engineering or ISD methodologies qualified as “formally trained”. Using this criterion, we decided to consider undergraduate degrees in computer science (CS) and information systems as “formal” training for ISD. Developers with undergraduate degrees in all other fields were considered “informally” trained or self-taught. The term “formal training” as used by us relates to formal education as defined by Roberts (2000) to mean any completed undergraduate degree. The interview questions, similarly, reflect our understanding of the IS development phases, and the importance of users participation and involvement in systems development. We prescribe to He’s (2004) view that the user has valuable contributions to make in the ISD process and hence it is important for developers to involve the user in ISD. 6

3 METHODOLOGY 3.1 Research Questions Developer’s styles of user interaction differ based on how they acquire technological skills related to ISD. Formally trained developers’ tend to have very structured views of user interaction based on development methodologies defined in IS literature between 1967 to 1977 (Fitzgerald, 2000). Informally trained developers have possibly been formally trained in an entirely different subject area and then migrated to an ISD role. Thus, we were interested in answering the following research question: How does developers’ style of user interaction differ based on whether the developer is formally or informally trained? Two related sub-questions that help in answering the above research question are: During which phases of the ISD processes do developers seek users’ knowledge? During which phases of the ISD processes do developers feel that the users’ knowledge is highly needed? In order to answer our main research question, we chose to interview two different groups of information systems developers, formally trained and informally trained developers as per our assumptions above. As we conducted the research, certain issues and concepts emerged which forced us to change our initial definitions of “formal” and “informal”. We discuss these issues and the evolving nature of our understanding of the terms formal and informal later in Section 3.4.4. We expected the outcome of this open-ended interview process would help us answer several specific questions related to developers’ beliefs of user interaction in the ISD process. From the 7

first group of interviewees, we expected to learn if there were differences between what is taught in school regarding the importance of user interaction and what is practiced by developers on a daily basis regarding capturing and applying users’ knowledge. From the second group of interviewees, we expected to learn how and why they became developers, where they received their ISD training, and how they capture and apply users’ knowledge in an ISD process. In addition, we also expected to learn what both groups considered as ISD phases, and how important they considered user interaction in ISD. 3.2 Rationale and Epistemology We analyzed both qualitative and quantitative research approaches to decide which methods were better suited to answer our research questions and the overall nature of the phenomenon we were interested in studying. Using quantitative research methods would have required us to quantify the phenomenon we were interested in studying by representing numerically the levels of our theoretical constructs and concepts (Straub, Gefen, & Boudreau, 2004). Qualitative research methods, on the other hand would enable us to examine phenomenon by understanding people and the social and cultural contexts within which they exist though interviews, participant observation or document analysis (Myers, 1997). Since there is no theory of how user involvement is viewed by formally and informally trained developers, we could not perform theory-testing or quantify the constructs that we thought were significant in this area. Thus, we decided to approach this research by examining the social actors’ (systems developers, in this case) perceptions and ideas about the significance of user interaction. The most important factors that influence the choice of qualitative methods include the nature of the research problem, the researcher’s theoretical lens, the degree of uncertainty surrounding the 8

phenomenon, the researcher’s skills, and academic politics (Trauth, 2001). To ensure that our research was well suited for qualitative analysis we explored those factors. In examining user interaction in information systems research, we did not have any a priori hypothesis about how formally and non-formally trained developers differ in how they perceive and practice user interaction. We were interested in examining and understanding a socially based phenomenon. Thus, the ontological perspective we adopted was concerned with social relations, processes and practices and was well matched to a qualitative research methodology (Mason, 2002). We could have chosen a positivist, interpretivist or a critical theoretical lens. The positivist ontological assumption is that there is an objective physical and social world that exists independent of the actions of humans. Positivist studies assume the existence of “a priori fixed relations within phenomenon which are typically investigated with structured instrumentation” (Trauth, 2001; p. 6). In the domain of information systems research, such studies have attempted to test theory, hypothesis, and formal propositions in order to draw inferences about phenomenon from the sample to the stated population (Orlikowski & Baroudi, 1991). Interpretivism is based on the ontological assumption that reality and our knowledge of it are socially constructed. Interpretive studies assume that “people create and associate their own subjective and intersubjective meanings as they interact with the world around them” (Trauth, 2001; p. 6). In IS research, interpretive methods are aimed at understanding the context within which an information system exists and how it influences and is influenced by this context (Walsham, 1993). Critical studies aim to critique the status quo of deep-seated structural contradictions within social systems with an aim to transform alienating and restrictive social conditions (Trauth, 2001). Critical studies in ISD consider information systems in their wider social context, 9

focusing on issues such as power, domination, conflict and contradiction (Howcroft & Trauth, 2004). The users’ minor role in IS development has been due to an underlying assumption that there is an objective truth in organizations that can be mapped to develop successful information systems (R. M. Wong & Tiainen). This positivist perspective has ignored the view that information systems development is a social process shaped by people, both users and developers. We embraced this interpretivist perspective in order to understand the importance of user interaction in the process of ISD. The investigation of social phenomena should be interpretivist in its orientation so that contextualized meaning can be revealed to enable socially-based phenomenon to be understood (Butler & Fitzgerald, 1997 ). The interactions between users and developers are socially-based phenomenon. We were interested in understanding what “user involvement and participation” means for systems developers and how their understanding of the term differs depending on their level of formal training, therefore we used an interpretive lens. In this case, there was a high degree of uncertainty surrounding the research problem as very little research explored the relationship between developers’ formal education and their perception of the importance of user interaction. Past research in this area has stressed how user interaction results in increased levels of user satisfaction and perceived usefulness of the system (Terry & Standing, 2004). However, research regarding the relationship between user interaction and success of information systems development is not grounded in theory nor substantiated by research data (Butler & Fitzgerald, 1997 ). The last factor pertinent to this research is the researchers themselves. Researchers familiar with qualitative research methods are more likely to use them. Members of the research team had past experience with qualitative interviewing, focus groups, and content analysis and felt comfortable 10

to use qualitative approaches for this study. 3.3 Research Design A methodological strategy is the logic by which one goes about answering research questions (Mason, 2002). Having decided on qualitative methods for investigating the phenomenon of interest, we attempted to link our research questions and rationale to research methods and practicalities. We adapted the table from (Mason, 2002) and linked her criteria with the research rationale described above to help us decide on a research method. How does developers’ Interviews of developers Attempt to discover Need two groups style of user difference of opinions of ISD interaction differ based Other Possibilities: between developers developers: on whether the observation, interviews based on their formally and developer is formally with users acquisition of informally or informally trained? technology skills. trained. During which phases Interviews of developers Attempt to elicit Need access to of the ISD processes personal standards of diverse pool of do developers seek Other Possibilities: user interactions from ISD developers users’ knowledge? observation, document developers. analysis During which phases Interviews of developers Attempt to elicit Need ISD of the ISD processes when developers developers who do developers feel that Other Possibilities: believe that users’ have worked on the users’ knowledge observation, document knowledge is needed. ISD portions with is highly needed? analysis users involved. Table 1: Developing a research design (Mason, 2002) Our ontological position suggested that developer’s knowledge, formal training, views, and experiences shape their perception of how important user involvement is to the process of ISD. Our interpretivist epistemology lead us to believe that via interviews we could construct the social reality of the phenomenon under study by getting first-hand accounts of how developers feel their training in formal or informal fields affects how they treat users in the process of developing systems. Talking to developers and observing their attitude towards user involvement 11

enabled us, as researchers, to be active and reflexive in the research process (Mason, 2002). The inclusion criterion for subjects in this study was that they should have been involved in the development of at least one information system, either during the course of their education (say as a graduate student research project) or in the course of working in industry. An information system, in general, consists of software, hardware, databases, servers, online services and integration tools. While this definition of information system definitely qualifies for the purposes of our research, we would also consider a software system or tool as an information system. An information system development life cycle for this study is composed of planning, analysis, design, implementation, and testing phases (Dennis et al., 2005), (Maciaszek, 2001). We were interested in interviewing a total of twelve subjects, six of whom were formally trained in ISD practices and techniques and another six informally trained. “Formal” training meant undergraduate degree in Computer Science (CS). Subjects with undergraduate degrees in all other fields were considered “informally” trained. These developers were self-taught in that they never had courses in software engineering or development techniques but had picked up ISD skills through books or online resources. Due to ease of accessibility, the subject included four current graduate students at the College of Information Science and Technology, three Penn State University Libraries staff and one Penn State Information Technology Services staff. Using a semi-structured interview method, we asked subjects open-ended questions related to the above mentioned themes. Talking to developers about how their training, formal or informal, has shaped their practice of user involvement in the ISD helped us to explore these issues in their social context. 12

3.4 Data Collection This research followed Mason’s suggestions for planning and conducting qualitative interviewing (Mason, 2002). She suggests we employ qualitative interviewing when we “want to give maximum opportunity for the construction of contextual knowledge” (Mason, 2002, p. 64) based on the participants’ experience. In this research, we were constructing knowledge about participants’ perception of user interaction by examining how developers’ training has influenced their ISD experience. An example of “constructing knowledge” is that instead of asking the interviewees to sort a given list of ISD phases, we would rather expect them to provide a list of ISD phases. 3.4.1 Subjects We designed the study to include twelve participants, six of whom were formally trained in ISD and the other six are informally trained. They were contacted via email (see email script in Appendix A), and asked to participate in semi-structured interview lasting no more than forty- five minutes. All participants contacted to participate in our study were Pennsylvania State University staff or graduate students, and colleagues of the authors. In addition, we sought people with technical skills and at least a year of professional experience in ISD. From the initial twelve emails sent out inviting people to participate in our study, ten replied confirming their interest in the study, one decided not to participate, and one did not respond the email. From these ten confirmed participants, we were able to schedule only eight interviews because of some time constraints. Fortunately, based on the eight interviewees’ academic background and professional experience, we were able to form even groups of interest. Taking into consideration the academic background, we had four participants formally trained and four participants informally trained. Considering professional experience, all participants had 13

experience with industry while four participants moved from industry into academia. Apart from the specific requirements above, this research considered that any other variable, such as age, gender, race, and nationality, would not be important for the scope of the research. Nevertheless, we had a well-balanced number of participants with different characteristics (men/ women, young/old, and different nationalities). 3.4.2 Interview Questions The interviews we designed for this research (see Appendix B) consisted of open-ended questions related to three main themes – participants’ education and skills, participants’ perception of what the ISD process is, and participants’ perception of user interaction in ISD. The first theme, regarding educational and skills, helped us to learn and identify the interviewee’s background in systems development. Although we had a broad idea about what a ‘formally’ trained developer would be, we relied on our findings to identify what kind of formal education actually influences the way developers perceive the user’s interaction in ISD. The development process theme covered information about the interviewee’s knowledge about ISD phases and sub-phases, as well as their relations with user’s participation and involvement in ISD. We expected that people with different educational backgrounds would have different opinions about the ISD phases. Lastly, the user participation and involvement theme would indicate how important the interviewee feels user interaction is within the ISD phases. For this theme, we were expecting to learn about the interviewee’s experience involving the user in different ISD phases. Although, it is widely known that users’ participation in ISD should be carried throughout all the ISD phases, we suspected that it would not happen in practice. Each interview question aims to understand and to cover a specific category within the themes. These categories are the initial source for our coding schema (see Appendix C). 14

3.4.3 Interview Process Each interview lasted approximately forty-five minutes, and was performed by at least two interviewers. One interviewer was responsible for conducting the interview and asking the open- ended questions. The other interviewer was responsible for taking notes, although, the interviews were also audio recorded. The second interviewer could also contribute with questions, comments, or clarifications. After each interview, the interviewers got together to discuss possible issues or suggest corrections for the next interview. Sub-section 3.4.4 presents a summary of the main issues found during the interview process. The environment, where the interviews took place, performed an important role for the accomplishment of the interviews. The collaborative space room was considered by the group as the best place to conduct interviews. It provided us with a quite place, and an excellent layout for talking to the interviewees. The interviewees’ office was relatively quite, except for some interruption of their officemates. Lastly, the interviewers’ laboratory was the worst place to interview people because it provided a significant level of distraction to the interviewee. The two laboratories used to do the interviews are shared with other graduate students, and consequently caused too much noise. 3.4.4 Issues The first interviews helped us to identify issues such as missing questions and topics that are relevant to the information we are seeking in this research. Instead of asking participants about their proficiency in computer languages and development methods/tools, it was more useful to ask them about their experience with the same. Based on the first interviews answers, we noticed that the term “proficiency” implied the participant’s current ability with a technology. On the 15

other hand, the term “experience” implied all abilities (present and past) with a technology. By switching the terms, we were able to generate more data about the participant’s experience. The sequence in which the interviewer asked questions was also changed to allow the conversation to flow smoothly, as well as to help the interviewees to better remember their experiences. For instance, we noticed that first asking the interviewees to list their experience with programming languages and development methodologies would help them to trigger their memory to remember how they learnt their skills. Asking the other way around was making the interviewees to spend extra time retrieving their memory in order to provide an accurate answer. During the interview process, we also came across some issues resulting from the authors’ lack of experience with conducting interviews. First, because some interviewees provided vague answers to the questions asked, we noticed a tendency for interviewers to lead the participants’ answers. A possible explanation for that is because the questions were not well formulated and led to multiple interpretations, or the interviewee was not elaborating enough and the answers were not producing the adequate information. Second, some interviewees had the tendency to speak a lot, and consequently dominated the interview. In these cases, we witnessed the tendency to add extra information to illustrate points, usually extending the conversation, and introducing many facts (relevant and irrelevant). Guided by our curiosity and the expectation of some useful data, we usually let the interviewee talk until we felt that we were not getting anymore relevant information. It was a challenge for the interviewers to interrupt the conversation and get it back on track. Third, some interviewees suggested us to hand out the questions in advance, so they would be better prepared to answer them. Having the questions in advance would help the interviewees to go through their past professional and academic experiences, and to remember relevant facts to answer the interview questions. Prior knowledge of the questions could also 16

sway the position of the interviewee in the realm of user interaction and its importance. Finally, some interviewees got easily distracted, and started to give us irrelevant information. For instance, an interviewee became interested for one of the author’s MAC laptop, and started to ask about its features. Another significant issue was defining where to draw the line between ‘formally’ and ‘informally’ trained developers. Our initial idea was to define people with degrees in computer science and information systems as formally trained and all others as informally trained. This presented a philosophical difficulty because although CS and IS programs may have some overlap in terms of the courses taught, the fields themselves have different underlying philosophies with respect to the importance of the user in the development process. This could mean that ISD training imparted in these fields is very different and hence we need to redefine our notion of ‘formally’ trained developer. Therefore the number and the type of such courses typically taken by participants’ from CS and IS helped us to define ‘formal’ training for the purposes of this research. 3.5 Role of Researchers All members of the research group took an active interest in all aspects of this study and worked well as a team. Through regular meetings members of the team discussed the research and the findings as they emerged, as well as contributed equally to writing reports, recruiting participants, and conducting interviews. Each researcher recruited four participants, conducted at least two interviews, took notes of at least two interviews, and coded four interviews. The approaches provided us with valuable opportunities to learn and perform different tasks in a qualitative research process. More information about the role of researchers can be found in sub- sections Preconceptions, Interview Process, and Data Analysis. 17

4 RESULTS 4.1 Data Analysis Data analysis began during the first interview and continued to refine itself throughout the entire data collection process. Since qualitative methods were employed, much of the analysis included the researchers internally processing the information and then elaborating their thoughts during meetings and across email. The analysis process included physical processing of data, data categorizing and coding, and analysis. 4.1.1 Processing of Data Physical processing of data included two main tasks: reviewing and transcribing interview audio and integrating additional researcher interview notes. During interviews one researcher was appointed interviewer and responsible for ensuring adequate coverage of all the main themes of our interview guide (see Appendix B). That researcher was also responsible for the interview transcription. This ensured that the interviewer gained a full working knowledge of the interview since they were typically slightly distracted by formulating questions and exploring additional opportunities the interviewee may have mentioned. The other researchers present for the interviews were responsible for note taking and then integrating those notes into the appropriate location within the transcripts. This was useful for the coding and analysis because it highlighted when and where areas of researcher enlightenment became apparent within the process. The revisiting of interviews and notes was an important factor in developing the researchers’ intimacy with the data. 4.1.2 Categorizing and Coding We were interested in understanding what “user involvement and participation” means for systems developers and how their understanding of the term differs depending on their level of 18

formal training, therefore we were using an interpretive lens to understand the data. Mason cautions that the main issue with using an interpretive lens is representing research participants’ views rather than imparting your own views onto the data they provide (Mason 2002, p.76). In order to counteract that tendency, we individually formulated our thoughts on emerging concepts and then held group meetings to check our approaches. During the research group meetings specific categories of interest became evident. Some of the categories were driven by literal data such as work experience, years, type of training while others were more interpretive in nature; user importance. Eleven high-level categories emerged from the group meetings and were utilized to create data codes. Those categories are described in detail in Appendix C. The allocations of interview participants in the identified categories are presented in Table 2. 19

Category Codes Participant Results Academic The participants included four formally trained with Computer Science Background degrees (two recent graduates, two experienced graduates) and four informally trained with degrees in Business, Engineering, Geology and Hotel Restaurant & Management. Acquired Skills All participants cited self-teaching as main approach to skill acquisition. Half of the formally and informally participants gained skills through coursework. One formally trained participant utilized all three venues for skill acquisition. Languages Only two of the formally trained participants had both types of experience while two of the formally and one informally trained participant had low- level programming experience and three informally trained participants had script/web experience. Information Unfortunately it was not until the researchers were half way through the System interview participants that it emerged as potentially important. For the Definition participants captured (two formally and three informally) only one of the formally could defined an IS and all of the informally had working definitions. Information The participants typically cited a waterfall or cyclical approach with only one System Phases formally trained participant unable to identify any phases to the ISD process. Career Path Four of the five graduate experience developers also had experience in the software industry while two of the four information technology experienced developers started their career in industry other than technology. Knowledge Three of the four direct type developers were also informally trained while Elicitation three developers were indirect and one developer had no interest in knowledge elicitation from users. Past Participation Two of the four formally trained and two of the four informally trained in ISD developers participated in all phases they identified. Past User In one extreme case, a developer admitted, without a bit of remorse, that Interaction when lacking appropriate details about requirements they would simply role play scenarios of the system in their head and then use their best guess to fill in the requirement gaps. Participant’s A majority of the participants did not see a difference between ideal and Ideal User current ISD, the few that would change the interaction were typically Interaction involved in low-level programming and would only change if the project necessitated user interaction. User Importance The participants were fairly dispersed along the levels of user importance: one None, three Low, one Medium, and three High level. Table 2: Applying Code Schema to interviews After we identified the eleven categories, two researchers applied the categories to the results at a 20

high level (Figure 1). Figure 1: High-Level Interview Coding Independently, the third researcher mapped the categories and codes to the transcripts and integrated notes (Figure 2). The qualitative software package Atlas was employed to perform content analysis of the interview transcripts. The low-level codes assigned during the content analysis were later mapped to the high-level codes developed by the other two researchers. This approach ensured corroboration of results by two independent coding techniques and helped decrease individual researcher biases that may have influenced data interpretation. 21

Figure 2: Atlas Coding 22

4.1.3 Analysis Through categorizing and coding of the interview data, three main factors emerged as promising determinants of developer’s user interaction beliefs: Academic background (formal and informal training), experiences and access to users. To analyze these factors we looked at the interpreted user importance levels versus the individual factors with the assumption that user importance levels are key to determining developer-user interaction during the ISD process. Examining formal and informal training versus our interpreted user importance illustrates that formal training (Computer Science) spread evenly across the importance levels while informally trained users had a higher level of user importance. High Medium Other Low Computer Science None 0 1 2 # of Participants Figure 3: Academic Background vs. User Importance This comparison leads us to believe that academic experience plays a smaller role than we originally assumed. This does not mean that academic experience plays no role in the developer- user interaction. There is also notable increase in user importance with participants that received an informal training in Information System Development. This may be due to their own experiences prior to becoming a developer. The second factor, developer’s experience, refers to the languages the developer has experienced 23

and the career path they took to get there. Two of the formally trained developers’ experience consisted of low-level programming and graduate work while the other two had experience in both low-level and script/web programming. All four of the informally trained developers had strictly script/web programming experience. High Medium Both Low Script/Web Low-Level None 0 1 2 # of Participants Figure 4: Experience vs. User Importance Low-level developer’s tended to have a lower level of user importance while the others had a high level of user importance. We believe this tendency of Low-level developer’s was a product of their work environment. The Low-level developers focused on more system based programming while the Script/Web developers focused on high level applications with a higher level of usability for gathering and displaying information. While users utilize system based programs they do not typically interact and extract information from those programs. System based programs are more task-driven than information distribution driven. 24

The final factor, developer’s access to user, refers to the level of insulation between the developer and the user during the ISD process. Throughout our interview and data collection process it became apparent that user interaction beliefs hinged on the role of the developer within the ISD process. If the developer obtained the user information indirectly (through a third party) then the level of importance was typically lower. We believe this was due to the insulation between the developer and user. Since the user is less prominent in the developer’s work, they are not as visible and therefore not deemed as important. High Medium Direct Low Indirect None 0 1 2 # of Participants Figure 5: User Knowledge Access vs. User Importance 4.2 Findings Our main research question was how developers’ style of user interaction differs based on whether the developer is formally or informally trained. We found that developers’ academic background played a small role in how important they perceived the user to be. In shaping developers’ perception of the importance of user involvement on-the-job experience and work context were seen to play a more important role than their academic background and education. Specifically, we found that developers with undergraduate degrees in computer science did not 25

perceive user involvement to be as important as did self-taught developers. This shows that formal education in software development does not emphasize the importance of user involvement in ISD. Self-taught developers perceived user involvement to be highly important mainly because their organizational context and work experience had taught them the importance of user involvement. Developers’ perception of users and the importance of interacting with them during the development of an information system were highly influenced by their work context. The organization’s work culture, development philosophy, freedom and constraints placed on developers with respect to resources and access to users were found to influence the way developers actually involved the user to a much larger extent than what they may or may not have learned through their formal education. We also found that developers working with Web-based or scripting applications perceived the user to be more important than programmers who write low-level code. Butler (2003) makes a distinction between Web-based information systems and “traditional” information systems with respect to the social and organizational issues associated with their development and implementation. Along those lines, we found that developers who write Web-based or scripting applications are more likely to perceive user involvement as important. It could be due to the fact that Web-based applications are more interactive and user satisfaction is much higher when the user is highly involved iteratively in the development process. Finally, the access to the user or “level of insulation” between the developer and the user affects developer-user interaction. The higher the “level of insulation” the developer has from the user the less important user interaction is for the developer. By “level of insulation” we mean organizational hierarchy which prevents the system developer from coming in direct contact with 26

the user for knowledge elicitation purposes. We found that low-level programmers were “insulated” from the user and obtained user requirements and needs via other members of the development team who actually interacted with the user. The organizational hierarchy prevented them from directly talking to users. Therefore, when they perceived insufficient or incomplete information from the user, they were unable to go back to the users and get more information. In this case, one formally trained developer said he would “make assumptions about how the common user is going to use” the system. Even then, being an expert, it was hard for him to imagine how a user without technical-knowledge would use the system he was designing. This example highlights that work context and organizational constraints influence how far developers involve the user much more than formal training does. 5 DISCUSSION In summary, developer’s training, whether it be formal or informal, plays only a small role in determining developer’s beliefs in user interaction within the ISD process. Work experience and context play a much larger role than academic experience because developers must conform to their current working environment. Developer’s in script and web based ISDs tend to interact with the user more which we believe increases their likelihood to deem the user as important to the system development. Lastly, the level of access to the user also plays a factor in user importance and therefore user interaction seeking because of the nature of low-level programming. 5.1 Implications The findings of our study have important implications for education and industry. They make us question the role of education in making systems developers aware of the importance of user involvement. Our research illustrates that developers with undergraduate degrees in computer 27

science did not receive any formal courses that stressed the importance of involving users. These formally trained developers were typically recruited in coding jobs. Among them, the low level programmers typically did not learn the importance of the user because of their insulation from them. This raises concerns about the inadequacy of traditional computer science curricula to address the need for teaching developers about the importance of user involvement. Computer science curriculum studies are still slow in embracing user-centered development methods (Kaasinen & Clarke 1998). User-centered design techniques should be taught as part of traditional CS courses as they stress on involving users throughout the development phases. According to Rosson & Carrol (2001), usability has been studied for the last three decades, and become a focal point in software development, especially in requirements analysis and interfaces design. Seffah & Andreevskaia (2003) found that most of systems analysts have little or no training in usability techniques and suggested that developers should be made aware of user- centered techniques. Also, our study raises concerns that organizations hiring computer science graduates are hiring systems developers who do not necessarily perceive users to be important to the process of ISD. Consequently, much of the usability training and education for developers is left to corporate training organizations (Kaasinen & Clarke 1998). This may lead to products being developed which have low usability and lead to low level of user satisfaction unless the organization has a culture of promoting user interaction in ISD. Thus, if organizations are interested in hiring developers who perceive users as important, they will have to either hire developers who have in their past job-experience treated users as important or train them to consider the same. 5.2 Contributions Past research into importance of user involvement in ISD has looked mostly at end-user training 28

as important for the success of information systems. However, very little research has looked at how developers’ formal educational background and training influence their perception of user interaction. This research highlights potential factors for future research within the realm of developer examination. It also illustrates that current educational efforts are not determining the ways developers choose to interact with users. 5.3 Limitations Due to time and resource constraints, our study has several areas for improvement. Even though we were interested in studying the differences between formally and informally trained developers, we struggled with the issue of defining “formally trained” developers throughout our research. We started out considering that developers who had courses in software engineering or ISD methodologies qualified as “formally trained”. Using this criterion, we decided to consider undergraduate degrees in computer science and information systems as “formal” training. However, the underlying educational philosophies of computer science and information systems fields with respect to the perception of the user differ considerably. Also, as we spoke to computer science graduates, we realized that most of them had not had any courses in software development methodologies or software engineering. Thus we changed our initial definition of formally trained to include only computer science graduates as they had formally learnt ISD skills as opposed to all other degree-holders who had learnt ISD skills on their own. Also, our study used a small sample of eight developers and the sample could be considered to be biased as most of the developers we talked to belonged to the same organization. The graduate students were mostly from the College of Information Sciences and Technology and the developers in industry were Penn State University Libraries or Information Technology Services staff. Since we found that work context plays a large role in perception of the user, using a more 29

diverse sample of developers would have lent more credibility to our research. 5.4 Future Research Our research highlights a gap in IS literature with respect to the developer’s role in user- developer interactions during the ISD process. In the future this research can be extended by using a larger sample size and talking to developers from diverse organizations. The distinction between Web-based and traditional developers with respect to their difference in perception of the user needs to be studied in more detail. Some unexpected issues emerged from our research other than the ones we were interested in addressing. We found that the age of developers affected their perception of user interaction. Older formally trained developers said that their education was so long ago that it had ceased to have much of an influence on their career today. Also, computer science degrees in the 70s and 80s did not include software methodology courses. Thus, older developers tend to be more influenced by their on-the-job experience than younger developers whose education is still fresh in their memory to be applied in their work. 6 CONCLUSION Our research preliminarily highlights a gap in the area of developer-user interactions. Although our sample group was small, we believe it was a significant source of evidence towards the determining factors in developer’s user interaction beliefs. The three main factors that emerged in this research were developers’ academic background, experience and level of insulation. With further exploration of these factors, we believe that valuable insights can be revealed to aid with future educational, employment, and research efforts. 7 REFERENCES Bailey, J. E., & Pearson, S. W. (1983). Development of a tool for measuring and analyzing computer user satisfaction. Management Science, 29(5), 530-545. 30

Barki, H., & Hartwick, J. (1989). Rethinking the Concept of user Involvement. MIS Quarterly, 13(1), 53-63 Barki, H., & Hartwick, J. (1994). Measuring User Participation, User Involvement, and User Attitude. MIS Quarterly, 18(1), 59-82. Butler, T., & Fitzgerald, B. (1997 ). A case study of user participation in the information systems development process. Proceedings of the eighteenth international conference on Information systems, Atlanta, Georgia, United States Cavaye, A. L. M. (1995). User Participation in Systems Development Revisited. Information and Management, 28, 311-323. Doll, W. J., & Torkzadeh, G. (1989). A Discrepancy Model of End-User Computing Involvment. Management Sciences, 35(10), 1151-1171. Emam, K., Quintin, S., Madhavji N. (1996). User participation in the requirements engineering process: An empirical study. Requirements Engineering, 1(1), 4. Fitzgerald, B. (2000). Systems Development Methodologies: the Problem of Tenses. Information Technology & People, 13(3), 174 -185. Grudin, J. (1991). Interactive Systems: Bridging the gaps between Developers and Users. IEEE Computer, 24(4), 59-69. Hartwick, J., & Barki, H. (1994). Explaning the role of user participation in Information System Use. Management Science, 40(4), 440-465. He, J. (2004). Knowledge Impacts of User Participation: A Cognitive Perspective. Paper presented at the SIGMIS, Tucson, AZ, USA. Howcroft, D. and E. M. Trauth (2004). The Choice of Critical Information Systems Research. Relevant Theory and Informed Practice - Looking Forward from a 20 Year Perspective on IS Research. Boston, MA, USA, Kluwer Academic Publishers Hwang, M. I., & Thorn, R. G. (1999). The Effect of User Engagement on System Success: A Meta-analytical Integration of Research Findings. Information and Management, 35, 229-236. Iivari, N. (2004). Enculturation of User Involvement in Software Development Organizations: An interpretative Case Study in the Product Development Context. ACM NordCHI, Finland, 287-296. Ives, B., Olson, M. (1984). User Involvement and MIS Success: A Review of Research. Management Science, 30(5), 586-603. Kautz, K., & Pries-Heje, J. (1999). Systems Development Education and Methodology Adoption. ACM SIGCPR Computer Personnel, 20(3), 6-26. Latzina, M., & Rummel, B. (2002). Collaboration-Based Usability training for Developers. Mensch & Computer 2002: Vom interaktiven Werkzeug zu kooperativen Arbeits- und Lernwelten. Stuttgart: B. G. Teubner, S. 285-291. Mason, J. (2002). Qualitative Researching (2nd Edition ed.). London, UK: Sage Publications. Myers, M. D. (1997). "Qualitative Research in Information Systems." MIS Quarterly 21(2): 31

241-242. Nosek, J. T. (2001). Justifying the Business Value of Information Systems Education: A Report on Multicultural Field Experiments. SIGCPR, San Diego, 205-211. Orlikowski, W. J. and J. J. Baroudi (1991). "Studying information technology in organizations: Research Approaches and Assumptions." Information Systems Research 2(1): 1-28. Roberts, E. (2000). Computing Education and the Information Technology Workforce. SIGCSE Bulletin, 32(2), 83-90. Robey, D., & Farrow, D. (1982). User Involvement in Information Systems Development: A Conflict Model and Empirical Test. Management Science, 28(1), 73-85. Rosove, P. E. (1969). Training Requirements for computer programmers in relation to system development phases. SIGCPR, Illinois, 65-76. Rosson, M. B., & Carroll, J. M. (2001). Usability Engineering: Scenario-Based Development of Human-Computer Interaction. Morgan Kaufmann. Seffah, A., & Andreevskaia, A. (2003). Empowering Software Engineers in Human-Centered Design. IEEE International Conference in Software Engineering, 653-658. Straub, D., D. Gefen, et al. (2004, January 7, 2005). "The ISWorld Quantitative, Positivist Research Methods Website." Retrieved February 5, 2006, from http://www.dstraub.cis.qsu.edu:88/quant/. Tait, P., & Vessey, I. (1988). The Effect of User Involvement on System Success: A Contingency Approach. MIS Quarterly, 91-108. Terry, J. and C. Standing (2004). "The Value of User Participation in E-Commerce Systems Development." Informing Science Journal 7 Trauth, E. M. (2001). Qualitative Research in IS: Issues and Trends. Hershey, PA, USA, Idea Group Publishing. Walsham, G. (1993). Interpreting Information Systems in Organizations. New York, NY, USA, John Wiley & Sons, Inc. Wilson, S., Bekker, M., Johnson, P., & Johnson, H. (1997). Helping and Hindering User Involvement – A Tale of Everyday Design. ACM-CHI’97, Atlanta, 178-185. Wong, E., & Tate, G. (1994). A Study of User Participation in Information Systems Development. Journal of Information Technology 9(1), 51. 32

8 APPENDIX A Email script for recruiting participants Dear <Participant>, We would like to request your voluntary participation in our study “Effects of Developers’ Training on User-Developer Interactions in Information Systems Development” (IRB# 22846). We believe that you are currently, or have in the past, worked on designing, developing, implementing or maintaining an information system. We would like to interview you about your experiences during this process, specifically about your involvement of the user. The questions in the interview will be related to the following main themes: your educational background, how you practice information systems development, where you learnt the skills you use, and when and how you involve the user. The interview will be audio-recorded with your permission. The interview will take place at a time and location convenient to you and will not exceed 45 minutes. This research may shed light on how you, as a developer, perceive user involvement as an important part of the information systems development process. Also, this could lead you to a better understanding of how your formal training in information systems development, or the lack thereof, leads you to involve the user. If you are interested in participating in this research, please reply to this email with possible times when you would be available to participate in the interview. If you have further questions about the research, please contact Andrey Soares at asoares@ist.psu.edu. This research is associated with the Spring Semester IST541: Qualitative Research in IST at The Pennsylvania State University. Thank you for your consideration, Jennifer McCauley, Sharoda Paul, and Andrey Soares Sharoda A. Paul, IST PhD Graduate Student 323 Information Sciences and Technology Building University Park, PA 16802 (814) 865-6846 spaul@ist.psu.edu Jennifer McCauley, IST Master Graduate Student 102 Paterno Library University Park, PA 16802 (814) 863-7098 jmccauley@ist.psu.edu Andrey Soares, IST PhD Graduate Student 309 Information Sciences and Technology Building University Park, PA 16802 (814) 865-6177 asoares@ist.psu.edu 33

9 APPENDIX B Interview Questions Guideline The following interview questions will cover three categories that we believe can help us to identify and to reveal how developers: learn their skills, understand systems development process, and perceive the user participation in ISD phases. Education and Skills • What is your academic background (i.e. Computer Science, Geography, etc)? • How did you learn the skills to become a system developer (i.e. Associate Degree, Bachelor Degree, Master Degree, PhD Degree, Technical/Professional Courses, or Self)? • Which programming languages and methodologies you have experience with (i.e. Java, C, Perl, Visual Basic, O-O, E-R, etc)? • What is Information Systems? • How long have you worked developing systems for? • List all job positions related to Information Systems Development that you have already performed (i.e. Programmer, systems analyst, Project Manager, etc)? Please indicate how long you spent working in each position. Development process • What are the phases and sub-phases of an Information Systems Development life cycle (i.e. analysis, implementation, etc)? • How do you learn about new suggestions and new implementations? • Who is responsible for designing interfaces and reports? Do you ask for user’s approval? • How much time developers usually spent testing the system? User Participation • What is the importance of users’ involvement in IS development? Explain. • Which activities of an ISD do users get involved in (i.e. requirements, etc)? • How important user suggestions are to an ISD process (i.e. very important, reasonable, and unimportant)? Which ISD phases? • Are users usually involved in quality assurance? • Have you worked with users for capturing their knowledge about the systems? o If yes:  How did you collect the user’s knowledge? Which ISD phases?  How did you save their knowledge? o If no:  Who was the information source?  How is the information captured?  How was the information passed to you? In what format?  Was the information given to you enough to understand the system?  Would you be able to explain how the system works? 34

10 APPENDIX C Coding Schema − Academic background - Undergraduate or graduate work within formal or informal areas relevant to ISD. o CS o IS o Business o Engineering o Other − Acquired Skills - Acquisition of skills used to build ISD’s were acquired: coursework, self-teaching, or training (sponsored by industry employer). o Courses o Self-training includes books, tutorials, web resources o Required training − Languages - Type of programming languages developer has utilized while participating in ISD: Low-level programming (Drivers, kernel, operating system based) or Scripting/Web based (Javascript, ColdFusion, etc) or both. o Programming o Scripting/web o Both − Definition of IS - This category was created to capture whether or not the participant could define an information system as an entity. o Can and what they are o Can’t − Career path - Capturing the path the developers took towards ISD experience. o Information Technology (Primary & Secondary) o Non-Technology o Undergraduate and Graduate work − IS phases – Meant to capture whether or not the developer could define specific phases of development, o Can and what they are o Can’t − Knowledge Elicitation - Within user interactions, vital information for ISD is elicited through varying approaches, we categorized these into two types: direct and indirect. The direct approaches involved the developer interacting with the user while the indirect approach includes involvement of a third party to gather and then translate/communicate requirements to the developer. − Past Participation in ISD - This category sought to identify the level of experience the developers had with ISD. o Match to part of the definition of IS − Past User Involvement - Designed to identify if participants actually practice what they “preach”. − Participant’s Ideal User Interaction - Politics motivate many areas of our lives and 35

development of ISD is no exception. This category was an opportunity for the participants to identify their ideal user interaction level within ISD without regard to their political situation. − User suggestion importance - The most interpretive category in our research, user importance was determined based on the researcher’s impression of the participant and not the participant’s declaration of user importance. o none (“I am the user”) o low (“I know the user exists, but are not necessary”) o medium (“When possible it is a bonus to involve the user”) o high (“I start with the user and end with the user”). 36

http://www.personal.psu.edu	13
http://static.slidesharecdn.com	2
http://www.lmodules.com	1

Effects of Developers’ Training on User-Developer Interactions in Information Systems Development

by Jennifer McCauley

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 16

Statistics