Research methods for socio-technical systems analysis (LSCITS EngD 2012)


Published on

Introduces research methods that may be used for the study of socio-technical systems

Published in: Technology
  • Hi Henock

    This version was written in 2012
    Are you sure you want to  Yes  No
    Your message goes here
  • I want to refer this article for my research paper and i couldn't find any date when it is written. please feel free to reply by email at
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • In practice – paper strips are thrownExtra things – paper cups
  • Research methods for socio-technical systems analysis (LSCITS EngD 2012)

    1. 1. Research methods for socio- technical analysis Ian SommervilleResearch methods, EngD course, 2012 Slide 1
    2. 2. Human and organisational issues • When designing, testing or evaluating critical systems we need to know about how that system will be used in real settings – What do users really do and why? – What potential is there for error? – How are errors avoided or mitigated? • There are many useful generic insights into human factors – For example the GEMS model • But systems engineers will inevitably need to understand human factors that are specific to the system they are constructing or maintaining – Specifics emerge from: the location, the people, their work, the technologies in use, etc. • Process models are too abstract to tell us what we need to know about what users doResearch methods, EngD course, 2012 Slide 2
    3. 3. Human and organisational issues • Because of the importance of human and organisational factors, systems engineering draws from fields such as – psychology – sociology – organisational studies • But draws from their methods as much as their results – Don‟t expect research done in other disciplines to provide all the necessary answers for questions in systems engineering – Don‟t expect that researchers from other disciplines will understand or be interested in technology systems. Some are, many aren‟t • Systems engineers have to – Find out about human and organisational factors themselves – Work closely with people from other disciplinesResearch methods, EngD course, 2012 Slide 3
    4. 4. The devil is in the detail • The fundamental principle of software design is abstraction – Find the commonalities between the entities in the system and represent these as software elements • This is based on the assumption that the common features are the most important and that stakeholders can adapt to them • Social science perspective is rather different. The focus is on the detail of a situation – The key argument is that the details are very significant indeed and that failure to pay attention to them leads to problemsResearch methods, EngD course, 2012 Slide 4
    5. 5. Studying socio-technical systems • Systems engineers need to find out about about human, social and organisational issue that are specific to the system they are working on and to the class of systems being developed • There are several useful ways to do this 1. Experiments 2. Archive research / Case studies 3. Interviews and focus groups 4. Simulation 5. Fieldwork • And several less useful ways (which are better than nothing) 1. Questionnaires and surveys 2. Imagining what users doResearch methods, EngD course, 2012 Slide 5
    6. 6. Case study analysis • Retrospective analysis, looking back at a system or incident (a case) that has occurred • In-depth study, based on a detailed analysis of a set of information sources • Based around one or more research questions e.g. – What were the factors in system X that led to catastrophic failure? – What management decisions contributed to the cost overrun in system Y? • Intention is to gain generally useful insights from specific casesResearch methods, EngD course, 2012 Slide 6
    7. 7. Examples of case studies • “Normal accidents‟ – The problem of nuclear reactor overheating at 3-mile Island • Denver airport baggage system • The London Ambulance Computer-aided Dispatch System • Airbus accident – Warsaw airport • Case studies tend to focus on system failure rather than system successesResearch methods, EngD course, 2012 Slide 7
    8. 8. Stages of case study analysis • Determine and define the research questions • Select the cases and determine data gathering and analysis techniques • Prepare and collect the data • Collect data in the „field‟ (if appropriate to do so) • Evaluate and analyse the data • Prepare the report of the case studyResearch methods, EngD course, 2012 Slide 8
    9. 9. Research questions • Establish the focus of the study by forming questions about the situation or the problem to be studied and so defining the study‟s purpose • The question or questions are then used to select material for analysis • Accident enquiries are focused case studies where the question is decided in advance i.e. – Why did accident X occur? • Danger of bias in establishing the research question – What management problems led to the collape of Lehmans? – What were the systemic factors that led to the collapse of Lehmans?Research methods, EngD course, 2012 Slide 9
    10. 10. Data gathering and analysis techniques • Case study selection – Often the most „typical‟ case study does not reveal the greatest insights – Choose case studies that are • Extreme - representative of an exceptional situation e.g. an accident • Critical - having strategic importance with respect to the general problem • Data to be analysed – Essential to use multiple sources of data • Interviews • Archives and reports • FieldworkResearch methods, EngD course, 2012 Slide 10
    11. 11. Data collection and analysis • Typically, case studies generate large amounts of qualitative data – Transcripts of interviews – Reports and associated notes • Look for opportunities for triangulation – Evidence from diverse sources that supports the same conclusions • Field data is also important – Look at the object of the case study as it is – The system in operation • Analyse the data in different ways and from different perspectives – Use quantitative data to corroborate qualitative dataResearch methods, EngD course, 2012 Slide 11
    12. 12. Case study issues • Generalisability – Much of our learning is through case studies so we can make generalisations from them. However, we have to be aware of the possibility that specific factors may be dominant in any one study and be careful about generalisation – Case studies can be useful for finding „black swans‟ i.e. contradicting received wisdom • Hindsight bias – Seeing events as more significant than they actually were because there is a clear link between these events and the research questions • Obsolescence of resultsResearch methods, EngD course, 2012 Slide 12
    13. 13. Fieldwork - Get Out of the Office!Research methods, EngD course, 2012 Slide 13
    14. 14. Fieldwork • Generic name for a set of techniques which involve collecting data directly from a worksite (the “field”), rather than relying on reports from other people • Encompasses a range of techniques, which have much in common: – Direct observation and discussion – Ethnography – Cultural and technology probesResearch methods, EngD course, 2012 Slide 14
    15. 15. Ethnography of work • Situated observation of work with a view to developing a detailed understanding of the way that work is done and how the actors respond to other actors and the context of the work • The aim is to develop a „rich picture‟ of the work done • Ethnomethodological ethnography – Ethnography that is intended to describe what work is rather than explain work in terms of some social theory – Approach proposed in the 1980s at Xerox, developed at Lancaster University in the 1990s. – Work at Lancaster related ethnography explicitly to systems design.Research methods, EngD course, 2012 Slide 15
    16. 16. Ethnography in practice • Looks at work in „real-time‟ rather than retrospectively • The ethnographer „lives in the workplace‟ and experiences it in the same way as the people involved in the work • Examines the processes and artefacts that take place in the workplace • The aim is to understand work as it really is rather than work as defined or retrospectively describedResearch methods, EngD course, 2012 Slide 16
    17. 17. Examples of ethnography • The UK en route air traffic control system • The London Underground control room • Preparing and printing government documents • Health systems – Managers in the health service – Breast cancer screening – NHS DirectResearch methods, EngD course, 2012 Slide 17
    18. 18. Fieldwork Methods • Get out of the office – Talk to people about their work – Observe practices – Visit settings of use • This can be part of an informal or formal study – How formal or informal you are depends on: • The formality of the results you wish to produce (Are you writing a study to be read by others, or just trying to get a feel for a setting?) • Your relationship to the people you research (Do they work in the same organisation as you? Are they customers?) • The setting (Some workplaces require different kinds of formalities, e.g. steel mill workers vs. hospital workers)Research methods, EngD course, 2012 Slide 18
    19. 19. Observe “ordinary practice” • Don‟t wait for things to go wrong – The routine and mundane are interesting (safe technology should be mundane and routine!) – Error is a judgement, are you in a position to judge what is an error? • What is supposed to happen and what really happens are often different – People break the rules – People have to go to extra effort to make the rules work • How is the system used in practice? • What extra things are necessary to make the system work? • Look and listen, but don‟t judge. Users know their work better than you do • Observation is sometimes referred to as “Ethnography”Research methods, EngD course, 2012 Slide 19
    20. 20. Fieldwork and systems engineering • “Traditional‟ ethnography is based on the ethnographer spending prolonged periods in the workplace – Problematical when there are deadlines for procurement and system specification • Various techniques have been developed to address this issue – Concurrent fieldwork – Evaluative fieldwork – Quick and dirty fieldworkResearch methods, EngD course, 2012 Slide 20
    21. 21. Concurrent fieldwork Ethnographic Systems study Debriefing development meetings System prototypeResearch methods, EngD course, 2012 Slide 21
    22. 22. Concurrent fieldwork • An approach that can be used in conjunction with iterative development or system prototyping – The field worker can act as a „surrogate user‟ because of their knowledge of the work and workplace – However, unlike users, they can be more objective about the system • The debriefing meetings involve the fieldworkers and system developers • The developers set questions for the fieldworker based on the prototype systemResearch methods, EngD course, 2012 Slide 22
    23. 23. Evaluative fieldwork Initial specification or design Ethnographic study Debriefing meetings Amended specification or designResearch methods, EngD course, 2012 Slide 23
    24. 24. Evaluative fieldwork • An approach which is geared to „sanity checking‟ assumptions about a system or a system specification • The field worker reads and understands the implications of the specification then makes observations of the processes to be suppprted • During the debriefing meetings, he or she discusses the issues and problems that arose during the fieldwork and the team discusses possible changes to the specification to address theseResearch methods, EngD course, 2012 Slide 24
    25. 25. Quick and dirty fieldwork Outline requirements Quick and dirty Debriefing Requirements fieldwork meetings engineering More detailed requirementsResearch methods, EngD course, 2012 Slide 25
    26. 26. Quick and dirty fieldwork • Short, focused studies to highlight the key issues of concern in the ways in which work is done • Aim of the work is to refine an initial specification • Fieldwork is concentrated in a short period (1 day-1 week), with no iterations after debriefing meetingsResearch methods, EngD course, 2012 Slide 26
    27. 27. Planning Fieldwork • Plan – But don‟t plan too much. Allow for the unexpected – How long you need is contingent on what you are looking at and how long people will put up with you – Who you observe is also contingent on how long you have, and who will have you. Don‟t always assume the „important‟ people are important (e.g. secretaries can often play critical roles) – Whether you make recordings or take notes depends on the setting, and whether and how you want to write up examples afterwards – Be ethical! This is not just about bureaucracy but about morality and common sense • Access Issues – You may need to approach and negotiate with people: managers, staff, trades-unionsResearch methods, EngD course, 2012 Slide 27
    28. 28. Writing up results • What you do after a field study depends on who the results are relevant to and how formal the work was – If it was sensitisation they may be nothing to do immediately, or you may need to do a presentation or write a report and discuss it (Discussion is very helpful) • The value of the sensitisation should be relevant later - it will influence subsequent decisions – If the work was more formal and for wider dissemination (i.e. you work in a large team, a long project and/or your work has wide relevance) you will need to write more formal reports. • First step should be to transcribe or write up what you have found • Analysis can be done in a number of ways and is usually qualitative rather than quantitative – Field-studies often lead to useful contextual information and often negative information rather than any functional or non-functional requirements.Research methods, EngD course, 2012 Slide 28
    29. 29. Informal notes about lab-books • She works over constant periods – maybe a day, maybe 10 minutes. • Has done collaborations in past, but not now. Works for a „boss‟. Seems like the usual working arrangement. She has two supervisors, one bioinformatician, one biologist. Biologist doesn‟t mind about the methods (as in paper) • The blasting etc will just be there in a paragraph. If you find errors you re-submit correct versions of sequences. • A lot of stuff looks similar as you work on it so a lab-book helps. • When is it useful? When writing up. Very useful when writing up rather than wading through files. When have collaborators for presentations or talks. When you have a student you can just show them the process. Time- you can see if someone published a suggestion before you went ahead and did it. When you want to submit a sequence. When you have competitors. A collaborator did ask the date once – I could go back. In bioinformatics I‟ve never had that.Research methods, EngD course, 2012 Slide 29
    30. 30. Key points • Case studies are a retrospective analysis technique that examine a single case in depth • Case studies rely on diverse information sources including (sometimes) fieldwork • Fieldwork is a way of finding out about human and organisational factors relevant to a system • Talk to users (stakeholders, developers, etc) and spend some time observing what they do • Shut up, look, listen. They are the experts, not you. • Discuss your findings informally with other developers, and write up your findings more formally if they are useful to a wide number of peopleResearch methods, EngD course, 2012 Slide 30