Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Privacy as a Safety Critical Concept


Published on

Privacy’s place in software engineering development is
tentatively made at best. Yet the effects of information leakage,
an improperly made audit and so on are known to be expensive in
monetary, legal and other terms. We present here learnings made
in adopting ideas from safety-critical systems and disciplines
where safety is a first-class concept, such as aviation and
medicine. We outline our rationale and describe our experiences
- both good and bad - and the necessary concepts that must be
understood for successful application of such ideas.

Published in: Technology
  • Be the first to comment

Privacy as a Safety Critical Concept

  1. 1. Privacy as a Safety Critical Concept Ian Oliver Nokia Networks Espoo, Finland Abstract—Privacy’s place in software engineering development is tentatively made at best. Yet the effects of information leakage, an improperly made audit and so on are known to be expensive in monetary, legal and other terms. We present here learnings made in adopting ideas from safety-critical systems and disciplines where safety is a first-class concept, such as aviation and medicine. We outline our rationale and describe our experiences - both good and bad - and the necessary concepts that must be understood for successful application of such ideas. I. INTRODUCTION This paper presents a number of themes in a discussion or essay format that support or build a basis of the theory that privacy should or must be treated as a safety critical concept in systems’ design. The techniques that need to be used and developed are effectively the very same tools and techniques that are used in the construction of safety-critical and fault tolerant systems. As privacy is fast becoming the number one issue (if it isn’t already) with regards to consumers’ data, the amount of effort in consumer advocacy and legal aspects is outstripping the ability of the technical community to keep up with the required architectures, platforms, design, implementation and techniques for achieving this. Indeed there is some kind of arms race going on here and I’m not sure it really is in the benefit of the consumer [1]. This paper presents a number of areas in an informal manner that must be considered when developing with privacy: from the engineering aspect and from the management aspect. II. INHERENT PRIVACY The Do Not Track technical community has come under criticism for not delivering a complete solution. This is not really 100% the fault of the W3C or the people developing these standards but rather the following lackings of a much wider community, including legal and standardisation: • understanding information systems (even in the theoreti- cal sense) and • a lack of applicable and relevant tools and techniques for the software engineering community who at the end of the day are the ones who will end up writing the code that implements the legal directives in some for or other. The term Inherent Safety (see [2]) is defined: Inherent safety is a concept particularly used in the chemical and process industries. An inherently safe process has a low level of danger even if things go wrong. It is used in contrast to safe systems where a high degree of hazard is controlled by protective systems. It should not be confused with intrinsic safety which is a particular technology for electrical systems in potentially flammable atmospheres. As perfect safety cannot be achieved, common prac- tice is to talk about inherently safer design. An inherently safer design is one that avoids hazards instead of controlling them, particularly by reducing the amount of hazardous material and the number of hazardous operations in the plant. Using this as a starting place we can propose a rewriting this principle in the privacy contenxt as shown below taking the extensions as proposed in [3] into consideration: • Minimize: reducing the amount of information/data present at any one time either in local storage, remote storage cache or network transfer • Substitute: replace one element of data with another of less privacy risk, eg: abstract GPS coordinates to city areas • Moderate: reduce the strength of a process to transform or analyse data, eg: reduce the amount of crossreferencing over a number of sets of data • Simplify: reduce the complexity of processes to control data, eg: single opt-out, one-time consent, simple ques- tions (and not asking the user what resources an app should have access too..) • Error Tolerance: design the system/software and pro- cesses to deal with worst cases, eg: misbehaving applica- tions sending too much data are blocked in some sense • Limit Effects: designing the system such that the effects of any leak of data is minimised, eg: properly anonymised
  2. 2. or pseduo-anonymised data sets, secure transport layers, encryption etc While admittedly not fully worked out we feel that this is more communicable and understandable to the software engineers than, say, Privacy by Design, which while lays out a good set of principles, is too high-level and abstract to map to the engineers and their day-to-day work. In some way the problem with the principles of Privacy by Design is that they can be interpreted much like the principles of the Agile Manifesto [4] which leads to some esoteric, bizarre and extreme ideas of what agile processes are - just take a look at some of the later writings of Ward Cunningham or Scott Ambler on the subject. One aspect of the inherent safety idea that particularly appeals is that it is more grounded in engineering and practical development rather than being a set of principles. Indeed much of the grounding for this work comes from a very practical need and development through sound engineering practice espoused by Trevor Klenz. His quote ”what you don’t have, can’t leak” applies equally to information as it does hazardous substances; Klentz’s book [5] maybe should become required reading along with Solove [6] and Nissenbaum [7]. As a further example, the HAZOP (Hazard and operability study) method(s) are purely formal methods in the true sense of the word in constructing a small, formally defined vocab- ulary and modelling standards - compare with process flow diagram in the chemical and process engineering with the data- flow diagram in software engineering for example. We finish with a reference to HACCP (Hazard analysis and critical control points) which itself has a number of principles (seven seems to be a common number), but here’s I’d like to concentrate for the moment on just two: Principle 2: Identify critical control points. A critical control point (CCP) is a point, step, or procedure in a food manufacturing process at which control can be applied and, as a result, a food safety hazard can be prevented, eliminated, or reduced to an acceptable level. Where do we control the flow of information? Is it near the user or far from? Is it honour based and so on? The further from the source of the information, the greater the chance of leakage (both in information and chemical systems). Principle 4: Establish critical control point moni- toring requirements. Monitoring activities are nec- essary to ensure that the process is under control at each critical control point. In the United States, the FSIS is requiring that each monitoring procedure and its frequency be listed in the HACCP plan. This is something that we as a privacy discipline are very bad with - do we ever monitor what information we keep in databases? Even the best intentions and best designs might still leak something, and this is especially true when working with large sets of information that can be cross referenced and fingerprinted. Maybe we should consider some kinds of information to be analogous to bacteria or viruses in a food handling situation? We therefore assert that in order to engineer information system correctly for privacy we must consider those systems to be safety-critical and treat them accordingly. III. LEARNINGS FROM SURGERY Surgical operating theatres are split into two parts: • the sterile field • the non-sterile surroundings Any non-sterile item entering the sterile field renders it non- sterile; and stringent efforts and protocols [8], [9] are made to ensure that this does not happen. The protocols above extend via simply analogy to information handling and information privacy: • Any body of data which is containing certain amounts and kinds of sensitive data we can consider to be non- sterile - assume for a moment that certain bits and bytes are infectious. • Everyone working with information is required to remain sterile and uncontaminated. • Information which is truly anonymous is sterile • Mixing two sets of information produces a single set of new information which is as at least as unclean as the dirtiest set of data mixed, and usually more so! • The higher the security classification the dirtier the infor- mation We can extend this latter point to specific information types, eg: location, personal data, or certain kinds of usages and purposes, eg: data for advertising or secondary data and so on. Extending our analogy further we can protect the sterile field in two ways:
  3. 3. • ensuring that everyone in contact with the sterile field is sterile • ensuring that the equipment entering the sterile field is sterile Basic data handling techniques are essential: • If two sets of data are to be mixed then ensure that the mixing occurs not in-situ but by generating a third data set kept separate from the two input sets • Data can be made more sterile by removing information content. But, be warned that certain kinds of obfuscation are not effective, eg: hashing or encryption of fields might just hide the information content of that field but not the information content of the whole data set • Keep sterile and non-sterile data-sets apart, physically if possible • Ensure that sterile and non-sterile data-sets have differing access permissions. Ideally different sets of people with access • Clean up after yourself: secure data deletion, overwriting of memory, cache purges etc. From a personnel point, in surgery precautions are made through restricting the persons inside the sterile field and even outside of this, basic precautions are taken in terms of protective clothing, handling, access to sterile items etc. While surgical attire might be overkill for office environments, the analogy here is that personnel with access to data have received the correct training and are aware of what data they can and can not use for various purposes. In a surgical environment, everything entering and leaving the sterile field is checked and recorded. In an information systems environment this means logging of access so that when a breach of the sterile field occurs the route of the pathogen and its nature can be effectively tracked and cleaned. IV. CHECKLISTS A colleague brought to my attention a publication from the PbD community on the subject of privacy engineering [10]. Overall the paper provides a good introduction into one aspect of what privacy engineering could be. But privacy engineering needs a huge amount of work, from all angles including the deeper mathematical basis as well as basic, software engineering, process, management etc. The current definition given above is: ”...privacy engineering is the discipline of under- standing how to include privacy as a non-functional requirement in systems engineering. While privacy may also appear as a functional requirement of a given system (such as the TOR anonymity system), for most systems, privacy is ancillary to the primary purposeof the system.” However non-functional requirements become very awkward functional requirements very quickly as the development pro- ceeds. In saying that this part is of particular interest and is regarding the use of checklists [11]: ”The checklist approach to privacy protection has been debated. Checklists have become important safety elements in airplane and medical procedures and are quite common in security auditing. However, their utility for privacy remains questionable. It might be possible to design privacy checklists for frequent and standardized use cases, but the breadth of potential projects makes a standard checklist for everything an unlikely tool.” Partly the paragraph is correct, the utility of checklists in privacy needs to be established, but, given what a checklist is and what is should be designed to achieve is independent of the area. Indeed the next statement that it might be possible to design privacy checklists for frequent and standardised use cases is exactly the cases where checklists do come into their own. Often repeated, critical phases of design - patterns if you like - are the areas where mistakes are frequently made; things are forgotten etc [12]. There are NO standard checklists - at least not for the expected usages hinted in the paper. If we compare with the WHO surgical checklists which work in similar, very open, high volatility environments there is a capitalised statement on the bottom of the checklist: THIS CHECKLIST IS NOT INTENDED TO BE COMPREHENSIVE. ADDITIONS AND MODIFI- CATIONS TO FIT LOCAL PRACTICE ARE EN- COURAGED. Ignore at your peril. Indeed the checklist in privacy WILL NOT give you a stan- dardised approach NOR will it give you the tool for checking your system [13]. It WILL give you a list of things that you should remember to complete at particular relevant points [14]. A checklist should be independent of your processes and partially independent of what ever tooling and techniques are being applied to the problem at hand. Furthermore, if the various items on a checklist are not completed it is not an indication that the process can not continue but rather a warning that important facts might not have been established
  4. 4. [15]. For example, on aircraft take-off there is a checklist item for the setting of flaps. This item may not necessarily specify the specific flap settings as this might vary by many conditions and the item can be ignored (and the flaps not set) and take- off can proceed - though this might be a particularly bad idea as in the Spanair Flight 5022 case. Interestingly in the paper checkilsts seem to be positioned against privacy impact assessments. One could better imagine that in the privacy checklist an entry “PIAs performed” is included, possibly with further qualifications as necessary [16]. It might be considered that such an entry is superfluous - who would forget to do the PIAs? For example pilots would never forget to lower flaps prior to take-off or a surgeon forget to check that the he/she is set up for the correct procedure to be performed, however experience tells us otherwise. Maybe the term checklist isn’t the best and this is where some confusion arises? At Great Ormond Street Hospital the term “Aide-Memoire” is used to reflect the function of the checklist and avoid the confusion with the ”tick-box mentality” and confusion with process. Privacy is a very wide area without well established and standardised use cases or at least not at the level of detail that say aviation is as is pointed out in the paper we lack. Indeed it is this lack of standardisation that makes the integration of checklists or aide-mmoires much more important. Some of our experiences with checklists in this area can be found here [17]. Not that particular study in Ontario, but another in Ontario regarding the Surgical Checklist and its “ineffectiveness”, which was rebutted by many including Atul Gwande which ended with this quote: Perhaps, however, this study will prompt greater attention to a fundamentally important question for health care reform broadly: how you implement an even simple change in systems that reduces errors and mortality like a checklist. For there is one thing we know for sure: if you dont use it, it doesnt work. Relating this back to my experiences in deploying and using checklists for privacy is that THEY ARE NOT A TOOL FOR IMPROVING PRIVACY DIRECTLY but a TOOL for organising your workflows, your actions and ensuring that all members of a team are actively cross-checking each other; and even then this is just a small part of the overall effect. Let us for a moment rewrite Gwande’s statement a little: Perhaps, however, this study will prompt greater attention to a fundamentally important question for privacy engineering: how you implement an even simple change in systems that reduces errors and non compliance like a checklist. For there is one thing we know for sure: if you dont use it, it doesnt work. In the paper [10] (emphasis mine): The checklist approach to privacy protection has been debated.[24] Checklists have become impor- tant safety elements in airplane and medical proce- dures and are quite common in security auditing. However, their utility for privacy remains question- able. It might be possible to design privacy check- lists for frequent and standardized use cases, but the breadth of potential projects makes a standard checklist for everything an unlikely tool. [24] Ian Oliver, Safety Systems Defining Moments Available at safety-defining-moments.html Indeed the two paragraphs on checklists and privacy impact assessments fails to properly understand the former and com- pares it against the latter which is a different kind of tool altogether. In fact, a privacy impact assessment should be done and this would be ensured or reminded by having it included as a point on a checklist for privacy. Indeed no mention was made, nor has been made of any “standardised checklist”. In fact there is a capitalised statement on the bottom of the checklist: THIS CHECKLIST IS NOT INTENDED TO BE COMPREHENSIVE. ADDITIONS AND MODIFI- CATIONS TO FIT LOCAL PRACTICE ARE EN- COURAGED. The point here in both cases: surgical and privacy engineering is that the checklist needs to be accompanied by a procedural and “societal” change for it be successful. One only needs to read about Pronovost’s work with a simple checklist and the changes surrounding that to understand how checklists work in practice - that and the other experiences presented in Gawande’s excellent book on the subject: The Checklist Man- ifesto. Our experiences can be read about in the presentation Flying Planes, Surgery and Privacy.
  5. 5. V. UNDERSTANDING ACCIDENTS The cause of 99.999...percent of accidents is easy to ascertain, quite simply it is the pilot’s/driver’s fault. In the cases of two recent aviation and railway accidents (Asiana 214 and the Galicia’s train crash) these were primarily caused by the pilot and driver respectively. Finding the causes of an accident don’t actually involve finding who is to blame but rather the whole context of what led up to the point where an accident was inevitable. And then even after that exploring what actually occurred during and after the accident. The reasoning here is that if we concentrate on who to blame then we will miss all the circumstances that allowed that accident to occur in the first place. For example., in the case of the Galician train crash it has been ascertained that the driver was speeding, on the phone and had a history of breaking the rules. However this misses the more subtle questions of why did the train derail, why could the driver break the speed limit, what was the reasoning of the phone call, why did the carriages fail catastrophically after the derailment, did the safety systems work sufficiently, what state was the signalling in, did the driver override systems, was the driver sufficiently trained etc etc etc. In other words, the whole context of the systems and organi- sation needs to be taken into account before appointing final blame; and even then very, very rarely is it single point of failure: the Swiss Cheese Model. If we apply this model to software engineering and specifically accidents such as hacking and data breaches we become very aware of how many holes in our computing Swiss cheese we have. Take a simple data breach where “hackers” have accessed a database on some server via an SQL injection via some web pages. If we apply our earlier thinking, it is obviously the fault of the ’stupid’ system administrators who can’t secure a system and the ’stupid’ software engineers who can’t write good code. Hindsight is great here isn’t it? To answer ‘who to blame?’ or better still ’why did things go wrong and how can we prevent this in the future?’ we need to put ourselves, as other accident investigators do, in the position of those software engineers, system administrators, architects, hackers and managers AT THE POINT IN TIME WHERE THEY ACTED, and NEVER in hindsight. Why did the trained, intelligent software engineer write code susceptible to SQLi in the first place? Maybe they were under time pressure, no proper, formal design documentation, coding standards, never trained to spot those kinds of errors? Actually just stating that we have a failure of discipline is already pointing to wholesale failures across our whole organisation rather than just one individual. Even if in the above case this was malice by the programmer, then why didn’t the testing pick this up? Why was the code released without testing or checking? Now we have a failure elsewhere too. Were there no procedures for this? Was the code signed-off by management with the risk in place? Now the net widens further across the organisation, and so on... In aviation there is a term to explain most accidents: loss of situational awareness, which when explored invariably ends up with a multitude of ‘wrong’ choices being made over a longer period of time rather than just at those few critical minutes or hours in the cockpit. I’m of the opinion that in software engineering that we almost always operate in a mode where we have no or little situational awareness. Our systems are complex, we lack formal models of our systems that clearly and concisely explain how the system works; indeed one of the maxims used by some practitioners of agile methods actively eschews the use of formality and modelling tools. Coupled with tight deadlines, a code-is-king mentality and rapidly and inconsistently changing requirements we have a fantastic recipe for disaster. Bringing this back to an aviation analogy again, consider Turkish Airlines Flight 1951 which crashed as Schiphol in 2009. It was initially easy to blame the pilots for allowng the plane to stall on final approach, but the whole accident investigation revealed deficiencies in the training, the approach procedures of Schiphol, a non-fault tolerant autothrottle and radar combination, a massively high workload situation for the pilots and ultimately a fault which manifested itself in precisely the behaviour that the pilots were requiring and expecting on their approach, that is the aircraft was losing speed. As an exercise, how does the above accident map to what we experience every day in software engineering? Given high workloads, changing requirements, inconsistent planning and deadlines to get something (anything!) out that sort of works and we start getting answers to why intelligent administrators and programmers make mistakes.
  6. 6. VI. SUMMARY Privacy in the context of the discpline of privacy engineering is developing slowly. Currently privacy does not feature as a first-class element in any information system design at the architectural or design levels. Privacy is considered a non- functional aspect which still remains at the legal level; and if implemented forms part of a very fragile compliance to the legal requirements. In order to progress we can not rely upon ‘luck’ or the ambudance of very skilled, privacy aware engineers - of which the are regrettably very few - or even security aware engineers who effectively build enough protections such that privacy professionals don’t have to get intimately involved in protecting a system. Reliance on ‘privacy compliance’ is little more than hoping a legal text will not be invalidated by a missing or superflous line of code. The thesis here is that for privacy to emerge as an engineering discipline there has to be a privacy-wide change in culture without a reinvention of the wheel, in that, techniques for creating safe systems already exist and these must brought into common use within privacy. ACKNOWLEDGMENTS The authors would like to thank the many colleagues at Nokia and Here for challenging and helping improve these ideas and concepts - not to mention taking onboard the ideas and ‘societal’ or cultural changes necessary. REFERENCES [1] I. Oliver, “An advertiser’s paradise: An adventure in a dystopian post- ‘do not track world?’,” in W3C Workshop: Do Not Track and Beyond, November 2012. [2] A.-M. Heikkil, “Inherent safety in process plant design. an index-based approach,” Ph.D. dissertation, Helsinki University of Technology (TKK), May 1999, vTT Publications 384. ISBN 951-38-5371-3. [3] F. I. Khan and P. R. Amoyette, “How to make inherent safety practice a reality,” Canadian Journal of Chemical Engineering, vol. 81, pp. 2–16, 2003. [4] K. Beck, M. Beedle, A. van Bennekum, A. Cockburn, W. Cunningham, M. Fowler, J. Grenning, J. Highsmith, A. Hunt, R. Jeffries, J. Kern, B. Marick, R. C. Martin, S. Mellor, K. Schwaber, J. Sutherland, and D. Thomas, “Manifesto for agile software development,” 2001. [5] T. A. Kletz, Plant Design for Safety A User-Friendly Approach. Hemi- sphere, New York, 1991. [6] D. J. Solove, “Conceptualizing Privacy,” California Law Review, vol. 90, no. 4, pp. 1087–1155, 2002. [7] H. Nissenbaum, “A contextual approach to privacy online,” Daedalus, vol. 140, no. 4, pp. 32–48, Fall 2011. [8] J. Roark, “Guidelines for maintaining the sterile field,” Infection Control Today, August 2003. [9] unk., “Best practices in maintaining the sterile,” Infection Control Today, November 2006. [10] A. Cavoukian, J. Cronk, and S. Shapiro, “Privacy engineering: Proac- tively embedding privacy, by design.” [11] A. Gawande, The Checklist Manifesto. Profile Books, 2011. [12] P. Pronovost, D. Needham, S. Berenholtz, D. Sinopoli, H. Chu, S. Cos- grove, B. Sexton, R. Hyzy, R. Welsh, G. Roth, J. Bander, J. Kepros, and C. Goeschel, “An Intervention to Decrease Catheter-Related Blood- stream Infections in the ICU,” New England Journal of Medicine, vol. 355, no. 26, pp. 2725–2732, 2006. [13] Implementation manual WHO Surgical Safety Checklist, World Health Organization, 2008, wHO/IER/PSP/2008.05. [14] T. Kern, Flight Discipline. McGraw-Hill Education, 1998. [15] C. L. Bosk, M. Dixon-Woods, C. A. Goeschel, and P. J. Pronovost, “Reality check for checklists,” The Lancet, vol. 374, no. 9688, pp. 444– 445, August 2009. [16] I. Oliver, Privacy Engineering: A Dataflow and Ontoligcal Approach. CreateSpace Independent Publishing, 2014. [17] I. Oliver and T. Kulmala, “Flying planes, surgery and pri- vacy,” privacy-external-version, April 2013.