Your SlideShare is downloading. ×
Software Engineering Research: Leading a Double-Agent Life.
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Software Engineering Research: Leading a Double-Agent Life.

13,057
views

Published on

Keynote Address, APSEC 2013. …

Keynote Address, APSEC 2013.
About software engineering research and its impact ...

Published in: Technology

2 Comments
4 Likes
Statistics
Notes
No Downloads
Views
Total Views
13,057
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
2
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Useful Software Engineering Research: Leading a Double-Agent Life Lionel Briand IEEE Fellow FNR PEARL Chair Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) University of Luxembourg APSEC, Bangkok, December 3rd, 2013
  • 2. SnT$Centre$ •  •  •  •  •  •  •  SnT centre, Est. 2009: Interdisciplinary, ICT security, reliability, and trust (SnT) Luxembourg city 220 scientists and Ph.D. candidates, 20 industry partners SVV Lab: Established January 2012, www.svv.lu 25 scientists (Research scientists, associates, and PhD candidates) Industry-relevant basic and applied research on system dependability: security, safety, reliability Six partners: Cetrel, CTIE, Delphi, SES, IEE, Hitec, … 2
  • 3. Software Everywhere 3
  • 4. Software Everywhere World Software Market: •  $296 Billion in 2013 •  $360 Billion in 2016 •  Annual growth of 6% 4
  • 5. Failures Everywhere … 5
  • 6. Failures Everywhere … McKinsey, U. Oxford, 2012: •  5400 large IT projects (> 15 $M) •  17% threatened the very existence of companies •  On average: 45% over budget, 56 percent less value 6
  • 7. Software Engineering Research Funding & Relevance •  Software Engineering (SE) research should be a top priority given its importance in society. •  But that is not the case anymore (with a few exceptions). •  Symptoms –  Listed priorities by research councils, relative funding –  University hiring, few software engineering departments –  Large centers or institutes being established or closed down •  May be partly related to (perceived) lack of relevance? –  Industry participation in leading SE conferences –  Application/Experience tracks not first class citizens –  A very small percentage of research work is ever used or even assessed on real industrial software –  Impact project (ACM SIGSOFT) 7
  • 8. Basili’s and Meyer’s Take •  Many (most?) of the advances in software engineering have come out of non-university sources •  “Academic research has had its part, honorable but limited.” (Meyer) •  Large scale labs don’t get funded, like they do in other engineering and scientific disciplines (Basili, Meyer) •  One significant difference though is that we cannot entirely recreate the phenomena we study within four walls •  Question: What is our responsibility in all this? 8
  • 9. Setting the Stage
  • 10. Engineering Research •  “Engineering: The application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines, processes, and systems.” (American Heritage Dictionary) •  Engineering research: innovative engineering solutions –  –  –  –  –  Problem driven Real world requirements Scalability Human factors, where it matters Economic tradeoffs and cost-benefit analysis 10
  • 11. Pasteur’s Quadrant Consideration of Use? No Quest for Fundamental Understandin g? Yes Yes Pure Basic Research (Bohr) Use-inspired Basic Research (Pasteur) No - Pure Applied Research (Edison) Donald Stokes, Pasteur’s Quadrant, 1997 11
  • 12. Software Engineering Research •  Pure basic research in software engineering? •  As an engineering discipline, all research work should contribute to: –  Knowledge discovery but also … –  Innovative (software) engineering solutions •  Pasteur’s quadrant: Research should driven by utility and its results be put to use … •  This requires a knowledge of engineering practice •  Are the majority of projects and papers doing so? •  If not, why is that? What can we do about it? 12
  • 13. Research Example
  • 14. A Representative Example •  Parnin and Orso (ISSTA, 2011) looked at automated debugging techniques •  50 years of automated debugging research •  Only 5 papers have evaluated automated debugging techniques with actual programmers •  Focus since ~2001: dozens of papers ranking program statements according to their likelihood of containing a fault •  Experiment –  How do programmers use the ranking? –  Do they see the bugs? –  Is the ranking important? 14
  • 15. Results from Parnin and Orso’s Study •  •  Only low performers strictly followed the ranking Only one out of 10 programmers who checked a buggy statement stopped the investigation •  Automated support did not speed up debugging •  Developers wanted explanations rather than recommendations •  We cannot abstract the human away in our research •  “… we must steer research towards more promising directions that take into account the way programmers actually debug in real scenarios.” 15
  • 16. What Happened? •  How people debug and what information they need is poorly understood –  Probably varies a great deal according to context and skills •  Researchers focused on providing a solution that was a mismatch for the actual problem •  That line of research became fashionable: a lot of (cool) ideas could be easily applied and compared, without involving human participants •  Resulted in many, many papers … •  Pasteur’s quadrant: That idea was never really put to use … •  Similar example: ISSTA 2012 paper on learning program invariants (Daikon) by Matt Staats et al. •  Many other examples 16
  • 17. Reconciling Relevance and Science in Software Engineering -Implementing Pasteur’s Quadrant
  • 18. Objectives •  How to implement Pasteur’s quadrant in software engineering? •  Go through recent and successful projects with industry partners –  Simula Research Laboratory, Oslo, Norway –  SnT centre, U. of Luxembourg •  Summarize what happened, our experience •  Two projects, in the automotive domain, described in more detail •  Draw conclusions and lessons learned –  Patterns for successful research –  Challenges and possible solutions 18
  • 19. Mode of Collaboration Adapted from Gorschek et al., IEEE Software 2006 19
  • 20. Projects$Overview$(<$5$years)$ Company Domain Objective Notation Automation ABB Robot controller Func. Safety Testing UML Model analysis for coverage criteria Cisco Video conference Testing (robustness) UML profile Metaheuristic search Kongsberg Maritime Fire and gas safety control system Safety certification SysML + traceability Model slicing algorithm Kongsberg Maritime Oil&gas, safety critical drivers CPU usage analysis UML+MARTE Constraint Solver FMC Subsea system Automated configuration UML profile Constraint solver WesternGeco Marine seismic acquisition Func. Testing UML profile + MARTE Metaheuristic search DNV Marine and Energy, certification body Compliance with safety standards UML profile Constraint verification SES Satellite operator Func. Testing UML profile Metaheuristic search Delphi Automotive systems Testing (safety +performance) Matlab/Simulink Metaheuristic search Lux. Tax department Legal & financial Legal Req. QA & testing UML profile Under investigation 20
  • 21. Project Example 1: Cisco •  •  •  •  •  Context: Video conferencing systems Original scientific problem: Modeling and test case generation, oracle, coverage strategy Practical observation: Access to test network infrastructure limited (emulate network traffic, etc.). Models get too large and complex. Modified research objectives: (1) How to select an optimal subset of test cases matching the time budget, (2) Modeling cross-cutting concerns References: Hemmati et al. (2010), Ali et al. (2011) 21
  • 22. Project Example 1: Cisco •  Context: Video conferencing systems •  Original scientific problem: Modeling and model-based test case generation, oracle, coverage strategy •  Practical observation: Access to test network infrastructure limited (emulate network traffic, etc.). Models get too large and complex. •  Modified research objectives: (1) How to select an optimal subset of test cases matching the time budget, (2) Modeling cross-cutting concerns •  References: Hemmati et al. (2010), Ali et al. (2011) 22
  • 23. Project Example 2: Siemens •  •  •  •  •  Context: 3-D image segmentation algorithms for medical applications (Siemens) Original scientific problem: Define specific test strategies for segmentation algorithms Practical observations: Algorithms are validated by using highly specialized medical experts. Expensive and slow. No obvious test oracle Modified research objective: Learning oracles for image segmentation algorithms in medical applications. Machine learning. Reference: Frouchni et al. (2011) 23
  • 24. Project Example 2: Siemens •  •  •  •  •  Context: 3-D image segmentation algorithms for medical applications (Siemens) Original scientific problem: Define specific test strategies for (constantly evolving) segmentation algorithms Practical observations: Test results are validated by using highly specialized medical experts. Expensive and slow. No obvious test oracle: non-testable systems Modified research objective: Learning oracles for image segmentation algorithms in medical applications. Machine learning. Reference: Frouchni et al. (2011) 24
  • 25. Project Example 3: FMC •  Context: Subsea integrated control systems •  Original scientific problem: integration in systems of systems •  Practical observations: Each subsea installation is unique (variant), the software configuration is extremely complex (thousands of configuration parameters), > 50% configuration faults •  Modified research objective: Real-time, automated support to the configuration process using domain specific product line modeling (focus on HW-SW dependencies) and constraint solving: : instant validation, guidance, inferred values •  Behjati et al. (2012) 25
  • 26. Project Example 3: FMC •  Context: Subsea integrated control systems •  Original scientific problem: integration in systems of systems •  Practical observations: Each subsea installation is unique (variant), the software configuration is extremely complex (thousands of configuration parameters), > 50% configuration faults •  Modified research objective: Real-time, automated support to the configuration process using domain specific product line modeling (focus on HW-SW dependencies) and constraint solving: instant validation, guidance, inferred values •  Behjati et al. (2012) 26
  • 27. Project Example 4: Kongsberg Maritime •  Context: safety-critical embedded systems in the energy and maritime sectors, e.g., fire and gas monitoring, process shutdown, dynamic positioning •  Original scientific problem: Model-driven engineering for failure-mode and effect analysis •  Practical observations: Certification meetings with thirdparty certifiers. Certification is lengthy, expensive, etc. Requirements-Design decisions traceability in large complex systems a priority. •  Modified research objective: Traceability between safety requirements and system design decisions. Solution based on SysML and a simple traceability language along with model slicing. •  Reference: Sabetzadeh et al. (2011) 27
  • 28. Project Example 4: Kongsberg Maritime •  Context: safety-critical embedded systems in the energy and maritime sectors, e.g., fire and gas monitoring, process shutdown, dynamic positioning •  Original scientific problem: Model-driven engineering for failure-mode and effect analysis •  Practical observations: Certification meetings with thirdparty certifiers: Certification is lengthy, expensive, etc. Requirements-Design decisions traceability in large complex systems is an issue. •  Modified research objective: Traceability between safety requirements and system design decisions. Solution based on SysML and a simple traceability language along with model slicing. •  Reference: Sabetzadeh et al. (2011), Nejati et al. (2012) 28
  • 29. Detailed Example 1 Testing Closed Loop Controllers R. Matinnejad et al., 2013 2
  • 30. Complexity$and$amount$of$so?ware$used$on$vehicles’$$ Electronic$Control$Units$(ECUs)$grow$rapidly$$ Comfort and variety More functions Faster time-to-market Safety and reliability Greenhouse gas emission laws Less fuel consumption 30
  • 31. Model-based control system development has three major stages Model-in-the-Loop Stage Software-in-the-Loop Stage Hardware-in-the-Loop Stage Simulink Modeling Code Generation and Integration Software Running on ECU Generic Functional Model MiL Testing Software Release SiL Testing HiL Testing 31
  • 32. Major$Challenges$in$MiLGSiLGHiL$TesIng$$ •  Manual test case generation •  Complex functions at MiL, and large and integrated software/embedded systems at HiL •  Lack of precise requirements and testing Objectives •  Test selection at HiL 32
  • 33. MiL$tesIng$ Requirements Individual Functions The ultimate goal of MiL testing is to ensure that individual functions behave correctly and timely on any hardware configuration 33
  • 34. A$Taxonomy$of$AutomoIve$FuncIons$ Computation Transforming unit convertors Calculating Controlling State-Based calculating positions, State machine controllers duty cycles, etc Continuous Closed-loop controllers (PID) Different testing strategies are required for different types of functions 34
  • 35. Dynamic Continuous Controllers are present in many embedded systems including ECUs 35
  • 36. Controller Plant Model and its Requirements Desired value + Error - Controller (SUT) Plant Model System output Actual value Desired Value & Actual Value (a) Liveness Smoothness (b) =< ~= 0 (c) Responsiveness w v y >= x z Desired Value Actual Value time time time 36
  • 37. Search$Elements$ •  Search: •  Inputs: Initial and desired values, configuration parameters •  (1+1) EA •  Search Objective: •  Example requirement that we want to test: liveness !  |Desired - Actual(final)|~= 0 For each set of inputs, we evaluate the objective function over the resulting simulation graphs: •  Result: •  worst case scenarios or values to the input variables that are more likely to break the requirement at MiL level •  stress test cases based on actual hardware (HiL) 37 37
  • 38. MiL-Testing of Continuous Controllers Objective Functions + Domain Expert Exploration Controllerplant model List of Regions Local Search Test Scenarios Overview Diagram 1.0 0.9 Desired Value Actual Value Initial Desired 0.8 0.7 0.6 0.5 0.4 0.3 Final Desired 0.2 0.1 0.0 0 1 time 2 38
  • 39. Conclusions$ •  Addressed a testing problem that has been largely ignored i.e., 31s. much horizontal axis of the diagrams in Figure 8 shows the number of •  We found Hence, theworse scenarios during MiL testing than our iterations instead of the computation In start both random search partner had found so far time.the addition, wefrom the exploration step. and (1+1) EA from the same initial point, i.e., worst case Overall in all the •  Much worse thanregions, (1+1) EA eventually reachesmore deterministic than ranrandom search EA is its plateau at a value higher than the random search plateau value. Further, (1+1) has smaller of testing is •  Theydom, i.e., the distribution of (1+1) EAthea HiL8).variance than that(e.g., Figure 8(d)), much are running them at (see Figure level, where random search, especially when reaching the plateau In some regions more expensive: MiL results ->faster than (1+1) EA, while in HiL other however, random reaches its plateau slightly test selection for some regions (e.g. Figure 8(a)), (1+1) EA is faster. We will discuss the relationship between •  But further landscape and the performance of (1+1) EA in RQ3. the region research is needed: RQ3. We drew the landscape for the 11 regions in our experiment. For example, Fig–  To 9 shows with the for two selected regions in Figures parameters ure deal the landscape many configuration 7(a) and 7(b). Specifically, Figure 9(a) shows the landscape for the region in Figure 7(b) where (1+1) EA is faster –  To dynamically adjust search algorithms in different than random, and Figure 9(b) shows the landscape for the region in Figure 7(a) where (1+1) EA is slower than different subregions withrandom search. lanscapes (a) (b) 0.40 0.20 0.39 0.19 0.38 0.18 0.37 0.17 0.36 0.16 0.35 0.15 0.34 0.14 0.33 0.13 0.32 0.12 0.31 0.11 0.30 0.10 0.70 0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.80 0.90 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1.00 Fig. 9. Diagrams representing the landscape for two representative HeatMap regions: (a) Landscape for the region in Figure 7(b). (b) Landscape for the region in Figure 7(a). 39
  • 40. Detailed Example 2 Minimizing CPU Time Shortage Risks in Integrated Embedded Software S. Nejati et al., 2013
  • 41. Today’s$cars$rely$on$integrated$systems$ •  Modular and independent development •  Many opportunities for division of labor and outsourcing •  Need for reliable and effective integration processes 41
  • 42. IntegraIon$process$in$the$automoIve$domain$ AUTOSAR Models sw runnables Glue AUTOSAR Models sw runnables 42
  • 43. CPU$Time$Shortage$in$Integrated$Embedded$$ So?ware$ •  Static cyclic scheduling: predictable, analyzable •  Challenge –  Many OS tasks and their many runnables run within a limited available CPU time •  The execution time of the runnables may exceed their time slot •  Our goal –  Reducing the maximum CPU time used per time slot to be able to •  Minimize the hardware cost •  Reduce the probability of overloading the CPU in practice •  Enable addition of new functions incrementally ✗ (a) 5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms ✔ (b) 5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms Fig. 4. Two possible CPU time usage simulations for an OS task with a 5ms cycle: (a) Usage with bursts, and (b) Desirable usage. to deadlin run passe by OS ta divisible 43 offset val 43 synchron
  • 44. Using&runnable&offsets&(delay&3mes)& Inserting runnables’ offsets 5ms 5ms 10ms 10ms 15ms 15ms 20ms 20ms 25ms 25ms 30ms 30ms 35ms 35ms 40ms 40ms ✗ ✔ Offsets have to be chosen such that the maximum CPU usage per time slot is minimized, and further, the runnables respect their period the runnables respect their time slot the runnables satisfy their synchronization constraints 44 44
  • 45. Meta$heurisIc$search$$algorithms$ -  The objective function is the max CPU usage of a 2s-simulation of runnables -  The search modifies one offset at a time, and updates other offsets only if timing constraints are violated -  Single-state search algorithms for discrete spaces (HC, Tabu) Simulation for the runnables case Case Study: an automotive software system with 430lowest maxin our usagestudy and HC runnables found by corresponding to the CPU 5.34 ms 2.13 ms Running the system without offsets Optimized offset assignment 45 45
  • 46. Comparing$different$search$algorithms$$ (ms) Best CPU usage (s) Time to find Best CPU usage 46 46
  • 47. Conclusions$ -  Though schedulability analysis is a well developed field, the problem we addressed was never defined in those terms (context) -  Search algorithms to compute offset values that reduce the max CPU time needed -  Positive evaluation results: Quick and significant differences -  Huge search space with constraints -  Current: Accounting for task time coupling constraints with multiobjective search " trade-off between relaxing coupling constraints and maximum CPU time 47 47
  • 48. What Have I Learned?
  • 49. Successful Research Patterns •  •  •  •  •  Successful: Innovative and high impact Inductive research: Working from specific observations in real settings to broader generalizations and theories –  Software development is highly diverse –  Problems and solutions are diverse too –  Context factors matter a great deal –  Generalization: Field studies and replications, analyze commonalities Scalability and practicality considerations must be part of the initial research problem definition Researching by doing: Hands-on research. Apply what exists in well defined, realistic context, with clear objectives. The observed limitations become the research objectives. Put new technology to actual use. Interdisciplinary: CS, Mathematics, Engineering, or non-technical domains 49
  • 50. So What? •  Making a conscious effort to understand the problem first –  A careful investigation often leads to surprises and a very different understanding of the issues at stake –  Precisely identify the requirements for an applicable solution –  More papers focused on understanding the problems –  Making experience/application tracks first class citizens in SE conferences 50
  • 51. So What? - cont’d •  Better relationships between academia and industry –  Different models •  Fraunhofer centres, Germany •  Research-based innovation centers in Norway •  SnT centre in Luxembourg •  Targeted grant programs •  Joint industry-academia labs (e.g., NASA SEL Lab) –  Mutually beneficial setting where software development is an object of study –  Exposing PhD students to industry practice, management and leadership: Ethical considerations (“Fix the PhD”, Nature, 2011) –  Incentives for SE academics 51
  • 52. So What? - Cont’d •  Work on end-to-end solutions: Pieces of solutions are interdependent. Necessary for impact, e.g., requirements and testing. •  Beyond professors and students –  Labs with interdisciplinary teams of professional scientists and engineers within or collaborating with universities –  Used to be the case with corporate research labs: Bell Labs, Xerox PARC, HP labs, NASA SEL, etc. –  Now: Fraunhofer (Germany), Simula (Norway), Microsoft Research (US), SEI (US), SnT (Luxembourg), FBK (Italy) –  Corporate labs versus publicly supported ones? –  Key point: The level of basic funding must allow high risk and rigorous research, performed by professional scientists, focused on impact in society (Use-inspired basic research) 52
  • 53. The “Classical” Model of Research and Innovation Basic Research Applied Research Innovation and Development
  • 54. An Effective, Collaborative Model of Research and Innovation Innovation & Development Basic Research Applied Research •  Basic and applied research take place in a rich context •  Basic Research is also driven by problems raised by applied research •  Main motivation for SnT’s partnership program B. Schneiderman, The Atlantic, Toward an Ecological Model of Research and Development, 2013
  • 55. Challenges
  • 56. Academic Challenges •  •  •  •  •  •  Our CS legacy … emancipating ourselves as an engineering discipline –  Electrical engineering: Subfield of Physics until late 19th century –  Software engineering departments? How cool is it? SE research is more driven by “fashion” than needs, a quest for silver bullets –  We can only blame ourselves Counting papers and how rankings do not help –  We are pressuring ourselves into irrelevance Taking academic tenure and promotion seriously –  What about rewarding impact? One’s research must cover a broader ground and be somewhat opportunistic – this pushes us out of our comfort zone Resources to support industry collaborations –  Large lab infrastructure, research engineers, time 56
  • 57. Industrial Challenges •  •  •  •  •  Distinguish the manifestations of a problem from its causes is not easy in practice Short term versus longer term goals (next quarter’s forecast is sometimes the priority) Industrial research groups are often disconnected from their own business units and external researchers may be perceived as competitors Company’s intellectual property regulations may conflict with those of the research institution Complexity of industrial systems and technology –  Cannot be transplanted in artificial settings for research - Need studies in real settings –  Substantial domain knowledge is required 57
  • 58. A Double-Agent Life Scientist, inquisitive but discrete Warning: No research here A new idea (as initially perceived by our partners) Practitioner (anonymous) 58
  • 59. Conclusions •  Software engineering is obviously important in all aspects of society, but academic software engineering research is not always perceived the same way •  The academic community, at various levels and in particular its leadership, is partly responsible for this •  How we take up the challenge of increasing our impact will determine the future of the profession •  There is some limited progress, but far too slow •  There are solutions, but no silver bullet •  We all have a role to play in this, as deans, department chairs, professors, scientists, reviewers, conference organizers, journal editors, etc. We can all be double-agents … 59
  • 60. SOUNDING BOARD Editor: Philippe Kruchten University of British Columbia pbk@ece.ubc.ca Embracing the Engineering Side of Software Engineering Lionel Briand I HAVE NOW been a professional researcher in software engineering for roughly 20 years. Throughout that time, I’ve worked at universities and in research institutes and collaborated on research projects with 30-odd private companies and public institutions. Over the years, I have increasingly questioned and reflected on the impact and usefulness of my research work and, as a result, made it a priority to combine my research with a genuine involvement in actual engineering problems. This short piece aims to reflect on my experiences in performing industry-relevant software engineering research across several countries and institutions. Not So Hot Anymore I suppose a logical start for this article is to assess, albeit concisely, the current state of software engineering research. As software engineering is widely taught in many universities, due in large part to a strong demand for software engineers in industry, the number of software engineering academics is substantial. The Journal of Systems and Software ranks researchers every year, usually accounting for roughly 4,000 individuals actively publishing in major journals. When I started my career, software engineering was defi nitely a hot topic in academia: funding was plentiful, and universities and research institutes were hiring in record numbers. This clearly isn’t the case anymore. Public funding for software engineering research has at best stagnated, and in many countries, declined significantly. 96 I E E E S O F T W A R E | P U B L I S H E D B Y T H E I E E E C O M P U T E R S O C I E T Y Hiring for research positions is limited and falls far below the number of software engineering graduates seeking research careers. Industry attendance at scientific software engineering conferences is roughly 10 percent, including the scientists from corporate research centers. Adding insult to injury, in many academic and industry circles, software engineering research isn’t even considered to be a real scientific discipline. I’ll spare you the numerous unpleasant comments about the credibility and scientific underpinning of software engineering research that I’ve heard over the years. This situation isn’t due to the subject matter’s lack of relevance. Software systems are pervasive in all industry sectors and have become increasingly complex and critical. The software engineering profession repeatedly tops job-ranking surveys. In many cases, most of a product’s innovation lies in its software components—for an example, think of the automotive industry. In all my recent industry collaborations, I’ve observed that all the issues and challenges traditionally faced in software development are becoming more acute. So how can we explain the paradox of being both highly relevant and increasingly underfunded and discredited? Looking for Some Answers Like other disciplines before us, because we’re a young and still-maturing engineering field, we lack the credibility of more continued on p. 93 0 74 0 -74 5 9 / 12 / $ 3 1. 0 0 © 2 0 12 I E E E 60
  • 61. Empirical Software Engineering •  •  •  •  Springer, 6 issues a year Both research papers and industry experience reports High impact factor among SE research journals “Applied software engineering research with a significant empirical component” 61
  • 62. Personal References •  •  •  •  •  •  •  •  •  Hemmati, Briand, Arcuri, Ali “An Enhanced Test Case Selection Approach for Model-Based Testing: An Industrial Case Study”, ACM FSE, 2010 Frouchni, Briand, Labiche, Grady, and Subramanyan, “Automating Image Segmentation Verification and Validation by Learning Test Oracles”, Information and Software Technology (Elsevier), 2011. Sabetzadeh, Nejati, Briand, Evensen Mills “Using SysML for Modeling of Safety-Critical Software– Hardware Interfaces: Guidelines and Industry Experience”, HASE, 2011 Sabetzadeh et al., “Combining Goal Models, Expert Elicitation, and Probabilistic Simulation for Qualification of New Technology”, HASE, 2011 Ali, Briand, Hemmati, “Modeling Robustness Behavior Using Aspect-Oriented Modeling to Support Robustness Testing of Industrial Systems”, Journal of Software and Systems Modeling (Springer), 2011. Behjati, Nejati, Yue, Gotlieb, Briand, “Model-based Automated and Guided Configuration of Embedded Software Systems”, ECMFA, 2012. Behjati, Yue, Briand, Selic. SimPL, “A Product-Line Modeling Methodology for Families of Integrated Control Systems, Information and Software Technology”, Information and Software Technology (Elsevier), 2012. Nejati et al., A SysML-Based Approach to Traceability Management and Design Slicing in Support of Safety Certification: Framework, Tool Support, and Case Studies, Information and Software Technology (Elsevier), 2012 Iqbal, Arcuri, Briand, “Empirical Investigation of Search Algorithms for Environment Model-Based Testing of Real-Time Embedded Software”, ACM ISSTA, 2012 62
  • 63. Personal References II •  •  •  •  •  •  •  Nejati, Di Alesio, Sabetzadeh, Briand, “Modeling and Analysis of CPU Usage in Safety-Critical Embedded Systems to Support Stress Testing”, ACM/IEEE MODELS 2012 Nejati, Sabetzadeh, Falessi, Briand, Coq, “A SysML-based approach to traceability management and design slicing in support of safety certification: Framework, tool support, and case studies”, Information & Software Technology, (Elsevier), 2012 Hemmati, Arcuri, L. Briand: Achieving scalable model-based testing through test case diversity. ACM Trans. Softw. Eng. Methodology, 2013. Panesar-Walawege, Sabetzadeh, Briand: Supporting the verification of compliance to safety standards via model-driven engineering: Approach, tool-support and empirical validation. Information & Software Technology (Elsevier), 2013 S. Nejati et al., “Minimizing CPU Time Shortage Risks in Integrated Embedded Software”, 28th IEEE/ACM International Conference on Automated Software Engineering, 2013 Sabetzadeh, Falessi, Briand, Di Alesio: A goal-based approach for qualification of new technologies: Foundations, tool support, and industrial validation. Reliability Eng. & System Safety, 2013 Briand et al., “Traceability and SysML Design Slices to Support Safety Inspections: A Controlled Experiment”, forthcoming in ACM Transactions on Software Engineering and Methodology, 2013 63
  • 64. Other References •  •  •  •  Bertrand Meyer’s blog: http://bertrandmeyer.com/2010/04/25/the-other-impediment-tosoftware-engineering-research/ Basili, “Learning Through Application: The maturing of the QIP in the SEL”, Making Software; What really works and why we believe it, Edited by Andy Oram and Greg Wilson, O’Reilly Publishers, 2011, pp. 65-78. T. Gorschek, P. Garre, S. Larsson, C. Wohlin. A Model for Technology Transfer in Practice. IEEE Software 23(6), 2006. Parnin and Orso, “Are Automated Debugging Techniques Actually Helping Programmers?”, ISSTA, 2011 64