The document discusses testing of closed-loop controllers in automotive systems. It notes the increasing complexity of automotive software and challenges in model-in-the-loop (MIL), software-in-the-loop (SIL), and hardware-in-the-loop (HIL) testing. It presents an approach to generate test cases for continuous controllers at the MIL level by representing requirements as objective functions and using a search-based approach to find worst-case scenarios. Experimental results found significantly worse scenarios than their industry partner, allowing them to generate better stress tests for HIL. The approach addresses a problem largely ignored in MIL testing of continuous controllers.

1. Useful Software Engineering Research:
Leading a Double-Agent Life
Lionel Briand
IEEE Fellow
FNR PEARL Chair
Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT)
University of Luxembourg
APSEC, Bangkok, December 3rd, 2013

2. SnT$Centre$
•
•
•
•
•
•
•
SnT centre, Est. 2009: Interdisciplinary, ICT
security, reliability, and trust (SnT)
Luxembourg city
220 scientists and Ph.D. candidates, 20
industry partners
SVV Lab: Established January 2012,
www.svv.lu
25 scientists (Research scientists,
associates, and PhD candidates)
Industry-relevant basic and applied research
on system dependability: security, safety,
reliability
Six partners: Cetrel, CTIE, Delphi, SES, IEE,
Hitec, …
2

3. Software Everywhere
3

4. Software Everywhere
World Software Market:
• $296 Billion in 2013
• $360 Billion in 2016
• Annual growth of 6%
4

5. Failures Everywhere …
5

6. Failures Everywhere …
McKinsey, U. Oxford, 2012:
• 5400 large IT projects (> 15 $M)
• 17% threatened the very
existence of companies
• On average: 45% over budget,
56 percent less value
6

7. Software Engineering Research Funding
& Relevance
• Software Engineering (SE) research should be a top priority given
its importance in society.
• But that is not the case anymore (with a few exceptions).
• Symptoms
– Listed priorities by research councils, relative funding
– University hiring, few software engineering departments
– Large centers or institutes being established or closed down
• May be partly related to (perceived) lack of relevance?
– Industry participation in leading SE conferences
– Application/Experience tracks not first class citizens
– A very small percentage of research work is ever used or even
assessed on real industrial software
– Impact project (ACM SIGSOFT)
7

8. Basili’s and Meyer’s Take
• Many (most?) of the advances in software engineering have
come out of non-university sources
• “Academic research has had its part, honorable but
limited.” (Meyer)
• Large scale labs don’t get funded, like they do in other
engineering and scientific disciplines (Basili, Meyer)
• One significant difference though is that we cannot entirely
recreate the phenomena we study within four walls
• Question: What is our responsibility in all this?
8

9. Setting the Stage

10. Engineering Research
• “Engineering: The application of scientific and
mathematical principles to practical ends such as the
design, manufacture, and operation of efficient and
economical structures, machines, processes, and
systems.” (American Heritage Dictionary)
• Engineering research: innovative engineering solutions
–
–
–
–
–
Problem driven
Real world requirements
Scalability
Human factors, where it matters
Economic tradeoffs and cost-benefit analysis
10

11. Pasteur’s Quadrant
Consideration of Use?
No
Quest for
Fundamental
Understandin
g?
Yes
Yes
Pure Basic
Research
(Bohr)
Use-inspired
Basic
Research
(Pasteur)
No
-
Pure Applied
Research
(Edison)
Donald Stokes, Pasteur’s Quadrant, 1997
11

12. Software Engineering Research
• Pure basic research in software engineering?
• As an engineering discipline, all research work should
contribute to:
– Knowledge discovery but also …
– Innovative (software) engineering solutions
• Pasteur’s quadrant: Research should driven by utility
and its results be put to use …
• This requires a knowledge of engineering practice
• Are the majority of projects and papers doing so?
• If not, why is that? What can we do about it?
12

13. Research Example

14. A Representative Example
• Parnin and Orso (ISSTA, 2011) looked at automated
debugging techniques
• 50 years of automated debugging research
• Only 5 papers have evaluated automated debugging
techniques with actual programmers
• Focus since ~2001: dozens of papers ranking program
statements according to their likelihood of containing a
fault
• Experiment
– How do programmers use the ranking?
– Do they see the bugs?
– Is the ranking important?
14

15. Results from Parnin and Orso’s Study
•
•
Only low performers strictly followed the ranking
Only one out of 10 programmers who checked a buggy
statement stopped the investigation
• Automated support did not speed up debugging
• Developers wanted explanations rather than recommendations
• We cannot abstract the human away in our research
• “… we must steer research towards more promising
directions that take into account the way programmers
actually debug in real scenarios.”
15

16. What Happened?
• How people debug and what information they need is poorly
understood
– Probably varies a great deal according to context and skills
• Researchers focused on providing a solution that was a
mismatch for the actual problem
• That line of research became fashionable: a lot of (cool) ideas
could be easily applied and compared, without involving human
participants
• Resulted in many, many papers …
• Pasteur’s quadrant: That idea was never really put to use …
• Similar example: ISSTA 2012 paper on learning program
invariants (Daikon) by Matt Staats et al.
• Many other examples
16

17. Reconciling Relevance and Science in Software
Engineering
-Implementing Pasteur’s Quadrant

18. Objectives
• How to implement Pasteur’s quadrant in software engineering?
• Go through recent and successful projects with industry partners
– Simula Research Laboratory, Oslo, Norway
– SnT centre, U. of Luxembourg
• Summarize what happened, our experience
• Two projects, in the automotive domain, described in more
detail
• Draw conclusions and lessons learned
– Patterns for successful research
– Challenges and possible solutions
18

19. Mode of Collaboration
Adapted from Gorschek et al., IEEE Software 2006
19

20. Projects$Overview$(<$5$years)$
Company
Domain
Objective
Notation
Automation
ABB
Robot controller
Func. Safety Testing
UML
Model analysis for
coverage criteria
Cisco
Video conference
Testing (robustness)
UML profile
Metaheuristic search
Kongsberg Maritime
Fire and gas safety
control system
Safety certification
SysML + traceability
Model slicing algorithm
Kongsberg Maritime
Oil&gas, safety critical
drivers
CPU usage analysis
UML+MARTE
Constraint Solver
FMC
Subsea system
Automated
configuration
UML profile
Constraint solver
WesternGeco
Marine seismic
acquisition
Func. Testing
UML profile + MARTE
Metaheuristic search
DNV
Marine and Energy,
certification body
Compliance with safety
standards
UML profile
Constraint verification
SES
Satellite operator
Func. Testing
UML profile
Metaheuristic search
Delphi
Automotive systems
Testing (safety
+performance)
Matlab/Simulink
Metaheuristic search
Lux. Tax department
Legal & financial
Legal Req. QA &
testing
UML profile
Under investigation
20

21. Project Example 1: Cisco
•
•
•
•
•
Context: Video conferencing systems
Original scientific problem: Modeling and test case generation, oracle,
coverage strategy
Practical observation: Access to test network infrastructure limited
(emulate network traffic, etc.). Models get too large and complex.
Modified research objectives: (1) How to select an optimal subset of
test cases matching the time budget, (2) Modeling cross-cutting
concerns
References: Hemmati et al. (2010),
Ali et al. (2011)
21

22. Project Example 1: Cisco
• Context: Video conferencing systems
• Original scientific problem: Modeling and model-based
test case generation, oracle, coverage strategy
• Practical observation: Access to test network
infrastructure limited (emulate network traffic, etc.). Models
get too large and complex.
• Modified research objectives: (1) How to select an optimal
subset of test cases matching the time budget, (2)
Modeling cross-cutting concerns
• References: Hemmati et al. (2010),
Ali et al. (2011)
22

23. Project Example 2: Siemens
•
•
•
•
•
Context: 3-D image segmentation algorithms for medical applications
(Siemens)
Original scientific problem: Define specific test strategies for
segmentation algorithms
Practical observations: Algorithms are validated by using highly
specialized medical experts. Expensive and slow. No obvious test
oracle
Modified research objective: Learning oracles for image segmentation
algorithms in medical applications. Machine learning.
Reference: Frouchni et al. (2011)
23

24. Project Example 2: Siemens
•
•
•
•
•
Context: 3-D image segmentation algorithms for medical applications
(Siemens)
Original scientific problem: Define specific test strategies for (constantly
evolving) segmentation algorithms
Practical observations: Test results are validated by using highly
specialized medical experts. Expensive and slow. No obvious test
oracle: non-testable systems
Modified research objective: Learning oracles for image segmentation
algorithms in medical applications. Machine learning.
Reference: Frouchni et al. (2011)
24

25. Project Example 3: FMC
• Context: Subsea integrated control systems
• Original scientific problem: integration in
systems of systems
• Practical observations: Each subsea
installation is unique (variant), the software
configuration is extremely complex
(thousands of configuration parameters), >
50% configuration faults
• Modified research objective: Real-time,
automated support to the configuration
process using domain specific product line
modeling (focus on HW-SW dependencies)
and constraint solving: : instant validation,
guidance, inferred values
• Behjati et al. (2012)
25

26. Project Example 3: FMC
• Context: Subsea integrated control systems
• Original scientific problem: integration in
systems of systems
• Practical observations: Each subsea
installation is unique (variant), the software
configuration is extremely complex (thousands
of configuration parameters), > 50%
configuration faults
• Modified research objective: Real-time,
automated support to the configuration
process using domain specific product line
modeling (focus on HW-SW dependencies)
and constraint solving: instant validation,
guidance, inferred values
• Behjati et al. (2012)
26

27. Project Example 4: Kongsberg Maritime
• Context: safety-critical embedded systems in the energy
and maritime sectors, e.g., fire and gas monitoring,
process shutdown, dynamic positioning
• Original scientific problem: Model-driven engineering for
failure-mode and effect analysis
• Practical observations: Certification meetings with thirdparty certifiers. Certification is lengthy, expensive, etc.
Requirements-Design decisions traceability in large
complex systems a priority.
• Modified research objective: Traceability between safety
requirements and system design decisions. Solution
based on SysML and a simple traceability language
along with model slicing.
• Reference: Sabetzadeh et al. (2011)
27

28. Project Example 4: Kongsberg Maritime
• Context: safety-critical embedded systems in the energy
and maritime sectors, e.g., fire and gas monitoring,
process shutdown, dynamic positioning
• Original scientific problem: Model-driven engineering for
failure-mode and effect analysis
• Practical observations: Certification meetings with thirdparty certifiers: Certification is lengthy, expensive, etc.
Requirements-Design decisions traceability in large
complex systems is an issue.
• Modified research objective: Traceability between safety
requirements and system design decisions. Solution
based on SysML and a simple traceability language
along with model slicing.
• Reference: Sabetzadeh et al. (2011), Nejati et al. (2012)
28

29. Detailed Example 1
Testing Closed Loop Controllers
R. Matinnejad et al., 2013
2

30. Complexity$and$amount$of$so?ware$used$on$vehicles’$$
Electronic$Control$Units$(ECUs)$grow$rapidly$$
Comfort and variety
More functions
Faster time-to-market
Safety and reliability
Greenhouse gas emission laws
Less fuel consumption
30

31. Model-based control system development has
three major stages
Model-in-the-Loop
Stage
Software-in-the-Loop
Stage
Hardware-in-the-Loop
Stage
Simulink Modeling
Code Generation
and Integration
Software Running
on ECU
Generic
Functional
Model
MiL Testing
Software
Release
SiL Testing
HiL Testing
31

32. Major$Challenges$in$MiLGSiLGHiL$TesIng$$
• Manual test case generation
• Complex functions at MiL, and large and integrated
software/embedded systems at HiL
• Lack of precise requirements and testing Objectives
• Test selection at HiL
32

33. MiL$tesIng$
Requirements
Individual Functions
The ultimate goal of MiL testing is to
ensure that individual functions
behave correctly and timely on any
hardware configuration
33

34. A$Taxonomy$of$AutomoIve$FuncIons$
Computation
Transforming
unit convertors
Calculating
Controlling
State-Based
calculating positions, State machine
controllers
duty cycles, etc
Continuous
Closed-loop
controllers (PID)
Different testing strategies are required for
different types of functions
34

35. Dynamic Continuous Controllers are present in
many embedded systems including ECUs
35

36. Controller Plant Model and its Requirements
Desired value +
Error
-
Controller
(SUT)
Plant
Model
System output
Actual value
Desired Value & Actual Value
(a)
Liveness
Smoothness
(b)
=<
~= 0
(c)
Responsiveness
w
v
y >=
x
z
Desired Value
Actual Value
time
time
time
36

37. Search$Elements$
• Search:
• Inputs: Initial and desired values, configuration parameters
• (1+1) EA
• Search Objective:
• Example requirement that we want to test: liveness
! |Desired - Actual(final)|~= 0
For each set of inputs, we evaluate the objective function over the resulting
simulation graphs:
• Result:
• worst case scenarios or values to the input variables that are
more likely to break the requirement at MiL level
• stress test cases based on actual hardware (HiL)
37
37

38. MiL-Testing of Continuous Controllers
Objective
Functions
+
Domain
Expert
Exploration
Controllerplant model
List of
Regions
Local Search
Test
Scenarios
Overview
Diagram
1.0
0.9
Desired Value
Actual Value
Initial Desired
0.8
0.7
0.6
0.5
0.4
0.3
Final Desired
0.2
0.1
0.0
0
1
time
2
38

39. Conclusions$
• Addressed a testing problem that has been largely ignored
i.e., 31s. much horizontal axis of the diagrams in Figure 8 shows the number of
• We found Hence, theworse scenarios during MiL testing than our
iterations instead of the computation
In
start both random search
partner had found so far time.the addition, wefrom the exploration step. and
(1+1) EA from the same initial point, i.e.,
worst case
Overall in all the
• Much worse thanregions, (1+1) EA eventually reachesmore deterministic than ranrandom search EA is its plateau at a value higher
than the random search plateau value. Further, (1+1)
has smaller
of testing is
• Theydom, i.e., the distribution of (1+1) EAthea HiL8).variance than that(e.g., Figure 8(d)), much
are running them at (see Figure level, where random search,
especially when reaching the plateau
In some regions
more expensive: MiL results ->faster than (1+1) EA, while in HiL other
however, random reaches its plateau slightly test selection for some
regions (e.g. Figure 8(a)), (1+1) EA is faster. We will discuss the relationship between
• But further landscape and the performance of (1+1) EA in RQ3.
the region research is needed:
RQ3. We drew the landscape for the 11 regions in our experiment. For example, Fig– To 9 shows with the for two selected regions in Figures parameters
ure deal the landscape many configuration 7(a) and 7(b). Specifically,
Figure 9(a) shows the landscape for the region in Figure 7(b) where (1+1) EA is faster
– To dynamically adjust search algorithms in different
than random, and Figure 9(b) shows the landscape for the region in Figure 7(a) where
(1+1) EA is slower than different
subregions withrandom search. lanscapes
(a)
(b)
0.40
0.20
0.39
0.19
0.38
0.18
0.37
0.17
0.36
0.16
0.35
0.15
0.34
0.14
0.33
0.13
0.32
0.12
0.31
0.11
0.30
0.10
0.70 0.71
0.72
0.73
0.74
0.75 0.76
0.77
0.78
0.79
0.80
0.90 0.91
0.92
0.93
0.94
0.95 0.96
0.97
0.98
0.99
1.00
Fig. 9. Diagrams representing the landscape for two representative HeatMap regions: (a) Landscape for the region in Figure 7(b). (b) Landscape for the region in Figure 7(a).
39

40. Detailed Example 2
Minimizing CPU Time Shortage Risks in
Integrated Embedded Software
S. Nejati et al., 2013

41. Today’s$cars$rely$on$integrated$systems$
• Modular and independent development
• Many opportunities for division of labor and
outsourcing
• Need for reliable and effective integration
processes
41

42. IntegraIon$process$in$the$automoIve$domain$
AUTOSAR Models
sw runnables
Glue
AUTOSAR Models
sw runnables
42

43. CPU$Time$Shortage$in$Integrated$Embedded$$
So?ware$
• Static cyclic scheduling: predictable, analyzable
• Challenge
– Many OS tasks and their many runnables run within a limited
available CPU time
• The execution time of the runnables may exceed their time slot
• Our goal
– Reducing the maximum CPU time used per time slot to be
able to
• Minimize the hardware cost
• Reduce the probability of overloading the CPU in practice
• Enable addition of new functions incrementally
✗
(a)
5ms
10ms
15ms
20ms
25ms
30ms
35ms
40ms
✔
(b)
5ms
10ms
15ms
20ms
25ms
30ms
35ms
40ms
Fig. 4. Two possible CPU time usage simulations for an OS task with a 5ms
cycle: (a) Usage with bursts, and (b) Desirable usage.
to deadlin
run passe
by OS ta
divisible
43
offset val
43
synchron

44. Using&runnable&offsets&(delay&3mes)&
Inserting runnables’ offsets
5ms
5ms
10ms
10ms
15ms
15ms
20ms
20ms
25ms
25ms
30ms
30ms
35ms
35ms
40ms
40ms
✗
✔
Offsets have to be chosen such that
the maximum CPU usage per time slot is minimized, and further,
the runnables respect their period
the runnables respect their time slot
the runnables satisfy their synchronization constraints
44
44

45. Meta$heurisIc$search$$algorithms$
- The objective function is the max CPU usage of a 2s-simulation of
runnables
- The search modifies one offset at a time, and updates other offsets
only if timing constraints are violated
- Single-state search algorithms for discrete spaces (HC, Tabu)
Simulation for the runnables
case
Case Study: an automotive software system with 430lowest maxin our usagestudy and HC
runnables found by
corresponding to the
CPU
5.34 ms
2.13 ms
Running the system without offsets
Optimized offset assignment
45
45

46. Comparing$different$search$algorithms$$
(ms)
Best CPU usage
(s)
Time to find
Best CPU usage
46
46

47. Conclusions$
- Though schedulability analysis is a
well developed field, the problem we
addressed was never defined in those
terms (context)
- Search algorithms to compute offset
values that reduce the max CPU time
needed
- Positive evaluation results: Quick and
significant differences
- Huge search space with constraints
- Current: Accounting for task time
coupling constraints with multiobjective search " trade-off between
relaxing coupling constraints and
maximum CPU time
47
47

48. What Have I Learned?

49. Successful Research Patterns
•
•
•
•
•
Successful: Innovative and high impact
Inductive research: Working from specific observations in real settings to
broader generalizations and theories
– Software development is highly diverse
– Problems and solutions are diverse too
– Context factors matter a great deal
– Generalization: Field studies and replications, analyze commonalities
Scalability and practicality considerations must be part of the initial research
problem definition
Researching by doing: Hands-on research. Apply what exists in well
defined, realistic context, with clear objectives. The observed limitations
become the research objectives. Put new technology to actual use.
Interdisciplinary: CS, Mathematics, Engineering, or non-technical domains
49

50. So What?
• Making a conscious effort to understand the problem
first
– A careful investigation often leads to surprises and
a very different understanding of the issues at
stake
– Precisely identify the requirements for an
applicable solution
– More papers focused on understanding the
problems
– Making experience/application tracks first class
citizens in SE conferences
50

51. So What? - cont’d
• Better relationships between academia and industry
– Different models
• Fraunhofer centres, Germany
• Research-based innovation centers in Norway
• SnT centre in Luxembourg
• Targeted grant programs
• Joint industry-academia labs (e.g., NASA SEL Lab)
– Mutually beneficial setting where software development is an
object of study
– Exposing PhD students to industry practice, management
and leadership: Ethical considerations (“Fix the PhD”,
Nature, 2011)
– Incentives for SE academics
51

52. So What? - Cont’d
• Work on end-to-end solutions: Pieces of solutions are
interdependent. Necessary for impact, e.g., requirements and
testing.
• Beyond professors and students
– Labs with interdisciplinary teams of professional scientists
and engineers within or collaborating with universities
– Used to be the case with corporate research labs: Bell Labs,
Xerox PARC, HP labs, NASA SEL, etc.
– Now: Fraunhofer (Germany), Simula (Norway), Microsoft
Research (US), SEI (US), SnT (Luxembourg), FBK (Italy)
– Corporate labs versus publicly supported ones?
– Key point: The level of basic funding must allow high risk
and rigorous research, performed by professional scientists,
focused on impact in society (Use-inspired basic research)
52

53. The “Classical” Model of Research and
Innovation
Basic
Research
Applied
Research
Innovation
and
Development

54. An Effective, Collaborative Model of Research
and Innovation
Innovation & Development
Basic Research
Applied
Research
• Basic and applied research take place in a rich context
• Basic Research is also driven by problems raised by
applied research
• Main motivation for SnT’s partnership program
B. Schneiderman, The Atlantic, Toward an Ecological Model of Research and Development, 2013

55. Challenges

56. Academic Challenges
•
•
•
•
•
•
Our CS legacy … emancipating ourselves as an engineering discipline
– Electrical engineering: Subfield of Physics until late 19th century
– Software engineering departments?
How cool is it? SE research is more driven by “fashion” than needs, a
quest for silver bullets
– We can only blame ourselves
Counting papers and how rankings do not help
– We are pressuring ourselves into irrelevance
Taking academic tenure and promotion seriously
– What about rewarding impact?
One’s research must cover a broader ground and be somewhat
opportunistic – this pushes us out of our comfort zone
Resources to support industry collaborations
– Large lab infrastructure, research engineers, time
56

57. Industrial Challenges
•
•
•
•
•
Distinguish the manifestations of a problem from its causes is not easy
in practice
Short term versus longer term goals (next quarter’s forecast is
sometimes the priority)
Industrial research groups are often disconnected from their own
business units and external researchers may be perceived as
competitors
Company’s intellectual property regulations may conflict with those of
the research institution
Complexity of industrial systems and technology
– Cannot be transplanted in artificial settings for research - Need
studies in real settings
– Substantial domain knowledge is required
57

58. A Double-Agent Life
Scientist,
inquisitive but
discrete
Warning: No
research here
A new idea
(as initially
perceived by our
partners)
Practitioner
(anonymous)
58

59. Conclusions
• Software engineering is obviously important in all aspects of
society, but academic software engineering research is not
always perceived the same way
• The academic community, at various levels and in particular its
leadership, is partly responsible for this
• How we take up the challenge of increasing our impact will
determine the future of the profession
• There is some limited progress, but far too slow
• There are solutions, but no silver bullet
• We all have a role to play in this, as deans, department chairs,
professors, scientists, reviewers, conference organizers, journal
editors, etc. We can all be double-agents …
59

60. SOUNDING BOARD
Editor: Philippe Kruchten
University of British Columbia
pbk@ece.ubc.ca
Embracing the
Engineering Side of
Software Engineering
Lionel Briand
I HAVE NOW been a professional researcher
in software engineering for roughly 20 years.
Throughout that time, I’ve worked at universities and in research institutes and collaborated on research projects with 30-odd private companies and public institutions. Over
the years, I have increasingly questioned and
reflected on the impact and usefulness of my
research work and, as a result, made it a priority to combine my research with a genuine involvement in actual engineering problems. This short piece aims to reflect on my
experiences in performing industry-relevant
software engineering research across several
countries and institutions.
Not So Hot Anymore
I suppose a logical start for this article is to
assess, albeit concisely, the current state of
software engineering research. As software
engineering is widely taught in many universities, due in large part to a strong demand
for software engineers in industry, the number of software engineering academics is substantial. The Journal of Systems and Software ranks researchers every year, usually
accounting for roughly 4,000 individuals actively publishing in major journals.
When I started my career, software engineering was defi nitely a hot topic in academia: funding was plentiful, and universities and research institutes were hiring in
record numbers. This clearly isn’t the case
anymore. Public funding for software engineering research has at best stagnated, and
in many countries, declined significantly.
96 I E E E S O F T W A R E | P U B L I S H E D B Y T H E I E E E C O M P U T E R S O C I E T Y
Hiring for research positions is limited and
falls far below the number of software engineering graduates seeking research careers.
Industry attendance at scientific software
engineering conferences is roughly 10 percent, including the scientists from corporate
research centers. Adding insult to injury, in
many academic and industry circles, software engineering research isn’t even considered to be a real scientific discipline. I’ll spare
you the numerous unpleasant comments
about the credibility and scientific underpinning of software engineering research that
I’ve heard over the years.
This situation isn’t due to the subject matter’s lack of relevance. Software systems are
pervasive in all industry sectors and have become increasingly complex and critical. The
software engineering profession repeatedly
tops job-ranking surveys. In many cases, most
of a product’s innovation lies in its software
components—for an example, think of the
automotive industry. In all my recent industry
collaborations, I’ve observed that all the issues and challenges traditionally faced in software development are becoming more acute.
So how can we explain the paradox of being both highly relevant and increasingly underfunded and discredited?
Looking for Some Answers
Like other disciplines before us, because
we’re a young and still-maturing engineering field, we lack the credibility of more
continued on p. 93
0 74 0 -74 5 9 / 12 / $ 3 1. 0 0 © 2 0 12 I E E E
60

61. Empirical Software Engineering
•
•
•
•
Springer, 6 issues a year
Both research papers and industry experience
reports
High impact factor among SE research journals
“Applied software engineering research with a
significant empirical component”
61

62. Personal References
•
•
•
•
•
•
•
•
•
Hemmati, Briand, Arcuri, Ali “An Enhanced Test Case Selection Approach for Model-Based Testing: An
Industrial Case Study”, ACM FSE, 2010
Frouchni, Briand, Labiche, Grady, and Subramanyan, “Automating Image Segmentation Verification and
Validation by Learning Test Oracles”, Information and Software Technology (Elsevier), 2011.
Sabetzadeh, Nejati, Briand, Evensen Mills “Using SysML for Modeling of Safety-Critical Software–
Hardware Interfaces: Guidelines and Industry Experience”, HASE, 2011
Sabetzadeh et al., “Combining Goal Models, Expert Elicitation, and Probabilistic Simulation for Qualification
of New Technology”, HASE, 2011
Ali, Briand, Hemmati, “Modeling Robustness Behavior Using Aspect-Oriented Modeling to Support
Robustness Testing of Industrial Systems”, Journal of Software and Systems Modeling (Springer), 2011.
Behjati, Nejati, Yue, Gotlieb, Briand, “Model-based Automated and Guided Configuration of Embedded
Software Systems”, ECMFA, 2012.
Behjati, Yue, Briand, Selic. SimPL, “A Product-Line Modeling Methodology for Families of Integrated
Control Systems, Information and Software Technology”, Information and Software Technology (Elsevier),
2012.
Nejati et al., A SysML-Based Approach to Traceability Management and Design Slicing in Support of Safety
Certification: Framework, Tool Support, and Case Studies, Information and Software Technology (Elsevier),
2012
Iqbal, Arcuri, Briand, “Empirical Investigation of Search Algorithms for Environment Model-Based Testing of
Real-Time Embedded Software”, ACM ISSTA, 2012
62

63. Personal References II
•
•
•
•
•
•
•
Nejati, Di Alesio, Sabetzadeh, Briand, “Modeling and Analysis of CPU Usage in Safety-Critical Embedded
Systems to Support Stress Testing”, ACM/IEEE MODELS 2012
Nejati, Sabetzadeh, Falessi, Briand, Coq, “A SysML-based approach to traceability management and
design slicing in support of safety certification: Framework, tool support, and case studies”, Information &
Software Technology, (Elsevier), 2012
Hemmati, Arcuri, L. Briand: Achieving scalable model-based testing through test case diversity. ACM
Trans. Softw. Eng. Methodology, 2013.
Panesar-Walawege, Sabetzadeh, Briand: Supporting the verification of compliance to safety standards via
model-driven engineering: Approach, tool-support and empirical validation. Information & Software
Technology (Elsevier), 2013
S. Nejati et al., “Minimizing CPU Time Shortage Risks in Integrated Embedded Software”,
28th IEEE/ACM International Conference on Automated Software Engineering, 2013
Sabetzadeh, Falessi, Briand, Di Alesio: A goal-based approach for qualification of new technologies:
Foundations, tool support, and industrial validation. Reliability Eng. & System Safety, 2013
Briand et al., “Traceability and SysML Design Slices to Support Safety Inspections: A Controlled
Experiment”, forthcoming in ACM Transactions on Software Engineering and Methodology, 2013
63

64. Other References
•
•
•
•
Bertrand Meyer’s blog:
http://bertrandmeyer.com/2010/04/25/the-other-impediment-tosoftware-engineering-research/
Basili, “Learning Through Application: The maturing of the QIP in the
SEL”, Making Software; What really works and why we believe it,
Edited by Andy Oram and Greg Wilson, O’Reilly Publishers, 2011, pp.
65-78.
T. Gorschek, P. Garre, S. Larsson, C. Wohlin. A Model for Technology
Transfer in Practice. IEEE Software 23(6), 2006.
Parnin and Orso, “Are Automated Debugging Techniques Actually
Helping Programmers?”, ISSTA, 2011
64