The document discusses the growing importance of modeling in various domains like science, engineering, and software. It notes that models are used everywhere from emergency response to financial decisions. The document argues that automated software engineering techniques are needed to best reason about complex models. It provides several examples of how optimization and evolutionary algorithms have been applied to problems in software engineering like program repair, product line configuration, spacecraft design, and avionics requirements engineering. The overall message is that modeling is a key emerging area and automated approaches are crucial for dealing with large, complex models.

1. Automated Software
Engineering
Cs791
Fall 2015
tjmenzie@ncsu.edu
Nov 21, 2014

2. Career advice
Learn Modeling
(it’s the next big thing in SE)
2

3. This kind of modeling? No
3

4. This kind of modeling? Nope
(though it
does look
pretty cool,
heh?)
4

5. Executable Software Models
5

6. Software models: are everywhere
• If you call an ambulance in London or New York,
– those ambulances are controlled by emergency
response models.
• If you cross the border Arizona to Mexico,
– A models determines if you are taken away for
extra security measures.
• If you default on your car loans,
– A model determines when (or if) someone to
repossess your car.
• If the stock market crashes,
– it might be that some model caused the crash.
6

7. Software models: are everywhere
• Karplus and Levitt
– 2013 Nobel prize in chemistry
– development of multi-scale
models for complex chemical
systems
– Explored complex chemical
reactions (e.g. split-second
changes of photosynthesis).
7
• Models are now a central tool
in scientific research.
– in physics, biology and other
fields of science
– complex simulations using
supercomputers.
• E.g. genomic map required
analyzing 80 trillion bytes
• E.g.. Other computational
modeling projects
– the rise and fall of native
cultures,
– subnuclear particles
– the Big Bang.

8. Question:
How to best reason
about models?
Answer:
automated software engineering
8

9. The Human Condition:
a balancing act between goals
• The human condition
– a constant balancing act between vaguely understood, often
competing, goals.
• For example:
– Lets build the software faster...
– ... with few bugs ...
– ... using less budget.
• For the consequences
of "better, faster, cheaper” :
– NASA's loss of the Mars Polar
Orbiter
– $300 million , wasted
9

10. Example
• Choices explored by satellite designers at NASA's Jet
Propulsion Laboratory.
– Dozens of experts in
propulsion,
communications,
guidance and control etc
– Week-long sessions
– Thrash out possibilities
for NASA's next deep
space mission.
• Groups sitting “here” can make decisions that impact
“there”, and they just do not realize.
10

11. Keeping track of all those decisions:
Lightweight requirements
notations
• E.g. Martin Feather’s DDP
language
– Mitigations …
– … that retire risks….
– …. That damage
requirements
• Multi-objective
optimization
– Find the cheapest
mitigations that enable the
most requirements
11

12. Keeping track of all those decisions:
Lightweight requirements
notations
• My optimizers out-performing
state of the art tools
– Faster
– Simpler policies
12
Data Mining for Very Busy People, Tim Menzies,
Ying Hu, IEEE Computer, October 2003

13. Keeping track of all those decisions:
Lightweight requirements
notations
• E.g. John
Mylopoulos‘
soft goals
• Competing
requirements
• Simulations to
explore all
the what-ifs
Chiang, Eliza, Menzies, Tim, Simulations for very early lifecycle quality evaluations, 13
Software Process: Improvement and Practice 7(3-4), 2003,

14. More lightweight
requirements notations
• E.g. Kang’s product lines
[Kang’90]
• Add in known constraints
– E.g. “if we use a camera
then we need a high
resolution screen”.
• Extract products
– Find subsets of the
product lines that satisfy
constraints.
– If no constraints, linear
time
– Otherwise, can defeat
state-of-the-art optimizers
[Pohl et at, ASE’11]
[Sayyad, Menzies ICSE’13].
Cross-Tree
Constraints
14
A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the
Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA.
November 2013.

15. Automated software engineering
and optimization
• Many SE activities are like optimization
problems [Harman,Jones’01].
• Due to computational complexity, exact optimization methods
can be impractical for large SBSE problems
• So researchers and practitioners use metaheuristic search to
find near optimal or good-enough solutions.
– E.g. simulated annealing [Rosenbluth et al.’53]
– E.g. genetic algorithms [Goldberg’79]
– E.g. tabu search [Glover86]
15

16. Optimization and
evolutionary algorithms
• Repeat till happy or exhausted
– Selection (cull the herd)
– Cross-over (the rude bit)
– Mutation (stochastic jiggle)
1
-- better on some
criteria, worse on none
-- generation[i+1] comes
from Pareto frontier of
generation[i]
2
3
5
4
Pareto frontier
7
6
Selection:
8
9
16

17. Long tradition of software,
simulation, and optimization
• Not one “best” solutions
– Its all trade-offs
– Satisficing = satisfy + sacrifice
• Problems have to be
explored via computer
simulations
– Try it out and see
– And approach pioneered
by John Von Neumann,
in the 1950s
17

18. Explosive growth of SE + optimization
Q: Why?
A: Thanks to Big Data, more access to more cpu.
18

19. Applications of Optimization in SE
1. Requirements Menzies, Feather, Bagnall, Mansouri, Zhang
2. Transformation Cooper, Ryan, Schielke, Subramanian, Fatiregun, Williams
3.Effort prediction Aguilar-Ruiz, Burgess, Dolado, Lefley, Shepperd
4. Management Alba, Antoniol, Chicano, Di Pentam Greer, Ruhe
5. Heap allocation Cohen, Kooi, Srisa-an
6. Regression test Li, Yoo, Elbaum, Rothermel, Walcott, Soffa, Kampfhamer
7. SOA Canfora, Di Penta, Esposito, Villani
8. Refactoring Antoniol, Briand, Cinneide, O’Keeffe, Merlo, Seng, Tratt
9. Test Generation Alba, Binkley, Bottaci, Briand, Chicano, Clark, Cohen, Gutjahr, Harrold, Holcombe, Jones,
Korel, Pargass, Reformat, Roper, McMinn, Michael, Sthamer, Tracy, Tonella,Xanthakis, Xiao,
Wegener, Wilkins
10. Maintenance Antoniol, Lutz, Di Penta, Madhavi, Mancoridis, Mitchell, Swift
11. Model checking Alba, Chicano, Godefroid
12. Probing Cohen, Elbaum
13. UIOs Derderian, Guo, Hierons
14. Comprehension Gold, Li, Mahdavi
15. Protocols Alba, Clark, Jacob, Troya
16. Component sel Baker, Skaliotis, Steinhofel, Yoo
17. Agent Oriented Haas, Peysakhov, Sinclair, Shami, Mancoridis 19

20. Example 1
(of four)
Automated program repair
20

21. Multi-objective Optimization,
in the 21st century, automated repair
A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each :
Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12
21

22. A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each :
Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12
22

23. A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each :
Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12
23

24. •
A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each :
Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12
24

25. Example 2
(of four)
Software Product Lines
25

26. More lightweight
requirements notations
• E.g. Kang’s product lines
[Kang’90]
• Add in known constraints
– E.g. “if we use a camera
then we need a high
resolution screen”.
• Extract products
– Find subsets of the
product lines that satisfy
constraints.
– If no constraints, linear
time
– Otherwise, can defeat
state-of-the-art optimizers
[Pohl et at, ASE’11]
[Sayyad, Menzies ICSE’13].
Cross-Tree
Constraints
26
A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the
Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA.
November 2013.

27. Problem: many competing goals
2 or 3 or 4 or 5 goals
Software engineering = navigating the user goals:
1. Satisfy the most domain constraints (0 ≤ #violations ≤ 100%)
2. Offers most features
3. Build “stuff” In least time
4. That we have used most before
5. Using features with least known defects
27
A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the
Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA.
November 2013.

28. Issues of scale up
• This model: 10 features, 8 rules
• [www.splot-research.org]:
ESHOP: 290 Features, 421 Rules
• LINUX kernel variability project
LINUX x86 kernel
6,888 Features; 344,000 Rules
Cross-Tree Constraints
28
A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the
Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA.
November 2013.

29. State of the Art (as of Nov’13)
Features
6888
SPLOT Linux (LVAT)
544
290
9
Henard
‘12
White ‘07, ‘08, 09a, 09b,
Pohl ‘11 Lopez-
Herrejon
‘11
Sayyad,
Menzies’13b
Sayyad,
Menzies’
13a
Velazco
‘13
Johansen
‘11
Benavides
‘05
Shi ‘10, Guo ‘11
Objectives
Single-goal Multi-goal
300,000+
clauses
29
A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the
Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA.
November 2013.

30. Example 3
(of four)
Space ships!
30

31. Skip re-entry
• My optimizers vs state of the art numeric optimizers
• My tools: ran 40 times faster
– Generated better solutions
• Powerful succinct explanation tool
31
Automatically Finding the Control Variables for Complex System Behavior Gregory Gay, Tim Menzies, Misty
Davies, and Karen Gundy-Burlet Journal - Automated Software Engineering, 2010 [PDF]

32. Example 4
(of four)
Avionics Requirements Engineering
32

33. 33
Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim
Menzies, Misty Davies, IEEE Trans Human Factors, to appear

34. WMC: GIT’s Work Models that
Compute [Kim’11]
• Cognitive models of the agents
(both pilots and computers)
– Late descent,
– Unpredicted rerouting,
– Different tailwind conditions
• Goal: validate operations
procedures (are they safe?)
• NASA’s analysts want to
explore 7000 scenarios.
– With current tools (NSGA-II)
– 300 weeks to complete
• Limited access to hardware
– Queue of researchers wanting
hardware access
– Hardware pulled away if in-flight
incidents for manned
space missions
Asiana Airlines
Flight 214
34
Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim
Menzies, Misty Davies, IEEE Trans Human Factors, to appear

35. Active Learning: Skip similar examples,
focus on the most different
4 mins (GALE) vs 7 hours (rest)
Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim
Menzies, Misty Davies, IEEE Trans Human Factors, to appear

36. Career advice
Learn Modeling
(it’s the next big thing in SE) 36

37. Question:
How to reason
about models?
Answer:
automated software engineering
Cs791, fall 2015
37