Software Speciﬁcation Discovery : A New Data Mining Approach
Software Speciﬁcation Discovery : A New Data Mining Approach
David Lo and Siau-Cheng Khoo
Department of Computer Science, National University of Singapore
Abstract nance  and 50% of the maintenance cost is due to com-
prehending or understanding an existing code base .
Software has been an ubiquitous component in our daily Hence, approximately 45% of software cost is due to difﬁ-
life. It ranges from large software systems like operating culty in understanding an existing system. This is especially
systems to small embedded systems like vending machines, true for software projects developed by many developers
both of which we frequently interact with. Software changes over a long period of time. A good indication on the amount
often during its lifespan; these cause difﬁculty in under- associated with software costs is the US GDP’s software
standing existing systems. Program comprehension is es- component which amounts to $216.0 billion at the second
timated to take up to 45% of software costs which goes up quarter 2007 alone . Considering the above factors, re-
to billions of dollars. One of the root causes of this prob- ducing maintenance cost by addressing the problem of lack,
lem is the fact that documented software speciﬁcation is of- incomplete and outdated speciﬁcation can potentially save
ten missing, incomplete or outdated. This causes difﬁculties a large amount of wasted resources.
in software maintenance efforts especially when a software On another related front, software dependability is a ma-
project involves many people or the project continues over a jor concern of software vendors and users. Software de-
long period of time. Lack of documented software speciﬁca- pendability related issues have caused the loss of billions of
tion also causes difﬁculty in testing or verifying the correct- dollars and even the loss of lives . As reported by US
ness of a software system. Incorrect software has caused the National Institute of Standards and Technology (NIST) in
loss of billions of dollars and even the loss of life. One solu- 2002, incorrect or buggy software has caused US economy
tion to address the above problems is speciﬁcation discov- to suffer $59.5 billion dollars loss annually . Another
ery, namely, automated extraction of software speciﬁcation case with Ariane 5, a rocket project by European Space
from program artifacts. In this paper, we describe our past Agency that exploded on its maiden voyage due to speci-
and current work in the domains of data mining, software ﬁcation and design errors, highlights the need of addressing
engineering and programming language in addressing the software dependability .
discovery of software speciﬁcations with the goal of reduc-
To ensure correctness of a software system, program ver-
ing software costs and improving software dependability.
iﬁcation tools  have been proposed. However, program
veriﬁer can only check speciﬁed properties (or speciﬁca-
tion). If a property is not documented properly, incomplete,
or unavailable, not much can be done in ensuring the cor-
It’s best if all programs and software projects are de- rectness of a software system. Also, the difﬁculty in for-
veloped with clear, precise and documented speciﬁcations. mulating a set of formal properties, which is the format re-
However, due to hard deadlines and ‘short-time-to-market’ quired by standard program veriﬁers, has been a barrier to
requirement , software products often come with poor, its wide-spread adoption in the industry .
incomplete and even no documented speciﬁcation. This One approach to address the problem of incomplete, out-
situation is further aggravated by the phenomenon termed dated and missing speciﬁcation is speciﬁcation discovery
as software evolution [6, 26]. As software evolves (i.e. – automated extraction of software speciﬁcation from pro-
changes) the documented speciﬁcation is often not updated. gram artifacts. Having a complete and updated software
This might render the original speciﬁcation of little use after speciﬁcation is a great help to lower software maintenance
several cycles of program evolution . costs and improve software dependability. Mined speciﬁca-
Incomplete, outdated and missing speciﬁcation has con- tion can be utilized for aiding program understanding. Fur-
tributed to high software maintenance costs. It has been thermore, it can be converted to run-time tests and input as
investigated that 90% of software cost is due to mainte- properties-to-verify to standard program veriﬁers.
Software speciﬁcation can be represented in various for- A Info
A – appendFile
malisms. One common formalism is automata (i.e., a transi- Info S 5 A S – storeFile
N – rename
X – Connect T
tion system with start and end nodes) . Another formal- G – Login 3 4 V – retrieveFile
C – changeWorkingDirectory
O – Logout N
ism is Linear Temporal Logic (LTL)  expressions which Y – Disconnect 6 S L – listFiles
T – setFileType
W- <init> S
can be directly accepted by standard program veriﬁers .
Also, formal versions of UML sequence diagram, namely 7 S
0 19 O
Message Sequence Chart (MSC)  and Live Sequence D C
W C D A
Chart (LSC) , have been active interests in the software G
8 9 I 10 11
20 Y 21
1 X 2 G
modeling community. Not only are the above formalisms G M
intuitive enough to aid program understanding, they are also G
formal enough to be veriﬁed via veriﬁcation techniques. G R
Hence, they address both program comprehension and de- 13
pendability issues. V
In this paper we describe brieﬂy three threads of our past 14
15 V I - listNames
D - deleteFile
and current work in recovering speciﬁcation (in the form of V
V M – makeDirectory
L R – removeDirectory
automata [28, 29], LTL [27, 31] and LSC [30, 32]) from 16 C 17 18 O
program execution traces via data mining approaches. A C
program execution trace can be simply viewed as a series of Figure 1. CVS Protocol
method signatures (or names) of methods invoked when a
system is executed. through pipelining of four functional components: Error-
The outline of this paper is as follows. Section 2 high- trace ﬁltering, clustering, learning, and automata merging
lights our work in automaton-based speciﬁcation mining. (c.f., ). We demonstrate that such an architecture im-
Section 3 highlights our work in mining Linear Temporal proves the quality of the mining result for two primary rea-
Logic (LTL) expressions. Section 4 describes our work in sons:
mining Live Sequence Charts (LSCs). Section 5 discusses
related work, and ﬁnally Section 6 concludes and discusses 1. Early identiﬁcation and ﬁltering of erroneous program
future work. execution traces can improve the quality of speciﬁca-
2 Mining Automata 2. Over-generalization which occurs at the learning stage
Initial studies on mining software speciﬁcation often rep- can be mitigated by localization of learning process to
resent a mined speciﬁcation in the form of an automaton groups of related program execution traces.
(e.g., [2, 11, 38, 4]). The work by Ammons et al. is We conduct experiments to support our reasoning. Our
one of the pioneer in automaton-based speciﬁcation mining experiments aim at deriving the API interaction protocol for
. In , a machine-learning approach (i.e., an automaton a CVS application built on top of the Jakarta Commons Net
learner referred to as sk-strings ) is employed to dis- FTP library . There are six common FTP interaction
cover program speciﬁcation in the form of an automata by scenarios in the CVS application: Initialization, multiple-
analyzing program execution traces. It is assumed that in- ﬁle upload, download, and deletion, multiple-directory cre-
put traces must “reveal strong hints of correct protocols” ation and deletion. All scenarios begin by connecting and
although they can also contain errors. logging-in to the FTP server. They end by logging-off and
Our work proposes novel data mining algorithms (outlier disconnecting from the FTP server. The client side only
detection and clustering techniques) to improve the quality maintains a record of ﬁles backed-up in the FTP server.
of work in . All these scenarios are depicted in the automata shown in
First, we deﬁne an array of metrics to objectively mea- Figure 1. The dashed boxes, from top to bottom, represent
sure the performance of an automaton-based speciﬁcation ﬁle upload, repository initialization, ﬁle deletion, directory
miner (c.f., ). The most important one is an accuracy creation, directory deletion, and ﬁle download scenario re-
metric measured by the notion of recall and precision. Pre- spectively.
cision and recall can then be deﬁned as the proportion of Precs Recall
sentences in Sinf that is accepted by Sorig and the propor- SMArTIC 0.484 0.981
tion of sentences in Sorig that is accepted by Sinf where SMArTIC without ﬁltering 0.426 1
Sorig is the original speciﬁcation and Sinf is the inferred SMArTIC without clustering 0.263 0.984
speciﬁcation. sk-strings 0.225 1.000
Next, we devise a novel architectural framework (re- As shown in the table above, SMArTIC improves the
ferred to as SMArTIC) that achieves speciﬁcation mining precision while maintaining a good recall in the CVS pro-
tocol inference. The precision of SMArTIC are more than F (unlock)
double the precision of sk-strings. Both ﬁltering and clus- Meaning: Eventually unlock is called
tering help in increasing precision while maintaining a good XF (unlock)
recall. Meaning: From the next event onwards, eventually
unlock is called
3 Mining Linear Temporal Logic G(lock → XF (unlock))
Meaning: Globally whenever lock is called, then from
In this section, we describe mining Linear Temporal the next event onwards, eventually unlock is called
Logic (LTL)  expressions (or rules) satisfying given
support and conﬁdence thresholds. LTL is one of the two Table 1. LTL Expressions and their Meanings
most commonly used formalisms accepted by standard pro-
gram veriﬁers  – the other one is Computational Tree rule; (Conﬁdence) The likelihood of the rule’s premise be-
Logic (CTL) . While an automata expresses a global ing followed by its consequent in the traces. Only rules
picture of software speciﬁcation (which might be complex), satisfying user-deﬁned thresholds of minimum support and
mined LTL expressions break the speciﬁcation into smaller conﬁdence are mined.
pieces each expressing strongly observed behavior which is An effective search space pruning strategy, inspired by
easier to be understood. closed pattern mining strategies [44, 41], is utilized to efﬁ-
Rules having the following format are the target of our ciently mine multi-event rules from traces. To prevent blow-
mining algorithm: up in the number of rules, only a representative sub-set of
rules containing non-redundant ones is generated.
’Whenever a series of events ESpre occurs, eventually A case study is performed on traces from the security
another series of events ESpost also occurs.’ component of JBoss Application Server . This shows
the usefulness of our mining technique in recovering the
The above can be denoted as ESpre → ESpost . ESpre and
underlying protocol that the system obeys, thus aiding pro-
ESpost are referred to as the premise and consequent of the
One of the mined rules (shown with abbreviated method
These set of rules can be expressed in LTL, and belong
signatures) is presented in Figure 2. It describes authenti-
to two of the most frequently used families of temporal
cation using Java Authentication and Authorization Service
logic expressions for veriﬁcation (i.e., response and chain-
(JAAS) for EJB within JBoss-AS. When authentication sce-
response) according to a survey by Dwyer et al. . Ex-
nario starts, conﬁguration information is ﬁrst checked to
amples of such rules include:
determine authentication service availability – this is de-
1. Resource Locking Protocol: Whenever a lock is ac- scribed by the premise of the rule. This is followed by:
quired, eventually it is released. invocations of actual authentication events, binding of prin-
2. Network Protocol: Whenever an HDLC connection cipal information to the subject being authenticated, and uti-
is made and an acknowledgement is received, eventu- lizations of subject’s principal and credential information in
ally a disconnection message is sent and an acknowl- performing further actions – these are described in the con-
edgement is received. sequent of the rule.
There are a number of LTL operators, among which we
are only interested in the operators ‘G’,‘F’ and ‘X’. The Premise Consequent
operator ‘G’ speciﬁes that globally at every point in time XLoginConfImpl.getConfEntry() ClientLoginModule.initialize()
a certain property holds. The operator ‘F’ speciﬁes that a AuthenticationInfo.getName() ClientLoginModule.login()
property holds either at that point in time or ﬁnally (even- SecAssocActs.setPrincipalInfo()
tually) it holds. The operator ‘X’ speciﬁes that a property SecAssocActs.pushSubjectCtxt()
holds at the next point in time. The three examples listed in SimplePrincipal.toString()
Table 1 illustrate the meaning and use of these operations. SecAssoc.getPrincipal()
The set of LTL expressions minable by our mining SecAssoc.getPrincipal()
framework is represented in the Backus-Naur Form (BNF) SecAssoc.getCredential()
Figure 2. A Rule from JBoss-Security
rules := G(prepost)
prepost := event → post|event → XG(prepost) We have performed another case study to discover spec-
post := XF (event)|XF (event ∧ XF (post)) iﬁcations of the transaction component of JBoss-AS. Also,
a separate study on the integration of mined rules with a
To identify strongly observed rules, we introduce: (Sup- program veriﬁer for bug discovery has been performed. In-
port) The number of traces exhibiting the premise of the terested readers might refer to [27, 31] for details.
4 Mining Live Sequence Charts or when frequent patterns are long.
In , we extend the algorithm to mine LSCs. To
Scenarios, depicted using variants of sequence diagrams, demonstrate and evaluate our approach, we present the re-
are popular means to specify the inter-object behavior of sults of a case study we have conducted using traces from
systems (see, e.g., ). They are included in the UML various components of Jeti , an open-source Java based
standard, and are supported by many modeling tools. In full featured instant messaging application. The results
particular, we are interested in mining modal scenarios pre- demonstrate the effectiveness of our mining technique in re-
sented using a UML2 compliant variant of Live Sequence covering non-trivial and expressive underlying interactions.
Chart (LSC) [12, 21]. LSC is an extension of International One of the mined LSCs involving sending of mes-
Telecommunication Union (ITU) standard on Message Se- sages when one client starts communicating with another
quence Chart . Different from standard UML sequence is shown in Fig. 3. The scenario starts whenever a user uses
diagram, MSC and LSC describe constraints on traces. the roster tree to select a party to communicate with. Then,
We address the mining problem in two steps. In , we the roster tree will initiate the chat and set up a chat win-
propose iterative pattern, extending work on sequential pat- dow. After several resources and identiﬁers of communi-
tern mining [1, 44, 41] and episode mining [33, 18]. An it- cating parties are obtained, eventually, an initial message is
erative pattern is not an LSC or MSC, however its semantics sent via the Backend/Connect/Output channel.
obeys the constraints of MSC and LSC. In , we extend
lsc Start chat
iterative pattern mining to mine LSC.
Iterative pattern is a series of events supported by a sig- 0: 0:Jeti 0:Chat 0:JID 1:JID 0: 0: 0:
Roster Windows Backend Connect Output
niﬁcant number of instances repeated within and across se- Tree
quences (or traces). Similar to sequential pattern mining, chat(..)
we consider a database of sequences rather than a single se- getResource()
quence. However, we also mine patterns occurring repeat- createThread()
edly within a sequence. This is similar in spirit to episode getMyJID()
mining, but we remove the restriction that related events getUser()
must happen closely together in a window. send(Packet)
The above differences are required for analyzing pro- send(Packet)
gram traces and inferring speciﬁcations or properties. Due
to loops and recursions, a trace can contain repeated occur- Figure 3. A Mined LSC: Start chat
rences of interesting patterns. Also, program properties are
often inferred from a set of traces instead of a single trace. 5 Related Work
Finally, important patterns for veriﬁcation, such as lock ac-
quire and release or stream open and close (c.f [45, 9]), of- For a complete description of related work, interested
ten have their events occur at some arbitrary distance away readers can refer to the related work sections of our past
from one another in a program trace. Hence, there is a need studies [29, 28, 27, 31, 30, 32]. In this section, only some
to ‘break’ the ‘window barrier’ in order to capture these pat- related studies will be highlighted.
terns of interest. Mining Automata. In , we extended the speciﬁca-
Iterative pattern obeys the following apriori property uti- tion miner described in . In another work, Whaley et
lized by depth-ﬁrst search sequential pattern miners (e.g., al. extract object-oriented component interface sequencing
FreeSpan  and PreﬁxSpan ) which states: constraints to form multiple ﬁnite state automatons .
If P is not frequent then P + +evs (where evs is a series Reiss et al. encode program execution traces as directed
of events) is also not frequent. acylic graphs to aid visualization and understanding of pro-
Due to possibly combinatorial number of frequent sub- grams . Arts et al. dynamically extract program mod-
sequences of a long pattern, it’s best to mine a closed set of els from Erlang programs as state graph models for model
patterns (c.f.,  & ). Closed pattern mining discov- checking and visualization . We believe that these and
ers patterns without any super-sequence having correspond- other similar miners can beneﬁt from our data-mining based
ing set of instances. In , we mine a closed set of iter- architecture of ﬁltering errors and clustering traces, with
ative patterns. A search space pruning strategy employed minimal changes.
by early identiﬁcation and pruning of non-closed patterns Mining LTL. Of the most relevance is the work on min-
is used to mine a closed set of iterative patterns efﬁciently. ing rule-based speciﬁcation [45, 42], where the rules have a
Our performance study on synthetic and real-world datasets similar semantics as our work  but are limited to two-
shows the success of our closed pattern miner: it runs with event rules (e.g., lock → unlock ). Their algorithms do
over an order of magnitude speedup (over mining a full set not scale for mining multi-event rules since they ﬁrst list
of frequent patterns) especially on low support thresholds all possible two-event rules and then check the signiﬁcance
of each rules. For rules of arbitrary lengths, the number of  W.-N. Chin, S.-C. Khoo, S. Qin, C. Popeea, and H. Nguyen. Verifying safety
possible rules is arbitrarily large. Our work generalizes their policies with size properties and alias controls. In ICSE, 2005.
 E. Clarke, O. Grumberg, and D. Peled. Model Checking. MIT Press, 1999.
work by mining a complete set of rules of arbitrary lengths  J. E. Cook and A. L. Wolf. Discovering models of software processes from
that satisfy given support and conﬁdence thresholds. To en- event-based data. ACM Trans. on Software Engineering and Methodology,
7(3):215–249, July 1998.
able efﬁcient mining, we devise a number of search space  W. Damm and D. Harel. LSCs: Breathing Life into Message Sequence
pruning strategies. Charts. J. on Formal Methods in System Design, 19(1):45–80, 2001.
 S. Deelstra, M. Sinnema, and J. Bosch. Experiences in software product
Mining LSCs. Several studies reverse engineers object families: Problems and issues during product derivation. In SPLC, 2004.
interactions from program traces and visualize them using  M. Dwyer, G. Avrunin, and J. Corbett. Patterns in property speciﬁcations for
ﬁnite-state veriﬁcation. In ICSE, 1999.
sequence diagrams (see, e.g., [15, 7]). These might seem  Eclipse Test and Performance Tools Platform. http://www.eclipse.org/tptp/.
similar to our work in . However, different from our  L. Erlikh. Leveraging legacy system dollars for e-business. IEEE IT Pro,
work, these studies simply represent the entire trace as a  GAO Report: Patriot Missile Defense – Software Problem Led to System
sequence diagram. On the other hand, we mine strongly Failure at Dhahran, Saudi.
observed LSCs expressed in the traces.  G. Garriga. Discovering unbounded episodes in sequential data. In PKDD,
 J. Han, J. Pei, B. Mortazavi-Asl, Q. Chen, U. Dayal, and M.-C. Hsu.
6 Conclusion & Future Work Freespan: Frequent pattern-projected sequential pattern mining. In KDD,
In this paper, we have proposed a new data mining ap-  D. Harel. From play-in scenarios to code: An achievable dream. IEEE Com-
puter, 34(1):53–60, 2001.
proach, namely speciﬁcation discovery, which is a process  D. Harel and S. Maoz. Assert and negate revisited: Modal semantics for uml
for automated extraction of software speciﬁcation from pro- sequence diagrams. Software and System Modeling, 2007.
 J. Hopcroft, R. Motwani, and J. Ullman. Introduction to Automata Thoery,
gram artifacts. Three separate threads of work mining dif- Language, and Computation. Addison Wesley, 2001.
ferent form of speciﬁcations (automata, LTL and LSC) from  M. Huth and M. Ryan. Logic in Computer Science. Cambridge, 2004.
 ITU-T. ITU-T Recommendation Z.120: Message Sequence Chart (MSC).
program execution traces have been brieﬂy described. Not 1999.
only can these speciﬁcation formalisms be easily under-  Jeti. Version 0.7.6 (Oct. 2006). http://jeti.sourceforge.net/.
 M. Lehman and L. Belady. Program Evolution - Processes of Software
stood by software developers, they are also formal enough Change. Academic Press, 1985.
to be veriﬁed by program veriﬁers. Hence, mining the above  D. Lo. A sound and complete speciﬁcation miner. In SIGPLAN PLDI Student
speciﬁcation formalisms can aid both program comprehen- Research Competition (2nd position). http://www.acm.org/src/winners.html,
sion and dependability.  D. Lo and S.-C. Khoo. QUARK: Empirical assessment of automaton-based
Our future research directions include: improving the speciﬁcation miners. In IEEE WCRE, 2006.
 D. Lo and S.-C. Khoo. SMArTIC: Toward building an accurate, robust and
scalability of the mining processes further, incorporating scalable speciﬁcation miner. In SIGSOFT FSE, 2006.
user-deﬁned constraints suitable for software developers,  D. Lo, S.-C. Khoo, and C. Liu. Efﬁcient mining of iterative patterns for
software speciﬁcation discovery. In KDD, 2007.
performing more case studies especially on more industrial  D. Lo, S.-C. Khoo, and C. Liu. Mining temporal rules from program execu-
systems, and exploring further intersection of techniques in tion traces. Int. Work. on Prog. Comprehension through Dynamic Analysis (to
data mining, software engineering and programming lan-  D. Lo, S. Maoz, and S.-C. Khoo. Mining modal scenario-based speciﬁca-
guage to improve speciﬁcation discovery process. tions from execution traces of reactive systems. In IEEE/SIGSOFT ASE (to
Acknowledgement. We would like to thank our co-authors  H. Mannila, H. Toivonen, and A. Verkamo. Discovery of frequent episodes in
event sequences. Data Mining and Knowledge Discovery, 1:259–289, 1997.
of some of our past studies referred to in this paper, namely,  Software Errors Cost U.S. Economy $59.5 Billion Annually.
Chao Liu and Shahar Maoz. Also we would like to thank http://www.nist.gov/public affairs/releases/n02-10.htm, 2002.
 J. Pei, J. Han, B. Mortazavi-Asl, H. Pinto, Q. Chen, U. Dayal, and M.-C. Hsu.
Glenn Ammons, Jiawei Han and David Harel for their ad- Preﬁxspan: Mining sequential patterns efﬁciently by preﬁx-projected pattern
vice on some of our past studies referred to in this paper. growth. In ICDE, 2001.
 A. V. Raman and J. D. Patrick. The sk-strings method for inferring pfsa.
In Proc. of the Work. on Automata Induction, Grammatical Inference and
References Language Acquisition at ICML, 1997.
 Red Hat Middleware, LLC. JBoss.com - JBoss application server.
 R. Agrawal and R. Srikant. Mining sequential patterns. In ICDE, 1995. http://www.jboss.org/products/jbossas.
 G. Ammons, R. Bodik, and J. R. Larus. Mining speciﬁcations. In SIGPLAN-  S. P. Reiss and M. Renieris. Encoding program executions. In ICSE, 2001.
SIGACT POPL, 2002.  T. Standish. An essay on software reuse. IEEE Trans. on Software Engineer-
 ARIANE 5 Failure - Full Report. ing, 5(10):494–497, 1984.
http://www.ima.umn.edu/∼arnold/disasters/ariane5rep.html, 1996.  The Apache Software Foundation. Jakarta Commons/Net.
 T. Arts and L. Fredlund. Trace analysis of erlang program. In Proc. of Erlang http://jakarta.apache.org/commons/net/.
 J. Wang and J. Han. BIDE: Efﬁcient mining of frequent closed sequences. In
 BEA: News Release: Gross Domestic Product.
 W. Weimer and G. Necula. Mining temporal speciﬁcations for error detection.
In TACAS, 2005.
Sep. 2007.  J. Whaley, M. Martin, and M. Lam. Automatic extraction of object oriented
 B. Boehm. Software Engineering Economics. Prentice-Hall, 1981. component interfaces. In ISSTA, 2002.
 L. C. Briand, Y. Labiche, and J. Leduc. Toward the reverse engineering of  X. Yan, J. Han, and R. Afhar. CloSpan: Mining closed sequential patterns in
uml sequence diagrams for distributed Java software. IEEE Trans. Software large datasets. In SDM, 2003.
Engineering, 32(9):642–663, 2006.  J. Yang, D. Evans, D. Bhardwaj, T. Bhat, and M.Das. Perracotta: Mining
 R. Capilla and J. Duenas. Light-weight product-lines for evolution and main- temporal API rules from imperfect traces. In ICSE, 2006.
tenance of web sites. In CSMR, 2003.