Teste combinatório de software
Upcoming SlideShare
Loading in...5
×
 

Teste combinatório de software

on

  • 675 views

Teste de software visa a detecção de erros (bugs) em software.Para aplicar o método de Testes Combinatórios, se cria uma bateria de testes em que através da especificação de valores para cada ...

Teste de software visa a detecção de erros (bugs) em software.Para aplicar o método de Testes Combinatórios, se cria uma bateria de testes em que através da especificação de valores para cada parâmetro de um programa ou aplicação, se garante que um grande número de combinações de valores de parâmetros sejam testados. Este é um método bastante eficaz de detecção de erros, já que foi verificado experimentalmente que a maioria dos erros de software são ocasionados pela interação entre alguns poucos parâmetros. Matrizes de cobertura têm sido amplamente estudadas e garantem vários níveis de cobertura de interações entre parâmetros ao mesmo tempo que minimizam o número de testes necessários. Nesta palestra daremos um apanhado geral da área de testes combinatórios e de construção de matrizes de cobertura.

Statistics

Views

Total Views
675
Views on SlideShare
675
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Teste combinatório de software Teste combinatório de software Presentation Transcript

  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Teste Combinat´rio de Software o Lucia Moura School of Electrical Engineering and Computer Science University of Ottawa lucia@eecs.uottawa.ca Palestra no SECCOM, INE, UFSC, 2012Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Trajet´ria geogr´fica e acadˆmica o a e Para ver todos os detalhes exibidos na tela, use o link Imprimir ao lado do mapa. Professora (2000-...) EECS, University of Ottawa, Ottawa, Canada PhD (1999) DCC, University of Toronto, Toronto, Canada Graduação e mestrado (1987, 1992) IME, USP, São Paulo, Brasil Imagens ©2012 TerraMetrics, Dados cartográficos ©2012 Google, INEGI, MapLink, Tele Atlas -Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Ottawa, Canad´ a To see all the details that are visible on the screen, use the Print link next to the map. EECS University of Ottawa Imagery ©2012 Cnes/Spot Image, DigitalGlobe, GeoEye, Map data ©2012 Google -Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Teste Combinat´rio de Software o Queremos testar um sistema: um programa um circuito um pacote que integra v´rios softwares a v´rias plataformas diferentes em que um pacote tenha que a funcionar um software altamente configur´vel a uma gui interface uma aplica¸˜o na “cloud” ca Queremos uma bateria de testes que dˆ uma boa cobertura do e sistema para detectar erros/bugs/falhas.Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Teste Combinat´rio de Software o Primeiro isolamos os parˆmetros do sistema e poss´ a ıveis valores os parˆmetros de entrada de um programa e seus valores a 5, 10, 20, 25, 30 <100,000 Introduction [100,000-250,000) 6 [250,000-450,000) > 450,000 17 opções Example 1.1.2. For our second example, we look at the circuit diagram in11 opções 1.1. Figure This is a small circuit, with only five inputs - a, b, c, d, and e - each of one bit, and three outputs. This circuit could be exhaustively tested using 25 = 32 tests, but based 6 opções on the fact that some of the inputs do not interact, doing so is unnecessary. We instead show that we can fully test the circuit using only eight tests by using a variable strength (5, 4, 11, 17, 6) covering array. We represent the circuit as a hypergraph, with one vertex for each input, as entradas de um circuito: 5 entradas bin´rias and one hyperedge for each output, as shown in Figure 1.2. a a NAND x b XNOR y c AND d XOR z NOT (2, 2, 2, 2, 2) AND e Figure 1.1: An example of a small circuit with five inputs and three outputs. componentes de uma plataforma e suas configura¸˜es co Component Web Browser Operating Connection Printer System Type Config Config: Netscape(0) Windows(0) LAN(0) Local (0) IE(1)Figure 1.1, taking each input Figure 1.2: The hypergraph that arises from the circuit in Macintosh(1) PPP(1) Networked(1) as a vertex and each output as a hyperedge overOther(2) the inputs contributing to Linux(2) its value. ISDN(2) Screen(2) Table 3: Four components, each with 3 configurations We now give a variable strength covering array for the circuit that requires only eight (3, 3, 3, 3) different test cases, with the guarantee that for each output, every possible combinationTeste Combinat´rio de Test Case Browser o Software of inputs have been tested together in some test case. OS Connection Printer Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Teste combinat´rio de pares (pairwise testing) o Testando um sistema com k = 4 componentes cada um com v = 3 valores: Component Web Browser Operating Connection Printer System Type Config Config: Netscape(0) Windows(0) LAN(0) Local (0) IE(1) Macintosh(1) ComponentPPP(1) Networked(1) WebOther(2) Browser Linux(2) Operating Connection Screen(2) ISDN(2) Printer Table 3: Four components, each with 3Type System configurations Config 4 Testar todas as possibilidades: 3Windows(0) Connection PrinterLocal (0) Config: Netscape(0) = 81 testes. Networked(1) Test Case Browser Macintosh(1) IE(1) OS LAN(0) PPP(1) 1 NetScape Linux(2) Windows LAN ISDN(2) Local Screen(2) Teste combinat´rio de pares de parˆmetros pode ser feito com 9 testes. o 2 Other(2) NetScape a Linux ISDN Networked Table 3: Four components, each with 3 configurations 3 NetScape Macintosh PPP Screen 4 IE Windows ISDN Screen Test Case Browser Macintosh Connection Networked 5 IE OS LAN Printer 16 NetScape Windows IE Linux LAN PPP Local Local 27 NetScape Windows Other Linux PPP ISDN Networked Networked 38 Other Linux NetScape Macintosh LAN PPP Screen Screen 49 Other IE Macintosh Windows ISDN ISDN Local Screen 5 Table 4: Test SuiteMacintosh Pairs from Table 3 IE to Cover all LAN Networked 6 IE Linux PPP Local 7 Other Windows PPP Networked agriculture and manufacturing [64]. It has entered the software testing community, appear- 8 Other Linux LAN Screen ing in practitioner’s guidebooks [70, 89], and provided in simple spreadsheet formats [40, 41]. The use of covering arrays in software testing was pioneered by Mandl Local Brownlie et 9 Other Macintosh ISDN [83] and al. [9, 102], and statistical foundations were explored in [46, 47, 48, 49, 82, 91]. Empirical Table 4: (examplo deinteractions in a software system Colbourn 2004) results indicate that testing Test Suite to Cover all Pairs from Table 3 indeed finds a of all pairwise large percentage of existing faults [45, 78]. Indeed, Burr et al. [11] provide more empirical agriculture and manufacturing [64]. It coverage leadsthe useful code coverage as well. Dalal results to show that this type of test has entered to software testing community, appear- Arranjo de cobertura com forca t = 2, k = 4 parametros, v = 3 valores ing et al. present empirical results to89], and provided in simple spreadsheet formats in a 41]. in practitioner’s guidebooks [70, argue that the testing of all pairwise interactions [40, software system finds a large percentage of the existing faults [45]. Dunietz et al. link the The use of covering arrays in software testing was pioneered by Mandl [83] and Brownlie et cada cobre todas as interacoes dois a dois com N = 9 tests. effectiveness of these methods to software code coverage. They show that high code block al. coverage is obtained when testing all two-way interactions, but higher48, 49, sizes 91]. needed [9, 102], and statistical foundations were explored in [46, 47, subset 82, are Empirical results good paththat testing of all pairwise interactions reports for three software systems. a for indicate coverage [54]. Kuhn et al. examined fault in a software system indeed findsTeste Combinat´rio de o Software Lucia Moura
  • shown to work across a variety of combinations of operating systems, databases, andIntrodu¸˜o network ca characteristics. Constru¸˜o de arranjos de cobertura ca Modelos generalizados Teste combinat´rio we hadpares operating system (Windows XP, Apple OS X, For example, suppose de an application that is intended to run on a variety of platforms comprised o five components: an (pairwise testing) of Red Hat Enterprise Linux), a browser (Internet Explorer, Firefox), protocol stack (IPv4, Arranjo ⋅processor (Intel,platforms.and a database tests, shownSybase, Oracle), possible of IPv6), a de cobertura: 3 ⋅ 2 ⋅ 2 ⋅ 2 3 = 72 possible AMD), With only 10 (MySQL, in Table 1, it is a total to for¸a every component5 parˆmetros, valores (3, 2, 2, least once, i.e., all possible test t = 2, k = interacting with every other component at 2, 3), N = 10 testes c a pairs of platform components are covered. Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHEL IE IPv6 AMD MySQL 8 RHEL Firefox IPv4 Intel Sybase 9 RHEL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle Table 1.taken from Khun, Kacker and Lei 2010) (example Pairwise test configurations testar todas as possibilidades (t = 5): 32 × 23 = 72 testes teste de pares (t = 2): 10 testes 7Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Teste combinat´rio de pares (pairwise testing) o Arranjo de cobertura: for¸a t = 2, k = 5 parˆmetros, valores (3, 2, 2, 2, 3), N = 10 testes c a (example taken from Khun, Kacker and Lei 2010) testar todas as possibilidades (t = 5): 32 × 23 = 72 testes teste de pares (t = 2): 10 testesTeste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Por que usar teste de pares (pairwise testing)? Economia: usamos um n´mero m´ u ınimo de testes. exemplo: k = 20 parˆmetros com v = 10 valores cada. a testar todas as combina¸˜es:1020 testes(em geral = v k ) co teste de pares:155 testes (em geral em O(v 2 log k)) Robustez: temos uma boa cobertura de erros na pr´tica. a a maior parte dos erros em software (75%-80%) s˜o causados a por certos valores ou por intera¸˜o de 2 valores. ca “Evaluating FDA recall class failures in medical devices... 98% showed that the problem could have been detected by testing the device with all pairs of parameter settings.” (Wallace and Kuhn, 2001) Cohen, Dalal, Fredman, Patton (1996) - AETG software Dalal, Karunanithi, Leaton, Patton, Horowicz (1999) Kuhn and Reilly (2002) cobertura de pares implicam outras medidas de cobertura. “Our initial trial of this was on a subset Nortel’s internal e-mail system where we able cover 97% of branches with less than 100 valid and invalid testcases, as opposed to 27 trillion exhaustive testcases.” (Burr and Young, 1998) “The block coverage obtained for [pairwise] was comparable with that achieved by exhaustively testing all factor combinations ...” (Dunietz et al., 1997) Cohen, Dalal, Fredman, Patton (1996, 1997) - AETG softwareTeste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Aumentando a for¸a da cobertura (t-way coverage) c podemos usar outros valores intermedi´rios de for¸a entre a c t = 2 (pairwise) e t = k (testar todo o espa¸o de parˆmetros). c a Defects Total o “tradeoff” ´ que aumentando o t, aumenta-se a robustez, e productivity more than doubling on average and more than tripling in found per hour 2.4X higher defects found 13% higher three projects. The groups using pair- mas tamb´m aumenta-se o n´mero de testes e u wise testing also achieved the same or higher quality in all 10 projects; Manual Pairwise Manual Pairwise estudos mostram que normalmente t ∈ [2, 6] ´ suficiente para all of the defects identified by the teams using manual test case selec- e (a) Testing method (b) Testing method tion methods were identified by the Figure 1. Summary of results from 10 projects. Pairwise combinatorial test case detectar todos os errors teams using combinatorial methods. In five projects, the combinatorial selection versus manual test case selection: (a) testing efficiency and (b) testing Kuhn, Wallace e Gallo (2004) quality. teams found additional defects that 100 had not been identified by the teams using manual methods. These proof-of-concept projects successfully demonstrated to the 75 teams involved that manual meth- ods of test case selection were not Cumulative percent nearly as effective as pairwise com- binatorial methods for finding the 50 largest number of defects in the least amount of time. Medical devices TESTING HIGHER-DEGREE Browser 25 Web server INTERACTIONS NASA distributed database Other empirical investigations have concluded that from 50 to 97 percent of software faults could be 0 1 2 3 4 5 6 identified by pairwise combinato- Interactions rial testing. However, what about the remaining faults? How many failures Figure 2. Cumulative error detection rate for fault-triggering conditions. Many faults could be triggered only by an unusual were caused by a single parameter value, a smaller proportion resulted from an interaction involving more than two interaction between two parameter values, and progressively fewer were triggered by three-, four-, five, and six -way interactions. parameters? In a 1999 study of faults arising from rare conditions, the National because it only guarantees that all by three-, four-, five,- and six-way Kuhn, Wallace e Gallo (2004) of parameter values will be interactions. Figure 2 summarizes Institute of Standards and Technology pairsTeste Combinat´rio de Software o reviewed 15 years of medical device tested. A particular four-way com- these results. Thus far, a fault trig- Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Teste combinat´rio de software, arranjos de cobertura o if it contains the minimum possible number of rows. Various authors transpose the array in Teste combinat´riocourse this isoccasionally constructpreference. In our discussions here, ca t the definition, and of o t-way requer of personal thede coberturaarray. for¸ we employ the N × k format, but a matter arranjos tranposed covering de for¸a teach. 3;has Nexample of a covering array =strength three with tenN = havinglinhas c = It isv = 13 rows. Here an = 2 s´ımbolos; k of 10 colunas; factors 13 two levels 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 0 0 0 1 1 0 1 1 0 1 0 1 0 0 1 0 0 0 1 1 1 0 0 0 0 1 1 0 0 1 0 0 1 0 0 0 1 0 1 0 1 1 1 0 1 1 0 1 0 0 1 0 1 0 0 0 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 0 0 0 0 0 1 1 1 0 1 0 0 0 1 1 1 0 1 Definicao: Arranjos objectCoberturain developing interaction tests when all factors This combinatorial de is fundamental have an equal number of levels. However, systems are typically not composed of components Um arranjo de cobertura the sameca t, of parameters v s´ (factors) that each have exactlyde for¸numberk fatores,(levels). To remove this fatorımbolos por limitation of covering arrays, the mixed-level covering array can be used. e tamanho N ,level covering array MCA (N;t,k, (v v),. .´, vumaanmatrixarray. × k com A mixed denotado CA(N ; t, k, , v , e )) is λ 1 .2 k N × k N Let {i , . . . , i } ⊆ {1, . . . , k}, and consider the subarray of size N ×t obtained by selecting columns 1 t s´ ımbolos. .deofumMCA. There arev-´rio G tal que em cada as rows, and an i ,. ,i 1 tthe alfabeto a v distinct t-tuples that could appear sub-arranjo t i=1 i t × N ,MCA requires smallest Nappear att leastcoberta pelo menos umav vez. )) cada the that eachem which suchonce. We coveringnotation CAN(t, k, (v , , . . . , v to denote t-tupla for G ´ a mixed use the array exists. e 1 2 k An early investigation of covering arrays appears implicitly in Marczewski [84]. R´nyi e o [105] determined sizes of covering arrays for the case t = v = 2 when N is even. KleitmanTeste Combinat´rio de Software Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Teste combinat´rio de software, arranjos de cobertura o Teste combinat´rio t-way requer arranjos de cobertura de for¸a t o c for¸a t = 3; v = 2 s´ c ımbolos; k = 10 colunas; N = 13 linhas Definicao: Arranjos de Cobertura Um arranjo de cobertura de for¸a t, k fatores, v s´ c ımbolos por fator e tamanho N , denotado CA(N ; t, k, v), ´ uma matrix N × k com e s´ ımbolos de um alfabeto v-´rio G tal que em cada sub-arranjo a t × N , cada t-tupla em Gt ´ coberta pelo menos uma vez. eTeste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Minimiza¸˜o de arranjos de cobertura ca Dados t (for¸a), k (n´mero de parˆmetros) and v (#valores). c u a Minimize N (#tests) CAN (t, k, v) = min{N : existe um CA(N ; t, k, v)}. Crescimento logar´ ımitico de arranjos de cobertura CAN (t = 2, k, v = 2) = {min N : N −1 ≥ k} = N/2 log k(1 + o(1)) (Katona 1973, Kleitman and Spencer 1973) t = 2, v > 2 fixed, k → ∞: CAN (t = 2, k, v) = v log k(1 + o(1)) 2 (Gargano, Korner and Vaccaro 1994) CAN (t, k, v = 2) ≤ 2t tO(log t) log k (Naor et al 1993,1996,1998) CAN (t, k, v) ≤ v t (t − 1) log k(1 + o(1)) (Godbole, Skipper and Sunley 1996)Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Minimiza¸˜o de arranjos de cobertura e crescimento ca logar´ ıtmo Dados t (for¸a), k (n´mero de componentes) e v (#valores). c u Minimize N (#testes) CAN (t, k, v) = min{N : existe um CA(N ; t, k, v)}. Para v e t fixos CAN (t, k, v) = O(log k). Use o m´todo guloso de densidade (Bryce & Colbourn 2007). e M´todo guloso de um-teste-por-vez garante N = O(log k). e ´ Otimo para teste de software: #testes cresce com o log do #parˆmetros! aTeste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Constru¸˜o de arranjos de cobertura m´ ca ınimos m´todos combinat´rios: recursivos e diretos e o Survey: Charlie Colbourn, “Combinatorial Aspects of Covering Arrays”, 2004 (34 paginas) algoritmos m´todos gulosos: e • AETG (D. Cohen, Dalal, Fredman, Patton 1996, 1997), um-teste-por-vez, tenta aproximar garantia logar´ ıtmica • m´todo guloso de densidade ( Bryce e Colbourn 2007), e um-teste-por-vez, garantia logar´ ıtmica • IPOG algoritm (J. Lei), ACTS tool/NIST (Khun e Kacker): alternˆncia de crescimento de linhas e colunas a m´todos de busca heur´ e ıstica • tabu search: Zekaoui (2006), Torres-Jimenez (2012) • simulated annealing: M. Cohen (2003-2008), Torres-Jimenez (2010-2012)Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Constru¸˜o de arranjos de cobertura ca M´todos pr´ticos, mais flex´ e a ıveis: m´todos gulosos (r´pidos, n´mero de testes n˜o ´ otimizado) e a u a e busca heur´ ıstica (mais lentos, resultados de melhor qualidade) M´todo para obter os melhores arranjos de cobertura e poss´ıvel: selecione os melhores resultados, usando combina¸˜o de: ca bons ingredientes (constru¸˜es diretas e buscas heur´ co ısticas) + e as melhores constru¸˜es combinat´rias recursivas co o Ver a tabela mantida por Colbourn com os melhores tamanhos de arranjos de cobertura conhecidos.Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Exemplo de bons ingredientes p/ usar em constru¸˜o ca recursiva arranjos ortogonais: CA(N = q 2 ; t = 2, k ≤ q + 1, v = q) (m´todo cl´ssico usando corpos finitos Fq ) e a 0000 2 3 6 0122 7 1220 6 7 6 7 2202 6 7 6 7 2021 6 7 6 7 6 6 0211 7 7 6 6 2110 7 7 4 1101 5 1012 (N otimo) metodo de LFSR para t = 3: CA(N = 2q 3 − 1; t = 3, k ≤ q 2 + q + 1, v = q) (Raaphorst, Moura, Stevens 2012) (N ´timo ou perto do ´timo) o oTeste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Exemplo de uma boa constru¸˜o recursiva: Produtos ca neste exemplo: parˆmetro t = 2 a CA(4,3) 1 2 3 4 1 0 0 0 0 CA(12=4*3,3) 2 0 1 1 2 1 2 3 4 5 6 7 8 9 10 11 12 3 0 2 2 1 4 1 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 5 1 1 0 1 0 1 1 2 0 1 1 2 0 1 1 2 6 1 2 1 0 0 2 2 1 0 2 2 1 0 2 2 1 7 2 0 1 1 1 0 2 2 1 0 2 2 1 0 2 2 8 2 1 2 0 1 1 0 1 1 1 0 1 1 1 0 1 9 2 2 0 2 1 2 1 0 1 2 1 0 1 2 1 0 2 0 1 1 2 0 1 1 2 0 1 1 size=9 2 1 2 0 2 1 2 0 2 1 2 0 2 2 0 2 2 2 0 2 2 2 0 2 CA(3,3) with 3 disjoint rows: 0 0 0 0 1 1 1 1 2 2 2 2 0 0 0 0 2 2 2 2 1 1 1 1 0 0 0 1 1 1 1 0 0 0 0 2 2 2 2 1 1 1 1 1 1 1 2 2 2 2 0 0 0 0 2 22 2 2 2 2 0 0 0 0 1 1 1 1 0 1 2 2 2 2 2 1 1 1 1 0 0 0 0 02 1 size=9+6=15 1 0 2 OD(3,3) 1 20 2 01 2 10 CA(N1,k1,g)+OD(N2,k2,g)= CA(N1+N2,k1*k2,g) size=6Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Generaliza¸oes de Arranjos de Cobertura c˜ Queremos constru¸˜es eficazes baseadas em teoria combinat´ria co o para lidar com restri¸˜es adicionais em teste de software: co N´mero de valores variado (v varia com a coluna: u v1 , v2 , . . . , vk ) Configura¸˜es proibidas co ACs evitando pares proibidos (CAFEs) e generaliza¸˜es co Varia¸˜es na for¸a co c ACs de for¸a vari´vel. c a Capacidade de determina¸˜o dos erros ca locating arrays, detecting arrays, error-locating arrays (ELAs) Outras generaliza¸˜es coTeste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados For example, suppose we had an application that is intended to run on a variety of platforms comprised of five components: an operating system (Windows XP, Apple OS X, N´mero de valores variados u Red Hat Enterprise Linux), a browser (Internet Explorer, Firefox), protocol stack (IPv4, IPv6), a processor (Intel, AMD), and a database (MySQL, Sybase, Oracle), a total of 3 ⋅ 2 ⋅ 2 ⋅ 2 ⋅ 3 = 72 possible platforms. With only 10 tests, shown in Table 1, it is possible to Mtest every component interacting= 5,every other 2, 2, 2, 3))least once, i.e., all possible CA(N = 10; t = 2, k with v = (3, component at pairs of platform components are covered. Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHEL IE IPv6 AMD MySQL 8 RHEL Firefox IPv4 Intel Sybase 9 RHEL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle Table 1. PairwiseKackerconfigurations (exemplo de Khun, test and Lei 2010) Constru¸˜es combinat´rias: co o Moura, Stardom, Stevens, Williams (2003) 7 Colbourn, Martirosyan, Mullen, Shasha, Sherwood, Yucas (2005)Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados N´mero de valores variados u M CA(N = 10; t = 2, k = 5, v = (3, 2, 2, 2, 3)) (exemplo de Khun, Kacker and Lei 2010) Constru¸˜es combinat´rias: co o Moura, Stardom, Stevens, Williams (2003) Colbourn, Martirosyan, Mullen, Shasha, Sherwood, Yucas (2005)Teste Combinat´rio de Software o Lucia Moura
  • the original configurations are underlined. Note that adding the constraint also resulted inIntrodu¸˜o ca reducing the number of Constru¸˜o de arranjos de coberturaThis will not always be Modelos generalizados ca test configurations by one. the case, depending on the constraints used, but it illustrates how constraints can help reduce the problem. Even if particular combinations are testable, the test team may consider some Configura¸oes proibidas/CAFEs c˜ combinations unnecessary, and constraints could be used to prevent these combinations, Internet explorer (IE) so podenumber of test configurations. proibidos: (OS X, IE) (RHL,IE) possibly reducing the ser usado com Windows (XP) Pares Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X Firefox IPv4 Intel Sybase 6 OS X Firefox IPv6 AMD Oracle 7 RHL Firefox IPv6 Intel MySQL 8 RHL Firefox IPv4 Intel Oracle 9 XP IE IPv4 AMD Sybase (exemplo de Khun, Kacker and Lei 2010) Figure 8. Test configurations for simple example with constraint. Modelo: grafo c/ 3+2+2+2+3 vertices, 2 arestas: {OS X, IE} {RHL,IE} Constru¸˜es Factors orias: 3.3.2 Cost co combinat´ • Danziger, Mendelsohn, Moura, Stevens (2008, is probably the most widely Using combinatorial methods to design test configurations 2009) • Maltais, Moura approach because it is quick and easy to do and typically delivers used combinatorial (2010) Aplica¸oes:improvements toattesting. costCombinatorial testing forand can parameters can significant c˜ provide better test coverage lower than conventional tests, input be extended to • Cohen, Dwyer, Shi (2007, much better assurance. high strength coverage to provide 2008) 3.4 Chapter SummaryTeste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados 1 1 1 0 0 0 1 0 For¸a vari´vel: AC em grafos c a 1 1 1 0 0 1 1 1 0 1 1 0 0 0 0 0 1 0 0 0 0 1 1 1 Testes cobrindo rela¸˜es de entrada-saida co (nota: a matriz est´ transposta) a Next, we transform into a test suite for ac- R cording to also cover the same combination for must step 5. . Since we I1 chose and its combination arbitrarily, we have proven B B I6 I2 that does cover all combinations of . 1 1 1 0 0 0 1 0 Y G The next theorem describes the relationship0 1 1 0 1 0 1 0 between I7 I8 at step 4 and 1 at step 5.1 1 0 0 0 1 0 I5 I3 R R Theorem 3.2 1 Suppose 0 1 1was 0 1 to0 0 us- reduced I4 ing steps 1 to1 of 1 algorithm. If 3 our 1 0 0 0and 1 are op- 0 B 1 1 0 1 timal test suites for and 0 1respectively then 0 0 . In addition, if then . Figure 1. The graph for the problem instance where 1 0 1 1 1 0 0 0 and , ,Proof. According to Theorem 3.1, 0 5 of our algorithm 1 0 0 0 step 1 1 1 , , , , . transforms into a test suite for whose size is (exemplo de Cheng, Dumitrescu, . Schroeder 2003) Hence, . On the other hand, when , can also be transformed into a test suite for . Let us now replace the param- Theorem 3.1in theis a test suite for each . That is, covers following manner. For , con- with their corresponding parameters all combinations of .each Colora¸˜o de grafos, homomorfismos de grafos ca eters in each in struct by setting equal to its value in . : , , That is, if we arrange the test cases in as columns in a , , array, and whose rows are indexed by , Proof.then the test suite for and is formed by considering only Let be a particular , , and • Meagher, tese de PhD (2005) . Applying reduction R1, combination indexed .by the input variables in . , let each denote the rows of For each Since us • Meagher, Stevens (2005), Meagher, Moura, Zekaoui (2007) by . The table below con- the color also in , this means that in rows Thus, is assigned to node the . in tains a test suite for . indexed by the input parameters in cover all the com- . Since covers test then we have a test suite there is • Cheng, Dumitrescu, Schroeder (2003), Cheng (2007) binations of . But , so the case created for that coverscoverscombination of , for also the all combinations 1 1 1 0 0 0 1 0 of .each . Consequently, the corresponding test case ofTeste Combinat´rio de Software 0 1 0 1 0 0 o 1 1 Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados For¸a vari´vel: AC em grafos c a Testes cobrindo rela¸˜es de entrada-saida co (nota: a matriz est´ transposta) a (exemplo de Cheng, Dumitrescu, Schroeder 2003) Colora¸˜o de grafos, homomorfismos de grafos ca • Meagher, tese de PhD (2005) • Meagher, Stevens (2005), Meagher, Moura, Zekaoui (2007) • Cheng, Dumitrescu, Schroeder (2003), Cheng (2007)Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados ACs de for¸a vari´vel c a VCAs = ACs em hipergrafos Se escolhe n´ ıveis de cobertura diferentes para diferentes conjuntos de parˆmetros. a • Cohen, Colbourn, Collofello, Gibbons, Mugridge (2003) • Raaphorst, Moura and Stevens (2011, 2012)Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Como localizar as intera¸oes que causam falha? c˜ Example: 0 0 0 0 F 0 1 1 0 F 0 2 2 2 P 1 0 2 1 P 1 1 0 2 P 1 2 1 0 P 2 0 1 2 P 2 1 2 0 P 2 2 0 1 P 0 0 1 1 P V´rios poss´ a ıveis conjuntos de intera¸˜es errˆneas ! co o Como construir um arranjo que nos indique as intera¸˜es que co causam erros?Teste Combinat´rio de Software o Lucia Moura
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Arranjos usados para localiza¸˜o de erros ca Yilmaz, Cohen and Porter (2006): arranjos de cobertura+ ´rvores de classifica¸˜o. a ca Colbourn and McClary (2008): locating arrays, detecting arrays. Martinez, Moura, Stevens and Panario (2010): error-locating arrays (ELAs) Chodoriwsky, Moura (2012): generalizacao de algoritmo adaptativo para t ≥ 3Teste Combinat´rio de Software o Lucia Moura
  • &(#%R3#$&.4$%6%>(."(%$6& A23/% &#$&.4+% $#,-."#% ),-./#,$% ),-./#% &(#% 9B% 6% &(#%Introdu¸˜o ca "23/% &#$&.4+5% >(1% /% "2 !""#$$% &% &(#% )*+#% ),-./#/% 01%˜o de "23/% &#$&.4+% -.,&3*2%7*"(.4#C%"2.#4&$%"44#"&%&(#%-.,&3*2%7*"(.4#%&(,3+(% &#$&.4+8% M(#,#% *,#% *2,#*/1% Constru¸a c &(#% arranjos de cobertura K8%AYZAVO@9YZ% Modelos generalizados ,#7&#% /#$D&)% *4/% &(#,% >*1$8% <3&% &(#% >*1% 6% 44=:;<% $#,-."#$5% &(#% #4&#,),.$#% /#$% &(#% 634"&.4*2% &#$&.4+% *4/% *))2."*&.4%$6&>*,#5%$&.22%4##/$%&%0#%.4$&*22#/%4%&(#%-.,&3*2% 6"3$%4%&(.$%*"&.-.&18%Y3,%> % % % M(.$% )*)#,% /#$",.0#/% 3,% ,#$#*,"(%6,%&(#%63&3,#%$6&>*,#%&#$&.4 )#,6,7*4"#% &#$&.4+8% 9&% .$% #*$1% &% *"(.#-#% 6,% &(#% 7*"(.4#% *4/% &(#4% .&% "*4% 0#% &#$&#/8% E.*+,*7% .$% *$% 622>% .4% &(#% *,#*% 6% "2 Outras generaliza¸˜es: aplica¸˜o em cloud computing co ca :;<=0*$#/%*))2."*&.4%$6&>*,#%),/3"&$%6%<?@%$&,3"&3,#8% &#$&.4+8% :#% .4&,/3"#/% &(#% /#6.4.&.4% 6% "23/% "7)3& FG.+8HIJ% *4/% &(#% /#6.4.&.4% 6% "23/% &#$&.4+8% !4/% &(#45% >#% *4*21L !A`ZY: :#%&(*4D%3,%"22#*+3 !"# # $%&(#)&*+,-.#/-01&2#32014.4(56# &(#%$&3/15%*4/%6,%&(#.,%"7 &(#%R3#$&.4$%6%>(."(%$6&>*,#%&#$&.4+%),]#"&$%"*4%/% A23/% &#$&.4+% $#,-."#% ),-./#,$% ),-./#% &(#% 9B% 6% &(#% X;G; "23/% &#$&.4+5% >(1% /% "23/$% &#$&.4+5% E8% Z3,7.5% &%:2$D.5% A8% [, (>% X8% /% "2 Jun &Meng (“Software Testing based on cloud computing” , 2011) -.,&3*2%7*"(.4#C%"2.#4&$%"44#"&%&(#%-.,&3*2%7*"(.4#%&(,3+(% aNb! &#$&.4+8% M(#,#% *,#% *2,#*/1% $#-#,*2% ),/3"&$% *-*.2*02#%.4%& c3$#665% *4/% E8% d*+, ,#7&#% /#$D&)% *4/% &(#,% >*1$8% <3&% &(#% >*1% 6% 44=:;<% "23/="7)3&.4+% $1$&#75f% discutem o modelo de usar cloud computing para teste de 6"3$%4%&(.$%*"&.-.&18%Y3,%>,D%.$%*.7#/%*&%/.4+%*%,#$#* i&(% 9;;;?!A% 94&#,4*&.4*2% *))2."*&.4%$6&>*,#5%$&.22%4##/$%&%0#%.4$&*22#/%4%&(#%-.,&3*2% &(#% [,./8% :*$(.4+&45% EA5% O 7*"(.4#% *4/% &(#4% .&% "*4% 0#% &#$&#/8% E.*+,*7% .$% *$% 622>% 6,%&(#%63&3,#%$6&>*,#%&#$&.4+%7#&(/$8% NQSjNHN8% #2*$&."% "7)3&#% aQb! !7*L4% software. FG.+8HIJ% % !A`ZY:V;E[;ZM% G.+8H%9**@%)2*&6,7%*,"(.&#"&3,#% % K.,&3*2.L*&.4%&#"(42+1% !-*.2*02#J(&&)J??*>$8*7*L48" aHb! !8% :#.$$5% eA7)3&.4+% .4% M(#%7*.4%),"#$$%6%"23/%&#$&.4+%*$%622>$J% ))8NUjQT5%Qhhk8% aSb! M88`.4+5% e!% $#26=&#$&.4+% *) :#%&(*4D%3,%"22#*+3#$%6,%&(#.,%&#"(4."*2%(#2)%/3, NI% O$#,$% 2+.4% "23/% &#$&.4+% ),-./#,P$% >#0$.&#% *4/% ,#+.$&#,%3$#,%.46,7*&.4C% /.$$#,&*&.45%G2,./*%94&#,4*&. Qhhi8% &(#%$&3/15%*4/%6,%&(#.,%"77#4&$%*4/%"4&.433$%$3)),& QI% O$#,$% *))21% 6,% &(#% &#$&% )2*&6,7% ,#$3,"#$5% aTb! ZE@O%E#)*,&7#4&%6%A7)3 [,3)5f%l*48%QhNh5%(&&)J??"$84/ *))2."*&.4$%4##/%&%/#$",.0#%&(#%"46.+3,*&.4%,#R3.,#7#4&$% aUb! MAVJ%(&&)J??&"28$3,"#6,+#84#& X;G;X;ZA;@% akb! l8% K*,.*8% FQhhi5% G#0,3*,1% Q 6% -.,&3*2% 7*"(.4#% #4-.,47#4&5% @3"(% *$% )#,*&.4+% $1$&#7% -#,$.45%0,>$#,%-#,$.45%7#7,1%$.L#5%(*,/%/.$D%$.L#5%(*,/% A23/8f% !7*L4 /.$D%$)##/5%4#&>,D%0*4/>./&(5%6.,#>*225%#&"8C% <2+8(&&)J??*>$8&1)#)*/8"7?* aNb!M(#% $#,-."#% ),-./#,% 6% "23/% &#$&.4+% ,#-.#>% &(#% [8% Y0#,&#22.5% @8% @7*4 HI% E8% Z3,7.5% X8% :2$D.5% A8% [,L#+,"L1D5% &(#="23/8(&72% c3$#665% *4/% E8% d*+,/4-5eM(#% #3"*21)&3$% )#4=$3 *))2."*&.4%*4/%"46.+3,#%&(#%&#$&%)2*&6,7C% SI% O4/#,% &(#% ,#4&*2% *+,##7#4&5% &(#% 3$#,% F42.4#I%ghiJ% B,"##/.4+$% 6% &(#% Q "23/="7)3&.4+% $1$&#75f% .4% AA[X9E% ),#=)*./%$#,-."#%6##C% % i&(% 9;;;?!A% 94&#,4*&.4*2% @17)$.37% 4% A23$&#,% A7)3&.4+% TI%O$#,$%2+.4%&(#%"23/%&#$&.4+%)2*&6,75%34&.2%&(#%#4/% &(#% [,./8% :*$(.4+&45% EA5% O@!J% 9;;;% A7)3&#,% @".#&15% Qhhi5 6%&(#%&#$&.4+C% NQSjNHN8% UI%!"",/.4+%&%&(#%62>%,%&.7#%&%)*1%&(#%*"&3*2%"$&$5% aQb! !7*L4% #2*$&."% "7)3&#% "23/% F!7*L4% ;AQI8% aY42. &(#%2#*$#%#4/$8% % !-*.2*02#J(&&)J??*>$8*7*L48"7?#"Q?8% 9K8%<O9VE%MW;%BX9K!M;%AVYOE%M;@M9Z[% Uma id´ia que proponho nesta palestra:aHb! !8% :#.$$5% eA7)3&.4+% .4% &(#% A23/$5f% 4#&:,D#,5% -28NN54 e G.+8H%9**@%)2*&6,7%*,"(.&#"&3,#% % K.,&3*2.L*&.4%&#"(42+1% M(#%7*.4%),"#$$%6%"23/%&#$&.4+%*$%622>$J% ))8NUjQT5%Qhhk8% BV!MGYX%YG%;ZM;XBX9@;% X#6#,#4"#% "23/% &#$&.4+% ./#*$5% #4&#,),.$#$% "*4% 03.2/% &(#.,% >4% .4&#,4*2% "23/% &#$&.4+% #4-.,47#4&% &% #4]1%6,% *3&47."% $6&>*,#5f% B( aSb! M88`.4+5% e!% $#26=&#$&.4+% *)),*"(% .&$% NI% O$#,$% 2+.4% "23/% &#$&.4+% ),-./#,P$% >#0$.&#% *4/% usu´rio usa arranjos de cobertura no passo 2 (pode ser invi´vel); ou a ,#+.$&#,%3$#,%.46,7*&.4C% a */-*4&*+#$8% A46./#4&.*2.&1% *4/% &(#,% ,#*$4$% 6,% &(#% /.$$#,&*&.45%G2,./*%94&#,4*&.4*2%O4.-#,$.&15%.*7.5%GV5%O@!5%!3 #^&#,4*2% 4#&>,D% "*4% 4&% *""#$$% &(#% $6&>*,#% X_E% Qhhi8% #4&#,),.$#%"23/%&#$&%7#&(/%"*4%*2$%"2."D%(#,#8% QI% O$#,$% *))21% 6,% &(#% &#$&% )2*&6,7% ,#$3,"#$5% aTb! ZE@O%E#)*,&7#4&%6%A7)3&#,%@".#4"#5%e@6&>*,#%M#$&.4+%X#$#* !3&7*&#/%"23/%&#$&.4+%#4-.,47#4&J%"($#%*%0#&&#,% sofisticar o passo 2: o usu´rio s´ fornece as componentes e a o *))2."*&.4$%4##/%&%/#$",.0#%&(#%"46.+3,*&.4%,#R3.,#7#4&$% [,3)5f%l*48%QhNh5%(&&)J??"$84/$38#/3?$&,+?8% )#,6,7*4"#%7*"(.4#%,%$#,-#,%&%.4$&*22%*3&7*&#/%&#$&.4+% $6&>*,#5%MAVJ%(&&)J??&"28$3,"#6,+#84#&?%V*$&%*""#$$#/J%*,"(%Nh5%QhNh8% aUb! 6,% #^*7)2#% V*/,34#,5% X0&% *4/% &(#,% &2$8% 6% -.,&3*2% 7*"(.4#% #4-.,47#4&5% @3"(% *$% )#,*&.4+% $1$&#7% e a for¸a desejada; o cloud configura¸˜es desejadas para o teste co c M#$&.4+% +,3)$% 2+% .4% &(.$% 7*"(.4#% &% /% &(#.,% >4% akb! l8% K*,.*8% FQhhi5% G#0,3*,1% QSI8% eK.,&3*2% @&,#$$=6,##% M#$&.4+% .4% 634"&.4*2%&#$&.4+%*4/%*3&7*&#/%&#$&.4+5%*4/%/%4&%4##/%&% -#,$.45%0,>$#,%-#,$.45%7#7,1%$.L#5%(*,/%/.$D%$.L#5%(*,/% .4$&*22%&(#%.4/#)#4/#4&%*3&7*&#/%&#$&.4+%#4-.,47#4&%.4%&(#% :#0% A23/8f% !7*L4% @#,- testing provider gera o arranjo de cobertura adequado /.$D%$)##/5%4#&>,D%0*4/>./&(5%6.,#>*225%#&"8C% <2+8(&&)J??*>$8&1)#)*/8"7?*>$?Qhhi?hQ?-.,&3*2=$&,#$$6,##=&#$&.4+ 2"*2%7*"(.4#$8% 32&.=7*"(.4#% #4-.,47#4&% &% 03.2/J% 3$.4+% -.,&3*2% &(#="23/8(&72% HI% M(#% $#,-."#% ),-./#,% 6%c˜o: &#$&.4+% ,#-.#>% &(#% das maquinas virtuais na cloud) (restri¸"23/% configura¸˜es a co 7*"(.4#% &#"(42+1% &% ")1% *4/% $&*,&% *% 4370#,% 6% -.,&3*2% *))2."*&.4%*4/%"46.+3,#%&(#%&#$&%)2*&6,7C% 7*"(.4#$%4%&(#%$#,-#,%R3."D215%&(#%&#$&%&#,7.4*2$%*""#$$%&% &(#.,% >4% -.,&3*2% 7*"(.4#$% &% 3)+,*/#% "3$&7#,% /*&*5% SI%o O4/#,% &(#% ,#4&*2% *+,##7#4&5% &(#% 3$#,% F42.4#I%Teste Combinat´rio de Software #^#"3&#%&(#%#4-.,47#4&%&#$&.4+5%$&,#$$%&#$&.4+5%)#,6,7*4"#% Lucia Moura &#$&.4+5% *4/% &(#,% $6&>*,#% &#$&.4+% 01% ,#7&#% /#$D&)% ,%
  • Introdu¸˜o ca Constru¸˜o de arranjos de cobertura ca Modelos generalizados Discuss˜o final a Teste combinat´rio de software ´ util e efetivo. o e Existem tools dispon´ ıveis para uso imediato em aplica¸˜es: co • ACTS do NIST (EUA) t ≤ 6 (open source, gratuito) • Hexawise: comercial t ≤ 6 (gratuito para uso acadˆmico, e nonprofit e compania com at´ 5 usu´rios, ≥$50.000/ano) e a • Testcover.com: gerador autom´tico (t = 2) na a cloud/software as a service (SaaS); (assinatura: $100/mˆs) e Existe pesquisa ativa na ´rea de algor´ a ıtmos e constru¸˜es co combinat´rias para otimizar o n´mero de testes em arranjos o u de cobertura. Temos tido algum sucesso em lidar com restri¸˜es adicionais. co Existe pesquisa ativa na ´rea de teste de software avaliando a a efetividade e adaptando teste combinat´rio de software aos o mais diversos tipos de aplica¸˜es. coTeste Combinat´rio de Software o Lucia Moura