Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing Variability-Intensive Systems, tutorial SPLC 2017, part Ii

11 views

Published on

Tutorial presented at the 21st International Systems and Software Product Line Conference

Published in: Software
  • Be the first to comment

  • Be the first to like this

Testing Variability-Intensive Systems, tutorial SPLC 2017, part Ii

  1. 1. www.unamur.be Testing Variability-intensive Systems (con’t) SPLC ’17, Sevilla, Spain
  2. 2. 2 gilles.perrouin@unamur.be 2 Part I (Until 16h) : sample configurations to test Feature modelling Sampling techniques Discussion on the JHipster case Part II (16h30-18h) : select behaviours to test A (short) introduction to behaviour modelling Behaviour-driven test case selection Test concretisation Open Discussion (18h-18h30) Tutorial Agenda
  3. 3. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite [X. Devroey, PhD thesis, 2017]
  4. 4. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite VIS Behaviour Specification [X. Devroey, PhD thesis, 2017]
  5. 5. Menu System
 under
 testSpecification Test case Output Pass or fail?
  6. 6. Menu System
 under
 testSpecification Test case Output Pass or fail?
  7. 7. Specification… of behaviour?
  8. 8. Specification… of behaviour? 87 6 1 3 close change tea serveTea 2 pay open 5soda serveSoda
  9. 9. Specification… of behaviour? 87 6 1 3 close change tea serveTea 2 pay open 5soda serveSoda Insert coin
  10. 10. Specification… of behaviour? 87 6 1 3 close change tea serveTea 2 pay open 5soda serveSoda Return change
  11. 11. Specification… of behaviour? 87 6 1 3 close change tea serveTea 2 pay open 5soda serveSoda Choose soda
  12. 12. Specification… of behaviour? 87 6 1 3 close change tea serveTea 2 pay open 5soda serveSoda Serve soda
  13. 13. Specification… of behaviour? 87 6 1 3 close change tea serveTea 2 pay open 5soda serveSoda Open compartment
  14. 14. Assessing Correctness
  15. 15. Model Checking Model and Property
  16. 16. Model Checking Model and Property Model Checker
  17. 17. Model Checking Model and Property Model Checker
  18. 18. Model Checking Model and Property Model Checker
  19. 19. Model Checking Model and Property Model Checker + counterexample
  20. 20. Variability-Intensive Behaviour Specification Optional feature Choose one or both VendingMachine v Tea t Free f Cancel c Soda s Beverages b
  21. 21. Variability-Intensive Behaviour Specification Optional feature Choose one or both VendingMachine v Tea t Free f Cancel c Soda s Beverages b • Feature model 
 denotes products • This one has 12
  22. 22. Variability-Intensive Behaviour Model Checking
  23. 23. Variability-Intensive Behaviour Model Checking =
  24. 24. Variability-Intensive Behaviour Model Checking == + counterexample + counterexample
  25. 25. Variability-Intensive Behaviour Model Checking • Too many to model individually • Too many to check individually ...
  26. 26. Variability-Intensive Behaviour Model Checking • Too many to model individually • Too many to check individually ... Lots of products built from the same assets
  27. 27. Variability-Intensive Behaviour Specification soda serveSodafree (d) Distributing soda for free skip pay soda serveSoda open tea serveTea (b) Selling tea and soda close change pay soda serveSoda open (a) Basic vending machine close change pay soda serveSoda open cancel return (c) With a cancel purchase function close change ... Instead of 12 transition systems: 
 
 
 
 
 
 
 VendingMachine v Tea t Free f Cancel c Soda s Beverages b [Classen et al., TSE ’13] [A. Classen, PhD thesis, 2011] [M. Cordy, PhD thesis, 2014]
  28. 28. Variability-Intensive Behaviour Specification soda serveSodafree (d) Distributing soda for free skip pay soda serveSoda open tea serveTea (b) Selling tea and soda close change pay soda serveSoda open (a) Basic vending machine close change pay soda serveSoda open cancel return (c) With a cancel purchase function close change ... Instead of 12 transition systems: 
 
 
 
 
 
 
 VendingMachine v Tea t Free f Cancel c Soda s Beverages b 87 6 5 1 pay/¬f 3 open/¬f cancel/creturn/c close/¬f change/¬f free/f soda/s serveSoda/s tea/t skip/f 2 4 serveTea/t + companion feature model: Write 1 featured transition system:
 [Classen et al., TSE ’13] [A. Classen, PhD thesis, 2011] [M. Cordy, PhD thesis, 2014]
  29. 29. Variability-Intensive Behaviour Specification soda serveSodafree (d) Distributing soda for free skip pay soda serveSoda open tea serveTea (b) Selling tea and soda close change pay soda serveSoda open (a) Basic vending machine close change pay soda serveSoda open cancel return (c) With a cancel purchase function close change ... Instead of 12 transition systems: 
 
 
 
 
 
 
 VendingMachine v Tea t Free f Cancel c Soda s Beverages b feature expressions 87 6 5 1 pay/¬f 3 open/¬f cancel/creturn/c close/¬f change/¬f free/f soda/s serveSoda/s tea/t skip/f 2 4 serveTea/t + companion feature model: Write 1 featured transition system:
 [Classen et al., TSE ’13] [A. Classen, PhD thesis, 2011] [M. Cordy, PhD thesis, 2014]
  30. 30. Exploring the state space of an FTS initial state, reachable by all products 87 6 5 1 3 cancel / creturn / c close / v change / v free / f soda / s serveSoda / s tea / t take / f 2 4 serveTea / t pay / ¬f open / ¬f 9 take / v
  31. 31. Exploring the state space of an FTS initial state, reachable by all products Transitions: 87 6 5 1 3 cancel / creturn / c close / v change / v free / f soda / s serveSoda / s tea / t take / f 2 4 serveTea / t pay / ¬f open / ¬f 9 take / v ➁ reachable by prods not having f.1 pay/¬f 2
  32. 32. Exploring the state space of an FTS initial state, reachable by all products 2 change/v 3 ➂ reachable by prods having v and not having f. Transitions: 87 6 5 1 3 cancel / creturn / c close / v change / v free / f soda / s serveSoda / s tea / t take / f 2 4 serveTea / t pay / ¬f open / ¬f 9 take / v ➁ reachable by prods not having f.1 pay/¬f 2
  33. 33. Exploring the state space of an FTS initial state, reachable by all products 2 change/v 3 ➂ reachable by prods having v and not having f. 3 soda/s 5 ➄ reachable by prods having v, s and not having f.... Transitions: 87 6 5 1 3 cancel / creturn / c close / v change / v free / f soda / s serveSoda / s tea / t take / f 2 4 serveTea / t pay / ¬f open / ¬f 9 take / v ➁ reachable by prods not having f.1 pay/¬f 2
  34. 34. Exploring the state space of an FTS initial state, reachable by all products 2 change/v 3 ➂ reachable by prods having v and not having f. 1 free/f 3 ➂ reachable by prods having f (or having v and not having f). 3 soda/s 5 ➄ reachable by prods having v, s and not having f.... Transitions: 87 6 5 1 3 cancel / creturn / c close / v change / v free / f soda / s serveSoda / s tea / t take / f 2 4 serveTea / t pay / ¬f open / ¬f 9 take / v ➁ reachable by prods not having f.1 pay/¬f 2
  35. 35. Exploring the state space of an FTS initial state, reachable by all products 2 change/v 3 ➂ reachable by prods having v and not having f. 1 free/f 3 ➂ reachable by prods having f (or having v and not having f). 3 soda/s 5 ➄ reachable by prods having v, s and not having f.... 3 soda/s 5 ➄ reachable by prods having f, s (or having v, s and not having f). ... Transitions: 87 6 5 1 3 cancel / creturn / c close / v change / v free / f soda / s serveSoda / s tea / t take / f 2 4 serveTea / t pay / ¬f open / ¬f 9 take / v ➁ reachable by prods not having f.1 pay/¬f 2
  36. 36. 87 6 5 1 pay/¬f 3 open/¬f cancel/creturn/c close/¬f change/¬f free/f soda/s serveSoda/s tea/t skip/f 2 4 serveTea/t Exploring the state space of an FTS = = = =
  37. 37. 87 6 5 1 pay/¬f 3 open/¬f cancel/creturn/c close/¬f change/¬f free/f soda/s serveSoda/s tea/t skip/f 2 4 serveTea/t Exploring the state space of an FTS = = = =
  38. 38. 87 6 5 1 pay/¬f 3 open/¬f cancel/creturn/c close/¬f change/¬f free/f soda/s serveSoda/s tea/t skip/f 2 4 serveTea/t Exploring the state space of an FTS = = = =
  39. 39. 87 6 5 1 pay/¬f 3 open/¬f cancel/creturn/c close/¬f change/¬f free/f soda/s serveSoda/s tea/t skip/f 2 4 serveTea/t Exploring the state space of an FTS = = = =
  40. 40. 87 6 5 1 pay/¬f 3 open/¬f cancel/creturn/c close/¬f change/¬f free/f soda/s serveSoda/s tea/t skip/f 2 4 serveTea/t Exploring the state space of an FTS = = = =
  41. 41. 87 6 5 1 pay/¬f 3 open/¬f cancel/creturn/c close/¬f change/¬f free/f soda/s serveSoda/s tea/t skip/f 2 4 serveTea/t Exploring the state space of an FTS = = = =
  42. 42. 87 6 5 1 pay/¬f 3 open/¬f cancel/creturn/c close/¬f change/¬f free/f soda/s serveSoda/s tea/t skip/f 2 4 serveTea/t Exploring the state space of an FTS = = = =
  43. 43. How much worth is it? (excerpt) Small SPL, 4 products 0s 8s 17s 25s 33s 42s 50s P1.1 P1.2 P1.3 P1.4 P1.5 P1.6 Medium SPL, 64 products 0s 25s 50s 75s 100s 125s 150s P1.1 P1.2 P1.3 P1.4 P1.5 P1.6 x TS 1 FTS (lower is better)
  44. 44. Getting Featured Transition Systems • Forward engineering: draw your (UML, Simulink …) models and automatically derive the FTS • Reverse engineering: build back the FTS from previous, logged executions of your systems
  45. 45. Forward engineering of FTS case study C++ code with IFDEFs Informal statecharts + variability out of sync model checking ProVeLines instance Visual statecharts + variability Formal -based DSL for statecharts + variability
  46. 46. https://github.com/xdevroey/yami Reverse engineering of FTS
  47. 47. LOG
  48. 48. LOG Karma_theme Facebook_plugin Wpstorecart_plugin Toolbox_theme Gtw_plugin Clockstone_theme Deep_blue_theme Max_theme Avenue_theme Admin_user Wp_sed_plugin Wp_bing_plugin Hotfix_plugin Mini_mail_dashboard_widget_plugin Theme_plugin Google_calendar_widget_plugin Zingiri_web_shop_plugin U_design_theme Purevision_theme Akismet_plugin Mm_forms_community_plugin Twentyten_theme OptimizePress_theme Plugins Thecartpress_plugin Sportpress_theme Braille_plugin Premium_Gallery_Manager_plugin Themes Lazy_seo_plugin Reflex_gallery_plugin Maintheme_theme Tell_a_friend_plugin Kiddo_theme a1_flash_gallery_plugin Yoko_theme Wp_404_plugin Formidable_plugin News_plugin_plugin Wp_mailinglist_plugin AgeRRNV_ageRRNV_splot_xml Likebtn_like_button_plugin Twentytwelve_theme Jetpack_plugin Twentyeleven_theme Legend: Optional Or FTS
  49. 49. 66.155.40.250 - - [08/Nov/2013:11:38:11 +0100] 
 "GET /index.php?p=potins&action=aimepas HTTP/1.1" 192.168.0.10 - - [08/Nov/2013:11:38:42 +0100] 
 "GET /index.php?p=potins&action=back HTTP/1.1"
 66.155.40.250 - - [08/Nov/2013:11:39:01 +0100] 
 "GET /index.php?p=accueil HTTP/1.1"
  50. 50. 66.155.40.250 - - [08/Nov/2013:11:38:11 +0100] 
 "GET /index.php?p=potins&action=aimepas HTTP/1.1" 192.168.0.10 - - [08/Nov/2013:11:38:42 +0100] 
 "GET /index.php?p=potins&action=back HTTP/1.1"
 66.155.40.250 - - [08/Nov/2013:11:39:01 +0100] 
 "GET /index.php?p=accueil HTTP/1.1" Session Session
  51. 51. 66.155.40.250 - - [08/Nov/2013:11:38:11 +0100] 
 "GET /index.php?p=potins&action=aimepas HTTP/1.1" 192.168.0.10 - - [08/Nov/2013:11:38:42 +0100] 
 "GET /index.php?p=potins&action=back HTTP/1.1"
 66.155.40.250 - - [08/Nov/2013:11:39:01 +0100] 
 "GET /index.php?p=accueil HTTP/1.1" Session Session 2gram TS
  52. 52. be.unamur.inference.main.Main
  53. 53. Bigram object used to build the TS be.unamur.inference.main.Main
  54. 54. Bigram object used to build the TS be.unamur.inference.main.Main
  55. 55. Pattern of the log entries in com.google.code.regexp format be.unamur.inference.main.Main
  56. 56. Pattern of the log entries in com.google.code.regexp format be.unamur.inference.main.Main
  57. 57. Use sessions to train the bigram be.unamur.inference.main.Main
  58. 58. Use sessions to train the bigram be.unamur.inference.main.Main
  59. 59. be.unamur.inference.main.Main Requests to include
  60. 60. be.unamur.inference.main.Main Requests to include
  61. 61. Requests to excludebe.unamur.inference.main.Main
  62. 62. Requests to excludebe.unamur.inference.main.Main
  63. 63. be.unamur.inference.main.Main
  64. 64. TS
  65. 65. TS PLUGIN_REGEXP = ".*/wp-content/plugins/([^/]+)/.*"; THEME_REGEXP = ".*/wp-content/themes/([^/]+)/.*"; ADMIN_REGEXP = ".*/wp-admin/.*";
  66. 66. TS PLUGIN_REGEXP = ".*/wp-content/plugins/([^/]+)/.*"; THEME_REGEXP = ".*/wp-content/themes/([^/]+)/.*"; ADMIN_REGEXP = ".*/wp-admin/.*"; FTS FM
  67. 67. How to define and select test cases for an SPL, 
 based on its behaviour?
  68. 68. Model-Based Testing (MBT) process Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite
  69. 69. Model-Based Testing (MBT) process Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite
  70. 70. Model-Based Testing (MBT) process Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  71. 71. Model-Based Testing (MBT) process Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  72. 72. Model-Based Testing (MBT) process Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case
  73. 73. Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite Model-Based Testing (MBT) process CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case
  74. 74. Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite Model-Based Testing (MBT) process CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case
  75. 75. Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite Model-Based Testing (MBT) process CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case insert_card
  76. 76. Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite Model-Based Testing (MBT) process CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case insert_card
  77. 77. Test suite selection tool Model Test scripts Mapping info. Concretizer Test execution environment System under test adaptor Abstract test suite Model-Based Testing (MBT) process CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card TestTestTestTestTestTestTestTestTestTest case Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case insert_card
  78. 78. Behavioural MBT of SPLs process FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite [Devroey et al., ISoLA ‘12]
  79. 79. Behavioural MBT of SPLs process FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite Structural-driven [Devroey et al., ISoLA ‘14] [Devroey et al., SPLAT@SPLC ‘14] [Devroey et al., ISoLA ‘12] Usage-driven [Devroey et al., VaMoS ’14] [Devroey et al., SoSyM ’17] Dissimilarity-driven [Devroey et al., VaMoS ’16]
  80. 80. Behavioural MBT of SPLs process FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suiteFeatured Mutants Model (FMM) [Devroey et al., FSE ‘14] [Devroey et al., ICSE ‘16] [Devroey et al., ISoLA ‘12] Equivalent mutants detection [Devroey et al., ICST ’17]
  81. 81. Behavioural MBT of SPLs process FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite [Devroey et al., ISoLA ‘12] Jeremy Vanhecke
  82. 82. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite
  83. 83. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite
  84. 84. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  85. 85. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  86. 86. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  87. 87. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  88. 88. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  89. 89. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  90. 90. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  91. 91. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  92. 92. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  93. 93. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  94. 94. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  95. 95. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  96. 96. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  97. 97. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  98. 98. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  99. 99. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  100. 100. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  101. 101. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card
  102. 102. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  103. 103. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Featured Transition System (FTS) Feature 
 Model (FM)
  104. 104. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Featured Transition System (FTS) Feature 
 Model (FM) Feature
 expression
  105. 105. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite
  106. 106. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite
  107. 107. Structural-driven selection
  108. 108. (Positive abstract) test case Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card t = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card)
  109. 109. (Positive abstract) test case Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card t = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card)
  110. 110. (Positive abstract) test case Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card t = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. (dd ∨ cr) ∧ (on ∧ pin) ∧ (on) ∧ CNF( )
  111. 111. (Positive abstract) test case Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card t = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. (dd ∨ cr) ∧ (on ∧ pin) ∧ (on) ∧ CNF( )prod(fts, t) =
  112. 112. Test suite and structural coverage • Test suite • s = (t1, t2, …, tn) • Structural coverage criterion • Associates an FTS and a test suite to a real value between 0 and 1
  113. 113. State/all-states Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card coverage(fts,s) = states visited by s / number of states in fts
  114. 114. State/all-states Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card coverage(fts,s) = states visited by s / number of states in fts
  115. 115. State/all-states Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card coverage(fts,s) = states visited by s / number of states in fts
  116. 116. Structural coverage criteria • State/all-states • Action/all-actions • Transition/all-transitions • Transition-pair/all-transition-pairs • Path/all-paths
  117. 117. VIBeS
  118. 118. Test suite selection [Devroey et al., SPLAT@SPLC ‘14] size products test suite coverage
  119. 119. size products test suite coverage Test suite selection [Devroey et al., SPLAT@SPLC ‘14] Minimal test suite
  120. 120. size products test suite coverage Test suite selection [Devroey et al., SPLAT@SPLC ‘14] P-minimal test suite
  121. 121. Product prioritisation • P-coverage • Coverage reached by sp⊆s, the test cases from s executable by p t1 t2 t3 t4 t5 t6 t7 t8 p = prod(fts,{t1,t2,t4,t5})
  122. 122. Product prioritisation • P-coverage • Coverage reached by sp⊆s, the test cases from s executable by p • P-coverage upper bound • Product p able to execute
 sp⊆s with the best coverage t1 t2 t3 t4 t5 t6 t7 t8 p = prod(fts,{t1,t2,t4,t5}) t1 t2 t3 t4 t5 t6 t7 t8 p = prod(fts, {t3,t4,t5,t8}) with best coverage(fts, {t3,t4,t5,t8})
  123. 123. Dissimilarity-driven selection
  124. 124. Dissimilarity between t1 and t2 t1 = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) t2 = (insert_card, select_app, check_PIN_offline, go_offline, update_card_info, remove_card)
  125. 125. Dissimilarity between t1 and t2 t1 = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) t2 = (insert_card, select_app, check_PIN_offline, go_offline, update_card_info, remove_card)
  126. 126. Dissimilarity between t1 and t2 t1 = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) t2 = (insert_card, select_app, check_PIN_offline, go_offline, update_card_info, remove_card) Dissa
  127. 127. Dissimilarity between t1 and t2 t1 = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) t2 = (insert_card, select_app, check_PIN_offline, go_offline, update_card_info, remove_card)
  128. 128. Dissimilarity between t1 and t2 Dissp t1 = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) t2 = (insert_card, select_app, check_PIN_offline, go_offline, update_card_info, remove_card)
  129. 129. Dissimilarity between t1 and t2 t1 = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) t2 = (insert_card, select_app, check_PIN_offline, go_offline, update_card_info, remove_card) Dissa Dissp
  130. 130. Dissimilarity between t1 and t2 ⊗ t1 = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) t2 = (insert_card, select_app, check_PIN_offline, go_offline, update_card_info, remove_card) Dissa Dissp
  131. 131. Dissimilarity selection + k: the number of test cases + d: the duration [Devroey et al., VaMoS ’16]
  132. 132. (1+1) evolutionary algorithm
  133. 133. (1+1) evolutionary algorithm s initialized with random test cases
  134. 134. (1+1) evolutionary algorithm Iterates during d
  135. 135. (1+1) evolutionary algorithm Sorted using local or global distance
  136. 136. (1+1) evolutionary algorithm New s is built by replacing last element by a random test case
  137. 137. (1+1) evolutionary algorithm If the fitness value is higher, the new s is kept
  138. 138. VIBeS
  139. 139. Product prioritisation • s = (t1, …, tn) • t1 more dissimilar than t2 (local distance) • t1 more dissimilar than (t2, …, tn) (global distance) • P-coverage upper bound • Product(s) able to execute the longest prefix of s
  140. 140. Usage-driven selection
  141. 141. Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0
  142. 142. Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0
  143. 143. Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0 Usage model
  144. 144. Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0
  145. 145. Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0 t = (insert_card, select_app, check_PIN_online, go_online, update_card_info, remove_card) Pr = 0.336
  146. 146. Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0 [Devroey et al., VaMoS ’14, SoSyM ’17]
  147. 147. Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0 Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case Selection Pr[min, max] [Devroey et al., VaMoS ’14, SoSyM ’17]
  148. 148. VIBeS
  149. 149. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Pr = 0.336
  150. 150. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Pr = 0.336
  151. 151. CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig PaymentSchema ps Connectivity c [1..*] opt. Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card Pr = 0.336
  152. 152. VIBeS toolbox DSL selection transformation execution mutation core fexpression <<uses>>
  153. 153. VIBeS toolbox DSL selection transformation execution mutation core fexpression <<uses>> Model definition public class CPTerminalFts extends FeaturedTransitionSystemDefinition { @Override protected void define() { initial("i"); from("i").action("insert_card") .to("CardIn"); from("cardIn").action("init_schema") .fexpr("dd || cr").to("App_init"); …}} Structural coverage selection FeaturedTransitionSystem fts = new CPTerminalFts().getTransitionSystem(); SolverFacade solver = new Sat4JSolverFacade(dimacsFile); TestSet set = allStatesSelection(fts, solver); Dissimilarity selection ... set = from(fts, solver) .withLocalMaxDistance( ftsDissimilarity(solver,levenshtein())) .during(10000) //sec. .generate(20); //test-cases
  154. 154. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite
  155. 155. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite
  156. 156. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite insert_card
  157. 157. AbsCon: a QTaste plugin Jeremy Vanhecke https://github.com/modji-be/AbsCon
  158. 158. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) <realisation id="Google testing"> … <tests> <test> <action>start</action> <action>goHomePage</action> <assert>onHomePage</assert> <action>inputSearchString</action> <assert>searchResultsPrinted</assert> <action>clickLink</action> <assert>pageLoaded</assert> <action>exit</action> </test> ... </tests> </realisation>
  159. 159. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) from qtaste import * from UiMappings import * #Actions definition def goHomePage(): googlePage.open() def inputSearchString(): searchBar.enterValue(testData.getValue("SEARCHVALUE")) ... #Asserts definition def searchResultsPrinted(): googlePage.waitForPage() if (not(navPicture.exists())): time.sleep(3) # wait for loading and retry return navPicture.exists() ... <realisation id="Google testing"> … <tests> <test> <action>start</action> <action>goHomePage</action> <assert>onHomePage</assert> <action>inputSearchString</action> <assert>searchResultsPrinted</assert> <action>clickLink</action> <assert>pageLoaded</assert> <action>exit</action> </test> ... </tests> </realisation>
  160. 160. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) from qtaste import * from UiMappings import * #Actions definition def goHomePage(): googlePage.open() def inputSearchString(): searchBar.enterValue(testData.getValue("SEARCHVALUE")) ... #Asserts definition def searchResultsPrinted(): googlePage.waitForPage() if (not(navPicture.exists())): time.sleep(3) # wait for loading and retry return navPicture.exists() ... <realisation id="Google testing"> … <tests> <test> <action>start</action> <action>goHomePage</action> <assert>onHomePage</assert> <action>inputSearchString</action> <assert>searchResultsPrinted</assert> <action>clickLink</action> <assert>pageLoaded</assert> <action>exit</action> </test> ... </tests> </realisation>
  161. 161. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) from qtaste import * from UiMappings import * #Actions definition def goHomePage(): googlePage.open() def inputSearchString(): searchBar.enterValue(testData.getValue("SEARCHVALUE")) ... #Asserts definition def searchResultsPrinted(): googlePage.waitForPage() if (not(navPicture.exists())): time.sleep(3) # wait for loading and retry return navPicture.exists() ... <realisation id="Google testing"> … <tests> <test> <action>start</action> <action>goHomePage</action> <assert>onHomePage</assert> <action>inputSearchString</action> <assert>searchResultsPrinted</assert> <action>clickLink</action> <assert>pageLoaded</assert> <action>exit</action> </test> ... </tests> </realisation>
  162. 162. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) from qtaste import * from UiMappings import * #Actions definition def goHomePage(): googlePage.open() def inputSearchString(): searchBar.enterValue(testData.getValue("SEARCHVALUE")) ... #Asserts definition def searchResultsPrinted(): googlePage.waitForPage() if (not(navPicture.exists())): time.sleep(3) # wait for loading and retry return navPicture.exists() ...
  163. 163. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) from qtaste import * from UiMappings import * #Actions definition def goHomePage(): googlePage.open() def inputSearchString(): searchBar.enterValue(testData.getValue("SEARCHVALUE")) ... #Asserts definition def searchResultsPrinted(): googlePage.waitForPage() if (not(navPicture.exists())): time.sleep(3) # wait for loading and retry return navPicture.exists() ...
  164. 164. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) from qtaste import * from UiMappings import * #Actions definition def goHomePage(): googlePage.open() def inputSearchString(): searchBar.enterValue(testData.getValue("SEARCHVALUE")) ... #Asserts definition def searchResultsPrinted(): googlePage.waitForPage() if (not(navPicture.exists())): time.sleep(3) # wait for loading and retry return navPicture.exists() ... SEARCHVALUE;LINKTOCLICK;BROWSER; unamur;UNamur;*firefox; unamur;UNamur;*chrome; unamur;UNamur;*opera; unamur;UNamur;*safari; ucl;UCL;*firefox; ucl;UCL;*chrome; ucl;UCL;*opera; ucl;UCL;*safari;
  165. 165. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv)
  166. 166. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) +
  167. 167. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv)
  168. 168. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv)
  169. 169. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) + exists(): boolean + getValue(): String + click() accessMethod accessValue WebPageElement + enterValue(String) WebEditBox WebButton WebLink WebRadio Button WebPictureWebText + start() + exit() WebBrowser api WebElement + open() + close() + getTitle(): String + waitForPage() address timeout WebPage *
  170. 170. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv) + exists(): boolean + getValue(): String + click() accessMethod accessValue WebPageElement + enterValue(String) WebEditBox WebButton WebLink WebRadio Button WebPictureWebText + start() + exit() WebBrowser api WebElement + open() + close() + getTitle(): String + waitForPage() address timeout WebPage *
  171. 171. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv)
  172. 172. Assertions and actions Verifications and operations (Operations.py) maps Interface model inst. (UiMapping.py) QTaste data mapping mechanism uses Test API SUT's interface CSV (TestData.csv)
  173. 173. Wrap up
  174. 174. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite
  175. 175. FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite
  176. 176. Dissimilarity selection ⊗ Jaccardp Hamminga × avg Dissp Dissa Jaccarda Dicea Antidicea Levenshteina [1..*] opt. [1..1] Product ∧ Action ⇒ ⊗ Dissimilarity Sort (1+1) EA Local Global + k: the number of test cases + d: the duration FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite
  177. 177. Dissimilarity selection ⊗ Jaccardp Hamminga × avg Dissp Dissa Jaccarda Dicea Antidicea Levenshteina [1..*] opt. [1..1] Product ∧ Action ⇒ ⊗ Dissimilarity Sort (1+1) EA Local Global + k: the number of test cases + d: the duration FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0 Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case Selection
  178. 178. Dissimilarity selection ⊗ Jaccardp Hamminga × avg Dissp Dissa Jaccarda Dicea Antidicea Levenshteina [1..*] opt. [1..1] Product ∧ Action ⇒ ⊗ Dissimilarity Sort (1+1) EA Local Global + k: the number of test cases + d: the duration FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0 Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case Selection https://github.com/xdevroey/yami
  179. 179. Dissimilarity selection ⊗ Jaccardp Hamminga × avg Dissp Dissa Jaccarda Dicea Antidicea Levenshteina [1..*] opt. [1..1] Product ∧ Action ⇒ ⊗ Dissimilarity Sort (1+1) EA Local Global + k: the number of test cases + d: the duration FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0 Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case Selection VIBeS toolbox DSL selection transformation execution mutation core fexpression <<uses>> Model definition public class CPTerminalFts extends FeaturedTransitionSystemDefinition { @Override protected void define() { initial("i"); from("i").action("insert_card") .to("CardIn"); from("cardIn").action("init_schema") .fexpr("dd || cr").to("App_init"); …}} Structural coverage selection FeaturedTransitionSystem fts = new CPTerminalFts().getTransitionSystem(); SolverFacade solver = new Sat4JSolverFacade(dimacsFile); TestSet set = allStatesSelection(fts, solver); Dissimilarity selection ... set = from(fts, solver) .withLocalMaxDistance( ftsDissimilarity(solver,levenshtein())) .during(10000) //sec. .generate(20); //test-cases https://github.com/xdevroey/yami
  180. 180. Dissimilarity selection ⊗ Jaccardp Hamminga × avg Dissp Dissa Jaccarda Dicea Antidicea Levenshteina [1..*] opt. [1..1] Product ∧ Action ⇒ ⊗ Dissimilarity Sort (1+1) EA Local Global + k: the number of test cases + d: the duration FTS model VIBeS Abstract test suite selection Products prioritization Mutation analysis Test scripts Mapping info. AbsCon Product config. Product adaptor derivation Feature model + Abstract test suite Init Card_in Aborted insert_card/ 1.0 abort/ 0.2 remove_card / 1.0 select_app / 0.8 App_init CH_verified check_PIN_online / 0.6 check_PIN _offline / 0.3 check_ signature / 0.1 Go_online Go_offline no_go / 0.1 go_offline / 0.2go_online / 0.7 Completed update_card_info / 1.0 update_card_info / 1.0 remove_card / 1.0 Init Card_in Aborted insert_card abort remove_card select_app / dd ∨ cr App_init CH_verified check_PIN_online / on ∧ pin check_PIN _offline / off ∧ pin check_ signature / ¬dd ∧ sig Go_online Go_offline no_go go_offline / offgo_online / on Completed update_card_info update_card_info remove_card CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest caseTest case Test case Selection VIBeS toolbox DSL selection transformation execution mutation core fexpression <<uses>> Model definition public class CPTerminalFts extends FeaturedTransitionSystemDefinition { @Override protected void define() { initial("i"); from("i").action("insert_card") .to("CardIn"); from("cardIn").action("init_schema") .fexpr("dd || cr").to("App_init"); …}} Structural coverage selection FeaturedTransitionSystem fts = new CPTerminalFts().getTransitionSystem(); SolverFacade solver = new Sat4JSolverFacade(dimacsFile); TestSet set = allStatesSelection(fts, solver); Dissimilarity selection ... set = from(fts, solver) .withLocalMaxDistance( ftsDissimilarity(solver,levenshtein())) .during(10000) //sec. .generate(20); //test-cases AbsCon: a QTaste plugin Jeremy Vanhecke https://github.com/xdevroey/yami
  181. 181. VIBeS https://github.com/xdevroey/vibes
  182. 182. https://amost2018.wordpress.com 14th Workshop on Advances 
 in Model Based Testing 
 @ICST 2018, Västerås, Sweden Paper submission: Jan. 12th, 2018 Workshop date: Apr. 13th, 2018

×