Your SlideShare is downloading. ×
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Testing of C software components using Models

159
views

Published on

The presentation discusses experiences of applying model-based testing on a NASA C software component, which is used in several NASA Flight Software missions.

The presentation discusses experiences of applying model-based testing on a NASA C software component, which is used in several NASA Flight Software missions.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
159
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Automated Test Generation for Flight Software (FSW)Dharmalingam Ganesan David McComasMikael Lindvall Nicholas YanchikChristoph Schulze Alan CudmoreGunnar Cortes Steve Slegel Goddard  Space  Flight  Center   Code  582,  Flight  So>ware    Fraunhofer  Center  Maryland   Systems    Branch   ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering  
  • 2. Problems with Current Testing•  Manual testing takes a lot of effort•  Manual testing could miss “corner-cases”•  Test cases difficult to maintain and reuse on multiple environments•  Tests are technical, typically not domain oriented –  and thus difficult to understand for non-technical stakeholders ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   2  
  • 3. Model based testing (MBT)•  Solution: Generate Test Cases using MBT! –  We incrementally model the behavior of the software to be tested •  Good and “Evil” usages •  Expected results are embedded in the model –  Based on the model, the test generator automatically generates test cases –  Any type of test cases can be generated •  Junit, Xunit, soapUI, Selenium, CuTest, etc.•  Extra benefits –  We have a very visual representation of the system –  Test cases are domain oriented and easier to understand for non- technical stakeholders –  Documentation of behavior ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   3  
  • 4. Hello World to MBT ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   4  
  • 5. Terminology  Nodes of the model are called states  Edges of the model are called transitions  Each test case is a path from the start state to the exit state  A random test is generated using a random walk from the start state to the exit state  A test suite is a collection of test cases ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   5  
  • 6. Architecture of MBT Model   List  of   abstract   acRons   ITestMonkey   List  of  abstract   input  data   provider   TestMonkeyImpl   ITestDataProvider   methods   Interface   System  Under  Test   TestDataProviderImpl   Model  is  agnosRc  to  the  test  execuRon  technology     ITestMonkey  interface  hides  the  test  execuRon  framework   TestMonkeyImpl  uses  interfaces  of  the  test  execuRon  framework   ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   6  
  • 7. Workflow ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   7  
  • 8. Flight Software Under Test•  Operating System Abstraction Layer (OSAL)•  Isolates flight software from operating systems and hardware•  Implementations for the real time systems RTEMS and VxWorks and posix compliant non-real time systems•  Used for mission critical embedded systems•  Provides support for file-system, tasks, queues, semaphores, interrupts, hardware abstraction, I/O ports and exception handling ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   8  
  • 9. NASA OSAL•  Why is it important that OSAL be bug free? –  Flight software is mission critical and needs to be of very high quality –  OSAL is the foundation of the core Flight Executive (cFE) –  OSAL is used in many NASA missions, e.g. the Lunar Renaissance Orbit –  If OSAL has issues, it might result in catastrophic failure ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   9  
  • 10. NASA OSAL – ArchitectureFlight  So>ware   ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   10  
  • 11. Applying MBT on OSAL•  Model captures OSAL’s expected behavior•  Test monkey has abstract instructions e.g. “CreateFileValid”, “CreateFileInvalid”•  We created and used a dynamic data provider •  Generates file and folder names •  Generates content for files ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   11  
  • 12. Applying MBT on OSAL ...•  OSAL‘s file system API‘s were modeled•  Several test cases in C were automatically generated •  We used the open source CuTest test execution framework ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   12  
  • 13. Example of a OSAL model ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   13  
  • 14. API spec of make directory/*-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐          Name:  OS_mkdir            Purpose:  makes  a  directory  specified  by  path.            Returns:  OS_FS_ERR_INVALID_POINTER  if  path  is  NULL                            OS_FS_ERR_PATH_TOO_LONG  if  the  path  is  too  long  to  be  stored  locally                            OS_FS_ERR_PATH_INVALID  if  path  cannot  be  parsed                            OS_FS_ERROR  if  the  OS  call  fails                            OS_FS_SUCCESS  if  success            Note:  The  access  parameter  is  currently  unused.  -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐*/    int32  OS_mkdir  (const  char  *path,  uint32  access);     ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   14  
  • 15. Sub-model of make directory pointer  =  openDirectoryValid();  CuAssertTrue(tc,pointer  !=  NULL);   ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   15  
  • 16. Sample generated Test in CuTestvoid  Testosal_Filesystem_min_2(CuTest*  tc)    {              status  =  makeFilesystemValid();            CuAssertIntEquals_Msg(tc,"Filesystem  could  not  be  created",                                                                                              OS_FS_SUCCESS,  status);              status  =  mountFilesystemValid();            CuAssertIntEquals_Msg(tc,"Filesystem  could  not  be  mounted",                                                                                            OS_FS_SUCCESS,  status);                      pointer  =  openDirectoryValid();          CuAssertTrue(tc,  pointer  !=  NULL);          …          status  =  removeFilesystemValid();          CuAssertIntEquals_Msg(tc,"Filesystem  could  not  be  removed”,  status);  }   ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   16  
  • 17. Issues found using MBT•  OSAL  is  a  high-­‐quality  product  (only  a  few  issues  detected!)  •  Issues  found  when  running  model  based  tests  on  the  Posix  implementaRon   of  OSAL:  •  File-­‐descriptors  a>er  removing  file-­‐system:   •  A>er  somewhat  long  tests  we  would  run  out  of  file-­‐descriptors   •  This  would  even  happen  with  a  newly  created  file-­‐system   •  Cause:  OSAL  does  not  remove  file-­‐descriptors  when  the  file-­‐system  is  removed  •  Wrong  error  codes  returned  and  unimplemented  features:   Test  scenario   Error  message   Expected   Actual    Expected  invalid   checkFileSystemNullName()   pointer  error   OS_FS_ERR_INVALID_POINTER   OS_FS_UNIMPLEMENTED    Expected   checkFileSystemOsCallFails()   filesystem  error   OS_FS_ERROR   OS_FS_UNIMPLEMENTED    Filesystem  Not   checkFilesystemValid()   Checked   OS_FS_SUCCESS   OS_FS_UNIMPLEMENTED    Filesystem  error   OS_FS_ERR_NAME_TOO_ copyFileLongSourceFilename()   code  expected   OS_FS_ERROR   LONG    Filesystem  error   copyFileNonExisRngSourceFile()   code  expected   OS_FS_ERROR   OS_FS_SUCCESS    Filesystem  error   OS_FS_ERR_NAME_TOO_ renameFileLongSourceFilename()     code  expected   OS_FS_ERROR   LONG   ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   17  
  • 18. MBT – some limitations•  Modeling requires specification of SUT•  Developers are typically not used to modeling•  Difficult to document individual test cases –  Note: Models summarize all test cases –  Some customers require document of each test case ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   18  
  • 19. Summary and Future Work•  MBT works for API-based testing!•  Some detected issues are serious (e.g., running out of file descriptors)•  MBT requires manual work for –  constructing models –  monkey and data provider implementations –  maintenance of above artifacts•  However, test cases are automatically generated•  Future: –  Evaluating it on requirements of cFE/CFS –  ROI estimation ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   19  
  • 20. Acknowledgement•  Lisa Montgomery - NASA SARP for supporting our applied research•  Sally Godfrey – NASA GSFC•  Thank you! ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   20  
  • 21. Acronyms•  API: Application Program Interface•  cFE: Core Flight Executive•  CFR: Code of Federal Regulations•  CFS: Core Flight Software•  FSW: Flight Software•  MBT: Model-based Testing•  OSAL: Operating System Abstraction Layer•  ROI: Return on Investment•  SUT: System Under Test ©  2011  Fraunhofer  USA,  Inc.    Center  for  Experimental  So>ware  Engineering   21  

×