Architecture-based Unit Testing of    the Flight Software Product LineFraunhofer CESE                         NASA Goddard...
Background• Unit testing deals with testing individual  modules in isolation  – Assuming the correctness of dependent    m...
Unit Testing Challenges in Practice• Isolating the module under test from other  modules• Isolating from special OS and ha...
Research Questions• What architectural design decisions  facilitate or impede unit testing?• How can we make use of the ar...
Reusable Analysis Questions• Can modules be tested independently of other  modules they use?• Can modules be unit tested w...
Technical Set-up for Analysis                              Source Code               Makefiles                            ...
Product Line Structure of the CFS         CFS Application A           CFS Application B        CFS Application C        … ...
Run-time Structure of the CFS                                         House-                                   Memory     ...
Missions using CFS/cFE Mission     Managed by         cFE         CFS Apps                              StatusCHIPS       ...
Dependencies of Core Modules          cFE - Core                                         Executive Service (ES)           ...
Unit Test Architecture of CFS• Core idea is to   – Define mocks/stubs for dependent modules   – Systematically control ret...
How Return Code is Controlled?• UT_SetRtn_t is the key data structure:  –   typedef struct {        uint32 count;        u...
Controlling the Return Code - Exampleextern UT_SetRtn_t    SB_CreatePipeRtn;// mock implementation of create pipeint32 CFE...
Binding to Mock Modules                The test suite for                 the Executive                   es_ut.c         ...
Testability and Module Secretsint32 CFE_ES_LoadLibrary(char *EntryPoint,                            char *LibName, …) {  b...
Same Return Code Impede Testingint32 CFE_SB_SendMsg(CFE_SB_Msg_t *MsgPtr) { /* check input parameter */if(MsgPtr == NULL){...
Testability, Abstract Interfaces and     Alternative Implementations                                         Software Bus ...
Some DataCore Module     # of Functions in Interface      # Directly invoked in Unit TestsSB                              ...
Conclusion• Programming to abstract interfaces is key  for improved testability• Internal details of modules have to be  o...
Acknowledgement• Lisa Montgomery of the NASA IV & V• Sally Godfrey of the NASA GSFC• All members of the CFS/cFE team      ...
Upcoming SlideShare
Loading in …5
×

Analysis of Testability of a Flight Software Product Line

796 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
796
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Analysis of Testability of a Flight Software Product Line

  1. 1. Architecture-based Unit Testing of the Flight Software Product LineFraunhofer CESE NASA Goddard Space Flight CenterDharmalingam Ganesan David McComasMikael Lindvall Maureen Bartholomew Steve Slegel Barbara Medina © 2010 Fraunhofer USA, Inc. 1 Center for Experimental Software Engineering
  2. 2. Background• Unit testing deals with testing individual modules in isolation – Assuming the correctness of dependent modules• Bugs found during unit testing are cheaper to fix than – Integration testing – System testing – Field reported defects © 2010 Fraunhofer USA, Inc. 2 Center for Experimental Software Engineering
  3. 3. Unit Testing Challenges in Practice• Isolating the module under test from other modules• Isolating from special OS and hardware features• Handling the variability or configuration dimension• Developing flexible build process (e.g., Makefiles) that facilitates unit testing © 2010 Fraunhofer USA, Inc. 3 Center for Experimental Software Engineering
  4. 4. Research Questions• What architectural design decisions facilitate or impede unit testing?• How can we make use of the architecture to define the architecture of test code?Approach: Case-study driven - NASA’s Core Flight Software Product Line (CFS) © 2010 Fraunhofer USA, Inc. 4 Center for Experimental Software Engineering
  5. 5. Reusable Analysis Questions• Can modules be tested independently of other modules they use?• Can modules be unit tested without access to special hardware and/or OS?• How are variation points of each module being handled during unit testing?• Is there a common look-and-feel in terms of the way the modules are unit tested?• How easy it is to set-up a unit test? (additional questions are listed in the paper) © 2010 Fraunhofer USA, Inc. 5 Center for Experimental Software Engineering
  6. 6. Technical Set-up for Analysis Source Code Makefiles 1 Data Extraction Test Suites (Automated Parsing) Call relation, Include relation, variation Makefiles of points for source test suites Call relation, code Include relation, variation points for test suites 2 Visualization, Query and AnalysisData extraction is automated using standard parsersData analysis and visualization are semi-automatic using tools: Relation Partition Algebra (RPA) query language SAVE-Light, SAVE Prefuse 6 © 2010 Fraunhofer USA, Inc. Center for Experimental Software Engineering
  7. 7. Product Line Structure of the CFS CFS Application A CFS Application B CFS Application C … Core Flight Executive (cFE) Services Layer Software Bus (SB) Executive Services (ES) Table Services (TBL) File Service (FS) Event Services (EVS) services cFE Board Support Hardware Abstraction OS Abstraction Layer Package (BSP) Layer Layer Board Support Package Real time OS Device Drivers (BSP) HardwareSee our SPLC 2009 paper for: the heritage of the CFS the business goals of the CFS architectural analysis of the CFS implementation © 2010 Fraunhofer USA, Inc. 7 Center for Experimental Software Engineering
  8. 8. Run-time Structure of the CFS House- Memory Memory keeping Checksum Manager Dwell* cFE core Applications Inter-task Message Router (SW Bus) Some CFS Applications Some Mission Applications Telemetry Command Software Time Executive Event Table File * * * Services * * * Output Ingest Bus Services Services Services Services Applications communicate by publishing/subscribing messages to the bus Core applications can also directly talk to each other using function calls Example Variation Points: Maximum number of applications using the SW bus Maximum number of messages in the SW bus Maximum number of messages an application can receive © 2010 Fraunhofer USA, Inc. 8 Center for Experimental Software Engineering
  9. 9. Missions using CFS/cFE Mission Managed by cFE CFS Apps StatusCHIPS GSFC ✓ Successfully demonstratedDSILCAS GSFC ✓ Integrated in FSW labGPM GSFC ✓ ✓ Integrated in FSW labLADEE ARC ✓ ✓ Integrated in FSW labLRO APL ✓ On orbitMMS GSFC ✓ ✓ Integrated in FSW labNanosat-6 AFRL/Cornell ✓ ✓ Integrated in FSW labRBSP APL ✓ Integrated in FSW labRLL APL ✓ Integrated in FSW labSolar APL ✓ Integrated in FSW labProbe Plus  Many missions depend on the quality of CFS/cFE and the reusable components it provides! © 2010 Fraunhofer USA, Inc. 9 Center for Experimental Software Engineering
  10. 10. Dependencies of Core Modules cFE - Core Executive Service (ES) Software Bus (SB) Event Service (EVS) File Service (FS) Table Service (TBL) Timing Service (Time) Reverse engineered from the source code of the CFS Boxes denote modules (i.e. directories) Arrows denote code relations (e.g. include, call)  How to test each module independently of other modules? © 2010 Fraunhofer USA, Inc. 10 Center for Experimental Software Engineering
  11. 11. Unit Test Architecture of CFS• Core idea is to – Define mocks/stubs for dependent modules – Systematically control return codes of dependent modules• Mocks are independent of other mocks• Mocks have the same signature as the real interface• Modules can be bound to mock modules or to real modules – Facilitates incremental integration testing too!11 © 2010 Fraunhofer USA, Inc. Center for Experimental Software Engineering
  12. 12. How Return Code is Controlled?• UT_SetRtn_t is the key data structure: – typedef struct { uint32 count; uint32 value; } UT_SetRtn_t;• Mocked functions are forced to return values based on the state of count• For every mocked function, there is an instance of UT_SetRtn_t © 2010 Fraunhofer USA, Inc. 12 Center for Experimental Software Engineering
  13. 13. Controlling the Return Code - Exampleextern UT_SetRtn_t SB_CreatePipeRtn;// mock implementation of create pipeint32 CFE_SB_CreatePipe (CFE_SB_PipeId_t *PipeIdPtr, uint16 Depth, void UT_SetRtnCode (UT_SetRtn_t *varPtr, char *PipeName) { int32 rtnVal, int32 cnt){ if (SB_CreatePipeRtn.count > 0) { varPtr->value = rtnVal; SB_CreatePipeRtn.count--; varPtr->count = cnt; if(SB_CreatePipeRtn.count == 0) { } return SB_CreatePipeRtn.value; } } return CFE_SUCCESS;} // forces CreatePipe to succeed in the first and fail in the second call UT_SetRtnCode (&SB_CreatePipeRtn, -1, 2); © 2010 Fraunhofer USA, Inc. 13 Center for Experimental Software Engineering
  14. 14. Binding to Mock Modules The test suite for the Executive es_ut.c Service (ES) Executive Service (ES) ut_evs_stubs ut_sb_stubs ut_time_stubs ut_tbl_stubs ut_fs_stubs ut_osapi_stubs ut_bsp_stubsExecutive Service module uses stub implementations of dependent modulesThe unit tests of ES control the return code of dependent modules to cover all paths © 2010 Fraunhofer USA, Inc. 14 Center for Experimental Software Engineering
  15. 15. Testability and Module Secretsint32 CFE_ES_LoadLibrary(char *EntryPoint, char *LibName, …) { boolean LibSlotFound = FALSE; for ( i = 0; i < CFE_ES_MAX_LIBRARIES; i++ ) { CFE_ES_MAX_LIBRARIES is a if ( CFE_ES_Global.LibTable[i].RecordUsed == FALSE ) { variation point LibSlotFound = TRUE; break; CFE_ES_Global is an internal global } variable of the ES module } if(LibSlotFound == FALSE) return CFE_ES_ERR_LOAD_LIB; …;}How do we test for more than the max. no. of loaded libraries scenario? /* Test for loading more than max number of libraries */ for (j= 0; j < CFE_ES_MAX_LIBRARIES; j++) { CFE_ES_Global.LibTable[j].RecordUsed = TRUE; } Return = CFE_ES_LoadLibrary("EntryPoint","LibName“, …); UT_Report(Return == CFE_ES_ERR_LOAD_LIB, "CFE_ES_LoadLibrary“, ”No free library slots"); Test code needs access to modules’ secrets! © 2010 Fraunhofer USA, Inc. 15 Center for Experimental Software Engineering
  16. 16. Same Return Code Impede Testingint32 CFE_SB_SendMsg(CFE_SB_Msg_t *MsgPtr) { /* check input parameter */if(MsgPtr == NULL){ Test code CFE_EVS_SendEventWithAppID(“Send Err:Bad input argument”,…); void Test_SendMsg_NullPtr(void){ return CFE_SB_BAD_ARGUMENT; …} ActRtn = CFE_SB_SendMsg(NULL); ExpRtn =MsgId = CFE_SB_GetMsgId(MsgPtr); CFE_SB_BAD_ARGUMENT;/* validate the msgid in the message */ if(ActRtn != ExpRtn){if(CFE_SB_ValidateMsgId(MsgId) != CFE_SUCCESS) { TestStat = CFE_FAIL; CFE_EVS_SendEventWithAppID(“Send Err:Invalid MsgId”, …); } return CFE_SB_BAD_ARGUMENT;} ExpRtn = 1; … ActRtn = UT_GetNumEventsSent();} if(ActRtn != ExpRtn){ TestStat = CFE_FAIL; } Test code cannot easily verify which … path was actually taken } Have to count the number of times log events (SendEvent)are used to justify the intended path © 2010 Fraunhofer USA, Inc. 16 Center for Experimental Software Engineering
  17. 17. Testability, Abstract Interfaces and Alternative Implementations Software Bus (SB) options linux/osapi.c rtems/osapi.c vxworks6/osapi.c Test/ut_osapi_stubs.c int32 OS_QueuePut(...){ int32 OS_QueuePut(...){ int32 OS_QueuePut(...){ int32 OS_QueuePut (...) { ... ... ... // Mock Implementation sendTo(...); rtems_message_queue_send(...); msgQSend (...); } ... ... ... } } }Software Bus needs services of OS QueuesCFS needs to run on several OS typesOS APIs are often different among OSFor improved testability, variations among APIs have to be abstractedBenefits: Mock light-weight implementations of APIs for unit testing Unit test code is free of variability issues to a large extent © 2010 Fraunhofer USA, Inc. 17 Center for Experimental Software Engineering
  18. 18. Some DataCore Module # of Functions in Interface # Directly invoked in Unit TestsSB 30 30ES 33 31  Each public function has at least oneEVS 7 7Time 24 24 dedicated test programTable 14 13File 5 5Core Module # of Functions Defined # Directly invoked in Unit TestsSB 86 45ES 117 68EVS 33 12 All internal functions are transitively testedTime 72 42Table 60 41File 11 11 Stub SB Stub ES Stub EVS Stub Time Stub TBL Stub FSSB NA 11 3 1 0 1ES 10 NA 4 3 1 4 Only a few stubs are needed becauseEVS 8 10 NA 1 0 1Time 9 8 2 NA 0 of well-defined abstract interfaces!TBL 9 15 3 1 NA 3FS 0 2 0 1 0 NA © 2010 Fraunhofer USA, Inc. 18 Center for Experimental Software Engineering
  19. 19. Conclusion• Programming to abstract interfaces is key for improved testability• Internal details of modules have to be open to facilitate unit testing – Apply architectural rules to prevent misusages• Complete graph of dependencies do NOT imply poor design and low testability• Treating testing as variation points help unit and incremental integration testing © 2010 Fraunhofer USA, Inc. 19 Center for Experimental Software Engineering
  20. 20. Acknowledgement• Lisa Montgomery of the NASA IV & V• Sally Godfrey of the NASA GSFC• All members of the CFS/cFE team © 2010 Fraunhofer USA, Inc. 20 Center for Experimental Software Engineering

×