This document discusses unit testing of the Core Flight Software (CFS) product line developed by NASA. It examines how the CFS architecture facilitates or impedes unit testing and how the architecture of test code can be defined based on the system architecture. The CFS uses a unit test architecture with mocks/stubs of dependent modules to enable isolated testing. It finds that defining abstract interfaces and exposing internal details controlled via architectural rules improves testability, and that complete dependency graphs do not inherently imply poor testability.

1. Architecture-based Unit Testing of
the Flight Software Product Line
Fraunhofer CESE NASA Goddard Space Flight Center
Dharmalingam Ganesan David McComas
Mikael Lindvall Maureen Bartholomew
Steve Slegel
Barbara Medina
© 2010 Fraunhofer USA, Inc. 1
Center for Experimental Software Engineering

2. Background
• Unit testing deals with testing individual
modules in isolation
– Assuming the correctness of dependent
modules
• Bugs found during unit testing are cheaper
to fix than
– Integration testing
– System testing
– Field reported defects
© 2010 Fraunhofer USA, Inc.
2
Center for Experimental Software Engineering

3. Unit Testing Challenges in Practice
• Isolating the module under test from other
modules
• Isolating from special OS and hardware
features
• Handling the variability or configuration
dimension
• Developing flexible build process (e.g.,
Makefiles) that facilitates unit testing
© 2010 Fraunhofer USA, Inc.
3
Center for Experimental Software Engineering

4. Research Questions
• What architectural design decisions
facilitate or impede unit testing?
• How can we make use of the architecture
to define the architecture of test code?
Approach: Case-study driven - NASA’s
Core Flight Software Product Line (CFS)
© 2010 Fraunhofer USA, Inc.
4
Center for Experimental Software Engineering

5. Reusable Analysis Questions
• Can modules be tested independently of other
modules they use?
• Can modules be unit tested without access to
special hardware and/or OS?
• How are variation points of each module being
handled during unit testing?
• Is there a common look-and-feel in terms of the
way the modules are unit tested?
• How easy it is to set-up a unit test?
 (additional questions are listed in the paper)
© 2010 Fraunhofer USA, Inc.
5
Center for Experimental Software Engineering

6. Technical Set-up for Analysis
Source Code Makefiles
1
Data Extraction
Test Suites (Automated Parsing)
Call relation, Include
relation, variation
Makefiles of points for source
test suites Call relation, code
Include relation,
variation points for
test suites
2 Visualization, Query and
Analysis
Data extraction is automated using standard parsers
Data analysis and visualization are semi-automatic using tools:
Relation Partition Algebra (RPA) query language
SAVE-Light, SAVE
Prefuse 6
© 2010 Fraunhofer USA, Inc.
Center for Experimental Software Engineering

7. Product Line Structure of the CFS
CFS Application A CFS Application B CFS Application C …
Core Flight Executive (cFE) Services Layer
Software Bus (SB) Executive Services (ES) Table Services (TBL)
File Service (FS)
Event Services (EVS)
services
cFE Board Support Hardware Abstraction
OS Abstraction Layer
Package (BSP) Layer Layer
Board Support Package
Real time OS Device Drivers
(BSP)
Hardware
See our SPLC 2009 paper for:
 the heritage of the CFS
 the business goals of the CFS
 architectural analysis of the CFS implementation
© 2010 Fraunhofer USA, Inc.
7
Center for Experimental Software Engineering

8. Run-time Structure of the CFS
House- Memory Memory
keeping Checksum
Manager Dwell
* cFE core Applications
Inter-task Message Router (SW Bus)
Some CFS Applications
Some Mission Applications
Telemetry Command Software Time Executive Event Table File
* * * Services * * *
Output Ingest Bus Services Services Services Services
Applications communicate by publishing/subscribing messages to the bus
Core applications can also directly talk to each other using function calls
Example Variation Points:
Maximum number of applications using the SW bus
Maximum number of messages in the SW bus
Maximum number of messages an application can receive
© 2010 Fraunhofer USA, Inc.
8
Center for Experimental Software Engineering

9. Missions using CFS/cFE
Mission Managed by cFE CFS Apps Status
CHIPS GSFC ✓ Successfully demonstrated
DSILCAS GSFC ✓ Integrated in FSW lab
GPM GSFC ✓ ✓ Integrated in FSW lab
LADEE ARC ✓ ✓ Integrated in FSW lab
LRO APL ✓ On orbit
MMS GSFC ✓ ✓ Integrated in FSW lab
Nanosat-6 AFRL/Cornell ✓ ✓ Integrated in FSW lab
RBSP APL ✓ Integrated in FSW lab
RLL APL ✓ Integrated in FSW lab
Solar APL ✓ Integrated in FSW lab
Probe Plus
 Many missions depend on the quality of CFS/cFE and
the reusable components it provides!
© 2010 Fraunhofer USA, Inc.
9
Center for Experimental Software Engineering

10. Dependencies of Core Modules
cFE - Core
Executive Service (ES)
Software Bus (SB) Event Service (EVS)
File Service (FS) Table Service (TBL)
Timing Service (Time)
Reverse engineered from the source code of the CFS
Boxes denote modules (i.e. directories)
Arrows denote code relations (e.g. include, call)
 How to test each module independently of other modules?
© 2010 Fraunhofer USA, Inc.
10
Center for Experimental Software Engineering

11. Unit Test Architecture of CFS
• Core idea is to
– Define mocks/stubs for dependent modules
– Systematically control return codes of
dependent modules
• Mocks are independent of other mocks
• Mocks have the same signature as the real
interface
• Modules can be bound to mock modules
or to real modules
– Facilitates incremental integration testing too!11
© 2010 Fraunhofer USA, Inc.
Center for Experimental Software Engineering

12. How Return Code is Controlled?
• UT_SetRtn_t is the key data structure:
– typedef struct {
uint32 count;
uint32 value;
} UT_SetRtn_t;
• Mocked functions are forced to return
values based on the state of count
• For every mocked function, there is an
instance of UT_SetRtn_t
© 2010 Fraunhofer USA, Inc.
12
Center for Experimental Software Engineering

13. Controlling the Return Code - Example
extern UT_SetRtn_t SB_CreatePipeRtn;
// mock implementation of create pipe
int32 CFE_SB_CreatePipe (CFE_SB_PipeId_t *PipeIdPtr,
uint16 Depth, void UT_SetRtnCode (UT_SetRtn_t *varPtr,
char *PipeName) { int32 rtnVal,
int32 cnt){
if (SB_CreatePipeRtn.count > 0) {
varPtr->value = rtnVal;
SB_CreatePipeRtn.count--; varPtr->count = cnt;
if(SB_CreatePipeRtn.count == 0) { }
return SB_CreatePipeRtn.value;
}
}
return CFE_SUCCESS;
}
// forces CreatePipe to succeed in the first and fail in the second call
UT_SetRtnCode (&SB_CreatePipeRtn, -1, 2);
© 2010 Fraunhofer USA, Inc.
13
Center for Experimental Software Engineering

14. Binding to Mock Modules
The test suite for
the Executive es_ut.c
Service (ES)
Executive
Service (ES)
ut_evs_stubs ut_sb_stubs ut_time_stubs ut_tbl_stubs ut_fs_stubs ut_osapi_stubs ut_bsp_stubs
Executive Service module uses stub implementations of dependent modules
The unit tests of ES control the return code of dependent modules to cover all paths
© 2010 Fraunhofer USA, Inc.
14
Center for Experimental Software Engineering

15. Testability and Module Secrets
int32 CFE_ES_LoadLibrary(char *EntryPoint,
char *LibName, …) {
boolean LibSlotFound = FALSE;
for ( i = 0; i < CFE_ES_MAX_LIBRARIES; i++ ) { CFE_ES_MAX_LIBRARIES is a
if ( CFE_ES_Global.LibTable[i].RecordUsed == FALSE ) { variation point
LibSlotFound = TRUE;
break; CFE_ES_Global is an internal global
} variable of the ES module
}
if(LibSlotFound == FALSE)
return CFE_ES_ERR_LOAD_LIB; …;
}
How do we test for more than the max. no. of loaded libraries scenario?
/* Test for loading more than max number of libraries */
for (j= 0; j < CFE_ES_MAX_LIBRARIES; j++) {
CFE_ES_Global.LibTable[j].RecordUsed = TRUE;
}
Return = CFE_ES_LoadLibrary("EntryPoint","LibName“, …);
UT_Report(Return == CFE_ES_ERR_LOAD_LIB,
"CFE_ES_LoadLibrary“, ”No free library slots");
Test code needs access to modules’ secrets!
© 2010 Fraunhofer USA, Inc.
15
Center for Experimental Software Engineering

16. Same Return Code Impede Testing
int32 CFE_SB_SendMsg(CFE_SB_Msg_t *MsgPtr) {
/* check input parameter */
if(MsgPtr == NULL){
Test code
CFE_EVS_SendEventWithAppID(“Send Err:Bad input argument”,…); void Test_SendMsg_NullPtr(void){
return CFE_SB_BAD_ARGUMENT; …
} ActRtn = CFE_SB_SendMsg(NULL);
ExpRtn =
MsgId = CFE_SB_GetMsgId(MsgPtr); CFE_SB_BAD_ARGUMENT;
/* validate the msgid in the message */ if(ActRtn != ExpRtn){
if(CFE_SB_ValidateMsgId(MsgId) != CFE_SUCCESS) { TestStat = CFE_FAIL;
CFE_EVS_SendEventWithAppID(“Send Err:Invalid MsgId”, …); }
return CFE_SB_BAD_ARGUMENT;
} ExpRtn = 1;
… ActRtn = UT_GetNumEventsSent();
} if(ActRtn != ExpRtn){
TestStat = CFE_FAIL;
}
Test code cannot easily verify which …
path was actually taken }
Have to count the number of times
log events (SendEvent)are used to
justify the intended path © 2010 Fraunhofer USA, Inc.
16
Center for Experimental Software Engineering

17. Testability, Abstract Interfaces and
Alternative Implementations
Software Bus (SB) options
linux/osapi.c rtems/osapi.c vxworks6/osapi.c Test/ut_osapi_stubs.c
int32 OS_QueuePut(...){ int32 OS_QueuePut(...){ int32 OS_QueuePut(...){ int32 OS_QueuePut (...) {
... ... ... // Mock Implementation
sendTo(...); rtems_message_queue_send(...); msgQSend (...); }
... ... ...
} } }
Software Bus needs services of OS Queues
CFS needs to run on several OS types
OS APIs are often different among OS
For improved testability, variations among APIs have to be abstracted
Benefits:
Mock light-weight implementations of APIs for unit testing
Unit test code is free of variability issues to a large extent
© 2010 Fraunhofer USA, Inc.
17
Center for Experimental Software Engineering

18. Some Data
Core Module # of Functions in Interface # Directly invoked in Unit Tests
SB 30 30
ES 33 31  Each public function has at least one
EVS 7 7
Time 24 24
dedicated test program
Table 14 13
File 5 5
Core Module # of Functions Defined # Directly invoked in Unit Tests
SB 86 45
ES 117 68
EVS 33 12 All internal functions are transitively tested
Time 72 42
Table 60 41
File 11 11
Stub SB Stub ES Stub EVS Stub Time Stub TBL Stub FS
SB NA 11 3 1 0 1
ES 10 NA 4 3 1 4 Only a few stubs are needed because
EVS 8 10 NA 1 0 1
Time 9 8 2 NA 0
of well-defined abstract interfaces!
TBL 9 15 3 1 NA 3
FS 0 2 0 1 0 NA
© 2010 Fraunhofer USA, Inc.
18
Center for Experimental Software Engineering

19. Conclusion
• Programming to abstract interfaces is key
for improved testability
• Internal details of modules have to be
open to facilitate unit testing
– Apply architectural rules to prevent misusages
• Complete graph of dependencies do NOT
imply poor design and low testability
• Treating testing as variation points help
unit and incremental integration testing
© 2010 Fraunhofer USA, Inc.
19
Center for Experimental Software Engineering

20. Acknowledgement
• Lisa Montgomery of the NASA IV & V
• Sally Godfrey of the NASA GSFC
• All members of the CFS/cFE team
© 2010 Fraunhofer USA, Inc.
20
Center for Experimental Software Engineering