User Guide of Regression Test & Validation Tool (v1.0)

1
User Guide of Regression Test & Validation Tool (v1.0)
Date June 17th
, 2014 Author Chen Fang
Location AEA Contact chenfang@autoliv.com
CONTENT
A. Introduction and Background ………………………………… 1
B. Tool Setup ………………………………… 4
C. Functions and Operations (For Tool User) ………………………………… 7
a. GUI Panels and Functions Introduction …………………………….. 8
i. Folder Selection and Regression Mode Selection ………….. 8
ii. Image Position Calibration ……….………………….. 10
iii. Setting Mode Selection ……….………………….. 12
iv. Exceptions Control ……….………………….. 17
b. Tool Applications …………………………… 19
i. CAN DAQ Data Label ……….………………….. 19
ii. Regression Test – Compare SIMULATION with CAN ……….. 26
iii. Regression Test – Compare SIMULATION with SIMULATION 31
D. Software Structure (For Tool Developer) ………………………………… 32
a. Graphical User Interface (GUI) …………………………….. 34
b. DAQ Data Loading and Processing …………………………….. 34
c. Image Processing – OpenCV Toolkit …………………………….. 35
d. EXCEL Interface – Object Linking and Embedding (OLE) Automation Technique
e. Parallel Computing …………………………….. 36
E. Future Work ………………………………… 37
a. SIMU to SIMU Regression Test …………………………….. 37
b. Vehicle Image Recognition …………………………….. 37
c. COM Object Packaging …………………………….. 37

2
A. Introduction and Background
The regression test and validation tool was developed to evaluate the sensor software and
points, helping engineers to select and deliver the optimal software and points set. Basically,
this tool can be used in two directions: (1) Software development; (2) Performance validation
and reporting.
During the development stage, engineers will usually at first locate some issue DAQ files, and
then try to modify the software (add patches) or tune the points in order to fix those issues.
However, fixing one issue in certain situation may potentially cause another issue in other
situations. For example, adding conditions to remove rainy false positives may potentially
increase the sunny false negatives. In this way, regression test provides us a method to quickly
check a large amount of files covering various scenarios for each software/points update.
During the performance validation and reporting stage, OEM or managers usually need to know
about the performance or the latest status of the released sensor software in time. Mounting the
released products on vehicle and collecting on-road sensor data is a traditional and robust way
to evaluate the sensor performance, but sometimes the data may not be enough or certain
testing scenarios may not be satisfied (rainy data for example). In this way, regression test
provides us a method to efficiently evaluate and validate the performance of released software
on large amount of historical data.
Based on user needs, some basic requirements that may have to be satisfied when developing
a good regression and validation test tool were summarized in table 1.
Table 1.1 Regressions and Validation Tool Requirement
Requirements Description Possible Solution
Speed 200 – 500 files within 1-2 hours Using C, codes optimization
Accuracy Points match, software version
control, signal synchronization
Pay attention to details
User-friendly Easy to use, no coding need GUI, reports auto-generation
Nice Report Concise, necessary statistics,
easy to read
Deep understanding about algorithm
and engineers’ work
This introduced regression and validation tool was developed trying to match above
requirements but still far from perfect. Setup tutorials will be presented in Section B. The
developer-level implementation details will be introduced in Section D. Following figure 1 shows
a tool GUI sample and figure 2 shows a final performance report sample. More details about
user-level operations will be introduced in Section C. In final Section E, some undergoing work
and potential future improvements were presented.

3
Figure 1.1 a Graphical User Interface (GUI) Sample of Regression Test Tool
Figure 1.2 a Final Performance Report Sample of Regression Test Tool

4
B. Tool Setup
a. System Requirement
Table 2.1 System Requirement Summary Table
Item Requirement Reason
OS
Platform
Windows (64bits or 32bits,
64bits preferred)
All mex files were developed under
64bits environment.
OS Toolkit Windows default SDK
Need to compile cpp files to mex filesC++
Compiler
MATLAB supported
Visual Studio 2010
recommendedMATLAB
Version
>=2007, no special toolbox
needed
All image processing and EXCEL
control functions were written in C++
Note that some windows systems may not have software development kit (SDK)
installed as default, but the latest SDK is available to download for free in Microsoft
download center http://www.microsoft.com/en-us/download/details.aspx?id=8279
Also Visual C++ Express edition also works if Visual Studio 2010 professional is not
available. The Express edition is also free to download in Microsoft download center
http://www.microsoft.com/en-us/download/details.aspx?id=40787
b. Setup Tutorial
The tool setup is straightforward and simple; following figures indicate the setup process
step by step.
Step 1: Copy or Unzip. Unzip the compressed file into any location of the PC. There will
be a folder called “Regression_Test_Tool_Release”. This folder can also be directly
copied from hard-drive.
Step 2: Build Package. Open MATLAB, direct into this unzipped folder and type “build”
in the command window and press “Enter” shown as figure 2.1
Figure 2.1 setup step 2 - build the package using keyboard

5
Or you can input “regressionGUI” in command window, and select “Build” button in the
popped out GUI window, shown as figure 2.2. Either way is OK.
Figure 2.2 setup step 2 – build the package in GUI window
Step 3: Wait for building. Wait for the completeness of the building process shown as
figure 2.3 and 2.4
Figure 2.3 setup step 3 – building process

6
Figure 2.4 setup step 3 – building process continued
Step 4: Done. Command “regressionGUI”, the GUI window will pop out.
Note that the building process will usually take around 1 minute for different PC systems.
The “build.m” script was created to remove the existing mex files, and to further compile
all cpp source files producing new mex files. The reason why all mex files should be re-
built again in each PC is that the windows DLLs may slightly vary from PC to PC.
Directly using the built mex files without re-building may cause incompatibility issues,
namely certain DLLs utilized to build and execute mex files in one PC may not exist in
another PC. So the most secure way is to compile all cpp files again for each new PC.
During this 1-minute building process, (1) existing mex files were removed; (2) mex files
for image processing were built (see the first progress bar in figure 2.3 – Compiling MEX
files); (3) mex files for EXCEL automation were built (see the second progress bar in
figure 2.4 – Build xxxx). The implementation details will be introduced in Section D.

7
C. Functions and Operations (For Tool User)
In this section, the separate functions embedded in each GUI panel will be introduced at first,
and then the applications of the entire GUI will be presented. To open the tool GUI, go to the
folder directory and command “regressionGUI” shown as figure 3.1, or find out the
“regressionGUI.m” file, right click it and select the “Run”. A GUI will appear shown as in
figure 3.2
Figure 3.1 DAQ data label step 1 – open GUI
Figure 3.2 the default GUI window
There are 4 panels on this GUI shown as in figure 3.2: (1) Folder Selection Panel; (2)
Regression Mode Panel; (3) Setting Mode Panel; (4) Image Position Calibration
Panel; the functions of each panel will be presented in following section a.

8
a. GUI Panels and Functions Introduction
i. Folder Selection and Regression Mode Selection
The regression Mode is the core panel of the entire tool, which will determine
the application type. Following table 3.1 give a brief introduction about each
application type determined by selecting the “Regression Mode”
Table 3.1 Introduction of Regression Modes
Regression Mode Description Enabled function
Regression Report
Use specified simulation software to
simulate the DAQ data and generate
regression reports comparing SIMU
and CAN performance
All
Summary Report
Convert the raw report generated by
using “Regression Report” mode to
organized summary report
“Browse” after “Summary
Report”; “BSD/LCA” mode
selection
DAQ Data Label
Report
Input DAQ data, and output DAQ
label report displaying necessary
CAN signals/plots/frames to help
label/check the DAQ data
“DAQ Folder” Selection;
“Image Position Calibration”;
“L/R/B-Side” Selection
Comparison Report
Use specified simulation software to
simulate the DAQ data and generate
regression reports comparing new
SIMU and old SIMU performance
All
Figure 3.3 Example for Selecting “DAQ data label” mode
Grayed
Grayed
Grayed

9
Note that the “Enabled function” in table 3.1 means that some GUI panel functions
will be enabled when selecting certain regression mode and some will be disabled.
Those disabled functions will be grayed therefore. An example is shown in figure 3.3
above when “DAQ data label” mode is selected. The “XPCDN Folder” selection
function, setting mode selection and software version are disabled and grayed.
The function of “Folder Selection” panel is presented in following table 3.2
Table 3.2 Introduction of Folder Selection
Folder Selection Description Requirement
XPCDN Folder
Select the folder containing XPCDN
software used to do simulation Select the folder, do NOT go
into (double click) the folder
when browsing the directory
DAQ Folder
Select the folder containing DAQ
date that needed to be simulated
Note that the DAQ data usually contains the sensor data (.dat), CAN data (.asc), and
video clips (.avi). Sometimes there is no video data. Following figure 3.4 shows an
example of selecting DAQ data folder. Click the “Browse” button; find the folder
containing all DAQ data needed, “2014_06_16_SouthfieldATS_GM_LCA” for
example. Left-single-click the folder and press the “Select Folder” button. DO NOT
double click to open it. The same principle goes to the “XPCDN Folder” selection.
Figure 3.4 Browse and select the DAQ data

10
ii. Image Position Calibration
This part is a bit tricky but very important. DAQ data recorders may sometimes
messed up the left and right video channels, and sometimes may use 2 cameras
and sometimes 4 cameras. This panel is therefore developed to link video frames
to correct side of sensor data, to tell the program that, for the left side sensor
data, please use the video from right channel for example.
Figure 3.5 Example of image position calibration
An example is shown in figure 3.5 above, once the DAQ data is selected
(assuming there are videos in DAQ data, exceptions will be considered later),
one file will be randomly selected and its video frames will be displayed under the
“Image Position Calibration” panel. This is done automatically. There is a slider
bar on the top of the frame, drag it to check other frames of the entire video.
Except the 0-Camera Mode, the 1-Camera Mode, 2-Camera Mode and 4-
Camera Mode need to be calibrated by users, the operation guide will be
presented below.
Slider Bar

11
Figure 3.6 2-Camera mode examples
If the entire image is combined of 2 sections (usually from 2 camera channels), refer to
above figure 3.6, firstly tick the “2-Camera Mode” checkbox, then two pop-up menus will
appear on the surface of the video frame. In the perspective of vehicle driver, select
left/right/back under each pop-up menu, and then the image position description will be
automatically filled into the text box (L/R-Side in above figure). Because the cameras are
always mounted to capture the objects behind, the car doors on the corner of the image
may help decide which side it is. Imaging that you are the driver and you will be able to
select it easily. In figure 3.6, two examples are given for users’ reference.
Figure 3.7 1-Camera/-Camera mode examples
The same principles apply for the 1-Camera and 4-Camera mode shown as in figure 3.7.
Notice that user should always choose 4-Camera mode if 4 sections (camera channel)
are observed even though 2 of them are invalid (blue in figure 3.7 for example);
Left Door
Right Door
Right Door Left Door

12
Under the “Image Position Calibration” panel, there is another function called
“Software Version” as shown in following figure 3.8. The default selection for this
sub-panel is “Auto”, which mean that it automatically detect the project type after
the after the DAQ data is selected. For example, in figure 3.8, string “Fiat” and
“Fiat_Dev” appear in the “SensorA4 File” and “SW Version” text boxes after we
select Fiat DAQ data. In very rare occasions, user needs to select “Manual”
mode, and manually select a project type different from that of selected DAQ
data. It can help users to simulate some DAQ data using different project’s
software.
Figure 3.8 Software Version sub-panel examples
iii. Setting Mode Selection
The setting mode panel is mainly used to set up the setting files or parameters
(points). To simulate any DAQ data, user need to specify the software (XPCDN
Folder), DAQ data (DAQ Folder), and the software parameters (Setting file). The
setting file is very important. Once the sensor software is fixed, engineers will
focus on tuning the parameter to optimize the software performance. All
parameters (points) are put in one MAT file; this panel can be used to select the
setting file or to update several points in the setting file.
There are 3 basic setting file selection modes, namely “SW Default Setting”,
“DAQ Data Setting”, and “Load in Setting File”. They are briefly introduced in
following table 3.3

13
Table 3.3 Setting file selection mode introduction
Setting Selection Description
SW Default
Setting
Use the setting file generate by the simulation software. All
simulation software can generate a set of default points.
DAQ Data
Setting
Use the setting file from the raw DAQ data. The settings stored in
DAQ data were used in real sensor when collecting those DAQ
data
Load in Setting
File
Use the user created setting file. Users can create their own
settings and load them in.
The “POINTS CHANGE” button is created to facilitate the behavior of updating
points. If there is no “POINTS CHANGE” function, user may have to copy original
setting file at first, change several points manually, and then create a new setting
file to do simulation. For example, if user wants to use DAQ data settings plus
several points change, he or she can select the “DAQ Data Setting” radio button
at first, and then click the “POINTS CHANGE” button. The points change GUI will
pop out as shown in following figure 3.9.
Figure 3.9 POINTS CHANGE GUI introductions
All the points in a setting file can be categorized into several fields; the fields’
names are displayed on the right part of the GUI as push buttons. Click one of
them, and all points belonging to that field will appear in the table. There are 5
columns for this table, which are introduced in following table 3.4

14
Table 3.4 Columns introduction of the points table
Column Description Editable
Field Name The points field names NO
Point Name The points names NO
Saved “NO” if user changes some points but haven’t saved
them into current settings
NO
Current Setting Contains the current status of settings value, which will
be used in simulation
NO
User Change User can edit this column YES
For example, if user wants to change the point “Straight_Rd_R” under the
“LCATAParams” field from 4900 to 5000.
Figure 3.10 change one point using POINTS CHANGE GUI
As shown in following figure 3.10, in the left figure, click “LCATAParams” push
button at first, and then find out the point “Straight_Rd_R” in the table. On the
column “User Change”, edit 5000 and press “Enter” (or click anywhere else), and
then it’s changed to 5000 and string “NO” will appear on the “Saved” column. But
the current setting hasn’t been updated yet. Refer to the right side figure, just
click “Save” button on the left-down corner, and the “Saved” column will become
“YES”, and the current setting will be changed to 5000.
By pressing the “Reload” button, all the current settings will be reloaded to be the
original value. In above figure 3.10 for example, if user has reached the status of
right side figure, pressing the “Reload” button will lead the points table back to
the status shown in the left side figure.
c

15
There is a special field called “VehicleSpecific” as shown in following figure 3.11.
The corresponding push button is always put at the end of the button group. Of
course user can edit any points of it as we did in figure 3.10. But this field is
special because engineers will usually change all or lots of vehicle specific points
at one time instead of just changing one or two of them if necessary during the
LCA parameter tuning stage. There are 220 vehicle specific points; manually
changing them one-by-one will be a disaster. For this reason, another button
“Load VehicleSpec” on the right-down corner is developed.
Figure 3.11 Change the Vehicle-Specific points
Figure 3.12 Load Vehicle Specific points

16
Click the “Load VehicleSpec” and a browse window as shown in figure 3.12 will
show up. Select one .nbpdxml type file and press “Open”. The .nbpdxml format is
a special format containing all vehicle specific points needed. It’s usually
provided by the engineer whoever is responsible of generating new vehicle
specific points.
Figure 3.13 Load Vehicle Specific points
After selecting one .nbpdxml file, all points contained by this file will be extracted
to the points table as shown in the figure 3.13 (left). The points will only be
extracted to “User Change” column at this point. Click “Save”, and then all those
points will be saved into current settings as shown in the right side figure above.
Besides setting file selection, L/R/B-Side can also be selected under the “Setting
Mode” panel. The “B-Side” mode under “Setting Mode” panel is selected as
default, which means that both of the left side sensor data and right side sensor
data will be processed. User can choose either “L-Side” (left side) or “R-Side”
(right side) if only one side data should be processed (another side data available
but corrupted or useless for example). Note that if one side data were missing or
unavailable at all, user can still choose “B-Side” mode, because program will only
process available data and automatically skip those missing sides.

17
iv. Exceptions Control
The interactions among different panels and functions are a bit complicated, and
user may sometimes load in wrong folder or select wrong mode. Exceptions are
used to control the user behavior boundary. All exceptions are put in following
table 3.5; user can always refer to this table for the specific reason when
encountering the warning/error message box.
Table 3.5 Exceptions list
Exception Reasons/Solutions
Pop out after selecting the DAQ data, which
means that there is no video file. It’s OK for LCA
mode when no video needed. But check the raw
DAQ data if you expect the existence of videos
Means that you select the invalid DAQ data
folder, this selected folder doesn’t contain any
sensor data. Check the raw DAQ data folder
Means that you select the invalid XPCDN folder,
program doesn’t detect the MEX file used to do
simulation. Check the XPCDN folder, make sure
project is compiled and MEX file is produced
Means that you select 1/2/4-Camera Mode but
didn’t select the Left/Right/Back in the pop-up
menu. Do NOT forget to calibrate images when
new DAQ data loaded.
Pops out when you specified processing L-Side,
but didn’t select a corresponding Left side image
in image position calibration panel.
Pops out when you specified processing R-Side,
but didn’t select a corresponding Rift side image
in image position calibration panel.
Pops out when you selected the “Summary
Report” regression mode but forgot browsing to
load the raw regression report, or load wrong
report
Pops out when you selected the “Load in Setting
File” setting mode but forgot browsing to load
the new setting file, or load wrong setting file

18
No Excel file (test plan) is found when
simulating LCA DAQ data, note that LCA
experiments are all controlled experiments with
test plans
More than one Excel file (test plan) are found
when simulating LCA DAQ data, note that LCA
experiments are all controlled experiments with
test plans

19
b. Tool Applications
In previous section a, different functions in separate panels were already introduced; in
this section, different tool applications will be presented step-by-step.
i. CAN DAQ Data Label
The “DAQ data labeling” function can help user quickly review the collected DAQ
sensor data, locating those issue files. This application can be used to analyze
both BSD and LCA data. Following table 3.6 indicates the step-by-step tutorials
for BSD DAQ Data Labeling.
Table 3.6 BSD mode DAQ Data Labeling guidance
Index Description
Step 1 Select the “DAQ Data Label Report” option in Regression Mode panel
Step 2 Select the “BSD” check box in Regression Mode panel
Step 3 Browse and select the DAQ folder in Folder Selection panel. (Select but do
NOT go into folder during browsing, refer to 3.a.i for guidance)
Step 4 Select “L/R/B-Side” in Setting Mode panel, keep the default “B-Side” if no
special requirement
Step 5 Calibrate the image position (Must do for BSD mode, refer to 3.a.ii for
guidance)
Step 6 Press “Run” button to run
Step 7
Open, label and save produced “DaqLabelReport”. It will be automatically
saved into the root directory of tool folder, and the report name will follow the
format “datafilename_DaqLabelReport_YYYYMMDDhhmmss.xls”. (Refer to
following table 3.7 and 3.13 for labeling guidance)
Step 8
Select “Summary Report” in Regression Mode panel, and browse to select
the labeled “DaqLabelReport” in step 7. Press “Run” button, and then the
final summary report will be saved into the root directory of tool folder
Note that in step 7, an example of “DaqLabelReport” is shown as following figure
3.14. The report contains title section and content section. Each data entry in the
content section represents one side of one DAQ file. User can drag up and down
the scroll bar to view the processing results of all DAQ files. The title section will
be fixed during scrolling. Note that the red marked area is the so-called “Label
Area”, which contains 3 cells for each data entry, namely the “EVENT”,
“TARGET”, and “CAN_LABEL”. User need to verify and edit these 3 cells based
on the signal plots and video frames. All features and the labeling guidance are
listed in table 3.7.

20
Figure 3.14 the DAQ Label Report sample
Table 3.7 BSD mode DAQ Label Report feature list
ID Feature Description
1 File_Name The DAQ file name processed
2 Sensor_Side L-left side sensor; R-right side sensor
3 Host_Speed (m/s) The host vehicle speed when the first alert is activated
4 EVENT FE-front entry; RE-rear entry
5 TARGET C/S/V-car, SUV, or van; SEMI-Semi. Truck
6 CAN_LABEL FP/FN/drop out/Late/out early/on early/sticky (table 3.13)
7 No_CAN_Alert Number of CAN alerts
8 Duration (sweep) Duration of each CAN alerts
9 Host_Speed_On_Off (m/s) The host vehicle speed at the point when alert is on and off
10 CAN_Front The status of FrontClassifier (0/1) when each alert is on
11 CAN_Slow The status of SlowClassifier (0/1) when each alert is on
12 CAN_MTI The status of MTI (0/1) when each alert is on
13 CAN_Dist (m) The relative range when each alert is on
14 CAN_XDist (m) The x-axis relative range when each alert is on
15 CAN_YDist (m) The y-axis relative range when each alert is on
16 CAN_Rel_Speed (m/s) The relative speed when each alert is on
17 CAN_Angle (degree) The angle when each alert is on
18 Upper Red – relative range
Upper Green – relative speed
Lower Blue – BSD alert
Lower thin Green – front classifier
Lower thin Black – slow classifier
Lower thin Red – MTI
19
Video frames corresponding to BSD alerts.
“Start-Middle-End” for one alert situation, “Start-
End-Start-End” for more than one alerts situation.
Content Section
Title Section
Label Area

21
After the final step 8, user can get the summary report, an example is shown as
following figure 3.15 and figure 3.16.
Figure 3.15 the DAQ Label summary report “Summary” sheet sample
All the statistics and summary tables are shown in the first sheet “Summary”, as
shown in figure 3.15. The summary contains the basic DAQ data information (left-up
corner), summary table (right-up corner), and other statistics (alert off host speed
and CAN alert duration). The “Summary” sheet format is not fixed; other detailed
analysis can be added in the future based on user need. The “ALERT OFF HOST
SPEED” and “CAN ALERT DURATION” are currently analyzed.
Figure 3.16 the DAQ Label summary report “Raw_Sum_Results” sheet sample

22
The detailed information about each file goes into other sheets, such as
“FPs_And_FNs”, “Drop_Outs”, and “Other_IOIs”. An example is shown in figure 3.16.
For these sheets, the features listed are consistent with those listed in
“DAQLabelReport”; refer to table 3.7 for more details.
Besides the BSD mode DAQ data labeling, this tool can also be applied in LCA
mode, which is very similar to that of BSD mode but slightly different. Following table
3.8 indicates the step-by-step tutorials for LCA DAQ Data Labeling.
Table 3.8 LCA mode DAQ Data Labeling guidance
Index Description
Step 1 Prepare the “Test Plan Table”, refer to following table 3.9 for guidance
Step 2 Select the “DAQ Data Label Report” option in Regression Mode panel
Step 3 Select the “LCA” check box in Regression Mode panel
Step 4 Browse and select the DAQ folder in Folder Selection panel. (Select but do
NOT go into folder during browsing, refer to 3.a.i for guidance)
Step 5 Select “L/R/B-Side” in Setting Mode panel, keep the default “B-Side” if no
special requirement
Step 6 Calibrate the image position (Optional for LCA mode, almost all LCA data
comes with no videos, refer to 3.a.ii for guidance)
Step 8
Wait for the completeness, and the final “DaqLabelReport” and
“DAQLabelSummaryReport” will be automatically saved into the root
directory of tool folder, and the report name will follow the format
“datafilename_DaqLabelReport_YYYYMMDDhhmmss.xls”, and
“datafilename_SummaryReport_YYYYMMDDhhmmss.xls”
Note that compared with BSD mode, there is no need to “Label” the raw report
shown as the step 7 in table 3.6, because LCA experiments are all controlled
experiment with “Test Plan”. For this reason, one “Test Plan” excel table must be
produced and put together with DAQ data before running. Following figure 3.17,
figure 3.18 and table 3.9 provide users with the relevant guidance.
Figure 3.17 an example of raw driver test plan

23
Shown as in figure 3.17, usually all collected LCA DAQ data comes with a driver-
produced test plan, showing the status of each collected DAQ file. What we need to
do is to convert this fancy table into a plain EXCEL table, as shown in following figure
3.18.
Figure 3.18 an example of converted plain test plan
There are 5 features need to be specified in this converted test plan. The feature
descriptions and converting guidance are shown in following table 3.9.
Table 3.9 Test plan producing guidance
Feature Description Location in raw test plan
File Index The DAQ file index The numbers under “File”
column
Host Speed Planed host vehicle speed The numbers under “H”
column
Target Speed Planed target vehicle speed The number under “T”
column
Curved Road 0 if tested on straight road, 1 if tested on
curved road
The title part of each page
Sensor Side 1-left, 2-right, 0-both (not specified) The title part of each small
table (0 if “Same Lane”)
Event
1-target overtakes host one lane over,
2-target overtakes host two lanes over,
0-target approaches host the same lane
The title part of each page
(red for event 1, blue for 2,
and brown for 0)

24
Note that for step 1, user must make sure that this produced test plan is the only
EXCEL file that put with LCA DAQ data. If there is no test plan or more than one test
plans putting together with the LCA DAQ data, “Error Message Box” with pop out
(Referring to the section C.a.iv Exceptions Control, table 3.5).
following figure 3.19 and figure 3.20.
Figure 3.19 an example of LCA summary report “Summary” sheet
Figure 3.19 an example of LCA summary report “Raw_Sum_Results” sheet
The format of LCA summary report is similar with that of BSD summary report, but
the features are different. Following table 3.10 lists all those features that are put into
the summary report for your reference.

25
Table 3.10 LCA mode DAQ Label Summary Report feature list
Summary Sheet
1 DAQ Data Information Report type, data, date, software, points change, weather
2 Summary Table No. of CAN alerts, FPs, and FNs
3 TTC Descriptive statistics and plots of TTC value
4 Range Descriptive statistics and plots of range value
5 Angle Descriptive statistics and plots of angle value
6 CAN alert duration Descriptive statistics and plots of CAN alert duration
Other Sheets
4 P_Host_Speed Planed host vehicle speed (refer to table 3.9)
5 P_Target_Speed Planed target vehicle speed (refer to table 3.9)
6 P_Curved_Road Planed curved road status (refer to table 3.9)
7 P_Sensor_Side Planed sensor side (refer to table 3.9)
8 P_Event Planed event (refer to table 3.9)
9 No_CAN_Alert Number of LCA CAN alerts
10 CAN_TTC(s) The TTC value when each LCA alert is activated
11 CAN_Dist (m) The range value when each LCA alert is activated
12 CAN_Rel_Speed (m/s) The relative speed when each LCA alert is activated
13 CAN_Angle (degree) The angle value when each LCA alert is activated
14 Duration (sweep) The duration (length) of each LCA alert
15 Host_Speed_On_Off (m/s) The host vehicle speed when each LCA alert is on and off

26
ii. Regression Test – Compare SIMULATION with CAN
The “Regression Report” function can help user simulate the collected DAQ
sensor data using developed software, comparing simulation performance with
CAN performance, therefore improving software. This application can be used to
analyze both BSD and LCA data. Following table 3.11 indicates the step-by-step
tutorials for BSD mode Regression Test.
Table 3.11 BSD mode CAN/SIMU regression test guidance
Index Description
Step 1 (Optional) Put the corresponding “DanView” report together with DAQ data if
it’s available
Step 2 Select the “Regression Report” option in Regression Mode panel
Step 3 Select the “BSD” check box in Regression Mode panel
Step 4 Browse and select the DAQ and XPCDN folders in Folder Selection panel.
(Select but do NOT go into folder during browsing, refer to C.a.i for guidance)
Step 5 Select “L/R/B-Side” and settings file in Setting Mode panel, keep the default
“B-Side” if no special requirement (refer to C.a.iii for guidance)
Step 6 Calibrate the image position (Must do for BSD mode, refer to C.a.ii for
guidance)
Step 8
Open, label and save produced “Raw Regression Report”. It will be
automatically saved into the root directory of tool folder, and the report name
will follow the format “datafilename_RegReport_YYYYMMDDhhmmss.xls”.
(Refer to following table 3.12 and 3.13 for labeling guidance)
Step 9
Select “Summary Report” in Regression Mode panel, and browse to select the
labeled “Raw Regression Report” in step 8. Press “Run” button, and then the
final summary report will be saved into the root directory of tool folder
Figure 3.20 an example of “DanView” report “Data” sheet

27
Note that the step 1 is optional because sometimes the “DanView” report is not
available when doing regression test. Specialist will use certain software to analyze
the collected videos frame by frame, and produce detailed report. This report is
usually more accurate in terms of LATE/EARLY/STICKY LED issues than regression
test report. So the regression test will extract those labeled information from
“DanView” report directly if it’s available. Above figure 3.20 shows an example of the
“Data” sheet in the “DanView” report, where every alert is analyzed and labeled.
Also note that in step 8, an example of “Raw Regression Report” is shown as
following figure 3.21. The report contains title section and content section. Each data
entry in the content section represents one side of one DAQ file. User can drag up
and down the scroll bar to view the processing results of all DAQ files. The title
section will be fixed during scrolling. Note that the yellow marked area is the so-
called “Label Area”, which contains 4 cells for each data entry, namely the “EVENT”,
“TARGET”, “CAN_LABEL”, and “SIMU_LABEL”. User need to verify and edit these 3
cells based on the signal plots and video frames. All information of this report is listed
in table 3.12 and the labeling rules are introduced in table 3.13.
Figure 3.21 an example of raw regression report
Note that the base color for each DAQ file entry is gray. If one side of DAQ file is
searched and found in the “DanView” report, all its relevant information, such as
EVENT, TARGET, and Comments will be extracted to regression report. These file
entries will be painted as green, which means that user can trust “EVENT”,
“TARGET”, and “CAN_LABEL” content during labeling. All features in this raw
regression report and the labeling guidance are listed in following table 3.12 for your
reference.
Label Area Green: DanView Info.
Gray: Regression Info.

28
Table 3.12 BSD mode raw regression test report feature list
4 EVENT FE-front entry; RE-rear entry
5 TARGET C/S/V-car, SUV, or van; SEMI-Semi. Truck
6 CAN_LABEL FP/FN/drop out/Late/out early/on early/sticky (table 3.13)
7 SIMU_LABEL FP/FN/drop out/Late/out early/on early/sticky (table 3.13)
8 No_CAN_Alert Number of CAN alerts
9 No_Simu_Alert Number of SIMULATION alerts
10 CAN/Simu_Mismatch (sweep) The alerts on/off mismatches between CAN and Simulation
11 CAN/Simu_Duration (sweep) Duration of CAN/SIMU alerts
12 CAN_Front The status of FrontClassifier (0/1) when CAN alert is on
13 CAN_Slow The status of SlowClassifier (0/1) when CAN alert is on
14 CAN_MTI The status of MTI (0/1) when CAN alert is on
15 SIMU_Front The status of FrontClassifier (0/1) when SIMU alert is on
16 SIMU _Slow The status of SlowClassifier (0/1) when SIMU alert is on
17 SIMU _MTI The status of MTI (0/1) when SIMU alert is on
18 CAN_Dist (m) The relative range when CAN alert is on
19 CAN_XDist (m) The x-axis relative range when CAN alert is on
20 CAN_YDist (m) The y-axis relative range when CAN alert is on
21 CAN_Rel_Speed (m/s) The relative speed when CAN alert is on
22 CAN_Angle (degree) The angle when CAN alert is on
23 CAN_Host_Speed_On_Off The host vehicle speed when CAN alert is on and off
24 SIMU_Dist (m) The relative range when SIMU alert is on
25 SIMU_Rel_Speed (m/s) The relative speed when SIMU alert is on
26 SIMU_Angle (degree) The angle when SIMU alert is on
27 SIMU_Host_Speed_On_Off The host vehicle speed when SIMU alert is on and off
28 Upper red line – CAN relative range
Upper colorful circles – SIMU relative range
Upper green line – CAN relative speed
Upper colorful triangles – SIMU relative speed
Lower dotted/solid blue – CAN/SIMU BSD alert
Lower dotted/solid green – CAN/SIMU front classifier
Lower dotted/solid black – CAN/SIMU slow classifier
Lower dotted/solid red – CAN/SIMU MTI
29
Video frames corresponding to BSD alerts.
“Start-Middle-End” for one alert situation, “Start-
End-Start-End” for more than one alerts situation.
Upper frame – Corresponding to CAN alerts
Lower frame – Corresponding to simulation alerts

29
Following table 3.13 shows the detailed rules about labeling, user should follow
these terms because program will read and recognize these terms when generating
summary report.
Table 3.13 BSD mode raw regression report label rules
ID Scenario Description
1 FPs There are alerts but no target (sometimes need to verify with
videos when target too fast, late instead of FP)
2 FNs There are target but no alerts (can’t find all FNs, will not display all
frames when there is 1 alert but 2 targets for example)
3 Drop outs Same target but more than one alert (sometimes need to check
video when there are multiple targets and alerts)
4 Late Light on late (frame not calibrated yet, judge by experience)
5 Sticky (LED) Light off late (frame not calibrated yet, judge by experience)
6 (Light) out early Light on early (frame not calibrated yet, judge by experience)
7 (Light) on early Light off early (frame not calibrated yet, judge by experience)
8 NA No alerts, no targets; or there are no videos
Note that in table 3.13, user can use the words under “Scenario” column, content
inside the bracket is optional, but words outside are key words that will be scanned
when generating summary report.
After the final step 9, user can find the summary report in the same directory as that
of raw regression report, as shown in following figure 3.22.
Figure 3.22 an example of summary report “Summary” sheet

30
Besides the BSD mode CAN/SIMU regression test, this tool can also be applied in
LCA mode, which is very similar to that of BSD mode but slightly different. Following
table 3.14 indicates the step-by-step tutorials for LCA CAN/SIMU regression test.
Table 3.14 LCA mode CAN/SIMU regression test guidance
Index Description
Step 1 Prepare the “Test Plan Table”, refer to table 3.9 for guidance
Step 2 Select the “Regression Report” option in Regression Mode panel
Step 3 Select the “LCA” check box in Regression Mode panel
Step 4 Browse and select the DAQ and XPCDN folders in Folder Selection panel.
(Select but do NOT go into folder during browsing, refer to C.a.i for guidance)
Step 5 Select “L/R/B-Side” and settings file in Setting Mode panel, keep the default
“B-Side” if no special requirement (refer to C.a.iii for guidance)
Step 6 Calibrate the image position (Optional for LCA mode, almost all LCA data
comes with no videos, refer to C.a.ii for guidance)
Step 8
Wait for the completeness, and the final “RegReport” and “SummaryReport”
will be automatically saved into the root directory of tool folder, and the report
name will follow the format
“datafilename_RegReport_YYYYMMDDhhmmss.xls”, and
“datafilename_SummaryReport_YYYYMMDDhhmmss.xls”
Note that compared with BSD mode, there is no need to “Label” the raw report
shown as the step 8 in table 3.11, because LCA experiments are all controlled
experiment with “Test Plan”. This part is same with the “DaqLabel” application
presented in section C.b.i. Refer to figure 3.17, figure 3.18 and table 3.9 for guidance.
following figure 3.23.
Figure 3.23 an example of LCA summary report “Summary” sheet

31
The format of LCA summary report is similar with that of BSD summary report, but
the features are different. Following table 3.15 lists all those features that are put into
the summary report for your reference.
Table 3.15 LCA mode regression test summary report feature list
Summary Sheet
1 DAQ Data Information Report type, data, date, software, points change, weather
2 Summary Table No. of CAN alerts, FPs, and FNs
3 TTC Descriptive statistics and plots of TTC value
4 Range Descriptive statistics and plots of range value
5 Angle Descriptive statistics and plots of angle value
6 CAN/SIMU alert mismatch Descriptive statistics and plots of CAN/SIMU alert mismatch
7 CAN/SIMU alert duration Descriptive statistics and plots of CAN/SIMU alert duration
Other Sheets
4 P_Host_Speed Planed host vehicle speed (refer to table 3.9)
5 P_Target_Speed Planed target vehicle speed (refer to table 3.9)
6 P_Curved_Road Planed curved road status (refer to table 3.9)
7 P_Sensor_Side Planed sensor side (refer to table 3.9)
8 P_Event Planed event (refer to table 3.9)
9 No_CAN_Alert Number of LCA CAN alerts
10 No_SIMU_Alert Number of LCA SIMU alerts
CAN/SIMU_Mismatch (sweep) The alerts on/off mismatches between CAN and Simulation
CAN/SIMU_Duration (sweep) Duration of CAN/SIMU alerts
10 CAN_TTC(s) The TTC value when CAN LCA alert is activated
11 CAN_Dist (m) The range value when CAN LCA alert is activated
12 CAN_Rel_Speed (m/s) The relative speed when CAN LCA alert is activated
13 CAN_Angle (degree) The angle value when CAN LCA alert is activated
14 CAN_Host_Speed_On_Off The host vehicle speed when CAN LCA alert is on and off
15 SIMU_TTC(s) The TTC value when SIMU LCA alert is activated
SIMU _Dist (m) The range value when SIMU LCA alert is activated
SIMU _Rel_Speed (m/s) The relative speed when SIMU LCA alert is activated
SIMU _Angle (degree) The angle value when SIMU LCA alert is activated
SIMU _Host_Speed_On_Off The host vehicle speed when SIMU LCA alert is on and off
iii. Regression Test – Compare SIMULATION with SIMULATION
Not Finished Yet

32
D. Software Structure (For Tool Developer)
In this section, the entire software structure and some important files of this tool are
presented in the perspective of tool developers, which will facilitate maintaining and
updating this tool for future developers. Tool users can only refer to section C, and skip this
section. Following figure 4.1indicates the basic software structure.
Figure 4.1 the software structure of regression test tool
In terms of the tool application, in the vertical direction of above figure, there are 4 modes,
namely (1) DAQ Data Label Report Mode, (2) Regression Report Mode, (3) Comparison
Report Mode, and (4) Summary Report Mode. All of these 4 applications have been
presented in section C. In the perspective of tool developer, all the tool programs can be
categorized into 3 layers, namely “GUI Layer”, “Data Layer”, and “Report Layer”. The
functions in different layers are independent with each other, which will facilitate the
debugging and updating process.
In following table 4.1, the descriptions about each layer and the classification of main
functions/scripts are presented for the reference of developer.

33
Table 4.1 introduction of software structure
ID Layer Description Main relevant functions
1 GUI
Creating user friendly
interface
regressionGUI.m; settingTableSupport.m;
build.m; cvMake.m
regressionMain.m
2 Data
Loading DAQ data and
reprocessing DAQ data
recalculationRep.m; daqLabelRep.m;
read_can_file.mexw64; readCV.m;
getSensorA4Function_Reg; pcdn_support.m;
readvideoheader.m
daqLabelRepMain.m; recalRepMain.m
3 Report
Generating the raw
regression/DAQ label
report and the summary
report
daqLabelRepBSD.m; daqLabelRepLCA.m;
excelExportDaqLabelBSD.mexw64;
excelExportDaqLabelLCA.mexw64;
recalRepBSD.m; recalRepLCA.m;
excelExportBSD.mexw64;
excelExportLCA.mexw64;
summaryRep.m; summaryRepMain.m;
summaryRepBSDDaqLabel.m;
summaryRepLCADaqLabel.m;
summaryRepBSD.m; summaryRepLCA.m;
xlswrite.m
Note that in table 4.1, some functions connecting two layers are the interface functions
between two layers, such as the “regressionMain.m” between the GUI layer and data layer.
The GUI layer takes in user’s inputs, packaged and passed them to the data layer. The
data layer takes in user’s inputs, load in DAQ data and reprocessing the DAQ data
based on user inputs. The reprocessing results and some user inputs are passed into
the report layer. The report layer takes in those reprocessed data and user inputs,
generating reports accordingly.

34
a. Graphical User Interface (GUI)
The main GUI function is “regressionGUI.m”. Another GUI function
“settingTableSupport.m” is used to generate the “POINTS CHANGE” sub-GUI.
These two GUI files are not automatically generated using MATLAB tool but
programmatically created. They follow the standard MATLAB GUI script structure,
namely, initialization, components, callbacks, and utility function. All sections are
clearly commented in the script.
In initialization part, the default value and structure of some important variables,
especially global variables are set. In components part, the position and layouts of all
components (pushbutton, radio button, axe etc.) are set. In callbacks part, the
callback functions of all components are programmed. In utility function part, all other
relevant functions are set. Developers can check different sections based on
debugging needs.
b. DAQ Data Loading and Processing
This part is the core part of entire regression test, containing the DAQ data
reprocessing algorithms. DAQ data need to be loaded in before being processed,
which is a straight-forward step but may cause various errors. There are 3 types of
DAQ data, namely the videos (AVI), CAN data (ASC), and sensor data (DAT). The
data reading functions and their descriptions are presented in following table 4.2.
Table 4.2 introductions of data read functions
Data Type Function Description
Video data readCV.m
readvideoheader.m
readCV.m reads specific frame of the video;
readvideoheader.m read the video information
CAN data read_can_file.cpp
read_can_file.mexw64
Read the raw .asc file as a structure containing
necessary CAN signals, it’s updated from
read_can_file.m file
Sensor data read_file.m
SPPlugin_Release
It is in the XPCDN folder, do not need to update
Note that the calling of these functions can be found in recalculationRep.m file and
daqLabelRep.m file.
The DAQ data reprocessing part is kept consistent with that of XPCDN software. The
processing structure is shown in following figure 4.2. The core function
“sensorA4_projectname_dev.mex64” can be build and found in the XPCDN folder. The
output will be sent to the “report layer” for further signal processing. There are 3 inputs for this
function, CAN data, sensor data and settings data. CAN data is directly read from ASC file,
and sensor data is from DAT file with some signal reprocessing and time stamp tuning work.
The most critical one is the settings data. As shown in figure 4.2, the settings can be from
DAQ data, software, or user input. User can make selection in the “Settings Mode” panel of
the GUI. The program basically follows this structure to generate inputs, reprocess data, and
produce outputs.

35
Figure 4.2 the DAQ data reprocessing structure
c. Image Processing – OpenCV Toolkit
Similar with the data visualization in XPCDN software, video frames will be extracted
and displayed for the regression test report, especially for the BSD mode. MATLAB
functions “VideoReader” and “read” are used in XPCDN software to show the video
frames of each DAQ file. However, these two functions are too slow when
processing hundreds of files. For this reason, the famous OpenCV library is brought
in and applied in regression test tool, which significantly improving the batch
processing efficiency. All of these functions are compiled MEX functions. Following
table 4.3 list a few of them used in video frame reading process.
Table 4.3 video reading OpenCV functions
function Example Description
VideoCapture videoObj =
cv.VideoCapture(videofilename);
Read in any given video files, and
output a video object
get NumberOfFrames =
videoObj.get(‘FrameCount’);
Get the video file information, such as
number of frames, size, time, etc.
delete videoObj.delete; Delete the video object created
set videoObj.set(‘PosFrames’, 100); Set the position of video reading
pointer, and other properties
read Frame = videoObj.read(); Read the frame out pointed by the video
frame pointer
These functions are easy to use and very fast. Refer to the website
http://www.cs.stonybrook.edu/~kyamagu/mexopencv/ for more information about
other functions, such as edge detection, blur, and resize.

36
All OpenCV source CPP files are put in the folder “mexopencv-master”, and they will
be compiled together to produce MEX files during the “Build” process (refer to
section B tool setup). These MEX files can therefore be easily used in MATLAB
script just as other MATLAB function. Also note that building these OpenCV image
processing CPP files to produce MEX files need the existence of lower layer
OpenCV package. This package is open source and saved in “opencv” folder in the
root directory.
d. EXCEL Interface – Object Linking and Embedding (OLE) Automation
Technique
This technique is developed and applied to improve the report generating speed.
MATLAB has a special COM application to communicate with window applications,
such as EXCEL, WORD, and PPT. But still, it’s too slow when importing hundreds of
image frames and plots from MATLAB to EXCEL. So the OLE technique is utilized to
solve this problem.
The OLE technique basically provides windows application developers with an
interface (a bunch of functions) to windows applications such as EXCEL and WORD.
It’s developed using C/C++, and developers can use them to easily create, read, and
manipulate EXCEL or WORD files. It works under the similar principles with the
MATLAB COM but allows user to take advantage of existing windows applications’
functionality directly using C/C++.
Take one file as an example, “excelExportBSD.cpp”. Open this file in Visual Studio,
developer can see a function “OLEMethod”, which is the core interface function used
to activate various EXCEL/WORD functionalities. Following this function, developer
can see a bunch of higher level functions such as “SetExcelSheetName”,
“HorizontalAlignment”, and “SetExcelValue”. These functions are developed based
on the basic function “OLEMethod” to encapsulate some frequently used operations.
Following the structure and template of existing functions, developer can create their
own new functions to manipulate the EXCEL/WORD file. For more information, refer
to the website http://www.codeproject.com/Articles/34998/MS-Office-OLE-
Automation-Using-C and http://msdn.microsoft.com/en-us/library/aa908601.aspx
e. Parallel Computing
This parallel computing is nothing new. I utilized the popular MATLAB parallel
computing toolbox to accelerate the DAQ data reprocessing. Hundreds of DAQ files
have to be reprocessed one-by-one without this toolbox, but this toolbox allows
developer to call several cores and reprocess more than 1 file simultaneously. Note
that not all users has parallel computing toolbox installed with their MATLAB, so the
program will at first search for the existing toolbox and will switch to original serial
mode automatically if no such toolbox is found.

37
E. Future Work
In this section, some future works are present. These works include unfinished functions,
and some advanced challenging tool improvements.
a. SIMU to SIMU Regression Test
This application is designed to compare the performance of two simulations.
Currently the simulation results can be compared with the CAN results. But
sometimes engineers will need to compare the results from two simulations. Of
course this can be done by using the CAN results as a benchmark, but it would be
nice if some kind of report can be generated to directly compare two simulations’
results. The DAQ data reprocessing part will not be changed, the challenging part is
to design the new comparison report.
b. Vehicle Image Recognition
So far, as shown in section C, user may have to label the BSD mode report manually,
which is sometimes time consuming, especially for those rainy data. This vehicle
image recognition technique is developed trying to reduce user’s labeling workload,
improving the tool efficiency. The principle is to borrow some mature object detection,
edge detection, and other image processing algorithms to pre-determine if there is
target during alert, or the target position. It will not be 100% accurate, but can give
user some valuable information during the labeling work.
c. COM Object Packaging
This work is designed to facilitate the tool installation and transferring process.
Currently this tool has to be copied and pasted if new user wants to use it. MATLAB
COM Object Packaging tool can help producing a single MATLAB installation file. A
simple double-click will install this file as an embedded MATLAB application.

User Guide of Regression Test & Validation Tool (v1.0)

Recommended

Recommended

More Related Content

What's hot

What's hot (9)

Similar to User Guide of Regression Test & Validation Tool (v1.0)

Similar to User Guide of Regression Test & Validation Tool (v1.0) (20)

User Guide of Regression Test & Validation Tool (v1.0)