2006-01-1600 Application of Robust Engineering Methods to ...


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

2006-01-1600 Application of Robust Engineering Methods to ...

  1. 1. SAE TECHNICAL PAPER SERIES 2006-01-1600 Application of Robust Engineering Methods to Improve ECU Software Testing Eldon G. Leaphart, Steve E. Muldoon and Jill N. Irlbeck Delphi Corporation 2006 SAE World Congress Detroit, Michigan April 3-6, 2006 400 Commonwealth Drive, Warrendale, PA 15096-0001 U.S.A. Tel: (724) 776-4841 Fax: (724) 776-5760 Web: www.sae.org
  2. 2. The Engineering Meetings Board has approved this paper for publication. It has successfully completed SAE's peer review process under the supervision of the session organizer. This process requires a minimum of three (3) reviews by industry experts. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of SAE. For permission and licensing requests contact: SAE Permissions 400 Commonwealth Drive Warrendale, PA 15096-0001-USA Email: permissions@sae.org Tel: 724-772-4028 Fax: 724-776-3036 For multiple print copies contact: SAE Customer Service Tel: 877-606-7323 (inside USA and Canada) Tel: 724-776-4970 (outside USA) Fax: 724-776-0790 Email: CustomerService@sae.org ISSN 0148-7191 Copyright  2006 SAE International Positions and opinions advanced in this paper are those of the author(s) and not necessarily those of SAE. The author is solely responsible for the content of the paper. A process is available by which discussions will be printed with the paper if it is published in SAE Transactions. Persons wishing to submit papers to be considered for presentation or publication by SAE should send the manuscript or a 300 word abstract to Secretary, Engineering Meetings Board, SAE. Printed in USA
  3. 3. 2006-01-1600 Application of Robust Engineering Methods to Improve ECU Software Testing Eldon G. Leaphart, Steve E. Muldoon and Jill N. Irlbeck Delphi Corporation Copyright © 2006 SAE International ABSTRACT software test methods using robust engineering are described. The orthogonal array test approach identifies Robust Engineering techniques developed by Taguchi noise factors for a specific algorithm anomaly condition have traditionally applied to the optimization of as opposed to the typical usage with assigned control engineering designs. Robust Engineering methods also factors. The noise space or range of noise factor may be applied to software testing of ECU algorithms. values is identified through a set of iterative tests. Thus, The net result is an approach capable of improving the an algorithm solution may be formulated which is robust software algorithm in one of two ways. First the over the range of noise factors that may be present as approach can identify the range of areas which prove opposed to a single anomaly condition. The overall goal problematic to the software such that a robust solution of using robust methods during the design phase is to may be developed. Conversely, the approach can be develop a design robust to external conditions early in used as a general strategy to verify that the software is the design phase. Given that testing always lags robust over the range of inputs tested. The robust completion of a given design, the goal of using the engineering methods applied to software testing utilize robust methods for ECU software testing is to more orthogonal array experiments to test software over a thoroughly test the design, thus preventing anomalies range of inputs. The actual software trials are best from surfacing during actual field operation. performed in the simulation environment and also via automated test hardware in the loop configurations in SOFTWARE DEVELOPMENT CYCLE realtime. This paper outlines a process for applying Robust Engineering methods to software testing. As a background it is useful to review the traditional “V” Experiments derived from the robust methodologies are software development cycle to indicate how it relates to then implemented using both software simulation and robust engineering methodologies. An example of the hardware in the loop environments. A case study is “V” diagram is shown in Figure 1. The left side of the “V” presented where the process is applied to improve the diagram describes the process of requirements capture robustness of an ECU software algorithm. Results are along with the various design phases of a project. The obtained from both using simulation and performing step of implementing the desired design is shown at the automated hardware in the loop tests. bottom of the diagram, and testing phases are shown on the right side of the diagram. Once the design requirements are defined, implementation is performed followed by testing phases to show that the completed INTRODUCTION implementation meets the original requirements. Iteration from the testing phase back to design and Robust Engineering techniques are typically applied to subsequent implementation may be necessary when optimize of engineering designs. Concepts including test results yield problems or show that requirements are parameter/control factor optimization experiments, two not met. step optimization analysis, prediction and confirmation tests, etc. have widely been used to improve robustness During the design phase of a project, robust engineering of hardware/mechanical, software, and process designs. principles may be applied to develop the design and In addition to using robust techniques throughout the show that performance is robust across chosen noise design phase, the test phase of a product can also factors. This implies that software control factors benefit from the application of robust methods. Using including logic constructs, performance gains, robust design principles specifically for software test calibrations, etc. are identified and varied to achieve the requires tailoring some of these methods due to unique proper output response. Output measurements are constraints of software. Consequently these differences performed such that signal / noise strategies are between traditional robust design processes vs. developed, calculated, and optimized. 1
  4. 4. The software test phase presents some unique characteristics that must be considered prior to applying robust engineering principles. Three characteristics that are discussed here are the following: DESIGN TEST PHASE PHASE • Software content is constant • Software performance is quantified as Pass/Fail • Software test effectiveness is a function of test input Robust Robust The first characteristic to consider is that software Design Test content either during test or over the lifetime of the Methods Methods product is constant. Ideally, software is not subject to IMPLEMENTATION noise factors such as environment, aging, and PHASE manufacturing. As such, during the testing phase, parameters of the software are not viewed as factors that may be modified. Figure 1: Software Development “V” Cycle The second characteristic to consider is that of measuring software performance. Although there may be several outputs which can be observed and measured, the software test phase only considers that ROBUST ENGINEERING SOFTWARE TEST the function under test passed or failed; e.g. PROCESS requirements are met or not met. This implies that for testing, signal to noise robust strategies are computed The traditional Robust Engineering design process may as a function of number of failures observed rather than be outlined in nine general steps. These steps are the measurement of an output variable from the process. following: Finally, the third characteristic to consider for this 1. Define scope of the project discussion is that software testing effectiveness is a 2. Define boundary of subsystem function of test inputs. Software is typically designed to perform a specific task or exhibit a specific response for 3. Define ideal function with input M, output y a given a set of input conditions. Software is often tested identified to insure that the desired functionality performs as 4. Develop signal and noise strategies intended. A more difficult task is to show that the software under test does not perform certain tasks 5. Define control factors and levels incorrectly for any possible set of inputs or operating 6. Formulate experiment conditions. Software can be tested exhaustively with a 7. Conduct experiment / data collection narrow set of inputs and shown to pass, only to fail or exhibit some undesirable behavior once in the field. This 8. Data Analysis (S/N ratios, sensitivity, response is because the field usage produces some sequence or tables, 2 step optimization, predictions) input characteristic which was not expected. Thus, some 9. Perform confirmation experiments and prior knowledge may be required to identify conditions document that the software is vulnerable to and thus can be tested against. Knowledge of how the software will be used in For software test, some of the traditional design steps the field is also extremely useful for identifying test are slightly modified to consider the software inputs. Once these conditions are known, subsequent characteristics described in the previous section. The tests may be developed. In summary some analysis of modified steps include definition of the basic or “ideal” the desired functionality, knowledge of possible function, a modified P-Diagram, formulation of problems and typical usage are recommended to serve experiment using the Orthogonal Test Array, iteration as starting point for robust software test development. step and test evaluation. Recognizing that software testing involves these unique BASIC FUNCTION characteristics, minor modifications to the traditional robust engineering design methods are proposed. The The first item to note differences between design and next section outlines this modified process. test is the ideal function. The ideal function is defined as the ideal relationship between input signal M and output response y based on the physics of the system under study. The output response is described as linear with the gain beta (β) assigned. With an expression defined as y=β M, the output response is measurable in objective 2
  5. 5. quantities, thus signal / noise ratios may be obtained. A MODIFIED P – DIAGRAM traditional ideal model is shown in Figure 2 below. A Parameter or P Diagram approach is used to describe the relationship between input signal, output response, control factors and noise factors for an engineering system. A general example of a P-Diagram is shown below in Figure 4. Part, y M Process, input output Function y β Noise Factors Control Factors environment Part, Process, aging Function manufacturing parameters M Figure 4: Parameter (P) - Diagram Figure 2: Traditional Ideal Function For software test purposes, Noise and Control factors are viewed differently and result in a modified P – Software testing requires an ideal function Diagram. Reiterating, during the test phase and field representation which takes into consideration an output operation software content is constant. Thus control response y which is either pass or fail. The resulting factors are not considered as variables other than relationship between y and M is nonlinear and a function possibly operating modes of the software. Also, in the of software logic designed to meet a requirement. Figure ideal sense, software is not subject to the typical noise 3 shows an example of the software test ideal function. factors of environment, aging, and manufacturing. It should be that noise factors still apply, however, they may be modeled as disturbances which add directly to the input signal M. These noise disturbances contribute to undesirable behavior in some cases. The goal of robust engineering applied to the software test phase is to identify areas where the software is not robust to a set pass of noise factors affecting the input, or conversely, show that for a given range of input signal noise disturbances, that the software exhibits no undesirable output. Figure 5 illustrates a Modified P-Diagram for the software y testing phase. fail Software y M Σ input Function output M Noise Factors Control Factors frequency program modes Figure 3: Software Test Ideal Function magnitude timing sequences etc Figure 5: Modified P-Diagram for Software Testing 3
  6. 6. ORTHOGONAL ARRAY EXPERIMENT SETUP software redesign, all orthogonal array iterations may be re-run to serve as a confirmation that software changes The experiment setup for both robust engineering design improve performance in the areas of concern. Most of phases and software test phases utilize the orthogonal the steps outlined in the iteration process also apply for array. An orthogonal array provides for a balanced set of the second approach where noise factor levels are experiment trials. This allows for conclusions to be selected from general observations of the software reached in a balanced manner. Typically, control factors scope. The exception is that iteration step 2 is not identified from the P-diagram are assigned to the required since this method assumes that a specific issue orthogonal array for experimentation. Software testing has not been identified from which to start from. requires that the noise factors identified from the Modified P-Diagram be assigned to the orthogonal array. The more formidable task with software test TEST EVALUATION experimental setup is determining what levels of noise factors to begin with to perform the experiment. The The final step with the robust engineering software test process of identifying the noise factor levels is process is to quantify the experimental test results for paramount to exploring the effects of the noise space the orthogonal array trials. For traditional robust design against software performance. Two approaches are projects, signal to noise ratio and sensitivity calculations suggested as part of this discussion. Iteration around are computed from the measured output responses of conditions producing a known or prior software anomaly each experimental trial. For the robust engineering is a more direct approach. A second approach is to software testing process, the measured results are in the identify a range of noise factors based on general discrete form of pass / fail or 0 / 1. observation of characteristics of the software objective under test. Both approaches require iteration to either Two approaches are available to quantify experimental hone in on the range of noise factors that is problematic, results in the discrete form, two way interaction or adequately confirm that the software performs response tables and signal to noise ratio computed as a adequately across all noise factors. function of number of failures. The response table approach requires that the number of failures for each 2 The iteration process for the first method where a known factor interaction of the orthogonal array test i.e. (Ai Bj) problem has been observed and analyzed is detailed be summed. The sum total for a two way interaction with the following steps. representing 100% failure is defined by the expression: 1. Identify noise factor parameters based on Size of Orthogonal Array / # of combinations of (Ai Bj) analysis of known problematic input. Metrics associated with the effectiveness of the software 2. Define for the first row of the Orthogonal Array testing may be obtained by totaling the number of 100% with noise factor levels which will reproduce the two way interaction failures, or simply tabulating total software anomaly. The measured response numbers of failures for the orthogonal array. An example should indicate a fail condition for this trial. template of a response table for an L8(27) orthogonal array test is shown below in Figure 6. 3. Perform remaining trials of Orthogonal Array covering all combinations as defined by the size A1 A2 B1 and type of the array. B2 Record data in B1 B2 terms of % failure 4. Based on results from Orthogonal Array Test C1 C2 change range of noise factor parameters to C1 C2 bound the problem area. This could either D1 require expanding the noise factor range around D2 D1 D2 the initial levels or concentrating or a tighter E1 range. E2 E1 E2 F1 5. Repeat steps 3 & 4 and tabulate experiment F2 results. Successive iterations should indicate F1 F2 G1 where noise factor levels affect the intended G2 software performance. Figure 6: Response Table for 2 Way Interactions Based on the orthogonal array test results, the software may be declared robust across the range of noise factors identified or conversely be subject to redesign to address areas which produced failures. In the case of a 4
  7. 7. An alternate method for quantifying discrete orthogonal experiment phase for software testing. Two methods array tests using signal to noise ratio calculations is used are pure software simulation and hardware in the described in [3]. In this approach, the total number of loop tests. errors vs. array size is computed. A signal to noise gain is computed from the expression shown as: SOFTWARE SIMULATION TEST ENVIRONMENT S/N = -20 Log10 [(Total Defects)/n + 1] Simulation of software can be defined as the capability to model and execute logic constructs representative of This expression will yield a S/N ratio of 0 db for a test realtime algorithms or processes. The simulation is without error. Tests which produce errors will result in a performed in processing environments that may differ negative S/N ration gain. This type of response is typical from the intended target application ie. microprocessor, of a “maximum the better” or “fewer errors the better” programming language, execution speed, etc. however, experiment design. Either signal to noise ratio or the calculation results should be equivalent. A response table calculations can be used to quantify simulation environment is ideal for developing test software test results during iteration steps or to show stimuli and evaluation criteria per robust engineering improvement between redesign confirmation trials. methods for software test. Matlab™ / SIMULINK™ simulation tools (from The MathWorks Inc.) are one such PROCESS SUMMARY tool used to simulate and test software. In summary, a process for Robust Engineering Software SIMULINK SIMULATION Test is defined. The process is based on Robust Engineering design steps, but differ slightly in their Matlab / SIMULINK is a flexible computational and application due to the unique characteristics of software. simulation tool capable of modeling a variety of physical Careful analysis of the software function under test is or software processes. Software can be modeled by recommended to select a set of noise parameters which constructing graphical building blocks to implement an will be most beneficial for providing desired test intended function or by executing compiled C language coverage. This process may require iteration if prior source code within a mechanism referred to as an s- knowledge of potential issues with the software / design Function. The method of using SIMULINK s-Functions to are less known. The advantage of using this method for implement compiled production C source code is software test is to show software compliance or thoroughly discussed in [6]. robustness over a wide range of inputs as opposed to testing at a single operating point. The Robust The SIMULINK environment allows for software modules Engineering Software Test process is summarized by or programs under test to be embedded into a single S- the following steps: function block. Additional blocks may be added to produce input signals and monitor output for the 1. Define scope of the project software under test. Figure 7 is shows a model layout 2. Define boundary of subsystem which has been constructed for test development 3. Define Pass / Fail function and Modified P - Diagram 4. Analyze input signal and input timing sequence characteristics for project and define noise factor 5. Determine initial noise factor levels 6. Formulate Orthogonal Array experiment 7. Conduct experiment / data collection via simulation or Hardware In Loop configurations 8. Data Analysis (2 way interaction response tables, S/N ratios, predictions) 9. Iterate steps 7 & 8 with modified noise factors Figure 7 : SIMULINK Test Model Layout 10. Perform re-design phase if necessary; address issues identified across noise space 11. Perform confirmation run; Repeat experiments ORTHOGONAL ARRAY SOFTWARE TEST with same iterations of noise factors to compute DEVELOPMENT improvements. The next section of this paper further details methods to Three elements required for simulation of an orthogonal implement and automate the orthogonal array array software test include input data sequences, 5
  8. 8. software under test, and the evaluation criteria. The modeling mode. Simulation models can be created using process for robust engineering software testing requires the C/C++ programming language or imported from that noise factor combinations be constructed as input to standard math tools like Matlab / SIMULINK. The the software under test. These input data characteristics simulation model is typically run in real time, executing such as frequency, magnitude, timing, sequence, etc the defined parameters and characteristics required for can be easily modeled with the available SIMULINK the targeted ECU environment, while the AutoSim input constructs. Data to be used as test stimuli may be and output hardware (designed for common automotive created given a description of desired characteristics or type signals (analog, digital, PWM, and serial requirements. Alternatively, actual field data may be communications), complete the simulator to ECU input / imported and applied to the software. The set of data output interface. A powerful and useful feature of the inputs can be easily arranged or scheduled per the AutoSim in a testing or development environment is the sequence as specified by the orthogonal array. ability to run user defined test scripts. These test scripts are typically created using C language and compiled for The next element required for the simulation of the direct execution on the AutoSim platform. The test orthogonal array software test is the actual software scripts allow complex combinations of input and output subject to test. SIMULINK offers flexibility in this area signals, as well as serial communications messages (i.e. relative to the representation of this software. The CAN, J1850, etc.) to be sequenced into the simulation in software may be modeled using the graphical blocks or a controlled and repeatable manner. conversely the s-Functions using compiled code may be used. The s-Function approach has advantages in that the source code for the target application is equivalent for a large extent to that executed in the model. This HIL SETUP DESCRIPTION eliminates possible error which may be introduced as a function of reproducing the code with the graphical block The HIL simulation set-up consists of the Pi AutoSim approach and may be more time efficient to develop a simulator chassis unit with I/O card complement, a “host” completed model. or user interface PC connected to the AutoSim, an automated signal line break-out box (AutoBOB), and a Lastly, data which propagates from the input blocks target ECU with wire harness. The AutoSim I/O and through the software s-Function to produce a desired ECU are powered from an external DC power supply. result must be evaluated per a pass / fail criteria. The DC supply provides internal power for the AutoSim Evaluation of this data output could be performed by I/O cards, as well as a source for controlled DC power visual inspection or analysis, however, it is much more for the ECU. The host PC is connected to the internal efficient to develop a routine to evaluate the pass fail AutoSim single board PC computer via ethernet. The criteria automatically. Orthogonal array testing requires host PC provides the user with a graphical user interface several successive trials to be conducted depending on (GUI) PC software application from which the simulation the matrix size. A devised routine to evaluate pass / fail modeling, testing script, and I/O configurations of the results, may be scripted to run as a part of each trial AutoSim are controlled. The ECU must be connected to simulation. the AutoSim through a specifically designed wiring harness, which matches the ECU signal lines to the With the three elements of data input, software, and required I/O card connections on the AutoSim chassis. evaluation criteria modeled, each orthogonal array trial The signal line wiring harness is routed through an may be run individually, or a script sequence may be automated break-out box or “AutoBOB” which is placed developed which performs all orthogonal array trials in between the ECU and the AutoSim I/O. The AutoBoB is successively for a given matrix in simulation. The a model or script controlled device capable of providing simulation environment serves as a test bed to develop DC power bus shorting and open circuit conditions for test requirements and confirm results. The components the signal lines. Serial communications interface developed in the simulation environment may be ported devices and ECU specific instrumentation are also to the HIL environment to complete testing of the accommodated as needed or required. Data results software with the target hardware. from executed test scripts are generally stored in a text file format. Figure 8 shows the HIL and Pi AutoSim hardware setup. HARDWARE IN LOOP TEST SIMULATION PI AUTOSIM CAPABILITY The Pi AutoSim is a bench-top simulator developed by Pi Technology Corp designed specifically for testing automotive electronic control units (ECUs) in a hardware-in-the-loop (HIL) environment. The simulator can be configured to be used in an open or closed loop 6
  9. 9. 5. Design the AutoSim test script logic flow in a manner that automatically cycles through the input trials in defined by the orthogonal array, covering all array combinations. PROBLEM ANALYSIS ROBUST A L16 (45) Orthogonal Array Test B C D E Determine Pass Fail Criteria glitch Actual steering signal initial glitch glitch voltage ENGINEERING duration cal Slope Result freq / amplitude voltage magnitude No. (loops) 1 = Fail 1 1Hz +/- 100 seg 3.2 0.6 2 0 (neg) 2 1Hz +/- 100 seg 2.8 0.65 3 1 (pos) 3 1Hz +/- 100 seg 2.4 0.8 4 0 (neg) 4 1Hz +/- 100 seg 1.8 1 1 1 (pos) 5 1.5Hz +/- 100 seg 3.2 0.65 4 1 ORTHOGONAL ARRAY 6 1.5Hz +/- 100 seg 2.8 0.6 1 0 7 1.5Hz +/- 100 seg 2.4 1 2 1 8 1.5Hz +/- 100 seg 1.8 0.8 3 0 TEST DESIGN 9 0.5Hz +/- 100 seg 3.2 0.8 1 1 10 0.5Hz +/- 100 seg 2.8 1 4 0 11 0.5Hz +/- 100 seg 2.4 0.6 3 1 12 0.5Hz +/- 100 seg 1.8 0.65 2 0 13 2.0Hz +/- 100 seg 3.2 1 3 0 14 2.0Hz +/- 100 seg 2.8 0.8 2 1 15 2.0Hz +/- 100 seg 2.4 0.65 1 0 16 2.0Hz +/- 100 seg 1.8 0.6 4 1 SIMULINK ENVIRONMENT CREATE DATA INPUTS VERIFY EXPECTED RESPONSE Mdl.ico AUTOSIM ENVIRONMENT TEST SCRIPT DESIGN AUTOMATED ITERITATIONS Net04.ico Figure 8: Pi AutoSim Hardware in the Loop Setup HIL ENVIRONMENT L16 (45) Orthogonal Array Test ORTHOGONAL ARRAY Determine A B C D E Pass Fail Criteria glitch Actual steering signal init glitch ial glitch voltage duration cal Slope Result f / amplitude req voltage magnitude Net04.ico No. (loops) 1 = Fail 1 1Hz +/- 100 seg 3.2 0.6 2 0( neg) TEST IMPLEMENTATION 2 1Hz +/- 100 seg 2.8 0.65 3 1( pos) 3 1Hz +/- 100 seg 2.4 0.8 4 0( neg) 4 1Hz +/- 100 seg 1.8 1 1 1( pos) 5 1.5Hz +/- 100 seg 3.2 0.65 4 1 6 1.5Hz +/- 100 seg 2.8 0.6 1 0 7 1.5Hz +/- 100 seg 2.4 1 2 1 8 1.5Hz +/- 100 seg 1.8 0.8 3 0 9 0.5Hz +/- 100 seg 3.2 0.8 1 1 REAL-TIME AUTOMATION OF ORTHOGONAL ARRAY 10 0.5Hz +/- 100 seg 2.8 1 4 0 11 0.5Hz +/- 100 seg 2.4 0.6 3 1 12 0.5Hz +/- 100 seg 1.8 0.65 2 0 13 2.0Hz +/- 100 seg 3.2 1 3 0 14 2.0Hz +/- 100 seg 2.8 0.8 2 1 15 2.0Hz +/- 100 seg 2.4 0.65 1 0 16 2.0Hz +/- 100 seg 1.8 0.6 4 1 TEST PASS / FAIL VERIFICATION REPORT The Pi AutoSim is ideally suited for performing the multiple, controlled iterations of the orthogonal array test process. In this application of the orthogonal array testing, the focus is the performance of a specific algorithm function performed by the ECU application software. By using the test scripting capability, a series Figure 9 : Robust Engineering S/W Test Process of varying input factors can be introduced to the ECU software in a systematic manner. The output response of the software may be monitored, and evaluated in real time. A general approach to automating the software CASE STUDY: AUTOMOTIVE SENSOR orthogonal array test using the AutoSim capability is PROCESSING described in the following steps and illustrated in Figure 9: The robust engineering software test process was applied to testing software for an automotive sensor 1. Perform Robust Engineering Software Test processing application. The technique proved effective process to formulate orthogonal array test in highlighting the range of conditions which would prove parameters. problematic to the algorithm. The benefit is that a robust solution can be developed to address the entire range of disturbance type inputs as opposed to only 2. Identify the pass / fail criteria for the sub-system implementing a solution for one specific operation function being tested in terms of measurable condition. simulation output signals. GENERAL OVERVIEW 3. Implement the orthogonal array test within the software simulation environment and iterate as An automotive angular position sensor is comprised of necessary to obtain initial confirmation results. multiple outputs which must be processed by an Electronic Control Unit (ECU) controller to determine a 4. Port input test scripts, data, requirements, etc single output signal proportional to angular position. Due and pass / fail criteria routines from software to the fact that the single output is a function of the simulation environment to HIL environment 7
  10. 10. multiple inputs, any disturbance on the inputs has the SINGLE SENSOR VOLTAGE OUTPUT capability of perturbing the output signal. Attempts have WITH SIMULATED DISTURBANCE been made to minimize the effect of the input disturbances through filtering, rate limiting, etc., disturbance however, this has proven to only be partially effective for initial voltage certain inputs. The robust engineering software test process was applied to test the algorithm over a wide disturbance magnitude range of inputs. The results indicate where Voltage improvements in the algorithm could be made to yield disturbance more robust performance. duration BASIC FUNCTION / MODIFIED P - DIAGRAM slope of signal proportional to input freq or rate The scope of the investigation is limited to the ECU software responsible for processing and converting the time sensor voltage outputs into a software variable proportional to angular position. An undesirable response was defined as a disturbance to the input of Figure 11 : Signal Disturbance Characteristics the ECU which results in a persistent offset in the angular position output. A modified P – Diagram of the problem is shown below ORTHOGONAL ARRAY EXPERIMENT SETUP in Figure 10. Disturbance characteristics for the input signal (output from sensor), signal frequency, signal An L16(45) Orthogonal Array was chosen to setup the magnitude, disturbance voltage, disturbance magnitude experiment test. This size matrix allows for 5 factors with and duration are identified with the P – Diagram. up to 4 levels each. The factors of signal frequency, signal magnitude, disturbance initial voltage, disturbance magnitude, disturbance duration, and disturbance slope were assigned as factors to the orthogonal array. Sensor Processing M Σ Software y Dummy treatment was used for Factor E (disturbance sensor Function angular slope) as only 2 levels 0 and 1 were applicable. For the voltage inputs position 1st iteration, Level 1 values were derived from sample (n) input data recorded from the ECU wherein the Noise Factors disturbance behavior had occurred. The remaining input signal frequency (rate) levels were selected as to provide a range around this input signal magnitude sensor input signal with disturbance single operating point which posed a problem. By disturbance initial voltage setting up the orthogonal array in this manner, the 1st disturbance magnitude (voltage) row experiment is trial is always expected to fail with the disturbance duration (loops) original ECU software. The remaining 15 additional experiment runs will reveal other interactions which also may cause the ECU software to fail. Table 1 below Figure 10: Modified P – Diagram Sensor Processing shows the table for the 1st iteration. Figure 11 provides an illustration of how these Factors disturbances affect the input signal. The slope of the initial disturbance disturbance signal with respect to time is proportional to the input input signal freq/ disturbance voltage duration cal disturbance slope am plitude frequency to the sensor. A disturbance can be viewed voltage m agnitude (m Sec) as a specific type of noise superimposed upon the input Level A B C D E signal to the controller. The noise has characteristics of 1 1H +/- 100 deg z 3.2 0.6 2 0 (neg) magnitude, duration, and an initial magnitude relative to 2 1.5H +/- 100 deg z 2.8 0.65 3 1 (pos) the range of the signal. These characteristics are 3 0.5H +/- 100 deg z 2.4 0.8 4 0 (neg) 4 2.0H +/- 100 deg z 1.8 1 1 1 (pos) identified as noise factors and will be used as input to the Orthogonal Array tests. Table 1 : 1st Iteration Factor / Level Assignment 8
  11. 11. The factor assignments for the full L16(45) matrix are output of the ECU software was also monitored using shown in Table 2. A SIMULINK s-function model was the Autosim. The test ECU has the ability to provide created to implement the ECU software. The various direct memory access to software data variables using inputs specific to the factor levels identified were the CAN Calibration Protocol (CCP). By monitoring and modeled in the SIMULINK environment and subjected to requesting specific memory location reads via the ECU the ECU software routine under test. CCP bus, internal software variables were captured in real time response to the corresponding controlled input data signals being generated by the text data files. The pass / fail criteria developed in the SIMULINK environment was also scripted within the AutoSim L16 (45) Orthogonal Array Test environment to automatically indicate the outcome of Determine each trial. The resulting configuration allowed for a very A B C D E Pass Fail repeatable, deterministic execution of the orthogonal Criteria array test matrix. initial disturbance disturbance input signal freq / Actual Result disturbance voltage duration cal Slope amplitude 1 = Fail No. voltage magnitude (mSec) 1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg) ITERATION TEST RESULTS 2 1Hz +/- 100 deg 2.8 0.65 3 1 (pos) Three iterations were necessary to successfully bound 3 1Hz +/- 100 deg 2.4 0.8 4 0 (neg) the input range of factor conditions which indicated 4 1Hz +/- 100 deg 1.8 1 1 1 (pos) problem areas with the existing ECU software. Following the 1st trial, the factor identified as disturbance slope was 5 1.5Hz +/- 100 deg 3.2 0.65 4 1 removed and replaced with input signal frequency. This 6 1.5Hz +/- 100 deg 2.8 0.6 1 0 allowed the input signal frequency and amplitude to be 7 1.5Hz +/- 100 deg 2.4 1 2 1 set independently. For the 2nd iteration the range of amplitude was selected over a broader range from 25 up 8 1.5Hz +/- 100 deg 1.8 0.8 3 0 to 100 Hz. This selection revealed that lower magnitudes 9 0.5Hz +/- 100 deg 3.2 0.8 1 1 below 75 Hz did not appear to cause problems for the ECU software. A final range was chosen which 10 0.5Hz +/- 100 deg 2.8 1 4 0 concentrated input amplitudes around 100 deg 11 0.5Hz +/- 100 deg 2.4 0.6 3 1 combined with frequencies above 1Hz. Experimental 12 0.5Hz +/- 100 deg 1.8 0.65 2 0 results indicated interactions between the disturbance criteria at higher frequencies and signal magnitudes that 13 2.0Hz +/- 100 deg 3.2 1 3 0 yielded undesirable results. 14 2.0Hz +/- 100 deg 2.8 0.8 2 1 With the problem successfully bounded, an alternative 15 2.0Hz +/- 100 deg 2.4 0.65 1 0 ECU software solution was developed and re-tested 16 2.0Hz +/- 100 deg 1.8 0.6 4 1 using the factors and levels that were used for the three iterations in the original tests. Response tables and signal to noise ratios were calculated based on the Table 2 : 1st Iteration Experiment Test number of experiment trial failures. This metric was used to quantify the difference of performance between trials, and also the amount of improvement from the original REALTIME HARDWARE IN LOOP EXPERIMENT design to the new design. An average 1.95 db S/N gain SETUP was observed between the two ECU designs. The final trial experiment with the redesigned ECU software still The setup for the HIL experiment consisted of the Pi indicated a problem interaction with large magnitudes Autosim, and utilized a controlled braking ECU and above 2 Hz. However, this condition was evaluated to be software (also shown in Figure 8). Input data files possible in simulation only, and outside the practical generated from the SIMULINK simulation of the operating constraints for the application. orthogonal array test used to test the ECU software were ported to the AutoSim environment. The input Table 3 provides these metrics for the various iterations data signals were generated / captured in a time for both designs of ECU software. Complete results for a stamped text data file. Through software mapping of sample iteration test are provided in the Appendix. the AutoSim analogue card outputs to the “command” data contained in the text data files, reproduction of REALTIME HARDWARE IN LOOP TEST RESULTS recorded sensor was performed. The ability to utilize this external text data as real time variables in the The various orthogonal array trials were performed using simulation required additional AutoSim model and script the HIL setup for iterations which illustrated the root functions to be developed by the testing team. The issues with the ECU software as well as confirmation 9
  12. 12. iterations with improved ECU software. The results from requirements that may be implemented in a software the software simulation step correlated with those from simulation environment and again confirmed with actual the HIL test results. The total runtime for each L16(45) HIL testing of the product ECU. The approach serves a matrix iteration for the HIL setups takes approximately 3 as a very repeatable, disciplined method to enhance the min., and time for set-up and completion of 1 complete level of ECU software verification. test matrix takes approximately 48 minutes. The development of the automated HIL test script becomes a check that can easily be applied to software packages as part of the final validation or detail software regression testing process. REFERENCES 1. ASI Consulting Group. Robust Engineering 5 L16 (4 ) Orthogonal Array Test Response Table Summary Practioner’s Course Class Notes. January 2004. Interaction Interaction 2. Deguchi, J, et. al. Efficient Tool for Debugging with Test S/N gain (db) S/N gain (db) S/N Failures Failures Description (initial design) (initial design) (re-design) (re-design) improvement the Orthogonal Array. R.E. for Software Application Iteration #1 35 -2.3620 10 -0.5266 1.8354 Course Handout. January 2004. Iteration #2 38 -1.9382 10 -0.5266 1.4116 3. Goldstein, S. Robust Testing of EW Systems. R.E. for Software Application Course Handout. January Iteration #3 71 -4.5449 40 -1.9382 2.6067 2004. 4. Taguchi, G., et. al.. Robust Engineering. McGraw Table 3 : Experimental Results Hill 2000. 5. Musa, J. Software Reliability Engineering. McGraw Hill 1999. 6. Wang, R., et. al. Unified Chassis Simulation CONCLUSION Architecture Delphi TechCon February 2004 7. Schuette, H., et. al. Hardware-in-the-LoopTesting of The application of robust engineering methodologies for Vehicle Dynamic Controllers – A Technical Survey ECU software testing provides a tool which allows for a SAE 2005-01-1660 more thorough investigation of possible interactions that may adversely affect software results. For the cases where a problem has been observed with existing ECU CONTACT software, the approach may be applied to identify the range or noise factor range which must be addressed. Eldon G. Leaphart, Engineering Manager – Diagnostics, This approach should help to prevent the undesirable Communications & System Software / Controlled Brakes cycle of creating and implementing solutions only to Delphi Corp., 12501 E. Grand River, MC 483-3DB-210, discover another undesirable scenario later during Brighton, MI, 48116-8326 Phone: (810)-494-4767, development or field operation. The method defines test Fax:(810)494-4458 email: eldon.g.leaphart@delphi.com 10
  13. 13. APPENDIX ITERATION # 1: ORIGINAL DESIGN Factors initial disturbance disturbance input signal freq / disturbance voltage duration cal Slope amplitude voltage magnitude (mSec) Level A B C D E 1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg) 2 1.5Hz +/- 100 deg 2.8 0.65 3 1 (pos) 3 0.5Hz +/- 100 deg 2.4 0.8 4 0 (neg) 4 2.0Hz +/- 100 deg 1.8 1 1 1 (pos) A B C D E initial disturbance disturbance input signal freq / disturbance voltage duration cal Slope 1 =Trouble amplitude No. voltage magnitude (mSec) 1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg) 1 2 1Hz +/- 100 deg 2.8 0.65 3 1 (pos) 0 3 1Hz +/- 100 deg 2.4 0.8 4 0 (neg) 1 4 1Hz +/- 100 deg 1.8 1 1 1 (pos) 0 5 1.5Hz +/- 100 deg 3.2 0.65 4 1 0 6 1.5Hz +/- 100 deg 2.8 0.6 1 0 1 7 1.5Hz +/- 100 deg 2.4 1 2 1 0 8 1.5Hz +/- 100 deg 1.8 0.8 3 0 1 9 0.5Hz +/- 100 deg 3.2 0.8 1 1 0 10 0.5Hz +/- 100 deg 2.8 1 4 0 0 11 0.5Hz +/- 100 deg 2.4 0.6 3 0 0 12 0.5Hz +/- 100 deg 1.8 0.65 2 1 0 13 2.0Hz +/- 100 deg 3.2 1 3 0 0 14 2.0Hz +/- 100 deg 2.8 0.8 2 1 0 15 2.0Hz +/- 100 deg 2.4 0.65 1 1 0 16 2.0Hz +/- 100 deg 1.8 0.6 4 0 1 Sum Failures 5 S/N -2.361986242 EXPERIMENT RESPONSE TABLE A1 A2 A3 A4 B1 1 B2 1 B3 1 B4 1 1 B1 B2 B3 B4 C1 1 1 1 1 1 C2 C3 1 1 1 C4 C1 C2 C3 C4 D1 1 D2 1 D3 1 1 1 1 1 1 D4 1 1 D1 D2 D3 D4 E1 1 1 E2 1 1 1 1 E3 1 1 1 1 1 1 E4 8 8 0 4 0 0 3 3 3 0 2 0 2 0 0 0 0 2 0 35 11
  14. 14. ITERATION #6 WITH REDESIGN (REPEAT ITERATION #1) Factors initial disturbance disturbance input signal freq / disturbance voltage duration cal Slope amplitude voltage magnitude (mSec) Level A B C D E 1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg) 2 1.5Hz +/- 100 deg 2.8 0.65 3 1 (pos) 3 0.5Hz +/- 100 deg 2.4 0.8 4 0 (neg) 4 2.0Hz +/- 100 deg 1.8 1 1 1 (pos) A B C D E steering signal initial glitch glitch voltage glitch duration Slope 1 =Trouble freq / amplitude voltage magnitude cal (loops) No. 1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg) 0 2 1Hz +/- 100 deg 2.8 0.65 3 1 (pos) 0 3 1Hz +/- 100 deg 2.4 0.8 4 0 (neg) 0 4 1Hz +/- 100 deg 1.8 1 1 1 (pos) 0 5 1.5Hz +/- 100 deg 3.2 0.65 4 1 0 6 1.5Hz +/- 100 deg 2.8 0.6 1 0 0 7 1.5Hz +/- 100 deg 2.4 1 2 1 0 8 1.5Hz +/- 100 deg 1.8 0.8 3 0 0 9 0.5Hz +/- 100 deg 3.2 0.8 1 1 0 10 0.5Hz +/- 100 deg 2.8 1 4 0 0 11 0.5Hz +/- 100 deg 2.4 0.6 3 1 0 12 0.5Hz +/- 100 deg 1.8 0.65 2 0 0 13 2.0Hz +/- 100 deg 3.2 1 3 0 0 14 2.0Hz +/- 100 deg 2.8 0.8 2 1 0 15 2.0Hz +/- 100 deg 2.4 0.65 1 0 1 16 2.0Hz +/- 100 deg 1.8 0.6 4 1 0 Sum Failures 1 S/N -0.526578774 EXPERIMENT RESPONSE TABLE A1 A2 A3 A4 B1 B2 B3 B4 1 B1 B2 B3 B4 C1 1 1 C2 C3 C4 C1 C2 C3 C4 D1 D2 D3 1 1 1 D4 D1 D2 D3 D4 E1 E2 1 1 1 1 E3 E4 0 0 0 4 0 0 0 0 3 0 2 0 0 0 0 0 0 1 0 10 12