CAA HUMS Research Project Meeting 16 th  March 2006 Presentation by:  Brian Larder, Rob Callan, Lorry Stoate
Agenda <ul><li>Minutes of 20 October meeting (accuracy and actions) </li></ul><ul><li>Overview of activities since last me...
Overview of activities since last meeting
Overview of activities since last meeting <ul><li>We have produced a software tool to manage the configuration for generat...
Overview of activities since last meeting <ul><li>Model Building </li></ul><ul><ul><li>Models have been built for all main...
Anomaly detection process development – model information extraction
Anomaly detection process development – model information extraction <ul><li>There are three key elements to anomaly detec...
Anomaly detection process development – model information extraction <ul><li>Alerting strategy </li></ul><ul><ul><li>The p...
Presentation and demonstration of the results of the off-line data analysis (phase 1)
Model Assessment <ul><li>The Anomaly modeling is the most complex element in the three stage process </li></ul><ul><ul><li...
Model Assessment <ul><li>The overall objective for the anomaly detection technology can be summarized by the following req...
Model Assessment <ul><li>What has been done to test the success of this objective (and to answer the questions from the pr...
Model Assessment <ul><li>Before proceeding with a review of the findings we need to keep in mind a few points </li></ul><u...
Model Assessment <ul><li>Does the anomaly modeling fundamentally work? </li></ul><ul><ul><li>The ProDAPS modeling diagnost...
Model Assessment <ul><li>The top chart shows a plot of the absolute values for FSA_SO1 from the training data on the Stbd ...
Model Assessment <ul><li>The chart on the left is the fleet distribution that has been identified by the model </li></ul><...
Revisit the Bevel Pinion fault case <ul><li>The global fitness on the Bevel Pinion case reveals some interesting informati...
Revisit the Bevel Pinion fault case <ul><li>The global fitness on the Bevel Pinion case reveals some interesting informati...
Model Assessment <ul><li>Review of selected cases based on current fleet data </li></ul>
Fan 15 blades
Accelerometer n°8  (12RK1) Accelerometer n°10 (12RK3) Accelerometer n°14  (12RK5) Accelerometer n°15 (18RK1) Accelerometer...
Recommendations for future research <ul><li>The activities of research and development often highlight the need for furthe...
Recommendations for future research <ul><li>The the activities of research and development often highlight the need for fu...
Review of live trial system development
Web-based anomaly detection system architecture <ul><li>The anomaly detection system will operate as a secure web server, ...
Summary of major activities and findings <ul><li>Live trial system development… </li></ul><ul><li>Activities </li></ul><ul...
Additional fault data?
Sources of fault data for system testing <ul><li>BHL AS332L IHUMS data </li></ul><ul><ul><li>Large quantity of historical ...
End of phase 1 report
Planning of live trial (phase 2)
Plan for 6 month trial <ul><li>Tasks to be completed prior to start of 6 month trial </li></ul><ul><ul><li>Completion of U...
HUMS for rotor systems
HUMS for rotor systems <ul><li>Contract amendment received from CAA on 18 January </li></ul><ul><li>Obtaining University s...
AOB and date of next meeting
Upcoming SlideShare
Loading in …5
×

16 03 06 Smiths Aeros 2084a

376
-1

Published on

Fashion, apparel, textile, merchandising, garments

Published in: Business, Lifestyle
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
376
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

16 03 06 Smiths Aeros 2084a

  1. 1. CAA HUMS Research Project Meeting 16 th March 2006 Presentation by: Brian Larder, Rob Callan, Lorry Stoate
  2. 2. Agenda <ul><li>Minutes of 20 October meeting (accuracy and actions) </li></ul><ul><li>Overview of activities since last meeting </li></ul><ul><li>Anomaly detection process development – model information extraction </li></ul><ul><li>Presentation and demonstration of the results of the off-line data analysis (phase 1) </li></ul><ul><li>Review of live trial system development </li></ul><ul><li>Additional fault data? </li></ul><ul><li>End of phase 1 report </li></ul><ul><li>Planning of live trial (phase 2) </li></ul><ul><li>HUMS for rotor systems </li></ul><ul><li>AOB </li></ul><ul><li>Date of next meeting </li></ul>
  3. 3. Overview of activities since last meeting
  4. 4. Overview of activities since last meeting <ul><li>We have produced a software tool to manage the configuration for generating anomaly models </li></ul><ul><ul><li>Specifying data source, model storage, model parameters, input indicators, etc was very tedious and holding back efficient progress </li></ul></ul><ul><ul><li>This tool allows the analyst to define a set of tasks that are then stored and executed as batch process </li></ul></ul><ul><ul><li>Tasks can also be placed on different available machines which is important because each task takes a lot of CPU time </li></ul></ul>
  5. 5. Overview of activities since last meeting <ul><li>Model Building </li></ul><ul><ul><li>Models have been built for all main, tail, intermediate and accessory gears </li></ul></ul><ul><ul><li>Fitness score predictions have been produced for all data </li></ul></ul><ul><ul><li>A lot of analysis has been undertaken to review the models (see later sections) </li></ul></ul><ul><li>Alerting threshold </li></ul><ul><ul><li>The strategy for setting a threshold for anomaly alerts has been studied in some detail </li></ul></ul><ul><ul><li>We have implemented an alerting strategy but this will need reviewing based on feedback </li></ul></ul><ul><li>Development of the live trial system </li></ul><ul><ul><li>To be demonstrated later </li></ul></ul>
  6. 6. Anomaly detection process development – model information extraction
  7. 7. Anomaly detection process development – model information extraction <ul><li>There are three key elements to anomaly detection </li></ul><ul><ul><li>Data pre-processing </li></ul></ul><ul><ul><ul><li>Used to emphasis data characteristics that are considered important such as trends </li></ul></ul></ul><ul><ul><ul><li>This can be difficult but it is a critical stage </li></ul></ul></ul><ul><ul><ul><ul><li>Pre-processing can be considered as tagging the data as interesting or not interesting </li></ul></ul></ul></ul><ul><ul><li>Anomaly modelling </li></ul></ul><ul><ul><ul><li>Main function is to highlight data of interest and to emphasise its significance </li></ul></ul></ul><ul><ul><ul><li>For HUMS data, model training needs to be robust to anomalous training </li></ul></ul></ul><ul><ul><ul><ul><li>In other words, the training data will contain anomalies due to sensor and general instrumentation problems that occur in practice </li></ul></ul></ul></ul><ul><ul><li>Alerting strategy </li></ul></ul><ul><ul><ul><li>Defining the events that lead to flagging the operator’s attention </li></ul></ul></ul>
  8. 8. Anomaly detection process development – model information extraction <ul><li>Alerting strategy </li></ul><ul><ul><li>The predicted fitness scores from the training data have been used to generate alerting thresholds (one per model) </li></ul></ul><ul><ul><li>The threshold is not based on a mean value plus so many standard deviations </li></ul></ul><ul><ul><ul><li>The fitness scores do not follow a normal distribution </li></ul></ul></ul><ul><ul><li>The strategy has been to examine the fitness scores using a statistical plot that can assist with identifying different regions of density </li></ul></ul><ul><ul><ul><li>The idea is to set the threshold at a point where there is change in density </li></ul></ul></ul><ul><ul><ul><li>The point should also lie towards the end of the cumulative distribution (i.e. such that the majority of data will not be in alert) </li></ul></ul></ul><ul><ul><li>Whilst the modelling is robust to anomalies in the training data, the threshold will be affected by the quality of the training data </li></ul></ul><ul><ul><li>This strategy will be used in the live trial but we need the live trial experience to refine it </li></ul></ul>
  9. 9. Presentation and demonstration of the results of the off-line data analysis (phase 1)
  10. 10. Model Assessment <ul><li>The Anomaly modeling is the most complex element in the three stage process </li></ul><ul><ul><li>The technology is independent of the other two stages of pre-processing and alerting </li></ul></ul><ul><ul><li>In our view it should provide more than a mechanism to flag anomalous data </li></ul></ul><ul><ul><ul><li>It should provide an insight into the ‘large data’ picture </li></ul></ul></ul><ul><li>We have addressed the following questions </li></ul><ul><ul><li>Does it do what it is designed to do? </li></ul></ul><ul><ul><li>What benefits does it bring? </li></ul></ul><ul><ul><li>The answers to these questions will be illustrated throughout today’s presentation </li></ul></ul><ul><ul><ul><li>It will become apparent that HUMS will clearly benefit from the application of the anomaly processing </li></ul></ul></ul><ul><ul><ul><ul><li>In our view, there is a proven need for this type of technology </li></ul></ul></ul></ul>
  11. 11. Model Assessment <ul><li>The overall objective for the anomaly detection technology can be summarized by the following requirement </li></ul><ul><ul><li>Anomaly models will be built from data that contain an undefined percentage of anomalies. The models should recognize outlying training data. The models response to outliers should be adaptive via configurable tuning – humans have to attach value to the anomalous information provided (i.e, ability to configure the alerting rate and trade off between false positives and costs) </li></ul></ul><ul><li>What has been done to test the success of this objective (and to answer the questions from the previous slide)? </li></ul><ul><ul><li>Models have been built for all 35 shafts (main, tail, intermediate and accessories) </li></ul></ul><ul><ul><ul><li>Resulted in 140 models </li></ul></ul></ul><ul><ul><ul><ul><li>Absolute and trend X 2 (8 indicators and M6) </li></ul></ul></ul></ul><ul><ul><ul><li>Fitness predictions have been produced against all of these models for </li></ul></ul></ul><ul><ul><ul><ul><li>Training, validation and current data </li></ul></ul></ul></ul>
  12. 12. Model Assessment <ul><li>What has been done to test the success of this objective (and to answer the questions from the previous slide)?… </li></ul><ul><ul><li>The training data approximates to about half that available </li></ul></ul><ul><ul><ul><li>For example, the training data contained 60 main gearboxes but in total there are now 115 </li></ul></ul></ul><ul><ul><ul><li>The validation data contains those gearboxes whose data were acquired before April 2004 and were not used for training </li></ul></ul></ul><ul><ul><ul><li>The current data consists of the current fleet and there will be some overlap with the validation data (where current components were in service before April 2004) </li></ul></ul></ul><ul><ul><li>The models have produced a huge amount of information and it is impossible to analyze all of this in the time available </li></ul></ul><ul><ul><ul><li>A selection of models have been explored in detail </li></ul></ul></ul><ul><ul><ul><li>We have directed most of our attention at the current fleet </li></ul></ul></ul>
  13. 13. Model Assessment <ul><li>Before proceeding with a review of the findings we need to keep in mind a few points </li></ul><ul><ul><li>The models have been built in a one-off process </li></ul></ul><ul><ul><ul><li>There has been no tuning but we have always stated there will be a need for this following the review and during the live trial </li></ul></ul></ul><ul><ul><ul><li>Initially we were struck by the quantity of low fitness scores being produced by the models and thought the model adaptation was probably too aggressive </li></ul></ul></ul><ul><ul><ul><ul><li>In some models this is true (hence the need for the ability to re-tune) but in most cases the modeling is pointing to genuinely outlying data – there happens to be a lot of it! </li></ul></ul></ul></ul><ul><ul><li>We can only re-tune once we have discussed some representative cases and established an initial policy of how we want the system to respond </li></ul></ul><ul><ul><li>Some low fitness scores are misleading </li></ul></ul><ul><ul><ul><li>This is not a modeling issue and it originates in the trend data pre-processing where there has been a step change because of maintenance to correct high vibration </li></ul></ul></ul>
  14. 14. Model Assessment <ul><li>Does the anomaly modeling fundamentally work? </li></ul><ul><ul><li>The ProDAPS modeling diagnostic tool has been used to explore model content and features </li></ul></ul><ul><ul><li>Anomalous fitness scores have been reviewed against the condition indicators (mainly covered in a later section) </li></ul></ul><ul><li>Model diagnostics </li></ul><ul><ul><li>The model diagnostics tool extracts a range of information and we shall see some of this </li></ul></ul><ul><ul><li>We have taken all of the 8 indicator models for the main gearbox and extracted what we call coverage statistics </li></ul></ul><ul><ul><ul><li>Basically we ask the model for a fleet view on all of the indicators </li></ul></ul></ul><ul><ul><ul><li>Sometimes the coverage shows that the tuning is too aggressive but often it illustrates that the modeling is doing a far superior job of representing the data than can be achieved using traditional parametric statistics </li></ul></ul></ul><ul><ul><ul><li>An example follows </li></ul></ul></ul>
  15. 15. Model Assessment <ul><li>The top chart shows a plot of the absolute values for FSA_SO1 from the training data on the Stbd Aft Fw </li></ul><ul><ul><li>The bars show a histogram of the data and whilst the bulk of the data approximates a normal distribution it contains a very long tail </li></ul></ul><ul><ul><li>The red line is the distribution computed from the data </li></ul></ul><ul><ul><ul><li>A single mode fleet distribution is a poor model of the data </li></ul></ul></ul><ul><ul><ul><li>This effect is repeated over many indicators and models </li></ul></ul></ul><ul><li>The histogram shows what looks to be a central distribution that is extended with a lot of outlying data </li></ul><ul><ul><li>The chart at the bottom shows the output of a ProDAPS cluster model </li></ul></ul><ul><ul><ul><li>This illustrates the impact of the outlying data </li></ul></ul></ul><ul><ul><ul><li>ProDAPS indicated that it would take in the order of 17 distributions to provide a good model of this data </li></ul></ul></ul><ul><li>The anomaly models are built using 8 indicators </li></ul><ul><ul><li>Ideally we would like the model to target the outlying data and find the distribution that characterises the main fleet </li></ul></ul><ul><ul><li>This is a very difficult challenge but it is the core objective we have set for the anomaly modelling </li></ul></ul>
  16. 16. Model Assessment <ul><li>The chart on the left is the fleet distribution that has been identified by the model </li></ul><ul><ul><li>It is clearly much more representative of the fleet data </li></ul></ul><ul><ul><li>It has clearly targeted outlying data </li></ul></ul><ul><li>The diagnostic tool can also take the fleet statistics and produce what we call a global fit </li></ul><ul><ul><li>For a component, the global fitness scores can look similar to the anomaly fitness scores but often they will contradict </li></ul></ul><ul><ul><li>The global fits in conjunction with the anomaly fits can be very informative </li></ul></ul><ul><li>We shall illustrate the global fit for FSA_SO1 on the Stbd Aft Fw and show what is driving the outliers </li></ul>
  17. 17. Revisit the Bevel Pinion fault case <ul><li>The global fitness on the Bevel Pinion case reveals some interesting information </li></ul><ul><ul><li>If we examine the absolute indicators for all data </li></ul></ul><ul><ul><ul><li>FSA_GE22 is the highest for this fault component </li></ul></ul></ul><ul><ul><ul><li>FSA_SON is suppressed </li></ul></ul></ul><ul><ul><ul><li>FSA_MS_2 does not look significant </li></ul></ul></ul><ul><ul><li>The diagnostics show, that although FSA_GE22 reaches the highest value of all data seen to date it is almost impossible to attach any significance to the trend when doing a simple fleet comparison </li></ul></ul><ul><ul><li>On the other hand </li></ul></ul><ul><ul><ul><li>The anomaly model, shows the significance on both the absolute and trend models </li></ul></ul></ul><ul><ul><ul><li>In terms of the absolute model </li></ul></ul></ul><ul><ul><ul><ul><li>FSA_MS_2 and FSA_GE22 are significant (see next slide) </li></ul></ul></ul></ul><ul><ul><ul><li>In terms of the trend model </li></ul></ul></ul><ul><ul><ul><ul><li>FSA_GE22 and FSA_SON are significant </li></ul></ul></ul></ul>
  18. 18. Revisit the Bevel Pinion fault case <ul><li>The global fitness on the Bevel Pinion case reveals some interesting information… </li></ul><ul><ul><li>The ProDAPS model diagnostics can indicate which indicators are driving the low fitness </li></ul></ul><ul><ul><ul><li>The plot to the right shows that FSA_MS_2 is more significant than FSA_GE22 in terms of absolute values </li></ul></ul></ul><ul><ul><ul><ul><li>This is lost in the fleet statistics </li></ul></ul></ul></ul>
  19. 19. Model Assessment <ul><li>Review of selected cases based on current fleet data </li></ul>
  20. 20. Fan 15 blades
  21. 21. Accelerometer n°8 (12RK1) Accelerometer n°10 (12RK3) Accelerometer n°14 (12RK5) Accelerometer n°15 (18RK1) Accelerometer n°16 (18RK2) Accelerometer n°12 (14RK2) Accelerometer n°11 (14RK1) Accelerometer n°13 (12RK4)
  22. 22. Recommendations for future research <ul><li>The activities of research and development often highlight the need for further work </li></ul><ul><li>Pre-processing </li></ul><ul><ul><li>There is a great deal of variability in HUMS data (e.g. due to various maintenance actions) </li></ul></ul><ul><ul><li>This variability could mask important trends because the trend pre-processing as it currently exists does a form of differencing and it does not distinguish between a developing trend and step change or ocurrence of noise in the data </li></ul></ul><ul><li>Model tuning </li></ul><ul><ul><li>The application of this technology is very new and we need the trial experience to refine the models </li></ul></ul><ul><li>Probabilistic alerting policy </li></ul><ul><ul><li>It is unrealistic to set a hard threshold to demarcate the interesting from the non-interesting </li></ul></ul><ul><ul><ul><li>We need a point at which an alert is triggered but we also need a type of anomaly index </li></ul></ul></ul><ul><ul><ul><ul><li>There could be a tendency to interpret fitness scores in a manner similar to reading a linear temperature scale but the distribution is not linear </li></ul></ul></ul></ul><ul><ul><ul><ul><li>A probabilistic measure is more appropriate </li></ul></ul></ul></ul><ul><ul><ul><ul><li>A probabilistic measure would also normalise the anomalies and assist with reasoning across shafts </li></ul></ul></ul></ul>
  23. 23. Recommendations for future research <ul><li>The the activities of research and development often highlight the need for further work… </li></ul><ul><li>Data mine the features of anomalous trends to test theoretical models </li></ul><ul><ul><li>ProDAPS diagnostics can point to the indicators that are driving trends </li></ul></ul><ul><ul><ul><li>It would be informative to mine these features to test established diagnostic knowledge and develop this </li></ul></ul></ul><ul><li>Reasoning </li></ul><ul><ul><li>More directed information could be provided by reasoning with anomaly outputs </li></ul></ul><ul><ul><ul><li>Fuse information across shafts to identify instrumentation issues </li></ul></ul></ul><ul><ul><ul><li>Reason about the nature of the anomaly – trends vs high variability vs step changes etc </li></ul></ul></ul><ul><ul><ul><li>Reason about the indicators driving the trend to provide more detail on the significance of anomalies </li></ul></ul></ul><ul><ul><ul><li>Case-based reasoning to search for any similar previous cases </li></ul></ul></ul>
  24. 24. Review of live trial system development
  25. 25. Web-based anomaly detection system architecture <ul><li>The anomaly detection system will operate as a secure web server, located at Smiths in Southampton. </li></ul><ul><ul><li>HUMS data is automatically transferred overnight from Bristow’s Web Portal to Southampton (Working). </li></ul></ul><ul><ul><li>The HUMS data is imported into the anomaly detection system’s data warehouse and analysed overnight (Working). </li></ul></ul><ul><ul><li>Bristow have a remote login to the system to view results at any time via a web browser (Tested). </li></ul></ul>
  26. 26. Summary of major activities and findings <ul><li>Live trial system development… </li></ul><ul><li>Activities </li></ul><ul><ul><li>Development of software components for live trial system </li></ul></ul><ul><ul><ul><li>Overnight Data processing </li></ul></ul></ul><ul><ul><ul><ul><li>Complete data flow (including Alerting) – has been working daily since 16 th February 2006. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Management of documentary data (gearbox changes, recording of trial findings etc.) – still to be implemented. </li></ul></ul></ul></ul><ul><ul><ul><li>User Interface (UI) - </li></ul></ul></ul><ul><ul><ul><ul><li>A Basic navigation is in place (drill down). </li></ul></ul></ul></ul><ul><ul><ul><ul><li>An Annotation strategy demonstrable. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Acknowledging Alerts demonstrable. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Historical Search strategy identified. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Speed of connection, and practicality tested with Bristow. </li></ul></ul></ul></ul><ul><li>Findings </li></ul><ul><ul><li>We have addressed all potential areas of risk in the data processing and UI. </li></ul></ul><ul><ul><li>We still have some features of the UI to complete for the live trial. </li></ul></ul>
  27. 27. Additional fault data?
  28. 28. Sources of fault data for system testing <ul><li>BHL AS332L IHUMS data </li></ul><ul><ul><li>Large quantity of historical data. </li></ul></ul><ul><ul><li>With BHL investigation of IHUMS indicator trends related to anomaly detection results, it is possible to perform a good assessment of anomaly detection capabilities using the existing data set. </li></ul></ul><ul><li>CHC Scotia AS332L IHUMS data </li></ul><ul><ul><li>Data for cracked AS332L bevel pinion. </li></ul></ul><ul><li>WHL AS332L MGB data from CAA seeded defect test programme </li></ul><ul><ul><li>Documentary information on this received from WHL. </li></ul></ul><ul><ul><li>Limited analysis coverage, but some fault related trends could be obtained. </li></ul></ul><ul><ul><li>Difficulties obtaining data from WHL. </li></ul></ul>
  29. 29. End of phase 1 report
  30. 30. Planning of live trial (phase 2)
  31. 31. Plan for 6 month trial <ul><li>Tasks to be completed prior to start of 6 month trial </li></ul><ul><ul><li>Completion of User Interface software </li></ul></ul><ul><ul><li>Model tuning / Data re-modelling </li></ul></ul><ul><ul><li>Definition of BHL operational procedures </li></ul></ul><ul><ul><ul><li>Daily review of anomaly detection </li></ul></ul></ul><ul><ul><ul><li>Feedback from follow-up of review </li></ul></ul></ul><ul><ul><ul><li>Reporting of faults, component changes and maintenance actions </li></ul></ul></ul><ul><ul><ul><li>System problem reporting </li></ul></ul></ul><ul><ul><li>Definition of Smiths support procedures </li></ul></ul><ul><ul><ul><li>Database update following component changes </li></ul></ul></ul><ul><ul><ul><li>Investigation of reported system problems and anomaly model behaviour </li></ul></ul></ul><ul><ul><ul><li>Configuration control for any model tuning </li></ul></ul></ul><ul><li>Trial start date </li></ul><ul><ul><li>Forecast for mid May </li></ul></ul><ul><li>Trial performance assessment </li></ul><ul><ul><li>Project review meeting after first month of operations? </li></ul></ul><ul><li>Decision on trial extension </li></ul>
  32. 32. HUMS for rotor systems
  33. 33. HUMS for rotor systems <ul><li>Contract amendment received from CAA on 18 January </li></ul><ul><li>Obtaining University support for study </li></ul><ul><ul><li>Selection of University of Glasgow (including rotorcraft group from Imperial College London, led by Dr Richard Brown.) </li></ul></ul><ul><ul><li>Arrangement of Confidentiality Agreement </li></ul></ul><ul><ul><li>Agreement of SOW (focus on literature review, mathematical modelling and laboratory test facilities) </li></ul></ul><ul><ul><li>Signing of a Research Agreement </li></ul></ul><ul><li>Project start </li></ul><ul><ul><li>Planned start date: 3 rd April </li></ul></ul><ul><ul><li>Planned Smiths resources </li></ul></ul><ul><ul><ul><li>BDL: Project management and liaison with UoG </li></ul></ul></ul><ul><ul><ul><li>SS: Accident data review </li></ul></ul></ul><ul><ul><ul><li>REC: Application of machine learning techniques </li></ul></ul></ul><ul><ul><ul><li>DLG: Mathematician support </li></ul></ul></ul>
  34. 34. AOB and date of next meeting
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×