Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski
•Download as PPTX, PDF•
2 likes•408 views
This talk is about data science and statistics applied to flight safety in commercial aviation worldwide. In the introductory part, we will stress the importance of monitoring your flight data and show you some real records coming from flight data recorders (aircraft “black boxes”). We will then explain how the data is recorded, downloaded, analysed, converted to safety events and finally, validated by experts in the field - flight data analysts. Data aggregation across many flights will result with statistical images of safety risks in airlines’ operations. However, this valuable tool can turn into a deadly weapon if used negligently – we’ll support this claim with examples. We are convinced the audience will know about some of these traps, regardless of the industry they are coming from, but hopefully there will be something valuable to take home, too. In the second part, we are saying goodbye to the data analyst and the statistician – the two dominant guys from the first part of the presentation. However, a data scientist will stem from valuable experiences and domain knowledges of the two. This guy will walk the audience through three simple, but working examples. The first one is about how we can improve the accuracy of automated analysis by using historic data and a probabilistic, Bayesian approach. The second example is about finding novel safety risks in airlines’ operations by using simple principal component analysis. Lastly, we’ll use a Markov model to detect aircraft which have changed behaviour with respect to frequency of data downloads, so we collect as many flight data as possible. We will try to make this chat as interesting and as interactive as we can and are looking forward to meeting you at this fun and interesting conference!
Recommended
Recommended
More Related Content
Similar to Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski
Similar to Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski (20)
More from Institute of Contemporary Sciences
More from Institute of Contemporary Sciences (20)
Recently uploaded
Recently uploaded (20)
Avoiding Black Boxes with Data Science part I & II - Marko Vasiljevski
- 1. Data Science Conference 11-12 October 2016 Belgrade, Serbia
- 3. Aircraft challenge Marko, don’t forget to show the aircraft to the audience
- 6. 3400 G shock for 6.5 ms 500 lb. Dropped from 10 ft with a ¼- inch-diameter contact point 1100 ºC flame for 30 minutes. 260 ºC for 10 hours Immersion in aircraft fluids for 24 hours Immersion in sea water for 30 days 5,000 pounds crush for 5 minutes on each axis Pressure equivalent to depth of 20,000 ft.
- 9. Avoiding Black Boxes with Data Science Raffaele Rainone & Marko Vasiljevski Data Science Conference 11-12 October 2016 Belgrade, Serbia
- 11. Hkjk;l “I don’t know…, pure mathematician, Python developer, data scientist, pizza lover, feeling a bit home sick for Italy, …”
- 12. Chatting about… Aviation Safety A story about flight data monitoring (FDM) A story of a flight data analyst A story of a flight safety statistician
- 14. Chatting about… Data science in (a bit of) action Jet-engine health - Working with Mr. Bayes Detecting safety concerns – PCA, a friend Finding cuckoo’s eggs – Mr. Markov’s chains
- 18. Flight Data Monitoring (FDM) what?! Part of safety management system (SMS) Airlines worldwide obliged to do FDM Spotting deviations from safe operation Non-punitive – learn from mistakes Confidential
- 24. 002AF5C0 00 00 76 07 04 01 64 08 06 01 3C 0C F2 0E 3C 0C ..v...d...<.ò.<. 002AF5D0 00 00 58 02 02 0C 00 0C 48 08 00 00 00 00 00 0C ..X.....H....... 002AF5E0 00 0C 00 00 00 00 E0 0F 00 00 00 0C 40 0C 00 0C ......à.....@... 002AF5F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90 01 ..............•. 002AF600 47 02 77 07 03 00 2B 0E DF 0F 02 0C A5 0F 78 00 G.w...+.ß...¥.x. 002AF610 55 04 D3 01 D8 0F EA 03 A4 0F EE 0F FD 0F 00 00 U.Ó.Ø.ê.¤.î.ý... 002AF620 08 00 10 00 00 00 08 00 11 00 00 00 06 00 1E 00 ................ 002AF630 F8 0F 10 00 00 00 8C 0B 0C 00 0A 00 00 00 00 00 ø.....Œ......... 002AF640 00 00 77 07 96 0F 64 08 06 01 3C 0C F2 0E 08 08 ..w.–.d...<.ò... 002AF650 00 00 00 00 01 00 AC 00 4C 08 01 08 00 00 00 00 ......¬.L....... 002AF660 00 00 04 02 D0 02 0D 02 12 02 00 0B 44 01 00 00 ....Ð.......D... 002AF670 A4 06 40 01 03 00 00 00 40 02 00 00 00 00 00 00 ¤.@.....@....... 002AF680 02 00 77 07 00 00 2B 0E DF 0F 04 0C A5 0F 00 08 ..w...+.ß...¥... 002AF690 54 04 D4 01 DA 0F EA 03 A4 0F 73 0E 00 00 01 00 T.Ô.Ú.ê.¤.s..... 002AF6A0 08 00 10 00 01 00 08 00 00 00 03 00 06 00 1D 00 ................ 002AF6B0 FB 0F 11 00 02 00 01 00 00 00 03 00 00 00 00 00 û............... 002AF6C0 01 00 76 07 00 00 64 08 06 01 3C 0C F2 0E 3C 0C ..v...d...<.ò.<. 002AF6D0 6A 02 21 03 03 00 00 00 48 08 00 00 00 00 00 00 j.!.....H....... 002AF6E0 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00 00 ................ 002AF6F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 FC 0F ..............ü. 002AF700 CC 08 77 07 03 00 2B 0E DF 0F 02 0C A5 0F 79 01 Ì.w...+.ß...¥.y. 002AF710 54 04 D4 01 D8 0F EA 03 A7 0F EE 0F FE 0F 01 00 T.Ô.Ø.ê.§.î.þ... 002AF720 08 00 00 00 00 00 00 00 11 00 02 00 06 00 00 00 ................ 002AF730 F8 0F 00 00 00 00 8C 0B 00 00 00 00 00 00 00 00 ø.....Œ......... 002AF740 00 00 76 07 94 0F 64 08 06 01 3C 0C F2 0E 08 08 ..v.”.d...<.ò... 002AF750 9A 0C 18 00 61 0C 68 01 48 08 F8 0F A4 0C 00 00 š...a.h.H.ø.¤... 002AF760 00 00 00 00 00 00 00 00 00 00 00 00 00 00 5B 02 ..............[. 002AF770 00 00 03 02 00 00 2B 0F F0 0F 2A 0F 00 00 00 00 ......+.ð.*..... 002AF780 00 00 77 07 00 00 2B 0E DF 0F 04 0C A5 0F 2C 00 ..w...+.ß...¥.,. 002AF790 54 04 D4 01 D8 0F EB 03 A6 0F 00 0D 00 00 01 00 T.Ô.Ø.ë.¦....... 002AF7A0 08 00 17 00 03 00 0F 00 00 00 00 00 06 00 1C 00 ................ 002AF7B0 F8 0F 10 00 02 00 01 00 00 00 03 00 00 00 00 00 ø............... What now? Data frames
- 27. The story of a flight data analyst
- 37. The story of a flight safety statistician
- 38. The story of a flight safety statistician STATISTICS
- 39. How do we monitor safety in FDM 0 1 2 3 4 5 6 7 8 Event Count 3.141592 65358979 32384626 43383384
- 41. Uncertainty - good companion Normal acceleration - Tdwn Normal acceleration – Lift-off Statistic Value Total count 80,663 Average 1.31 (Min, Max) (1.03, 2.40) Range 1.37 Standard deviation 0.10 Statistic Value Total count 80,663 Average 1.19 (Min, Max) (0.66, 1.63) Range 0.97 Standard deviation 0.05 Wider histogram, less confidence in mean value Narrower histogram, more confidence in mean value
- 42. How apples relate to flight safety
- 43. Micromanaging events 0 1 2 3 4 5 6 7 8 9 10 High speed taxiing High lateral g taxiing Flap overspeed Airspeed high 10000-5000 ft Excessive breaking Top 5 Events – January 2016
- 44. Micromanaging events 0 1 2 3 4 5 6 7 8 9 10 High speed taxiing High lateral g taxiing Flap overspeed Airspeed high 10000- 5000 ft Excessive breaking Pull up Stick shaker Top 7 Events – January 2016
- 45. Most severe safety event – a riddle It doesn’t happen in the air It doesn’t happen at the gate It happens in the office
- 46. Micromanaging events 0 1 2 3 4 5 6 7 8 9 10 High speed taxiing High lateral g taxiing Flap overspeed Airspeed high 10000- 5000 ft Excessive breaking Pull up Stick shaker Top 7 Events – January 2016
- 47. Not looking at your data!
- 48. Trending?
- 49. Correlation?
- 51. Enough, it’s a data science conference!
- 52. Improving current state Jet-engine health with Mr. Bayes
- 55. Engine performance monitoring N1 N2 Fuel Flo w Exhaust Gas Temperature
- 56. Engine performance monitoring Fuel Flo w Exhaust Gas Temperature
- 58. End of take-off flow
- 59. End of take-off flow
- 60. End of take-off flow
- 61. End of take-off flow
- 62. End of take-off flow
- 63. End of take-off flow
- 64. End of take-off flow
- 65. End of take-off – a problem If not detected – problems with engine health Difficult to do with classical signal processing What if a parameter fails (not so rarely)
- 66. End of take-off – a solution Time at vertical navigation mode selected (VNAV) Drop/rise in fuel flow (FF) and gas temperature (EGT) Search for changes around that point in time Gather the knowledge Calculate probability for EOT for unseen flight
- 67. End of take-off – a solution Time at vertical navigation mode selected (VNAV) Drop/rise in fuel flow (FF) and gas temperature (EGT) Search for changes around that point in time Gather the knowledge Calculate probability for EOT for unseen flight
- 68. End of take-off – a solution
- 69. End of take-off – a solution
- 70. End of take-off – a solution
- 71. End of take-off – a solution
- 72. End of take-off – a solution If crazy about this, search for PyData London 2016 video M. Vasiljevski & R. Rainone “Python flying at 40,000 feet”
- 74. Finding new concerns Novelty detection with principal component analysis
- 78. Standardising mean = 0 standard deviation = 1
- 82. Formally… Previous 700 x 123 matrix is A Multiply transpose of A with A to get covariance matrix, C. It’s 123 x 123 Do singular value decomposition of C to find eigenvectors and eigenvalues Eigen vectors coincide with directions of highest variance Keep just first couple of vectors to reduce dimensionality
- 93. Helping business Flight data upload monitor based on Markov chains
- 94. Flight data upload data data data
- 95. Flight data upload data data data
- 96. Flight data upload data data data
- 97. Flight data upload data data data
- 98. Flight data upload data data data
- 99. Flight data upload data data data State1 = M x State0 State2 = M x State1 = M x (M x State0) = M2 x State0 StateN = MN x State0 Don’t have to know states, just powers of M (probabilities)
- 100. Flight data upload data data data
- 101. Flight data upload data data data
- 102. Flight data upload data data data
- 103. Flight data upload data data data
- 104. Flight data upload data data data
- 105. Flight data upload data data data
- 107. Credits
- 112. MILAN
- 115. Main takeaways from this talk
- 116. LEARN
- 117. HAVE FUN
- 118. SHARE KNOWLEDGE
- 119. THINK
- 120. PLAY
- 121. THANK YOU & SAFE FLYING
Editor's Notes
- Depending on where we are in the world means various standards that FDRs need to comply to; visibility in wreckage (orange)
- The crash was caused primarily by the aircraft's automated reaction which was triggered by a faulty radio altimeter. This caused the autothrottle to decrease the engine power to idle during approach. The crew noticed this too late to take appropriate action to increase the thrust and recover the aircraft before it stalled and crashed.[9] Boeing has since issued a bulletin to remind pilots of all 737 series and BBJ aircraft of the importance of monitoring airspeed and altitude, advising against the use of autopilot or autothrottle while landing in cases of radio altimeter discrepancies.
- “I don’t know…, pure mathematician, Python developer, data scientist, pizza lover, feeling a bit home sick for Italy, …” Holds a PhD in theoretical mathematics, more specifically, in group theory. He is also a very capable developer and can easily communicate results of analyses. All of that makes him a great data scientist. Last but not least, he cooks great pasta carbonara - main reason for employing him.
- “I don’t know…, pure mathematician, Python developer, data scientist, pizza lover, feeling a bit home sick for Italy, …” Holds a PhD in theoretical mathematics, more specifically, in group theory. He is also a very capable developer and can easily communicate results of analyses. All of that makes him a great data scientist. Last but not least, he cooks great pasta carbonara - main reason for employing him.
- Intro: Legal requirement, explain SMS, how the airlines are obliged
- Aircraft flying
- By the way, is it trying to stop or is accelerating down the rwy? Where is the second black box?
- Aircraft on the ground
- Every manufacturer, a/c type has different set of parameters recorded. Even the same a/c type can have different parameters – like OS, imagine same file on multiple computers – different locations, but still comparable when clicked
- Data cleansing, Different frequencies, Parameter naming convention important
- Why is there missing data – normal or problem with the parameter?; change in behaviour of acceleration normal – why?
- Why is there missing data – normal or problem with the parameter?; change in behaviour of acceleration normal – why?
- Why is there missing data – normal or problem with the parameter?; change in behaviour of acceleration normal – why?
- Why is there missing data – normal or problem with the parameter?; change in behaviour of acceleration normal – why?
- Mainly with charts and numbers coming from the data
- Very often forgotten We live in the data-driven world We used to rely on our hunch feeling and on flying planes Nowadays, we are overwhelmed by quantities of data and automation (autopilot) We are deceived by charts and numbers The solution: somehow combine previous knowledge and data that are constantly being stored in databases across the globe
- Insert image
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- Examples of 1) missing data; 2) corrupt parameter
- The first outlier – remarkably difficult to analyse, outlier due to E1N1 Hi after TDWN, causes: AT engaged until a couple of seconds after TDWN in high cross wind at 45% (pilot to handle control column instead to monitor the speed); concerns: deep, baloon, hard landings
- Contrary to the previous example, AT was diengaged, landing into high cross wind at 45 degrees
- AT mach mode engaged during the landing – poor airmanship, too low speed, can stall
- Firm landing, The 1.9g peak was a result of the deployment of speed brakes. The trace shows a perfect example of normal-g trace – two close peaks after KTI (two wheels), and then smaller one corresponding to the nose wheel, and third, more pronounced peak for the breaks
- Examples of 1) missing data; 2) corrupt parameter