Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Application of the Actor Model to Large Scale NDE Data Analysis

208 views

Published on

The Actor model of concurrent computation discretizes a problem into a series of independent units or actors that interact only through the exchange of messages. Without direct coupling between individual components, an Actor-based system is inherently concurrent and fault-tolerant. These traits lend themselves to so-called “Big Data” applications in which the volume of data to analyze requires a distributed multi-system design. For a practical demonstration of the Actor computational model, a system was developed to assist with the automated analysis of Nondestructive Evaluation (NDE) datasets using the open source Myriad Data Reduction Framework. A machine learning model trained to detect damage in two-dimensional slices of C-Scan data was deployed in a streaming data processing pipeline. To demonstrate the flexibility of the Actor model, the pipeline was deployed on a local system and re-deployed as a distributed system without recompiling, reconfiguring, or restarting the running application.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Application of the Actor Model to Large Scale NDE Data Analysis

  1. 1. Application of the Actor Model to Large Scale NDE Data Analysis SPIE Smart Structures and NDE for Industry 4.0 4 - 8 March 2018
  2. 2. Emphysic® Chris Coughlin
  3. 3. • Distributed Processing Architectures • Actor Model • Defect Detection Algorithm • Sample Results • Q & A Agenda Introduction Myriad Desktop UI Emphysic Actor Model for NDE Analysis
  4. 4. Comparing Distributed Processing Models Architecture Apache Spark Batch Processing Model (aka Map-Reduce) Apache Storm Stream Processing Model Akka Actor Processing Model Emphysic Actor Model for NDE Analysis
  5. 5. • Lightweight • 1 actor ~ 300 bytes in RAM • Fault-tolerant • “Let it crash” • Configurable • Understandable Benefits of Actor Model Emphysic Actor Model for NDE Analysis
  6. 6. • Actor-based “pipeline parallelism” structure • Algorithm is divided into a series of concurrent stages • Each stage in the algorithm consists of a central routing Actor, one or more worker Actors, and a work queue • Output of one stage  input to subsequent stage • Pyramid Actor blurs and subsamples data, sends each step to a Window Actor • For each Window the Window Actor sends to a Defect Scanner Actor • Defect Scanner Actor sends to Reporter Actor Overview Architecture Defect Detection Structure Emphysic Actor Model for NDE Analysis
  7. 7. • Blur • Convolve with a blur kernel, usually • Box or • Gaussian • Usually approximated w. 3 Box filter passes • Must account for edges • Subsample • Also known as down-sampling or decimation • Take every nth element Pyramid Actor Algorithms & Components Pyramid algorithm Emphysic Actor Model for NDE Analysis
  8. 8. Gaussian Pyramid Algorithms & Components STEP 1 STEP 2 STEP 3 STEP 4 80×80 40×40 20×20 10×10 Emphysic Actor Model for NDE Analysis
  9. 9. • Scan across each dataset • Each window is scanned independently for defect signals • Tradeoffs: • Speed of scan affected by size of input data and • Size of window • Amount of overlap (smaller step size) • Step size makes it more likely to detect ROI but also more likely to find the same ROI multiple times Window Actor Algorithms & Components Sliding Window Algorithm Emphysic Actor Model for NDE Analysis
  10. 10. • Simple Interface – get data, return True if defect found • Bundle online learning algorithm with (optional) preprocessor into a single small (~ 10kB) binary package, or • Parallelize existing algorithms • No need to port to Java, can call external code (Python, MATLAB, C++, etc.) with system call Defect Scan Actor Algorithms & Components Defect scanner interface Emphysic Actor Model for NDE Analysis
  11. 11. • Compiles results of defect scanning • Every stage in the process adds metadata to the message • Data ingestion – data source • Pyramid – scaling factor • Sliding Window – position within scaled data • Defect detection – ROI found • Metadata allows the Reporting stage to find ROI relative to the original input Reporter Actor Algorithms & Components Reporting ROI Results Emphysic Actor Model for NDE Analysis
  12. 12. • Training data • 2-D slices of ultrasonic • 15x15 elements • Model • Passive Aggressive learning algorithm • Sobel edge detection preprocessing • Pipeline • 423 workers • 1 Ingestor • 2 Scalers • 4 Pre-processors • 128 Sliding • 256 Defect • 32 Reporters Using A Model Demonstration Emphysic Actor Model for NDE Analysis
  13. 13. • Sample Input • 33 separate data files (CSV, JPEG, TIFF, etc.) • 60 million data points • Single System Single Process (SSSP) • Eight cores 32GB RAM • 1 process • Single System Multiprocess (SSMP) • Eight cores 32GB RAM • 184 processes • Multisystem Multiprocess (MSMP) • Eight cores 32GB RAM local • Eight cores 32GB RAM remote (Azure VM) • 88 local processes 128 remote (216 total) Trial Number Architecture SSSP SSMP MSMP 1 302.66 106.69 107.28 2 299.16 99.43 106.94 3 297.00 111.87 106.11 4 303.22 110.20 106.05 5 299.39 103.83 106.13 Mean Processing Time [s] 300.28 106.40 106.50 Mean Throughput [Points Per Second] 2.07E+05 5.87E+05 5.85E+05 Sample Throughputs Emphysic Actor Model for NDE Analysis
  14. 14. Sample Throughputs Emphysic Actor Model for NDE Analysis
  15. 15. Sample Throughputs – Doubled Input Emphysic Actor Model for NDE Analysis
  16. 16. ANY QUESTIONS? Emphysic Actor Model for NDE Analysis

×