Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Digitale Fabriek (deel 5)A practical approach for theimplementation of new services2012-02-14
Background+ Flanders’ Mechatronics Technology Centre    Partners: machine builders like Atlas Copco, Bekaert, CNH, Dana… ...
Problem Statement+ Goal   Offer new services based on machine / product usage info+ New service kinds      Reporting    ...
Challenge 1: Is my data available? (1/2)+ Usage data    Logs are produced on the machine by the machine software.    Is ...
Challenge 1: Is my data available? (2/2)+ So, you’re stuck with low-level log files      Created by the developer based o...
Challenge 2: Where to interpret my data?                    Machine Office        Original                                ...
Challenge 3: How to interpret my data?+ Difficult? Depends on your needs…+ Easy steps    Filtering    Statistics    His...
ESP & CEP: Intro+ ESP = Event Stream Processing+ CEP = Complex Event Processing+ Background    Financial fraud detection ...
ESP & CEP: How does it work? (1/2)+ First some definitions    Event        Happens at a given time        Carries some ...
ESP & CEP: How does it work? (2/2)+ Trivial data filtering     Events can be filtered on their contents      “select id, ...
ESP & CEP: Raising the abstraction+ Raising abstraction + detect errors    Initial log file contains low-level events   ...
ESP & CEP: Example+ Example: Robot with moving arm    Initial log stream contains robot arm movements:     ArmDockedInBas...
Overview+ Problem Statement+ Challenges   1. Is my data available?      You only have low-level logs   2. Where to analyz...
LogAn tool+ FMTC created two products    LogAn Studio        Definition of patterns        Off-line matching of the pat...
Concrete usage examples+ Usage 1: Helpdesk improvement    Problem: Log files not optimally used        Content too detai...
Concrete usage examples+ Usage 2: Log restructuring / documentation    Problem: Logs difficult to understand        Log ...
Concrete usage examples+ Usage 3: Online monitoring    Problem: Associate alarms with complex events        Alarms trigg...
Concrete usage examples+ Usage 4: Online filtering    Problem: What data should be sent to backoffice?        Huge log-f...
Overview+ Problem Statement+ Challenges   1. Is my data available?      You only have low-level logs   2. Where to analyz...
Limitations+ ESP & CEP    Defining patterns quickly becomes complex.     Knowing the semantics is important.      Not su...
Future abilities of LogAn+ Integration of pattern mining    Today, patterns are defined manually.     Expert “adds” his k...
Conclusion+ Machine does NOT produce the desired information    Too detailed for direct use    Too big to send to back-o...
Contact                   Johan Van Noten                     Project Leader                johan.vannoten@fmtc.be        ...
Upcoming SlideShare
Loading in …5
×

2012 02-14-digitale fabriek v - Een praktische aanpak voor implementatie van nieuwe services - Johan Van Noten, FMTC

465 views

Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

2012 02-14-digitale fabriek v - Een praktische aanpak voor implementatie van nieuwe services - Johan Van Noten, FMTC

  1. 1. Digitale Fabriek (deel 5)A practical approach for theimplementation of new services2012-02-14
  2. 2. Background+ Flanders’ Mechatronics Technology Centre  Partners: machine builders like Atlas Copco, Bekaert, CNH, Dana…  Concrete joint research projects  Field of Mechatronics  Mechanics  Electronics  Software  We do not develop commercial products+ Cooperation with KULeuven – Sirris+ Johan Van Noten  Software background  a.o. project concerning log analysis © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p2
  3. 3. Problem Statement+ Goal Offer new services based on machine / product usage info+ New service kinds  Reporting  Diagnostics / error alerts  Maintenance scheduling  Prognostics  Error analysis  Usage advice towards customer  Market analysis  …+ All you need is data… © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p3
  4. 4. Challenge 1: Is my data available? (1/2)+ Usage data  Logs are produced on the machine by the machine software.  Is this exactly what I need? Could happen, but probably not.+ Can I ask my developer to produce the exact data?  Too late: machine’s software is already in the field.  Wrong focus: data collection and processing is not the developer’s task  Too complex: mixing functionality and data processing complexifies software  Impossible: software could be 3rd party, you can’t control its features Don’t count on “readily available processed data”. © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p4
  5. 5. Challenge 1: Is my data available? (2/2)+ So, you’re stuck with low-level log files  Created by the developer based on passed experience  Partially for debugging, partially for reporting  Reflects execution flow of the machine  Potentially multiple sets of data (corresponding to machine parts)+ Does this match my desired data?  Nope… ? Derivable? Way too detailed Desired data log data (statistics or other) © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p5
  6. 6. Challenge 2: Where to interpret my data? Machine Office Original Analysed data data Original Analysed data Reduced data transfer needs Original data Analysed. data. © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p6
  7. 7. Challenge 3: How to interpret my data?+ Difficult? Depends on your needs…+ Easy steps  Filtering  Statistics  Historical data querying  Traditional techniques apply+ Typically more challenging  Time based relationships: “Did this error occur within two minutes after system startup?”  Sliding window queries: “Did the user push this button when the machine’s rpm exceeded 600 for at least 2 minutes?” ESP & CEP can be the answer © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p7
  8. 8. ESP & CEP: Intro+ ESP = Event Stream Processing+ CEP = Complex Event Processing+ Background  Financial fraud detection  Network intrusion detection  Monitoring of complex application servers+ Strengths  Can do the usual filtering and statistics  Strong in time relationships  “A followed by B within 10s”  “All occurances of A between B and C where C occurs < 5’ after B”  Works on ethernal streams  “Mean value of A during the last 5 seconds” © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p8
  9. 9. ESP & CEP: How does it work? (1/2)+ First some definitions  Event  Happens at a given time  Carries some information about what happened  Pattern  Potentially complex relationship between events  Pattern matching  Looking for occurrences of given patterns in a data set+ Working principle  You define a set of patterns that express your needs.  Data flows through a set of existing queries (≠ SQL: query works on set of existing data) © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p9
  10. 10. ESP & CEP: How does it work? (2/2)+ Trivial data filtering  Events can be filtered on their contents “select id, pressure from [every ValveEvent(pressure > 5) ]” Matches each time such a ValveEvent appears on the stream.  Considers events one by one (no relationships)  Good for data reduction, selection of interesting events+ Complex Event Processing (CEP) When a CEP pattern matches, this match can be considered as the occurence of a “higher order event”.  Relates multiple events by time, causality, origin…  “every a=ValveEvent -> b=ValveEvent(pressure > 2*a.pressure) where timer.within(3s)]”  Each match is considered as a new event of newly created stream “SuddenPressureIncrease”. © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p10
  11. 11. ESP & CEP: Raising the abstraction+ Raising abstraction + detect errors  Initial log file contains low-level events  Based on CEP patterns, higher level events can be derived  These higher level events can then be re-used in other CEP patterns. Pattern 1 Pattern 2 © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p11
  12. 12. ESP & CEP: Example+ Example: Robot with moving arm  Initial log stream contains robot arm movements: ArmDockedInBase, ArmMovesUp, ArmMovesDown, ArmTurns  Pattern 1: Raising abstraction  new derived event kind RobotAction = “ArmDockedInBase  followed by any number of ArmMovesUp, ArmMovesDown, ArmTurns  followed by ArmDockedInBase”  Each RobotAction contains: total execution time , total length traveled.  Pattern 2: Detecting error  Selects some of the newly generated RobotAction events: e.g. select * from RobotAction where totalLength/totalTime > 2m/s  The selected events indicate a usage of the robot over specification. Second query would be more complex without the first one. © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p12
  13. 13. Overview+ Problem Statement+ Challenges 1. Is my data available?  You only have low-level logs 2. Where to analyze my data?  Locally on machine or at office 3. How to analyze my data?  ESP & CEP make it easier LogAn tool + concrete usage examples © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p13
  14. 14. LogAn tool+ FMTC created two products  LogAn Studio  Definition of patterns  Off-line matching of the patterns  Graphical analysis of the results  LogAn Matcher  Online matching of the patterns+ Based on ESP & CEP implementation:  Esper  By EsperTech (http://www.espertech.com)+ Note: next examples mention LogAn, but principle is valid for all ESP & CEP. © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p14
  15. 15. Concrete usage examples+ Usage 1: Helpdesk improvement  Problem: Log files not optimally used  Content too detailed for helpdesk agents  Escalation to developers needed  disturbing  Solution  Developer analyses occurred issue through LogAn  Developer adds expert knowledge as a pattern to LogAn  LogAn knows patterns for all kinds of issues  Helpdesk agent offers new log file to LogAn  LogAn immediately shows occurrences of previously identified patterns © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p15
  16. 16. Concrete usage examples+ Usage 2: Log restructuring / documentation  Problem: Logs difficult to understand  Log is flat list of events  Too detailed / not clearly structured  Even developer does not easily navigate  Solution  Developer once adds patterns that clarify structure  Huge production log gets structured in production day, batch, job, task…  New log can now easily navigate through detected structure © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p16
  17. 17. Concrete usage examples+ Usage 3: Online monitoring  Problem: Associate alarms with complex events  Alarms trigger on complex events  Difficult to program in traditional ways  Solution  On-line LogAn Matcher post-processes the machine’s output  Same patterns as off-line  Can be configured to produce the desired info without touching the machine’s core code.  Can fire locally (alerts, local optimization…) © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p17
  18. 18. Concrete usage examples+ Usage 4: Online filtering  Problem: What data should be sent to backoffice?  Huge log-file data is too big to be sent  It can not be determined up-front what needs to be sent  Solution  Dynamically define which (processed) data needs to be sent to back-office ~ Technically similar to previous example.+ Usage 5: Pre-process data for analysis  Problem: Log data is too detailed for traditional analysis methods  Data mining, statistics etc require desired data objects  Logs only contain them in a derivable way  Deriving them with classic techniques is difficult  Solution:  Use ESP & CEP to pre-process the data to higher level data objects © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p18
  19. 19. Overview+ Problem Statement+ Challenges 1. Is my data available?  You only have low-level logs 2. Where to analyze my data?  Locally on machine or at office 3. How to analyze my data?  ESP & CEP make it easier+ LogAn tool+ Concrete usage examples Limitations & Future abilities © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p19
  20. 20. Limitations+ ESP & CEP  Defining patterns quickly becomes complex. Knowing the semantics is important.  Not suitable for sporadic use  Highly scalable & computationally intensive. Also used for server applications.  Not suitable for embedded environments. Requires PC alike environment+ Esper core is GPLv2  Gnu Public License v2 will be inherited by your software.  If required, make sure you have a commercial license. © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p20
  21. 21. Future abilities of LogAn+ Integration of pattern mining  Today, patterns are defined manually. Expert “adds” his knowledge to the tool by means of patterns.  Sometimes no-one knows which pattern to look for.  Algorithms exist to do pattern mining: Reveal certain patterns that seem to occur in your data.  LogAn will assist you in preparing your data & mining patterns.+ Assisted definition of patterns  FMTC wants to reduce the complexity of the pattern definition  This reduces the abilities as well  Full capabilities will be kept open but easy scenarios will be offered. © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p21
  22. 22. Conclusion+ Machine does NOT produce the desired information  Too detailed for direct use  Too big to send to back-office+ ESP & CEP techniques allow you to  Powerfully filter your data  Define complex patterns that relate events+ Typical uses  Increase the abstraction level  Easier next level patterns  Match on complex error patterns  Powerful alerting  Pre-process data to reduce amount  Send relevant info only  Pre-process data for analysis  Determine analysable data © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p22
  23. 23. Contact Johan Van Noten Project Leader johan.vannoten@fmtc.be +32 498 91 94 03 Flanders’ Mechatronics Technology Centre vzw Celestijnenlaan 300D 3001 Heverlee © FMTC vzw 2012 • STRICTLY CONFIDENTIAL • p23

×