Your SlideShare is downloading. ×
0

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Health & Status Monitoring (2010-v8)

573

Published on

This is a variant of a talk that I gave at Predictive Analytics World in February 2010.

This is a variant of a talk that I gave at Predictive Analytics World in February 2010.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
573
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Health & Status Monitoring: Two Case Studies
    Robert Grossman Open Data Group
    February 18, 2010
    1
  • 2. 1. Introduction
    2
  • 3. Traditional Approach
    Two types of variation:
    Common cause of variation (noise) occur as normal part of manufacturing process
    Special cause of variation represents a potential problem
    3
  • 4. 3 s
    Shewhart control chart used by NIST for calibrating the standard KG.
    4
    Source: NIST
  • 5. Shewhart / Deming Cycle
    Plan – identify opportunity or problem and make a plan.
    Do – implement the change on a small scale and collect the data.
    Check – perform a statistical analysis and check if there was an impact.
    Act – if there was an impact, broaden the scale and continuously improve your results.
    5
  • 6. Case Study 1. Data Center
    Thousands of servers
    Complex workloads
    Large variations are normal
    Problems make the front page
    6
  • 7. Case Study 2. Payments Network
    Billion+ cards
    100+ million terminals
    Millions of merchants
    Thousands of transactions per second
    Thousands of member banks
    Data highly heterogeneous
    Variations among products
    Variations among cardholders
    Variations among merchants
    Variations among banks
    Variation among payment networks
    7
  • 8. The Challenge Today
    Many sources and data feeds
    Data is complex and highly heterogeneous
    High volume, streaming data from around the world
    Multiple parties involved, each of which can modify the data in subtle ways
    8
  • 9. Health & Status Monitoring Systems
    9
  • 10. 2. The Technology
    10
  • 11. 11
    Observed Model
    Baseline Model
    CUSUM models
    GLR models
  • 12. Build more than 104 Models: One for Each Cell in Cube of Models
    15,000+ separate baselines
    • Build separate model for each bank (1000+)
    • 13. Build separate model for each geographical region (6 regions)
    • 14. Build separate model for each different type of merchant (over 800 types of merchants)
    • 15. For each distinct cube, build a distinct model
    Geospatialregion
    Type of Transaction
    Bank
    Modeling using Cubes of Models (MCM)
    12
  • 16. learning sets
    data updates
    1. data
    collection
    3. on-line scoring
    2. off-line modeling
    Entity/Feature Database
    Data Mining
    System
    PMML
    models
    features
    Model Consumer
    Data Mining
    Mart
    events
    Rules
    candidate alerts
    Operational systems, data feeds, warehouses, …
    4. reporting
    Dashboard engine
    reports
    13
  • 17. Augustus
    Augustus is an open source data mining platform:
    Used to estimate baselines for over 15,000 separate segmented models
    Used to score high volume operational data and issue alerts for follow up investigations
    Augustus is PMML compliant
    Augustus scales with
    Volume of data
    Real time transaction streams (15,000/sec+)
    Number of segmented models (10,000+)
    14
  • 18. Greedy Meaningful/Manageable Balancing (GMMB) Algorithm
    Breakpoint
    • More alerts
    • 19. Alerts more meaningful
    • 20. To increase alerts, add breakpoint to split cubes,order by number of new alerts, & select one or more new breakpoints
    • 21. Fewer alerts
    • 22. Alerts more manageable
    • 23. To decrease alerts, remove breakpoint,order by number of decreased alerts, & select one or more breakpoints to remove
    One model for each cell in data cube
    15
  • 24. 3. Case Studies
    16
  • 25. Case Study 1
    Open Cloud Testbed Monitor
    17
  • 26. Results
    Dozens of separate statistical baselines models developed and deployed.
    Effective for discovering nodes that are hindering effective use of OCC’s large data cloud.
    Dead nodes are easy to identify and remove.
    Removing just one or two “slow” nodes from a pool of 100 nodes can improve overall performance by 15% - 20+%.
    18
  • 27. Dashboard
    19
  • 28. Case Study 2
    Account
    Issuing Bank
    Payments Network
    Merchant
    Acquiring Bank
    20
  • 29. Program Structure
    Strategic objective identified early:
    “Identify and ameliorate data interoperability issues to improve the approval rate of valid transactions and the disapproval rate of invalid transactions, ...”
    Report quarterly to CIOs’ council with third-party endorsed monetary benefits summarized on an executive dash board
    Introduced data governance program early in project
    Developed payment transaction monitor that produced candidate alerts
    Set up investigation process to screen alerts and investigate those of interest
    Developed reference models and appropriate standards
    21
  • 30. Results
    ROI
    5.1x Year 1 (over 6 months)
    7.3x Year 2 (12 months)
    10.0x Year 3 (12 months)
    Over 15,500 separate statistical baselines models developed and deployed.
    Also developed appropriate rules-based models to make work of analysts more efficient.
    22
  • 31. 4. Summary
    23
  • 32. Business Process
    Strategic Objective
    Dashboard
    Governance
    Modeling Process
    Reference Model
    Investigative Process
    Monitor - produces candidate alerts
    InvestigativeProcess
    candidate alerts
    events
    program alerts
    24
  • 33. Some Lesson Learned
    Business Processes
    Importance of “C”-level executive support, dashboard reports, and a data governance program
    Modeling Processes
    Critical to build as many statistical models as the data required; used open source Augustus software for this
    Architecture separated offline modeling and online scoring
    Post processing with business rules to control workflow to analysts
    Investigative Processes
    It is not about the models and alerts – it is about optimizing the analysts’ workload and derived business value
    Small changes in report designs had large impact in the effectiveness of the alerts
    25
  • 34. Summary
    26
  • 35. For More Information
    Learn about Health and status monitoring
    • Open Data Group
    • 36. www.opendatatgroup.com
    Robert Grossman
    grossman.info at gmail.com
    rgrossman.com (blog)
    27
  • 37. References
    Joseph Bugajski, Chris Curry, Robert L. Grossman, David Locke, Steve Vejcik, Detecting Changes in Large Data Sets of Payment Card Data: A Case Study, Proceedings of The Thirteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2007), ACM, 2007
    Joseph Bugajski and Robert L. Grossman, An Alert Management Approach to Data Quality: Lessons Learned from the Visa Data Authority Program, Proceedings of the 12th International Conference on Information Quality, (ICIQ 2007).
    Walter A. Shewhart, Statistical Method from the Viewpoint of Quality Control, Dover, 1986.
    H. Vincent Poor and Olympia Hadjiliadis, Quickest Detection, Cambridge University Press, 2009.
    Augustus is an open source system available from augustus.googlecode.com.
    28

×