Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hadoop to the Enterprise

Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hadoop to the Enterprise

  • Login to see the comments

Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hadoop to the Enterprise

  1. 1. From Fraud Detection to Big Data Platform: Bringing Hadoop to the Enterprise at Fiducia & GAD IT AG Daniel Schmitt & Florian Herrmann October 25th 2016
  2. 2. 2 About us 9/15/2016World of Watson 2016 Daniel Schmitt (1985) • Karlsruhe, Germany • Business Intelligence Dep. at Fiducia & GAD IT AG since 2009 Topics Business Analytics Design and Implementation, Reporting, Planning and all topics related to Analytics Experience Apache Hadoop, Cognos BI, Cognos TM1, GeoInformation Systems, Cognos Enterprise Planning etc
  3. 3. 3 About us 9/15/2016World of Watson 2016 Florian Herrmann (1988) • Karlsruhe, Germany • Database Development Dep. at Fiducia & GAD IT AG since 2013 Topics Data Modelling and Database Design for core banking system, Performance Optimization and in-house consulting for all topics related to DBs Experience Apache Hadoop, Database Systems (DB2-Family, Oracle) etc
  4. 4. 1. The Challenge 2. The Solution 3. The Lessons Learned4. The Blueprint
  5. 5. 1. The Challenge Make the fraudsters shiver
  6. 6. 6 Fiducia & GAD IT AG at a glance 9/15/2016World of Watson 2016 Computer Center Services Integration Platform Competence Center Leading Banking System
  7. 7. 7 Fiducia & GAD IT AG at a glance 9/15/2016World of Watson 2016 167,000 workstations in banks 6,300m accounting entries per year 79m active accounts 36,000 self-service terminals 550m ATM cash withdrawals per year
  8. 8. Requirements • Evaluation of all user initiated online transactions on fraud suspicion • Integration in core banking system and existing banking processes (Fiducia & GAD is just the service provider not the owner!) • Model based on customer behavior • Flexible system design for a fast reaction on new fraud patterns 8 Fraud Detection for online banking 9/15/2016World of Watson 2016
  9. 9. 9 Fraud Detection for online banking 9/15/2016World of Watson 2016 Millions of transactions per day Up to 100 transactions per second Evaluation in less than 100 milliseconds System adjustment in minutes Be prepared for new datasources or -formats
  10. 10. 10 Fraud Detection for online banking 9/15/2016World of Watson 2016 Transaction handling Fraud Detection System Development of evaluation models Storage of all transactions Evaluation in milliseconds Flexible adjustment Evaluate Transaction Accounting and Forwarding
  11. 11. 2. The Solution One elephant to rule them all
  12. 12. The Solution 9/15/2016World of Watson 201612 Velocity Realtime evaluation of incoming data. Access on large data volume within milliseconds Variety Transactional data won’t be enough in foreseeable future Volume Store millions of transaction details each day over years Flexibility Quick response on changing fraud patterns. Integration of complex data structures. Model development based on current events
  13. 13. The Solution 13 9/15/2016World of Watson 2016 Pig Spark Hive Data Access Storm Phoenix HBase Governance Sqoop Kafka Flume Hadoop & YARN Operations Security RangerKnox Oozie Zoo- keeper Ambari
  14. 14. The Solution 14 9/15/2016World of Watson 2016 Cognos Bi Fidoop Gateway Big SQL Kafka Core Banking System Storm Realtime Processing Datasources R-Studio HiveHBase Spark Jobs …Java App Ambari Knox Ranger … Hadoop (IOP)
  15. 15. The Solution 15 9/15/2016World of Watson 2016 Potential Use Cases Master Worker Big Data Plattform Fraud Detection Usecase 2 Usecase 3
  16. 16. 3. The Lessons Learned What a year with the elephant taught us
  17. 17. The Lessons Learned 17 9/15/2016World of Watson 2016 - one has to manage things like hardware configuration, network architecture, disksizes, security and more - getting even the development skills can take much time (not to mention the understanding of a distributed system) - there is a bunch of components to get used to Hadoop is complex
  18. 18. The Lessons Learned 18 9/15/2016World of Watson 2016 Support means - vendor support - external support - (and maybe) internal support Support is a key to success
  19. 19. The Lessons Learned 19 9/15/2016World of Watson 2016 Even standard tasks can generate big effort or cause a deadlock - the advantage of fast feature availability comes with a price - some features are theoretically available but not enterprise ready - Hadoop is not an “out-of-the-box” tool
  20. 20. The Lessons Learned 20 9/15/2016World of Watson 2016 Open source within a distribution comes with a price Advantages: stability, component interoperability, easy installation, support … The price: Seeing fixed issues to be available in a project but not in your distribution can be frustrating Bugs and feature requests are difficult to handle as there is a distributor and a open source project
  21. 21. The Lessons Learned 21 9/15/2016World of Watson 2016 New technologies require a change of thinking - a distribution of open source projects isn’t a single vendor tool - establishing a distributed platform can require new processes or procedures - sometimes building up a new thing can help to get rid of old junk
  22. 22. The Lessons Learned 22 9/15/2016World of Watson 2016 Costs: hardware as for a cluster you have to buy servers software support as open source is free but not “for free” external support if you don’t have all skills (and you’ll need a lot) integration as a new platform has to be integrated decently Establish Hadoop as a plattform generates relevant inital costs
  23. 23. 4. The Blueprint How to get the elephant started
  24. 24. The Blueprint 24 9/15/2016World of Watson 2016 Take a simple use case (if possible)Hadoop is complex
  25. 25. The Blueprint 25 9/15/2016World of Watson 2016 Use as few components as possible Support is a key to success Hadoop is complex
  26. 26. The Blueprint 26 9/15/2016World of Watson 2016 In the beginning start with a security that is as simple as possible Hadoop is complex
  27. 27. The Blueprint 27 9/15/2016World of Watson 2016 Try to be agile in development as building up a plattform will be sophisticated Even standard tasks can generate big effort or cause a deadlock
  28. 28. The Blueprint 28 9/15/2016World of Watson 2016 Be sure to have good management support for budget decisions and escalations Establish Hadoop as a plattform generates relevant inital costs New technologies require a change of thinking
  29. 29. The Blueprint 29 9/15/2016World of Watson 2016 Concentrate on relevant parts and avoid to much additional effort where possible (buildtools etc) Establish Hadoop as a plattform generates relevant inital costs
  30. 30. The Blueprint 30 9/15/2016World of Watson 2016 Calculate with training time and bugfixing Even standard tasks can generate big effort or cause a deadlock Hadoop is complex
  31. 31. The Blueprint
  32. 32. Thank You

×