Downloaded 85 times
Uploaded byDataStax
The Big Data Ecosystem for Financial Services
1) The document discusses the use of big data and modern data architectures in the financial services industry. 2) It notes challenges such as aging IT systems, high costs, and lack of resources that are barriers to using big data. 3) Reference architectures are presented from IBM, Oracle, and DataStax that incorporate technologies like Hadoop, Spark, and Cassandra to power applications like customer 360, fraud detection, and real-time analytics.
More Related Content
Similar to The Big Data Ecosystem for Financial Services
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
The Big Data Ecosystem for Financial Services
- 1. Martin Van Ryswyk-EVP of Engineering, Datastax The “Big Data” Ecosystem for Financial Services
- 2. 53% can’t distinguishbetween banks – 71% would rather go to the dentist – Digital is the preferred channel – 1 in 3 willing to switch banks © 2015 DataStax, All Rights Reserved. 2 Source: The Millennial Disruption Index (http://bit.ly/1gp93Q9)
- 3. IT budgets underpressure – Services and labor costs have exploded – ROE at historically low 6% Source: Managing Complexity (http://owy.mn/1UNwPPy) © 2015 DataStax, All Rights Reserved. 3
- 4. “lack of staffresources continues to be the number one barrier for financial services organizations in 2014.” © 2015 DataStax, All Rights Reserved. 4 Source: 2014 Payments Fraud Survey Summary of Results (http://bit.ly/1POwxkF)
- 5. “58% of FinancialServices organisations see Risk and Operational Management as the drivers for using Big Data. 66% see fragmentation of Systems/IT complexity as a significant hurdle in achieving this.” © 2015 DataStax, All Rights Reserved. 5 Source: https://datafloq.com/read/financial-services-firms-leveraging-big-data-infog/352
- 6. The New DatabaseMarket Legacy Post Relational TransactionalAnalytical © 2015 DataStax, All Rights Reserved. 6
- 7. Architecture of aModern Application © 2015 DataStax, All Rights Reserved. 7 Web App Queue Ingest Operational Database
- 8. Architecture of aModern Application © 2015 DataStax, All Rights Reserved. 8 Web App Queue Ingest Operational Database Solr
- 9. Architecture of aModern Application © 2015 DataStax, All Rights Reserved. 9 Web App Queue Ingest Operational Database Hadoop Solr Queue Hive Search Batch Processing
- 10. Architecture of aModern Application © 2015 DataStax, All Rights Reserved. 10 Web App Queue Ingest Spark Streaming Operational Database Hadoop Solr Queue Hive Search Batch Processing
- 11. © 2015 DataStax,All Rights Reserved. 11
- 12. IBM Real-time AnalyticsReference Architecture Source: Explore the advanced analytics platform (http://ibm.co/1PLAneB)© 2015 DataStax, All Rights Reserved. 12
- 13. Oracle Reference Architecture Source:Big Data in Financial Services and Banking (http://bit.ly/1GP8mmE) © 2015 DataStax, All Rights Reserved.
- 14. Hadoop Real-time AnalyticsReference Architecture Source: Designing Fraud-Detection Architecture That Works Like Your Brain Does (http://bit.ly/1OeAnDb)© 2015 DataStax, All Rights Reserved. 14
- 15. “70% of Hadoopdeployments will not meet cost savings and revenue generation objectives due to skills and integration challenges.” © 2015 DataStax, All Rights Reserved. 15 Source: Hadoop promises not yet paying off (http://tek.io/1MrHtSx)
- 16. © 2015 DataStax,All Rights Reserved. 16 DataStax Enterprise is the database for cloud applications
- 17. HTTP Application MessageQueue Streamin g Analytics Batch Analytics Real-time DSE Real-time Analytics Reference Architecture © 2015 DataStax, All Rights Reserved. 17
- 18. Flying a planeis not the same as building it… © 2015 DataStax, All Rights Reserved.
- 19. Gartner ODBMS MQ2015 © 2015 DataStax, All Rights Reserved. 19
- 20. DataStax Use Cases •MDM: Customer 360, Product Catalog • Personalization and Recommendation • Internet of Things (IoT) and Time Series • Fraud Detection • List Management • Messaging • Inventory Management • Authentication © 2015 DataStax, All Rights Reserved. 20
- 21. Customer 360° /User Profile • Consolidate data, reduce complexity • Alleviate strain on legacy systems • Integrated data sovereignty • Eliminate downtime • Recommend higher margin products • Customer personalization • Audit, and regulatory compliance • Full text indexing, document search © 2015 DataStax, All Rights Reserved. 21 “DataStax has allowed us to break the complexities of our legacy systems, empowering us to leverage data to innovate and make more informed decisions so we can provide a truly personalized and premium experience to our customers.”
- 22. “DataStax Enterprise givesus the power to change information in our supporting systems into actionable insights… we can not only scale but give our users a real-time, engaging customer experience.” • Financial product recommendation • Provide the “small bank” feel through personalization • Real-time recommendation using transaction data, search, and browsing history. • Measure engagement and satisfaction to prevent flight • Website and mobile metrics 22 Recommendation, Engagement, and Customer Churn Prevention © 2015 DataStax, All Rights Reserved.
- 23. Payments and Transactions •Transaction history tagging, search and budgeting, cash flow prediction • Product recommendation based on purchasing history. • Data safe against loss of node, rack, or data center. • Shed load from legacy systems to reduce cost and provide better user experience. “We had reached the limits of our scalability. We wanted to remove all single points of failure. We had to be active-active. That's why we chose Cassandra.” © 2015 DataStax, All Rights Reserved. 23
- 24. Regulatory Compliance andCapital Adequacy “The platform enables traders to capture vast amounts of versioned market data which is then used for risk management purposes in real time.” © 2015 DataStax, All Rights Reserved. 24 • Capture and analyze every trade, every position for every trader. • Real-time risk analysis. • Globally replicated, 100% uptime. • Low latency, high ingest.
- 25. Market Tick andTime Series Data • Capture high fidelity time series information at high ingest rates, and low latencies • 100% uptime even in the event of data center loss © 2015 DataStax, All Rights Reserved. 25 “We primarily use Cassandra as a distributed data store for Financial Instrument and Time- Series data, where the ‘master’ data stores cannot scale globally and provide the high availability needed to support a global application with heavy user demand.”
- 26. Mission Critical Applications “DSEis a perfect match… easing availability challenges by being active- active and having an always-on architecture” © 2015 DataStax, All Rights Reserved. 26 Source: Exploiting Hotel Cassandra (http://bit.ly/1mf0KRE)
Editor's Notes
- #3 Banks are in a crisis. 53% of consumers can't distinguish between the offerings of the large banks 71% of consumers in the US would rather visit the dentist than go to a bank And although 99% of people are banked in the western world, 1 in 3 are unhappily banked. Being unhappily banked means being willing to switch banks in the next 90 days if given a compelling reason. That compelling reason could be a compelling product offering from a rival bank, or a bad experience with their existing bank. Banks are amongst the least loved brands by by the under 35 segment. At the same time consumers are asking for more. 68% say that how we interact with money will be totally different within 5 years 70% say that how we do payments will be radically different within 5 years 33% say they won't need a bank at all Most expect this future to be mobile and online first. Challenger Commercial Teaching Step = #1 The Warmer: layout the key challenges. Point to industry statistics. Raise the impending challenge …
- #4 At the same time when technology is expected to be critical to the future of banking, their return on IT spending is abysmal. It's on par with utilities. This is down from 30% ROE prior to the global financial crisis. When you dig into the numbers it's because of complexity and an explosion in services spending. Challenger Commercial Teaching Step = #1 The Warmer: layout the key challenges. Point to industry statistics. Raise the impending challenge …
- #5 The complexity of solutions, and the skills required to build or run these solutions is proving to be a major hurdle. It's gotten so bad that 51% of retail or financial services firms have no mobile anti-fraud strategy despite it's impending rollout. Challenger Commercial Teaching Step = #1 The Warmer: layout the key challenges. Point to industry statistics. Raise the impending challenge …
- #6 The complexity of solutions, and the skills required to build or run these solutions is proving to be a major hurdle. It's gotten so bad that 51% of retail or financial services firms have no mobile anti-fraud strategy despite it's impending rollout. Challenger Commercial Teaching Step = #1 The Warmer: layout the key challenges. Point to industry statistics. Raise the impending challenge …
- #7 Lets first start with what are we, and how we help address this issue. DataStax is a database vendor. The database market is still divided into two camps, transactional real-time workloads, and analytical data warehousing workloads. What's changed is the players and the scale at which we operate. DataStax is an operational data store in direct competition with Oracle or DB2. It's not a data-warehousing product like Hadoop. DataStax is designed for low latency, customer facing, real-time queries. Challenger Commercial Teaching Step = #1 The Warmer: layout the key challenges. Point to industry statistics. Raise the impending challenge …
- #8 To help place the database in context you need to know what a basic application looks like You've got a web application that takes actions from a user, and those actions or requests go into a queue. You've then got workers that take these requests and either read or write data from the database, and optionally respond to the user. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight The architecture slides help provide context as to why a platform or holistic response is necessary to address the complexity problem
- #9 That works for a while, but you're going to want to add search capabilities so that you can handle complex queries and Google like full text search or recommendation that users have become accustom too. People typically address this by adding another system like Solr or ElasticSearch and do synchronization between the main database and the search database. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight The architecture slides help provide context as to why a platform or holistic response is necessary to address the complexity problem
- #10 But that only gets you so far, soon you'll need to start offering true recommendations in order to drive up customer engagement, or do cross selling to increase revenue. In order to address that need people typically add a Hadoop cluster, and that adds a whole lot more complexity and cost. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight The architecture slides help provide context as to why a platform or holistic response is necessary to address the complexity problem
- #11 And soon you realize that batch recommendations run every couple of days or months aren't very good because they're not contextual. They don't reflect what the user is actually doing. To provide good personalized experience or recommendations you need to do streaming or real-time recommendations. To handle this people typically add yet another system such as Spark Streaming, or Storm. Soon you've got at least 8 separate systems supporting one "basic" application. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight The architecture slides help provide context as to why a platform or holistic response is necessary to address the complexity problem
- #12 And this is just one project making technology decisions. Every team is doing this independently with no standardization, typically with no support contract. It's resulted in a sprawl of vendors with no ability to gain efficiencies or develop best practices. Every team is making the same mistakes over and over again. Challenger Commercial Teaching Step = #3 Rational Drowning – this is the business case for why the reframes are worth customers time & insert the fear/uncertainty/doubt
- #13 And it's not just me making things up. Here is a diagram from IBMs blog that is the functional equivalent of the system I just showed you. It's even more complex with even more moving parts. I don't want to just pick on IBM, Oracle is just as bad. Each of these boxes in another system, with a different operation run book, with another team supporting it. Each box is a staffing challenge and a drag on agility. Every box is a liability. And remember this is just anti-fraud, not a transaction, not a recommendation, not personalization, not session management, not scheduling, not caching. This is why these solutions are so expensive. Too many moving pieces too many teams to staff, too many things to learn. THIS is just too complex. Complexity means missed deadlines, spiraling costs, projects never actually gets to production. The key to eliminating complexity is eliminating the number of systems required to deliver the required functionality. If you can use the same system to access the data through multiple vectors it eliminates external systems, and therefore limits complexity. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight IBM and Hadoop architecture slides help reframe that it is about the architecture.
- #14 When you look at existing solutions built on top of relational technologies you begin to understand why. Here is a realtime analytics reference diagram from IBM. I don't want to just pick on IBM, Oracle is just as bad. Each of these boxes in another system, with a different operation run book, with another team supporting it. Each box is a staffing challenge and a drag on agility. Every box is a liability. And remember this is just anti-fraud, not a transaction, not a recommendation, not personalization, not session management, not scheduling, not caching. This is why these solutions are so expensive. Too many moving pieces too many teams to staff, too many things to learn. THIS is just too complex. Complexity means missed deadlines, spiraling costs, projects never actually gets to production. The key to eliminating complexity is eliminating the number of systems required to deliver the required functionality. If you can use the same system to access the data through multiple vectors it eliminates external systems, and therefore limits complexity.
- #15 Just dropping in a NoSQL solution won't help. Here is the same reference architecture from Cloudera. It's just as complex. They may help with scale, but they don't address the underlying complexity issues. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight IBM and Hadoop architecture slides help reframe that it is about the architecture.
- #16 As a result 70% of Hadoop deployments won't meet revenue or project goals. The big elephant in the room isn't just the Hadoop logo, it's that most big data solutions aren't delivering on their promise. Challenger Commercial Teaching Step = #3 Rational Drowning – this is the business case for why the reframes are worth customers time & insert the fear/uncertainty/doubt
- #18 As a result you end up with a much simpler solution. If you compare the two slides you'll see that all of the same functionality is present. The difference is fewer components, and less complexity. NO ETL between point solutions You don't have to bolt on replication. You don't have to bolt on integration between many point solutions. It's provided for you by DSE. Most of the complexity in the other solutions comes from trying to synchronize data between multiple point solutions. Requests come in through an application, and are persisted to a message queue. The streaming nodes consume messages from the queue, persist and to themselves. We'll go into this in more detail, but it's important for you to see it so that you have some context. We use Cassandra as the underlying data store. Cassandra handles all of the replication, and high availability. Challenger Commercial Teaching Step = #6 Your Solution – our solution is unique and better than anyone else
- #19 But there is a big difference between “developing an application using Cassandra” and “Developing Cassandra” They are different skills… both valuable, both require a lot of training and talented engineers
- #20 The completeness of this vision is why Gartner put us in the leader quadrant. Challenger Commercial Teaching Step = #6 Your Solution – our solution is unique and better than anyone else
- #21 These capabilities have allowed us to penetrate deeply into financial services for mission critical applications extraordinarily quickly. I just want to give you a brief overview of some logos and use cases before we dive in. Real Time Payments Engine: Allied Payment Post Trade Risk Reporting/Capital Adeqquacy: Credit Suisse, UBS, BNP Paribas Global Reference Data Replication: UBS Market Data Capture/Replay: Blue Mountain, Sirca Post Trade Surveillance: UBS, Credit Suisse Real Time Fraud Analytics: Can’t share names – BofA – Fraud Analytics as part of the transactional flow. Not going to wait a minute for a period of time to determine fraud at the atm or time of a credit card transaction Forward Curve Analytics: UBS Challenger Commercial Teaching Step = #4 Emotional Impact – is telling stories about other companies and/or introducing the high-level challenges we solve Multi-Channel, Fraud, etc.
- #22 The first being Customer 360. Customer 360 allows you to do two things. Consolidate and simply user profile information. Challenger Commercial Teaching Step = #4 Emotional Impact – is telling stories about other companies and/or introducing the high-level challenges we solve Multi-Channel, Fraud, etc.
- #23 Challenger Commercial Teaching Step = #4 Emotional Impact – is telling stories about other companies and/or introducing the high-level challenges we solve Multi-Channel, Fraud, etc.
- #24 Challenger Commercial Teaching Step = #4 Emotional Impact – is telling stories about other companies and/or introducing the high-level challenges we solve Multi-Channel, Fraud, etc.
- #25 Challenger Commercial Teaching Step = #4 Emotional Impact – is telling stories about other companies and/or introducing the high-level challenges we solve Multi-Channel, Fraud, etc.
- #26 Challenger Commercial Teaching Step = #4 Emotional Impact – is telling stories about other companies and/or introducing the high-level challenges we solve Multi-Channel, Fraud, etc.
- #27 I really do want to emphasize the mission critical nature of the applications that we support. If you're serious about supporting mission critical, real time applications there really is no other choice in the market. Challenger Commercial Teaching Step = #6 Your Solution – our solution is unique and better than anyone else (product and in this case, services)