DataStax Enterprise in Practice (Field Notes)
•Download as PPTX, PDF•
1 like•535 views
This document discusses several use cases from banks and financial institutions using DataStax Enterprise (DSE). It describes challenges faced and approaches taken for: 1. A global merchant transactions use case involving 100TB of credit card charges and migrating from mainframe to DSE for scalability and high availability. 2. A global payments use case replacing an Exadata dependency and ingesting over 5,000 transactions per second into DSE. 3. A global billing use case scaling an Oracle Coherence deployment with DSE to aggregate usage data. It also discusses challenges around new technologies, data modeling, hardware requirements, and internal politics that were addressed when implementing DSE.
![Company Confidential© 2016 DataStax, All Rights Reserved.
What are all the other banks doing?
What do you mean SSDs?
What’s the difference with [MongoDB]?
How many nodes will I need?
How do I load data from [Oracle]?
What are all the other banks doing?
DS
E](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Similar to DataStax Enterprise in Practice (Field Notes)
Similar to DataStax Enterprise in Practice (Field Notes) (20)
More from DataStax
More from DataStax (20)
Recently uploaded
Recently uploaded (20)
DataStax Enterprise in Practice (Field Notes)
- 1. DataStax Enterprise Field Notes: DSE in Practice Caroline George caroline.george@datastax.com
- 2. Company Confidential© 2016 DataStax, All Rights Reserved. What are all the other banks doing? What do you mean SSDs? What’s the difference with [MongoDB]? How many nodes will I need? How do I load data from [Oracle]? What are all the other banks doing? DS E
- 3. Company Confidential© 2016 DataStax, All Rights Reserved. About Me
- 4. Company Confidential© 2016 DataStax, All Rights Reserved. Global Merchants Transactions Use Case • 100TB of credit card charges • Move to distributed REST Web Services • Migrate off mainframe / DB2 • Reduce cost Why DSE • 100% up time • Scalability as business grows
- 5. Company Confidential© 2016 DataStax, All Rights Reserved. Challenge #1: Hardware 1. Private Cloud • 4-core CPU • 16 GB RAM • 600 GB Attached SAN 2. AWS • i2.2xlarge • 8-core CPU • 64 GB RAM • 2 x 800 GB SSDs 3. Final Hardware • Cisco C240 • 12-core CPU • 64 GB RAM • 8 x 400 GB SSDs
- 6. Company Confidential© 2016 DataStax, All Rights Reserved. Challenge #2: Data Model Queries • Credit card charges for a merchant for a date range • Credit card charges details equal to a specific amount for a merchant • Credit card charges by charge reference number Approach 2-step Look up
- 7. Company Confidential© 2016 DataStax, All Rights Reserved. Global Payments Use Case • Replace dependency on Exadata • Ingestion of payment data at a rate > 5,000 tps • Support schema evolution Why DSE • No down time • Projected growth: 100TB • Scalability as data retention increases
- 8. Company Confidential© 2016 DataStax, All Rights Reserved. Challenge #1: Data Model Queries • Payments by status [for a date range] • Payments by credit value date • Payments by payment ID • Payments by currency • Payments by region • Completed payments • Cancelled payments • Processed payments … Approach 1. DSE Search 2. Partition by time bucket wide partitions cfstats
- 9. Company Confidential© 2016 DataStax, All Rights Reserved. Global Billing • Move off Oracle Coherence Why DSE • Scale to bring B2B customers onboard Use Case • Aggregate usage data in 15 min intervals • 3 million records every 15 mins • Send alerts if customer exceeds thresholds
- 10. Company Confidential© 2016 DataStax, All Rights Reserved. Workflow
- 11. Company Confidential© 2016 DataStax, All Rights Reserved. Challenge #1: New Technologies Technologies Approach 1. http://academy.datastax.com 2. Onsite training 3. Consulting 4. Github
- 12. Company Confidential© 2016 DataStax, All Rights Reserved. Global Customer 360 Use Case • Scale as they bring more data points onboard • Centralized repo of customer activity Why DSE • Fast ingestion rate • Better customer interaction • 22 million clients / 2,400 branches / 4,700 ATMs • Personalization
- 13. Company Confidential© 2016 DataStax, All Rights Reserved. Architecture ● New account on-boarding ● Recommendation/personalization ● Omnichannel ● Cross-selling ● Fraud detection ● Single source of truth ● Cloud applications ● Web services ● Microservices ● Mobile ● Lambda architecture Analytics Operational Search
- 14. Company Confidential© 2016 DataStax, All Rights Reserved. Challenge #1: Internal Politics Politics • Infrastructure • Onboarding new technology • Coordinating across business lines • Budget Approach • Involve Infrastructure/Security/DBAs early • COE
- 15. Company Confidential© 2016 DataStax, All Rights Reserved. Who else? 15
Editor's Notes
- Payments & Transactions ING SPOKE PUBLICLY AT EUROPE SUMMIT 2014 – CAN MENTION BY NAME - global transaction payment: each country has own processing format > need to centralize & unify format - biggest challenge: always be available + scale effectively - 25k TPS real time + batch - < 25 ms response time - migrate from mainframe - no downtime possible - using Solr Fraud Prevention & Detection BofA - spoke at Summit but didn't get to hear it so not sure if they spoke about this use case specifically. Detect if the same debit card is being used at 2 separate ATMs at the same time or unusual spending activity, ie Alex just dropped $5k at Victoria Secrets Feedzai - Developed a fraud detection application running on DSE using Spark & Mllib – machine learning for more accurate detection, DSE for scalability since customers will have different amount of data Capital One - spoke at Summit 2015 but about their reporting platform so can mention by name but not specifically for this fraud use case for account opening (checking/saving) > user has to wait a few days before approval > can lose momentum/users prior to Feedzai > crude decision matrix > turned down more people than should have > didn’t catch enough fraudster Machine learning: selects 2/3k variables that are most valuable > from internal data + from customer’s input + customer saying w/o saying i.e. discrepancies in applications/pattern of usage (long wait during application, # of application from device) ~ 10mio banking customers Summit reporting platform: https://vimeopro.com/user35188327/cassandra-summit-2015/video/143823838 Using Feedzai b/c: model more precisely > can integrate machine learning > more accurately detect fraud vs not-fraud fast > 5 mili seconds to make decisions > can process more complex model in real time Real-time Market Risk / Regulatory Compliance & Reporting CS SPOKE PUBLICLY AT EUROPE SUMMIT 2014 – CAN MENTION BY NAME Talking points- after 2009 crisis - centralized risk mgt system risk system -> Hippo trading UI -> Hippo Results store -> Hippo 2 challenges 1. HA - new system = only 1 risk system -> impacts all bizs if down. hedge risk based on this system 2. Scale - ability to bring new groups on Credit Suisse Group, a Switzerland-based multinational financial services holding company, handles millions of global equity derivatives trading transactions per day. In order to support this massive volume of activity, Credit Suisse looked to establish a technology infrastructure that could address availability and scalability challenges inherent with their Oracle and MongoDB environment. Further amplifying these challenges was the need to meet extreme government and regulatory requirements which required tight controls over all data and trading activities. Credit Suisse realized that they needed a new database platform that could quickly scale to meet their needs, ensure 100% uptime and provide the necessary monitoring capabilities to meet regulatory compliance standards. So they turned to DataStax. Since migrating to DataStax, Credit Suisse has seen their operational costs reduced by 4X, while being able to easily handle extreme data volumes to the tune of over 56 million reads and 24 million writes per day. All without worry about performance or availability issues. Customer 360° & Portfolio Analysis Amex – DO NOT MENTION BY NAME migrate from mainframe -> cost savings, more flexible 30TB of data / 25m trans/day / 10% growth per year GMDL is the data platform for all POA merchant data supporting a wide spectrum of international and domestic applications. As GMDL has been maturing as a platform over the course of the last two years, it has become obvious that as the merchant side of the business continues to grow and Amex continues to add more online and interactive functionality, the cost of maintaining and delivering the charge data to the merchant customers continues to grow and expand over time. This growth curve is not sustainable long term, and must be curtailed. Addtionally, the volume of data that is returned for many of the company’s largest and most profitable merchant customers is so expansive that many of the current online systems time-out and fail to deliver on the request from the customer. Recommendations & Personalization Better service customers by offering products and services that match customer activity profile Centralized source of customer activity: 1) phone support 2) web 3) mobile) 4) in person, ie at branch Ability for branch manager to know when customer walks into branch that they already spoke with phone support or had issues with website to provide better service Global Document Search JPMC – DO NOT MENTION BY NAME Very silo organization, each group stores documents in their own format -> difficult to find information Store document information in Cassandra & use Solr for searching -> more efficient, centralized Geospatial Search Ie, branch location – via Solr geospatial functionality