As a leader in the financial industry, Capital One applications generate huge amounts of data that require fast and accurate handling, storage and analysis. We are transforming how we report operational data to our internal users so that they can make quick and precise business decisions to serve our customers. As part of this transformation, we are building a new Go-based data processing framework that will enable us to transfer data from multiple data stores (RDBMS, files, etc.) to a single NoSQL database - Cassandra. This new NoSQL store will act as a reporting database that will receive data on a near real-time basis and serve the data through scorecards and reports. We would like to share our experience in defining this fast data platform and the methodologies used to model financial data in Cassandra.

1. Using Cassandra In Building A Reporting Platform
Javed Roshan – Director, Data Services
Mukaram Aziz – Sr. Manager, Data Services

2. 1 Use Case
2 New Data Platform
3 Design Decisions
4 Solution Stack
5 Challenges
2© 2015. All Rights Reserved.

3. Use Case
• Fast Data requirements in an Operational Space
– Metrics and Reports for intra-day business decisions
– Process Monitoring
• Current Landscape
– Multiple data sources
– Traditional batched ETL
– Multiple data destinations
– Reporting Tools
• Opportunity Areas
– Make reports near real time
– Achieve 99.99% SLAs
– Time to market delivery
– Make enhancements inexpensive
3© 2015. All Rights Reserved.
Existing
Data Sources RDBMS, Files
ETL File Based
Data Distribution Files
Data Destination RDBMS
Reporting Tools Various

4. New Data Platform
• Platform
– Data Distribution: Kafka
– Data Processing: Go / Docker
– Data Store: Cassandra
– AWS
• Design Decisions
– Move data when available
– Transform when all data available
• Cassandra
– CAP: Emphasis on A & P with tunable C
– Wide row tables
– Linear scalability to handle large data sizes
– Out of the box multi-DC deployment
4© 2015. All Rights Reserved.
Existing New
Data Sources RDBMS, Files RDBMS, Files
ETL File based Go / Docker
Data Distribution Files Kafka
Data Destination RDBMS Cassandra
Reporting Tools Various Streamlined

5. Design Decisions
• Data Modeling
– Partition Key/Size
– “Read” Response time
– Handling Consistency
– Collection Columns: Sets & Maps
– Logical separation of raw & processed data
– All lookup data in a single table
• Indexes
– Primary, Inverted, Secondary, DSE Search Indexes
• DSE Search
– range-queries
– regular-expression
– non-equality
– faceted
5© 2015. All Rights Reserved.

6. Design Decisions
• Consistency
– W Consistency + R Consistency > Replication Factor
• Indexes
6© 2015. All Rights Reserved.
Data Access Options
Data /
Index
Storage
Response
Time
Maintain Cardinality Search Consistency
Primary Key Data High V Fast App High Limited Tunable
Duplicate Data (Primary Key) Data High V Fast App High Limited Tunable
Inverted Index Index Low Fast App High Limited Tunable
Secondary Index Index Low Medium System Low Limited Tunable
DSE Search Index Medium
Slow
(relative)
System Any Versatile One (R)

7. Benchmarking: Indexes
0.02 0.04
1.6
0.6
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Primary Index Inverted Index DSE Search Secondary Index
7© 2015. All Rights Reserved.
Timeinseconds
Index Type
• 22.6 million rows
• 6 node cluster

8. Performance
• 3 Replication Factor
• Write Heavy
– Increased concurrent writes to 64 (from 32)
– Decreased concurrent reads to 16 (from 32)
– Size-tiered compaction strategy
• Cassandra cluster with DSE Search enabled on all nodes
• Virtual nodes set to 16
• All caches disabled except filter cache
• EC2 Snitch on AWS – 3 AZs
• DSE Search soft auto-commit max time to 10s
8© 2015. All Rights Reserved.

9. Solution Stack
9© 2015. All Rights Reserved.
MESOSPHERE
MARATHON
DOCKER
GO
SERVICES
INGESTION
DOCKER
GO
SERVICES
PROCESSORKAFKA
API
SERVER
CONSUL

10. Solution Stack: Plug-In Framework
10© 2015. All Rights Reserved.
• Go Service: Plugins chained in a single process
• Packaged & deployed in a Docker Container
• Bootstrapped from a config
• 100% developed in-house
RUNNER
PLUGIN
IN
CHANNEL
OUT
CHANNEL
RUNNER
PLUGIN
IN
CHANNEL
OUT
CHANNEL
RUNNER
PLUGIN
IN
CHANNEL
OUT
CHANNEL
GO
SERVICE

11. Solution Stack
11© 2015. All Rights Reserved.
MESOSPHERE
MARATHON
DOCKER
GO
SERVICES
INGESTION
DOCKER
GO
SERVICES
PROCESSORKAFKA
API
SERVER
CONSUL

12. Solution Stack
12© 2015. All Rights Reserved.
MESOSPHERE
MARATHON
DOCKER
GO
SERVICES
INGESTION
DOCKER
GO
SERVICES
PROCESSORKAFKA
API
SERVER
CONSUL

13. Solution Stack
13© 2015. All Rights Reserved.
MESOSPHERE
MARATHON
DOCKER
GO
SERVICES
INGESTION
DOCKER
GO
SERVICES
PROCESSORKAFKA
API
SERVER
CONSUL

14. Solution Stack
14© 2015. All Rights Reserved.
MESOSPHERE
MARATHON
DOCKER
GO
SERVICES
INGESTION
DOCKER
GO
SERVICES
PROCESSORKAFKA
API
SERVER
CONSUL

15. Solution Stack
15© 2015. All Rights Reserved.
MESOSPHERE
MARATHON
DOCKER
GO
SERVICES
INGESTION
DOCKER
GO
SERVICES
PROCESSORKAFKA
API
SERVER
CONSUL

16. 16© 2015. All Rights Reserved.
• Cassandra
• Data Storage
• Go-Based Plugin Framework
• Go services for data Ingestion & Processing
• Docker
• Packaging and deployment
• Mesosphere
• Single view of infrastructure
• Marathon
• Launch containers
• Kafka
• Data transfer and distribution
• Consul
• Service discovery and configuration management
• Jenkins
• Continuous Integration
Solution Stack

17. Benchmarking: Data Processing
17© 2015. All Rights Reserved.
• Test for a functional group
• Cassandra: 6 node cluster
• Kafka: 6 node cluster
• Go Services: 3
• Primary Data Source: Oracle
• Time: 360 minutes
• Data Size: 1 year
Description Measure
Total rows processed 450 million
De-normalized rows 11.8 million
Rate of processing (Go Services) ~300k tps
Rate of processing (Platform) ~21k tps
% time waiting on data ingestion 75%

18. Challenges
• Not all query patterns are known in advance
• Index rebuilds are costly
• Business adjusting to near real-time data
• Operational support adjustments
• Backup/Restore
• Finding Talent – We are hiring!
18© 2015. All Rights Reserved.

19. Thank you