Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The role of NoSQL in the Next Generation of Financial Informatics

659 views

Published on

The slide deck for a webinar that was hosted by Bob's Guide and had guest speaker Steve Yatko from OKTAY Technology

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The role of NoSQL in the Next Generation of Financial Informatics

  1. 1. The Role of NoSQL In Next Generation Financial Informatics December 10, 2015
  2. 2. 2 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. ■ Market Overview ■ The role of NoSQL in Financial Services ■ Customer Use Cases ■ Fraud ■ Order Management ■ How you can achieve Speed at Scale Agenda
  3. 3. Steve Yatko CEO, Oktay Technology Financial Services Market Overview
  4. 4. 4 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Hours - Days Analytics Store Operational Store (Landing Zone – Fast Ingest) MarkLogic ElasticSearch Deep Archive Apache Storm Apache Spark Apache Ignite Project Heron Ingest/ Gateway Tibco EMS JMS 60 East 29 West TCP FTP IBM MQ Data Warehouse (Legacy) Teradata DB2 WH Actian(ParAccel) Pivotal Astor Data/TD Neteeza/DB2 WH Exadata Mark Logic Veritca Data Lake (100+ Petabyte) Tiered External Storage Flash Arrays to Warm/Cold/Frozen Hadoop MapReduce Redis IMDGrid Hazelcast Aerospike Graph Db Large Footprint Front Office Real-Time Processing On The Wire “Fast Data’ Hours Days-Years No SQL HBase MongoDB Cassandra Aerospike Time Series ETL In Memory Cache Data Operating System – YARN, MESOS Deeper Analysis World Class Big Data Ecosystem: “Service Oriented Information” HDFS Interface Small Footprint Surveillance Trading Analytics Metrics, Monitoring and Alerts Search ComplianceRT Discovery & Visualization Oktay Technology LLC Confidential RAM-first Storage Flash-enabled Storage SQL-Based MemSQL Volt DB Kafka In Memory Fil;esystem Tachyon IgniteFS
  5. 5. Brian Bulkowski Founder and CTO, Aerospike The Role of NoSQl in Next Generation Financial Informatics
  6. 6. 6 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. 2010 Aerospike 2.0 deploys with first at-scale customers 2011 Funding, company launch 2013 Aerospike 3.0 adds indexes, analytics integration 2014 Open Source 2015 Deployments in Financial Services, Telecom, etc Aerospike History
  7. 7. 7 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Modern Scale Out Architecture Research Warehouse Long-term cold storage HDFS BASED App Servers Fast, stateless Load Balancer Simple, stateless High Performance NoSQL Operational Key Value Session, authentication, account status, cookies, deviceID, IP address, location, segments, trades, debits, billing, prices... Real-Time Decisions Best sellers, top scores, trending tweets
  8. 8. 8 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Powering Pioneers Across the Internet Mobile Advertising Omni-Channel Marketing Search, Video, Social, Gaming Web Advertising
  9. 9. 9 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. 1 Million Writes/Sec on Google Compute New results: 20 nodes, and 4M reads per second • Aerospike hits 1M writes/sec with 6x fewer servers than Cassandra • Delivers consistent low latency with no jitter for both read and write workloads
  10. 10. 10 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Shared-Nothing System, High Availability • Every node in a cluster is identical, handles both transactions and long running tasks • Clients give rich semantics in multiple languages and connect over the network • Data is replicated synchronously with within the local cluster • Data is replicated asynchronously across data centers • Primary key hash RIPE MD160 ( 20 byte ) for extreme collision resistance ( DHT ), Red black tree for records within hash bucket • Scatter-gather B+ secondary index
  11. 11. 11 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. The NoSQL Pattern Applied to Inline Fraud Intervention
  12. 12. 12 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Fraud Prevention 4% of online transactions are fraudulent False positives cost millions in lost sales Global criminal agents New opportunities: Data sources Algorithms Speed at Scale Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
  13. 13. 13 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Operational Scale Problem LEGACY DATABASE (Mainframe) XDR Decision Engine DATA WAREHOUSE/ DATA LAKE LEGACY RDBMS HDFS BASED BUSINESS TRANSACTIONS Request for Payment ( Mobile Queries ) ( Recommendation ) ( And More ) High Performance NoSQL 500 Business Trans per sec 5000 Calculations per sec X = “REAL-TIME BIG DATA” “DECISIONING” 2.5 M Database Transactions per sec
  14. 14. 14 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Advanced Fraud Scoring Anything can be a “feature” POS type, item sold, price of item, time of day for given items Browser type, screen resolution, item + browser But you’ll need recent data And cutting edge libraries ( your private algorithms )
  15. 15. 15 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Example IP Addresses 2B to 4B addresses Unknown number of networks Different risk models for different network ranges Recent behavior required
  16. 16. 16 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Example – IP based scoring ID START_IP END_IP Score 1 192.168.4.1 192.168.4.26 20 2 192.168.4.7 192.168.17.0 2 SELECT id, score WHERE START_IP >= this_ip AND END_IP <= this_ip Not so bad! Too slow, and too simple
  17. 17. 17 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Using SQL ID IP TIMESTAMP BEHAVIOR 1 192.168.4.1 2015-06-02 12:00 transaction 2 192.168.4.7 2015-06-02 12:01 tweet Add the most recent 5 minutes of traffic for each IP address and lookup by previous network INSERT ip, time, behavior FOR ( network ) SELECT behavior WHERE ip <= network.start_id AND ip >= network.end_id AND timestamp > now - 600 You can do it in SQL! Too slow, call the DBAs.
  18. 18. 18 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. With a NoSQL KVS key Network list 192.168.4.0 [ net1, net2, net3 ] 192.168.5.0 null ID START_IP END_IP Score net1 192.168.4.1 192.168.4.26 20 net2 192.168.4.7 192.168.17.0 2 Key on Class C address List of networks Batch lookup networks Filter on client That’s messy. Two network round trips, it’ll go fast.
  19. 19. 19 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. With a NoSQL KVS – add behavior Already have filtered list of networks For each network, insert into capped or time limited collection key Network list 192.168.4.0 [ net1, net2, net3 ] 192.168.5.0 null ID START_IP END_IP Score behavior net1 192.168.4.1 192.168.4.26 20 { collection } net2 192.168.4.7 192.168.17.0 2 { collection } That’s really messy! It’s fast and easy to explain.
  20. 20. 20 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Use Cutting Edge Libraries in Your Favorite App Environment If it’s in SQL, it’s too late
  21. 21. 21 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Modern Scale Out Architecture App Servers Fast, stateless Load Balancer Simple, stateless High Performance NoSQL Operational Key Value Session, authentication, account status, cookies, deviceID, IP address, location, segments, trades, debits, billing, prices... Real-Time Decisions Best sellers, top scores, trending tweets Scale and Compute Data Enabled by Fast Networks Least Vendor Lock-in
  22. 22. 22 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. The NoSQL Pattern Applied to Order Management
  23. 23. 23 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Financial Services – Order Management Systems Challenge • Must update stock prices, show balances on 300 positions, process 250M transactions, 2M updates/day • Scale-out requirement • Multi-asset aggregation for combined: Trading Risk Surveillance Customer • Real-time data view and computation Speed at Scale
  24. 24. 24 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Financial Services – Order Management Systems Individual data silos • No global, integrated data view • No simple post-order visibility • Often built on non-scalable technologies • Little shared operational infrastructure DB1 DB2 DB3 Order Management System Order Management System Order Management System FIXED INCOMEEQUITIES DERIVATIVES
  25. 25. 25 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Financial Services – Order Management Systems Shared Data OMS •True multi-asset aggregation •Supports real-time risk models •Immediate customer visibility of order state •Natural NoSQL data model including parent-child and fills •Microsecond performance with persistence •Scale-out architecture Order Management System Order Management System Order Management System FIXED INCOMEEQUITIES DERIVATIVES
  26. 26. 26 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Financial Services – Order Management Systems Hybrid order management •Existing order management code unchanged •Order states placed onto message bus •Recorded in unified NoSQL storage •Achieve unified multi-asset aggregated view •Storm and Spark integration •Beyond RAM-based processing solutions DB1 DB2 DB3 Order Management System Order Management System Order Management System FIXED INCOMEEQUITIES DERIVATIVES MESSAGE BUS (KAFKA, etc)
  27. 27. 27 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. How Much Data Can I Use?
  28. 28. 28 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Multiple Engagements 10B to 50B objects 300 T before HA replication 2 M TPS 50 milliseconds per 5,000 queries Local and remote DR
  29. 29. 29 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. 1 Million TPS on Flash per server Options for storage on a database before Aerospike:  RAM, which was fast, but allowed very limited storage  Disk, which allowed for a lot of storage, but was limited in speed Intel achieved 1M TPS using 4 Intel P3700 SDs with 1.6 TB capacity on a single Aerospike server. The cost per GB is a fraction of the cost of RAM, while still having very high performance.
  30. 30. 30 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Aerospike Has the Experience Flash optimized from the beginning 5 9’s reliability ( longest running cluster: 5 minutes of outage in 4.5 years ) Multi-million TPS clusters in production today Enterprise Support of your POC & NoSQL design Used in financial services, telecom, gaming ( and adtech of course )
  31. 31. 31 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. Questions? @aerospikedb brian@aerospike.com @bbulkow
  32. 32. 32 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved. High Performance NoSQL Database Powering New Opportunities at Scale @aerospikedb NEXT STEPS: See how much you can save with Aerospike: http://www.aerospike.com/tco-calculator/ Ready to get started? http://www.aerospike.com/quick-start/ If you have any questions or want to further explore if Aerospike is right for you, contact us: info@aerospike.com

×