Your SlideShare is downloading. ×
Flash Economics and Lessons learned from operating low latency platforms at high throughput with Aerospike NoSQL.
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Flash Economics and Lessons learned from operating low latency platforms at high throughput with Aerospike NoSQL.

1,269

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,269
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
30
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 1 Aerospike aer . o . spike [air-oh- spahyk] noun, 1. tip of a rocket that enhances speed and stability FLASH ECONOMICS AND LESSONS LEARNED FROM OPERATING LOW LATENCY AND HIGH TPS PLATFORMS IN-MEMORY NOSQL DR. V. SRINIVASAN FOUNDER, VP ENGINEERING & OPERATIONS GITPRO APRIL 12, 2014
  • 2. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 2 REQUIREMENTS FOR INTERNET ENTERPRISES 1. Know who the Interaction is with ■ Monitor 200+ Million US Consumers, 5+ Billion mobile devices and sensors 2. Determine intent based on current context ■ Page views, search terms, game state, last purchase, friends list, ads served, location 3. Respond now, use big data for more accurate decisions ■ Display the most relevant Ad ■ Recommend the best product ■ Deliver the richest gaming experience ■ Eliminate fraud… 4. Service can NEVER go down!
  • 3. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 3 INTERNET ENTERPRISES RETAIL E-COMMERCE MOBILE OMNI CHANNEL GAMING WEB VIDEO SOCIAL SEARCH EMAIL
  • 4. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 4 Response time: Hours, Weeks TB to PB Read Intensive TRANSACTIONS (OLTP) Response time: Seconds Gigabytes of data Balanced Reads/Writes ANALYTICS (OLAP) STRUCTURED DATA Response time: Seconds Terabytes of data Read Intensive BIG DATA ANALYTICS Real-time Transactions Response time: < 10 ms 1-20 TB Balanced Reads/Writes 24x7x365 Availability UNSTRUCTURED DATA REAL-TIME BIG DATA DATABASE LANDSCAPE
  • 5. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 5 Aerospike recognized as the only company in the Visionaries Quadrant in Gartner's Magic Quadrant for Operational Database Management Systems Gartner, Magic Quadrant for Operational Database Management Systems Donald Fienberg et al.October 23, 2013 This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available at www.aerospike.com . Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
  • 6. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 6 MILLIONS OF CONSUMERS BILLIONS OF DEVICES AEROSPIKE CLUSTER APP SERVERS RDBMS DATA WAREHOUSE SEGMENTS WRITE REAL-TIME CONTEXT READ RECENT CONTENT PROFILE STORE Cookies, email, deviceID, IP address, location, segments, clicks, likes, tweets, search terms... REAL-TIME ANALYTICS Best sellers, top scores, trending tweets BATCH ANALYTICS Discover patterns, segment data: location patterns, audience affinity TYPICAL REAL-TIME DATABASE DEPLOYMENT TRANSACTIONS WRITE CONTEXT
  • 7. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 7 KEY CHALLENGES 1. Handle extremely high rates of read/write transactions over persistent data 2. Avoid hot spots to maintain tight latency SLAs 3. Provide immediate consistency with replication 4. Ensure long running tasks do not slow down transactions 5. Scale linearly as data sizes and workloads increase 6. Add capacity with no service interruption
  • 8. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 8 SYSTEM ARCHITECTURE FOR 100% UPTIME
  • 9. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 9 SHARED-NOTHING SYSTEM:100% DATA AVAILABILITY ■ Every node in a cluster is identical, handles both transactions and long running tasks ■ Data is replicated synchronously with immediate consistency within the cluster ■ Data is replicated asynchronously across data centers OHIO Data Center
  • 10. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 10 ROBUST DHT TO ELIMINATE HOT SPOTS How Data Is Distributed (Replication Factor 2) ■ Every key is hashed into a 20 byte (fixed length) string using the RIPEMD160 hash function ■ This hash + additional data (fixed 64 bytes) are stored in RAM in the index ■ Some bits from this hash value are used to compute the partition id ■ There are 4096 partitions ■ Partition id maps to node id based on cluster membership cookie-abcdefg-12345678 182023kh15hh3kahdjsh Partition ID Master node Replica node … 1 4 1820 2 3 1821 3 2 4096 4 1
  • 11. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 11 REAL-TIME PRIORITIZATION TO MEET SLA 1. Write sent to row master 2. Latch against simultaneous writes 3. Apply write to master memory and replica memory synchronously 4. Queue operations to disk 5. Signal completed transaction (optional storage commit wait) 6. Master applies conflict resolution policy (rollback/ rollforward) master replica 1. Cluster discovers new node via gossip protocol 2. Paxos vote determines new data organization 3. Partition migrations scheduled 4. When a partition migration starts, write journal starts on destination 5. Partition moves atomically 6. Journal is applied and source data transactions continue Writing with Immediate Consistency Adding a Node
  • 12. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 12 INTELLIGENT CLIENT TO MAKE APPS SIMPLER Shield Applications from the Complexity of the Cluster■ Implements Aerospike API ■ Optimistic row locking ■ Optimized binary protocol ■ Cluster tracking ■ Learns about cluster changes, partition map ■ Gossip protocol ■ Transaction semantics ■ Global transaction ID ■ Retransmit and timeout ■ Linear scale ■ No extra hop ■ No load balancers
  • 13. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 13 OTHER DATABASE OS FILE SYSTEM PAGE CACHE BLOCK INTERFACE SSD HDD BLOCK INTERFACE SSD SSD OPEN NVM SSD OTHER DATABASE AEROSPIKE FLASH OPTIMIZED IN-MEMORY DATABASE Ask me and I’ll tell you the answer.Ask me. I’ll look up the answer and then tell it to you. AEROSPIKE HYBRID MEMORY SYSTEM™ • Direct device access • Large Block Writes • Indexes in DRAM • Highly Parallelized • Log-structured FS “copy-on-write” • Fast restart with shared memory FLASH OPTIMIZED HIGH PERFORMANCE
  • 14. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 14 Storage type DRAM & NoSQL SSD & DRAM Storage per server 180 GB (196 GB Server) 2.4 GB (4 x 700 GB) TPS per server 500,000 500,000 Cost per server $8,000 $11,000 Server costs $1,488,000 $154,000 Power/server 0.9 kW 1.1 kW Power (2 years) $0.12 per kWh ave. US $352,000 $32,400 Maintenance (2 years) $3,600 per server $670,000 $50,400 Total $2,510,000 $236,800 FLASH PROVIDES DRAM-LIKE PERFORMANCE WITH MUCH LOWER COMPLEXITY & TCO Actual customer analysis. Customer requires 500K TPS, 10 TB of storage, with 2x replication factor. 186 SERVERS REQUIRED 14 SERVERS REQUIRED OTHER DATABASES ONLY
  • 15. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 15 HOT ANALYTICS BY ROW
  • 16. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 16 SECONDARY INDEXES IN MEMORY ■ Fast ■ Indexes in DRAM, Data on Flash ■ No hotspots, Index-Data balanced across the cluster ■ Parallel processing across nodes, cores & SSDs ■ Reliable ■ Index and Data co-located to manage data migrations and guarantee ACID ■ Lock-free MVCC Secondary Index Primary Index Record Values DRAM SSD Server Client
  • 17. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 17 LOW SELECTIVITY INDEX QUERIES 1. Query sent to ALL nodes in parallel “SCATTER” 2. Secondary Index keys in DRAM ■ Map to Primary keys in DRAM ■ Co-located with Record on SSD 3. Records read in parallel from ALL SSDs 4. Parallel read results aggregated on node 5. Results from ALL nodes aggregated client-side “GATHER” Secondary Keys Primary Keys Records R1, R2 DRAM SSD Server Client … Keys Keys R3, R4 R5, R4 V1 V2 V3 V4 V5 V6
  • 18. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 18 SQL & NoSQL ➤ Secondary index  Equality, Range, IN (,,,), Compound  e.g. WHERE group_id = 1234, WHERE last_activity > 1349293398, WHERE branch_id IN (5,6,7,8) ➤ Filters  SQL: Where clause with non-indexed “AND”s (e.g. “AND gender=„M‟ ”)  NOSQL: Map step ➤ Aggregation  SQL: GROUP BY, ORDER BY, LIMIT, OFFSET  NOSQL: Reduce step Secondary Key Primary Key Record Filter Map Aggregate DRAM SSD Aggregate Client Client Server Reduce Aggregate Query
  • 19. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 19 ROW BASED SCHEDULING ■ Due to caching and blocks, most system resource consumption is per row ( Flash is in-memory ) ■ Rows are fine grained ■ Scheduler is “local” only ■ Deadline scheduling ■ Per-query priority (per transaction timeout) Secondary Index Primary Index Record Values SSD Server Client Hot analytics Operational Priority Q
  • 20. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 20 LESSONS LEARNED
  • 21. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 21 Native Flash  Performance 0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 Balanced Read-Heavy Aerospike Cassandra MongoDB Couchbase 2.0* *We were forced to exclude Couchbase...since when run with either disk or replica durability on it was unable to complete the test.” – Thumbtack Technology 0 2.5 5 7.5 10 0 50,000 100,000 150,000 200,000 AverageLatency,ms Throughput, ops/sec Balanced Workload Read Latency Aerospike Cassandra MongoDB 0 4 8 12 16 0 50,000 100,000 150,000 200,000 AverageLatency,ms Throughput, ops/sec Balanced Workload Update Latency Aerospike Cassandra MongoDB HIGH THROUGHPUT LOW LATENCY Throughput,TPS
  • 22. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 22 High Availability Through Clustering & Replication 1 32 4 5 Phases 1) 100KTPS – 4 nodes 2) Clients at Max 3) 400KTPS – 4 nodes 4) 400KTPS – 3 nodes 5) 400KTPS – 4 nodes Aerospike Node Specs: CentOS 6.3 Intel i5-2400@ 3.1 GHz (Quad core) 16 GB RAM@1333 MHz
  • 23. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 23 LESSONS 1. Keep architecture simple ■ No hot spots (e.g., robust DHT) ■ Scales up easily (e.g., easy to size) ■ Avoids points of failure (e.g., single node type) 2. Avoid manual operation – automate, automate! ■ Self-managed cluster responds to node failures ■ Data rebalancing requires no intervention ■ Real-time prioritization allows unattended system operation 3. Keep system asynchronous ■ Shared nothing – nodes are autonomous ■ Async writes across data centers ■ Independent tuning parameters for different classes of tasks
  • 24. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 24 LESSONS (cont’d) 4. Monitor the Health of the System Extensively ■ Growth in load sneaks up on you over weeks ■ Early detection means better service ■ Most failures can be predicted (e.g., capacity, load, …) 5. Size clusters properly ■ Have enough capacity ALWAYS! ■ Upgrade SSDs every couple years ■ Reduce cluster sizes to make operations simple 6. Have geographically distributed data centers ■ Size the distributed data centers properly ■ Use active-active configurations if possible ■ Size bandwidth requirements accurately
  • 25. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 25 LESSONS (CONT’D) 7. Have plan for unforeseen situations ■ Devise scenarios and practice during normal work time ■ Ensure you can do rolling upgrades during high load time ■ Make sure that your nodes can restart fast (< 1 minute) 8. Constantly test and monitor app end-to-end ■ Application level metrics are more important than DB metrics ■ Most issues in a service are due to a combination of application, network, database, storage, etc. 9. Separate online and offline workloads ■ Reserve real-time edge database for transactions and hot analytics queries (where newest data is important) ■ Avoid ad-hoc queries on on-line system ■ Perform deep analysis in offline system (Hadoop) 10. Use the Right Data Management System for the job ■ Fast NoSQL DB for real-time transactions and hot analytics on rapidly changing data ■ Hadoop or other comparable systems for exhaustive analytics on mostly read-only data
  • 26. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 26 1. Scaling the Internet of Everything 2. Pushing the limits of modern hardware 3. No data loss (ACID) and No downtime MODERN REAL-TIME DATA PLATFORM APPSERVERAEROSPIKESERVER REAL-TIME BIG DATA APPLICATION AEROSPIKE SMART CLIENT™ • APIs (C, C#, Java, PHP, Python, Ruby, Erlang…) • Transactions, Cluster awareness EXTENSIBLE DATA MODEL • Str, Int, Lists, Maps • Lookups, Queries, Scans • Aerospike Alchemy Framework™ with User Defined Functions and Distributed Aggregations MONITORING & MANAGEMENT • Aerospike Monitoring Console™ • Command Line Tools • Plugins- Nagios, Graphite, Zabbix AEROSPIKE SMART CLUSTER™ AEROSPIKE HYBRID MEMORY SYSTEM™ PROXIMITY & REDUNDANCY Cross Data Center Replication™ (XDR) REAL-TIME ENGINE APP/WEB SERVER AEROSPIKE CLUSTER Written in „C‟, Patents pending
  • 27. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 27 SUPPORT FOR REAL-TIME BIG DATA APPS Rapid Development Complete Customizability ➤ Support for popular languages and tools  ASQL and Aerospike Client in Java, C#, Ruby, Python.. ➤ Complex data types  Nested documents (map, list, string, integer)  Large (Stack, Set, List) Objects ➤ Queries  Single record  Batch multi-record lookups  Equality and range  Aggregations and MapReduce ➤ User Defined Functions (UDFs)  In-DB processing ➤ Aggregation Framework  UDF Pipeline  MapReduce ++ ➤ Time Series Queries  Just 2 IOPs for most r/w independent of object size
  • 28. © 2014 Aerospike, Inc. All rights reserved. Confidential. | GITPRO – April 12, 2014 | 28 QUESTIONS? info@aerospike.com www.aerospike.com

×