You Snooze You Lose or How to Win in Ad Tech?

727 views
676 views

Published on

Presentation from Adtech Hacked
Aerospike's highly reliable and scalable database, using NoSQL and In-memory technology, presentation slides given at Stack Exchange on April 10th with NSOne and advertising technology luminaries.
AdTech Gets Hacked in Lower Manhattan
Stack Exchange, 110 William St 28th Floor,
New York, NY 10038

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
727
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

You Snooze You Lose or How to Win in Ad Tech?

  1. 1. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 1 Aerospike aer . o . spike [air-oh- spahyk] noun, 1. tip of a rocket that enhances speed and stability YOU SNOOZE YOU LOSE OR HOW TO WIN IN AD TECH? THE ONLY FLASH-OPTIMIZED DATABASE BRIAN BULKOWSKI FOUNDER, CTO, PRODUCT STACK EXCHANGE MEETUP APRIL 10, 2014
  2. 2. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 2 Aerospike: the gold standard for high throughput, low latency, high reliability transactions Performance • Over ten trillion transactions per month • 99% of transactions faster than 2 ms • 150K TPS per server Scalability • Billions of Internet users • Clustered Software • Automatic Data Rebalancing Reliability • 50 customers; zero service down- time • Immediate Consistency • Rapid Failover; Data Center Replication Price/Performance • Makes impossible projects affordable • Flash-optimized • 1/10 the servers required
  3. 3. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 3 Aerospike Proven in Production ■  AppNexus - #2 RTB after Google ■  45 Billion auctions per day ■  2M QPS ■  3 12 server clusters ■  4.8T Flash per server ■  120K read TPS, 60K write TPS ■  Chango – #2 Search after Google ■  Sees more Searches than Yahoo! + bing ■  Data on 300 Million users ■  TradeDesk – first Ad Exchange ■  Facebook Exchange partner ■  FBX serves 25% of Ads on the Internet ■  Snapdeal ■  2 servers replace 10 mongo servers ■  10GB data ■  “changed our company” “Aerospike has operated without interruptions and easily scaled to meet our performance demands.” – Mike Nolet, CTO, AppNexus © 2013 Aerospike. All rights reserved. Pg. 3
  4. 4. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 4 MILLIONS OF CONSUMERS BILLIONS OF DEVICES AEROSPIKE CLUSTER APP SERVERS RDBMS DATA WAREHOUSE SEGMENTS WRITE REAL-TIME CONTEXT READ RECENT CONTENT PROFILE STORE Cookies, email, deviceID, IP address, location, segments, clicks, likes, tweets, search terms... REAL-TIME ANALYTICS Best sellers, top scores, trending tweets BATCH ANALYTICS Discover patterns, segment data: location patterns, audience affinity TYPICAL REAL-TIME DATABASE DEPLOYMENT TRANSACTIONS WRITE CONTEXT
  5. 5. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 5 KEY CHALLENGES 1.  Handle extremely high rates of read/write transactions over persistent data 2.  Avoid hot spots to maintain tight latency SLAs 3.  Provide immediate consistency with replication 4.  Ensure long running tasks do not slow down transactions 5.  Scale linearly as data sizes and workloads increase 6.  Add capacity with no service interruption
  6. 6. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 6 SYSTEM ARCHITECTURE FOR 100% UPTIME
  7. 7. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 7 SHARED-NOTHING SYSTEM:100% DATA AVAILABILITY ■  Every node in a cluster is identical, handles both transactions and long running tasks ■  Data is replicated synchronously with immediate consistency within the cluster ■  Data is replicated asynchronously across data centers OHIO Data Center © 2013 Aerospike. All rights reserved Pg. 7
  8. 8. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 8 ROBUST DHT TO ELIMINATE HOT SPOTS How Data Is Distributed (Replication Factor 2) ■  Every key is hashed into a 20 byte (fixed length) string using the RIPEMD160 hash function ■  This hash + additional data (fixed 64 bytes) are stored in RAM in the index ■  Some bits from this hash value are used to compute the partition id ■  There are 4096 partitions ■  Partition id maps to node id based on cluster membership cookie-abcdefg-12345678 182023kh15hh3kahdjsh Partition ID Master node Replica node … 1 4 1820 2 3 1821 3 2 4096 4 1 © 2013 Aerospike. All rights reserved Pg. 8
  9. 9. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 9 REAL-TIME PRIORITIZATION TO MEET SLA 1.  Write sent to row master 2.  Latch against simultaneous writes 3.  Apply write to master memory and replica memory synchronously 4.  Queue operations to disk 5.  Signal completed transaction (optional storage commit wait) 6.  Master applies conflict resolution policy (rollback/ rollforward) master replica 1.  Cluster discovers new node via gossip protocol 2.  Paxos vote determines new data organization 3.  Partition migrations scheduled 4.  When a partition migration starts, write journal starts on destination 5.  Partition moves atomically 6.  Journal is applied and source data deleted transactions continue Writing with Immediate Consistency Adding a Node
  10. 10. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 10 INTELLIGENT CLIENT TO MAKE APPS SIMPLER ■  Implements Aerospike API ■  Optimistic row locking ■  Optimized binary protocol ■  Cluster tracking ■  Learns about cluster changes, partition map ■  Gossip protocol ■  Transaction semantics ■  Global transaction ID ■  Retransmit and timeout ■  Linear scale ■  No extra hop ■  No load balancers © 2013 Aerospike. All rights reserved Pg. 10
  11. 11. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 11 OTHER DATABASE OS FILE SYSTEM PAGE CACHE BLOCK INTERFACE SSD HDD BLOCK INTERFACE SSD SSD OPEN NVM SSD OTHER DATABASE AEROSPIKE FLASH OPTIMIZED IN-MEMORY DATABASE Ask me and I’ll tell you the answer.Ask me. I’ll look up the answer and then tell it to you. AEROSPIKE HYBRID MEMORY SYSTEM™ •  Direct device access •  Large Block Writes •  Indexes in DRAM •  Highly Parallelized •  Log-structured FS “copy-on-write” •  Fast restart with shared memory FLASH OPTIMIZED HIGH PERFORMANCE
  12. 12. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 12 Storage type DRAM & NoSQL SSD & DRAM Storage per server 180 GB (196 GB Server) 2.4 GB (4 x 700 GB) TPS per server 500,000 500,000 Cost per server $8,000 $11,000 Server costs $1,488,000 $154,000 Power/server 0.9 kW 1.1 kW Power (2 years) $0.12 per kWh ave. US $352,000 $32,400 Maintenance (2 years) $3,600 per server $670,000 $50,400 Total $2,510,000 $236,800 THE BOTTOM LINE Actual customer analysis. Customer requires 500K TPS, 10 TB of storage, with 2x replication factor. 186 SERVERS REQUIRED 14 SERVERS REQUIRED OTHER DATABASES ONLY
  13. 13. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 13 Only up in 2013, 2014 Everyone wants that “Facebook architecture” Facebook and Apple bought at least $200+M in FusionIO cards in 2012 + = $200+M © 2013 Aerospike. All rights reserved Pg. 13
  14. 14. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 14 © 2012 Aerospike. All rights reserved. Pg. 14 Measure your drives! Aerospike Certification Tool (ACT) http://github.com/aerospike/act Transactional database workload Reads: 1.5KB (can’t batch / cache reads, random) Writes: 128K blocks (log based layout) (plus defragmentation) Turn up the load until latency is over required SLA
  15. 15. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 15 ➤  Super Storm Sandy 2012 §  NYC down for 17 hours §  Back up and synched in 1 hour via Aerospike Cross-Data Center Replication (XDR) Replication that Works “Aerospike allows us to handle business continuity and reliability across 4 data centers seamlessly. And we can now expand our deployment to new data centers in less than a week.” - Elad Efraim, CTO
  16. 16. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 16 HOT ANALYTICS BY ROW
  17. 17. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 17 ➤  Namespaces (policy containers) §  Determine storage - DRAM or Flash §  Determine replication factor §  Contain records and sets ➤  Sets (tables) of records §  Arbitrary grouping ➤  Records (rows) §  Max 128k, contain key and bins §  Bin with same name can contain values of different types u  String, integer, bytes (raw, blob, etc) u  list ( an ordered collection of values ) u  map ( a collection of keys and values ) §  Bins can be added anytime NOSQL EXTENSIBILITY
  18. 18. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 18 DISTRIBUTED QUERIES 1.  “Scatter” requests to all nodes 2.  Indexes in DRAM for fast map of secondary à primary keys 3.  Indexes co-located with data to guarantee ACID, manage migrations 4.  Records read in parallel from all SSDs using lock free concurrency control 5.  Aggregate results on each node 6.  “Gather” results from all nodes on client STREAM AGGREGATIONS 1.  Push Code/ Security Policies/ Rules to Data with UDFs 2.  Pipe Query results through UDFs to Filter, Transform, Aggregate.. Map, Reduce REAL-TIME ANALYTICS on OPERATIONAL DATA (No ETL) ➤  In Database, within the same Cluster ➤  On the same Data, on XDR Replicated Clusters Real-time Analytics on Operational Data
  19. 19. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 19 LESSONS LEARNED
  20. 20. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 20 NATIVE FLASH à PERFORMANCE ■  Low Latency at High Throughput 0 2.5 5 7.5 10 0 50,000 100,000 150,000 200,000 AverageLatency,ms Throughput, ops/sec Balanced Workload Read Latency (Full view) Aerospike Cassandra MongoDB 0 4 8 12 16 0 50,000 100,000 150,000 200,000 AverageLatency,ms Throughput, ops/sec Balanced Workload Update Latency (Full view) Aerospike Cassandra MongoDB 0 1 2 3 4 0 75,000 150,000 225,000 300,000 AverageLatency,ms Throughput, ops/sec Read-Heavy Workload Read Latency (Full view) Aerospike Cassandra MongoDB 0 6 12 18 24 0 75,000 150,000 225,000 300,000 AverageLatency,ms Throughput, ops/sec Read-Heavy Workload Update Latency (Full view) Aerospike Cassandra MongoDB © 2013 Aerospike. All rights reserved Pg. 20
  21. 21. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 21 LESSONS 1.  Keep architecture simple ■  No hot spots (e.g., robust DHT) ■  Scales up easily (e.g., easy to size) ■  Avoids points of failure (e.g., single node type) 2.  Avoid manual operation – automate, automate! ■  Self-managed cluster responds to node failures ■  Data rebalancing requires no intervention ■  Real-time prioritization allows unattended system operation 3.  Keep system asynchronous ■  Shared nothing – nodes are autonomous ■  Async writes across data centers ■  Independent tuning parameters for different classes of tasks © 2013 Aerospike. All rights reserved Pg. 21
  22. 22. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 22 LESSONS (cont’d) 4.  Monitor the Health of the System Extensively ■  Growth in load sneaks up on you over weeks ■  Early detection means better service ■  Most failures can be predicted (e.g., capacity, load, …) 5.  Size clusters properly ■  Have enough capacity ALWAYS! ■  Upgrade SSDs every couple years ■  Reduce cluster sizes to make operations simple 6.  Have geographically distributed data centers ■  Size the distributed data centers properly ■  Use active-active configurations if possible ■  Size bandwidth requirements accurately © 2013 Aerospike. All rights reserved Pg. 22
  23. 23. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 23 LESSONS (CONT’D) 7.  Have plan for unforeseen situations ■  Devise scenarios and practice during normal work time ■  Ensure you can do rolling upgrades during high load time ■  Make sure that your nodes can restart fast (< 1 minute) 8.  Constantly test and monitor app end-to-end ■  Application level metrics are more important than DB metrics ■  Most issues in a service are due to a combination of application, network, database, storage, etc. 9.  Separate online and offline workloads ■  Reserve real-time edge database for transactions and hot analytics queries (where newest data is important) ■  Avoid ad-hoc queries on on-line system ■  Perform deep analysis in offline system (Hadoop) 10.  Use the Right Data Management System for the job ■  Fast NoSQL DB for real-time transactions and hot analytics on rapidly changing data ■  Hadoop or other comparable systems for exhaustive analytics on mostly read-only data © 2013 Aerospike. All rights reserved Pg. 23
  24. 24. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 24 AEROSPIKE REAL-TIME BIG DATA PLATFORM Rapid Development Complete Customizability ➤  Support for popular languages and tools §  ASQL and Aerospike Client in Java, C#, Ruby, Python.. ➤  Complex data types §  Nested documents (map, list, string, integer) §  Large (Stack, Set, List) Objects ➤  Queries §  Single record §  Batch multi-record lookups §  Equality and range §  Aggregations and MapReduce ➤  User Defined Functions (UDFs) §  In-DB processing ➤  Aggregation Framework §  UDF Pipeline §  MapReduce ++ ➤  Time Series Queries §  Just 2 IOPs for most r/w independent of object size
  25. 25. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 25 HOW TO GET AEROSPIKE? Free Community Edition Enterprise Edition ➤  For developers looking for speed and stability and transparently scale as they grow ➤  No transaction limits ➤  No time limit ➤  No production limit ➤  Data per cluster limit ➤  Community support ➤  For mission critical apps needing to scale right from the start §  Unlimited number of nodes, clusters, data centers §  Cross data center replication §  Premium 24x7 support §  Priced by TBs of unique data (not replicas) © 2013 Aerospike. All rights reserved Pg. 25
  26. 26. © 2013 Aerospike, Inc. All rights reserved. Confidential. | <Title of Presentation> | 26 QUESTIONS? brian@aerospike.com www.aerospike.com © 2013 Aerospike. All rights reserved Pg. 26

×