FollowJoin Us!< meta description=“Aerospike is aRule CaptainWord {strings:$header = {D0 CF 11 E0 A1 B1 1A E1}$author = {00...
© 2012 Aerospike. All rights reserved. Confidential Pg. 2PR, Case Studies (Slideshare, videos), blogsFeature 2 Customers/M...
© 2012 Aerospike. All rights reserved. Confidential Pg. 3“oltp”“transactional”“operational”“front end”“olap”“batch”“wareho...
© 2012 Aerospike. All rights reserved. Confidential Pg. 410101101011010101010011010101101010101010101101011101010110101010...
© 2012 Aerospike. All rights reserved. Confidential Pg. 5AdvertisingRecent activityCurrent predictionsAdvertising: recent ...
Flash 10T example10x fewer servers, 13x better TCO© 2012 Aerospike. All rights reserved. Confidential Pg. 6Storage type SS...
© 2012 Aerospike. All rights reserved. Confidential Pg. 7200GRedis with DRAM:10 servers @ $4K = $40KAerospike with Flash:3...
© 2012 Aerospike. All rights reserved. Confidential Pg. 8( 55% of $440M revinue estimate, reported inquarterly FusionIO ea...
FollowJoin Us!< meta description=“Aerospike is aRule CaptainWord {strings:$header = {D0 CF 11 E0 A1 B1 1A E1}$author = {00...
© 2012 Aerospike. All rights reserved. Confidential Pg. 10MeasureAerospike Certification Tool (ACT)http://github.com/aeros...
© 2012 Aerospike. All rights reserved. Confidential Pg. 11Micron P320h – ACT results[root@144.bm-general.dev.nym2 act]#lat...
© 2012 Aerospike. All rights reserved. Confidential Pg. 12Test data – the next generation6K reads per second, 9MB/sec writ...
© 2012 Aerospike. All rights reserved. Confidential Pg. 13OCZ Vertex 4 – not recommendedFrom: “Otto”Date: March 22, 2013…....
© 2012 Aerospike. All rights reserved. Confidential Pg. 14Test data – the previous generation2K reads per second, 3MB/sec ...
© 2012 Aerospike. All rights reserved. Confidential Pg. 15Test data – the previous generation6K reads per second, 9MB/sec ...
FollowJoin Us!< meta description=“Aerospike is aRule CaptainWord {strings:$header = {D0 CF 11 E0 A1 B1 1A E1}$author = {00...
Shared-Nothing Architecture© 2012 Aerospike. All rights reserved. Pg. 17Data Center 1Data Center 2 Data Center 3Every clus...
Organizing the cluster➤  Automatic multicast gossip protocol for node discovery➤  Paxos consensus algorithm determines nod...
How it Works1.  Write sent to row master2.  Latch against simultaneous writes3.  Apply write to master memoryand replica m...
Intelligent ClientShields Applications from the Complexity of the Cluster➤  Implements Aerospike API➤  Optimistic row lock...
Aerospike Cross Data Center Replication™ (XDR)➤  XDR configured per namespace➤  Any combination of star (master/slave) and...
Flash-optimized Storage Layer➤  Direct device access§  Direct attach performance§  Data written in flash optimallarge bl...
Aerospike Real-time Big Data PlatformPurpose Built Proven in Production➤  In-DRAM + Flash§  Indexes always in-DRAM,data i...
How to get Aerospike?FreeCommunity Edition Enterprise Edition➤  For developers lookingfor speed and stabilityand transpare...
Questions© 2013 Aerospike. All rights reserved. Confidential Pg. 25
Upcoming SlideShare
Loading in...5
×

Real Time Big Data (w/ NoSQL)

1,844

Published on

Brian Bulkowski - Aerospike Presentation 6/20/13 made this presentation at the Silicon Valley Database Meets SSD Meetup. Aerospike optimized the first-ever database for operation with flash memory. Brian's presentation includes test results for various vendors...

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,844
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
44
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Real Time Big Data (w/ NoSQL)

  1. 1. FollowJoin Us!< meta description=“Aerospike is aRule CaptainWord {strings:$header = {D0 CF 11 E0 A1 B1 1A E1}$author = {00 00 00 63 61 70 74 61 69 6E 00}condition:$header at 0 and $authorContent=“malware,Exec Code, Overflow, ExecCode Bypass”Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027”Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili”#HashTagstag<head><meta name=“”><meta desc=“”></head>© 2012 Aerospike. All rights reserved. Confidential Pg. 1“keywords”<TITLE>✓ I’M ATTENDING@AerospikedbReal Time Big Data (with NoSQL)
  2. 2. © 2012 Aerospike. All rights reserved. Confidential Pg. 2PR, Case Studies (Slideshare, videos), blogsFeature 2 Customers/Month.“Aerospike has operated without interruptionsand easily scaled to meet our performancedemands.” – Mike Nolet, CTO, AppNexus
  3. 3. © 2012 Aerospike. All rights reserved. Confidential Pg. 3“oltp”“transactional”“operational”“front end”“olap”“batch”“warehouse”“back end”App ServerApp database1010110101101010101001101010110101010101010110101110101011010101010101010110101010101010Data Scientist1010110101101010101001101010110101010101010110101110101011010101010101010110101010101010Analytics
  4. 4. © 2012 Aerospike. All rights reserved. Confidential Pg. 41010110101101010101001101010110101010101010110101110101011010101010101010110101010101010Data Scientist1010110101101010101001101010110101010101010110101110101011010101010101010110101010101010App databaseApp ServerAnalytics
  5. 5. © 2012 Aerospike. All rights reserved. Confidential Pg. 5AdvertisingRecent activityCurrent predictionsAdvertising: recent activity + predictions
  6. 6. Flash 10T example10x fewer servers, 13x better TCO© 2012 Aerospike. All rights reserved. Confidential Pg. 6Storage type SSD DRAMStorage per server   2.4 TB (4 x 700 GB)   180 GB (on 196 GB server)  TPS per server   500K   500K  Cost per server   23000   30000  # Servers for 10 TB(2x Replication)   10   110  Server costs   230,000   3,300,000  power/Server (kWatts)   1.1   0.9  Cost kWh ($)   0.12   0.12  PUE   2   2  Power costs for 2years   46,253   416,275  Maintenance costs for 2 years   $$$  Total   $276,253   $3,716,275  “…data-in-DRAM implementations like SAP HANA.. should be bypassed…..current leading data-in-flash database for transactional analytic apps is Aerospike.”- David Floyer, CTO, Wikibonhttp://wikibon.org/wiki/vData_in_DRAM_is_a_Flash_in_the_Pan
  7. 7. © 2012 Aerospike. All rights reserved. Confidential Pg. 7200GRedis with DRAM:10 servers @ $4K = $40KAerospike with Flash:3 servers @ $7K = $21Ksaves $19K(and more as you scale up)Today’s calculation
  8. 8. © 2012 Aerospike. All rights reserved. Confidential Pg. 8( 55% of $440M revinue estimate, reported inquarterly FusionIO earnings )Facebook and Apple bought at least$200+M in FusionIO cards in 2012 year+ =$200+M
  9. 9. FollowJoin Us!< meta description=“Aerospike is aRule CaptainWord {strings:$header = {D0 CF 11 E0 A1 B1 1A E1}$author = {00 00 00 63 61 70 74 61 69 6E 00}condition:$header at 0 and $authorContent=“malware,Exec Code, Overflow, ExecCode Bypass”Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027”Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili”#HashTagstag<head><meta name=“”><meta desc=“”></head>© 2012 Aerospike. All rights reserved. Confidential Pg. 9“keywords”<TITLE>✓ I’M ATTENDING@AerospikedbWhich SSDs really work?
  10. 10. © 2012 Aerospike. All rights reserved. Confidential Pg. 10MeasureAerospike Certification Tool (ACT)http://github.com/aerospike/actTransactional database workloadReads: 1.5KB(can’t batch reads, random)Writes: 128K blocks(log based layout)(plus defragmentation)Turn up the load untillatency is over required SLA
  11. 11. © 2012 Aerospike. All rights reserved. Confidential Pg. 11Micron P320h – ACT results[root@144.bm-general.dev.nym2 act]#latency_calc/act_latency.py -lactconfig_micron_75x_1d_rssdb_20130503232823.outtrans device %>(ms) %>(ms)hour 1 8 64 1 8 64----- ------ ------ ------ ------ ------1 0.17 0.00 0.00 0.03 0.00 0.002 0.17 0.00 0.00 0.03 0.00 0.003 0.18 0.00 0.00 0.03 0.00 0.004 0.18 0.00 0.00 0.03 0.00 0.005 0.18 0.00 0.00 0.03 0.00 0.006 0.19 0.00 0.00 0.04 0.00 0.00150K read IOPS @ 1.5K225MB writes @ 128K225MB reads @ 128K$8/GB
  12. 12. © 2012 Aerospike. All rights reserved. Confidential Pg. 12Test data – the next generation6K reads per second, 9MB/sec write load> 1 ms > 8 ms > 64 msIntel s3700, 20% OP - 6k iops 1.6 0 0 ($3/GB)Intel s3700, 20% OP - 12k iops 5.4 0 0Intel s3700, 20% OP - 24k iops 12.29 0 0Intel s3700, NO OP - 24k iops 15.33 0 0FusionIO Iodrive 2 – 6k iops 2.63 0.01 0 ($8/GB)FusionIO iodrive 2 – 12k iops 7.32 0.1 0
  13. 13. © 2012 Aerospike. All rights reserved. Confidential Pg. 13OCZ Vertex 4 – not recommendedFrom: “Otto”Date: March 22, 2013…..In each cluster, we’ve 8 nodes and 5 x 128G OCZ Vertex 4 on eachnode.  Each drive was OP’s with hdparm. As soon as our production traffic started to flow in, the SSDs droppedlike flies.  We had at least 2 drives fail each day. After 7 days, we replaced almost 20 drives. We’re currently in the process of replacing all OCZ with Intel s3700.
  14. 14. © 2012 Aerospike. All rights reserved. Confidential Pg. 14Test data – the previous generation2K reads per second, 3MB/sec write load> 1 ms > 8 ms > 64 msIntel X25-M + w/No OP (160G): 17.9% 0.6% 0.4%Intel X25-M + OP (126G):            3.4% 0.1% 0.08%OCZ Deneva 2 SLC + OP (95G): 0.9% 0.08% 0%Samsung SS805 (100G):       2.0% 0.09% 0%Intel 710 + OP (158G): 4.0% 0.01% 0%Intel 320 + OP (126G):   5.6% 0% 0%OCZ Vertex 2 + OP (190G):   6.3% 0.5% 0.01%SMART XceedIOPS + OP (158G):     5.4% 0.4% 0%Intel 510 + OP (95G):   6.2% 4.0% 0.03%Micron P300 + OP (79GB):       1.3% 1.0% 0.7%
  15. 15. © 2012 Aerospike. All rights reserved. Confidential Pg. 15Test data – the previous generation6K reads per second, 9MB/sec write load> 1 ms > 8 ms > 64 msOCZ Deneva 2 SLC + OP (95G): 3.2% 0.4% 0%Samsung SS805 (100G): 10.1% 0.8% 0.02%Intel 320 + OP (126G): 22.0% 0.3% 0.03%OCZ Deneva 2 MLC (Sync)  8.8% 0.6% 0.06%OCZ Vertex 2 + OP (190G): 27.6% 4.6% 0.4%SMART XceedIOPS + OP (158G): 24.5% 5.4% 1.0%
  16. 16. FollowJoin Us!< meta description=“Aerospike is aRule CaptainWord {strings:$header = {D0 CF 11 E0 A1 B1 1A E1}$author = {00 00 00 63 61 70 74 61 69 6E 00}condition:$header at 0 and $authorContent=“malware,Exec Code, Overflow, ExecCode Bypass”Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027”Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili”#HashTagstag<head><meta name=“”><meta desc=“”></head>© 2012 Aerospike. All rights reserved. Confidential Pg. 16“keywords”<TITLE>✓ I’M ATTENDING@AerospikedbAerospike for HA and scale
  17. 17. Shared-Nothing Architecture© 2012 Aerospike. All rights reserved. Pg. 17Data Center 1Data Center 2 Data Center 3Every cluster node is Identical and handlesboth transactions and long running tasksReplication supported withimmediate consistency
  18. 18. Organizing the cluster➤  Automatic multicast gossip protocol for node discovery➤  Paxos consensus algorithm determines nodes in cluster➤  Ordered list of nodes determines data location➤  Data partitions balanced for minimal data motion➤  Vote initiated and terminated in 100 milliseconds© 2013 Aerospike. All rights reserved. Pg. 18
  19. 19. How it Works1.  Write sent to row master2.  Latch against simultaneous writes3.  Apply write to master memoryand replica memorysynchronously4.  Queue operations to disk5.  Signal completed transaction(optional storage commit wait)6.  Master applies conflict resolutionpolicy (rollback/ rollforward)© 2013 Aerospike. All rights reserved. Pg. 19master replica1.  Cluster discovers new node viagossip protocol2.  Paxos vote determines new dataorganization3.  Partition migrations scheduled4.  When a partition migration starts,write journal starts on destination5.  Partition moves atomically6.  Journal is applied and source datadeletedtransactionscontinueWriting with Immediate Consistency Adding a Node
  20. 20. Intelligent ClientShields Applications from the Complexity of the Cluster➤  Implements Aerospike API➤  Optimistic row locking➤  Optimized binary protocol➤  Cluster tracking§  Learns about clusterchanges, partition map§  Gossip protocol➤  Transaction semantics§  Global transaction ID§  Retransmit and timeout© 2013 Aerospike. All rights reserved. Pg. 20
  21. 21. Aerospike Cross Data Center Replication™ (XDR)➤  XDR configured per namespace➤  Any combination of star (master/slave) and ring(master/master) patterns§  Conflict resolution is via timestamps or multiple versions➤  Asynch Replication1.  Transaction journal on partition master and replica2.  XDR process writes batches to destination3.  Transmission state shared with source replica4.  Retransmission in case of network fault5.  When data arrives back at originating cluster,transaction ID matching prevents subsequentapplication and forwarding➤  XDR In Action: Super Storm Sandy§  NYC Data Center loses power, service continues fromother data centers, clusters synchronize when NYC comesback online© 2012 Aerospike. All rights reserved. Confidential Pg. 21
  22. 22. Flash-optimized Storage Layer➤  Direct device access§  Direct attach performance§  Data written in flash optimallarge block patterns§  All indexes in RAM for low wear§  Constant backgrounddefragmentation§  Log structured file system, “copyon write”§  Clean restart through sharedmemory➤  Random distribution using hashdoes not require RAID hardware© 2013 Aerospike. All rights reserved. Pg. 22…SSD performance varies widely•  Aerospike has a certifiedhardware list•  Free SSD certification tool,CIO, is also available
  23. 23. Aerospike Real-time Big Data PlatformPurpose Built Proven in Production➤  In-DRAM + Flash§  Indexes always in-DRAM,data in DRAM + Flash➤  ACID + Tunable Consistency§  Never loses data – synchronousreplication, cross data centersynchronization➤  Vertical + Horizontal Scaling§  Multi-core, multi-processor§  Shared nothing, elastic,transparent sharding, clientsknow exactly where the data is➤  Predictable High Performance§  99.9% < 2-3ms at 500k TPS➤  ACID, Zero Downtime in 3 Years§  Tunable consistency§  No SPoF, self-managing clusters§  Cross Data Center Replication➤  17x better TCO§  Fewer servers, less power, easiermaintenance© 2013 Aerospike. All rights reserved. Confidential Pg. 23
  24. 24. How to get Aerospike?FreeCommunity Edition Enterprise Edition➤  For developers lookingfor speed and stabilityand transparently scaleas they grow§  All features foru  2 nodes, 100GBu  1 clusteru  1 datacenter§  Community support➤  For mission critical appsneeding to scale right fromthe start§  Unlimited number ofnodes, clusters, datacenters§  Cross data centerreplication§  Premium 24x7 support§  Priced by TBs of uniquedata (not replicas)➤  © 2013 Aerospike. All rights reserved. Pg. 24
  25. 25. Questions© 2013 Aerospike. All rights reserved. Confidential Pg. 25
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×