Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Running a High Performance NoSQL Database on Amazon EC2 for Just $1.68/Hour

1,383 views

Published on

Rajkumar Iyer and Sunil Sayyaparaju reveal how their team proved that cost-effective, high performance in the cloud isn’t a myth. They will walk through the 10-step process to efficiently set up high-performance instances on Amazon EC2 with Aerospike.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Running a High Performance NoSQL Database on Amazon EC2 for Just $1.68/Hour

  1. 1. IN-MEMORY NOSQL, Now OPEN SOURCE! 1MTPS ON AMAZON @ $1.68/HR RAJKUMAR IYER Aerospike aer . o . spike [air-oh- spahyk] noun, 1. tip of a rocket that enhances speed and stability © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 1
  2. 2. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 2 AGENDA ■ Share knowledge we gained while trying to get best out of Amazon running Aerospike database. ■ List out steps and Demonstrate 1 Million TPS on a single instance applying the above principle in Amazon ■ Keep it Interactive
  3. 3. INSTANCE TYPES Instance Name Comment R3.* Memory Intensive C3.* Compute Intensive M3.* Balance between R and C T* A very low end beginner machine M1.* Previous Generation. Low network. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 3
  4. 4. NETWORK TYPES Moderate - Upper bound of 100MBps) High - Ranges from 300Mbps to 1.8Gbps 10 Gigabit - Upper bound of 8.8 Gbps) Instance Type Network Type *.large, *.xlarge Moderate *.2xlarge Moderate - High *.4xlarge High *.8xlarge 10 Gbps © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 4
  5. 5. OLD GENERATION INSTANCES ■ m1.* and t1.* has low to moderate network. ■ Are being retired. ■ Reliability Guarantee are lowest of all. ■ Can be used for developer activity. ■ Not for running high performance service. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 5
  6. 6. NEW GENERATION INSTANCES ■ C3.* R3.* and I3.* has Moderate to 10 Gbps network Capability ■ *.4xlarge and above instances are shared with lot less other instances. ■ Reliability Guarantee is on higher side. Barring Availability zone failure. ■ Can be used for Real Service. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 6
  7. 7. VIRTUAL MACHINE TYPE ■ PV *.8xlarge though rated at 10Gbps they upper bounded at 2.2 Gpbs (iperf result) ■ PV overhead at high load is around 40% of entire CPU utilization. ■ HVM has much better CPU utilization and network utilization. ■ HVM nearly outperform PV instance by 3x. On apple-to-apple comparison of C3.8xlarge HVM gave 215ktps vs 85ktps on PV. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 7
  8. 8. NETWORK PACKET PROCESSING ■ Soft Interrupt Processing (% si in top) is one of the largest CPU drains in the client-server network based system. ■ In AWS a NIC is 2 queues as exposed. Interrupt for which can be only be serviced on two cores. This causes core bottleneck at high throughput rates. ■ Single NIC peaks at 250ktps processing. Trick is to use multiple NICs ■ Interrupt coalescing improves CPU utilization a little but did not translate into better processing. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 8
  9. 9. NETWORK PACKET PROCESSING ■ Receive Packet Steering to spread IRQ processing over multiple cores. Single NIC with RPS enabled can push up to 800ktps with interrupts spread over 8 cores. echo ff | sudo tee /sys/class/net/eth0/queues/rx-[0-1]/rps_cpus ■ ENIs provides a way of adding multiple (virtual) NICs to an instance. IRQ processing for each spread over different core is optimal. echo 1 | sudo tee /proc/irq/<irq number>/smp_affinity ■ Using ENIs with private IPs is free of cost. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 9
  10. 10. MISCELLANEOUS ■ Having instances in the same PLACEMENT GROUP lets instances have low-latency, full bisection 10 Gbps network. ■ Instances have to be in a VPC to be in same placement © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 10 group ■ Shared vs Dedicated tenancy in the new generation instances did not seem to matter as we experimented. ■ To avoid CPU stealing, avoid over user of CPU. Typically running with large number of threads.
  11. 11. SETUP INSTANCES ■ Setup Security group with following ports open. 3000-3003 [Internode communication] 8081 [Aerospike Monitoring Console] 22 [ssh to the host ] ■ Setup c3.8xlarge HVM instances in a VPC. Set it up with Placement Group ■ Attach 3 additional ENI to the server instance. Setup RPS on each one of them. ■ Setup 5 c3.2xlarge instances to run as a client. Put them in same VPC and Placement group as server instance. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 11
  12. 12. SETUP AEROSPIKE SERVER ■ On server instance. Run afterburner.sh (cd /opt/aerospike/bin; sudo ./afterburner.sh) to set the optimal number of threads on each Aerospike server. ■ Configure the in-memory namespace to use 32GB of RAM (/etc/aerospike/aerospike.conf) ■ Start the Aerospike Server sudo service aerospike start ■ Start the Aerospike Monitoring Console on one of the client instances. sudo service amc start © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 12
  13. 13. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 13 FIRE THE LOAD ■ Load the server with data using Java benchmark client. cd <java client>/benchmarks ./run_benchmarks -z 64 -n test -w I -o S:10 -b 10 -l 23 -k 10000000 -latency 5,1 -h YOUR_AWS_INTERNAL_IP ■ Run the clients with 100% read load. Point each client to a separate private IP on different ENIs. cd <java client>/benchmarks ./run_benchmarks -z 64 -n test -w RU,100 -o S:10 -b 10 -l 23 -k 10000000 -latency 5,1 -h YOUR_AWS_INTERNAL_IP
  14. 14. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 14 RESULT 4 interfaces with proper irq affinity and 8 client processes on 4 client instances, we were able to get up to 960ktps on a single c3.8xlarge server instance.
  15. 15. CLUSTER SETUP ■ 4 node Aerospike cluster with data in memory © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 15 (RAM) ■ 40 Million unique objects in cluster. ■ 100 bytes record size; 10 bins of 10 bytes each. ■ Replication Factor 2 with synchronous replication for immediate consistency.
  16. 16. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 16 INSTANCES Instance Type Comment m1.large Previous Generation m3.xlarge Current Generation without Enhanced networking r3.2xlarge Current Generation with Enhanced Networking r3.large Current Generation low end
  17. 17. © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 17 NODES vs TPS
  18. 18. COST ANALYSIS © 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 18

×