Benchmark Comparison Lunacloud x Amazon ec2 x Rackspace cloud

  • 272 views
Uploaded on

An independent US-based cloud analyst group released the CloudSpecs Performance Report in which it underlines that "Lunacloud outperforms Amazon and Rackspace consistently over a period of 30 days". …

An independent US-based cloud analyst group released the CloudSpecs Performance Report in which it underlines that "Lunacloud outperforms Amazon and Rackspace consistently over a period of 30 days". This CloudSpecs Performance Report compares cloud servers of Amazon EC2, Rackspace Cloud Servers and Lunacloud. Performance is measured over a period of 30 days to collect data on server performance capabilities as well as consistency in performance. The CloudSpecs system runs open source benchmark tests to measure performance in CPU, RAM, disk, and internal network. Some of the key findings were that in the general server test, Lunacloud outperforms Amazon and Rackspace consistently over a period of 30 days; Lunacloud disk performance scores up to 8x better than Amazon or Rackspace disk performance; and Lunacloud’s internal network throughput is 2x faster than Rackspace’s internal network throughput, and significantly more stable than Amazon in a period of 30 days. http://www.lunacloud.com/en/lunacloud-outperforms-amazon-ec2

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
272
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
11
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. CLOUDSPECS PERFORMANCE REPORT LUNACLOUD, AMAZON EC2, RACKSPACE CLOUD AUTHOR: KENNY LI NOVEMBER 2012
  • 2. EXECUTIVE SUMMARY This publication of the CloudSpecs Performance Report compares cloud servers of Amazon EC2, Rackspace Cloud Servers (not OpenStack Cloud), and Lunacloud. Performance is measured over a period of 30 days to collect data on server performance capabilities as well as consistency in performance. The CloudSpecs system runs open source benchmark tests to measure performance in CPU, RAM, disk, and internal network. The software is set to automatically run 4 times a day for 7 days a week. The report is comprised of:  1 general server performance test  5 CPU performance tests    3 Disk performance tests 2 RAM performance tests 7 network performance tests Key Findings    In the general server test, Lunacloud outperforms Amazon and Rackspace consistently over a period of 30 days. Lunacloud disk performance scores up to 8x better than Amazon or Rackspace disk performance. This is an important consideration for database performance. Lunacloud’s  internal  network  throughput  is  2x  faster  than   Rackspace’s  internal  network  throughput,  and  significantly  more   stable than Amazon in a period of 30 days. With applications requiring more than one server, internal network speed may be considered a bottleneck in many cases if other components of the server are optimized.   Amazon  EC2’s  CPU  performance  generally  scores  slightly  better   than Rackspace and Lunacloud CPU performance. CPU is a vital component of the server, and application performance depends significantly on CPU functionality. Rackspace’s  RAM  is  more  consistent  in  performance  than   Amazon or Lunacloud, though it scores the lowest in performance over a period of 30 days. Though the processor speed is an important factor in server performance, RAM plays an equally important role. Because modern CPUs can perform many of the applications run today, bottleneck instances may actually occur with RAM. METHODOLOGY Cloud Spectator sets up two servers on each cloud provider that meets the following minimum requirement: Primary Server Secondary Server CPU Cores RAM in GB Disk in GB 4 vCPUs 16 50 CPU Cores RAM in GB Disk in GB 1 vCPU 4 50 Because Amazon and Rackspace offer packaged cloud servers with preset allocations of CPU, RAM, and disk, the closest package that meets the minimum requirements is selected. AMAZON The servers are set up on Amazon EC2 availability zone US-East-1a. Virtualized server OS is Amazon Linux. Primary Server Secondary Server CPU Cores RAM in GB Disk in GB 4 vCPUs 15 1,690 CPU Cores RAM in GB Disk in GB 1 vCPUs 3.75 410 CPU Cores RAM in GB Disk in GB 2 vCPUs 4 160 RACKSPACE The  servers  are  set  up  in  Rackspace’s  Dallas  (DFW)  data  center. Virtualized server OS is Debian 6. Primary Server Secondary Server CPU Cores RAM in GB Disk in GB 4 vCPUs 15.872 620 2
  • 3. LUNACLOUD The  servers  are  set  up  in  Lunacloud’s  Virtualized  server  OS  is  Debian  6. Primary Server Secondary Server CPU Cores RAM in GB Disk in GB 4 vCPUs 16 50 CPU Cores RAM in GB Disk in GB 1 vCPUs 4 50 Cloud Spectator maintains root access to all servers. Using SSH, the CloudSpecs application is installed on to the servers. Both Primary and Secondary servers are located within the same virtual private network. Once the CloudSpecs performance application is installed, it automatically runs 4 times a day, 365 days a year. PERFORMANCE TESTS Cloud Spectator runs basic, open source benchmark tests that are precise and accurate for measuring the performance of a cloud server. Tests included come from the Unixbench Suite (https://code.google.com/p/byte-unixbench) and Phoronix Test Suite (http://www.phoronix-test-suite.com). The benchmarks for network tests and the disk benchmark for local file copy may be reproduced using the Linux command line interface. The network tests assume $SERVER is the IP of the secondary server and $CLIENT is the IP of the primary server. Command line instructions for the internal network tests are included below in each of the tests within the network description. DATA COLLECTION The CloudSpecs test results for this publication run from October 17, 2012 to November 16, 2012 (a total of 30 days). Cloud Spectator obtains cloud servers by either purchasing the server space directly from the client, or the client reimburses Cloud Spectator for data collected relevant to that active project. CLOUDSPECS SCORE The CloudSpecs Score represents the true value a cloud user receives when using a server. A higher score means better value. The highest value performer receives a score of 100 and other providers are pegged in relativity to value. Value is calculated based on performance over time and the cost of the Primary Server hourly (730 hours per month). Provider Amazon EC2 East Rackspace Lunacloud Cost per Hour $0.64 $0.90 $0.32 Below  is  the  formula  process  for  calculating  the  CloudSpecs  Score  (Provider’s  CloudSpecs  Score  is  the  calculation for the final CloudSpecs score of each provider). The CloudSpecs score is rounded off to the nearest whole number. The CloudSpecs score in this report were calculated over a period of 30 days:  provider_value(P) = [Provider test score over a period of time] / [Provider price]  best_provider_value = max(provider_values)  Provider’s  CloudSpecs  Score = 100 * provider_value(P) / best_provider_value The  Sectional  CloudSpecs  Score  is  calculated  by  averaging  the  CloudSpecs  scores  of  each  section’s  available  tests. 3
  • 4. PERFORMANCE SUMMARY General Server Performance UnixBench (Score) Provider Lunacloud Amazon Rackspace 30-day Average 1,305 1,037 955 CloudSpecs Score 100 40 26 CPU Performance 7-Zip Compression (MIPS) Provider Lunacloud Amazon Rackspace 30-day Average 4,498 6,777 6,152 Video Encoding Using x264 (FPS) CloudSpecs Score 100 75 49 Provider Lunacloud Amazon Rackspace 30-day Average 29 52 47 CloudSpecs Score 100 90 58 Linux Kernel Compilation (Seconds to Compile) Audio Encoding Using LAME MP3 (Seconds to Compile) Provider Lunacloud Amazon Rackspace Provider Lunacloud Amazon Rackspace 30-day Average 449 234 251 CloudSpecs Score 100 96 64 30-day Average 25 31 32 CloudSpecs Score 100 48 28 Apache Kernel Compilation (Seconds to Compile) Provider Lunacloud Amazon Rackspace 30-day Average 69 67 63 CloudSpecs Score 100 55 37 Disk Performance DBench (MiB/sec) Provider Lunacloud Amazon Rackspace 30-day Average 399 129 95 Iozone (MiB/sec) CloudSpecs Score 100 16 8 Provider Lunacloud Amazon Rackspace 30-day Average 1,871 960 716 CloudSpecs Score 100 26 14 File Copy (MiB/sec) Provider Lunacloud Amazon Rackspace 30-day Average 754 26 39 CloudSpecs Score 100 2 2 RAM Performance STREAM (MiB/sec) Provider Lunacloud Amazon Rackspace 30-day Average 11,114 15,497 6,648 RAMSpeed SMP (MiB/sec) CloudSpecs Score 100 70 21 Provider Lunacloud Amazon Rackspace 30-day Average 7,665 7,676 5,501 CloudSpecs Score 100 50 25 4
  • 5. Internal Network Performance Network Throughput with Iperf (Mibit/sec) Apache WGET (MiBit/sec) Provider Lunacloud Amazon Rackspace Provider Lunacloud Amazon Rackspace 30-day Average 767 384 396 CloudSpecs Score 100 25 18 FTP to RAM (MiBit/sec) Provider Lunacloud Amazon Rackspace 30-day Average 916 446 187 30-day Average 933 279 187 CloudSpecs Score 100 31 8 Nginx WGET (MiBit/sec) CloudSpecs Score 100 24 7 FTP to Disk (MiBit/sec) Provider Lunacloud Amazon Rackspace 30-day Average 864 528 188 Provider Lunacloud Amazon Rackspace 30-day Average 891 517 187 CloudSpecs Score 100 29 7 Network SCP (MiBit/sec) CloudSpecs Score 100 15 7 Provider Lunacloud Amazon Rackspace 30-day Average 169 118 184 CloudSpecs Score 100 35 39 5
  • 6. GENERAL SERVER PERFORMANCE Cloud Spectator runs the UnixBench benchmark suite to gauge overall performance of a cloud server. Over the test period of 30 days, Lunacloud outperforms Amazon and Rackspace in the UnixBench test. UNIXBENCH Description The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system. Why It Matters The UnixBench is a suite of approximately 10 tests that can measure how many instructions per second a CPU can perform. If CPU A performs 2x better than CPU B at only 1.5x the cost, then CPU A provides more performance per unit of cost. Provider Lunacloud Amazon Rackspace 30-day Average 1,305 1,037 955 CloudSpecs Score 100 40 26 6
  • 7. CPU PERFORMANCE 7ZIP COMPRESSION Description This is a test of 7-Zip using p7zip with its integrated benchmark feature, measured in million instructions per second (MIPS). The test consists of compressing a file with random data using the 7-zip program and then dividing the number of CPU instructions executed during the compression by the number of seconds. The result is then divided by 1 million to calculate the value in MIPS. Why It Matters The 7-zip compression test results are important in comparison because the 7-zip compression algorithm uses many kinds of CPU instructions, resulting in an accurate overall score reflecting the performance of the CPU. The fact that it is a compression program is irrelevant in the test. Provider Lunacloud Amazon Rackspace 30-day Average 4,498 6,777 6,152 CloudSpecs Score 100 75 49 TIMED LINUX KERNEL COMPILATION Description This test times how long (in seconds) it takes to build the Linux kernel. Unlike the 7-zip compression, this is a compilation source code. Though many compilation tests exist, an advantage of the Linux Kernel compilation is the portability and compatibility with a majority of Linux distributions. Why It Matters Because of the variety of CPU instructions and the length of time for the test, the Linux kernel compilation can accurately measure  a  CPU’s  performance. Provider Lunacloud Amazon Rackspace 30-day Average 449 234 251 CloudSpecs Score 100 96 64 7
  • 8. TIMED APACHE COMPILATION Description This test times how long (in seconds) it takes to build the Apache HTTP Server. Though many compilation tests exist, an advantage of the Apache compilation is the portability and compatibility with a majority of Linux distributions. Why It Matters Because of the variety of CPU instructions and the length of time for the test, the Apache compilation can accurately measure  a  CPU’s  performance. Provider Lunacloud Amazon Rackspace 30-day Average 69 67 63 CloudSpecs Score 100 55 37 X264 VIDEO ENCODING Description The x264 test benchmarks the speed it takes a server to encode a movie file using the x264 H.264/AVC encoder. Why It Matters When results are shown alone, they have little meaning, but the process requires a large amount of computations, so when compared side-by-side using results from several cloud providers, the test provides an accurate insight of CPU performance. Provider Lunacloud Amazon Rackspace 30-day Average 29 52 47 CloudSpecs Score 100 90 58 8
  • 9. LAME MP3 ENCODING Description LAME is an MP3 encoder licensed under the LGPL. This test measures the time (in seconds) required to encode a WAV file to MP3 format. Why It Matters Similar to the x264 test, results alone have little meaning, but the process requires a large amount of computations, so when compared side-by-side using results from several cloud providers, the test provides accurate insight of CPU performance. Provider Lunacloud Amazon Rackspace 30-day Average 25 31 32 CloudSpecs Score 100 48 28 9
  • 10. DISK PERFORMANCE DBENCH Description DBench is a benchmark designed by the Samba project as a free alternative to netbench, but dbench contains only file-system calls for testing the disk performance. Why It Matters DBench can be used to stress a filesystem or a server to see which workload it becomes saturated and can also be used for prediction analysis to determine how many concurrent clients/applications performing this workload the server can handle before response starts to lag. Provider Lunacloud Amazon Rackspace 30-day Average 399 129 95 CloudSpecs Score 100 16 8 FILE COPY Description This measures the performance of the operating system's 'cp' command, followed by a 'sync' (to flush any cache), executed on the local storage device. The file size is 10GB, and is copied to the same disk measured in mebibytes per second. Why It Matters File copy measures read and write performance of the disk, which is important when processing massive amounts of data. Provider Lunacloud Amazon Rackspace 30-day Average 754 26 39 CloudSpecs Score 100 2 2 10
  • 11. IOZONE FILESYSTEM BENCHMARK Description Iozone benchmarks file IO performance of a server for the operations: Read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread ,mmap, aio_read, aio_write Why It Matters Iozone can collect data on a broad range of file system performance to locate hot or cold spots in the file systems. Provider Lunacloud Amazon Rackspace 30-day Average 1,871 960 716 CloudSpecs Score 100 26 14 11
  • 12. RAM TESTS STREAM Description The STREAM benchmark is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s) and the corresponding computation rate for simple vector kernels. The STREAM benchmark is specifically designed to work with datasets much larger than the available cache on any given system, so that the results are (presumably) more indicative of the performance of very large, vector style applications. Why It Matters STREAM measures memory bandwidth to see how the RAM accommodates to the processes running on the server. Provider Lunacloud Amazon Rackspace 30-day Average 11,114 15,497 6,648 CloudSpecs Score 100 70 21 RAMSPEED SMP Description This benchmark tests the system memory (RAM) performance. Why It Matters RAMspeed results correlate highly with STREAM results. Though the algorithms are different, RAMspeed SMP also measures the RAM accommodation to processes running on the server. Provider Lunacloud Amazon Rackspace 30-day Average 7,665 7,676 5,501 CloudSpecs Score 100 50 25 12
  • 13. NETWORK PERFORMANCE TESTS NETWORK THROUGHPUT WITH IPERF Description Iperf is a commonly used network testing tool that can create TCP and UDP data streams and measure the throughput of a network that is carrying them. Our test consists of transferring as much data as possible through the local network (where possible, private network IPs were used) for 120 seconds over TCP port 5001. Why It Matters Iperf tests the quality of the network. Using this test between two cloud servers in the same data center of the same provider, results can be collected on how efficient the internal network speed is, which is important for servers inside the same virtual private network that may be required to transfer data between or among each other quickly and efficiently to effectively run applications. Code wget 'http://downloads.sourceforge.net/project/iperf/iperf2.0.5.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fiperf%2F&ts=1341586993&use_mirror=ignum' -O iperf.tgz tar zxvf iperf.tgz cd iperf-2.0.5 ./configure make ./src/iperf -s -p 5001 wget 'http://downloads.sourceforge.net/project/iperf/iperf2.0.5.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fiperf%2F&ts=1341586993&use_mirror=ignum' -O iperf.tgz tar zxvf iperf.tgz cd iperf-2.0.5 ./configure make ./src/iperf -c $SERVER -p 5001 -t 120 Provider Lunacloud Amazon Rackspace 30-day Average 767 384 396 CloudSpecs Score 100 25 18 13
  • 14. NETWORK FTP TO RAMDISK Description Using vsftpd (Very Secure FTP Daemon) on Server and lftp on Client, the test transfer a 1GB file from RAM drive to RAM drive. Why It Matters Transmitting from a RAM disk bypasses the disk in the network test, preventing a possible bottleneck at the disk level and enabling a more accurate measurement of the internal network transfer speed. Code mkdir -p /mnt/ramfs chmod 0555 /mnt/ramfs mount -t ramfs -o size=1050mb ramfs /mnt/ramfs dd if=/dev/zero of=/mnt/ramfs/file.big bs=1MiB count=1024 time lftpget -d ftp://$SERVER/file.big Provider Lunacloud Amazon Rackspace 30-day Average 916 446 187 CloudSpecs Score 100 24 7 14
  • 15. NETWORK FTP TO HARD DISK Description Using vsftpd (Very Secure FTP Daemon) on Server and lftp on Client, the test transfer a 5GB file from hard disk to hard disk. Why It Matters The File Transfer Speed of the Disk is similar to the FTP speed of the RAM disk, but in a manner that measures the speed efficiency of a disk-to-disk transfer, rather than RAM-to-RAM. Code dd if=/dev/zero of=/tmp/file.big bs=5MiB count=1024 time lftpget -d ftp://$SERVER/file.big Provider Lunacloud Amazon Rackspace 30-day Average 933 279 187 CloudSpecs Score 100 15 7 15
  • 16. NETWORK HTTP Apache Description Using apache2 as Server and wget as Client, transfer of a 1 GB file. Why It Matters Performing a more natural test, the Apache Network HTTP, Nginx Network HTTP, and Network SCP tests all combine the internal network with CPU usage to see how fast servers perform when they have added CPU usage. Code dd if=/dev/zero of=/tmp/file.big bs=1MiB count=1024 time wget http://$SERVER/file.big Provider Lunacloud Amazon Rackspace 30-day Average 864 528 188 CloudSpecs Score 100 31 8 16
  • 17. NETWORK HTTP NGINX Description Using nginx (a web server application) as server and wget as Client, transfer of a 1 GB file. Why It Matters Performing a more natural test, the Apache Network HTTP, Nginx Network HTTP, and Network SCP tests all combine the internal network with CPU usage to see how fast servers perform when they have added CPU usage. Code dd if=/dev/zero of=/tmp/file.big bs=1MiB count=1024 time wget http://$SERVER/file.big Provider Lunacloud Amazon Rackspace 30-day Average 891 517 187 CloudSpecs Score 100 29 7 17
  • 18. NETWORK SCP Description Copy of a 5GB file using Secure Copy (SCP). Why It Matters Performing a more natural test, the Apache Network HTTP, Nginx Network HTTP, and Network SCP tests all combine the internal network with CPU usage to see how fast servers perform when they have added CPU usage. Code dd if=/dev/zero of=/home/guest/file.big bs=5MiB count=1024 time scp guest@$SERVER:file.big . Provider Lunacloud Amazon Rackspace 30-day Average 169 118 184 CloudSpecs Score 100 35 39 About Cloud Spectator Cloud Spectator is the premier cloud analyst group focused on infrastructure pricing and server performance. Since 2011, Cloud Spectator has monitored the cloud Infrastructure industry on a global scale and continues to produce research reports for businesses to make informed purchase decisions by leveraging its CloudSpecs utility, an application that automates live server performance tests 4 times a day, 365 days a year with use of open source benchmark tests. The data is compared relatively with pricing from each provider to allow business decision-makers to smart decisions. Cloud Spectator 800 Boylston Street 16TH Floor Boston, MA 02199 Website: www.cloudspectator.com Phone: (USA) 1-617-300-0711 18