• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Cosbench apac
 

Cosbench apac

on

  • 453 views

 

Statistics

Views

Total Views
453
Views on SlideShare
453
Embed Views
0

Actions

Likes
2
Downloads
9
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Cosbench apac Cosbench apac Presentation Transcript

    • COSBench: A Benchmark Tool for Cloud Object Storage Services 段建钢 英特尔亚太研发中心 云基础架构实验室 2012/81
    • Agenda • Cloud Storage in tomorrow’s Data Center • COSBench Introduction • Case Study to evaluate OpenStack* swift performance with COSBench • Summary2
    • Agenda • Cloud Storage in tomorrow’s Data Center • COSBench Introduction • Case Study to evaluate OpenStack* swift performance with COSBench • Summary3
    • Data Centers are Evolving Compute • Data centers are built Flexible upon three fundamental Workloads pillars: – Compute – Storage Virtualized – Networking Storage Infrastructure • All three are critical for Open efficient data center Platforms Network operations Common – Balanced in Fabrics performance and utilization A Balanced Data Center is Essential for Efficiency4
    • IDC Storage Capacity Growth† EB 90 Structured data (23.6% CAGR) Traditional enterprise database 80 Replicated data (24.2% CAGR) 70 Backups 60 Data warehouses 50 Unstructured data (54.8% CAGR) Archives 40 Content Depots (75.6% CAGR) 30 Web 20 Email Document sharing 10 Social network content (pictures/videos) 2009 2010 2011 2012 2013 2014 2012 Deployment ~7.6 million drives Estimate: ~500,000 storage systems‡ †Source: IDC, Worldwide Enterprise Storage Systems 2010–2014 Forecast: Recovery, Efficiency, and Digitization Shaping Customer Requirements for Storage Systems, Doc ‡Source: Internal estimates based on the IDC Worldwide Enterprise Storage Systems Forecast # 223234., May 20115
    • Usage Models Dictate the Solutions Business DB PerformanceStorage Performance Random small (OLTP, OLAP) Storage Content distribution network (CDN) Requirement (Objects per second) Application data store (e.g. e-mail, VM/Boot, Sharepoint*) Large Relational DB (e.g. NoSQL, non ACID) Large analytics (e.g Hadoop*/HDFS) Sequential Large High performance compute (e.g. pNFS, Luster*) Capacity Storage Cloud Object storage COSBench (e.g. photos/videos) Backup and archive (server and client) Gigabytes Terabytes Petabytes Exabytes Storage Capacity Requirement Key Storage Usage Models Have Differing Requirements Thus Need New Benchmarks 6
    • Agenda • Cloud Storage in tomorrow’s Data Center • COSBench Introduction • Case Study to evaluate OpenStack* swift performance with COSBench • Summary7
    • COSBench Introduction • COSBench is an Intel developed benchmark to measure Cloud Object Storage Service performance • Cloud end user can use COSBench to compare different public Cloud Object Storage service performance • Cloud builder can use it to – Compare different Hardware/Software Stacks – Identify bottleneck and make optimization COSBench is the IOMeter for Cloud Object Storage service8
    • COSBench Key Component Config.xml: – define workload with flexibility. Web Console Controller Controller: – Control all drivers Config.xml – Collect and aggregate stats. COSBench Driver: – generate load w/ config.xml parameters. Driver Driver – can run tests w/o controller. Web Console: Controller Node – Manage controller – Browse real-time stats Storage – Communication is based on HTTP (RESTful style) Cloud Storage Node9
    • Web Console Driver list Workload List History list Intuitive UI to get Overview.10
    • Workload Configuration Flexible load control object size distribution Read/Write Operations Workflow for complex stages Flexible configuration parameters is capable of complex Cases11
    • Performance Metrics Throughput (Operations/s): the operations completed in one second Response Time (in ms): the duration between operation initiation and completion. Bandwidth (KB/s): the total data in KiB transferred in one second Success Ratio (%): the ratio of successful operations12
    • Agenda • Cloud Storage in tomorrow’s Data Center • COSBench Introduction • Case Study to evaluate OpenStack* swift performance with COSBench • Summary13
    • OpenStack* Swift overview OpenStack* is open source software to build private and public clouds. OpenStack Object Store (Swift): Create petabytes of reliable storage using standard servers †Source: docs.openstack.org14
    • OpenStack* Swift Overview Entities  RING  physical location (zone/device/partition/…) Proxy Node Proxy Server Account Container Object Ring Ring Ring Account Server Container Server Object Server Storage Node metadata Object Account Container file DB DB15
    • Test Configuration16
    • Test OpenStack* Swift performance Software and workloads used in performance tests may have been optimized for performance only on Intel® microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those17 factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance.
    • Swift characterization - Insufficient processing power throttles overall performance. Baseline (ObjectSize=64KB, Concurrency=512, 12 Disks per node) Op/s RT (ms) BW (KB/s) Performance 5,644 91 361,220 Proxy Node 6 Node 7 Node 8 Node 9 Node 10Client r/s 90.31 91.16 90.64 91.01 91.17 w/s 0.01 0.01 0.01 0.00 0.01 rKB/s 5,633 5,378 5,384 5,379 5,381 wKB/s 0.09 0.09 0.09 0.08 0.08 data disk await 5.16 5.05 5.13 5.16 5.14 rxkB/s 356,225 1,813 1,826 1,844 1,757 1,710 Internal txkB/s 8,356 71,910 73,393 74,265 74,559 72,641 rxkB/s 3,524 External txkB/s 357,506 user% 79.70 14.35 1.42 cpu% system% 19.40 4.26 1.74 iowait% 0.00 21.85 0.13 Software and workloads used in performance tests may have been optimized for performance only on Intel® microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those18 factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance.
    • Agenda • Cloud Storage in tomorrow’s Data Center • COSBench Introduction • Case Study to evaluate OpenStack* swift performance with COSBench • Summary19
    • Summary • New storage Usage model rises for Cloud Computing age, which need new benchmark • COSBench is a new benchmark developed by Intel to measure Cloud Object Storage service performance • COSBench is useful to analyze Cloud Object Service system performance, identify bottleneck and conduct optimization20
    • Next Step and call for action • We are developing COSBench to support more Cloud Object Storage service • Our final goal is to open source COSBench to make it available for industry and community use to make better Object Storage service design • We will continue to use COSBench to analyze the optimize OpenStack* Swift performance and share back our finding to community • Any question, feedback, please contact me at: • Jiangang.duan@intel.com21
    • backup22
    • Storage Layout How data is stored in each node ? /dev/swift/a /dev/swift/b /dev/swift/c • accounts • accounts • accounts • containers • containers • containers • objects • objects • objects • async_pending • async_pending • async_pending object 1025 1027 partition DG3 12C 1026 G1J 45A1…12C SFT3…12C hash suffix hash23
    • GET/HEAD@Proxy Node How proxy node cooperate with storage nodes to obtain object data ? Consult the Retrieve Ring for container candidate Return the information nodes result Perform GET/HEAD A&A using candidate pre-hooked 200 server(s) facilities 0 (R&H) no response 5xx 314 server error ? 507 404 unmodified disk error 412 file not found precondition failed or not synchronized24
    • PUT@Proxy Node – Part I How proxy cooperate with storage nodes to create an object ? Consult the Retrieve Ring for Check container candidate various information nodes constraints Perform Create the A&A using timestamp pre-hooked header facilities 413 411 404 400 Invalid Path Object Length Container Too Large Required not Found Invalid Object Name25
    • PUT@Proxy Node – Part II How proxy cooperate with storage nodes to create an object ? Try making Forward 3 Phased Workflow R conns to data to storage storage Return the servers servers result Assign Collect each conn resps from a container storage server 0 201 servers Time out: 86400 secs no response 5xx 408 time out server error 507 422 disk error data corrupted26
    • Disclaimers INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTELS TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTELS PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm%20 This document contains information on products in the design phase of development. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Copyright © 2012 Intel Corporation. All rights reserved.27