Performance Testing of Big Data Applications - Impetus Webcast

2,600 views

Published on

Impetus webcast "Performance Testing of Big Data Applications" available at http://lf1.me/cqb/

This Impetus webcast talks about:
• A solution approach to measure performance and throughput of Big Data applications
• Insights into areas to focus for increasing the effectiveness of Big Data performance testing
• Tools available to address Big Data specific performance related challenges

Published in: Technology

Performance Testing of Big Data Applications - Impetus Webcast

  1. 1. Performance Testing of Big Data Applications Mustafa Batterywala, Performance Architect Larry Pearson,Vice President Marketing Impetus Technologies, Inc.
  2. 2. Key Big Data Technologies • MapReduce – Apache Hadoop, Cloudera, Hortonworks, MapR etc. • NoSQL – Cassandra, Mongo DB, Oracle NoSQL, Neo4j etc. • Messaging Queues – Kafka, ActiveMQ, RabbitMQ, ZeroMQ etc. • Search – Lucene, Elastic Search, Solr © 2013 Impetus Technologies Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  3. 3. Agenda • Big Data Performance Testing Focus Areas • Challenges • Performance Testing Approach • Solutions © 2013 Impetus Technologies Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  4. 4. Performance Test Focus Areas © 2013 Impetus Technologies Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  5. 5. Performance Testing Challenges • • • • • • © 2013 Impetus Technologies Diverse technologies Unavailability of tools Test scripting Test environment Limited monitoring solutions Lack of diagnostic solutions Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  6. 6. Performance Testing Approach © 2013 Impetus Technologies Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  7. 7. Performance Testing Solutions • Performance Test Tools – YCSB (Yahoo Cloud Serving Benchmark), SandStorm, JMeter • Monitoring Tools – Nagios, Zabbix, Ganglia, JMX utilities • Diagnostic Tools (APM) – visualVM, AppDynamics, Compuware © 2013 Impetus Technologies Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  8. 8. Critical Performance Parameters • • • • • • • © 2013 Impetus Technologies Data Storage Commit Logs Concurrency Caching JVM Parameters MapReduce Configurations Message Queue Configurations Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  9. 9. Real World Experience The Application • Online Social networking website with almost 100 million registered users across the globe • Hundreds of thousands of users are online at any given time The Challenges • • • • Develop a near real time analytics solution to analyze the user feedback and interactions • Real time monitoring of test environment for bottleneck identification © 2013 Impetus Technologies Solution uses Kafka, HBase and Hive as major technologies SLA to support 50K messages per minute Optimize and tune the Kafka clusters for maximum throughout Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  10. 10. Real World Experience Impetus Contributions • • • • • • Proposed and implemented performance test strategy for analytics solution • Real time monitoring using SandStorm to identify performance bottlenecks Identified key performance components namely Kafka and Hbase for focus testing Proposed SandStorm as performance testing tool based on project requirements Prepared Kafka clients to simulate expected data ingestion volumes Optimized and tune single Kafka cluster in EC2 on medium instance Executed tests with varying message rate and reached to the max throughput of 50k messages per minute using 3 server cluster Benefits Realized • • • © 2013 Impetus Technologies Optimum hardware utilization for complete solution Zero performance issues on Go live Maximum throughput of Kafka servers Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  11. 11. Summary • Test each component of the eco-system – Data Ingestion Rate, Throughput, System Resources • Automate test environment in Cloud • Appropriate tools and plug-ins – Cloud v/s On-premise solutions – Custom Test Harness – Monitoring and Diagnostic Plug-ins • Failover and Reliability testing © 2013 Impetus Technologies Recorded webinar is available at http://lf1.me/cqb/ For more Info contact bigdata@impetus.com
  12. 12. Thank You Recorded version available at http://lf1.me/cqb/ For more info and/or a Demo of our Big Data performance testing solution bigdata@impetus.com @impetustech

×