Neev QA Offering

Neev QA Offerings
Seeing the Forest and the Trees

The Agenda
• Why Performance Matters for Digital Businesses?
• Where Neev can help? A few Use Cases
• Why Neev as a QA Partner?
• Capabilities
I. Functional (Web / Mobile / Cloud)
II. Non-Functional Testing
III. Tools (Web / Mobile)
• Agile Testing Expertise
• Performance/load Testing
• Automated Testing
• Case Studies
• About Neev
• Summary

Why Performance Matters for Digital Businesses?
A few statistics about the business impact of performance at major internet
companies:
• Amazon and Walmart increased revenue 1% for every 100ms of improvement
• Microsoft found that Bing searches that were 2 seconds slower resulted in a 4.3% drop
in revenue per user
• When Mozilla shaved 2.2 seconds off their landing page, Firefox downloads increased
15.4% (60 million more downloads)
• Making Barack Obama’s website 60% faster increased donation conversions by 14% (30
million more dollars)
• Shopzilla sped up average page load time from 6 seconds to 1.2 seconds, and increased
revenue by 12% and page views by 25%
Fast Application
Performance
Positive / Hassle-
Free User
Experience
Positive Affect
towards the Brand
Improved Brand
Source: http://www.appdynamics.com/blog/devops/load-testing-tools-explained-the-server-side/
Equity

Where Neev can help? A few Scenarios
• Pre-Release Testing: Have a release coming up in a month? Would you like an
Independent Testing Team to test the application/product thoroughly and
come back with suggestions?
• Agile Test Partner: Looking for an independent testing partner that can
rigorously test your sprint increments, Sprint by Sprint, and maintain an
automated test suite at a test coverage of over 70%?
• Pre-Launch Multi-Platform Mobile App Testing: Looking for a Testing partner
who understands the nuances of the Mobile Ecosystem and get your Apps
store-ready?
• Long Term Testing Partner: Looking for a Testing partner that can help you get
your Regression Tests on the Cloud and run them on-demand?
• Pre-Release Scale Testing: Want to test if the application can scale as per your
expectations.
• Tool Centered Testing : Looking for a Testing partner specializing in tools.
Neev – A Partner for your Everyday & Mission-Critical Testing Needs

Why Neev as a QA Partner?
• Fast Go-to-market: Hundreds of start ups have worked with Neev to go live with
their version 1.0 of their product., eg., InMobi, Redbus, Apalya Myplex.tv
• Wide Expertise in Tech Ecosystems: Neev understands the Web, Mobile, Cloud
ecosystem in a way that few companies do.
• Process Orientation: Strong in agile, scrum processes – Sprint planning,
retrospectives, product backlog grooming, etc.
• Tools & Test Automation: Comprehensive experience in Test Automation and
Load Test Tools.
• Strong Partnerships – A strong ecosystem of partners – Amazon Web Services,
AppDynamics, Splunk, Sauce Labs.
• Strong Domain Expertise: Neev has expertise in handling Testing projects across
multiple domains - eCommerce, Mobility, Education, Healthcare, Social Media.
All Round Expertise with excellent Tools / Partners ecosystem

Regression
Neev Testing Capabilities
Tests
followed by
Sanity Testing
Automa
tion
Testing
Perform
ance
Testing

Functional Testing Capabilities at a Glance
Interface & Usability Testing
SDK Testing User Acceptance Testing
Manual Testing
System Testing
Installation Testing
Automation/Tool based Testing
Smoke/Sanity Testing
Integration Testing
Black Box and White box Testing
Unit Testing
Pre User Acceptance Testing
Automated & Regression Testing
Component Testing
Experience across diverse Testing needs.

Non-Functional Testing Capabilities at a Glance
Experience across diverse Testing needs.

Tools Expertise for Web Application Testing
QA Activity Tools Used
Test Management Visual Studio 2010, Fitnesse, RedMine, Test Link
Test Automation Tools (Open
Source)
Selenium RC, WebDriver, Watir, Watij, SauceLabs, Sahi, Canoo
Functional Testing Tools (Open
Source)
Fit, FitNesse, Cucumber, RSpec
Unit Testing Frameworks JUnit, NUnit, SpringUnit, GrailsUnit Test, PHPUnit
Issue Tracking BugZilla, RedMine, TRAC, JIRA
Code Management CVS, Git, SVN, VSS, TFS
Build Automation Continuum, Maven, Make, Ant, Cruise Control, Cruise Control.NET
Performance Test Tool Apache JMeter, WebLoad
Expertise on Best-in-class Testing tools

Tools Expertise for Mobile Application Testing
Test Management Redmine, testlink
Test Automation Tools Robotium, Appium, monkeyrunner
Unit Testing Frameworks JUnit, OCUnit
Issue Tracking BugZilla, JIRA, Trac
Code Management SVN, Git
Build Automation Hudson, Ant
Performance Test Tool Apache JMeter, LoadRunner, memory profiling
through xcode
Expertise in Best-in-class Testing tools

Key Challenges & Best Practices for Mobile Testing
Mobile Testing Best Practices distilled
Key
challenges
Device diversity
OS fragmentation
Network environment
Best practices followed
•Write unit test cases
•Early testing using emulators
•Create acceptance environment by prioritizing the target OS & devices
•Create automated test scripts for smoke & regression testing
•Use of paid services such as “deviceanywhere”
•Create continuous integration system
•Perform testing using various devices
•Perform testing using WiFi, 3G network
•Perform usability testing
•Perform security testing
•Perform random testing

Agile Testing Best Practices
• Test Driven Development : Use of Rspec, Cucumber, Selenium, Sauce Labs, Unit Test frameworks (jUnit,
TestNG, spork and watchr) to make the code more maintainable, robust and bug free.
• Continuous Integration:
– Use of Teamcity CI server to help prevent integration problems.
– Git for source control
• Code Metrics and Performance benchmarks:
– Measure quality of code to provide better insights into the code quality.
– Use of tools like rubocop, rails_best_practices, metric_fu,mini_profiler and brakeman to benchmark
the page loads and optimize them to the fullest.
• Agile Project Management Tools:
– Use of Agile project management tools like Redmine to track and manage our agile projects and
ensure delivery.
– Follows Scrum practices.
• UI Test Automation: Use of Selenium / Sauce based Test Automation frameworks to automate end user
testing
• Cloud best practices: AWS Consulting partner with a great understanding of cost drivers on the Cloud.
Agile is in our DNA

Agile is in our DNA
Agile Testing Process

Progressive Regression
Regression suite development cycle will always be one step behind the product
development cycle and it will ensure that existing product features are not
impacted by the current development.
Agile is in our DNA

A Typical QA Environment
Development QA Sandbox Pre-production
environment
Isolation of Test Environments and Well-defined Gates
Production
•Unit Testing Manual testing of
•Features
•Bug fixes
•Patches
•Hotfixes
Regression test with
•Existing configuration
•Existing data
Minimal Load Test
Regression
& Load Test
Regression test
•Features
•Components
•Functionalities
Detailed Load Test
Security Test

Test Management Mercury Test Director, Rational Test Manager, Visual
Studio 2010, Borland Caliber RM, Quality Center,
StarTeam.
Test Automation Tools (Std / Commercial) QTP, WinRunner, LoadRunner, Visual Studio Tests,
SilkTest, Rational Test
Test Automation Tools (open source) Selenium, Watir
Functional Testing Tools (open source) Fit, FitNesse, Cucumber
Unit Testing Frameworks JUnit, NUnit, SpringUnit, GrailsUnit Test, PHPUnit
Issue Tracking BugZilla, Rational ClearQuest, JIRA, Visual Studio TFS
Code Management CVS, PVCS, SVN, Rational ClearCase, VSS, MS TFS, Git,
Perforce
Build Automation Continuum, Maven, Make, Ant, Cruise Control, Nant,
Cruise Control.NET, Visual Studio TFS
Performance Test Tool Apache JMeter, LoadRunner, Visual Studio Load
Testing, WebLoad, SilkPerformer, OpenSTA
Tools / Technologies

Use cases for Performance / Load Testing
• Pre-go-live validation of application’s ability to meet
performance SLAs
• Benchmark maximum operating capacity (Throughput,
concurrent connections, average response time, etc.,) of the
system under both steady-state and peak load conditions.
• Identify performance bottlenecks on application, database
and infrastructure that could potentially cause performance
degradations.
• Sizing & planning the production infrastructure needed for
the system’s optimal performance.
• Measuring Scaling Limits and Potential Break Points of the
system
• Continuous Improvement and system maintenance.

Load Test Design Considerations
• Type of Application to be load tested – The load test design may vary depending on the
type of application. For example, an application can be any of the following types:
internet –based consumer applications, Transactional / workflow based systems, CMS
systems, Batch Processing Applications etc., Each of these types of application will
require a different approach and thought process when designing the load test.
• Typical and Peak Load Expectations: Understanding the expected load traffic under both
typical as well as peak load conditions is a key design consideration for load tests. This
enables the team to select the right load generator tools for the given application under
test.
• Baselined Performance Metrics : It is important to lay down clearly what the key
performance metrics to be tracked, monitored and base-lined are. This enables the team
to select the right monitoring tools for the right data.

Load Test Design Considerations
• Technical and Deployment Architectures: The production architecture of the system
under test is a key design consideration. For example various architectural elements such
as server clustering, 2 / 3 tier architecture, Use of CDNs, Database replication / sharing
set up, etc., will introduce different considerations when designing the load test.
• Simulating Geographically distributed user load : Real users may be geographically
distributed across different continents all over the world. It is important for the load test
to mimic the real world user traffic as closely as possible by generating a geographically
distributed load.

Tools & Platforms
1. Apache Jmeter
2. Amazon Web Services
3. Apache Benchmark (ab)
4. 3rd Party Services (Saucelabs, Blazemeter)
5. APM Solutions (New Relic, AppDynamics)
6. Custom scripts for capturing custom metrics
Tools and Technologies

Methodology and Approach
The above time estimates are based on a typical experience. This may vary
depending on the exact nature of application involved.

Deliverables
1. Load Test Plan
• Load test objectives, metrics to be captured and test procedures to be executed
• Test Infrastructure Plan
2. Load Test Scripts
• Recorded and validated load test scripts
3. Load Test Report and Recommendations
• Performance benchmark metrics of the application under test
• Recommended Infrastructure size for production
• Projected Cost of infrastructure and Cost-to-Performance ratio.

Automated Testing using Selenium
and other Innovative Frameworks

Selenium Automation Experience
• Actively deployed Selenium as automation tool of choice across 75% of our
EDCs.(Extended Development Centers)
• Strong Java/J2EE based test automation experience.
• Developed our own Selenium-based automation framework using Selenium,
TestNG, jUnit, ANT integrating all best practices.
• Java Developer resources assigned to Selenium development with experience
in OOAD, Agile development, etc.
• Integrated Selenium into:
• Continuous Integration servers
• Build, QA and Release processes.
• Daily email alerts and notification system.
We have developed our own Selenium-based automation framework

Selenium + TestNG Based Automation - Key Highlights
• Modularized Test Suites and Test
Scripts with automated test bed
set up.
• Backed by Selenium RC Server
for Test scripting in standard
languages like Java, Python,
Ruby.
• Integrated with TestNG, Junit for
easy grouping of test suites and
test scripts.
• Integration for automated build
of the entire Test Automation
Project.
• Automated generation and
mailing of the HTML reports.
Automation with Selenium + TestNG
Release
Candidate

Innovative Testing Framework using Fitnesse, Xebium & Monkeytalk
• The idea was to create an environment where a Business
Analyst can understand what a Test Engineer is testing and so
can the customer. To create an environment where customers
or Business Analysts should be able to create their own
acceptance test cases with little or no help from the test
engineer.
• We found out that most of the components were already in
place and just needed a glue to fit them together. Fitnesse,
Xebium and Monkeytalk were what we bumped into.
• More information in our blog here.
Automation with Fitnesse, Xebium & Monkeytalk
Release
Candidate

Case Study – AppDynamics EDC
An Application Performance Management Product
• AppDynamics is an application performance measurement tool,
monitoring JVM's(Java) and CLRs(.NET) in production.
• Supports applications across a wide range of technologies and platforms.
• The solution helps to reduce mean-time-to-resolution by up to 90%.
• Started in 2008 in California, AppDynamics is a funded start-up.

AppDynamics EDC - The Customer Need
• The Product's one of the many USP's is the very low
overhead(2%). Measurement of the performance of any
new build is immensely important.
Performance Testing
• The product has to support applications on a
combination of technologies. Each and every
functionality need to be tested before the release.
Functional Testing
• A large pool of test cases has been created till date. An
automation framework has to be developed that will
incorporate testing on these test cases.
Automation Framework
Development
• To gain confidence of the customer, a proof of concept
demonstrating the capabilities of the product has to be
created.
Proof of Concept
Applications

AppDynamics EDC - The Solution … (1)
overhead it has(2%). Measurement of the performance
of any new build is immensely important.
Performance Testing
Functional Testing
Development
created.
Proof of Concept
Applications
Neev created an automation
framework that is used to obtain the
overhead added by agents running
on JVM’s and CLR’s.

AppDynamics EDC -The Solution …. (2)
Performance Testing
Functional Testing
Neev created test applications using a
combination of technologies. These applications
are employed on a daily basis to perform a
rigorous test on the product features. For
instance, an application to test Java Messaging
Service, which would run on Jboss with Cassandra
as back-end
Development
created.
Proof of Concept
Applications

AppDynamics EDC - The Solution …. (3)
A section of Neev team is focused on
building a framework, which would
facilitate the automation of test cases.
This brought down the sanity &
regression testing period significantly.
Performance Testing
Functional Testing
Development
created.
Proof of Concept
Applications

AppDynamics EDC - The Solution …. (4)
Performance Testing
Neev develops an application specific to a
potential customer. The idea is to simulate
the production environment of the
customer, and then demonstrate the
product's prowess and capabilities in that
environment.
Functional Testing
Development
created.
Proof of Concept
Applications

AppDynamics EDC - A typical day for the Neev team
Regression Tests
followed by Sanity
Testing
Automation
Testing
Performance
Testing
POCs as
& when
required

AppDynamics EDC - Our journey so far …

Case Study – eMids
Selenium Automation Intervention
 eMids was looking for QA partner to automate Regression Tests for an application which they
were developing for a reputed health-care organization.
 Constraints – Cost, Improve Coverage, Reduce turnaround time, Predictable build quality,
Scalable framework, Reliable regression test runs.
 Neev deployed its Selenium framework with Java / TestNG for automation which was
completely Open Source compared to other solutions which were License based.
 Neev covered all the 7 modules of the application by automating more than 300 test cases.
 In the process of automation Neev uncovered more than 30 new bugs.
 The process improved the turn around time of QA and in return improved the productivity by
60%.
No. Of Modules No. of Test Cases Time to
Automate
Duration of Automated
Test Run
7 300+ 4 weeks 50 mins

Offshore Onsite
Project
Manager
QA Lead
QA 3
(Selenium)
QA 4
(Selenium)
QA 1
(Selenium)
QA 2
(Selenium)
Project
Sponsor
(Client)
Onsite
Project
Coordinator
eMids EDC Team Structure

Case Study – eMids
Key Challenges
• Short Term Intervention Program
 Focused Short Term Regression Test Automation intervention program.
 Application was in production. No major feature changes were expected. Customer wanted to reduce
the periodic regression test cycle time by at least 50%.
 7 key modules were identified. Close to 300 test cases were automated within a short period of time.
• No documentation
 Detailed Documentation of the application feature description was not available.
 For the 7 modules identified as candidates for automation, the team performed a thorough analysis
and documented all key test cases. The test cases and scenarios were signed off by the customer.
• Zero Prior Automation
 As there were no automated cases, the team started by developing an Automation Framework using
Selenium and jUnit, that can be flexible, extensible and maintainable.
 Test Cases were organized into a comprehensive suite which can be triggered using an ANT target
during build time. Thus, every time a new build was generated, the entire automation suite was
executed and the results were recorded and analyzed.

Cloud-based Mobile Testing for Tripflick
Solution Highlights
• 'TripFlick' is a Singapore-based tourist
information services company aspiring to
become a global leader in the information
services market for the tourism industry.
• Perfecto Mobile is a global provider of cloud-based
testing, automation and monitoring
solutions for mobile applications and websites
utilizing a wide selection of REAL and emulated
mobile devices.
• The mobile market is extremely fragmented
and dynamic. At any point in time, there are
dozens of devices which are relevant to the
target market. Thus, we had to test on a
number of devices.
• Neev utilized services of Perfecto Mobile for its
client 'TripFlick' to ensure the quality of their
mobile app before launching.
• Neev integrated services of Perfecto mobile for
'TripFlick' and tested Tripflick app in 30+
Devices.

Load Testing Success Stories – KEH
About the Client: Founded in the year 1979 in Atlanta, USA, KEH is the world’s largest dealer of used
photographic (digital and film) products. KEH lets anyone in any corner of the world sell or trade equipment with
a global customer base.
Approach: The staging environment served as the test bed for all performance tests conducted in this project.
JMeter was used to generate the load required. 1 Amazon EC2 small instance in US Zone acted as the JMeter
controller. 5 Amazon EC2 small instances in US Zone acted as JMeter agents. Both the Application Server and the
Database server were configured with monitoring systems (Monyog, Manage Engine - App Manager) that
captured CPU, Memory, Disk usage, Number of connections etc.
Insights:
• Even at high load of 37500 users, the application home page responded at an average time of 20ms.
• Based on the average response time of 20 ms, we predict that the home page can sustain a load of up to
50,000 users ramping up in 1000 seconds. i.e, up to 50 concurrent users.
• There is always a clean throughput. Meaning requests do not queue up at the server end.
• The number of open DB connections remain untouched. Meaning, the database Is not impacted by the home
page hits. All home page requests are served from the server cache without accessing the database.
Recommendations Provided:
• Presently an order is at least 10 clicks away for all users. We can look at optimizing the order submit flow.
• Review the IIS connections setting and the Database connections setting to tune it according to expected user
traffic.
• Consider setting Up sufficient monitoring during application failure / ISS server failure.

About Neev
Magento
Hybris Commerce
SaaS Applications
Adobe Marketing Cloud
Custom Development
Key Company Highlights
300+ team with experience
in managing offshore,
distributed development.
Neev Technologies
established in Jan ’05
VC Funding in 2009 By
Basil Partners
Part of Publicis Groupe
Hybris and Adobe CQ
centers of Excellence
Offices at Bangalore,
Gurgaon, Pune, Mumbai
Member of NASSCOM
Mobile Cloud
iPhone
Android
PhoneGap
Windows Phone
HTML5 Apps
Web
AWS
Rackspace
Joyent
Heroku
Google Cloud Platform
Digital Marketing, CRM, Analytics (Omni-Channel)
User Interface Design and User Experience Design
Performance Consulting Practices
Quality Assurance & Testing
Outsourced Product Development
Click here to know more about us

Project Management Tools
Redmine
Jira/Trac
Deployment Tools
Jenkins
Emma/Cobertura
QA Tools
Selenium
Jmock
Knowledge Capture Tools
Neev Hub
QA to Staging to Production
environments
Daily Stand-ups
Neev Project Methodology

The Neev Edge
• End-to-end consultative approach for software solutions through needs assessment,
process consulting and strategic advice.
• Internal QMS are ISO 9001-2008 certified and CMM level 3 compliant.
• Continuous process and service level improvements through deployment of best-of-breed
processes and technologies.
• International Standards and best practices on Project Management including PMI, ISO
and Prince-2.
• Proven EDC Model of delivery to provide predictable results.
• Scrum based Agile development methodology.

Neev Information Technologies Pvt. Ltd. sales@neevtech.com
India - Bangalore
The Estate, # 121,6th Floor,
Dickenson Road
Bangalore-560042
Phone :+91 80 25594416
India - Pune
Office No. 4 & 5, 2nd floor, L-Square,
Plot No. 8, Sanghvi Nagar, Aundh,
Pune - 411007.
Phone :+91 20 64103338
For more info on our offerings, visit www.neevtech.com

Neev QA Offering

More Related Content

What's hot

Similar to Neev QA Offering

More from Neev Technologies

Recently uploaded

Neev QA Offering