This document discusses how big data and tools like Hadoop can help testing teams. It defines big data as large, complex datasets that are difficult to process using traditional tools. Hadoop uses HDFS for storage and MapReduce for processing. A example is given of using MapReduce to count words in a file. The document then describes a problem of testing a stock exchange adapter where the daily data was 150GB. It proposes using Hadoop to standardize and compare output files to test the adapter's performance. Finally, it lists other ways testing teams could use big data tools like for repeated tests, customer feedback analysis, and adding intelligence to test data.

1. Confidential | Copyright © QAAgility Technologies
Big Data - Hadoop and
MapReduce - new age
tools for aid to testing
and QA
by Aditya Garg

2. Big Data - Hadoop and
MapReduce - new age tools
for aid to testing and QA
Topic for the presentation

3. What is this
Confidential | Copyright © QA Agility Technologies

4. 1. How to test Big Data
applications ?
2. How can QA and Testing
team use Big Data tools
for their testing needs ?
What are we going to discuss ?

5. 1. How to test Big Data
applications ?
2. How can QA and Testing
team use Big Data tools
for their testing needs ?
What are we going to discuss ?

6. Confidential | Copyright © QA Agility Technologies
What is Big Data ?
Is it just too much Hype or
reality ?

7. Here is latest one from yesterday on #Bigdata

8. Confidential | Copyright © QA Agility Technologies
Let us start with what
exactly is BigData

9. Which Search Engine do you use ?
https://www.cirrusinsight.com/blog/how-much-data-does-google-store
http://searchstorage.techtarget.com/definition
/Kilo-mega-giga-tera-peta-and-all-that
How much data does Google store ?

11. Key Points in Big Data
1.Volume – Data Explosion
2.Velocity
3.Variety
4.Veracity

12. Ref: IBM.com
Key Points in Big Data

13. Definition
Big datais the term for a collection
of data sets so large and complex
that it becomes difficult to
process using on-hand database
management tools or traditional
data processing applications. The
challenges include capture,
curation, storage, search,
sharing, transfer, analysis, and
visualization.
http://www.forbes.com/sites/gilpress/2014/09/03/12-big-data-definitions-whats-
yours/#379879e621a9
Ref: goo.gl/iWZhjJ

14. Big Data Application
1. Finance
2. Insurance
3. Health Care
4. Agriculture
5. Defense
6. Manufacturing
7. Aero Space
8. Oil and Gas
9. Advertisement and Marketing
10.Election Campaigns
11. List goes on --- applicability across industries

15. http://snip.ly/UKNB#http://bit.ly/1OF5nhF

16. Big Data Application
http://www.forbes.com/sites/bernardmarr/2016/02/03/how-the-super-bowl-uses-big-data-to-
change-the-game/?

17. Big Data Application
http://andrewshamlet.com/2015/12/03/who-will-win-the-2016-us-presidential-nominations/

18. Lets go back to definition
Big datais the term for a collection
of data sets so large and complex that
it becomes difficult to process using
on-hand database management
tools or traditional data processing
applications. The challenges include
capture, curation, storage, search,
sharing, transfer, analysis, and
visualization.

19. Confidential | Copyright © QA Agility Technologies
Tools solving Big Data
Challenge

20. Tool solving the Big Data Challenge

21. *Source Udacity
Hadoop – Key components HDFS and MR

22. *Source Udacity
1. Sqoop takes data from
regular RDBMS and
puts it into HDFS
2. Flume ingests data
into HDFS as it is
generated by external
systems
3. HBASE is real time
database on top of
HDFS
4. Hue is a graphical
front end to the
cluster
5. Oozie is workflow
management tool
6. Mahout is Machine
Learning library
Hadoop Ecosystem

23. HDFS
• HDFS stands for Hadoop Distributed File
System, which is the storage system used
by Hadoop. The following is a high-level
architecture that explains how HDFS
works.

24. Map Reduce
Ref: Emanuele Della Valle
@manudellavalle

25. Confidential | Copyright © QA Agility Technologies
Understanding MapReduce
Demo – Word Count
Given an input file, count
unique words

26. WordCount – Map Reduce
Reference : http://wearecloud.cz/media/files/prezentace-biz/Big%20Data%20v%20Cloudu.ppt

27. Confidential | Copyright © QA Agility Technologies
How can QA and Testing
team use Big Data tools
for their testing needs ?

28. Confidential | Copyright © QA Agility Technologies
Problem Statement and
Solution using Hadoop
and MapReduce

29. MTBT – Multicast Tick by Tick Adapter
Input was exchange feed – Output given to HFT Engine
Exchange TAP
– Co-location
servers listen
to it at high
speed
Legacy Adaptor (3rd Party)
connects to the TAP – and
converts to a format which
can be used by HFT
Platforms (Algorithmic
Trading Platforms)
New Adaptor – being made
Inhouse – to increase the
speed by 10 Times
HFT
Engine
MTBT - Adaptor

30. MTBT – Multicast Tick by Tick Adapter
•Client was trying to build a brand new MTBT
Exchange Adaptor
•The adaptor was being developed in C and Unix and
was to run in a co-location with NSE (National Stock
Exchange)
•The new adaptor was supposed to increase the
overall speed by more than 10 times from the existing
adaptor
•The Goal was to test the new adaptor

31. Input Output
Output over time
MTBT - Adaptor
S
a
m
p
l
e
S
a
m
p
l
e
S
a
m
p
l
e
S
a
m
p
l
e
S
a
m
p
l
e
Do A Reverse
Comparison
MTBT – Testing Strategy - Sampling

32. Input Output
Output over time
MTBT - Adaptor Challenges
--------------------------------------------------
1. Manually next to impossible
2. Even few seconds samples were
running into large MegaBytes (MB)
files
3. Manually impossible to compare
the legacy records with the New
code processed records
4. Daily processed data ran into 150
Giga Bytes (GB) plus files
MTBT – Challenges

33. Input Output
Output over time
MTBT - Adaptor BIG DATA Problem
--------------------------------------------------
1. LARGE 150 GB files (legacy and New
applications) – VOLUME
2. Testing to compare the output and
measure the functional
effectiveness in real time data
environment – VELOCITY
3. Packet drops may happen –
(VERACITY)
4. Variety was not there – except the
format of the output file generated
was not in similar format – the
content/information was there
MTBT – It was a BIG DATA Testing
problem

34. MTBT – SOLUTION
1 Reduce LEGACY MTBT - Output file into a standard format
2 Reduce NEW INHOUSE MTBT output file into a standard format
3 Compare the two files
4 Generate Report

35. Confidential | Copyright © QA Agility Technologies
QA team can use the tools in multiple scenarios
1. Beta Testing
2. Repeated execution effectiveness –
applying analytics ( R)
3. Capturing Customer feedback and
channeling the same for smarter test
execution
4. Extracting relevant information from
repeated regression cycles from QC
5. Adding intelligence on the data generated
by the testing team
Other scenarios – Big Data Tool
implementation

36. Thank you and Jai Hind
Questions ?
@adigIndia
@AgileTA
#GTR2016

37. Contact
Please contact us at info@QAAgility.com
Confidential | Copyright © QAAgility Technologies
MUMBAI
711, Rupa Solitaire
MBP,Mahape
Navi Mumbai-400701
DENMARK
1Lindebo 7 Lej -42,
2630Tasstrup, Copenhagen
+45.7164.0278
denmark@qaagility.com
USA
200E Campus ViewBlvd.
Suite200,Columbus, OH