The document discusses testing challenges with big data and proposes the test pyramid approach. It provides an overview of big data concepts like Hadoop, HDFS, MapReduce and HBase. Testing big data is challenging due to long feedback cycles and data dependencies. The test pyramid advocates automating basic tests like unit tests while using techniques like sampling and test automation to counter long test cycles. It also provides a high level example of using a Hive query validator as part of the testing approach.

1. TESTING BIG DATA
Adoniram Mishra
Benazeer Khan

2. Agenda
❖ What is Big Data?
❖ Hadoop Ecosystem
❖ Testing Challenges
❖ Test Pyramid

3. What is Big Data?

5. What is Hadoop

6. Hadoop Ecosystem

7. HDFS

8. MapReduce

9. High Level Example
Data
Source
HBase HiveData
Source
Data
Source
I
M
P
O
R
T HQL
UI
Reports
Other
applications
M
R
M
R
M
R

10. Testing Challenges

11. Long Feedback Cycle

12. Data

13. Deadlines

14. Automation

15. Countering Challenges
❖ Sampling.
❖ Basic testing concepts - Boundary Value testing, Equivalence class
partitioning.
❖ Automation

16. Nightly Build
Tests
Hive Query
Validator
Unit Tests
UI Test
Test Pyramid

17. Hive Query Validator
Hive Query Validator
Query 1
Query 2
Query 3
Correct
Correct
Incorrect

18. Nightly tests & Smoke Tests
SMOKE
MODULE 3MODULE 1
MODULE 2
MODULE 4

20. About Us
● Adoniram Mishra (adoniram@thoughtworks.com)
● Benazeer Khan (benak@thoughtworks.com)
Follow us at:
Facebook: https://www.facebook.com/groups/vodqa/
Twitter: https://twitter.com/vodqancr