1. Big Data
Storage POC
ASHA CAMPER SINGH
SOFTWARE ENGINEER INTERN - SUMMER 2015

2. GOAL
Store HBase Dedupe HBase Write SequenceFile

3. GOAL 1: STORE
CSV
POJO
HBase

4. GOAL 1: STORE
CSV
POJO
HBase
TEST_RESULT_KEY
STATUS
CLUSTER_KEY
KNOWN_ISSUE_KEY
BASELINE_OUTCOME
UPGRADED_OUTCOME
BASELINE_MESSAGE
UPGRADED_MESSAGE
IS_DATA_SILO
IS_COMPILE_FAILURE
NAMESPACE
......

5. GOAL 1: STORE
CSV
POJO
HBase
Entity
“TEST_RESULT_KEY” à “foo”
“STATUS” à “foo”
“CLUSTER_KEY” à “foo”
“KNOWN_ISSUE_KEY” à “foo”
“BASELINE_OUTCOME” à “foo”
“UPGRADED_OUTCOME” à “foo”
“BASELINE_MESSAGE” à “foo”
“UPGRADED_MESSAGE” à “foo”
“IS_DATA_SILO” à “foo”
“IS_COMPILE_FAILURE” à “foo”
“NAMESPACE” à “foo”
...... à .....
UPSERT INTO
TEST_RESULT ...

6. GOAL 1: STORE
CSV
Map Reduce
/usr/asha/CSVs
Map
/usr/asha/CSVs/1.csv
/usr/asha/CSVs/2.csv
/usr/asha/CSVs/3.csv
/usr/asha/CSVs/4.csv
/usr/asha/CSVs/5.csv
………
/usr/…/1.csv 34223
/usr/…/2.csv 98702
/usr/…/3.csv 982
/usr/…/4.csv 12570
/usr/…/5.csv 8787
………
Reduce
Input.txt Output.txt
absolutePath()
Entity
UPSERT INTO
TEST_RESULT ...

7. GOAL 2: DEDUPE
F-P F-F
Perm
Regex
“NOT_A_BUG” “DUPLICATE_OF
_BASELINE”
“DUPLICATE
_PERM_REGEX”

8. GOAL 2: DEDUPE F-P
RESULT
_KEY
ANALYSIS CORE_INFO …
STATUS …
BASELINE_
OUTCOME
UPGRADED_
OUTCOME
… …
foo1 TEST TEST
foo2 COMPILE
foo3 TEST
foo4

9. GOAL 2: DEDUPE F-P
RESULT
_KEY
ANALYSIS CORE_INFO …
STATUS …
BASELINE_
OUTCOME
UPGRADED_
OUTCOME
… …
foo1 TEST TEST
foo2 NOT_A_BUG COMPILE
foo3 NOT_A_BUG TEST
foo4

10. GOAL 2: DEDUPE F-P
UPSERT INTO TEST_RESULT (RESULT_KEY, STATUS)
SELECT RESULT_KEY, ‘NOT_A_BUG’
FROM TEST_RESULT
WHERE (BASELINE_OUTCOME = ‘TEST’ OR
BASELINE_OUTCOME = ‘COMPILE’) AND
UPGRADED_OUTCOME IS NULL

11. GOAL 2: DEDUPE F-F
RESULT
_KEY
ANALYSIS CORE_INFO …
STATUS …
BASELINE_
OUTCOME
UPGRADED_
OUTCOME
BASELINE_
MESSAGE
UPGRADED_
MESSAGE
…
foo1 TEST TEST ABC ABC
foo2 COMPILE JKL
foo3 TEST ABC
foo4 TEST TEST ABC XYZ

12. GOAL 2: DEDUPE F-F
RESULT
_KEY
ANALYSIS CORE_INFO …
STATUS …
BASELINE_
OUTCOME
UPGRADED_
OUTCOME
BASELINE_
MESSAGE
UPGRADED_
MESSAGE
…
foo1
DUP_OF_
BASELINE
TEST TEST ABC ABC
foo2 COMPILE JKL
foo3 TEST ABC
foo4 TEST TEST ABC XYZ

13. GOAL 2: DEDUPE F-F
UPSERT INTO TEST_RESULT (RESULT_KEY, STATUS)
SELECT RESULT_KEY, ‘DUPLICATE_OF_BASELINE’
FROM TEST_RESULT
WHERE (BASELINE_OUTCOME = ‘TEST’ AND
UPGRADED_OUTCOME = ‘TEST’) AND
BASELINE_MESSAGE = UPGRADED_MESSAGE
* TODO: implement pattern matching between the messages
instead of only strict equality

14. GOAL 2: DEDUPE
Perm
Regex
base_regex1, up_regex1
base_regex2, up_regex2
base_regex3, up_regex3
base_regex4, up_regex4
base_regex5, up_regex5
base_regex6, up_regex6
………
<base_regex1,
up_regex1>
<base_regex2,
up_regex2>
<base_regex3,
up_regex3> …
KNOWN_ISSUE__C
SCOPE = ‘PERMANENT’

15. GOAL 2: DEDUPE
Perm
Regex
<base_regex1,
up_regex1>
<base_regex2,
up_regex2>
<base_regex3,
up_regex3>
<base_regex4,
up_regex4>
<base_regex4,
up_regex4> …

16. GOAL 2: DEDUPE
Perm
Regex
<base_regex1,
up_regex1> … Column Filter
<base_regex1>
Column Filter
<up_regex1>
Scanner

17. GOAL 2: DEDUPE
Perm
Regex
Scanner
RESULT
_KEY
ANALYSIS CORE_INFO …
STATUS …
BASELINE_
OUTCOME
UPGRADED_
OUTCOME
BASELINE_
MESSAGE
UPGRADED_
MESSAGE
…
foo1 TEST TEST ABC ABC
foo2 COMPILE JKL
foo3 TEST ABC
foo4 TEST TEST ABC XYZ
……
Results STATUS = ‘DUPLICATE_PERM_REGEX’

18. ANALYSIS
Psuedo-distributed mode
~1,000,000 entries

19. ANALYSIS
Psuedo-distributed mode
~1,000,000 entries
HAMMER
Total (EU5) 8-14 hours
THOR/SCREWDRIVER/JAGHAMMER/
VOLDEMORT/GROND/SHARUR
Total < 1 hour

20. ANALYSIS
THOR/SCREWDRIVER/JAGHAMMER/VOLDEMORT/
GROND/SHARUR
Type of Deduping Time (seconds)
F - P 26.678
F - F 26.742
PermRegex 2369.15
Psuedo-distributed mode
~1,000,000 entries

21. GOAL 3: WRITE
HBase Write SequenceFile
NOT_A_BUG ✗
DUPLICATE_OF_BASELINE ✗
DUPLICATE_PERM_REGEX ✗
IS_COMPILE_FAILURE ✗
IS_DATA_SILO ✔
<RESULT_KEY, UPGRADED_MESSAGE>

22. GOAL 3: WRITE
NOT_A_BUG ✗
DUPLICATE_OF_BASELINE ✗
DUPLICATE_PERM_REGEX ✗
IS_COMPILE_FAILURE ✗
IS_DATA_SILO ✔
SELECT * FROM TEST_RESULT WHERE
Results

23. GOAL 3: WRITE
Results
<RESULT_KEY, UPGRADED_MESSAGE>
write
SequenceFile

24. Store HBase Dedupe HBase Write SequenceFile
✔ ✔✔
Thank you