Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[2A3]Big Data Launching Episodes

2,927 views

Published on

DEVIEW 2014 [2A3]Big Data Launching Episodes

Published in: Technology
  • Be the first to comment

[2A3]Big Data Launching Episodes

  1. 1. Big Data Launching Episodes 안성화 Manager / Data Tech Lab SK Telecom
  2. 2. CONTENTS 1. Accessibility 2. Expansion 3. Lessens Learned 4. Future
  3. 3. 1. Accessibility 10 GB/Hour 100 MB/Hour GroupBy & Sum SKT 최초의 Hadoop 시스템
  4. 4. Accessibility MapReduce에서 Hive로 1. Group By & Sum ! ! ! ! 2. UDF & UDAF Map ! Group By Key별 수집 Reduce ! Group By Key별 Sum Map ! UDF Reduce ! UDAF Select key1, sum(key1) From Table Group by key1; Select udf(key1) From Table; Select key1, udaf(key1) From Table Group by key1;
  5. 5. Accessibility MapReduce에서 Hive로 3. Transform ! Map ! Transform Reduce ! Transform FROM ( FROM records2 MAP year, temperature, quality USING 'is_good_quality.py' AS year, temperature) map_output REDUCE year, temperature USING 'max_temperature_reduce.py' AS year, temperature; Hadoop Definitive Guide
  6. 6. Accessibility MapReduce에서 Hive로 4. GUI (Hue) http://gethue.com/wp-content/uploads/2014/03/hue-3.6.png
  7. 7. Accessibility insert overwrite table tmp_daily_recommendation partition(silo) select x.probe_mgmt_string, x.marang_made_spring, x.marang_made_nm, x.bf_m2_series, x.chg_dev_silo, x.prob_simple, x.prob_simple_ko, x.prob_outs, x.prob_outs_ko, x.choosen, x.silo from ( select a.probe_mgmt_string, e.marang_made_spring, e.marang_made_nm, e.bf_m2_series, e.chg_dev_silo, a.mouse_prob_simple as prob_simple, a.mouse_rank_simple as prob_simple_ko, a.mouse_prob_outs as prob_outs, a.mouse_rank_outs as prob_outs_ko, 'chg' as choosen, trim(' 20140927 ') as silo from (select distinct probe_mgmt_string from tmp_old_report) d right outer join (select probe_mgmt_string, mouse_prob_simple, mouse_prob_outs, mouse_rank_simple, mouse_rank_outs from tmp_comp_post_prob_new where silo = trim(' 20140927 ') and pado_spring in ('1','2') and testman_sample_spring >= '021' and months >= 15 and event_type = 'event1' and mouse_rank_simple <= mouse_rank_outs) a on a.probe_mgmt_string = d.probe_mgmt_string join (select s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm, round(avg(s.bf_m2_series),0) as bf_m2_series, max(case when s.prev_intro_chg_silo like '#%' then s.probe_test_silo else s.prev_intro_chg_silo end) as chg_dev_silo from (select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst where silo = trim(' 20140927 ') union all select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst_enc where silo = trim(' 2014-09-27 ')) s where s.pado_spring in ('1','2') and s.probe_st_spring = 'song' and s.testman_sample_spring >= '21' and (s.bf_m2_series + s.bf_m3_series + s.bf_m4_series) >= 60000 group by s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm) e on a.probe_mgmt_string = e.probe_mgmt_string left outer join (select probe_mgmt_string, min(chg_silo) as chg_silo from master_mers_evthist where chg_silo between date_add(concat(substr(trim(' 20140927 '),1,4),'-',substr(trim(' 20140927 '),5,2),'-',substr(trim(' 20140927 '),7,2)),-60) and trim(' 20140927 ') and ((probe_chg_spring in ('1','2') and probe_chg_result_spring in ('11', '12')) or probe_chg_spring in ('1','2')) group by probe_mgmt_string) c on a.probe_mgmt_string = c.probe_mgmt_string where c.probe_mgmt_string is null and d.probe_mgmt_string is null order by prob_simple_ko asc limit 10000 union all select a.probe_mgmt_string, e.marang_made_spring, e.marang_made_nm, e.bf_m2_series, e.chg_dev_silo, a.mouse_prob_simple as prob_simple, a.mouse_rank_simple as prob_simple_ko, a.mouse_prob_outs as prob_outs, a.mouse_rank_outs as prob_outs_ko, 'out' as choosen, trim(' 20140927 ') as silo from (select distinct probe_mgmt_string from tmp_old_report) d right outer join (select probe_mgmt_string, mouse_prob_simple, mouse_prob_outs, mouse_rank_simple, mouse_rank_outs from tmp_comp_post_prob_new where silo = trim(' 20140927 ') and pado_spring in ('1','2') and testman_sample_spring >= '021' and months >= 15 and event_type = 'event1' and mouse_rank_simple > mouse_rank_outs) a on a.probe_mgmt_string = d.probe_mgmt_string join (select s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm, round(avg(s.bf_m2_series),0) as bf_m2_series, max(case when s.prev_intro_chg_silo like '#%' then s.probe_test_silo else s.prev_intro_chg_silo end) as chg_dev_silo from (select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst where silo = trim(' 20140927 ') union all select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst_enc where silo = trim(' 2014-09-27 ')) s where s.pado_spring in ('1','2') and s.probe_st_spring = 'song' and s.testman_sample_spring >= '021' and (s.bf_m2_series + s.bf_m3_series + s.bf_m4_series) >= 60000 group by s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm) e on a.probe_mgmt_string = e.probe_mgmt_string left outer join (select probe_mgmt_string, min(chg_silo) as chg_silo from master_mers_evthist where chg_silo between date_add(concat(substr(trim(' 20140927 '),1,4),'-',substr(trim(' 20140927 '), 5,2),'-',substr(trim(' 20140927 '),7,2)),-60) and trim(' 20140927 ') and ((probe_chg_spring in ('1','2') and probe_chg_result_spring in ('11','12')) or probe_chg_spring in ('31','32')) group by probe_mgmt_string) c on a.probe_mgmt_string = c.probe_mgmt_string where c.probe_mgmt_string is null and d.probe_mgmt_string is null order by prob_outs_ko asc limit 10000) x
  8. 8. Accessibility 6491.4 3200
  9. 9. Accessibility Job ID … Map % Map Total Maps Completed Job_1 12% 120,000 12,000 Job_2 0% 512 0 … … … … …
  10. 10. Accessibility Fair Scheduler로 Queue별 Quota 설정 ! ! tom ! jerry ! default 30% 30% 20% 40% 40% Fair Share Over Fair Share 0% 50% 100% Load 1! (Transform) 특정 Queue만 사용할 경우 다소 억제 다수의 Queue가 동시에 사용될 경우 여전히 문제 독점 사용 문제 해결 set mapred.job.queue.name=tom;
  11. 11. Accessibility code value SELECT code, sum(value) a | 1 a | 38 a | 45 b | 9 a | 34 a | 12 a | 78 FROM Table GROUP BY code; Mapper Mapper Mapper a! Reducer b! Reducer 왜 99%에서 안 끝나죠?!!!
  12. 12. Accessibility select /*+ MAPJOIN(b) */ count(*) from tableA a join tableB b on (a.id = b.id); 원래보다 너무 느려요!!!
  13. 13. Accessibility hadoop fs -­‐text xxx.snappy > xxx.gzip hadoop fs -­‐put xxx.gzip / fasdjlkfjlak UnSplittable!! sjdfljasdfjl Only 1 Mapper!! kasdjfljau82 n381qslfj832 9ruqw9ufoiau 8qwue899288u q98r912ioquq
  14. 14. Accessibility http://www.bbc.co.uk/bitesize/ks3/maths/shape_space/2d_shapes/revision/3/
  15. 15. Accessibility insert overwrite table tmp_daily_recommendation partition(silo) select x.probe_mgmt_string, x.marang_made_spring, x.marang_made_nm, x.bf_m2_series, x.chg_dev_silo, x.prob_simple, x.prob_simple_ko, x.prob_outs, x.prob_outs_ko, x.choosen, x.silo from ( select a.probe_mgmt_string, e.marang_made_spring, e.marang_made_nm, e.bf_m2_series, e.chg_dev_silo, a.mouse_prob_simple as prob_simple, a.mouse_rank_simple as prob_simple_ko, a.mouse_prob_outs as prob_outs, a.mouse_rank_outs as prob_outs_ko, 'chg' as choosen, trim(' 20140927 ') as silo from (select distinct probe_mgmt_string from tmp_old_report) d right outer join (select probe_mgmt_string, mouse_prob_simple, mouse_prob_outs, mouse_rank_simple, mouse_rank_outs from tmp_comp_post_prob_new where silo = trim(' 20140927 ') and pado_spring in ('1','2') and testman_sample_spring >= '021' and months >= 15 and event_type = 'event1' and mouse_rank_simple <= mouse_rank_outs) a on a.probe_mgmt_string = d.probe_mgmt_string join (select s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm, round(avg(s.bf_m2_series),0) as bf_m2_series, max(case when s.prev_intro_chg_silo like '#%' then s.probe_test_silo else s.prev_intro_chg_silo end) as chg_dev_silo from (select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst where silo = trim(' 20140927 ') union all select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst_enc where silo = trim(' 2014-09-27 ')) s where s.pado_spring in ('1','2') and s.probe_st_spring = 'song' and s.testman_sample_spring >= '21' and (s.bf_m2_series + s.bf_m3_series + s.bf_m4_series) >= 60000 group by s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm) e on a.probe_mgmt_string = e.probe_mgmt_string left outer join (select probe_mgmt_string, min(chg_silo) as chg_silo from master_mers_evthist where chg_silo between date_add(concat(substr(trim(' 20140927 '),1,4),'-',substr(trim(' 20140927 '),5,2),'-',substr(trim(' 20140927 '),7,2)),-60) and trim(' 20140927 ') and ((probe_chg_spring in ('1','2') and probe_chg_result_spring in ('11', '12')) or probe_chg_spring in ('1','2')) group by probe_mgmt_string) c on a.probe_mgmt_string = c.probe_mgmt_string where c.probe_mgmt_string is null and d.probe_mgmt_string is null order by prob_simple_ko asc limit 10000 union all select a.probe_mgmt_string, e.marang_made_spring, e.marang_made_nm, e.bf_m2_series, e.chg_dev_silo, a.mouse_prob_simple as prob_simple, a.mouse_rank_simple as prob_simple_ko, a.mouse_prob_outs as prob_outs, a.mouse_rank_outs as prob_outs_ko, 'out' as choosen, trim(' 20140927 ') as silo from (select distinct probe_mgmt_string from tmp_old_report) d right outer join (select probe_mgmt_string, mouse_prob_simple, mouse_prob_outs, mouse_rank_simple, mouse_rank_outs from tmp_comp_post_prob_new where silo = trim(' 20140927 ') and pado_spring in ('1','2') and testman_sample_spring >= '021' and months >= 15 and event_type = 'event1' and mouse_rank_simple > mouse_rank_outs) a on a.probe_mgmt_string = d.probe_mgmt_string join (select s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm, round(avg(s.bf_m2_series),0) as bf_m2_series, max(case when s.prev_intro_chg_silo like '#%' then s.probe_test_silo else s.prev_intro_chg_silo end) as chg_dev_silo from (select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst where silo = trim(' 20140927 ') union all select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst_enc where silo = trim(' 2014-09-27 ')) s where s.pado_spring in ('1','2') and s.probe_st_spring = 'song' and s.testman_sample_spring >= '021' and (s.bf_m2_series + s.bf_m3_series + s.bf_m4_series) >= 60000 group by s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm) e on a.probe_mgmt_string = e.probe_mgmt_string left outer join (select probe_mgmt_string, min(chg_silo) as chg_silo from master_mers_evthist where chg_silo between date_add(concat(substr(trim(' 20140927 '),1,4),'-',substr(trim(' 20140927 '), 5,2),'-',substr(trim(' 20140927 '),7,2)),-60) and trim(' 20140927 ') and ((probe_chg_spring in ('1','2') and probe_chg_result_spring in ('11','12')) or probe_chg_spring in ('31','32')) group by probe_mgmt_string) c on a.probe_mgmt_string = c.probe_mgmt_string where c.probe_mgmt_string is null and d.probe_mgmt_string is null order by prob_outs_ko asc limit 10000) x
  16. 16. Accessibility 1,000,000,000,000
  17. 17. 1,000,000,000,000 100 TB Accessibility
  18. 18. Accessibility 1,000,000,000,000 100 TB 4 days 4 hours
  19. 19. Accessibility insert overwrite table tmp_daily_recommendation partition(silo) select x.probe_mgmt_string, x.marang_made_spring, x.marang_made_nm, x.bf_m2_series, x.chg_dev_silo, x.prob_simple, x.prob_simple_ko, x.prob_outs, x.prob_outs_ko, x.choosen, x.silo from ( select a.probe_mgmt_string, e.marang_made_spring, e.marang_made_nm, e.bf_m2_series, e.chg_dev_silo, a.mouse_prob_simple as prob_simple, a.mouse_rank_simple as prob_simple_ko, a.mouse_prob_outs as prob_outs, a.mouse_rank_outs as prob_outs_ko, 'chg' as choosen, trim(' 20140927 ') as silo from (select distinct probe_mgmt_string from tmp_old_report) d right outer join (select probe_mgmt_string, mouse_prob_simple, mouse_prob_outs, mouse_rank_simple, mouse_rank_outs from tmp_comp_post_prob_new where silo = trim(' 20140927 ') and pado_spring in ('1','2') and testman_sample_spring >= '021' and months >= 15 and event_type = 'event1' and mouse_rank_simple <= mouse_rank_outs) a on a.probe_mgmt_string = d.probe_mgmt_string join (select s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm, round(avg(s.bf_m2_series),0) as bf_m2_series, max(case when s.prev_intro_chg_silo like '#%' then s.probe_test_silo else s.prev_intro_chg_silo end) as chg_dev_silo from (select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst where silo = trim(' 20140927 ') union all select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst_enc where silo = trim(' 2014-09-27 ')) s where s.pado_spring in ('1','2') and s.probe_st_spring = 'song' and s.testman_sample_spring >= '21' and (s.bf_m2_series + s.bf_m3_series + s.bf_m4_series) >= 60000 group by s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm) e on a.probe_mgmt_string = e.probe_mgmt_string left outer join (select probe_mgmt_string, min(chg_silo) as chg_silo from master_mers_evthist where chg_silo between date_add(concat(substr(trim(' 20140927 '),1,4),'-',substr(trim(' 20140927 '),5,2),'-',substr(trim(' 20140927 '),7,2)),-60) and trim(' 20140927 ') and ((probe_chg_spring in ('1','2') and probe_chg_result_spring in ('11', '12')) or probe_chg_spring in ('1','2')) group by probe_mgmt_string) c on a.probe_mgmt_string = c.probe_mgmt_string where c.probe_mgmt_string is null and d.probe_mgmt_string is null order by prob_simple_ko asc limit 10000 union all select a.probe_mgmt_string, e.marang_made_spring, e.marang_made_nm, e.bf_m2_series, e.chg_dev_silo, a.mouse_prob_simple as prob_simple, a.mouse_rank_simple as prob_simple_ko, a.mouse_prob_outs as prob_outs, a.mouse_rank_outs as prob_outs_ko, 'out' as choosen, trim(' 20140927 ') as silo from (select distinct probe_mgmt_string from tmp_old_report) d right outer join (select probe_mgmt_string, mouse_prob_simple, mouse_prob_outs, mouse_rank_simple, mouse_rank_outs from tmp_comp_post_prob_new where silo = trim(' 20140927 ') and pado_spring in ('1','2') and testman_sample_spring >= '021' and months >= 15 and event_type = 'event1' and mouse_rank_simple > mouse_rank_outs) a on a.probe_mgmt_string = d.probe_mgmt_string join (select s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm, round(avg(s.bf_m2_series),0) as bf_m2_series, max(case when s.prev_intro_chg_silo like '#%' then s.probe_test_silo else s.prev_intro_chg_silo end) as chg_dev_silo from (select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst where silo = trim(' 20140927 ') union all select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst_enc where silo = trim(' 2014-09-27 ')) s where s.pado_spring in ('1','2') and s.probe_st_spring = 'song' and s.testman_sample_spring >= '021' and (s.bf_m2_series + s.bf_m3_series + s.bf_m4_series) >= 60000 group by s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm) e on a.probe_mgmt_string = e.probe_mgmt_string left outer join (select probe_mgmt_string, min(chg_silo) as chg_silo from master_mers_evthist where chg_silo between date_add(concat(substr(trim(' 20140927 '),1,4),'-',substr(trim(' 20140927 '), 5,2),'-',substr(trim(' 20140927 '),7,2)),-60) and trim(' 20140927 ') and ((probe_chg_spring in ('1','2') and probe_chg_result_spring in ('11','12')) or probe_chg_spring in ('31','32')) group by probe_mgmt_string) c on a.probe_mgmt_string = c.probe_mgmt_string where c.probe_mgmt_string is null and d.probe_mgmt_string is null order by prob_outs_ko asc limit 10000) x 4 days 4 hours
  20. 20. http://www.ldn.net.au/wp-content/uploads/2013/09/boost-your-marketing-effectiveness.png
  21. 21. 1월 2월 3월 4월 http://fc06.deviantart.net/fs70/f/2009/357/f/c/Hopelessness_by_sarafim.jpg
  22. 22. Accessibility Hive에서 Tajo/Impala로 ! 100 ! 75 50 25 0 April May June July Data Size / Date Impala Tajo
  23. 23. Accessibility insert overwrite table tmp_daily_recommendation partition(silo) select x.probe_mgmt_string, x.marang_made_spring, x.marang_made_nm, x.bf_m2_series, x.chg_dev_silo, x.prob_simple, x.prob_simple_ko, x.prob_outs, x.prob_outs_ko, x.choosen, x.silo from ( select a.probe_mgmt_string, e.marang_made_spring, e.marang_made_nm, e.bf_m2_series, e.chg_dev_silo, a.mouse_prob_simple as prob_simple, a.mouse_rank_simple as prob_simple_ko, a.mouse_prob_outs as prob_outs, a.mouse_rank_outs as prob_outs_ko, 'chg' as choosen, trim(' 20140927 ') as silo from (select distinct probe_mgmt_string from tmp_old_report) d right outer join (select probe_mgmt_string, mouse_prob_simple, mouse_prob_outs, mouse_rank_simple, mouse_rank_outs from tmp_comp_post_prob_new where silo = trim(' 20140927 ') and pado_spring in ('1','2') and testman_sample_spring >= '021' and months >= 15 and event_type = 'event1' and mouse_rank_simple <= mouse_rank_outs) a on a.probe_mgmt_string = d.probe_mgmt_string join (select s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm, round(avg(s.bf_m2_series),0) as bf_m2_series, max(case when s.prev_intro_chg_silo like '#%' then s.probe_test_silo else s.prev_intro_chg_silo end) as chg_dev_silo from (select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst where silo = trim(' 20140927 ') union all select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst_enc where silo = trim(' 2014-09-27 ')) s where s.pado_spring in ('1','2') and s.probe_st_spring = 'song' and s.testman_sample_spring >= '21' and (s.bf_m2_series + s.bf_m3_series + s.bf_m4_series) >= 60000 group by s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm) e on a.probe_mgmt_string = e.probe_mgmt_string left outer join (select probe_mgmt_string, min(chg_silo) as chg_silo from master_mers_evthist where chg_silo between date_add(concat(substr(trim(' 20140927 '),1,4),'-',substr(trim(' 20140927 '),5,2),'-',substr(trim(' 20140927 '),7,2)),-60) and trim(' 20140927 ') and ((probe_chg_spring in ('1','2') and probe_chg_result_spring in ('11', '12')) or probe_chg_spring in ('1','2')) group by probe_mgmt_string) c on a.probe_mgmt_string = c.probe_mgmt_string where c.probe_mgmt_string is null and d.probe_mgmt_string is null order by prob_simple_ko asc limit 10000 union all select a.probe_mgmt_string, e.marang_made_spring, e.marang_made_nm, e.bf_m2_series, e.chg_dev_silo, a.mouse_prob_simple as prob_simple, a.mouse_rank_simple as prob_simple_ko, a.mouse_prob_outs as prob_outs, a.mouse_rank_outs as prob_outs_ko, 'out' as choosen, trim(' 20140927 ') as silo from (select distinct probe_mgmt_string from tmp_old_report) d right outer join (select probe_mgmt_string, mouse_prob_simple, mouse_prob_outs, mouse_rank_simple, mouse_rank_outs from tmp_comp_post_prob_new where silo = trim(' 20140927 ') and pado_spring in ('1','2') and testman_sample_spring >= '021' and months >= 15 and event_type = 'event1' and mouse_rank_simple > mouse_rank_outs) a on a.probe_mgmt_string = d.probe_mgmt_string join (select s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm, round(avg(s.bf_m2_series),0) as bf_m2_series, max(case when s.prev_intro_chg_silo like '#%' then s.probe_test_silo else s.prev_intro_chg_silo end) as chg_dev_silo from (select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst where silo = trim(' 20140927 ') union all select probe_mgmt_string, marang_made_spring, marang_made_nm, prev_intro_chg_silo, probe_test_silo, bf_m2_series, bf_m3_series, bf_m4_series, pado_spring, probe_st_spring, testman_sample_spring from master_mers_mst_enc where silo = trim(' 2014-09-27 ')) s where s.pado_spring in ('1','2') and s.probe_st_spring = 'song' and s.testman_sample_spring >= '021' and (s.bf_m2_series + s.bf_m3_series + s.bf_m4_series) >= 60000 group by s.probe_mgmt_string, s.marang_made_spring, s.marang_made_nm) e on a.probe_mgmt_string = e.probe_mgmt_string left outer join (select probe_mgmt_string, min(chg_silo) as chg_silo from master_mers_evthist where chg_silo between date_add(concat(substr(trim(' 20140927 '),1,4),'-',substr(trim(' 20140927 '), 5,2),'-',substr(trim(' 20140927 '),7,2)),-60) and trim(' 20140927 ') and ((probe_chg_spring in ('1','2') and probe_chg_result_spring in ('11','12')) or probe_chg_spring in ('31','32')) group by probe_mgmt_string) c on a.probe_mgmt_string = c.probe_mgmt_string where c.probe_mgmt_string is null and d.probe_mgmt_string is null order by prob_outs_ko asc limit 10000) x 4 days 4 hours 2 days 2 hours
  24. 24. 2. Expansion
  25. 25. Expansion Current System 분석 Job Detail 저장 표준화 ETL, Cleansing, Lineage 등 저장 20 PB 저장 능력 공급 원활한 데이터 공급 (Real Time / Batch) 프로세싱 분석 R, Python 등 분석 중심 DW/ Realtime Low Latency, Event Processing 현황 Data Size (압축 / Origin) Day 50 TB / 250 TB Year 18.25 PB / 91.25 PB Job Type 저장 표준화/저장/공급 프로세싱 분석/Real Time
  26. 26. Expansion Jupiter (분석) 3 PB BigBang! Saturn (저장) 20 !PB Neptune (Real Time) 1 PB Flume
  27. 27. Expansion Saturn(저장) Cluster Topology 2G 40G DS 10G X 2 Bonding .! .! .! . .! .! .! . .! .! .! .! .! .! .! .! . Rack awareness 4TB X 12 AS
  28. 28. Expansion : Saturn (저장) Disk Fault at Datanode High IO Low IO Eject! RoundRobin Available Space
  29. 29. Expansion : Saturn (저장) High Temperature at Datanode Disk Controller 참고 쓰는 중 http://rlv.zcache.com/suppressed_laughing_yellow_smiley_face_stickers-r200e51f37ff941a38208de69f6c51657_v9waf_8byvr_512.jpg
  30. 30. Expansion : Saturn (저장) + Flume Saturn(저장) Cluster Topology + Flume .! .! .! .! .! .! .! .! .! Flume Compress & Send / 1 minute Dynamic Frequency Scaling Maximum Performance
  31. 31. Expansion : Saturn (저장) + Flume Saturn(저장) Cluster Topology + Flume Flume Sending… Sudden Fault Disk at DataNode Eventual Sending using SSD
  32. 32. Expansion : Neptune (DW) Neptune(DW) Cluster Topology 10G 40G X 2 20G Bonding .! .! .! . .! .! .! . .! .! .! .! .! .! .! .! . Rack awareness 1TB X 23 SAS DS AS
  33. 33. Expansion : Neptune (DW) Bandwidth가 높을 때는 Network 필수적으로 점검할 사항 20G Bonding $ ifconfig! …! eth0 Link encap:Ethernet HWaddr 38:EA:A7:38:53:24 ! UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1! RX packets:7170284853 errors:2456 dropped:25019 overruns:0 frame:2456! TX packets:31088639355 errors:0 dropped:0 overruns:0 carrier:0! collisions:0 txqueuelen:10000 ! RX bytes:41081083208513 (37.3 TiB) TX bytes:40786177694493 (37.0 TiB)! … GBIC $ ethtool -S eth0! …! rx_crc_errors : 2456! …
  34. 34. Expansion : Neptune (DW) Bandwidth가 높을 때는 Network 필수적으로 점검할 사항 https://c0da80aa54a5e1ed7d2b945327c31140a345bfe8.googledrive.com/host/0BxotWZXnwSAGSS1qRE02eWVrU28/2013-07-kernel-networking-ring-buffer. png
  35. 35. Expansion : Neptune (DW) Bandwidth가 높을 때는 Network 필수적으로 점검할 사항 # ethtool -­‐g eth0 Ring parameters for eth0: Pre-­‐set maximums: RX: 4096 (최대) RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 512 (현재) RX Mini: 0 RX Jumbo: 0 TX: 512 잦은 Frame Packet Drop 발생 # ethtool -­‐G eth0 rx 2048
  36. 36. Expansion : Neptune (DW) Bandwidth가 높을 때는 Network 필수적으로 점검할 사항 rmem_max wmem_max tcp_mem socket Receive! Buffer Send! Buffer tcp_rmem tcp_wmem net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 204800 204800 16777216 net.ipv4.tcp_wmem = 204800 204800 16777216 net.ipv4.tcp_wmem 204800 204800 16777216 R S R S
  37. 37. Expansion Jupiter (분석) 3 PB BigBang! Saturn (저장) 20 !PB Neptune (Real Time) 1 PB Flume
  38. 38. Expansion : Yarn Multiple Processing Engine들의 Resrouce Mgmt를 위해 NodeManager yarn.nodemanager.resource.memory-mb! Node Manager에서 관리하는 전체 메모리 yarn.nodemanager.resource.cpu-vcores! Node Manager에서 관리하는 CPU Core수 Resource Manager yarn.scheduler.minimum-allocation-mb! 각 Node Manager에 할당할 수 있는 Container당 ! 최소 메모리
  39. 39. Expansion : Yarn Multiple Processing Engine들의 Resrouce Mgmt를 위해 Resource Manager 최소메모리 : 3G NodeManager 각 Node당 단 1개의 Container만 생성 이렇게 일주일 운영 전체메모리 : 3G 사용가능 Core수 : 18 Container TaskJVM! (ex. TaskTracker Fork 1개)
  40. 40. Expansion : Yarn Multiple Processing Engine들의 Resrouce Mgmt를 위해 Resource Manager 최소메모리 : 3G NodeManager 각 Node당 18 개의 Container 생성 가능 전체메모리 : 54 G 사용가능 Core수 : 18 Container TaskJVM! (ex. TaskTracker Fork 1개)
  41. 41. Expansion : Compress Snappy가 좋다고 하길래 용량이 넉넉해서 마음껏 사용! Raw Data 250 TB/day Snappy 90 TB/day 32 PB/year 용량이 부족 Snappy 90 TB/day GZip 50 TB/day 2달 걸림.
  42. 42. Expansion : NameNode HA Automatic FailOver면 안심해도 되는줄 Zookeeper Timeout : 60초 NameNode GC하는데 3분 30초 걸림 Standby로 FailOver했는데, ! Hadoop Client들이 원래 Active로만 연결 전 Cluster 장애 Zookeeper Timeout : 10분
  43. 43. Expansion : Too Many CLOSE_WAIT MR V1 & Datanode 1. connect TaskTracker DataNode 2. block 요청 3. send block 4. close CLOSE_WAIT FIN_WAIT 2시간 내로 없어지지 않음. client socket port 고갈 TT Restart
  44. 44. 3. Lessons Learns
  45. 45. Lessons Learned Accessibility 1. 누구나 쉽게 접근할 수 있어야 한다. 2. 프로그램은 할 줄 몰라도 동작원리는 알아야 한다. 3. 쉬우면 많은 사람들이 접근한다. 4. 누구나 분석가가 되어간다.
  46. 46. Lessons Learned Expansion 1. Network는 Hadoop의 혈관과 같다. 2. Yarn은 아직 사용하기 시기 상조다. 설정 정보가 너무 많고, 상관 관계도 너무 복잡하다. 3. Hadoop 이중화는 반드시 Client도 확인해야 한다. 4. Hadoop 이중화가 그렇다고 정말 안전하지도 않다. 5. Yahoo 2,000대는 아마도 디스크가 작았던 것 같다. 6. 아직 해야할 일이 많다.
  47. 47. 4. Future
  48. 48. Approximate Query Engine Blink DB 처럼 select sum(val1) from table where key=’a’ within 3 seconds select sum(val1) from table where key=’a’ Error Rate 10%
  49. 49. Approximate Query Engine Zoomable Data Navigation select sum(val1) from table where age between 1 and 10 within 1 seconds select sum(val1) from table where age between 1 and 10 within 10 seconds
  50. 50. Q&A
  51. 51. THANK YOU

×