Coscup 2013 : Continuous Integration on top of hadoop

1,271
-1

Published on

CI on hadoop

Coscup 2013 Taipei

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,271
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
28
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Coscup 2013 : Continuous Integration on top of hadoop

  1. 1. Continuous Integration on top of Hadoop Wisely Chen and Neal Lee Saturday, August 3, 13
  2. 2. Agenda • Who I am • Problem • Solution • Demo • Q&A Saturday, August 3, 13
  3. 3. Who I am • Wisely Chen ( thegiive@gmail.com ) • Release manager of Yahoo![Taiwan] shopping and data team • Loves to promote open source tech in Taiwan • Hadoop Summit 2013 San Jose • Ruby and Rails : Coscup 2006, Ubisunrise 2007, OSDC 2007 • Puppet : PHPConf 2012 , RubyConf 2012 • Release Practice :Webconf 2013, Coscup 2012 Saturday, August 3, 13
  4. 4. Who I am • Neal Lee (@neal_lee) • Data Engineer at Yahoo![Taiwan] data team • Aims to build a easy to use self-service BI platform connecting to Hadoop. Saturday, August 3, 13
  5. 5. EC Data Team 拍賣/商城/購物中心 站台 流量/點擊/使用者行為 追蹤 Transactional data Tracking data Data Highway Data Warehouse/ Data Mart Data Infra BI Platform Report Recommendation API Machine Learning Serve Saturday, August 3, 13
  6. 6. Problem : Debug Saturday, August 3, 13
  7. 7. Problem : Performance Saturday, August 3, 13
  8. 8. Solution Saturday, August 3, 13
  9. 9. Continuous Integration Saturday, August 3, 13
  10. 10. Continuous Integration • A software engineering practice • Maintain code repos • Automate the build • Make the build self-testing • Everyone commit to the baseline everyday • Every commit should be a build • Test in a clone of production environment • Make it easy to get the latest deliverables • Everyone can see the result of latest build • Automate deployment Saturday, August 3, 13
  11. 11. We focus on • A software engineering practice • Maintain code repos • Automate the build • Make the build self-testing • Everyone commits to the baseline everyday • Every commit should be a build • Test in a clone of production environment • Make it easy to get the latest deliverables • Everyone can see the result of latest build • Automate deployment Saturday, August 3, 13
  12. 12. CI on Hadoop Flow Code Unit Test Performance Test Deploy Doc Execution Saturday, August 3, 13
  13. 13. One Click Deploy Commit Unit Test Performance Test Deploy Doc Execution Saturday, August 3, 13
  14. 14. Toolset Commit Unit Test Performance Test Deploy Doc Execution Vaidya BASH Saturday, August 3, 13
  15. 15. System diagram CI Master GitHub Alpha CI Slave Beta Cluster Hadoop JobTracker CI Slave Hadoop node Hadoop node Hadoop node Hadoop node Slave Node Prod ClusterGateway Saturday, August 3, 13
  16. 16. Unit Test Commit Unit Test Performance Test Deploy Doc Execution Saturday, August 3, 13
  17. 17. PigUnit • A simple xUnit framework • No cluster set up is required in local mode • Unit testing, regression testing, and rapid prototyping on the fly Saturday, August 3, 13
  18. 18. Using PigUnit • After • Coding • Write PigUnit test case • Run local PigUnit test • Push to cluster • Run Pig on cluster • Get right result ! • Before • Coding • Manual local test • Push to cluster • Run Pig on cluster • Get right result ! Saturday, August 3, 13
  19. 19. Unit test is live doc • Unit test is runnable live doc • Pass test case and meet previous requirement Saturday, August 3, 13
  20. 20. Flexible • Pig can use PigUnit • MapReduce can use MapUnit • Hive can use hive_test Saturday, August 3, 13
  21. 21. Performance Test Commit Unit Test Performance Test Deploy Doc Execution Saturday, August 3, 13
  22. 22. Vaidya • Rule based performance diagnosis of M/R jobs • Extensible framework • You can add your own rules • Write complex rules using existing rules Saturday, August 3, 13
  23. 23. Performance Test Pig Job Pig Job History Vaidya Vaidya Rule 4 Pig Job Conf Notify User 3 Performance result Next CI Stage 1 1 2 2 2 5 1. Exec pig job with sampling data on beta server 2. Vaidya read job history,conf,rule to check performance problem 3. If ok, create performance result 4. If job has performance issue, notify user 5. Go to next CI stage Sampling data 1 Saturday, August 3, 13
  24. 24. Vaidya Rule<Diagnos)cTest> <Title><![CDATA[Balanaced Reduce Partitioning]]></Title>  <ClassName>        <![CDATA[                      org.apache.hadoop.vaidya.postexdiagnosis.tests.BalancedReducePar77oning        ]]> </ClassName> <Descrip)on>          <![CDATA[This  rule  tests  as  to  how  well  the  input  to  reduce  tasks  is  balanced]]> </Descrip)on> <Importance><![CDATA[High]]></Importance> <SuccessThreshold><![CDATA[0.20]]></SuccessThreshold> <Prescrip)on><![CDATA[advice]]></Prescrip)on> </Diagnos)cTest> See  if  the  reduce  job  is   balance  or  not   Rule  importance Diagnose  success   threshold Test  Java  Class Saturday, August 3, 13
  25. 25. Deploy Commit Unit Test Performance Test Deploy Doc Execution Saturday, August 3, 13
  26. 26. Deploy • Deploy to production cluster • Easy to rollback • Create a git tag • Auto doc generating • Each release should map to a ticket • Auto comment in Bugzilla Saturday, August 3, 13
  27. 27. Auto comment in bugzilla Repo url Release Note Issue status change Saturday, August 3, 13
  28. 28. Auto create git tag Release Note [Bug xxx] log.... Git Tag Saturday, August 3, 13
  29. 29. DEMO Saturday, August 3, 13
  30. 30. Demo • Demo1 : Unit test fail • Demo2 : Unit test success • Demo3 : Check performance test • Demo4 :Auto generate Doc • Demo5 : Notify user Saturday, August 3, 13
  31. 31. Demo Saturday, August 3, 13
  32. 32. Conclusion • CI will revolutionize your workflow • CI will boost your productivity Saturday, August 3, 13
  33. 33. Saturday, August 3, 13
  34. 34. Logic Debug • Map/Reduce  job  oJen  takes  a  lot  of  )me  for  execu)on • Repeated  Map/Reduce  execu)on  cost    a  lot  of  )me   during  logic  debugging  phase • Need  a  way  to  find  out  logic  problem  before   execu)on  produc)on  job • Coding Manual Test Exec Get Bug Saturday, August 3, 13
  35. 35. Performance • Map/Reduce  performance  is  hard  to  es)mate  before  execu)on   • Production Grid computing resource is shared by allYahoos • Bad performance will affect otherYahoos Grid jobs • Putting bad performance code on production grid is guilty • We manually investigate the job performance before we actually execute it on production Grid Coding Manual Test Manual investgate Get Bug Saturday, August 3, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×