YAPCEurope2014-myfinder

1,677 views
1,657 views

Published on

talk slieds for YAPC::Europe 2014

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,677
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
1
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

YAPCEurope2014-myfinder

  1. 1. How to: more faster “make test” YAPC::Europe 2014
  2. 2. who am i Name: Tatsuro Hisamori twitter, PAUSE, etc..: myfinder Came from: Tokyo / JAPAN I’m called “maa-in-der(まいんだー)” from engineers in japan.
  3. 3. who am i please call me “maa- in-der” “(まいんだー)” when you call my name.
  4. 4. why I came here - I was awarded the best runner-up talk of YAPC::Asia 2013 - talk page link - today’s my talk is a part of that talk.
  5. 5. I’m working at - Internet Ad Tech Company at Tokyo Japan. - that company has a RTB business brand “doubles” in U.S. and Europe. - https://www.dobleas.com/
  6. 6. RTB? - Real-time Bidding - programmatic instantaneous auction - similar to financial markets "High Frequency Trade"
  7. 7. RTB? - We Developed that system by Perl. - Talk about the “make test” of this system.
  8. 8. Today’s topics - Test case(.t files) is increased to more than 1000 - problems with this situation. - the method and system to be executed at a high speed. - how to develop that test system.
  9. 9. we have Problems - more than 40 minutes uses to test one branch. - execution of the “make test” is serialized - time-consuming - wait and wait and wait….. - motivation is lost ;-(
  10. 10. we have Problems uggggggg…...
  11. 11. we have Problems - and developers will not to run the “make test”. - and the risk of accidents will increase. - and Product will not be able to maintenance ;- (
  12. 12. we have Problems oh thats NO FUTURE ;-(
  13. 13. Improvement aka KAIZEN - requirement - faster more and more of course. - that can be scaled out and easy maintenance. - without changing the test already exeists.
  14. 14. Strategies of Implementations - run across multiple servers. - distributed a ".t" file-by-file basis. - easy to add server. - provisioning by puppet. - easy to maintenance. - deploy by capistrano “cap shell”.
  15. 15. implementation image Master Node Cluster Node Test List First count the number of test files(.t files). shuffle and assign test cases to nodes. Second execute tests. receive results from nodes and summarize. Therd notification and visualize.
  16. 16. First - count and shuffle uses File::Find, List:: MoreUtils - and assign .t files to cluster node. - see gist.
  17. 17. Second - Parallel::ForkManager, Net::OpenSSH - see gist.
  18. 18. Third - notification to developers and visualize. - eMail, HipChat API, Result Visualize tool - “email sample gist” - “HipChat sample screenshot"
  19. 19. Third - HipChat notify - from notify bot
  20. 20. Third - Visualize tool - we uses “Ukigumo::Server” - Ukigumo::Server is very Simple WebApp - http://search.cpan.org/dist/Ukigumo-Server/
  21. 21. Ukigumo::Server
  22. 22. Ukigumo::Server
  23. 23. Ukigumo::Server
  24. 24. Ukigumo::Server - see Ukigumo-chan, this is notification icon. - smile face when success “make test” - angry face when failed “make test”
  25. 25. Ukigumo::Server ↘
  26. 26. Ukigumo::Server - very easy to understand the results. - Ukigumo-chan designed by MIKIHOSHI (PAUSE) - he is my Colleague
  27. 27. running environment - we made 16 nodes cluster by surplus PC parts - the servers has been worked production env until years ago :D
  28. 28. running environment - If you try this, you should to use the AWS EC2 :-) - use the spot instance in particular. - it’s cheap :-)
  29. 29. result - reduced about 40 min -> 3 min - becomes even faster by adding more servers. - we got more faster “make test” cluster!!!
  30. 30. conclusion - talked about more faster “make test”. - First, it’s simple, made from well-known perl modules. - Second, it’s very scalable. - Therd, easy to summarize and notifications and visualize.
  31. 31. one more thing. - a nice surprise. Developers has came "git push first, if them make a new branch" therefor developer activities became more visible.
  32. 32. a nice suprise "high-speed make test" will change the development culture.
  33. 33. More Informations - this story has been serialized in the Japanese magazine Web. - bit.ly/faster-make-test
  34. 34. thank you for listening. have fun! last day of YAPC::Europe!!

×