Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Build your application in seconds and optimize workflow as much as you can using dev ops

2,683 views

Published on

Building an application is a very intense and complicated process. Sometimes it could lead to unacceptable results when you can wait for temporary product eternity. Tools could be different, applications could be different, but techniques will be the same.

Optimization is very important thing even when your process is standartized and strong. During that seesion I'll talk about:
- Build is the most valuable product in DevOps
- Tests, Sniffers, Performance tests and other things are minor in comparison to builds
- How to get rid of long waits for small changes or fixes
- How to don't waste time for waiting for build
- How to incorporate measurement tools
- How to solve feature branch hell and don't spent tons of time for merge conflicts
- Make builds for enterprise and big data databases
- Other interesting things from DevOps live :)

Optimisation strategy shouldn’t be strict and shouldn’t ruin current process or block the team from performing operations. Given those answers, we can move forward like a thunder and achieve whatever we want.

Published in: Data & Analytics
  • Be the first to comment

Build your application in seconds and optimize workflow as much as you can using dev ops

  1. 1. Alexander Schedrov aka sanchiz DrupalCamp Lviv, Sep 2016 Build your application in seconds and optimize workflow as much as you can using DevOps
  2. 2. What is build? Build is the process of integrating, building and compiling the software that is produced.
  3. 3. – Кузьма Иванович, водовоз “Без воды - и не туды, и не сюды”
  4. 4. – Кузьма Иванович, водовоз “Без воды - и не туды, и не сюды”
  5. 5. – Щедров Александр, билдовоз :) “Без билда - и не туда, и не сюда”
  6. 6. How to accelerate builds?
  7. 7. What is a big project?
  8. 8. Big data?
  9. 9. Big codebase?
  10. 10. Big team?
  11. 11. Big client?
  12. 12. Big money!
  13. 13. Big money!
  14. 14. What is the most valuable thing for client in project?
  15. 15. Clean and structurized code?
  16. 16. Custom modules?
  17. 17. Tests coverage?
  18. 18. Product, speed and deliverability!
  19. 19. – Andrii Podanenko, FFW “Success depends on how fast you can show changes to client.”
  20. 20. Our project
  21. 21. In terms of build • $$$ :) • Huge database • ~300mb *.sql.gz • ~ 4gb *.sql • D8 multisite - 3 sites • Automated tests • Code sniffers
  22. 22. In terms of workflow • Builds queue • ~50 per day • Server space • 80GB droplet • ~10 active builds + 5 feature dev sites
  23. 23. Old build timeline
  24. 24. PR build timeline
  25. 25. Primary build timeline
  26. 26. + 2 subsites :(
  27. 27. Let The Machines Do The Work
  28. 28. Lazy builders
  29. 29. Primary build and secondary builds 1. Primary build 2. Other builds in parallel • Slave sites • Sniffers • Tests
  30. 30. Lazy builder timeline
  31. 31. Post build actions
  32. 32. GitHub API for lazy builders sudo jo state=pending target_url=http://URL/job/PR_BUILDER_TESTS/$ {BUILD_NUMBER}/console description="Tests run has been started." context="CIBox tests" > pending.json curl -u [bot_name]:[token] https://api.github.com/repos/[org]/ [repo]/statuses/${PR_SHA1} --request POST --data @pending.json Pending status
  33. 33. GitHub API for lazy builders # Run tests. cd /var/www/build${PARENT_BUILD_NUMBER} ansible-playbook tests.yml -i 'localhost,' --connection=local # Set status and post comment to GitHub. cd ${WORKSPACE} sudo jo body="Behat test results file URL: http://URL/build$ {PARENT_BUILD_NUMBER}/build_reports/behat_report.html" > comment.json curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/ [repo]/issues/${PR_ID}/comments --request POST --data @comment.json sudo jo state=success target_url=http://URL/job/PR_BUILDER_TESTS/$ {BUILD_NUMBER}/console description="Tests run has been finished." context="CIBox tests" > success.json curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/ [repo]/statuses/${PR_SHA1} --request POST --data @success.json Tests + success status
  34. 34. https://wiki.jenkins-ci.org/display/JENKINS/Post+build+task
  35. 35. GitHub API for lazy builders cd ${WORKSPACE} sudo jo state=failure target_url=http://URL/job/PR_BUILDER_TESTS/$ {BUILD_NUMBER}/console description="Tests run has been failed." context="CIBox tests" > failed.json curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/ [repo]/statuses/${PR_SHA1} --request POST --data @failed.json Failed status
  36. 36. Build hierarchy Primary Build Slave 1 Slave 2 Sniffers Tests Comment + status on GitHub Comment + status on GitHub Comment + status on GitHub Comment + status on GitHub Comment + status on GitHub
  37. 37. Database optimisation
  38. 38. Optimize SQL import SQL import is the most intensive operation • Warm up databases for build? • Bandwidth speed for downloading dumps locally? • Maintenance?
  39. 39. Virtualization with Docker
  40. 40. Create Docker Image 1. Run local Docker repository(per project) 2. Pull base image cibox/mysql 3. Import database from production dump 4. Create/Update image after import 5. Push to local Docker repository(per project)
  41. 41. Build pipeline 1.Pull changes from Docker Hub + Custom Repository 2.Run Docker container 3.Specify in settings container’s IP 4.Container will be killed via cleaner
  42. 42. Benefits 1. Seconds to spin up container 2. Non-blocking operations for concurrent builds 3. Pull only changes(commits) to local environment 4. Update Docker Image daily
  43. 43. New build timeline
  44. 44. New build timeline
  45. 45. New Primary build timeline
  46. 46. Drush cr optimization
  47. 47. Why it faster now? • ~2 seconds to spin up container • ~2 seconds to clear Drupal 8 cache • Non-Blocking operation in filesystem • Isolated MySQL • Run fast tests on build • Run full tests on demo/dev site
  48. 48. Statistics Start date
  49. 49. Numbers • Primary build • ~15 mins -> ~3 mins • Overall time • ~3 mins before testing of main product • ~6-8 mins before testing of secondary products • ~6-8 mins before sniffers & test results
  50. 50. Metrics - measure your success
  51. 51. https://wiki.jenkins-ci.org/display/JENKINS/Timestamper
  52. 52. Time in comments
  53. 53. https://wiki.jenkins-ci.org/display/JENKINS/Global+Build +Stats+Plugin
  54. 54. Remote access API http://[JENKINS_URL]/job/[JOB_NAME]/api/json?tree=allBuilds[*]
  55. 55. DevOps life
  56. 56. – Chang Xiao, FFW “I swear to god our vagrant is like the nerdiest thing ever.” “It should be like making a ANSCII sandwich, brewing some tea, waiting for your local environment to be fully baked.”
  57. 57. Drupal.org: https://www.drupal.org/u/sanchiz GitHub: https://github.com/Sanchiz Blog: http://sanchiz.net Email: alexander.schedrov@gmail.com Twitter: @alexschedrov Thank you!

×