Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Python - code quality and production monitoring

2,895 views

Published on

About code quality & production monitoring

Published in: Engineering

Python - code quality and production monitoring

  1. 1. Code Quality to Monitoring production: use cases and best practices David Melamed Pyweb meetup - September 21, 2014
  2. 2. Agenda What is good code? How to measure quality code? Continuous Integration How to efficiently monitor production?
  3. 3. About me Full-stack developer at Cloudlock Fond of technology & automation Code Quality Evangelist Geek on my free time
  4. 4. About Cloudlock Founded: 2011 Headquarters: Waltham, Mass. (U.S.) R&D Headquarters: Tel Aviv Employees: 80+ Funding: $28M+ Investors Selected customers 700+ Customers 9,600 Unique Apps Discovered Native Cloud Security Solutions for SaaS Applications 5 million Users under Management 13 billion objects processed daily
  5. 5. What is good code?
  6. 6. Good code is… Easy to read & understand Easy to extend Easy to maintain Testable
  7. 7. How to write good code?
  8. 8. from xkcd
  9. 9. Good code requires constant REFACTORING
  10. 10. The Code Quality Star Coding Standards Code Review Automated Tests Code Complexity Architecture Design Source
  11. 11. Code should be easy to read
  12. 12. Code should be readable Avoid comments but avoid obvious ones (the code should be self-explanatory) Avoid comments too verbose Comments need to be maintained
  13. 13. Coding Standards
  14. 14. The Zen of Python (PEP20) Beautiful is better than ugly Explicit is better than implicit Simple is better than complex Complex is better than complicated Readability counts
  15. 15. Coding Style: Readability Counts “Programs must be written for people to read, and only incidentally for machines to execute.” Abelson & Sussman, Structure and Interpretation of Computer Programs
  16. 16. PEP8: Style guide for Python Indentation Maximum line length Blank lines Redundant white spaces Imports Comments Naming conventions …
  17. 17. How to check /enforce PEP8 Pycharm (Tools > Flake8) Git pre-commit hook (http://goo.gl/bteYkq) Jenkins job for continuous integration
  18. 18. Naming is key
  19. 19. Naming best practises Adopt a convention and stick with it (fetch vs. retrieve vs. get) Use intention-revealing names Avoid misleading readers Make meaningful distinctions Use searchable names Avoid mental mapping Use domain names From Clean code, R. Martin
  20. 20. Code Architecture
  21. 21. Best Practices in Code Architecture OOP DRY SOLID Loose Coupling
  22. 22. Don’t Repeat Yourself (aka DRY) http://www.sonarqube.org/manage-duplicated-code-with-sonar/
  23. 23. SOLID principles • SRP: Single Responsibility Principle • OCP: Open/closed principle • LSP: Liskov substitution principle • ISP: Interface segregation principle • DIP: dependency inversion principle
  24. 24. http://www.codeproject.com/Articles/93369/How-I-explained-OOD-to-my-wife
  25. 25. http://www.codeproject.com/Articles/93369/How-I-explained-OOD-to-my-wife
  26. 26. http://www.codeproject.com/Articles/93369/How-I-explained-OOD-to-my-wife
  27. 27. http://www.codeproject.com/Articles/93369/How-I-explained-OOD-to-my-wife
  28. 28. http://www.codeproject.com/Articles/93369/How-I-explained-OOD-to-my-wife
  29. 29. Loosely Coupled Components
  30. 30. Main principles Split components into repositories Web app: use clean layers view / endpoint (presentation) service (business logic) DAO (model) Use message bus for communication inter-components (i.e. RabbitMQ) Use daemons / workers for long-running tasks Use events / signals to decouple unrelated actions Microservices
  31. 31. Architecture at Cloudlock Nginx AWS VPC Web Server API Server Message Bus (RabbitMQ) RDS PostgreSQL ElastiCache Redis Authentication Gateway Worker Worker Worker Logs
  32. 32. Code Reviews
  33. 33. Tools for code review Github Pull Requests Review Board Gerrit Crucible
  34. 34. Code documentation
  35. 35. Code Testing
  36. 36. Different types of tests Manual Test Unit Test Integration Test System Test Acceptance/Regression Test (end-to-end) Smoke Test
  37. 37. Unit Test Test if small, distinct and isolated portions of code are working as expected, provided some input Mock your dependencies (database, logger, service) Should be very fast No need to test EVERYTHING (i.e. private methods) Frameworks: unittest, pytest and nose
  38. 38. Pytest No need for base class (works both for classes and methods) More pythonic than unittest Fixtures (conftest.py) Parametrized tests Markers Various plugins for coverage, bdd, django, twisted… Used for unit tests, integration tests and end to end tests
  39. 39. Integration Tests Test communication between services You may mock some of the dependencies depending on what you test
  40. 40. System Tests Global test, more functional of the whole system TestApp (Pyramid app) using a test database Call the API and test the expected response code
  41. 41. End-to-end tests Splinter: web driver abstraction (firefox, selenium) splinter-pytest: fixtures for browser Plugin for rerun flaky tests, tests with steps Use Selenium Grid to test several browsers
  42. 42. Smoke Tests Make sure the system is alive Tests that the components are working Tests that the components can communicate Health check / status page
  43. 43. Measuring code quality
  44. 44. Code Quality Metrics Code Standards Violations (pylint /flake8/pyflakes) LOC (Lines Of Code) Cyclomatic Complexity Unit Tests Coverage
  45. 45. Code Coverage Makes sense only for unit tests Calculates the coverage based on the paths run by the tests 100% coverage is not a goal on its own May be misleading regarding the code quality
  46. 46. Unit Test Diff Coverage
  47. 47. Mutation Analysis From Mutation analysis vs. code coverage in automated assessment of students’ Testing Skills (P. Ihantola)
  48. 48. Pylint Report ====== 85 statements analysed. ! Duplication ----------- ! +-------------------------+------+---------+-----------+ | |now |previous |difference | +=========================+======+=========+===========+ |nb duplicated lines |0 |0 |= | +-------------------------+------+---------+-----------+ |percent duplicated lines |0.000 |0.000 |= | +-------------------------+------+---------+-----------+ !!! Messages by category -------------------- ! +-----------+-------+---------+-----------+ |type |number |previous |difference | +===========+=======+=========+===========+ |convention |18 |18 |= | +-----------+-------+---------+-----------+ |refactor |2 |2 |= | +-----------+-------+---------+-----------+ |warning |1 |1 |= | +-----------+-------+---------+-----------+ |error |4 |4 |= | +-----------+-------+---------+-----------+ !!! Messages -------- ! +-----------------------------+------------+ |message id |occurrences | +=============================+============+ |line-too-long |10 | +-----------------------------+------------+ |no-name-in-module |4 | +-----------------------------+------------+ |missing-docstring |4 | +-----------------------------+------------+ Global evaluation ----------------- Your code has been rated at 5.18/10 (previous run: 5.18/10, +0.00) ! Raw metrics ----------- ! +----------+-------+------+---------+-----------+ |type |number |% |previous |difference | +==========+=======+======+=========+===========+ |code |90 |63.83 |90 |= | +----------+-------+------+---------+-----------+ |docstring |30 |21.28 |30 |= | +----------+-------+------+---------+-----------+ |comment |5 |3.55 |5 |= | +----------+-------+------+---------+-----------+ |empty |16 |11.35 |16 |= | +----------+-------+------+---------+-----------+ !!! Statistics by type ------------------ ! +---------+-------+-----------+-----------+------------+---------+ |type |number |old number |difference |%documented |%badname | +=========+=======+===========+===========+============+=========+ |module |1 |1 |= |0.00 |0.00 | +---------+-------+-----------+-----------+------------+---------+ |class |2 |2 |= |100.00 |0.00 | +---------+-------+-----------+-----------+------------+---------+ |method |1 |1 |= |100.00 |0.00 | +---------+-------+-----------+-----------+------------+---------+ |function |6 |6 |= |50.00 |33.33 | +————+-------+-----------+-----------+------------+---------+ !!!!!!!!!!
  49. 49. Test Automation
  50. 50. Why do you need automated tests? ! Optimise speed, efficiency and quality Catch the bugs upfront Safety net for refactoring
  51. 51. On the Road to Continuous Integration
  52. 52. Several “Continuous” processes Continuous Integration Continuous Deployment Continuous Delivery
  53. 53. Several platforms for CI Jenkins Travis CI Circle CI Shippable BuildBot
  54. 54. Continuous Integration Flow Feature Branch Develop Branch Git Push Updates Github PR Several jobs: - pep8 compliance - unit test - integration test - end to end test Merge PR Pull Request Tag Deployment Code Review
  55. 55. Jenkins dashboard
  56. 56. Automated Deployment
  57. 57. Pick the right tool to deploy AWS ElasticBeanstalk AWS OpsWorks Chef Puppet SaltStack Ansible
  58. 58. Ansible http://terry.im/wiki/terry/attachments/29786135/37552129.png
  59. 59. It works only on my machine
  60. 60. The Docker Project Run everywhere Run everything Advantages: • Pricing • Speed • Unifying environments • Testability / reproducibility • Easily add/replace components
  61. 61. Docker on your Mac http://www.warski.org/blog/2014/05/spray-server-docker-container/docker-stack-5/
  62. 62. Production Monitoring
  63. 63. Production Monitoring AWS CloudWatch End-to-end tests running in Jenkins StackDriver Log analysis (streams)
  64. 64. Questions?
  65. 65. Python developers WANTED Contact: dmelamed@cloudlock.com

×