Downloaded 48 times
Uploaded byDavid Melamed
Python - code quality and production monitoring
This document summarizes best practices for code quality and monitoring production systems. It discusses what constitutes good code through practices like coding standards, code reviews, testing and architecture. It also covers continuous integration, deployment and monitoring through tools like Jenkins, Docker and AWS CloudWatch. The presenter is a developer from Cloudlock who advocates for code quality and shares their experiences.
More Related Content
Similar to Python - code quality and production monitoring
Python - code quality and production monitoring
- 1. Code Quality toMonitoring production: use cases and best practices David Melamed Pyweb meetup - September 21, 2014
- 2. Agenda What isgood code? How to measure quality code? Continuous Integration How to efficiently monitor production?
- 3. About me Full-stackdeveloper at Cloudlock Fond of technology & automation Code Quality Evangelist Geek on my free time
- 4. About Cloudlock Founded:2011 Headquarters: Waltham, Mass. (U.S.) R&D Headquarters: Tel Aviv Employees: 80+ Funding: $28M+ Investors Selected customers 700+ Customers 9,600 Unique Apps Discovered Native Cloud Security Solutions for SaaS Applications 5 million Users under Management 13 billion objects processed daily
- 5. What is goodcode?
- 6. Good code is… Easy to read & understand Easy to extend Easy to maintain Testable
- 7. How to writegood code?
- 8.
- 9. Good code requiresconstant REFACTORING
- 10. The Code QualityStar Coding Standards Code Review Automated Tests Code Complexity Architecture Design Source
- 11. Code should beeasy to read
- 13. Code should bereadable Avoid comments but avoid obvious ones (the code should be self-explanatory) Avoid comments too verbose Comments need to be maintained
- 14.
- 15. The Zen ofPython (PEP20) Beautiful is better than ugly Explicit is better than implicit Simple is better than complex Complex is better than complicated Readability counts
- 16. Coding Style: ReadabilityCounts “Programs must be written for people to read, and only incidentally for machines to execute.” Abelson & Sussman, Structure and Interpretation of Computer Programs
- 17. PEP8: Style guidefor Python Indentation Maximum line length Blank lines Redundant white spaces Imports Comments Naming conventions …
- 18. How to check/enforce PEP8 Pycharm (Tools > Flake8) Git pre-commit hook (http://goo.gl/bteYkq) Jenkins job for continuous integration
- 19.
- 22. Naming best practises Adopt a convention and stick with it (fetch vs. retrieve vs. get) Use intention-revealing names Avoid misleading readers Make meaningful distinctions Use searchable names Avoid mental mapping Use domain names From Clean code, R. Martin
- 23.
- 24. Best Practices inCode Architecture OOP DRY SOLID Loose Coupling
- 25. Don’t Repeat Yourself(aka DRY) http://www.sonarqube.org/manage-duplicated-code-with-sonar/
- 26. SOLID principles •SRP: Single Responsibility Principle • OCP: Open/closed principle • LSP: Liskov substitution principle • ISP: Interface segregation principle • DIP: dependency inversion principle
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33. Main principles Splitcomponents into repositories Web app: use clean layers view / endpoint (presentation) service (business logic) DAO (model) Use message bus for communication inter-components (i.e. RabbitMQ) Use daemons / workers for long-running tasks Use events / signals to decouple unrelated actions Microservices
- 35. Architecture at Cloudlock Nginx AWS VPC Web Server API Server Message Bus (RabbitMQ) RDS PostgreSQL ElastiCache Redis Authentication Gateway Worker Worker Worker Logs
- 36.
- 38. Tools for codereview Github Pull Requests Review Board Gerrit Crucible
- 39.
- 41.
- 42. Different types oftests Manual Test Unit Test Integration Test System Test Acceptance/Regression Test (end-to-end) Smoke Test
- 43. Unit Test Testif small, distinct and isolated portions of code are working as expected, provided some input Mock your dependencies (database, logger, service) Should be very fast No need to test EVERYTHING (i.e. private methods) Frameworks: unittest, pytest and nose
- 45. Pytest No needfor base class (works both for classes and methods) More pythonic than unittest Fixtures (conftest.py) Parametrized tests Markers Various plugins for coverage, bdd, django, twisted… Used for unit tests, integration tests and end to end tests
- 46. Integration Tests Testcommunication between services You may mock some of the dependencies depending on what you test
- 47. System Tests Globaltest, more functional of the whole system TestApp (Pyramid app) using a test database Call the API and test the expected response code
- 48. End-to-end tests Splinter:web driver abstraction (firefox, selenium) splinter-pytest: fixtures for browser Plugin for rerun flaky tests, tests with steps Use Selenium Grid to test several browsers
- 49. Smoke Tests Makesure the system is alive Tests that the components are working Tests that the components can communicate Health check / status page
- 50.
- 52. Code Quality Metrics Code Standards Violations (pylint /flake8/pyflakes) LOC (Lines Of Code) Cyclomatic Complexity Unit Tests Coverage
- 53. Code Coverage Makessense only for unit tests Calculates the coverage based on the paths run by the tests 100% coverage is not a goal on its own May be misleading regarding the code quality
- 54. Unit Test DiffCoverage
- 55. Mutation Analysis FromMutation analysis vs. code coverage in automated assessment of students’ Testing Skills (P. Ihantola)
- 56. Pylint Report ====== 85 statements analysed. ! Duplication ----------- ! +-------------------------+------+---------+-----------+ | |now |previous |difference | +=========================+======+=========+===========+ |nb duplicated lines |0 |0 |= | +-------------------------+------+---------+-----------+ |percent duplicated lines |0.000 |0.000 |= | +-------------------------+------+---------+-----------+ !!! Messages by category -------------------- ! +-----------+-------+---------+-----------+ |type |number |previous |difference | +===========+=======+=========+===========+ |convention |18 |18 |= | +-----------+-------+---------+-----------+ |refactor |2 |2 |= | +-----------+-------+---------+-----------+ |warning |1 |1 |= | +-----------+-------+---------+-----------+ |error |4 |4 |= | +-----------+-------+---------+-----------+ !!! Messages -------- ! +-----------------------------+------------+ |message id |occurrences | +=============================+============+ |line-too-long |10 | +-----------------------------+------------+ |no-name-in-module |4 | +-----------------------------+------------+ |missing-docstring |4 | +-----------------------------+------------+ Global evaluation ----------------- Your code has been rated at 5.18/10 (previous run: 5.18/10, +0.00) ! Raw metrics ----------- ! +----------+-------+------+---------+-----------+ |type |number |% |previous |difference | +==========+=======+======+=========+===========+ |code |90 |63.83 |90 |= | +----------+-------+------+---------+-----------+ |docstring |30 |21.28 |30 |= | +----------+-------+------+---------+-----------+ |comment |5 |3.55 |5 |= | +----------+-------+------+---------+-----------+ |empty |16 |11.35 |16 |= | +----------+-------+------+---------+-----------+ !!! Statistics by type ------------------ ! +---------+-------+-----------+-----------+------------+---------+ |type |number |old number |difference |%documented |%badname | +=========+=======+===========+===========+============+=========+ |module |1 |1 |= |0.00 |0.00 | +---------+-------+-----------+-----------+------------+---------+ |class |2 |2 |= |100.00 |0.00 | +---------+-------+-----------+-----------+------------+---------+ |method |1 |1 |= |100.00 |0.00 | +---------+-------+-----------+-----------+------------+---------+ |function |6 |6 |= |50.00 |33.33 | +————+-------+-----------+-----------+------------+---------+ !!!!!!!!!!
- 57.
- 58. Why do youneed automated tests? ! Optimise speed, efficiency and quality Catch the bugs upfront Safety net for refactoring
- 60. On the Roadto Continuous Integration
- 61. Several “Continuous” processes Continuous Integration Continuous Deployment Continuous Delivery
- 62. Several platforms forCI Jenkins Travis CI Circle CI Shippable BuildBot
- 63. Continuous Integration Flow Feature Branch Develop Branch Git Push Updates Github PR Several jobs: - pep8 compliance - unit test - integration test - end to end test Merge PR Pull Request Tag Deployment Code Review
- 64.
- 67.
- 68. Pick the righttool to deploy AWS ElasticBeanstalk AWS OpsWorks Chef Puppet SaltStack Ansible
- 69.
- 70. It works onlyon my machine
- 71. The Docker Project Run everywhere Run everything Advantages: • Pricing • Speed • Unifying environments • Testability / reproducibility • Easily add/replace components
- 72. Docker on yourMac http://www.warski.org/blog/2014/05/spray-server-docker-container/docker-stack-5/
- 73.
- 75. Production Monitoring AWSCloudWatch End-to-end tests running in Jenkins StackDriver Log analysis (streams)
- 76.
- 77. Python developers WANTED Contact: dmelamed@cloudlock.com