SlideShare a Scribd company logo
1 of 24
Download to read offline
© 2024
A Guide to a Modern
Project Setup
PyConDE / PyData 2024, April 22nd
Florian Wilhelm
Streamlining
Python
Development
Mathematical Modelling
Modern Data Warehousing & Analytics
Personalisation & RecSys
Uncertainty Quantification & Causality
Python Data Stack
OSS Contributor & Creator of PyScaffold
Dr. Florian Wilhelm
• HEAD OF DATA SCIENCE
FlorianWilhelm.info
florian.wilhelm@inovex.de
FlorianWilhelm
‣ Application Development (Web Platforms, Mobile
Apps, Smart Devices and Robotics, UI/UX
design,Backend Services)
‣ Data Management and Analytics (Business
Intelligence, Big Data, Searches, Data Science
and Deep Learning, Machine Perception and
Artificial Intelligence)
‣ Scalable IT-Infrastructures (IT Engineering, Cloud
Services, DevOps, Replatforming, Security)
‣ Training and Coaching (inovex Academy)
is an innovation and quality-driven
IT project house with a focus on
digital transformation.
Using technology to
inspire our clients.
And ourselves.
Berlin · Karlsruhe · Pforzheim · Stuttgart · München · Köln · Hamburg · Erlangen
www.inovex.de
1. Introduction:
a. What makes a good project setup?
b. How do we achieve it?
2. Streamlined Project Setup:
a. configuration with pyproject.toml
b. tooling with hatch, ruff, mypy, pytest, …
3. Conclusion
Agenda
Introduction
1. efficient development
2. easy collaboration
3. seamless build & deployment
What makes a streamlined Python Project Setup?
1. Conventions
a. project structure
b. code formatting, e.g., pep8, black, ruff
c. documentation, e.g., Sphinx, mkdocs
2. Automation
a. dependency & environment management
b. building & publishing
c. versioning, e.g., semantic versioning
d. testing, linting/formatting, type checking
3. Easy to Use!
Concrete Requirements for those Goals
Semantic Versioning
‣ tells developers what to
expect
‣ avoids dependency hell for
developers using your
software
‣ necessary for requirement
specifiers like ~= 2.21 or
^2.2.21 (Poetry only)
More Details: https://www.geeksforgeeks.org/introduction-semantic-versioning/ and https://semver.org/
This is not a talk about the best Package Management Tool
Source: An unbiased evaluation of environment management and packaging tools (https://www.inovex.de/de/blog/)
Streamlined Project Setup
‣ reproducibly building & publishing packages
‣ robust environment management with support for
custom scripts
‣ easy Python management, replacing pyenv
‣ easy semantic versioning based on Git tags
‣ sophisticated testing within various environments,
replacing tox
🐣 Hatch, the extensible Python project manager
Ofek Lev
‣ folders for
∙ source files
∙ documentation
∙ tests
‣ human-readable information
∙ README.md
∙ …
‣ configuration files
∙ pyproject.toml
∙ …
Project Directory Structure
‣ defines the build system
‣ metadata about your project
for PyPI
‣ configuration for (almost) all
tools
∙ pytest
∙ mypy
∙ ruff
∙ coverage
All-in-One Configuration with pyproject.toml
Scripts in pyproject.toml for automation of tasks, e.g.
∙ running unit-tests with our without coverage, debugging,
∙ building the documentation,
∙ running the linters, code checks, mypy,
∙ …
Automation with Scripts!
> hatch run test:cov
‣ replaces tons of tools
‣ easy configuration via
pyproject.toml
‣ extremely fast
‣ over 700 plugins
Code Quality: Linting & Formatting
Ruff
flake8
autoflake
pydocstyle
…
Why mypy?
Type Checking: Are you my type?
compile-time type checking finds many errors in
advance, often edge cases.
type declaration act as machine-checked
documentation, thus enhancing the dev
experience.
Mypy Example
> hatch run lint:typing
pytest
‣ defacto standard for unit testing
‣ powerful features like fixtures, etc.
‣ tons of useful plugins, e.g.:
∙ pytest-cov for coverage
∙ pytest-recording for mocking calls to external services
∙ pytest-sugar to make it easier on the eyes
Testing with pytest & hatch
hatch & tox
‣ isolated environments for testing different Python versions and
dependency combinations
Avoiding human-errors by automated checks on every git commit
Automated QA with pre-commit
‣ Automatic and reproducible testing
‣ Publishing packages based on git tags
‣ Established branching strategy, e.g. GithubFlow
for efficient collaboration
‣ Scalability and Adaptability when needed
‣ Automated deployments, building of
documentation etc.
Automation with CI/CD
More Details: Data Science in Production: Packaging, Versioning and Continuous Integration (https://www.inovex.de/de/blog/)
Conclusion
‣ unified configuration in pyproject.toml
‣ standardized folder structure with
src-layout and useful README.md
‣ easy package management and
automation with hatch
‣ automated QA with ruff, pytest,
pre-commit, mypy, CI/CD
‣ proper documentation with mkdocs
‣ automation & conventions are key!
https://github.com/FlorianWilhelm/the-hatchlor
Check out the Hatchlor!
⭐
CHEERS TO THE COMMUNITY
Credits & Resources
‣ Ofek Lev, the creator of hatch, for is
awesome work in his spare time ❤
‣ Michael Hofmann from inovex who
made these awesome slides
© 2023
Thank you!
Dr. Florian Wilhelm
Head of Data Science
PyConDE / PyData 2024
inovex.de
florian.wilhelm@inovex.de
@inovexlife
@inovexgmbh

More Related Content

What's hot

What is Continuous Integration? | Continuous Integration with Jenkins | DevOp...
What is Continuous Integration? | Continuous Integration with Jenkins | DevOp...What is Continuous Integration? | Continuous Integration with Jenkins | DevOp...
What is Continuous Integration? | Continuous Integration with Jenkins | DevOp...Edureka!
 
DevOps on AWS: DevOps Day San Francisco
DevOps on AWS: DevOps Day San FranciscoDevOps on AWS: DevOps Day San Francisco
DevOps on AWS: DevOps Day San FranciscoAmazon Web Services
 
Version Control History and Git Basics
Version Control History and Git BasicsVersion Control History and Git Basics
Version Control History and Git BasicsSreedath N S
 
Code review vs pull request
Code review vs pull requestCode review vs pull request
Code review vs pull requestBryan Liu
 
Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...
Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...
Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...Edureka!
 

What's hot (7)

Deep Dive on Serverless Stack
Deep Dive on Serverless StackDeep Dive on Serverless Stack
Deep Dive on Serverless Stack
 
What is Continuous Integration? | Continuous Integration with Jenkins | DevOp...
What is Continuous Integration? | Continuous Integration with Jenkins | DevOp...What is Continuous Integration? | Continuous Integration with Jenkins | DevOp...
What is Continuous Integration? | Continuous Integration with Jenkins | DevOp...
 
DevOps on AWS: DevOps Day San Francisco
DevOps on AWS: DevOps Day San FranciscoDevOps on AWS: DevOps Day San Francisco
DevOps on AWS: DevOps Day San Francisco
 
Version Control History and Git Basics
Version Control History and Git BasicsVersion Control History and Git Basics
Version Control History and Git Basics
 
Code review vs pull request
Code review vs pull requestCode review vs pull request
Code review vs pull request
 
Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...
Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...
Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...
 
Git presentation
Git presentationGit presentation
Git presentation
 

Similar to Streamlining Python Development: A Guide to a Modern Project Setup

DevOps & DevSecOps in Swiss Banking
DevOps & DevSecOps in Swiss BankingDevOps & DevSecOps in Swiss Banking
DevOps & DevSecOps in Swiss BankingAarno Aukia
 
DevSecOps - Security in DevOps
DevSecOps - Security in DevOpsDevSecOps - Security in DevOps
DevSecOps - Security in DevOpsAarno Aukia
 
Wie macht man aus Software einen Online-Service in der Cloud
Wie macht man aus Software einen Online-Service in der CloudWie macht man aus Software einen Online-Service in der Cloud
Wie macht man aus Software einen Online-Service in der CloudAarno Aukia
 
Secrets of Successful Cloud Foundry Adopters
Secrets of Successful Cloud Foundry AdoptersSecrets of Successful Cloud Foundry Adopters
Secrets of Successful Cloud Foundry AdoptersVMware Tanzu
 
Velocity NY 2018 "The Cloud Native Developer Workflow"
Velocity NY 2018 "The Cloud Native Developer Workflow"Velocity NY 2018 "The Cloud Native Developer Workflow"
Velocity NY 2018 "The Cloud Native Developer Workflow"Daniel Bryant
 
microXchg 2019: "Creating an Effective Developer Experience for Cloud-Native ...
microXchg 2019: "Creating an Effective Developer Experience for Cloud-Native ...microXchg 2019: "Creating an Effective Developer Experience for Cloud-Native ...
microXchg 2019: "Creating an Effective Developer Experience for Cloud-Native ...Daniel Bryant
 
Cloud Native Development
Cloud Native DevelopmentCloud Native Development
Cloud Native DevelopmentManuel Garcia
 
Weave GitOps - continuous delivery for any Kubernetes
Weave GitOps - continuous delivery for any KubernetesWeave GitOps - continuous delivery for any Kubernetes
Weave GitOps - continuous delivery for any KubernetesWeaveworks
 
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...Vietnam Open Infrastructure User Group
 
Next Level DevOps Implementation with GitOps
Next Level DevOps Implementation with GitOpsNext Level DevOps Implementation with GitOps
Next Level DevOps Implementation with GitOpsRamadoni Ashudi
 
DevOps LA Meetup Intro to Habitat
DevOps LA Meetup Intro to HabitatDevOps LA Meetup Intro to Habitat
DevOps LA Meetup Intro to HabitatJessica DeVita
 
Is Automation Necessary for the CC Survival?
Is Automation Necessary for the CC Survival?Is Automation Necessary for the CC Survival?
Is Automation Necessary for the CC Survival?Javier Tallón
 
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with ConcourseContinuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with ConcourseVMware Tanzu
 
DevSecOps: Bringing security to the DevOps pipeline
DevSecOps: Bringing security to the DevOps pipelineDevSecOps: Bringing security to the DevOps pipeline
DevSecOps: Bringing security to the DevOps pipelineAarno Aukia
 
Next gen software operations models in the cloud
Next gen software operations models in the cloudNext gen software operations models in the cloud
Next gen software operations models in the cloudAarno Aukia
 
DevSecOps: Bringing security to the DevOps pipeline
DevSecOps: Bringing security to the DevOps pipelineDevSecOps: Bringing security to the DevOps pipeline
DevSecOps: Bringing security to the DevOps pipelineAarno Aukia
 
Continues Integration and Continuous Delivery with Azure DevOps - Deploy Anyt...
Continues Integration and Continuous Delivery with Azure DevOps - Deploy Anyt...Continues Integration and Continuous Delivery with Azure DevOps - Deploy Anyt...
Continues Integration and Continuous Delivery with Azure DevOps - Deploy Anyt...Janusz Nowak
 
Continuous Lifecycle London 2018 Event Keynote
Continuous Lifecycle London 2018 Event KeynoteContinuous Lifecycle London 2018 Event Keynote
Continuous Lifecycle London 2018 Event KeynoteWeaveworks
 
Top 10 Python Frameworks for App Development
Top 10 Python Frameworks for App DevelopmentTop 10 Python Frameworks for App Development
Top 10 Python Frameworks for App DevelopmentKateWood30
 

Similar to Streamlining Python Development: A Guide to a Modern Project Setup (20)

DevOps & DevSecOps in Swiss Banking
DevOps & DevSecOps in Swiss BankingDevOps & DevSecOps in Swiss Banking
DevOps & DevSecOps in Swiss Banking
 
DevSecOps - Security in DevOps
DevSecOps - Security in DevOpsDevSecOps - Security in DevOps
DevSecOps - Security in DevOps
 
Wie macht man aus Software einen Online-Service in der Cloud
Wie macht man aus Software einen Online-Service in der CloudWie macht man aus Software einen Online-Service in der Cloud
Wie macht man aus Software einen Online-Service in der Cloud
 
Secrets of Successful Cloud Foundry Adopters
Secrets of Successful Cloud Foundry AdoptersSecrets of Successful Cloud Foundry Adopters
Secrets of Successful Cloud Foundry Adopters
 
Velocity NY 2018 "The Cloud Native Developer Workflow"
Velocity NY 2018 "The Cloud Native Developer Workflow"Velocity NY 2018 "The Cloud Native Developer Workflow"
Velocity NY 2018 "The Cloud Native Developer Workflow"
 
microXchg 2019: "Creating an Effective Developer Experience for Cloud-Native ...
microXchg 2019: "Creating an Effective Developer Experience for Cloud-Native ...microXchg 2019: "Creating an Effective Developer Experience for Cloud-Native ...
microXchg 2019: "Creating an Effective Developer Experience for Cloud-Native ...
 
Cloud Native Development
Cloud Native DevelopmentCloud Native Development
Cloud Native Development
 
Weave GitOps - continuous delivery for any Kubernetes
Weave GitOps - continuous delivery for any KubernetesWeave GitOps - continuous delivery for any Kubernetes
Weave GitOps - continuous delivery for any Kubernetes
 
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
 
Next Level DevOps Implementation with GitOps
Next Level DevOps Implementation with GitOpsNext Level DevOps Implementation with GitOps
Next Level DevOps Implementation with GitOps
 
DevOps LA Meetup Intro to Habitat
DevOps LA Meetup Intro to HabitatDevOps LA Meetup Intro to Habitat
DevOps LA Meetup Intro to Habitat
 
Is Automation Necessary for the CC Survival?
Is Automation Necessary for the CC Survival?Is Automation Necessary for the CC Survival?
Is Automation Necessary for the CC Survival?
 
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with ConcourseContinuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
Continuous Delivery: Fly the Friendly CI in Pivotal Cloud Foundry with Concourse
 
DevSecOps: Bringing security to the DevOps pipeline
DevSecOps: Bringing security to the DevOps pipelineDevSecOps: Bringing security to the DevOps pipeline
DevSecOps: Bringing security to the DevOps pipeline
 
Next gen software operations models in the cloud
Next gen software operations models in the cloudNext gen software operations models in the cloud
Next gen software operations models in the cloud
 
DevSecOps: Bringing security to the DevOps pipeline
DevSecOps: Bringing security to the DevOps pipelineDevSecOps: Bringing security to the DevOps pipeline
DevSecOps: Bringing security to the DevOps pipeline
 
Continues Integration and Continuous Delivery with Azure DevOps - Deploy Anyt...
Continues Integration and Continuous Delivery with Azure DevOps - Deploy Anyt...Continues Integration and Continuous Delivery with Azure DevOps - Deploy Anyt...
Continues Integration and Continuous Delivery with Azure DevOps - Deploy Anyt...
 
DevOps-Roadmap
DevOps-RoadmapDevOps-Roadmap
DevOps-Roadmap
 
Continuous Lifecycle London 2018 Event Keynote
Continuous Lifecycle London 2018 Event KeynoteContinuous Lifecycle London 2018 Event Keynote
Continuous Lifecycle London 2018 Event Keynote
 
Top 10 Python Frameworks for App Development
Top 10 Python Frameworks for App DevelopmentTop 10 Python Frameworks for App Development
Top 10 Python Frameworks for App Development
 

More from Florian Wilhelm

Unlocking the Power of Integer Programming
Unlocking the Power of Integer ProgrammingUnlocking the Power of Integer Programming
Unlocking the Power of Integer ProgrammingFlorian Wilhelm
 
WALD: A Modern & Sustainable Analytics Stack
WALD: A Modern & Sustainable Analytics StackWALD: A Modern & Sustainable Analytics Stack
WALD: A Modern & Sustainable Analytics StackFlorian Wilhelm
 
Forget about AI and do Mathematical Modelling instead!
Forget about AI and do Mathematical Modelling instead!Forget about AI and do Mathematical Modelling instead!
Forget about AI and do Mathematical Modelling instead!Florian Wilhelm
 
An Interpretable Model for Collaborative Filtering Using an Extended Latent D...
An Interpretable Model for Collaborative Filtering Using an Extended Latent D...An Interpretable Model for Collaborative Filtering Using an Extended Latent D...
An Interpretable Model for Collaborative Filtering Using an Extended Latent D...Florian Wilhelm
 
Honey I Shrunk the Target Variable! Common pitfalls when transforming the tar...
Honey I Shrunk the Target Variable! Common pitfalls when transforming the tar...Honey I Shrunk the Target Variable! Common pitfalls when transforming the tar...
Honey I Shrunk the Target Variable! Common pitfalls when transforming the tar...Florian Wilhelm
 
Matrix Factorization for Collaborative Filtering Is Just Solving an Adjoint L...
Matrix Factorization for Collaborative Filtering Is Just Solving an Adjoint L...Matrix Factorization for Collaborative Filtering Is Just Solving an Adjoint L...
Matrix Factorization for Collaborative Filtering Is Just Solving an Adjoint L...Florian Wilhelm
 
Uncertainty Quantification in AI
Uncertainty Quantification in AIUncertainty Quantification in AI
Uncertainty Quantification in AIFlorian Wilhelm
 
Performance evaluation of GANs in a semisupervised OCR use case
Performance evaluation of GANs in a semisupervised OCR use casePerformance evaluation of GANs in a semisupervised OCR use case
Performance evaluation of GANs in a semisupervised OCR use caseFlorian Wilhelm
 
Bridging the Gap: from Data Science to Production
Bridging the Gap: from Data Science to ProductionBridging the Gap: from Data Science to Production
Bridging the Gap: from Data Science to ProductionFlorian Wilhelm
 
How mobile.de brings Data Science to Production for a Personalized Web Experi...
How mobile.de brings Data Science to Production for a Personalized Web Experi...How mobile.de brings Data Science to Production for a Personalized Web Experi...
How mobile.de brings Data Science to Production for a Personalized Web Experi...Florian Wilhelm
 
Deep Learning-based Recommendations for Germany's Biggest Vehicle Marketplace
Deep Learning-based Recommendations for Germany's Biggest Vehicle MarketplaceDeep Learning-based Recommendations for Germany's Biggest Vehicle Marketplace
Deep Learning-based Recommendations for Germany's Biggest Vehicle MarketplaceFlorian Wilhelm
 
Deep Learning-based Recommendations for Germany's Biggest Online Vehicle Mark...
Deep Learning-based Recommendations for Germany's Biggest Online Vehicle Mark...Deep Learning-based Recommendations for Germany's Biggest Online Vehicle Mark...
Deep Learning-based Recommendations for Germany's Biggest Online Vehicle Mark...Florian Wilhelm
 
Declarative Thinking and Programming
Declarative Thinking and ProgrammingDeclarative Thinking and Programming
Declarative Thinking and ProgrammingFlorian Wilhelm
 
Which car fits my life? - PyData Berlin 2017
Which car fits my life? - PyData Berlin 2017Which car fits my life? - PyData Berlin 2017
Which car fits my life? - PyData Berlin 2017Florian Wilhelm
 
PyData Meetup Berlin 2017-04-19
PyData Meetup Berlin 2017-04-19PyData Meetup Berlin 2017-04-19
PyData Meetup Berlin 2017-04-19Florian Wilhelm
 
Explaining the idea behind automatic relevance determination and bayesian int...
Explaining the idea behind automatic relevance determination and bayesian int...Explaining the idea behind automatic relevance determination and bayesian int...
Explaining the idea behind automatic relevance determination and bayesian int...Florian Wilhelm
 

More from Florian Wilhelm (16)

Unlocking the Power of Integer Programming
Unlocking the Power of Integer ProgrammingUnlocking the Power of Integer Programming
Unlocking the Power of Integer Programming
 
WALD: A Modern & Sustainable Analytics Stack
WALD: A Modern & Sustainable Analytics StackWALD: A Modern & Sustainable Analytics Stack
WALD: A Modern & Sustainable Analytics Stack
 
Forget about AI and do Mathematical Modelling instead!
Forget about AI and do Mathematical Modelling instead!Forget about AI and do Mathematical Modelling instead!
Forget about AI and do Mathematical Modelling instead!
 
An Interpretable Model for Collaborative Filtering Using an Extended Latent D...
An Interpretable Model for Collaborative Filtering Using an Extended Latent D...An Interpretable Model for Collaborative Filtering Using an Extended Latent D...
An Interpretable Model for Collaborative Filtering Using an Extended Latent D...
 
Honey I Shrunk the Target Variable! Common pitfalls when transforming the tar...
Honey I Shrunk the Target Variable! Common pitfalls when transforming the tar...Honey I Shrunk the Target Variable! Common pitfalls when transforming the tar...
Honey I Shrunk the Target Variable! Common pitfalls when transforming the tar...
 
Matrix Factorization for Collaborative Filtering Is Just Solving an Adjoint L...
Matrix Factorization for Collaborative Filtering Is Just Solving an Adjoint L...Matrix Factorization for Collaborative Filtering Is Just Solving an Adjoint L...
Matrix Factorization for Collaborative Filtering Is Just Solving an Adjoint L...
 
Uncertainty Quantification in AI
Uncertainty Quantification in AIUncertainty Quantification in AI
Uncertainty Quantification in AI
 
Performance evaluation of GANs in a semisupervised OCR use case
Performance evaluation of GANs in a semisupervised OCR use casePerformance evaluation of GANs in a semisupervised OCR use case
Performance evaluation of GANs in a semisupervised OCR use case
 
Bridging the Gap: from Data Science to Production
Bridging the Gap: from Data Science to ProductionBridging the Gap: from Data Science to Production
Bridging the Gap: from Data Science to Production
 
How mobile.de brings Data Science to Production for a Personalized Web Experi...
How mobile.de brings Data Science to Production for a Personalized Web Experi...How mobile.de brings Data Science to Production for a Personalized Web Experi...
How mobile.de brings Data Science to Production for a Personalized Web Experi...
 
Deep Learning-based Recommendations for Germany's Biggest Vehicle Marketplace
Deep Learning-based Recommendations for Germany's Biggest Vehicle MarketplaceDeep Learning-based Recommendations for Germany's Biggest Vehicle Marketplace
Deep Learning-based Recommendations for Germany's Biggest Vehicle Marketplace
 
Deep Learning-based Recommendations for Germany's Biggest Online Vehicle Mark...
Deep Learning-based Recommendations for Germany's Biggest Online Vehicle Mark...Deep Learning-based Recommendations for Germany's Biggest Online Vehicle Mark...
Deep Learning-based Recommendations for Germany's Biggest Online Vehicle Mark...
 
Declarative Thinking and Programming
Declarative Thinking and ProgrammingDeclarative Thinking and Programming
Declarative Thinking and Programming
 
Which car fits my life? - PyData Berlin 2017
Which car fits my life? - PyData Berlin 2017Which car fits my life? - PyData Berlin 2017
Which car fits my life? - PyData Berlin 2017
 
PyData Meetup Berlin 2017-04-19
PyData Meetup Berlin 2017-04-19PyData Meetup Berlin 2017-04-19
PyData Meetup Berlin 2017-04-19
 
Explaining the idea behind automatic relevance determination and bayesian int...
Explaining the idea behind automatic relevance determination and bayesian int...Explaining the idea behind automatic relevance determination and bayesian int...
Explaining the idea behind automatic relevance determination and bayesian int...
 

Recently uploaded

TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024Stephen Perrenod
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfFIDO Alliance
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...ScyllaDB
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024Lorenzo Miniero
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentationyogeshlabana357357
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdfMuhammad Subhan
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform EngineeringMarcus Vechiato
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfalexjohnson7307
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewDianaGray10
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxFIDO Alliance
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!Memoori
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...ScyllaDB
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch TuesdayIvanti
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxFIDO Alliance
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Skynet Technologies
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 

Recently uploaded (20)

TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 

Streamlining Python Development: A Guide to a Modern Project Setup

  • 1. © 2024 A Guide to a Modern Project Setup PyConDE / PyData 2024, April 22nd Florian Wilhelm Streamlining Python Development
  • 2. Mathematical Modelling Modern Data Warehousing & Analytics Personalisation & RecSys Uncertainty Quantification & Causality Python Data Stack OSS Contributor & Creator of PyScaffold Dr. Florian Wilhelm • HEAD OF DATA SCIENCE FlorianWilhelm.info florian.wilhelm@inovex.de FlorianWilhelm
  • 3. ‣ Application Development (Web Platforms, Mobile Apps, Smart Devices and Robotics, UI/UX design,Backend Services) ‣ Data Management and Analytics (Business Intelligence, Big Data, Searches, Data Science and Deep Learning, Machine Perception and Artificial Intelligence) ‣ Scalable IT-Infrastructures (IT Engineering, Cloud Services, DevOps, Replatforming, Security) ‣ Training and Coaching (inovex Academy) is an innovation and quality-driven IT project house with a focus on digital transformation. Using technology to inspire our clients. And ourselves. Berlin · Karlsruhe · Pforzheim · Stuttgart · München · Köln · Hamburg · Erlangen www.inovex.de
  • 4. 1. Introduction: a. What makes a good project setup? b. How do we achieve it? 2. Streamlined Project Setup: a. configuration with pyproject.toml b. tooling with hatch, ruff, mypy, pytest, … 3. Conclusion Agenda
  • 6. 1. efficient development 2. easy collaboration 3. seamless build & deployment What makes a streamlined Python Project Setup?
  • 7. 1. Conventions a. project structure b. code formatting, e.g., pep8, black, ruff c. documentation, e.g., Sphinx, mkdocs 2. Automation a. dependency & environment management b. building & publishing c. versioning, e.g., semantic versioning d. testing, linting/formatting, type checking 3. Easy to Use! Concrete Requirements for those Goals
  • 8. Semantic Versioning ‣ tells developers what to expect ‣ avoids dependency hell for developers using your software ‣ necessary for requirement specifiers like ~= 2.21 or ^2.2.21 (Poetry only) More Details: https://www.geeksforgeeks.org/introduction-semantic-versioning/ and https://semver.org/
  • 9. This is not a talk about the best Package Management Tool Source: An unbiased evaluation of environment management and packaging tools (https://www.inovex.de/de/blog/)
  • 11. ‣ reproducibly building & publishing packages ‣ robust environment management with support for custom scripts ‣ easy Python management, replacing pyenv ‣ easy semantic versioning based on Git tags ‣ sophisticated testing within various environments, replacing tox 🐣 Hatch, the extensible Python project manager Ofek Lev
  • 12. ‣ folders for ∙ source files ∙ documentation ∙ tests ‣ human-readable information ∙ README.md ∙ … ‣ configuration files ∙ pyproject.toml ∙ … Project Directory Structure
  • 13. ‣ defines the build system ‣ metadata about your project for PyPI ‣ configuration for (almost) all tools ∙ pytest ∙ mypy ∙ ruff ∙ coverage All-in-One Configuration with pyproject.toml
  • 14. Scripts in pyproject.toml for automation of tasks, e.g. ∙ running unit-tests with our without coverage, debugging, ∙ building the documentation, ∙ running the linters, code checks, mypy, ∙ … Automation with Scripts! > hatch run test:cov
  • 15. ‣ replaces tons of tools ‣ easy configuration via pyproject.toml ‣ extremely fast ‣ over 700 plugins Code Quality: Linting & Formatting Ruff flake8 autoflake pydocstyle …
  • 16. Why mypy? Type Checking: Are you my type? compile-time type checking finds many errors in advance, often edge cases. type declaration act as machine-checked documentation, thus enhancing the dev experience.
  • 17. Mypy Example > hatch run lint:typing
  • 18. pytest ‣ defacto standard for unit testing ‣ powerful features like fixtures, etc. ‣ tons of useful plugins, e.g.: ∙ pytest-cov for coverage ∙ pytest-recording for mocking calls to external services ∙ pytest-sugar to make it easier on the eyes Testing with pytest & hatch hatch & tox ‣ isolated environments for testing different Python versions and dependency combinations
  • 19. Avoiding human-errors by automated checks on every git commit Automated QA with pre-commit
  • 20. ‣ Automatic and reproducible testing ‣ Publishing packages based on git tags ‣ Established branching strategy, e.g. GithubFlow for efficient collaboration ‣ Scalability and Adaptability when needed ‣ Automated deployments, building of documentation etc. Automation with CI/CD More Details: Data Science in Production: Packaging, Versioning and Continuous Integration (https://www.inovex.de/de/blog/)
  • 21. Conclusion ‣ unified configuration in pyproject.toml ‣ standardized folder structure with src-layout and useful README.md ‣ easy package management and automation with hatch ‣ automated QA with ruff, pytest, pre-commit, mypy, CI/CD ‣ proper documentation with mkdocs ‣ automation & conventions are key!
  • 23. CHEERS TO THE COMMUNITY Credits & Resources ‣ Ofek Lev, the creator of hatch, for is awesome work in his spare time ❤ ‣ Michael Hofmann from inovex who made these awesome slides
  • 24. © 2023 Thank you! Dr. Florian Wilhelm Head of Data Science PyConDE / PyData 2024 inovex.de florian.wilhelm@inovex.de @inovexlife @inovexgmbh