SlideShare a Scribd company logo
1 of 16
Migrating our monolith to
Python 3
Django Bulgaria Meetup Group
18th of November, 2019
About Me
Ilian Iliev
Platform Engineer at Lifesum
ilian@ilian.io
www.ilian.io
The Status Quo
● One monolith running Python 2.7
● 7 years old codebase
● MySQL, Django, Celery, Redis
● 38M users all around the world
The Plan
● Make the code Python 3 compatible
● Have it run on our test environments for a while
● Do a round of manual testing
● Ship it
I. Set it up
● Not all of our dependencies were Python 3 compatible
● Suds
● Add tests to mock the calls
● Replace it with zeep
● Verify
II. Make it run
● Failing imports
● Local imports
● Six
○ Moved libs
○ six.iteritems
○ Six.text_type
● Setup CircleCI to test both 2.7 and 3.6
III. Make tests pass
● six.iteritems
● six.text_type
● hashlib requires bytes
● exception.message
● xrange
● dict_keys
● filter
● Comparing with None
● round behaving differently
IV. Upgrade migrations & Manual test
● Custom column names existed as bytes in migrations
● Same for choices and help_text
● Some third party apps generated new migrations
● Manual tests
V. Release
VI. Rollback and aftermath
● Sentry could not handle the error load
● Redis is an issue
○ Via Django
○ Direct usage
● Python 3 incompatible code not covered by tests
● filter
● Lesson learned - Monday morning == Friday evening
VI. Second Try
Questions?
Thank you

More Related Content

What's hot

Python fundamental course curriculum
Python fundamental course curriculumPython fundamental course curriculum
Python fundamental course curriculumEngiGyan .
 
Why Rust? - Matthias Endler - Codemotion Amsterdam 2016
Why Rust? - Matthias Endler - Codemotion Amsterdam 2016Why Rust? - Matthias Endler - Codemotion Amsterdam 2016
Why Rust? - Matthias Endler - Codemotion Amsterdam 2016Codemotion
 
Deep drive into rust programming language
Deep drive into rust programming languageDeep drive into rust programming language
Deep drive into rust programming languageVigneshwer Dhinakaran
 
Python training-course-content
Python training-course-contentPython training-course-content
Python training-course-contentTraining Institute
 
TDC2016SP - Groovy como você nunca viu
TDC2016SP - Groovy como você nunca viuTDC2016SP - Groovy como você nunca viu
TDC2016SP - Groovy como você nunca viutdc-globalcode
 
Rust Programming Language
Rust Programming LanguageRust Programming Language
Rust Programming LanguageJaeju Kim
 

What's hot (8)

Python fundamental course curriculum
Python fundamental course curriculumPython fundamental course curriculum
Python fundamental course curriculum
 
I believe in rust
I believe in rustI believe in rust
I believe in rust
 
effective java
effective javaeffective java
effective java
 
Why Rust? - Matthias Endler - Codemotion Amsterdam 2016
Why Rust? - Matthias Endler - Codemotion Amsterdam 2016Why Rust? - Matthias Endler - Codemotion Amsterdam 2016
Why Rust? - Matthias Endler - Codemotion Amsterdam 2016
 
Deep drive into rust programming language
Deep drive into rust programming languageDeep drive into rust programming language
Deep drive into rust programming language
 
Python training-course-content
Python training-course-contentPython training-course-content
Python training-course-content
 
TDC2016SP - Groovy como você nunca viu
TDC2016SP - Groovy como você nunca viuTDC2016SP - Groovy como você nunca viu
TDC2016SP - Groovy como você nunca viu
 
Rust Programming Language
Rust Programming LanguageRust Programming Language
Rust Programming Language
 

Similar to Migrating our monolith to Python 3

Python Linters at Scale.pdf
Python Linters at Scale.pdfPython Linters at Scale.pdf
Python Linters at Scale.pdfJimmy Lai
 
Testing Django APIs
Testing Django APIsTesting Django APIs
Testing Django APIstyomo4ka
 
Continuous delivery of Windows micro services in the cloud
Continuous delivery of Windows micro services in the cloud Continuous delivery of Windows micro services in the cloud
Continuous delivery of Windows micro services in the cloud Owain Perry
 
Launchpad: Lessons Learnt
Launchpad: Lessons LearntLaunchpad: Lessons Learnt
Launchpad: Lessons LearntTim Penhey
 
Migrating python.org to buildbot 9 and python 3
Migrating python.org to buildbot 9 and python 3Migrating python.org to buildbot 9 and python 3
Migrating python.org to buildbot 9 and python 3Craig Rodrigues
 
How we realized SOA by Python at PyCon JP 2015
How we realized SOA by Python at PyCon JP 2015How we realized SOA by Python at PyCon JP 2015
How we realized SOA by Python at PyCon JP 2015hirokiky
 
Introduction to GitHub Actions
Introduction to GitHub ActionsIntroduction to GitHub Actions
Introduction to GitHub ActionsKnoldus Inc.
 
The Five Stages of Enterprise Jupyter Deployment
The Five Stages of Enterprise Jupyter DeploymentThe Five Stages of Enterprise Jupyter Deployment
The Five Stages of Enterprise Jupyter DeploymentFrederick Reiss
 
Python: Thanks for the memories
Python: Thanks for the memoriesPython: Thanks for the memories
Python: Thanks for the memoriesDanil Ineev
 
Programming with Python - Basic
Programming with Python - BasicProgramming with Python - Basic
Programming with Python - BasicMosky Liu
 
Dmytro Dziubenko "Developer's toolchain"
Dmytro Dziubenko "Developer's toolchain"Dmytro Dziubenko "Developer's toolchain"
Dmytro Dziubenko "Developer's toolchain"Fwdays
 
A Python Petting Zoo
A Python Petting ZooA Python Petting Zoo
A Python Petting Zoodevondjones
 
OpenStack Cinder On-Boarding Education - Boston Summit - 2017
OpenStack Cinder On-Boarding Education - Boston Summit - 2017OpenStack Cinder On-Boarding Education - Boston Summit - 2017
OpenStack Cinder On-Boarding Education - Boston Summit - 2017Jay Bryant
 
Spark and S3 with Ryan Blue
Spark and S3 with Ryan BlueSpark and S3 with Ryan Blue
Spark and S3 with Ryan BlueDatabricks
 
GeoServer Developers Workshop
GeoServer Developers WorkshopGeoServer Developers Workshop
GeoServer Developers WorkshopJody Garnett
 
Python and Pytorch tutorial and walkthrough
Python and Pytorch tutorial and walkthroughPython and Pytorch tutorial and walkthrough
Python and Pytorch tutorial and walkthroughgabriellekuruvilla
 

Similar to Migrating our monolith to Python 3 (20)

Python Linters at Scale.pdf
Python Linters at Scale.pdfPython Linters at Scale.pdf
Python Linters at Scale.pdf
 
Testing Django APIs
Testing Django APIsTesting Django APIs
Testing Django APIs
 
Koji and pulp
Koji and pulpKoji and pulp
Koji and pulp
 
Continuous delivery of Windows micro services in the cloud
Continuous delivery of Windows micro services in the cloud Continuous delivery of Windows micro services in the cloud
Continuous delivery of Windows micro services in the cloud
 
Launchpad: Lessons Learnt
Launchpad: Lessons LearntLaunchpad: Lessons Learnt
Launchpad: Lessons Learnt
 
Migrating python.org to buildbot 9 and python 3
Migrating python.org to buildbot 9 and python 3Migrating python.org to buildbot 9 and python 3
Migrating python.org to buildbot 9 and python 3
 
How we realized SOA by Python at PyCon JP 2015
How we realized SOA by Python at PyCon JP 2015How we realized SOA by Python at PyCon JP 2015
How we realized SOA by Python at PyCon JP 2015
 
Go at Skroutz
Go at SkroutzGo at Skroutz
Go at Skroutz
 
Introduction to GitHub Actions
Introduction to GitHub ActionsIntroduction to GitHub Actions
Introduction to GitHub Actions
 
The Five Stages of Enterprise Jupyter Deployment
The Five Stages of Enterprise Jupyter DeploymentThe Five Stages of Enterprise Jupyter Deployment
The Five Stages of Enterprise Jupyter Deployment
 
Python: Thanks for the memories
Python: Thanks for the memoriesPython: Thanks for the memories
Python: Thanks for the memories
 
Programming with Python - Basic
Programming with Python - BasicProgramming with Python - Basic
Programming with Python - Basic
 
Git 1
Git 1Git 1
Git 1
 
Dmytro Dziubenko "Developer's toolchain"
Dmytro Dziubenko "Developer's toolchain"Dmytro Dziubenko "Developer's toolchain"
Dmytro Dziubenko "Developer's toolchain"
 
A Python Petting Zoo
A Python Petting ZooA Python Petting Zoo
A Python Petting Zoo
 
OpenStack Cinder On-Boarding Education - Boston Summit - 2017
OpenStack Cinder On-Boarding Education - Boston Summit - 2017OpenStack Cinder On-Boarding Education - Boston Summit - 2017
OpenStack Cinder On-Boarding Education - Boston Summit - 2017
 
Spark and S3 with Ryan Blue
Spark and S3 with Ryan BlueSpark and S3 with Ryan Blue
Spark and S3 with Ryan Blue
 
GeoServer Developers Workshop
GeoServer Developers WorkshopGeoServer Developers Workshop
GeoServer Developers Workshop
 
Hacktoberfest 2022
Hacktoberfest 2022Hacktoberfest 2022
Hacktoberfest 2022
 
Python and Pytorch tutorial and walkthrough
Python and Pytorch tutorial and walkthroughPython and Pytorch tutorial and walkthrough
Python and Pytorch tutorial and walkthrough
 

Recently uploaded

A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationHelp Desk Migration
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfVictor Lopez
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Gáspár Nagy
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAlluxio, Inc.
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Krakówbim.edu.pl
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...rajkumar669520
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfQ-Advise
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationWave PLM
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAlluxio, Inc.
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfDeskTrack
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfsteffenkarlsson2
 
Studiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting softwareStudiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting softwareinfo611746
 
What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersEmilyJiang23
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfmbmh111980
 
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityAPVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityamy56318795
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024Shane Coughlan
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...Alluxio, Inc.
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems ApproachNeo4j
 

Recently uploaded (20)

A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data Migration
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
 
5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM Integration
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
AI Hackathon.pptx
AI                        Hackathon.pptxAI                        Hackathon.pptx
AI Hackathon.pptx
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
 
Studiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting softwareStudiovity film pre-production and screenwriting software
Studiovity film pre-production and screenwriting software
 
What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java Developers
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
 
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityAPVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 

Migrating our monolith to Python 3

  • 1. Migrating our monolith to Python 3 Django Bulgaria Meetup Group 18th of November, 2019
  • 2. About Me Ilian Iliev Platform Engineer at Lifesum ilian@ilian.io www.ilian.io
  • 3. The Status Quo ● One monolith running Python 2.7 ● 7 years old codebase ● MySQL, Django, Celery, Redis ● 38M users all around the world
  • 4. The Plan ● Make the code Python 3 compatible ● Have it run on our test environments for a while ● Do a round of manual testing ● Ship it
  • 5. I. Set it up ● Not all of our dependencies were Python 3 compatible ● Suds ● Add tests to mock the calls ● Replace it with zeep ● Verify
  • 6. II. Make it run ● Failing imports ● Local imports ● Six ○ Moved libs ○ six.iteritems ○ Six.text_type ● Setup CircleCI to test both 2.7 and 3.6
  • 7. III. Make tests pass ● six.iteritems ● six.text_type ● hashlib requires bytes ● exception.message ● xrange ● dict_keys ● filter ● Comparing with None ● round behaving differently
  • 8. IV. Upgrade migrations & Manual test ● Custom column names existed as bytes in migrations ● Same for choices and help_text ● Some third party apps generated new migrations ● Manual tests
  • 10.
  • 11.
  • 12. VI. Rollback and aftermath ● Sentry could not handle the error load ● Redis is an issue ○ Via Django ○ Direct usage ● Python 3 incompatible code not covered by tests ● filter ● Lesson learned - Monday morning == Friday evening
  • 14.

Editor's Notes

  1. Code was PEP8 compatible, which helped a lot
  2. We started in February
  3. Six - fixes moved modules compatibility Local import - In Python 3 you need to explicitly specify if you want module or package level import. We also had print statements
  4. Remember filter, we will get back to it later
  5. Remember filter, we will get back to it later
  6. Remember filter, we will get back to it later
  7. Remember filter, we will get back to it later
  8. Remember filter, we will get back to it later
  9. Remember filter, we will get back to it later
  10. Remember filter, we will get back to it later
  11. Remember filter, we will get back to it later