Building Good Containers for Python Applications
•
0 likes•101 views
This document discusses best practices for building Python applications in containers. It recommends containers be fast, small, secure, and usable. Specific strategies covered include using binary wheels to minimize build steps, building Python from source in a separate stage for caching, using multi-stage builds to optimize layers and size, and generating self-contained binary wheels. The conclusion emphasizes thinking through container strategies before implementation to achieve performance and portability.
Report
Share
Report
Share
Download to read offline
Recommended
It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.
Presented by: Igor Seletskiy
Presented at the All Things Open 2021
Raleigh, NC, USA
Raleigh Convention Center
Abstract: IT Teams know the drill. New security bulletins, new issues, new patches to deploy. Schedule another maintenance operation and prepare for system downtime.
There is a better way to do things. Live patching has been around in the Linux Kernel for some time now, but adoption has not been ideal so far - either because of a lack of trust in the technology or just lack of awareness - or sysadmins just enjoy interrupting their workloads or users.
Live patching consists of two aspects. First, there has to be a mechanism for function redirection in the kernel. As in many things, the kernel actually provides three different subset of tools that provide this functionality - kprobes, fprobes and Livepatching. Secondly, Live Patching relies on a set of tools to generate the actual patches to deploy, replacing the old code with new one. This is arguably the most involved part: you need to fit your new code in the proper space, you can’t overwrite other unrelated code and you need to maintain compatibility with other functions. If you change your parameter list, for example, its game over - something will break in the worst possible way.
In this talk we’ll go over issues like Consistency model, patch generation, deployment mechanisms and identify situations that are ideal candidates for live patching instead of traditional patching operations.
The New York Times: Sustainable Systems, Powered by Python
The document discusses Python tools and techniques for building sustainable systems at the New York Times. It describes applications they build using a Python-based microservices framework called Photon for ingesting and managing photos. It then summarizes several Python packages they use for installation and running applications, ensuring code quality, adding features, providing resilient runtimes, and generating automated documentation.
Modern Tools for Building Progressive Web Apps
Presented at: Open Source South Carolina Meetup
Presented by: John M. Wargo, Principal Program Manager, Visual Studio App Center @ Microsoft
Source Control with Domino Designer 8.5.3 and Git (DanNotes, November 28, 2012)
See my blog post about the presentation:
http://per.lausten.dk/blog/2012/11/source-control-with-domino-designer-8-5-3-and-git-my-talk-at-dannotes-november-2012.html
How to Work Efficiently in a Hybrid Git-Perforce Environment
Many companies face the challenge of supporting Git and Perforce together in their company. This presentation will describe the challenges Trend Micro faced and how they enabled a hybrid Git-Perforce environment. Additionally, learn three practices in using Perforce which make their work more efficient.
TYPO3 & Composer
Composer is a dependency manager for PHP that allows loading code packages from various locations into a PHP project. It resolves dependencies and allows updating packages easily. The document discusses how to use Composer to manage extensions in TYPO3 projects by defining extensions and versions in a composer.json file. It also covers creating custom Composer repositories and using PHPCI for continuous integration with Composer and TYPO3.
Testing cloud and kubernetes applications - ElasTest
Kubernetes applications are complex distributed systems composed by several microservices. When some end to end test is failing in these kind of applications, root cause is difficult without good observability tools. In this presentation, several tools are presented to make easier root cause analysis of cloud and kubernetes applications. One of the most interesting ones is ElasTest, a platform that integrates several open source tools to provide observability to e2e testing of complex distributed systems.
Introduction to Git(BitBucket) , Continuous Integration (Bamboo) & Confluence
Git is a distributed version control system that allows teams to manage multiple versions of documents and programs. Bitbucket provides Git and Mercurial hosting and integrates with Jira. Continuous integration with Bamboo automates building and testing code changes frequently to catch errors early. Confluence provides a central place for teams to share and collaborate on information.
Recommended
It’s 2021. Why are we -still- rebooting for patches? A look at Live Patching.
Presented by: Igor Seletskiy
Presented at the All Things Open 2021
Raleigh, NC, USA
Raleigh Convention Center
Abstract: IT Teams know the drill. New security bulletins, new issues, new patches to deploy. Schedule another maintenance operation and prepare for system downtime.
There is a better way to do things. Live patching has been around in the Linux Kernel for some time now, but adoption has not been ideal so far - either because of a lack of trust in the technology or just lack of awareness - or sysadmins just enjoy interrupting their workloads or users.
Live patching consists of two aspects. First, there has to be a mechanism for function redirection in the kernel. As in many things, the kernel actually provides three different subset of tools that provide this functionality - kprobes, fprobes and Livepatching. Secondly, Live Patching relies on a set of tools to generate the actual patches to deploy, replacing the old code with new one. This is arguably the most involved part: you need to fit your new code in the proper space, you can’t overwrite other unrelated code and you need to maintain compatibility with other functions. If you change your parameter list, for example, its game over - something will break in the worst possible way.
In this talk we’ll go over issues like Consistency model, patch generation, deployment mechanisms and identify situations that are ideal candidates for live patching instead of traditional patching operations.
The New York Times: Sustainable Systems, Powered by Python
The document discusses Python tools and techniques for building sustainable systems at the New York Times. It describes applications they build using a Python-based microservices framework called Photon for ingesting and managing photos. It then summarizes several Python packages they use for installation and running applications, ensuring code quality, adding features, providing resilient runtimes, and generating automated documentation.
Modern Tools for Building Progressive Web Apps
Presented at: Open Source South Carolina Meetup
Presented by: John M. Wargo, Principal Program Manager, Visual Studio App Center @ Microsoft
Source Control with Domino Designer 8.5.3 and Git (DanNotes, November 28, 2012)
See my blog post about the presentation:
http://per.lausten.dk/blog/2012/11/source-control-with-domino-designer-8-5-3-and-git-my-talk-at-dannotes-november-2012.html
How to Work Efficiently in a Hybrid Git-Perforce Environment
Many companies face the challenge of supporting Git and Perforce together in their company. This presentation will describe the challenges Trend Micro faced and how they enabled a hybrid Git-Perforce environment. Additionally, learn three practices in using Perforce which make their work more efficient.
TYPO3 & Composer
Composer is a dependency manager for PHP that allows loading code packages from various locations into a PHP project. It resolves dependencies and allows updating packages easily. The document discusses how to use Composer to manage extensions in TYPO3 projects by defining extensions and versions in a composer.json file. It also covers creating custom Composer repositories and using PHPCI for continuous integration with Composer and TYPO3.
Testing cloud and kubernetes applications - ElasTest
Kubernetes applications are complex distributed systems composed by several microservices. When some end to end test is failing in these kind of applications, root cause is difficult without good observability tools. In this presentation, several tools are presented to make easier root cause analysis of cloud and kubernetes applications. One of the most interesting ones is ElasTest, a platform that integrates several open source tools to provide observability to e2e testing of complex distributed systems.
Introduction to Git(BitBucket) , Continuous Integration (Bamboo) & Confluence
Git is a distributed version control system that allows teams to manage multiple versions of documents and programs. Bitbucket provides Git and Mercurial hosting and integrates with Jira. Continuous integration with Bamboo automates building and testing code changes frequently to catch errors early. Confluence provides a central place for teams to share and collaborate on information.
Testing fácil con Docker: Gestiona dependencias y unifica entornos
Docker es una tecnología que permite empaquetar el software de forma que se pueda ejecutar de forma sencilla y rápida, sin instalación y en cualquier sistema operativo. Es como tener cualquier programa instalado en su propia máquina virtual, pero arranca mucho más rápido y consume menos recursos. Docker está cambiando la forma en la que desplegamos software, pero también está afectando al propio proceso de desarrollo y particularmente al testing.
En este taller pondremos en práctica cómo usar Docker para facilitar la implementación de diferentes tipos de tests y su ejecución tanto en el portátil como en el entorno de integración continua. Aunque las técnicas que veremos se podrán aplicar en cualquier lenguaje de programación, los ejemplos estarán basados en Java y en JavaScript.
A successful Git branching model
This document outlines a Git branching model to allow for smooth development with isolated features. The model uses main branches of master and develop, and support branches of feature, release, and hotfix. Feature branches isolate new work and merge back to develop. Release branches prepare releases and merge to both develop and master. Hotfix branches address urgent bugs in master and also merge to both develop and master. This model aims to allow multiple developers to work independently while reducing conflicts and enabling flexible releases and rollbacks.
FluentD vs. Logstash
Presented by: Justin Reock
Presented at the All Things Open 2021
Raleigh, NC, USA
Raleigh Convention Center
Abstract: In our FluentD vs. Logstash comparison blog we talked about the importance of easily capturing, parsing, and visualizing log data at enterprise scale. We looked at the approaches FluentD and Logstash take to accomplish these tasks and defined particular areas of complexity and challenge that users face. Participants in this demo-driven webinar will watch as a system is configured for log analysis using both approaches, highlighting the strengths and weaknesses of each technology in the process.
Companion Blog: https://www.openlogic.com/blog/fluentd-vs-logstash
Integrating Git, Gerrit and Jenkins/Hudson with Mylyn
This document discusses integrating Git, Gerrit, and Jenkins/Hudson with the Mylyn task-focused interface in Eclipse. It describes each tool and common workflows. Mylyn allows viewing only tasks relevant to the developer's current context to avoid information overload. The document demonstrates integrating these tools with Mylyn and encourages contributing to related open source projects.
P2 Introduction
The document provides an overview of using p2 to facilitate agile software development. It discusses how p2 can be used to manage software update strategies, provision Eclipse, RCP and OSGi applications, and install and manage new features and configurations. The document also outlines topics to be covered, including the p2 architecture, building products with p2, and hands-on exercises for using p2.
Introduction to Containers: From Docker to Kubernetes and everything in-between
This document provides an introduction to containers using Docker and Kubernetes. It begins with an overview of containers, images, and layers. It then discusses Docker, including how to build images with Dockerfiles and compose containers together. The document also introduces Kubernetes for managing container clusters. It outlines an agenda covering containers, Docker, Kubernetes, Helm, Istio, GitOps and monitoring tools.
Git essentials
Don't be a git - the essentials you should know about git to use it correctly
Presentation by Otto Kekäläinen held at Vincit Teatime on Nov 11th 2015
http://www.vincitteatime.fi/
February OpenNTF Webinar: Introduction to Ansible for Newbies
This talk is for Domino admins and developers who would like to learn Ansible basics. Ansible is an automation engine to automate deployments. HCL provides a set of Ansible playbooks and roles to deploy a complete HCL Connections 7 environment. Come learn what Ansible is and why you should use it in this webinar.
The speaker will be:
Christoph Stoettener, HCL Ambassador
[RHFSeoul2017]6 Steps to Transform Enterprise Applications
The document provides a 6 step approach to transforming enterprise applications:
1. Re-organizing to DevOps;
2. Implementing self-service, on-demand infrastructure;
3. Automating deployments using tools like Puppet, Chef, and Kubernetes;
4. Establishing continuous integration and deployment pipelines;
5. Adopting advanced deployment techniques like blue-green deployments;
6. Moving to a microservices architecture.
(Declarative) Jenkins Pipelines
This document summarizes a Jenkins pipeline for testing and deploying Chef cookbooks. The pipeline is configured to automatically scan a GitHub organization for any repositories containing a Jenkinsfile. It will then create and manage multibranch pipeline jobs for each repository and branch. The pipelines leverage a shared Jenkins global library which contains pipeline logic to test and deploy the Chef cookbooks. This allows for standardized and reusable pipeline logic across all Chef cookbook repositories.
Moving 1,000 Users & 100 Branches into Streams
MathWorks has approximately 100 products derived from a single large code base, with over 1,000 developers contributing changes to almost one million source files. Their products are used to develop safety-critical systems, so managing a continuous influx of changes while guaranteeing quality and correctness is challenging. Learn how MathWorks, unable to use a simple model of “component as a directory,” created an elegant system using virtual streams and the Perforce broker in their ongoing efforts to modularize their code base.
2013 10-28 php ug presentation - ci using phing and hudson
This document discusses using Phing and Hudson for continuous integration of PHP projects. Phing is a build tool similar to Ant for PHP projects that allows running tests, building packages, and more via XML configuration files. Hudson is an open source continuous integration server that can be used to run Phing build scripts and publish test reports. The document provides examples of using Phing for tasks like linting, testing, deploying, and generating documentation and packages, as well as configuring Hudson jobs to regularly run builds, track changes, and publish results.
Provisioning environments. A simplistic approach
This document discusses provisioning development environments using DevOps tools like Vagrant and Puppet. It introduces Vagrant as a way to create reproducible and portable virtual environments. Puppet is then presented as an automation tool to configure and manage the resources and applications within those environments. The document provides a high-level example of using these tools together to provision a load-balanced web application backed by MongoDB. It aims to demonstrate how DevOps practices like automation and cooperation between development and operations can help address challenges like frequent deployment and onboarding new team members.
Introduction to Git for developers
The document provides a brief history of revision control systems including SCCS, RCS, CVS, Subversion, and distributed systems like Git, Mercurial, and Bazaar. It discusses the problems with earlier systems that motivated the creation of Git, including issues with CVS and Subversion. It describes how Linus Torvalds created Git to address these problems and support fast, distributed, and non-linear development workflows.
Codifying the Build and Release Process with a Jenkins Pipeline Shared Library
These are my slides from my Jenkins World 2017 talk, detailing a war story of migrating 150-200 Freestyle Jobs for build and release, into ~10 line Jenkinsfiles that heavily leverages Jenkins Pipeline Shared Libraries (https://jenkins.io/doc/book/pipeline/shared-libraries/)
Writing Commits for You, Your Friends, and Your Future Self
Presented at Open Source 101 2022
Presented by Victoria Dye, GitHub
Abstract: For many developers, a Git commit is just a step in their source control ritual: an arbitrary save point at the end of a work day, the Nth attempt at finding the right syntax for a CI workflow, or a satisfying bookend to fixing a bug. But commits provide far more than that: they tell the story of how each line of code was created and modified. Despite their unassuming nature, they are one of the best ways to answer the question "why did I write that?"
The story told by a repository's commit history is both richer and more essential in the open source world. Well-structured commits can guide you in the implementation of a new feature, break down complex reviews into easily-digestible increments, and provide an invaluable resource to future contributors hoping to extend functionality or find the root cause of a bug.
This talk will show how high-quality commit organization can benefit you now and in the future. First, we will introduce strategies and tools to tell a clear story through commits. Later, we will demonstrate how to use an informative commit history while reviewing code and investigating bugs.
Smarter deployments with octopus deploy
This document discusses Octopus Deploy, a deployment automation tool. It describes Octopus Deploy's architecture and 7 step deployment process. The process includes declaring environments, creating application packages, defining projects, creating deployment processes with steps and variables, releasing packages, and deploying releases to environments. Octopus Deploy supports features like automated deployments, rollbacks, configuration transformations, and integration with build pipelines. It provides visibility through audit logs and manages deployments across development, test, and production environments.
Git for IBM Notes Designer
A presentation I did to bring a Notes developer not familiar with Git source control, some examples of navigating common tasks.
WTF is GitOps and Why You Should Care?
Slides from OpenSource101.com Talk (https://opensource101.com/sessions/wtf-is-gitops-why-should-you-care/)
If you’re interested in learning more about Cloud Native Computing or are already in the Kubernetes community you may have heard the term GitOps. It’s become a bit of a buzzword, but it’s so much more! The benefits of GitOps are real – they bring you security, reliability, velocity and more! And the project that started it all was Flux – a CNCF Incubating project developed and later donated by Weaveworks (the GitOps company who coined the term).
Pinky will share from personal experience why GitOps has been an essential part of achieving a best-in-class delivery and platform team. Pinky will give a brief overview of definitions, CNCF-based principles, and Flux’s capabilities: multi-tenancy, multi-cluster, (multi-everything!), for apps and infra, and more.
Pinky will cover a little of Flux’s microservices architecture and how the various components deliver this robust, secure, and trusted open source solution. Through the components of the Flux project, users today are enjoying compatibility with Helm, Jenkins, Terraform, Prometheus, and more as well as with cloud providers such as AWS, Azure, Google Cloud, and more.
Join us for this informative session and get all of your GitOps questions answered by an end user in the community!
Speaker: Priyanka (aka “Pinky”) is a Developer Experience Engineer at Weaveworks. She has worked on a multitude of topics including front end development, UI automation for testing and API development. Previously she was a software developer at State Farm where she was on the delivery engineering team working on GitOps enablement. She was instrumental in the multi-tenancy migration to utilize Flux for an internal Kubernetes offering. Outside of work, Priyanka enjoys hanging out with her husband and two rescue dogs as well as traveling around the globe.
Automatic codefixes
When the daily duty of delivering working software is done with ease, and also secondary tasks like working tests are done well, there’s always more work waiting at the ‘nice to have’ priority level. Things like code style, valid Composer files, updated dependencies or various other meta data that isn’t at all mission critical, but always provides a certain level of annoyance if not maintained properly.
I’ll show you our way how to deal with such a situation when maintaining about 140 distinct repositories with PHP software. At this scale automation is the only choice, and we not only do it for testing, but for these maintenance tasks as well. We have created a single point of attack from where we can influence all our repositories and it’s code, and we do it in a way that is not as intrusive as pre-commit or pre-receive hooks, by using pull requests.
SFSCON23 - Denver Gingerich - How do you really do GPL enforcement
SFSCON23 - Denver Gingerich - How do you really do GPL enforcementSouth Tyrol Free Software Conference
Enforcing the General Public License (GPL) to bring real software freedom to people can be very challenging in practice, but many of the steps in the process are straight-forward. As the only organization enforcing the GPL for Linux, Software Freedom Conservancy (SFC) receives a huge quantity of GPL violation reports, and needs to triage each one as the beginning of our process. The next step is called the "CCS check" (complete corresponding source check), a crucial but barely known activity that determines whether some candidate source code actually corresponds to the device/binaries that the candidate was provided for. We will discuss the CCS check in detail, providing examples and tips for doing your own checks.
Lastly, we'll cover the offer check, something everyone can do to help in SFC's efforts to bring real software right-to-repair to every device running Linux. Whether you want to check offers for source code, review a CCS candidate, or go even further with additional GPL enforcement work of your own, this talk will set you up for success.Python overview
Python was created in 1990 by Guido van Rossum as a hobby project. It is a highly portable, interpreted programming language that is designed to be easy to learn and read. Python code tends to be shorter and more readable than comparable code in other languages like C or Java due to its clear syntax and use of whitespace.
More Related Content
What's hot
Testing fácil con Docker: Gestiona dependencias y unifica entornos
Docker es una tecnología que permite empaquetar el software de forma que se pueda ejecutar de forma sencilla y rápida, sin instalación y en cualquier sistema operativo. Es como tener cualquier programa instalado en su propia máquina virtual, pero arranca mucho más rápido y consume menos recursos. Docker está cambiando la forma en la que desplegamos software, pero también está afectando al propio proceso de desarrollo y particularmente al testing.
En este taller pondremos en práctica cómo usar Docker para facilitar la implementación de diferentes tipos de tests y su ejecución tanto en el portátil como en el entorno de integración continua. Aunque las técnicas que veremos se podrán aplicar en cualquier lenguaje de programación, los ejemplos estarán basados en Java y en JavaScript.
A successful Git branching model
This document outlines a Git branching model to allow for smooth development with isolated features. The model uses main branches of master and develop, and support branches of feature, release, and hotfix. Feature branches isolate new work and merge back to develop. Release branches prepare releases and merge to both develop and master. Hotfix branches address urgent bugs in master and also merge to both develop and master. This model aims to allow multiple developers to work independently while reducing conflicts and enabling flexible releases and rollbacks.
FluentD vs. Logstash
Presented by: Justin Reock
Presented at the All Things Open 2021
Raleigh, NC, USA
Raleigh Convention Center
Abstract: In our FluentD vs. Logstash comparison blog we talked about the importance of easily capturing, parsing, and visualizing log data at enterprise scale. We looked at the approaches FluentD and Logstash take to accomplish these tasks and defined particular areas of complexity and challenge that users face. Participants in this demo-driven webinar will watch as a system is configured for log analysis using both approaches, highlighting the strengths and weaknesses of each technology in the process.
Companion Blog: https://www.openlogic.com/blog/fluentd-vs-logstash
Integrating Git, Gerrit and Jenkins/Hudson with Mylyn
This document discusses integrating Git, Gerrit, and Jenkins/Hudson with the Mylyn task-focused interface in Eclipse. It describes each tool and common workflows. Mylyn allows viewing only tasks relevant to the developer's current context to avoid information overload. The document demonstrates integrating these tools with Mylyn and encourages contributing to related open source projects.
P2 Introduction
The document provides an overview of using p2 to facilitate agile software development. It discusses how p2 can be used to manage software update strategies, provision Eclipse, RCP and OSGi applications, and install and manage new features and configurations. The document also outlines topics to be covered, including the p2 architecture, building products with p2, and hands-on exercises for using p2.
Introduction to Containers: From Docker to Kubernetes and everything in-between
This document provides an introduction to containers using Docker and Kubernetes. It begins with an overview of containers, images, and layers. It then discusses Docker, including how to build images with Dockerfiles and compose containers together. The document also introduces Kubernetes for managing container clusters. It outlines an agenda covering containers, Docker, Kubernetes, Helm, Istio, GitOps and monitoring tools.
Git essentials
Don't be a git - the essentials you should know about git to use it correctly
Presentation by Otto Kekäläinen held at Vincit Teatime on Nov 11th 2015
http://www.vincitteatime.fi/
February OpenNTF Webinar: Introduction to Ansible for Newbies
This talk is for Domino admins and developers who would like to learn Ansible basics. Ansible is an automation engine to automate deployments. HCL provides a set of Ansible playbooks and roles to deploy a complete HCL Connections 7 environment. Come learn what Ansible is and why you should use it in this webinar.
The speaker will be:
Christoph Stoettener, HCL Ambassador
[RHFSeoul2017]6 Steps to Transform Enterprise Applications
The document provides a 6 step approach to transforming enterprise applications:
1. Re-organizing to DevOps;
2. Implementing self-service, on-demand infrastructure;
3. Automating deployments using tools like Puppet, Chef, and Kubernetes;
4. Establishing continuous integration and deployment pipelines;
5. Adopting advanced deployment techniques like blue-green deployments;
6. Moving to a microservices architecture.
(Declarative) Jenkins Pipelines
This document summarizes a Jenkins pipeline for testing and deploying Chef cookbooks. The pipeline is configured to automatically scan a GitHub organization for any repositories containing a Jenkinsfile. It will then create and manage multibranch pipeline jobs for each repository and branch. The pipelines leverage a shared Jenkins global library which contains pipeline logic to test and deploy the Chef cookbooks. This allows for standardized and reusable pipeline logic across all Chef cookbook repositories.
Moving 1,000 Users & 100 Branches into Streams
MathWorks has approximately 100 products derived from a single large code base, with over 1,000 developers contributing changes to almost one million source files. Their products are used to develop safety-critical systems, so managing a continuous influx of changes while guaranteeing quality and correctness is challenging. Learn how MathWorks, unable to use a simple model of “component as a directory,” created an elegant system using virtual streams and the Perforce broker in their ongoing efforts to modularize their code base.
2013 10-28 php ug presentation - ci using phing and hudson
This document discusses using Phing and Hudson for continuous integration of PHP projects. Phing is a build tool similar to Ant for PHP projects that allows running tests, building packages, and more via XML configuration files. Hudson is an open source continuous integration server that can be used to run Phing build scripts and publish test reports. The document provides examples of using Phing for tasks like linting, testing, deploying, and generating documentation and packages, as well as configuring Hudson jobs to regularly run builds, track changes, and publish results.
Provisioning environments. A simplistic approach
This document discusses provisioning development environments using DevOps tools like Vagrant and Puppet. It introduces Vagrant as a way to create reproducible and portable virtual environments. Puppet is then presented as an automation tool to configure and manage the resources and applications within those environments. The document provides a high-level example of using these tools together to provision a load-balanced web application backed by MongoDB. It aims to demonstrate how DevOps practices like automation and cooperation between development and operations can help address challenges like frequent deployment and onboarding new team members.
Introduction to Git for developers
The document provides a brief history of revision control systems including SCCS, RCS, CVS, Subversion, and distributed systems like Git, Mercurial, and Bazaar. It discusses the problems with earlier systems that motivated the creation of Git, including issues with CVS and Subversion. It describes how Linus Torvalds created Git to address these problems and support fast, distributed, and non-linear development workflows.
Codifying the Build and Release Process with a Jenkins Pipeline Shared Library
These are my slides from my Jenkins World 2017 talk, detailing a war story of migrating 150-200 Freestyle Jobs for build and release, into ~10 line Jenkinsfiles that heavily leverages Jenkins Pipeline Shared Libraries (https://jenkins.io/doc/book/pipeline/shared-libraries/)
Writing Commits for You, Your Friends, and Your Future Self
Presented at Open Source 101 2022
Presented by Victoria Dye, GitHub
Abstract: For many developers, a Git commit is just a step in their source control ritual: an arbitrary save point at the end of a work day, the Nth attempt at finding the right syntax for a CI workflow, or a satisfying bookend to fixing a bug. But commits provide far more than that: they tell the story of how each line of code was created and modified. Despite their unassuming nature, they are one of the best ways to answer the question "why did I write that?"
The story told by a repository's commit history is both richer and more essential in the open source world. Well-structured commits can guide you in the implementation of a new feature, break down complex reviews into easily-digestible increments, and provide an invaluable resource to future contributors hoping to extend functionality or find the root cause of a bug.
This talk will show how high-quality commit organization can benefit you now and in the future. First, we will introduce strategies and tools to tell a clear story through commits. Later, we will demonstrate how to use an informative commit history while reviewing code and investigating bugs.
Smarter deployments with octopus deploy
This document discusses Octopus Deploy, a deployment automation tool. It describes Octopus Deploy's architecture and 7 step deployment process. The process includes declaring environments, creating application packages, defining projects, creating deployment processes with steps and variables, releasing packages, and deploying releases to environments. Octopus Deploy supports features like automated deployments, rollbacks, configuration transformations, and integration with build pipelines. It provides visibility through audit logs and manages deployments across development, test, and production environments.
Git for IBM Notes Designer
A presentation I did to bring a Notes developer not familiar with Git source control, some examples of navigating common tasks.
WTF is GitOps and Why You Should Care?
Slides from OpenSource101.com Talk (https://opensource101.com/sessions/wtf-is-gitops-why-should-you-care/)
If you’re interested in learning more about Cloud Native Computing or are already in the Kubernetes community you may have heard the term GitOps. It’s become a bit of a buzzword, but it’s so much more! The benefits of GitOps are real – they bring you security, reliability, velocity and more! And the project that started it all was Flux – a CNCF Incubating project developed and later donated by Weaveworks (the GitOps company who coined the term).
Pinky will share from personal experience why GitOps has been an essential part of achieving a best-in-class delivery and platform team. Pinky will give a brief overview of definitions, CNCF-based principles, and Flux’s capabilities: multi-tenancy, multi-cluster, (multi-everything!), for apps and infra, and more.
Pinky will cover a little of Flux’s microservices architecture and how the various components deliver this robust, secure, and trusted open source solution. Through the components of the Flux project, users today are enjoying compatibility with Helm, Jenkins, Terraform, Prometheus, and more as well as with cloud providers such as AWS, Azure, Google Cloud, and more.
Join us for this informative session and get all of your GitOps questions answered by an end user in the community!
Speaker: Priyanka (aka “Pinky”) is a Developer Experience Engineer at Weaveworks. She has worked on a multitude of topics including front end development, UI automation for testing and API development. Previously she was a software developer at State Farm where she was on the delivery engineering team working on GitOps enablement. She was instrumental in the multi-tenancy migration to utilize Flux for an internal Kubernetes offering. Outside of work, Priyanka enjoys hanging out with her husband and two rescue dogs as well as traveling around the globe.
Automatic codefixes
When the daily duty of delivering working software is done with ease, and also secondary tasks like working tests are done well, there’s always more work waiting at the ‘nice to have’ priority level. Things like code style, valid Composer files, updated dependencies or various other meta data that isn’t at all mission critical, but always provides a certain level of annoyance if not maintained properly.
I’ll show you our way how to deal with such a situation when maintaining about 140 distinct repositories with PHP software. At this scale automation is the only choice, and we not only do it for testing, but for these maintenance tasks as well. We have created a single point of attack from where we can influence all our repositories and it’s code, and we do it in a way that is not as intrusive as pre-commit or pre-receive hooks, by using pull requests.
What's hot (20)
Testing fácil con Docker: Gestiona dependencias y unifica entornos
Testing fácil con Docker: Gestiona dependencias y unifica entornos
Integrating Git, Gerrit and Jenkins/Hudson with Mylyn
Integrating Git, Gerrit and Jenkins/Hudson with Mylyn
Introduction to Containers: From Docker to Kubernetes and everything in-between
Introduction to Containers: From Docker to Kubernetes and everything in-between
February OpenNTF Webinar: Introduction to Ansible for Newbies
February OpenNTF Webinar: Introduction to Ansible for Newbies
[RHFSeoul2017]6 Steps to Transform Enterprise Applications
[RHFSeoul2017]6 Steps to Transform Enterprise Applications
2013 10-28 php ug presentation - ci using phing and hudson
2013 10-28 php ug presentation - ci using phing and hudson
Codifying the Build and Release Process with a Jenkins Pipeline Shared Library
Codifying the Build and Release Process with a Jenkins Pipeline Shared Library
Writing Commits for You, Your Friends, and Your Future Self
Writing Commits for You, Your Friends, and Your Future Self
Similar to Building Good Containers for Python Applications
SFSCON23 - Denver Gingerich - How do you really do GPL enforcement
SFSCON23 - Denver Gingerich - How do you really do GPL enforcementSouth Tyrol Free Software Conference
Enforcing the General Public License (GPL) to bring real software freedom to people can be very challenging in practice, but many of the steps in the process are straight-forward. As the only organization enforcing the GPL for Linux, Software Freedom Conservancy (SFC) receives a huge quantity of GPL violation reports, and needs to triage each one as the beginning of our process. The next step is called the "CCS check" (complete corresponding source check), a crucial but barely known activity that determines whether some candidate source code actually corresponds to the device/binaries that the candidate was provided for. We will discuss the CCS check in detail, providing examples and tips for doing your own checks.
Lastly, we'll cover the offer check, something everyone can do to help in SFC's efforts to bring real software right-to-repair to every device running Linux. Whether you want to check offers for source code, review a CCS candidate, or go even further with additional GPL enforcement work of your own, this talk will set you up for success.Python overview
Python was created in 1990 by Guido van Rossum as a hobby project. It is a highly portable, interpreted programming language that is designed to be easy to learn and read. Python code tends to be shorter and more readable than comparable code in other languages like C or Java due to its clear syntax and use of whitespace.
Design and Evolution of cyber-dojo
presented at NorDev conference 2016 (Norwich) and also at Agile on the Beach conference 2016 (Falmouth)
Ci For The Web 2.0 Guy Or Gal
This document provides instructions for setting up a continuous integration environment using Ubuntu Linux, Ruby on Rails, CruiseControl, JsUnit and Selenium. It includes steps to install the necessary software like Ubuntu, Ruby, Rails, MySQL, Subversion and other tools. It also outlines creating a sample Rails application, importing it to Subversion and configuring CruiseControl for continuous integration. The goal is to have a working CI environment that can be easily replicated and used on real projects.
Pinto+Stratopan+Love
A presentation in three acts: The newest Pinto features; Stratpan -- Managing Perl modules in the cloud; Why you'll love using Pinto/Stratopan.
Understanding and building Your Own Docker
A lot of people use Docker/rkt, but very often we do not have time to actually understand how they work. So today in half-hour I will show you in a nutshell how that works. My hope is that even after you know how to build a container engine, I can still convince you that the existing tools are worth spending $MM to create and use.
Pwnstaller
Will presented on using Pyinstaller and introducing Pwnstaller, a tool he created to dynamically generate unique Python payload executables. Pyinstaller packages Python scripts into standalone executables but its loader binaries could be detected by antivirus. Pwnstaller obfuscates and recompiles the Pyinstaller loader source each time to avoid static signatures. It has been integrated into Veil-Evasion so Python payloads benefit from dynamically generated unique loaders that are harder for antivirus to detect.
WordPress Security Blitz
The document discusses basic WordPress security strategies and plugins to consider. It recommends picking a reliable host, splitting domains and hosting, backing up sites regularly, keeping WordPress and plugins updated, restricting wp-admin access, using strong passwords, and installing security-focused plugins like Akismet, Bad Behavior, and WP Security Scan. Regular server log checks, non-standard database prefixes, avoiding the "admin" user, file permission management, and .htaccess restrictions are also advised. Testing backups, database cleanup, and security audits are suggested along with using a staging server and services like CloudFlare.
Closing the gap between Distros(devs) and their Users(ops)
Kris Buytaert discusses the gap between software developers (devs) and operations teams (ops) when using Linux distributions. There is often a lack of communication and different priorities, with devs focused on getting the latest code quickly and ops concerned with stability, security and deployability at scale. Buytaert provides examples of issues, such as packages containing PHP code directly in /etc, and distributions packaging outdated or broken upstream software. He advocates for better collaboration between distributions, upstream projects and power users through activities like meetups, user advisory boards, and tools to more easily build and test packages. The goal is to improve culture, automation, measurement and sharing between all parties.
Возможности интерпретатора Python в NX-OS
The document discusses a webinar presented by Cisco TAC Engineer Anton Tugai about the capabilities of the Python interpreter in NX-OS. Some key points:
- Tugai gave a presentation on trends in Cisco SDN and current solutions.
- The webinar covered an introduction to Python, how Python is integrated into NX-OS, examples, and a demonstration.
- Native Python interpreter is available on Nexus switches starting from certain software versions, allowing Python scripts to run directly on the switch and execute CLI commands.
Modern Perl for the Unfrozen Paleolithic Perl Programmer
Modern Perl for the Unfrozen Paleolithic Perl Programmer discusses many changes to Perl and its ecosystem since 2001 when the caveman programmer fell into a glacier. Some key changes include Perl version 5.22, new language features like defined-or and subroutine signatures, and tools like Perlbrew, Plenv and cpanm. Modern web development uses Plack/PSGI instead of CGI.pm. The speaker recommends modules like JSON::MaybeXS, Moose/Moo and websites like CPAN Ratings, MetaCPAN and Perl Weekly to stay up-to-date.
Automated Deployment With Phing
Website deployment is a tedious and intricate task that lends itself to human error (oops, did I forget to update the DB schema?). Using Phing in conjunction with deployment techniques can greatly reduce the risk of human error and increase productivity. The presentation will cover using Phing to sync files, run tasks, migrate databases, target configuration, and other deployment techniques.
From Config Management Sucks to #cfgmgmtlove
This document summarizes Kris Buytaert's talk on the evolution of config management from the 1990s to present day. It discusses early approaches like manual installations and system imaging tools. It then covers the rise of infrastructure as code using tools like Puppet, Chef, and Docker. The talk addresses challenges like getting operations teams to adopt new methods and complexities that can arise from dependencies and modules. It promotes treating infrastructure like code with development practices for versioning, testing, and continuous integration/deployment.
A quick overview of why to use and how to set up iPython notebooks for research
A quick overview of why to use and how to set up iPython notebooks for research in the Amaral lab. Example notebook is a gist at:
http://nbviewer.ipython.org/gist/anonymous/f8e6d8985d2ea0e4bab1
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
This presentation on Kamaelia at Euro OSCON 2006, and specifically focusses
on a particular system - Kamaelia Macro which is essentially a system for
timeshifting pretty much everything.
Dear compiler please don't be my nanny v2
Nothing bad would happen. The compiler would be able to infer the type is float in this case. Strong typing is not always necessary or helpful.
Pyhton-1a-Basics.pdf
This document provides an introduction to the Python programming language. It discusses what Python is, why it was created, its basic features and uses. Python is an interpreted, object-oriented programming language that is designed to be readable. It can be used for tasks such as web development, scientific computing, and scripting. The document also covers Python basics like variables, data types, operators, and input/output functions. It provides examples of Python code and discusses best practices for writing and running Python programs.
Getting to push_button_deploys
A rundown of the tools and procedures that I have implemented at Moovweb to increase the velocity of software deployment.
Janus Workshop @ ClueCon 2020
Slides for the 60 minutes workshop I presented at the virtual edition of ClueCon 2020 (ClueCon Deconstructed). The many slides cover different aspects in Janus, ranging from configuration, to plugins, how to write your own plugin, core features, recording, monitoring, and so on. Unfortunately I didn't have enough time to talk about everything, but slides should be easy to follow anyway.
Django Dev Environment Howto
This document provides guidance on setting up a development environment for Python and Django projects. It discusses choosing PostgreSQL as the database, installing Python packages and dependencies using pip, virtualenv and virtualenvwrapper, ensuring the necessary C compilers are installed to build extensions, and options for installing pre-built binaries when a C compiler is not desired. Potential issues around version mismatches, architectures, and security are also addressed.
Similar to Building Good Containers for Python Applications (20)
SFSCON23 - Denver Gingerich - How do you really do GPL enforcement
SFSCON23 - Denver Gingerich - How do you really do GPL enforcement
Closing the gap between Distros(devs) and their Users(ops)
Closing the gap between Distros(devs) and their Users(ops)
Modern Perl for the Unfrozen Paleolithic Perl Programmer
Modern Perl for the Unfrozen Paleolithic Perl Programmer
A quick overview of why to use and how to set up iPython notebooks for research
A quick overview of why to use and how to set up iPython notebooks for research
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
More from All Things Open
Building Reliability - The Realities of Observability
Presented at the ATO RTP Meetup
Presented by Jeremy Proffit, Director of DevSecOps & SRE for Customer Care and Communications, Ally
Title: Building Reliability - The Realities of Observability
Abstract: Join me as we discuss true observability, learn what works and what doesn't. We'll not only discuss dashboards, monitoring and alerting, but how these can be built by automation or included in your IAC modules. We'll talk about how to properly alert staff based on priority to keep your staff and yourself sane. And even discuss architecture and how it impacts reliably and why serverless isn't always the best at being reliable.
Modern Database Best Practices
Presented at the ATO RTP Meetup
Presented by Peter Zaitsev, Founder of Percona
Title: Modern Database Best Practices
Abstract: There are now more Database choices available for developers than ever before - there are general purpose databases and specialized databases, single node and distributed databases, Open Source, Proprietary databases and databases available exclusively in the cloud. In this presentation we will cover the best practices of choosing database(s) for your applications, best practices as it comes to application development as well as managing those databases to achieve best possible performance, security, availability at the lowest cost.
Open Source and Public Policy
All Things Open 2023
Presented at All Things Open 2023
Presented by Deb Bryant - Open Source Initiative, Patrick Masson - Apereo Foundation, Stephen Jacobs - Rochester Institute of Technology, Ruth Suehle - SAS, & Greg Wallace - FreeBSD Foundation
Title: Open Source and Public Policy
Abstract: New regulations in the software industry and adjacent areas such as AI, open science, open data, and open education are on the rise around the world. Cyber Security, societal impact of AI, data and privacy are paramount issues for legislators globally. At the same time, the COVID-19 pandemic drove collaborative development to unprecedented levels and took Open Source software, open research, open content and data from mainstream to main stage, creating tension between public benefit and citizen safety and security as legislators struggle to find a balance between open collaboration and protecting citizens.
Historically, the open source software community and foundations supporting its work have not engaged in policy discussions. Moving forward, thoughtful development of these important public policies whilst not harming our complex ecosystems requires an understanding of how our ecosystem operates. Ensuring stakeholders without historic benefit of representation in those discussions becomes paramount to that end.
Please join our open discussion with open policy stakeholders working constructively on current open policy topics. Our panelists will provide a view into how oss foundations and other open domain allies are now rising to this new challenge as well as seizing the opportunity to influence positive changes to the public’s benefit.
Topics: Public Policy, Open Science, Open Education, current legislation in the US and EU, US interest in OSS sustainability, intro to the Open Policy Alliance
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
This document summarizes a presentation about graph-quilt, an open source GraphQL orchestrator library. It discusses the challenges of building a GraphQL orchestrator to unify data from multiple services. Graph-quilt addresses this by allowing services to register their GraphQL schemas and composing them into a unified schema. It also supports features like remote schema extensions, authorization, and adapting existing REST APIs. The presenters believe graph-quilt provides a flexible way to build GraphQL gateways and help more clients adopt GraphQL.
The State of Passwordless Auth on the Web - Phil Nash
Presented at All Things Open 2023
Presented by Phil Nash - Sonar
Title: The State of Passwordless Auth on the Web
Abstract: Can we get rid of passwords yet? They make for a poor user experience and users are notoriously bad with them. The advent of WebAuthn has brought a passwordless world closer, but where do we really stand?
In this talk we'll explore the current user experience of WebAuthn and the requirements a user has to fulfil to authenticate without a password. We'll also explore the fallbacks and safeguards we can use to make the password experience better and more secure. By the end of the session you'll have a vision of how authentication could look in the future and a blueprint for how to build the best auth experience today.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Total ReDoS: The dangers of regex in JavaScript
Presented at All Things Open 2023
Presented by Phil Nash - Sonar
Title: Total ReDoS: The dangers of regex in JavaScript
Abstract: Regular expressions are complicated and can be hard to learn. On top of that, they can also be a security risk; writing the wrong pattern can open your application up to denial of service attacks. One token out of place and you invite in the dreaded ReDoS.
But how can a regular expression cause this? In this talk we’ll track down the patterns that can cause this trouble, explain why they are an issue and propose ways to fix them now and avoid them in the future. Together we’ll demystify these powerful search patterns and keep your application safe from expressions that behave in a way that is anything but regular.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
What Does Real World Mass Adoption of Decentralized Tech Look Like?
Presented at All Things Open 2023
Presented by Karl Mozurkewich - Storj
Title: What Does Real World Mass Adoption of Decentralized Tech Look Like?
Abstract: We delve into the transformative potential of decentralized technology. Beginning with a brief overview of the rise of centralization with the advent of the internet and the counter-shift marked by blockchain we explore the intrinsic characteristics of decentralized and distributed systems, such as trustless operations, peer-to-peer networks, and enterprise application scalability. Various sectors, including finance, supply chains, media and entertainment, data science and cloud infrastructure are on the brink of disruption. The societal implications are vast, with the potential for greater individual empowerment, a greener planet and more viable resource utilization, but concerns about data security persist.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
How to Write & Deploy a Smart Contract
Presented at All Things Open 2023
Presented by Anastasia Lalamentik - Kaleido
Title: How to Write & Deploy a Smart Contract
Abstract: In this talk, Anastasia Lalamentik, Full Stack Engineer at Kaleido, will walk through how Ethereum smart contracts work and go over related concepts like gas fees, the Ethereum Virtual Machine (EVM), the block explorer, and the Solidity programming language. This is vital to anyone who wants to build a blockchain app and is a great introduction to blockchain technology for newcomers to the space.
By the end of the talk, attendees will better understand how to:
- Write a simple smart contract
- Deploy their smart contract to an Ethereum test network through the latest tools like Hardhat and the MetaMask wallet
- Test interactions with their deployed smart contract and ensure that everything is working properly
Additionally, participants will get to interact with Anastasia's deployed smart contract at the end of the talk. Anastasia’s past talks have attracted and have been attended by a diverse group of participants with a range of experience in the space.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Presented at All Things Open 2023
Presented by Paul Brebner - Instaclustr (by Spot by NetApp)
Title: Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Abstract: In this talk we’ll build a Drone delivery application, and then use it to do some Machine Learning “on the fly”.
In the 1st part of the talk, we'll build a real-time Drone Delivery demonstration application using a combination of two open-source technologies: Uber’s Cadence (for stateful, scheduled, long-running workflows), and Apache Kafka (for fast streaming data).
With up to 2,000 (simulated) drones and deliveries in progress at once this application generates a vast flow of spatio-temporal data.
In the 2nd part of the talk, we'll use this platform to explore Machine Learning (ML) over streaming and drifting Kafka data with TensorFlow to try and predict which shops will be busy in advance.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
DEI Challenges and Success
Presented at the All Things Open 2023 Inclusion and Diversity in Open Source Event
Presented by Efraim Marquez-Arreaza - Red Hat
Title: DEI Challenges and Success
Abstract: In today's world, many companies and organizations have Diversity, Equity and Inclusion (DEI) communities. Red Hat Unidos is a DEI community focused on advocating for the Hispanic/Latine community. In this talk, we would like to share our challenges and success during the past 4-years and plans for the future.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Scaling Web Applications with Background
Presented at All Things Open 2023
Presented by Lydia Cupery - HubSpot
Title: Scaling Web Applications with Background Jobs: Takeaways from Generating a Huge PDF
Abstract: Do you need to perform time-consuming or CPU-intensive processes in your web application but are concerned about performance? That’s where background jobs come in. By offloading resource-intensive tasks to separate worker processes, you can improve the scalability of your web application.
In this talk, I'll share my experience of using background jobs to scale our web application. I'll discuss the challenges my team faced that led us to adopt background jobs. Then, I'll share practical tips on how to design background jobs for CPU-intensive or time-consuming processes, such as generating huge PDFs and batch emailing. I'll wrap up by going over the performance and cost tradeoffs of background jobs.
I'll use Typescript, Express, and Heroku as examples in this talk, but the concepts and best practices that I'll share are applicable to other languages and tools.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Supercharging tutorials with WebAssembly
Presented at All Things Open 2023
Presented by Robert Aboukhalil - CZI
Title: Supercharging tutorials with WebAssembly
Abstract: sandbox.bio is a free platform that features interactive command-line tutorials for bioinformatics. This talk is a deep-dive into how sandbox.bio was built, with a focus on how WebAssembly enabled bringing command-line tools like awk and grep to the web. Although these tools were originally written in C/C++, they all run directly in the browser, thanks to WebAssembly! And since the computations run on each user's computer, this makes the application highly scalable and cost-effective.
Along the way, I'll discuss how WebAssembly works and how to get started using it in your own applications. The talk will also cover more advanced WebAssembly features such as threads and SIMD, and will end with a discussion of WebAssembly's benefits and pitfalls (it's a powerful technology, but it's not always the right tool!).
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Using SQL to Find Needles in Haystacks
Presented at All Things Open 2023
Presented by K.S. Bhaskar - YottaDB LLC
Title: Using SQL to Find Needles in Haystacks
Abstract: Database journal files capture every update to a database. A database of a few hundred GB can generate GBs worth of journal files every minute at busy times. Troubleshooting and forensices, especially of rare and intermittent problems, such as which process made what update and when, is an exercise of finding needles in haystacks. A similar problem exists with syslogs. A solution is to load the journal files and syslogs into a database, and use SQL to query the database. Bhaskar will present and demonstrate this with a 100% FOSS stack.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Configuration Security as a Game of Pursuit Intercept
The document discusses configuration security as a game of pursuit-evasion and intercept. It was presented by Wes Widner, Principal Engineer at Automox. The document includes a JSON policy snippet with an ID, statement, actions, effects, resources, and principal allowing the GetObject action on all objects in an S3 bucket for all principals. It has page numbers at the bottom indicating it is from a larger presentation.
Scaling an Open Source Sponsorship Program
Presented at All Things Open 2023
Presented by Carol Huang & Mike Fix - Stripe
Title: Scaling an Open Source Sponsorship Program
Abstract: We already know this: the open-source ecosystem needs further monetary investment from the companies that benefit most from it. Likewise, companies say they want to participate in these initiatives, but find it hard to dedicate resources to open source funding when there isn’t a clear ROI.
This talk discusses how the Open Source Program Office at Stripe built a scalable, sustainable open source sponsorship model that aligns internal company incentives with those of open source maintainers and the community at large. We go over the unique “platformization” of our OSPO that allowed us to create multiple funding models, such as BYOB (Bring Your Own Budget), and share lessons learned from this experience as well as other OSPOs.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Build Developer Experience Teams for Open Source
Presented at All Things Open 2023
Presented by Arundeep Nagaraj - Amazon Web Services (AWS)
Title: Build Developer Experience Teams for Open Source
Abstract: Open Source has become the default strategy for many IT organizations and Enterprises. However, the constant challenge with Open Source leaders of these organizations has been -
How is my product's developer experience?
Is this the right metric to track?
How can I scale my team to support our products better?
How can I add automation to scale redundant workflows?
If my product involves working with developers, how can I scale to the complexity of the requests and reduce Engineering bandwidth?
The challenges within support of open source products continues to magnify depending on the end user persona whether they are consumers or contributors to your product. Consumers utilize your product, SDK's and API's and are blocked with using it or run into issues, whereas contributors are advanced users of your software that understands the codebase to provide a meaningful contribution back to the product.
The answer to the above is to look at Open Source support as a first-class citizen of your corporate support strategy. To employ the right level of developer focused support as opposed to traditional infrastructure based support is key to scale to the amount of developers using your product. Supporting customers in the open involves more than pure support - building customer / developer experiences (DX) in the open (across platforms and communities) that pivots over the ability of your product's users or developers to be focused on the end-to-end value add. This helps with your active developer growth and retention of users.
Key Takeaways:
- IT leaders of Open Source will learn to employ strategies to build a DX team that engages on multiple platforms
- Work on identifying accurate metrics for product and organization
- Innovate on platforms such as Discord to build a bot and a dashboard
- Ability to leverage customer feedback and iterate over the customer success flywheel
- Distinguish between DX and Developer Advocacy (DA)
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Deploying Models at Scale with Apache Beam
Presented at All Things Open 2023
Presented by Danny McCormick - Google
Title: Deploying Models at Scale with Apache Beam
Abstract: Apache Beam is an open source tool for building distributed scalable data pipelines. This talk will explore how Beam can be used to perform common machine learning tasks, with a heavy focus on running inference at scale. The talk will include a demo component showing how Beam can be used to deploy and update models efficiently on both CPUs and GPUs for inference workloads.
An attendee can expect to leave this talk with a high level understanding of Beam, the challenges of deploying models at scale, and the ability to use Beam to easily parallelize their inference workloads.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Sudo – Giving access while staying in control
Presented at All Things Open 2023
Presented by Peter Czanik - One Identity
Title: Sudo – Giving access while staying in control
Abstract: Sudo is used by millions to control and log administrator access to systems, but using the default configuration only, there are plenty of blind spots. Using the latest features in sudo let you watch some previously blind spots and control access to them. Here are four major new features, which arrived since the 1.9.0 release, allowing you see your blind spots:
- configuring a working directory or chroot within sudo often makes full shell access redundant
- JSON-formatted logs give you more details on events and are easier to act on
- relays in sudo_logsrvd make session recording collection more secure and reliable
- you can log and control sub-commands executed by the command run through sudo
Let us take a closer look at each of these.
Previously, there were quite a few situations where you had to give users full shell access through sudo. Typical examples include when you need to run a command from a given directory, or running commands in a chroot environment. You can now configure the working directory or the chroot directory and give access only to the command the user really needs.
Logging is a central role of sudo, to see who did what on the system. Using JSON-formatted log messages gives you even more information about events. What is even more: structured logs are easier to act on. Setting up alerting for suspicious events is much easier when you have a single parser to configure for any kind of sudo logs. You can collect sudo logs not only by local syslog, but also by using sudo_logsrvd, the same application used to collect session recordings.
Speaking of session recordings: instead of using a single central server, you can now have multiple levels of sudo_logsrvd relays between the client and the final destination. This allows session collection even if the central server is unavailable, providing you with additional security. It also makes your network configuration simpler.
Finally, you can log sub-commands executed from the command started through sudo. You can see commands started from a shell. No more unnoticed shell access from text editors. Best of all: you can also intercept sub-commands.
These are just a few of the most prominent features helping you to watch and control previous blind spots on your systems. See these and other possibilities in action in some live demos during our presentation.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Presented at All Things Open 2023
Presented by Christine Abernathy - F5, Inc.
Title: Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Abstract: As Artificial Intelligence (AI) and Machine Learning (ML) applications continue to surge, it is crucial to be aware of and address the security risks associated with these technologies. In this talk, Christine will explore AI/ML failure modes, threats, and mitigation strategies. She will guide you through the fundamentals of ML models then introduce you to key security challenges such as adversarial attacks, data poisoning, model inversion, model stealing, and membership inference attacks, using real-world examples to demonstrate their potential impact.
Christine will also discuss privacy and ethical considerations in ML, touching upon techniques like federated learning and shedding light on the current regulatory landscape surrounding security risks. If you are developing AI/ML applications or incorporating AI/ML components into your technology stack, check out this talk. You will walk away with a deeper understanding of the current AI/ML security landscape and a toolkit to help you address these risks, enabling you to build safer, more secure, and privacy-aware applications.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Presented at All Things Open 2023
Presented by Carlos Santana - AWS
Title: Securing Cloud Resources Deployed with Control Planes on Kubernetes using Governance and Policy as Code
Abstract: Are you concerned about the security of your cloud resources deployed on Kubernetes? Are you struggling to ensure compliance with regulatory requirements while managing your cloud infrastructure? If yes, then this talk is for you!
We will discuss how to secure cloud resources deployed with Crossplane on Kubernetes using Governance and Policy as Code. We will explore how to leverage Governance and Policy as Code tools like Rego, Kyverno, and OPA to ensure security and compliance.
By the end of this talk, you will have a better understanding of the challenges associated with securing cloud resources deployed with Crossplane or ACK on Kubernetes, the importance of Governance and Policy as Code in ensuring security and compliance, and why it is critical to use open source and open standards in these technologies.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
More from All Things Open (20)
Building Reliability - The Realities of Observability
Building Reliability - The Realities of Observability
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
The State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil Nash
What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Configuration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit Intercept
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Recently uploaded
"What does it really mean for your system to be available, or how to define w...
We will talk about system monitoring from a few different angles. We will start by covering the basics, then discuss SLOs, how to define them, and why understanding the business well is crucial for success in this exercise.
Astute Business Solutions | Oracle Cloud Partner |
Your goto partner for Oracle Cloud, PeopleSoft, E-Business Suite, and Ellucian Banner. We are a firm specialized in managed services and consulting.
From Natural Language to Structured Solr Queries using LLMs
This talk draws on experimentation to enable AI applications with Solr. One important use case is to use AI for better accessibility and discoverability of the data: while User eXperience techniques, lexical search improvements, and data harmonization can take organizations to a good level of accessibility, a structural (or “cognitive” gap) remains between the data user needs and the data producer constraints.
That is where AI – and most importantly, Natural Language Processing and Large Language Model techniques – could make a difference. This natural language, conversational engine could facilitate access and usage of the data leveraging the semantics of any data source.
The objective of the presentation is to propose a technical approach and a way forward to achieve this goal.
The key concept is to enable users to express their search queries in natural language, which the LLM then enriches, interprets, and translates into structured queries based on the Solr index’s metadata.
This approach leverages the LLM’s ability to understand the nuances of natural language and the structure of documents within Apache Solr.
The LLM acts as an intermediary agent, offering a transparent experience to users automatically and potentially uncovering relevant documents that conventional search methods might overlook. The presentation will include the results of this experimental work, lessons learned, best practices, and the scope of future work that should improve the approach and make it production-ready.
A Deep Dive into ScyllaDB's Architecture
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless.
As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency.
We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
This LF Energy webinar took place June 20, 2024. It featured:
-Alex Thornton, LF Energy
-Hallie Cramer, Google
-Daniel Roesler, UtilityAPI
-Henry Richardson, WattTime
In response to the urgency and scale required to effectively address climate change, open source solutions offer significant potential for driving innovation and progress. Currently, there is a growing demand for standardization and interoperability in energy data and modeling. Open source standards and specifications within the energy sector can also alleviate challenges associated with data fragmentation, transparency, and accessibility. At the same time, it is crucial to consider privacy and security concerns throughout the development of open source platforms.
This webinar will delve into the motivations behind establishing LF Energy’s Carbon Data Specification Consortium. It will provide an overview of the draft specifications and the ongoing progress made by the respective working groups.
Three primary specifications will be discussed:
-Discovery and client registration, emphasizing transparent processes and secure and private access
-Customer data, centering around customer tariffs, bills, energy usage, and full consumption disclosure
-Power systems data, focusing on grid data, inclusive of transmission and distribution networks, generation, intergrid power flows, and market settlement data
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
ScyllaDB monitoring provides a lot of useful information. But sometimes it’s not easy to find the root of the problem if something is wrong or even estimate the remaining capacity by the load on the cluster. This talk shares our team's practical tips on: 1) How to find the root of the problem by metrics if ScyllaDB is slow 2) How to interpret the load and plan capacity for the future 3) Compaction strategies and how to choose the right one 4) Important metrics which aren’t available in the default monitoring setup.
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
These are the slides for the presentation, "Component Testing: Bridging the gap between frontend applications" that was presented at QA or the Highway 2024 in Columbus, OH by Zachary Hamm.
GNSS spoofing via SDR (Criptored Talks 2024)
In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security.
This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing.
The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.
Containers & AI - Beauty and the Beast!?!
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Keywords: AI, Containeres, Kubernetes, Cloud Native
Event Link: https://meine.doag.org/events/cloudland/2024/agenda/#agendaId.4211
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation Functions to Prevent Interaction with Malicious QR Codes.
Aim of the Study: The goal of this research was to develop a robust hybrid approach for identifying malicious and insecure URLs derived from QR codes, ensuring safe interactions.
This is achieved through:
Machine Learning Model: Predicts the likelihood of a URL being malicious.
Security Validation Functions: Ensures the derived URL has a valid certificate and proper URL format.
This innovative blend of technology aims to enhance cybersecurity measures and protect users from potential threats hidden within QR codes 🖥 🔒
This study was my first introduction to using ML which has shown me the immense potential of ML in creating more secure digital environments!
"Choosing proper type of scaling", Olena Syrota
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
Session 1 - Intro to Robotic Process Automation.pdf
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
Recently uploaded (20)
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Building Good Containers for Python Applications
- 1. Building Containers for Python Applications Moshe Zadka – https://cobordism.com 2021
- 2. Acknowledgement of Country Belmont (in San Francisco Bay Area Peninsula) Ancestral homeland of the Ramaytush Ohlone people
- 3. What is good
- 4. What is good I To crush your enemies
- 5. What is good I To crush your enemies I To see them driven before you
- 6. What is good I To crush your enemies I To see them driven before you I Um, wrong slides
- 7. What is good I Fast
- 8. What is good I Fast I Small
- 9. What is good I Fast I Small I Secure
- 10. What is good I Fast I Small I Secure I Usable
- 11. Specifying the requirements Let’s be more concrete I Keep up to date
- 12. Specifying the requirements Let’s be more concrete I Keep up to date I Reproducible builds
- 13. Specifying the requirements Let’s be more concrete I Keep up to date I Reproducible builds I No compilers in prod
- 14. Specifying the requirements Let’s be more concrete I Keep up to date I Reproducible builds I No compilers in prod I Keep size (reasonably) small
- 15. Up to date Install security updates
- 16. Up to date Install security updates But when?
- 17. Up to date Install security updates But when? Depends!
- 18. Reproducible builds Same code gives same results
- 19. Reproducible builds Same code gives same results ...mostly
- 20. No compilers in prod A common anti-pattern
- 21. No compilers in prod A common anti-pattern ...surprisingly easy to get wrong!
- 23. Size I Diminishing returns I Cost savings
- 24. Support binary wheels Installing and building
- 25. Support binary wheels Installing and building Faster
- 26. Support binary wheels Installing and building Faster Simplifies images
- 27. Not run as root General hygiene
- 28. Minimal privileges Especially avoid permissions to pip install
- 30. Base OS The distro wars are back?
- 31. Base - size Most modern distros have a decent minimal server
- 32. Base - size Most modern distros have a decent minimal server ...but Debian is easiest to get smallest.
- 33. Base - LTS/support Usually around 5 years
- 34. Base - LTS/support Usually around 5 years Gives you time to upgrade!
- 35. Base - Volatility How much change? Security? Backports? Fixes?
- 37. Ubuntu LTS: 5 years (Universe, Multiverse, etc...) Fairly conservative
- 38. Alpine (probably not) Uses musl, not manylinux compatible
- 39. Rolling releases (probably not) Up to date, but...
- 40. Rolling releases (probably not) Up to date, but... updates can change major versions!
- 42. How to get Python? So many options...
- 43. Not system Python Distros aim Python at distro packages
- 44. Not system Python Distros aim Python at distro packages not user programs.
- 45. Appropriate repositories Famous examples: deadsnakes PPA for Ubuntu
- 46. pyenv Builds and installs Python
- 47. python-build Builds and installs Python
- 48. Source RUN c o n f i g u r e [ . . . ] RUN make RUN make i n s t a l l
- 49. Source RUN c o n f i g u r e [ . . . ] RUN make RUN make i n s t a l l Build from source + Debian?
- 50. Source RUN c o n f i g u r e [ . . . ] RUN make RUN make i n s t a l l Build from source + Debian?python: images are basically that!
- 51. Trade-offs Control vs. Work vs. Problems
- 52. Versions Support multiple for upgrade path
- 53. Versions Support multiple for upgrade path 2-3
- 54. Container multistage build (quick recap) Only one stage output
- 55. Container multistage build (quick recap) Only one stage output other stages help
- 56. FROM Use previous stage as starting image
- 57. COPY –from Copy files from previous stage
- 58. Stages a as modules FROM ubuntu as s e c u r i t y −updates RUN add−apt−r e p o s i t o r y ppa : deadsnakes /ppa RUN apt−get update RUN apt−get upgrade FROM s e c u r i t y −updates as with −38 RUN apt−get i n s t a l l python3 .8 FROM s e c u r i t y −updates as with −39 RUN apt−get i n s t a l l python3 .9
- 59. Separate build and runtime Especially when building from source!
- 60. Separate build and runtime Especially when building from source! FROM ubuntu as b u i l d e r # i n s t a l l b u i l d dependencies # b u i l d Python i n t o / opt /myorg/ python FROM ubuntu as as runtime COPY −−from=b u i l d e r / opt /myorg/ python / opt /myorg/ python
- 61. Optimizing layers Put everything under /opt/myorg Use one COPY −−from=...
- 62. Optimizing size After building Python, remove: I Tests I Builder dependencies (in runtime) I ....and more
- 63. Binary wheels I Build with builder I Copy to runtime I Install in virtual environment
- 64. Binary wheels (alt) I Build with builder I Install in virtual environment I Copy virtual environment to runtime
- 65. Patchelf Used to make wheels self-contained Newest version needed
- 66. Auditwheel Use pip to install
- 67. Self-contained binary wheels Run auditwheel r e p a i r −−platform l i n u x x 8 6 6 4
- 68. Self-contained binary wheels Run auditwheel r e p a i r −−platform l i n u x x 8 6 6 4 No need for binary dependencies!
- 69. Portable binary wheels I Oldest supported?
- 70. Portable binary wheels I Oldest supported? Example: auditwheel r e p a i r −−platform manylinux 2 27 x86 64
- 71. Generating binary wheels Build instructions in docs
- 72. Generating binary wheels Build instructions in docs Build dependencies
- 75. Optimizing caching Where to build wheel?
- 76. Optimizing caching Where to build wheel? What invalidates caching?
- 77. Conclusion I Wrong easier than right
- 78. Conclusion I Wrong easier than right I But right is amazing
- 79. Conclusion I Wrong easier than right I But right is amazing I Think before you docker
- 80. Further Resources Itamar’s series – https://pythonspeed.com/docker/