Puppet Camp LA 2015 talk covering: packages, package managers, puppet, and tips, tricks, and puppet modules for setting up secure package repositories.
Package manages and Puppet - PuppetConf 2015ice799
This talk will begin by explaining what a package manager is and how package managers work, at a high level. Next, we'll observe the common patterns seen on the internet of compiling software in a Puppet manifest and discuss why this not ideal. This talk will conclude by showing how you can add package repositories to your infrastructure using Puppet and what settings are important for ensuring secure access to remote package repositories.
Infrastructure as code might be literally impossible part 2ice799
The document discusses various issues with infrastructure as code including complexities that arise from software licenses, bugs, and inconsistencies across tools and platforms. Specific examples covered include problems with SSL and APT package management on Debian/Ubuntu, Linux networking configuration difficulties, and inconsistencies in Python packaging related to naming conventions for packages containing hyphens, underscores, or periods. Potential causes discussed include legacy code, lack of time for thorough testing and bug fixing, and economic pressures against developing fully working software systems.
my talk from highload++ 2013 -- talking about scaling compiled applications but from the point of view of scaling up from supporting 1 platform to supporting MANY platforms.
in other words: given an application that supports ubuntu 10.04, what sort of systems, tips, and tricks are needed to help scale support to other ubuntus, redhats, centos, windows, etc.
Infrastructure as code might be literally impossibleice799
The document discusses many challenges and issues with using infrastructure as code. It notes that code operates outside of one's frame of reference unless every line is read thoroughly. It provides examples of bugs that have caused major performance problems and outlines vulnerabilities in common package management systems. The document argues that true reproducible infrastructure will require better computer system building and being honest about technology's limitations.
Puppet Camp LA 2015: Package Managers and Puppet (Beginner)Puppet
This document discusses package managers, Puppet, and creating software package repositories. It recommends using tools like createrepo and reprepro to create RPM and Debian package repositories, and signing them with GPG for security. Puppet modules can automate repository creation and configuration. The document emphasizes the importance of secure practices like GPG signing, HTTPS, and installing necessary verification libraries. Overall it provides guidance on best practices for managing and distributing software packages across infrastructure.
Packaging is the Worst Way to Distribute Software, Except for Everything Elsemckern
As part of the 2014 USENIX Release Engineering Summit West, I presented a talk about packaging software and what's wrong with current trends.
Here's the abstract:
Reliably distributing software is a notoriously difficult problem, and almost every operating system and programming language vendor has tried to solve it. This has led to a herd of packaging systems, almost none of which are cross-compatible; some manage system-level software, while others focus on extending their own language (often by trampling on system-level software). And like all competing standards, every packaging system comes with its own sharp corners, dull edges, and hidden idiosyncrasies to deal with along the path to packaging happiness. In an attempt to answer the question "How do I install this software and ensure that its dependencies are fulfilled?", some novel solutions have begun to see popular adoption. But a lot of these newer tools and techniques tread the same ground as their predecessors while overlooking the lessons that were learned along the way.
I'll talk about the state of native packaging systems on some popular platforms (Debian/Ubuntu, RHEL/CentOS/Fedora, and Mac OS X), packaging systems for popular languages (Ruby, Python, Perl, and Node) and the ways that developers are attempting to work around the limitations of these systems. I'll review the reasons that tools like curlbash, FPM, and omnibus packages have become popular by sharing lessons I've learned while working through these systems. While this will be an amusing presentation, I'll show how native packages can address the concerns that have pushed Release Engineers and Developers away. I will also talk about what native packaging systems can learn from the next generation of packaging tools.
The original abstract is available here:
https://www.usenix.org/conference/ures14west/summit-program/presentation/mckern
The document discusses best practices for writing a C/C++ Python extension in 2017. It covers available options like ctypes, cffi, Cython, and SWIG. It then focuses on building a binary Python extension using ctypes, including debugging crashes by generating core files and using lldb/gdb. It also discusses memory issues and using valgrind and clang sanitizers. It recommends abusing Python unit tests for testing C code. Finally, it covers shipping the extension, including manylinux wheels, testing wheels on different Linux distributions with Docker, and publishing source and wheel distributions.
Puppet Camp LA 2015 talk covering: packages, package managers, puppet, and tips, tricks, and puppet modules for setting up secure package repositories.
Package manages and Puppet - PuppetConf 2015ice799
This talk will begin by explaining what a package manager is and how package managers work, at a high level. Next, we'll observe the common patterns seen on the internet of compiling software in a Puppet manifest and discuss why this not ideal. This talk will conclude by showing how you can add package repositories to your infrastructure using Puppet and what settings are important for ensuring secure access to remote package repositories.
Infrastructure as code might be literally impossible part 2ice799
The document discusses various issues with infrastructure as code including complexities that arise from software licenses, bugs, and inconsistencies across tools and platforms. Specific examples covered include problems with SSL and APT package management on Debian/Ubuntu, Linux networking configuration difficulties, and inconsistencies in Python packaging related to naming conventions for packages containing hyphens, underscores, or periods. Potential causes discussed include legacy code, lack of time for thorough testing and bug fixing, and economic pressures against developing fully working software systems.
my talk from highload++ 2013 -- talking about scaling compiled applications but from the point of view of scaling up from supporting 1 platform to supporting MANY platforms.
in other words: given an application that supports ubuntu 10.04, what sort of systems, tips, and tricks are needed to help scale support to other ubuntus, redhats, centos, windows, etc.
Infrastructure as code might be literally impossibleice799
The document discusses many challenges and issues with using infrastructure as code. It notes that code operates outside of one's frame of reference unless every line is read thoroughly. It provides examples of bugs that have caused major performance problems and outlines vulnerabilities in common package management systems. The document argues that true reproducible infrastructure will require better computer system building and being honest about technology's limitations.
Puppet Camp LA 2015: Package Managers and Puppet (Beginner)Puppet
This document discusses package managers, Puppet, and creating software package repositories. It recommends using tools like createrepo and reprepro to create RPM and Debian package repositories, and signing them with GPG for security. Puppet modules can automate repository creation and configuration. The document emphasizes the importance of secure practices like GPG signing, HTTPS, and installing necessary verification libraries. Overall it provides guidance on best practices for managing and distributing software packages across infrastructure.
Packaging is the Worst Way to Distribute Software, Except for Everything Elsemckern
As part of the 2014 USENIX Release Engineering Summit West, I presented a talk about packaging software and what's wrong with current trends.
Here's the abstract:
Reliably distributing software is a notoriously difficult problem, and almost every operating system and programming language vendor has tried to solve it. This has led to a herd of packaging systems, almost none of which are cross-compatible; some manage system-level software, while others focus on extending their own language (often by trampling on system-level software). And like all competing standards, every packaging system comes with its own sharp corners, dull edges, and hidden idiosyncrasies to deal with along the path to packaging happiness. In an attempt to answer the question "How do I install this software and ensure that its dependencies are fulfilled?", some novel solutions have begun to see popular adoption. But a lot of these newer tools and techniques tread the same ground as their predecessors while overlooking the lessons that were learned along the way.
I'll talk about the state of native packaging systems on some popular platforms (Debian/Ubuntu, RHEL/CentOS/Fedora, and Mac OS X), packaging systems for popular languages (Ruby, Python, Perl, and Node) and the ways that developers are attempting to work around the limitations of these systems. I'll review the reasons that tools like curlbash, FPM, and omnibus packages have become popular by sharing lessons I've learned while working through these systems. While this will be an amusing presentation, I'll show how native packages can address the concerns that have pushed Release Engineers and Developers away. I will also talk about what native packaging systems can learn from the next generation of packaging tools.
The original abstract is available here:
https://www.usenix.org/conference/ures14west/summit-program/presentation/mckern
The document discusses best practices for writing a C/C++ Python extension in 2017. It covers available options like ctypes, cffi, Cython, and SWIG. It then focuses on building a binary Python extension using ctypes, including debugging crashes by generating core files and using lldb/gdb. It also discusses memory issues and using valgrind and clang sanitizers. It recommends abusing Python unit tests for testing C code. Finally, it covers shipping the extension, including manylinux wheels, testing wheels on different Linux distributions with Docker, and publishing source and wheel distributions.
The document provides an overview of a talk given by Stephen Wallace on using Puppet for system administrators. The talk introduces Puppet as a tool to help system administrators achieve goals like availability, scalability, predictability and reducing workload. It addresses common concerns that system administrators have with Puppet, such as the need to learn programming. The talk demonstrates how Puppet can be used in a simple way and provides references for further learning.
This document summarizes Yoshiki Shibukawa's presentation on building multi-platform GUI applications with Go. Shibukawa explored several approaches: wrapping existing toolkits, creating a new toolkit in Go, and a hybrid C++/Go approach. While wrapping toolkits is intuitive, maintaining wrappers is difficult. Creating a new toolkit in Go allows perfect integration but supporting multiple platforms is challenging. The hybrid approach uses C++ for GUI with Go handling logic, which provides better integration but requires managing inter-process communication. Shibukawa concluded more work is needed to find the best solution and that GUI programming remains difficult, potentially requiring C++ involvement.
Augeas, swiss knife resources for your puppet treeJulien Pivotto
This document provides an overview of Puppet resources for managing files, including the File resource, Concat module, Exec commands, and Augeas. The File resource is most commonly used and works for many situations. The Concat module offers more flexibility but also more complexity. Exec commands with sed/grep should generally be avoided. Augeas provides a powerful way to edit configuration files while preserving formatting and only changing what is needed.
Infrastructure as code might be literally impossible / Joe Domato (packageclo...Ontico
HighLoad++ 2017
Зал «Мумбай», 7 ноября, 12:00
Тезисы:
http://www.highload.ru/2017/abstracts/2918.html
This talk will begin by briefly examining what it means for infrastructure to be represented as code. We'll examine some fundamental software components required for automating infrastructure such as GPG, package managers, SSL, and more. We'll examine some interesting failure cases for these tools and how these shortcomings might make infrastructure as code impossible, for now.
Puppet at DemonWare - Ruaidhri Power - Puppetcamp Dublin '12Puppet
This talk will describe the evolution of how we've used Puppet at Demonware, a subsidiary of Activision Blizzard, to run the infrastructure of some of the world's biggest games, supporting millions of concurrent users for titles such as Call of Duty.
Ruaidhri Power of DemonWare at PuppetCamp Dublin '12. http://www.puppetlabs.com
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)Tim Bunce
Slides of my talk on Devel::NYTProf and optimizing perl code at the Italian Perl Workshop (IPW09). It covers the new features in NYTProf v3 and a new section outlining a multi-phase approach to optimizing your perl code.
30 mins long plus 10 mins of questions. Best viewed fullscreen.
Modern Perl for the Unfrozen Paleolithic Perl ProgrammerJohn Anderson
Modern Perl for the Unfrozen Paleolithic Perl Programmer discusses many changes to Perl and its ecosystem since 2001 when the caveman programmer fell into a glacier. Some key changes include Perl version 5.22, new language features like defined-or and subroutine signatures, and tools like Perlbrew, Plenv and cpanm. Modern web development uses Plack/PSGI instead of CGI.pm. The speaker recommends modules like JSON::MaybeXS, Moose/Moo and websites like CPAN Ratings, MetaCPAN and Perl Weekly to stay up-to-date.
John presents several tools and techniques he uses to automate tasks and maintain consistency across systems in order to maximize his productivity while developing software. Some of the key tools and strategies he discusses include: App::MiseEnPlace for managing directory structures and symlinks; smartcd for running scripts when entering or leaving directories; building critical tools like Perl, Node.js, and Git from source instead of relying on system versions; and keeping his entire $HOME directory under revision control with GitGot. He emphasizes automating repetitive tasks, maintaining consistency across systems, and not having to think about tools or environments.
A presentation given at DeveloperWeek in San Francisco by Zack Argyle. It goes through important concepts in building out reusable React components, releasing it to Github, and publishing it to NPM. There are best practices and suggestions with an example component.
This document provides an overview and introduction to Puppet, an open source tool for configuration management. It discusses what Puppet is, how it works, its main components like the Puppet Master, Puppet Agent, and Facter for gathering system facts. It also covers Puppet manifests which define configurations declaratively using the Puppet DSL language.
FUSE allows processes to mount their own private filesystems without requiring root privileges. Some examples of FUSE filesystems include encrypted volumes using encfs and remote filesystems mounted over SSH using sshfs. These filesystems can be mounted automatically and only be visible to the mounting process, providing security and privacy for personal data even from the root user.
Puppet Camp Atlanta 2014: DEV Toolsets for Ops (Beginner) - Puppet
The document discusses best practices for developing Puppet code in a DevOps environment. It recommends keeping code in branches in version control systems like Git or SVN, writing syntax-checked and tested code locally in virtual environments, and sharing code publicly through tools like Puppet Forge and privately through Hiera. Automating processes like testing, deployments, and sharing code are also emphasized.
find & improve some bottleneck in Debian project (DebConf14 LT)Hideki Yamane
This document discusses identifying and addressing bottlenecks in the Debian project to improve efficiency and attract more contributors. It proposes adding a preprocessing step for new packages where trusted contributors could provide an initial review to catch obvious issues early. This would transition the new package review process from a serial model dependent on ftpmasters to a parallel model utilizing hundreds of contributors. Metrics like daily package intake and time in the new queue would help evaluate the success of these changes.
Deploying 3 times a day without a downtime @ Rocket Tech Summit in BerlinAlessandro Nadalin
A look at how we try to make our architecture robust, resilient and fun to work with: Namshi is not github or spotify but... ...imitation is the sincerest form of flattery!
Bootstrapping Puppet and Application Deployment - PuppetConf 2013Puppet
"Bootstrapping Puppet and Application Deployment" by Robert de Macedo Soares, Application Security Engineer, BusinessWire.
Presentation Overview: A dive into the problems faced when first launching Puppet across existing, heterogeneous servers, outlining possible solutions using our experience as an example. In addition, this session will touch on application management and deployment using subversion and rake tasks, what works and what is a little rough around the edges.
Speaker Bio: Robert is an engineer who has spent the past several years attempting to automate away the need for the work that he does. Focusing on server automation and security work for BusinessWire, Robert also develops web services such as tee.ms, a chat service, and designs and develops games. Trism, which he co-designed, was nominated for Cellular Game of the Year by the Academy of Interactive Arts & Sciences in the 2009 Interactive Achievement Awards.
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013Puppet
"Testing for Ops: Going Beyond the Manifest" by Christopher Webber, Infrastructure Engineer, Demand Media.
Presentation Overview: This talk aims to show the value of rspec-puppet for those who come from a more Ops-centric background. The focus will be on using tests to go beyond just rewriting manifests in rspec. Instead the focus will be on scenarios like: - Are the baseline security measures in place? - Do the differences between dev and prod get reflected? - Are the config elements that are core to the application present? In addition, tests will help to be a place to help document the oddities of our configurations and ensuring that minor changes don't result in catastrophe.
Speaker Bio: After beginning his career at UC Riverside supporting enterprise operations and bioinformatics research, Chris is now rocking being an infrastructure engineer at Demand Media in Santa Monica. He currently supports large high-traffic sites like eHow.com, LiveSTRONG.com, and Cracked.com. Chris enjoys attending local meetups, writing new Puppet modules, and creating small tools to make his team's lives a little easier. Find him on Twitter as @cwebber.
Fine-tuning your development environment means more than just getting your editor set up just so -- it means finding and setting up a variety of tools to take care of the mundane housekeeping chores that you have to do -- so you have more time to program, of course! I'll share the benefits of a number of yak shaving expeditions, including using App::GitGot to batch manage _all_ your git repos, App::MiseEnPlace to automate getting things _just_ so in your working environment, and a few others as time allows.
Delivered at OpenWest 2016, 13 July 2016
Porting a command line tool to Android involves cross-compiling the code using the Android NDK toolchain, which may require patching the code to address issues like different file paths, endianness, and library dependencies. While compiling and running static binaries is straightforward, dynamic binaries require position-independent executable (PIE) support added in Android 5. Calling native executables from Android code requires using Runtime.exec() or ProcessBuilder and parsing output streams. Special care needs to be taken to avoid security issues like command injection when passing untrusted inputs to native programs run as root on Android.
about Debian "squeeze" @201002 OSC TokyospringHideki Yamane
The document discusses the upcoming Debian 6.0 release codenamed "Squeeze". It provides details on the development process, expected release timeline in 2010, and highlights some of the new components and packages that will be included such as Linux 2.6.32, GNOME 2.28, KDE 4.3, and Perl 5.10. It also discusses the process for moving packages from testing to stable and addressing release critical bugs.
Commonly in startups, you probably don't be worry about how your application artefacts are managed to be deployed in a server: compressed files or JAR/WAR files combined with a set of scripts. However, for larger companies that uses not only one or two servers, but a entire pool of machines, it requires some control and organisation to deploy apps. You can handle this case with some automation / configuration management tools like Ansible, Puppet, Chef, of course. But using a set of scripts can be hard to maintain all deployment steps, from artefact update to rollback.
If you distribute your software using a Operational System package manager like YUM (CentOS) or APT (Debian, Ubuntu), your deployment will be easier because these tools handles software changes in a good way. It sounds good, but package binaries is a pain for developers or newer sysadmins that don't have knowledge about how YUM/APT works. It requires creation of one or a lot of configuration files to be updated and when you have multiple artefacts probably packaging them will be a nightmare.
Here enter fpm-cookery, a Ruby gem responsible to package artefacts based on recipes, much easy to use. It will be your swiss knife for anything that you need to package, from common binaries not distributed in RPM/DEB formats to programming languages from source, compiling when needed.
Using NuGet the way you should
Consuming NuGet packages, that’s what everyone does. Open source projects create NuGet packages and post them on NuGet.org. Meanwhile, all of us are still working with shared projects and fighting relative paths, versioning and so on. In this talk, we’ll use Visual Studio, NuGet and TeamCity to work with NuGet the way you should. Project references must die! Add Package Reference and good continuous integration is everything you will ever need.
The document provides an overview of a talk given by Stephen Wallace on using Puppet for system administrators. The talk introduces Puppet as a tool to help system administrators achieve goals like availability, scalability, predictability and reducing workload. It addresses common concerns that system administrators have with Puppet, such as the need to learn programming. The talk demonstrates how Puppet can be used in a simple way and provides references for further learning.
This document summarizes Yoshiki Shibukawa's presentation on building multi-platform GUI applications with Go. Shibukawa explored several approaches: wrapping existing toolkits, creating a new toolkit in Go, and a hybrid C++/Go approach. While wrapping toolkits is intuitive, maintaining wrappers is difficult. Creating a new toolkit in Go allows perfect integration but supporting multiple platforms is challenging. The hybrid approach uses C++ for GUI with Go handling logic, which provides better integration but requires managing inter-process communication. Shibukawa concluded more work is needed to find the best solution and that GUI programming remains difficult, potentially requiring C++ involvement.
Augeas, swiss knife resources for your puppet treeJulien Pivotto
This document provides an overview of Puppet resources for managing files, including the File resource, Concat module, Exec commands, and Augeas. The File resource is most commonly used and works for many situations. The Concat module offers more flexibility but also more complexity. Exec commands with sed/grep should generally be avoided. Augeas provides a powerful way to edit configuration files while preserving formatting and only changing what is needed.
Infrastructure as code might be literally impossible / Joe Domato (packageclo...Ontico
HighLoad++ 2017
Зал «Мумбай», 7 ноября, 12:00
Тезисы:
http://www.highload.ru/2017/abstracts/2918.html
This talk will begin by briefly examining what it means for infrastructure to be represented as code. We'll examine some fundamental software components required for automating infrastructure such as GPG, package managers, SSL, and more. We'll examine some interesting failure cases for these tools and how these shortcomings might make infrastructure as code impossible, for now.
Puppet at DemonWare - Ruaidhri Power - Puppetcamp Dublin '12Puppet
This talk will describe the evolution of how we've used Puppet at Demonware, a subsidiary of Activision Blizzard, to run the infrastructure of some of the world's biggest games, supporting millions of concurrent users for titles such as Call of Duty.
Ruaidhri Power of DemonWare at PuppetCamp Dublin '12. http://www.puppetlabs.com
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)Tim Bunce
Slides of my talk on Devel::NYTProf and optimizing perl code at the Italian Perl Workshop (IPW09). It covers the new features in NYTProf v3 and a new section outlining a multi-phase approach to optimizing your perl code.
30 mins long plus 10 mins of questions. Best viewed fullscreen.
Modern Perl for the Unfrozen Paleolithic Perl ProgrammerJohn Anderson
Modern Perl for the Unfrozen Paleolithic Perl Programmer discusses many changes to Perl and its ecosystem since 2001 when the caveman programmer fell into a glacier. Some key changes include Perl version 5.22, new language features like defined-or and subroutine signatures, and tools like Perlbrew, Plenv and cpanm. Modern web development uses Plack/PSGI instead of CGI.pm. The speaker recommends modules like JSON::MaybeXS, Moose/Moo and websites like CPAN Ratings, MetaCPAN and Perl Weekly to stay up-to-date.
John presents several tools and techniques he uses to automate tasks and maintain consistency across systems in order to maximize his productivity while developing software. Some of the key tools and strategies he discusses include: App::MiseEnPlace for managing directory structures and symlinks; smartcd for running scripts when entering or leaving directories; building critical tools like Perl, Node.js, and Git from source instead of relying on system versions; and keeping his entire $HOME directory under revision control with GitGot. He emphasizes automating repetitive tasks, maintaining consistency across systems, and not having to think about tools or environments.
A presentation given at DeveloperWeek in San Francisco by Zack Argyle. It goes through important concepts in building out reusable React components, releasing it to Github, and publishing it to NPM. There are best practices and suggestions with an example component.
This document provides an overview and introduction to Puppet, an open source tool for configuration management. It discusses what Puppet is, how it works, its main components like the Puppet Master, Puppet Agent, and Facter for gathering system facts. It also covers Puppet manifests which define configurations declaratively using the Puppet DSL language.
FUSE allows processes to mount their own private filesystems without requiring root privileges. Some examples of FUSE filesystems include encrypted volumes using encfs and remote filesystems mounted over SSH using sshfs. These filesystems can be mounted automatically and only be visible to the mounting process, providing security and privacy for personal data even from the root user.
Puppet Camp Atlanta 2014: DEV Toolsets for Ops (Beginner) - Puppet
The document discusses best practices for developing Puppet code in a DevOps environment. It recommends keeping code in branches in version control systems like Git or SVN, writing syntax-checked and tested code locally in virtual environments, and sharing code publicly through tools like Puppet Forge and privately through Hiera. Automating processes like testing, deployments, and sharing code are also emphasized.
find & improve some bottleneck in Debian project (DebConf14 LT)Hideki Yamane
This document discusses identifying and addressing bottlenecks in the Debian project to improve efficiency and attract more contributors. It proposes adding a preprocessing step for new packages where trusted contributors could provide an initial review to catch obvious issues early. This would transition the new package review process from a serial model dependent on ftpmasters to a parallel model utilizing hundreds of contributors. Metrics like daily package intake and time in the new queue would help evaluate the success of these changes.
Deploying 3 times a day without a downtime @ Rocket Tech Summit in BerlinAlessandro Nadalin
A look at how we try to make our architecture robust, resilient and fun to work with: Namshi is not github or spotify but... ...imitation is the sincerest form of flattery!
Bootstrapping Puppet and Application Deployment - PuppetConf 2013Puppet
"Bootstrapping Puppet and Application Deployment" by Robert de Macedo Soares, Application Security Engineer, BusinessWire.
Presentation Overview: A dive into the problems faced when first launching Puppet across existing, heterogeneous servers, outlining possible solutions using our experience as an example. In addition, this session will touch on application management and deployment using subversion and rake tasks, what works and what is a little rough around the edges.
Speaker Bio: Robert is an engineer who has spent the past several years attempting to automate away the need for the work that he does. Focusing on server automation and security work for BusinessWire, Robert also develops web services such as tee.ms, a chat service, and designs and develops games. Trism, which he co-designed, was nominated for Cellular Game of the Year by the Academy of Interactive Arts & Sciences in the 2009 Interactive Achievement Awards.
Testing for Ops: Going Beyond the Manifest - PuppetConf 2013Puppet
"Testing for Ops: Going Beyond the Manifest" by Christopher Webber, Infrastructure Engineer, Demand Media.
Presentation Overview: This talk aims to show the value of rspec-puppet for those who come from a more Ops-centric background. The focus will be on using tests to go beyond just rewriting manifests in rspec. Instead the focus will be on scenarios like: - Are the baseline security measures in place? - Do the differences between dev and prod get reflected? - Are the config elements that are core to the application present? In addition, tests will help to be a place to help document the oddities of our configurations and ensuring that minor changes don't result in catastrophe.
Speaker Bio: After beginning his career at UC Riverside supporting enterprise operations and bioinformatics research, Chris is now rocking being an infrastructure engineer at Demand Media in Santa Monica. He currently supports large high-traffic sites like eHow.com, LiveSTRONG.com, and Cracked.com. Chris enjoys attending local meetups, writing new Puppet modules, and creating small tools to make his team's lives a little easier. Find him on Twitter as @cwebber.
Fine-tuning your development environment means more than just getting your editor set up just so -- it means finding and setting up a variety of tools to take care of the mundane housekeeping chores that you have to do -- so you have more time to program, of course! I'll share the benefits of a number of yak shaving expeditions, including using App::GitGot to batch manage _all_ your git repos, App::MiseEnPlace to automate getting things _just_ so in your working environment, and a few others as time allows.
Delivered at OpenWest 2016, 13 July 2016
Porting a command line tool to Android involves cross-compiling the code using the Android NDK toolchain, which may require patching the code to address issues like different file paths, endianness, and library dependencies. While compiling and running static binaries is straightforward, dynamic binaries require position-independent executable (PIE) support added in Android 5. Calling native executables from Android code requires using Runtime.exec() or ProcessBuilder and parsing output streams. Special care needs to be taken to avoid security issues like command injection when passing untrusted inputs to native programs run as root on Android.
about Debian "squeeze" @201002 OSC TokyospringHideki Yamane
The document discusses the upcoming Debian 6.0 release codenamed "Squeeze". It provides details on the development process, expected release timeline in 2010, and highlights some of the new components and packages that will be included such as Linux 2.6.32, GNOME 2.28, KDE 4.3, and Perl 5.10. It also discusses the process for moving packages from testing to stable and addressing release critical bugs.
Commonly in startups, you probably don't be worry about how your application artefacts are managed to be deployed in a server: compressed files or JAR/WAR files combined with a set of scripts. However, for larger companies that uses not only one or two servers, but a entire pool of machines, it requires some control and organisation to deploy apps. You can handle this case with some automation / configuration management tools like Ansible, Puppet, Chef, of course. But using a set of scripts can be hard to maintain all deployment steps, from artefact update to rollback.
If you distribute your software using a Operational System package manager like YUM (CentOS) or APT (Debian, Ubuntu), your deployment will be easier because these tools handles software changes in a good way. It sounds good, but package binaries is a pain for developers or newer sysadmins that don't have knowledge about how YUM/APT works. It requires creation of one or a lot of configuration files to be updated and when you have multiple artefacts probably packaging them will be a nightmare.
Here enter fpm-cookery, a Ruby gem responsible to package artefacts based on recipes, much easy to use. It will be your swiss knife for anything that you need to package, from common binaries not distributed in RPM/DEB formats to programming languages from source, compiling when needed.
Using NuGet the way you should
Consuming NuGet packages, that’s what everyone does. Open source projects create NuGet packages and post them on NuGet.org. Meanwhile, all of us are still working with shared projects and fighting relative paths, versioning and so on. In this talk, we’ll use Visual Studio, NuGet and TeamCity to work with NuGet the way you should. Project references must die! Add Package Reference and good continuous integration is everything you will ever need.
Open Source Tools for Leveling Up Operations FOSSET 2014Mandi Walls
This document discusses using open source tools to improve operations workflows and processes. It introduces various tools including Git for version control, packaging tools like FPM, and testing tools like Nagios plugins. The document advocates applying principles from development like testing, version control, and automation to make operations processes more reliable, transparent and reduce risk.
This document discusses deploying software at scale through automation. It advocates treating infrastructure as code and using version control, continuous integration, and packaging tools. The key steps are to automate deployments, make them reproducible, and deploy changes frequently and consistently through a pipeline that checks code, runs tests, builds packages, and deploys to testing and production environments. This allows deploying changes safely and quickly while improving collaboration between developers and operations teams.
The document discusses best practices for building and packaging compiled applications across different platforms. It recommends setting up a build server like Jenkins to enable reproducible builds. Built artifacts should be backed up, such as copying to S3. The document also discusses using tools like chroot, mock, and pbuilder to build in clean environments. It recommends static linking of dependencies where possible and capturing debug symbols to aid in debugging. Automated testing of install/uninstall and correctness is also recommended to catch errors on supported platforms.
This document provides an overview and introduction to various package managers for different programming languages and platforms including: RubyGems, NPM, Bower, NuGet, Paket, Chocolatey/OneGet. It discusses what each package manager is used for, how to install packages, manage dependencies, publish packages, and includes demonstrations of common commands.
Que nos espera a los ALM Dudes para el 2013?Bruno Capuano
The document discusses challenges with application lifecycle management (ALM) and recommends adopting agile practices like Scrum and Kanban to improve project predictability, lower costs, and increase team responsiveness. It emphasizes establishing continuous integration using automated testing, version control like Git, and configuration management. Adopting practices like test-driven development, behavior-driven development, and continuous integration can help address typical ALM problems like lack of visibility, ineffective communication, undefined requirements, and inadequate testing.
This document provides an overview of automated server deployment and configuration using Ansible. It discusses traditional server provisioning processes versus modern approaches using infrastructure as code and configuration management software. It introduces key concepts in Ansible like idempotence and provides examples of installing Apache web server using Ansible playbooks and modules. The document recommends Ansible as an easy to learn configuration management tool and outlines steps to get started, including installing Ansible, configuring inventory files, using modules and writing playbooks. It also discusses using Ansible to manage Docker images and containers.
This document discusses using Vagrant and Chef together to create consistent development environments that match production. It notes that development and production environments often differ, causing bugs. Vagrant allows creating and managing virtual machines easily. Chef automates server configuration through recipes and community cookbooks. The document provides basic instructions for installing Vagrant and VirtualBox then cloning a sample project using Vagrant, Chef and a Git submodule to provision a virtual machine. It concludes by inviting questions.
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & HadoopOlinData
Dennis Matotek, Technical Lead Platforms at Experian Hitwise Australia, gave an excellent presentation on setting up puppet using vagrant, puppet and testing, including a full demo of rspec-puppet and Jenkins.
From PuppetCamp Southeast Asia 2012 in Kuala Lumpur, Malaysia. Hadoop in a box - from playground to production Desc: How Vagrant, Puppet and other tools can be used to move your manifest from test bed to production.
Dennis Matotek, Technical Lead Platforms at Experian Hitwise Australia, gave an excellent presentation on setting up puppet using vagrant, puppet and testing, including a full demo of rspec-puppet and Jenkins.
Thursday, June 12th 2014
Discussing strategies in Rails development for keeping multiple application environments as consistent as possible for the best development, testing, and deployment experience.
This lecture discusses the different techniques used to install, uninstall and upgrade software packages in Linux and the associated tools
Video for this lecture on youtube:
http://www.youtube.com/watch?v=pFqdupd9wKk
Check the other Lectures and courses in
http://Linux4EnbeddedSystems.com
or Follow our Facebook Group at
- Facebook: @LinuxforEmbeddedSystems
Lecturer Profile:
Ahmed ElArabawy
- https://www.linkedin.com/in/ahmedelarabawy
This document describes perlbrew, a tool for installing and switching between multiple Perl versions. It allows installing Perls from tarballs or git, and builds them with customizable options. Perlbrew lists installed Perls, allows switching between them, and keeps Perl installations and libraries isolated. It has benefits like easier cleanups and isolated environments for different apps. The tool is actively developed on GitHub with contributions from many.
The document describes an approach to managing CI/CD pipelines for over 400 .NET solution repositories using Cake, a build automation system for .NET. A master repository is used to define common build, test and package tasks that each individual repository can inherit from. This avoids duplicating code and allows centralized management of the pipelines. An infrastructure is also described that includes Jenkins for running builds, Slack for notifications, Elasticsearch for telemetry, and a custom dashboard for visualizing status.
'Intro to Infrastructure as Code' - DevOps BelfastJohn Fitzpatrick
This document provides an introduction to infrastructure as code and Chef. It discusses how infrastructure as code treats infrastructure configuration like code that can be version controlled, tested, and managed declaratively. It then introduces Chef concepts like resources, recipes, cookbooks, and how they allow defining infrastructure in a declarative way using Ruby DSL. The document uses examples like package, file, template and service resources to illustrate how Chef models infrastructure as code.
A story of how we went about packaging perl and all of the dependencies that our project has.
Where we were before, the chosen path, and the end result.
The pitfalls and a view on the pros and cons of the previous state of affairs versus the pros/cons of the end result.
Conda is a cross-platform package manager that lets you quickly and easily build environments containing complicated software stacks. It was built to manage the NumPy stack in Python but can be used to manage any complex software dependencies.
Similar to Chef Conf 2015: Package Management & Chef (20)
Boost Your Savings with These Money Management AppsJhone kinadey
A money management app can transform your financial life by tracking expenses, creating budgets, and setting financial goals. These apps offer features like real-time expense tracking, bill reminders, and personalized insights to help you save and manage money effectively. With a user-friendly interface, they simplify financial planning, making it easier to stay on top of your finances and achieve long-term financial stability.
Photoshop Tutorial for Beginners (2024 Edition)alowpalsadig
Photoshop Tutorial for Beginners (2024 Edition)
Explore the evolution of programming and software development and design in 2024. Discover emerging trends shaping the future of coding in our insightful analysis."
Here's an overview:Introduction: The Evolution of Programming and Software DevelopmentThe Rise of Artificial Intelligence and Machine Learning in CodingAdopting Low-Code and No-Code PlatformsQuantum Computing: Entering the Software Development MainstreamIntegration of DevOps with Machine Learning: MLOpsAdvancements in Cybersecurity PracticesThe Growth of Edge ComputingEmerging Programming Languages and FrameworksSoftware Development Ethics and AI RegulationSustainability in Software EngineeringThe Future Workforce: Remote and Distributed TeamsConclusion: Adapting to the Changing Software Development LandscapeIntroduction: The Evolution of Programming and Software Development
Photoshop Tutorial for Beginners (2024 Edition)Explore the evolution of programming and software development and design in 2024. Discover emerging trends shaping the future of coding in our insightful analysis."Here's an overview:Introduction: The Evolution of Programming and Software DevelopmentThe Rise of Artificial Intelligence and Machine Learning in CodingAdopting Low-Code and No-Code PlatformsQuantum Computing: Entering the Software Development MainstreamIntegration of DevOps with Machine Learning: MLOpsAdvancements in Cybersecurity PracticesThe Growth of Edge ComputingEmerging Programming Languages and FrameworksSoftware Development Ethics and AI RegulationSustainability in Software EngineeringThe Future Workforce: Remote and Distributed TeamsConclusion: Adapting to the Changing Software Development LandscapeIntroduction: The Evolution of Programming and Software Development
The importance of developing and designing programming in 2024
Programming design and development represents a vital step in keeping pace with technological advancements and meeting ever-changing market needs. This course is intended for anyone who wants to understand the fundamental importance of software development and design, whether you are a beginner or a professional seeking to update your knowledge.
Course objectives:
1. **Learn about the basics of software development:
- Understanding software development processes and tools.
- Identify the role of programmers and designers in software projects.
2. Understanding the software design process:
- Learn about the principles of good software design.
- Discussing common design patterns such as Object-Oriented Design.
3. The importance of user experience (UX) in modern software:
- Explore how user experience can improve software acceptance and usability.
- Tools and techniques to analyze and improve user experience.
4. Increase efficiency and productivity through modern development tools:
- Access to the latest programming tools and languages used in the industry.
- Study live examples of applications
What to do when you have a perfect model for your software but you are constrained by an imperfect business model?
This talk explores the challenges of bringing modelling rigour to the business and strategy levels, and talking to your non-technical counterparts in the process.
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...kalichargn70th171
Visual testing plays a vital role in ensuring that software products meet the aesthetic requirements specified by clients in functional and non-functional specifications. In today's highly competitive digital landscape, users expect a seamless and visually appealing online experience. Visual testing, also known as automated UI testing or visual regression testing, verifies the accuracy of the visual elements that users interact with.
A neural network is a machine learning program, or model, that makes decisions in a manner similar to the human brain, by using processes that mimic the way biological neurons work together to identify phenomena, weigh options and arrive at conclusions.
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISTier1 app
Are you ready to unlock the secrets hidden within Java thread dumps? Join us for a hands-on session where we'll delve into effective troubleshooting patterns to swiftly identify the root causes of production problems. Discover the right tools, techniques, and best practices while exploring *real-world case studies of major outages* in Fortune 500 enterprises. Engage in interactive lab exercises where you'll have the opportunity to troubleshoot thread dumps and uncover performance issues firsthand. Join us and become a master of Java thread dump analysis!
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Paul Brebner
Closing talk for the Performance Engineering track at Community Over Code EU (Bratislava, Slovakia, June 5 2024) https://eu.communityovercode.org/sessions/2024/why-apache-kafka-clusters-are-like-galaxies-and-other-cosmic-kafka-quandaries-explored/ Instaclustr (now part of NetApp) manages 100s of Apache Kafka clusters of many different sizes, for a variety of use cases and customers. For the last 7 years I’ve been focused outwardly on exploring Kafka application development challenges, but recently I decided to look inward and see what I could discover about the performance, scalability and resource characteristics of the Kafka clusters themselves. Using a suite of Performance Engineering techniques, I will reveal some surprising discoveries about cosmic Kafka mysteries in our data centres, related to: cluster sizes and distribution (using Zipf’s Law), horizontal vs. vertical scalability, and predicting Kafka performance using metrics, modelling and regression techniques. These insights are relevant to Kafka developers and operators.
Alluxio Webinar | 10x Faster Trino Queries on Your Data PlatformAlluxio, Inc.
Alluxio Webinar
June. 18, 2024
For more Alluxio Events: https://www.alluxio.io/events/
Speaker:
- Jianjian Xie (Staff Software Engineer, Alluxio)
As Trino users increasingly rely on cloud object storage for retrieving data, speed and cloud cost have become major challenges. The separation of compute and storage creates latency challenges when querying datasets; scanning data between storage and compute tiers becomes I/O bound. On the other hand, cloud API costs related to GET/LIST operations and cross-region data transfer add up quickly.
The newly introduced Trino file system cache by Alluxio aims to overcome the above challenges. In this session, Jianjian will dive into Trino data caching strategies, the latest test results, and discuss the multi-level caching architecture. This architecture makes Trino 10x faster for data lakes of any scale, from GB to EB.
What you will learn:
- Challenges relating to the speed and costs of running Trino in the cloud
- The new Trino file system cache feature overview, including the latest development status and test results
- A multi-level cache framework for maximized speed, including Trino file system cache and Alluxio distributed cache
- Real-world cases, including a large online payment firm and a top ridesharing company
- The future roadmap of Trino file system cache and Trino-Alluxio integration
How Can Hiring A Mobile App Development Company Help Your Business Grow?ToXSL Technologies
ToXSL Technologies is an award-winning Mobile App Development Company in Dubai that helps businesses reshape their digital possibilities with custom app services. As a top app development company in Dubai, we offer highly engaging iOS & Android app solutions. https://rb.gy/necdnt
Odoo releases a new update every year. The latest version, Odoo 17, came out in October 2023. It brought many improvements to the user interface and user experience, along with new features in modules like accounting, marketing, manufacturing, websites, and more.
The Odoo 17 update has been a hot topic among startups, mid-sized businesses, large enterprises, and Odoo developers aiming to grow their businesses. Since it is now already the first quarter of 2024, you must have a clear idea of what Odoo 17 entails and what it can offer your business if you are still not aware of it.
This blog covers the features and functionalities. Explore the entire blog and get in touch with expert Odoo ERP consultants to leverage Odoo 17 and its features for your business too.
An Overview of Odoo ERP
Odoo ERP was first released as OpenERP software in February 2005. It is a suite of business applications used for ERP, CRM, eCommerce, websites, and project management. Ten years ago, the Odoo Enterprise edition was launched to help fund the Odoo Community version.
When you compare Odoo Community and Enterprise, the Enterprise edition offers exclusive features like mobile app access, Odoo Studio customisation, Odoo hosting, and unlimited functional support.
Today, Odoo is a well-known name used by companies of all sizes across various industries, including manufacturing, retail, accounting, marketing, healthcare, IT consulting, and R&D.
The latest version, Odoo 17, has been available since October 2023. Key highlights of this update include:
Enhanced user experience with improvements to the command bar, faster backend page loading, and multiple dashboard views.
Instant report generation, credit limit alerts for sales and invoices, separate OCR settings for invoice creation, and an auto-complete feature for forms in the accounting module.
Improved image handling and global attribute changes for mailing lists in email marketing.
A default auto-signature option and a refuse-to-sign option in HR modules.
Options to divide and merge manufacturing orders, track the status of manufacturing orders, and more in the MRP module.
Dark mode in Odoo 17.
Now that the Odoo 17 announcement is official, let’s look at what’s new in Odoo 17!
What is Odoo ERP 17?
Odoo 17 is the latest version of one of the world’s leading open-source enterprise ERPs. This version has come up with significant improvements explained here in this blog. Also, this new version aims to introduce features that enhance time-saving, efficiency, and productivity for users across various organisations.
Odoo 17, released at the Odoo Experience 2023, brought notable improvements to the user interface and added new functionalities with enhancements in performance, accessibility, data analysis, and management, further expanding its reach in the market.
Measures in SQL (SIGMOD 2024, Santiago, Chile)Julian Hyde
SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries.
SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL.
To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context.
A talk at SIGMOD, June 9–15, 2024, Santiago, Chile
Authors: Julian Hyde (Google) and John Fremlin (Google)
https://doi.org/10.1145/3626246.3653374
WWDC 2024 Keynote Review: For CocoaCoders AustinPatrick Weigel
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
3. hi, I’m joe!
• i think these things are cool:
• computer programs
• reproducible builds / infrastructure
• automation
• configuration management
• tahdig*
* an rice food
4. packagecloud.io
• I work on packagecloud.io
• packagecloud makes it easy to upload,
download, store, and delete software packages
• you should use it, it’s cool.
• it’s a perfect companion to Chef Delivery
6. “packagecloud:enterprise has solved the problem
of distributing public and private package
repositories. We’re extremely satisfied with the
support and we trust it with all the GitLab Omnibus
package downloads of more than 1TB per week."
Sytse Sijbrandij, CEO & Founder
9. Why?
• Central to maintaining, building, and testing
infrastructure.
• Packages are a primitive in Chef.
• Understanding where packages come from, and how
to store them properly is a requirement for
infrastructure of any size.
• Packages and packaging are much trickier than they
seem!
10. Overview
• what is a package?
• what is a package manager?
• ./configure && make && make install pattern
• open source tools for package repositories
• HOWTO manage repos in your infra with Chef
11. What is a package?
Beck Gusler, https://flic.kr/p/4A15jm
12. What is a package?
• A package generally consists of:
• metadata (version, architecture, deps, etc)
• files to be written to the filesystem (/usr/sbin/
nginx, etc)
14. Common package types
• RPM packages
• Used on CentOS, RHEL, Scientific Linux, Fedora, …
• files typically have the “.rpm” file extension
• can be inspected, installed, and removed with rpm
• are actually a:
• header structure (binary data)
• CPIO archive
17. Common package types
• Deb packages:
• Used on Ubuntu, Debian, Knoppix, …
• files typically have the “.deb” file extension
• can be inspected, installed, and removed with
dpkg
18. Common package types
• Deb packages:
• are actually an AR archive with:
• version file: the debian format version
• data.tar.gz: the actual files to write to the filesystem
• control.tar.gz: the package metadata
• Can be GPG signed, but signatures are never checked!
20. Common package types
• There are lots more! (ruby gems, npm, java,
python, …)
• Some packaging systems also have source
packages.
21. What is a source package?
• A source package consists of:
• metadata (version, architecture(s), build deps,
etc).
• source files (C source, C++ source, py scripts,
etc).
• Allows you to rebuild a binary package easily.
22. Install packages with chef
Use the ‘package’ resource to install packages:
package "zlib1g" do
action :install
end
23. Install packages with chef
Specify the version you want by setting ‘version’:
package "zlib1g" do
version "1:1.2.8-1"
action :install
end
24. Summary
• Packages are a collection of files with metadata.
• The metadata usually has info like:
• architecture
• version
• dependency info
• and more.
• Installation is easy if you don’t have dependencies.
26. Dependencies
• Installing 1 package is as easy as:
• dpkg -i filename.deb
• rpm -ivh filename.rpm
• Of course, you should use chef instead :D
• But what if your program needs other programs?
• For example: nginx depends on libssl, zlib, …
29. Package manager
• A package manager is a collection of software
that allows you to:
• install, upgrade, remove packages
• query package info from local system or repos
• Some tools include more advanced features like
mirroring or more advanced caching features.
31. • yum (Yellowdog Updater, Modified)
• Common on RHEL, CentOS, Fedora, …
• Used for installing, removing, configuring, and
querying RPM packages and dependencies.
Common package managers
33. Common package managers
• APT (Advanced Package Tool)
• Common on Debian, Ubuntu, KNOPPIX, …
• Used for installing, removing, configuring, and
querying Debian packages and dependencies.
34. Install packages with chef
• When you install packages with chef, chef will
automatically detect which package manager to
use.
• You won’t need to worry about which command
to run, or what options to pass; chef will take
care of that for you!
35. Summary
• package managers help you install software and
associated dependencies
• easily remove, upgrade, and query packages
• Chef will automatically detect the system’s
package manager when you install a package.
37. A problem
• You run Ubuntu 10.04 LTS
• You want to install redis
• Ubuntu 10.04 comes with redis-server 1.2.0-1
• That’s too old! You need 2.8.19!
• So, now what?
38. Common (not great) solution
• A common solution to this sort of problem is
building redis (or ruby, or …) from source in your
chef cookbook
• Like this….
39. execute ‘compile redis' do
cwd ‘/tmp/redis’
command ‘make clean && make’
end
!
execute ‘install redis' do
cwd ‘/tmp/redis’
command ‘make install’
end
Common (not great) solution
40. Why?
• It’s easy!
• ./configure && make && make install
• It works!
• I’m using chef so it’s reproducible!
41. But…
• What happens if you need to:
• completely remove Redis?
• install a security update?
• install a new version?
• install the same exact Redis on 200 machines?
42. The not-so great side
• Not all Makefiles have uninstall targets, so you
have to remove files manually
• Leaving artifacts on the filesystem can cause
really, really hard to debug problems later
• If the build process changes version to version,
it can be painful to rollback
43. The not-so great side
• Rebuilding the same source does not necessarily
get you the same byte-for-byte binary
• If the binaries aren’t identical, you can end up
with bugs in some of the compiled binaries but
not others
• Painful to recreate source builds inside of chef
• Makes writing tests for cookbooks painful
44. Make a package
• Install the same binary on every machine
• When the package is removed, all installed files are
removed
• Versioning of build process built in (with most tools)
• Keep your chef cookbooks about config management
• Your build steps are “factored out” into the package
45. Your new chef recipe
package "redis" do
action :install
end
46. Your package
• Your build steps get encapsulated in the package
itself
• Makes iterating on the build more straight forward
• Don’t need to run (potentially) a huge number of
cookbooks every time you do a build
53. Tradeoffs
• Once you learn how to make packages you can
build reproducible infrastructure much more
easily
• You can use your prod environment in dev and
test
• You can more easily build tests for your
infrastructure with kitchen.ci
56. Package repositories
• Major linux distributions keep repositories of
packages for users:
• EPEL
• Ubuntu / Debian official repositories
• You can store a package and its dependencies to
make it easy to install them all on your infrastructure
58. Package repositories
• createrepo: creates yum repositories
• reprepro: creates apt repositories
• Many other free tools available!
• Read the documentation carefully. Lots of tricky
options.
• I’ll show some examples to get you started!
61. GPG is important
• Using GPG to sign the generated repository
guarantees that you generated the repository.
• This is important.
• This means that no one else modified, removed, or
inserted a package other than you.
• GPG signing the repository is not a very well known
security measure, but it is incredibly important!
• This is NOT the same as using rpmsign/rpm --sign.
62. Secure YUM repos
• Sign repository metadata with GPG
• Sign packages with GPG (use rpmsign)
• Serve repositories over SSL
• Enable all the right options for SSL verification,
repository GPG checking, AND package GPG
checking.
63. Wouldn’t it be cool to do all
that with Chef instead?
Good news: you can!
64. createrepo via chef
Chef can create YUM repositories for you!
$ knife cookbook site install yumrepo_server
65. yumrepo_server 'creates my yum repo' do
action :create
dir 'relative/yum/repo/path'
remote_source "http://upstream.com/path"
packages %w(pkg1.rpm pkg2.rpm pkg3.rpm)
end
createrepo via chef
66. You still need to GPG sign
the repository yourself :(
execute ‘gpg sign yum metadata' do
cwd ‘relative/yum/repo/path/repodata’
command ‘gpg --detach-sign —armor repomd.xml’
end
68. Add YUM repos with chef
most people never turn on repo_gpgcheck or
sslverify, or set the ssl certificate path, but you
should!!
yum_repository ‘my_repo' do
description “packagecloud.io is better than this”
baseurl “https://myurl.com/repo“
gpgkey ‘http://myurl.com/gpg.pub.key'
gpgcheck true
repo_gpgcheck true
sslverify true
sslcacert “/etc/pki/tls/certs/ca-bundle.crt”
action :create
end
69. But that’s not all!
• You MUST have the ‘pygpgme’ package
installed on the system that will verify the
signatures.
• Without pygpgme, yum will not be able to verify
signatures!
• Some versions of CentOS / RHEL do not
automatically install pygpgme with yum!!
70. Make sure to install pygpgme
package "pygpgme" do
action :install
end
74. reprepro
• You can add more sections if you need more code
names (lucid, trusty, etc).
• SignWith specifies which GPG key to use for signing
repository metadata
• You can get your gpg key ID by looking at the output
of gpg —list-keys
• This is not the same as using debsigs/debsign !!!
80. Add APT repos with chef
apt_repository 'repo' do
uri ‘http://repo.com/ubuntu/'
arch 'amd64'
distribution 'precise'
components ['main']
key ‘http://repo.com/ubuntu/archive.key'
end
$ knife cookbook site install apt
81. But that’s not all!
• You MUST have the ‘apt-transport-https’ package
installed on the system if your repository is served
over HTTPS!
• Without apt-transport-https, you can’t install
packages over HTTPS.
• You definitely want this.
82. Make sure to install apt-transport-https
package “apt-transport-https“ do
action :install
end
84. Success
• You can now use kitchen.ci to test your
infrastructure.
• Determine if the packages you need are actually
installed after your cookbooks have run.
• Determine if the repositories you added are
actually added after your cookbooks have run.
• Don’t need to wait forever for Ruby, redis, et al to
build during a test run.
85. BEST OF ALL !!!!
• You can now run Chef on your development VM
using the same cookbooks you use in
production
• The cookbooks are applied and you are running
the same exact binaries you run in production
• Won’t catch ALL production bugs, but getting
closer to production during development is
super useful
86. Summary
• Creating package repositories can be tricky. Make
sure to GPG sign repository metadata.
• 99% of package repositories get this wrong.
• Carefully read the documentation of createrepo and
reprepro.
• Make sure to install necessary libraries for verifying
signatures and accessing repositories via HTTPS.
• Always serve up your repositories over HTTPS.
87. Use chef to automate your
infrastructure.
Use packagecloud.io to
deliver software.