Advanced software development processes and deployment automation are necessary for any large software development team. However, even small teams and lone developers can benefit from these techniques.
Let me show you how with a few tools that are powerful, simple to understand, and free to use. I will cover solutions for CLI & GUI package management, machine imaging, virtualization, and configuration management. You will see examples that can create a unified software deployment approach for both local development environments & production cloud instances.
Related blog post: http://www.jbarker.com/blog/2014/ebay-berlin-devops-cloud
Building a Docker Swarm cluster on ARM by Dieter Reuter and Stefan Scherer Docker, Inc.
In this training you'll learn how to build a physical Docker Swarm cluster with Raspberry Pi's. We'll guide you through the setup process and you’ll learn how to use Docker and Docker Swarm to complete the cluster. You'll learn how to build and deploy a distributed application and ship it as Docker containers to your cluster. In the end you have built a portable datacenter which can be used for testing and live demos as well.
- building a hardware cluster with Raspberry Pi’s
- install and set up HypriotOS, a Debian-based Linux system
- installing Docker and Docker Swarm to connect all cluster nodes
- build a distributed application, the Docker voting app
- ship the app with Docker Compose to your cluster
- let’s run and test your app
In this training you'll learn how to build a physical Docker Swarm cluster with Raspberry Pi's. We'll guide you through the setup process and you’ll learn how to use Docker and Docker Swarm to complete the cluster. You'll learn how to build and deploy a distributed application and ship it as Docker containers to your cluster. In the end you have built a portable datacenter which can be used for testing and live demos as well.
- building a hardware cluster with Raspberry Pi’s
- install and set up HypriotOS, a Debian-based Linux system
- installing Docker and Docker Swarm to connect all cluster nodes
- build a distributed application, the Docker voting app
- ship the app with Docker Compose to your cluster
- let’s run and test your app
Sydney based cloud consultancy Cloudten's Richard Tomkinson shows how masterless Puppet can be used in concert with AWS's services including Lambda to automate server builds and manage code deployments
Use docker to ease data science software installation for learning purposes in your own laptop or for deploying into production using AWS services.
This was in a data science event organized by Celerative at Galvanize, San Francisco 28/11/2017
The use of serverless architecture has very quickly become an accepted approach for organizations deploying cloud applications, with a plethora of choices available for deployment.
Even traditionally conservative organizations are making partial use of some serverless technologies.
Most of the discussion goes to Functions as a Service (e.g., AWS Lambda, Google Cloud Functions, Azure Functions) while the appropriate patterns for use are still emerging.
Building a Docker Swarm cluster on ARM by Dieter Reuter and Stefan Scherer Docker, Inc.
In this training you'll learn how to build a physical Docker Swarm cluster with Raspberry Pi's. We'll guide you through the setup process and you’ll learn how to use Docker and Docker Swarm to complete the cluster. You'll learn how to build and deploy a distributed application and ship it as Docker containers to your cluster. In the end you have built a portable datacenter which can be used for testing and live demos as well.
- building a hardware cluster with Raspberry Pi’s
- install and set up HypriotOS, a Debian-based Linux system
- installing Docker and Docker Swarm to connect all cluster nodes
- build a distributed application, the Docker voting app
- ship the app with Docker Compose to your cluster
- let’s run and test your app
In this training you'll learn how to build a physical Docker Swarm cluster with Raspberry Pi's. We'll guide you through the setup process and you’ll learn how to use Docker and Docker Swarm to complete the cluster. You'll learn how to build and deploy a distributed application and ship it as Docker containers to your cluster. In the end you have built a portable datacenter which can be used for testing and live demos as well.
- building a hardware cluster with Raspberry Pi’s
- install and set up HypriotOS, a Debian-based Linux system
- installing Docker and Docker Swarm to connect all cluster nodes
- build a distributed application, the Docker voting app
- ship the app with Docker Compose to your cluster
- let’s run and test your app
Sydney based cloud consultancy Cloudten's Richard Tomkinson shows how masterless Puppet can be used in concert with AWS's services including Lambda to automate server builds and manage code deployments
Use docker to ease data science software installation for learning purposes in your own laptop or for deploying into production using AWS services.
This was in a data science event organized by Celerative at Galvanize, San Francisco 28/11/2017
The use of serverless architecture has very quickly become an accepted approach for organizations deploying cloud applications, with a plethora of choices available for deployment.
Even traditionally conservative organizations are making partial use of some serverless technologies.
Most of the discussion goes to Functions as a Service (e.g., AWS Lambda, Google Cloud Functions, Azure Functions) while the appropriate patterns for use are still emerging.
Arash Kaffamanesh presents a lightning talk during Build A Cloud Day on Platform as a Service. While most open source solution in the Cloud as focused on the IaaS layer, the Platform layer is starting to see some increase development and testing. RedHat OpenShift is seeing great traction but software like Appscale, Stackato and Cloud Fundry are also being used. In this presentation Arash presented several protoypes of deploying a Platform as a Service solution on top of a CloudStack based cloud. Not shon in the slides is the fact that Stratos from WS02 is being voted as an Apache project in the incubator. the PaaS area is heating up.
How to Improve Your Image Builds Using Advance Docker BuildDocker, Inc.
Nicholas Dille, Haufe-Lexware + Docker Captain -
Docker continues to be the standard tool for building container images. For more than a year Docker ships with BuildKit as an alternative image builder, providing advanced features for secret and cache management. These features help to make image builds faster and more secure. In this session, Docker Captain Nicholas Dille will teach you how to use Buildkit features to your advantage.
Are you still using FTP to deploy your code? Are you still manually performing the same steps of deploying a feature, again and again? How many hours have you spent on ssh-ing into the server, pulling the repo, migrating the database, reloading the web server and so on, for each deployment?? Ever wondered if there is a process as simple as a single click to perform all these steps for you?
Automated Deployment does exactly these things for you. It takes the burden of remembering all the steps required in each deployment process and execute it smoothly.
Meet up presentation on Continuous Integration with Docker on Amazon Web Services (AWS). The presentation covers benefits of Docker on AWS along with advanced Docker patterns and lessons learned.
Containerize All the (Multi-Platform) Things! by Phil EstesDocker, Inc.
There are literally tens of thousands of images available today in the public DockerHub repository. These images cover almost every possible distro and common open source tool, server, or application that exists today. But there has been one drawback--all of these images assume you are on the most commonly used platform: 64-bit Intel Linux. In late 2015 and 2016 the Docker development community and Docker distribution team have enabled a new registry image specification that supports packaging multiple architecture and OS layer images within the same repository name and tag. This allows a common image, say "mysql:latest", to contain references to images for all supported OS and architectures that the packager determines to make available. Now a `docker run` or `docker pull` of "mysql:latest" will work seamlessly across all supported architectures. In this talk we will demonstrate the packaging and running of a multi-architecture containerized application on several different supported Docker platforms like ARM, POWER, and System z.
Ansible, Vagrant and Packer are tools that allow you to model production systems on your laptop. This talk was given by Bas Meijer at GOTO Amsterdam 2015. Source code at https://github.com/bbaassssiiee/vagransible
Arash Kaffamanesh presents a lightning talk during Build A Cloud Day on Platform as a Service. While most open source solution in the Cloud as focused on the IaaS layer, the Platform layer is starting to see some increase development and testing. RedHat OpenShift is seeing great traction but software like Appscale, Stackato and Cloud Fundry are also being used. In this presentation Arash presented several protoypes of deploying a Platform as a Service solution on top of a CloudStack based cloud. Not shon in the slides is the fact that Stratos from WS02 is being voted as an Apache project in the incubator. the PaaS area is heating up.
How to Improve Your Image Builds Using Advance Docker BuildDocker, Inc.
Nicholas Dille, Haufe-Lexware + Docker Captain -
Docker continues to be the standard tool for building container images. For more than a year Docker ships with BuildKit as an alternative image builder, providing advanced features for secret and cache management. These features help to make image builds faster and more secure. In this session, Docker Captain Nicholas Dille will teach you how to use Buildkit features to your advantage.
Are you still using FTP to deploy your code? Are you still manually performing the same steps of deploying a feature, again and again? How many hours have you spent on ssh-ing into the server, pulling the repo, migrating the database, reloading the web server and so on, for each deployment?? Ever wondered if there is a process as simple as a single click to perform all these steps for you?
Automated Deployment does exactly these things for you. It takes the burden of remembering all the steps required in each deployment process and execute it smoothly.
Meet up presentation on Continuous Integration with Docker on Amazon Web Services (AWS). The presentation covers benefits of Docker on AWS along with advanced Docker patterns and lessons learned.
Containerize All the (Multi-Platform) Things! by Phil EstesDocker, Inc.
There are literally tens of thousands of images available today in the public DockerHub repository. These images cover almost every possible distro and common open source tool, server, or application that exists today. But there has been one drawback--all of these images assume you are on the most commonly used platform: 64-bit Intel Linux. In late 2015 and 2016 the Docker development community and Docker distribution team have enabled a new registry image specification that supports packaging multiple architecture and OS layer images within the same repository name and tag. This allows a common image, say "mysql:latest", to contain references to images for all supported OS and architectures that the packager determines to make available. Now a `docker run` or `docker pull` of "mysql:latest" will work seamlessly across all supported architectures. In this talk we will demonstrate the packaging and running of a multi-architecture containerized application on several different supported Docker platforms like ARM, POWER, and System z.
Ansible, Vagrant and Packer are tools that allow you to model production systems on your laptop. This talk was given by Bas Meijer at GOTO Amsterdam 2015. Source code at https://github.com/bbaassssiiee/vagransible
An overview on docker and container technology behind it. Lastly, we discuss few tools that might come handy when dealing with large number of containers management.
Do you usually pay to develop your Node.js app in a cluster from a cloud provider? You won't do that anymore after you see how you can develop locally in a cloud environment. You will see how using minikube can help you scale your app along with how you plan on updating apps in real time.
Automated Infrastructure and Application ManagementClark Everetts
Managing application infrastructure is an error prone, tedious, and often manual process leading to late hours spent troubleshooting self-inflicted oversights. Clark will introduce an open source Chef cookbook automating many steps, which utilizes a server side SDK to painlessly deploy PHP applications, and also show how the process can be managed leveraging Zend Server. Attendees will walk away with a complete toolset to implement quickly in their own projects.
What is this Docker and Microservice thing that everyone is talking about? A primer to Docker and Microservice and how the two concepts complement each other.
Today’s cutting edge companies have software release cycles measured in days instead of months. This agility is enabled by the DevOps practice of continuous integration and delivery, which automates building, testing, and deploying all code changes. This automation helps you catch bugs sooner and accelerates developer productivity. In this session, we’ll share the processes followed by Amazon engineers and discuss how you can bring them to your company by using a set of application lifecycle management tools from AWS: the newly announced AWS CodeBuild service, AWS CodePipeline, and AWS CodeDeploy.
DCEU 18: Building Your Development PipelineDocker, Inc.
Oliver Pomeroy - Solution Engineer, Docker
Laura Frank Tacho - Director of Engineering, CloudBees
Enterprises often want to provide automation and standardisation on top of their container platform, using a pipeline to build and deploy their containerized applications. However this opens up new challenges… Do I have to build a new CI/CD Stack? Can I build my CI/CD pipeline with Kubernetes orchestration? What should my build agents look like? How do I integrate my pipeline into my enterprise container registry? In this session full of examples and “how-to”s, Olly and Laura will guide you through common situations and decisions related to your pipelines. We’ll cover building minimal images, scanning and signing images, and give examples on how to enforce compliance standards and best practices across your teams.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Łukasz Chruściel
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
GraphSummit Paris - The art of the possible with Graph TechnologyNeo4j
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeAftab Hussain
Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.
- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia
Zoom is a comprehensive platform designed to connect individuals and teams efficiently. With its user-friendly interface and powerful features, Zoom has become a go-to solution for virtual communication and collaboration. It offers a range of tools, including virtual meetings, team chat, VoIP phone systems, online whiteboards, and AI companions, to streamline workflows and enhance productivity.
2. Why automation?
Automation in the Small: Code to Cloud | Jay Barker 2
Go Faster,
Save Time
flic.kr/p/nYT2s4
Complexity
& Scale
flic.kr/p/28i54x
Relax
flic.kr/p/3tj1VR
Be Flexible
flic.kr/p/4wNdu5
4. About your presenter
• Jay Barker
• 10+ years as software engineer & manager
• Manager, Software Development
@ mobile.de, an ebay company
• Further details & contact information
www.jbarker.com
Automation in the Small: Code to Cloud | Jay Barker 4
5. Automation in the Small:
Code to Cloud
• Automation techniques for individuals and tiny teams
• A set of integrated tools, just one example of many ways to
automate local development & cloud deployment
• Related code & documentation are on Github
– github.com/jbarker/dev-automation-small
– Further details in README.md file
– Some ‘demo’ steps make use of this repository
• This presentation is on SlideShare
– slideshare.net/jbrkr/automation-smallcodeclouddevops
Automation in the Small: Code to Cloud | Jay Barker 5
6. The Tools
Automation in the Small: Code to Cloud | Jay Barker 6
Homebrew
packages
brew.sh
VirtualBox
virtualization
virtualbox.org
Packer
imaging
packer.io
Ansible
provisioning
ansible.com
8. Use a Package Manager
• Take control of software updates with a package manager
• Linux has built-in options, e.g. apt, yum
• For Windows, consider Chocolatey
• For Max OS X, consider Homebrew with Cask
– Some ‘demo’ steps assume both are installed
• Demo: install Homebrew & Cask
$
ruby
-‐e
"$(curl
-‐fsSL
https://
raw.github.com/Homebrew/homebrew/go/install)"
$
brew
config
$
brew
install
caskroom/cask/brew-‐cask
brew.sh / caskroom.io
Automation in the Small: Code to Cloud | Jay Barker 8
9. Automated package
installation
• Demo: install the latest stable git from the community to
override the old version bundled by Apple
$
brew
install
git
$
git
-‐-‐version
>
git
version
2.0.4
• Demo: install & run a GUI application, Github for Mac
$
brew
cask
install
github
$
github
-‐-‐help
$
github
git-scm.com / mac.github.com
Automation in the Small: Code to Cloud | Jay Barker 9
11. Automation in the Small: Code to Cloud | Jay Barker 11
Use a local Virtualization app
• Run virtual machines (VMs) on your host machine
• Install a virtualization app running on your host machine
• One free, cross-platform option is VirtualBox
• Other options include: Parallels Desktop, VMWare Fusion
• Demo: install VirtualBox for Mac
$
brew
cask
install
virtualbox
$
virtualbox
virtualbox.org
12. Automation in the Small: Code to Cloud | Jay Barker 12
Use local VMs for testing
• Example: testing web applications with Internet Explorer on
Windows
• Use the ievms project for automated VMs installation from
Microsoft’s official source, modern.ie
• Demo: clone the ievms project, then download & install
various versions of Windows with Internet Explorer
Warning: this may take awhile due to large downloads!
$
git
clone
https://github.com/xdissent/
ievms.git
$
cd
ievms
$
./ievms.sh
github.com/xdissent/ievms
14. Automation in the Small: Code to Cloud | Jay Barker 14
Use a Provisioning tool
• Let’s create a new custom virtual machine
• We can use Packer to provision this virtual machine
• Demo: install Packer using Homebrew
$
brew
tap
homebrew/binary
$
brew
install
packer
packer.io
15. Automation in the Small: Code to Cloud | Jay Barker 15
Provision a new local VM
• For the operating system, let’s use Debian 7.x x86_64
Network installation ISO
• Tip: Be sure to configure SSH access to your VM!
– Recommended is access by public/private key pair
– Your public key may already be available in a
convenient location
• e.g. github.com/example-username.keys
• Demo: provision the VM with Packer
Warning: this may take awhile due to large downloads!
$
cd
packer
$
packer
build
-‐only=virtualbox-‐iso
-‐force
debian-‐7.x-‐amd64.json
debian.org
16. Automation in the Small: Code to Cloud | Jay Barker 16
Import & start the new VM
• Import your virtual machine into VirtualBox
• Start your virtual machine in headless mode (no UI!)
• Demo: import the new VM created by Packer
$
VBoxManage
import
packer-‐debian-‐7.x-‐amd64-‐
virtualbox/packer-‐debian-‐7.x-‐amd64.ovf
$
VBoxManage
list
vms
• Demo: launch the new VM with VirtualBox
$
VBoxHeadless
-‐s
"packer-‐debian-‐7.x-‐amd64"
17. Automation in the Small: Code to Cloud | Jay Barker 17
Use a Configuration
Management tool
• Configure your VMs the same way, every time
• Use a configuration management tool to define & maintain
machine configurations
• Let’s use Ansibile to configure the local virtual machine
• Other options include: Chef, Puppet, Salt Stack
• Demo: install Ansible using Homebrew
$
brew
install
ansible
ansible.com
18. Automation in the Small: Code to Cloud | Jay Barker 18
Configure your new VM
• Ansible will setup your machine using configuration that
you define in a YAML file
• Ansible requires SSH access to your VM
– You should configure a public/private key pair for access
when creating your VM
• Demo: ping your running VM with Ansible
$
cd
ansible
$
ansible
-‐i
inventory.ini
all
-‐m
ping
• Demo: configure your running VM with Ansible
$
ansible
-‐i
inventory.ini
all
-‐m
setup
-‐vvvv
20. Choose a Cloud provider
Automation in the Small: Code to Cloud | Jay Barker 20
• Many popular choices for cloud providers. Choose a cloud
provider that best fits your needs!
• Possible considerations
– common operating system offering, like Debian
– integration with preferred tools, like Packer
• Some options with both Debian & Packer integration:
– DigitalOcean
• digitalocean.com/community/tags/debian
– Amazon EC2
• wiki.debian.org/Cloud/AmazonEC2Image
– Google Compute Engine
• wiki.debian.org/Cloud/GoogleComputeEngineImage
– OpenStack
• wiki.debian.org/OpenStack
21. Configure your Cloud server
Automation in the Small: Code to Cloud | Jay Barker 21
• Use the same tools for cloud development and local
development
• Provision a custom machine with Packer
Many builders supported out-of-the-box
• Configure your machine with Ansible
Works great via standard SSH
• Test in a Virtual Machine locally with VirtualBox
Host in all popular desktop operating systems
• Test in cloud instances with your preferred provider
• Deploy to indentical cloud instances for production use
?
23. In summary + Q&A
• Automation: Go Faster, Save Time / Be Flexible /
Complexity & Scale / Relax
• Package management for software installation & upgrades
– Homebrew & Cask on Mac OS X
– Similar options on Linux or Windows
• Virtualization for local development & testing
– VirtualBox for running custom & prepared machines
• Provisioning & configuration management
– Packer & Ansible with Debian Linux
• Configure, test, & deploy to cloud instances
– Your preferred provider with standard tools
flic.kr/p/iVLZt
Automation in the Small: Code to Cloud | Jay Barker 23