This presentation discusses using Terraform and Consul-Template to implement "Immutablish" infrastructure at Trulia. Immutablish infrastructure allows some changes, unlike immutable infrastructure, through dynamically generated configuration files. Terraform is used to define Consul keys for template variables and generate unique scripts for each host. Consul-Template monitors Consul for changes and reloads services when configurations are updated. While more flexible than immutable setups, Immutablish infrastructure adds operational complexity and may not work for all stateful services.
Introductory Overview to Managing AWS with TerraformMichael Heyns
From the AWS NZ Auckland Community Meetup - May 4th 2017
https://www.meetup.com/AWS_NZ/events/236169428/
We get a first look at Hashicorp's Terraform and how to use it for Infrastructure as Code with Amazon Web Services.
We'll also share how it fits in with our current CI/CD workflow at the Invenco cloud services team
Sample code available at https://github.com/beanaroo/aws_nz_meetup-terraform_intro
This talk is a very quick intro to Docker, Terraform, and Amazon's EC2 Container Service (ECS). In just 15 minutes, you'll see how to take two apps (a Rails frontend and a Sinatra backend), package them as Docker containers, run them using Amazon ECS, and to define all of the infrastructure-as-code using Terraform.
Terraform, is no doubt very flexible and powerful. The question is, how do we write Terraform code and construct our infrastructure in a reproducible fashion that makes sense? How can we keep code DRY, segment state, and reduce the risk of making changes to our service/stack/infrastructure?
HashiCorp’s infrastructure management tool, Terraform, is no doubt very flexible and powerful. The question is, how do we write Terraform code and construct our infrastructure in a reproducible fashion that makes sense? How can we keep code DRY, segment state, and reduce the risk of making changes to our service/stack/infrastructure?
This talk describes a design pattern to help answer the previous questions. The talk is divided into two sections, with the first section describing and defining the design pattern with a Deployment Example. The second part uses a multi-repository GitHub organization to create a Real World Example of the design pattern.
Introductory Overview to Managing AWS with TerraformMichael Heyns
From the AWS NZ Auckland Community Meetup - May 4th 2017
https://www.meetup.com/AWS_NZ/events/236169428/
We get a first look at Hashicorp's Terraform and how to use it for Infrastructure as Code with Amazon Web Services.
We'll also share how it fits in with our current CI/CD workflow at the Invenco cloud services team
Sample code available at https://github.com/beanaroo/aws_nz_meetup-terraform_intro
This talk is a very quick intro to Docker, Terraform, and Amazon's EC2 Container Service (ECS). In just 15 minutes, you'll see how to take two apps (a Rails frontend and a Sinatra backend), package them as Docker containers, run them using Amazon ECS, and to define all of the infrastructure-as-code using Terraform.
Terraform, is no doubt very flexible and powerful. The question is, how do we write Terraform code and construct our infrastructure in a reproducible fashion that makes sense? How can we keep code DRY, segment state, and reduce the risk of making changes to our service/stack/infrastructure?
HashiCorp’s infrastructure management tool, Terraform, is no doubt very flexible and powerful. The question is, how do we write Terraform code and construct our infrastructure in a reproducible fashion that makes sense? How can we keep code DRY, segment state, and reduce the risk of making changes to our service/stack/infrastructure?
This talk describes a design pattern to help answer the previous questions. The talk is divided into two sections, with the first section describing and defining the design pattern with a Deployment Example. The second part uses a multi-repository GitHub organization to create a Real World Example of the design pattern.
A Hands-on Introduction on Terraform Best Concepts and Best Practices Nebulaworks
At our OC DevOps Meetup, we invited Rami Al-Ghami, a Sr. Software engineer at Workday to deliver a presentation on a Hands-On Terraform Best Concepts and Best Practices.
The software lifecycle does not end when the developer packages their code and makes it ready for deployment. The delivery of this code is an integral part of shipping a product. Infrastructure orchestration and resource configuration should follow a similar lifecycle (and process) to that of the software delivered on it. In this talk, Rami will discuss how to use Terraform to automate your infrastructure and software delivery.
My talk at FullStackFest, 4.9.2017. Become more familiar with managing infrastructure using Terraform, Packer and deployment pipeline. Code repository - https://github.com/antonbabenko/terraform-deployment-pipeline-talk
https://www.youtube.com/watch?v=IeweKUdHJc4
My presentation from Hashiconf 2017, discussing our use of Terraform, and our techniques
to help make it safe and accessible.
Slides form Config Management Camp, looking at how you can take a collaborative GitFlow approach to Terraform using Remote State, Modules and Dynamically Generated Credentials using Vault
Terraform: Configuration Management for Cloud ServicesMartin Schütte
Hashicorp's Terraform provides a declarative notation (like Puppet) to describe various cloud resources. It is an open-source tool, provider-independent, and thus able to combine resources from multiple cloud platforms and to be extended through plugins.
The talk demonstrates how to describe a small web application with Terraform, showing how easily all related components can be started, updated, and stopped. It also shows how to organise larger projects using modules and gives an introduction to writing plugins for one’s own services.
A presentation from Hashiconf 2016.
Terraform is a wonderful tool for describing infrastructure as code. It’s fast, flexible, automatically resolves dependencies, and is rapidly improving.
But in some ways, Terraform is flexible like AWS is flexible. You can do pretty much anything, but it’s also easy to shoot yourself in the foot if you aren’t careful.
In the past year, we’ve started managing thousands of resources with Terraform, allowing a lot more of the dev team to change the underlying infrastructure. During that time, we’ve learned a lot about how to set up our terraform modules so that they are easy to manage and reuse.
This talk will cover how we manage tfstate, separate environments, specific module definitions, and how use terraform to boot new services in production. I’ll also discuss the challenges we’re currently facing, and how we plan to attack them going forward.
Listen up, developers. You are not special. Your infrastructure is not a beautiful and unique snowflake. You have the same tech debt as everyone else. This is a talk about a better way to build and manage infrastructure: Terraform Modules. It goes over how to build infrastructure as code, package that code into reusable modules, design clean and flexible APIs for those modules, write automated tests for the modules, and combine multiple modules into an end-to-end techs tack in minutes.
You can find the video here: https://www.youtube.com/watch?v=LVgP63BkhKQ
A comprehensive walkthrough of how to manage infrastructure-as-code using Terraform. This presentation includes an introduction to Terraform, a discussion of how to manage Terraform state, how to use Terraform modules, an overview of best practices (e.g. isolation, versioning, loops, if-statements), and a list of gotchas to look out for.
For a written and more in-depth version of this presentation, check out the "Comprehensive Guide to Terraform" blog post series: https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca
This beginning terraform workshop will teach you how to safely create and provision Infrastructure as Code (IAC) using Hashicorp Terraform in an AWS environment. In this class you will learn how to setup and install terraform. You will also be given a walkthrough of Terraform fundamentals. You will be lead through the process of deploying a single server, deploying a cluster and setting up a load balancer. You will also learn how to author Terraform Modules, work with Route53 and how to manage DNS.
Requirements. You will need to have an AWS account set up already with Terraform v0.9.3 installed. You will also need to have git install to download the workshop material.
You can find more informaiton on how to install terraform here: https://www.terraform.io/intro/getting-started/install.html. You can sign up for an AWS account here: https://aws.amazon.com/account/
https://github.com/jasonvance/terraform-introduction
Slides for my talk at the London Perl Workshop in Nov 2013, featuring the Devel::SizeMe perl module.
See also the screencast at https://archive.org/details/Perl-Memory-Profiling-LPW2013
A Hands-on Introduction on Terraform Best Concepts and Best Practices Nebulaworks
At our OC DevOps Meetup, we invited Rami Al-Ghami, a Sr. Software engineer at Workday to deliver a presentation on a Hands-On Terraform Best Concepts and Best Practices.
The software lifecycle does not end when the developer packages their code and makes it ready for deployment. The delivery of this code is an integral part of shipping a product. Infrastructure orchestration and resource configuration should follow a similar lifecycle (and process) to that of the software delivered on it. In this talk, Rami will discuss how to use Terraform to automate your infrastructure and software delivery.
My talk at FullStackFest, 4.9.2017. Become more familiar with managing infrastructure using Terraform, Packer and deployment pipeline. Code repository - https://github.com/antonbabenko/terraform-deployment-pipeline-talk
https://www.youtube.com/watch?v=IeweKUdHJc4
My presentation from Hashiconf 2017, discussing our use of Terraform, and our techniques
to help make it safe and accessible.
Slides form Config Management Camp, looking at how you can take a collaborative GitFlow approach to Terraform using Remote State, Modules and Dynamically Generated Credentials using Vault
Terraform: Configuration Management for Cloud ServicesMartin Schütte
Hashicorp's Terraform provides a declarative notation (like Puppet) to describe various cloud resources. It is an open-source tool, provider-independent, and thus able to combine resources from multiple cloud platforms and to be extended through plugins.
The talk demonstrates how to describe a small web application with Terraform, showing how easily all related components can be started, updated, and stopped. It also shows how to organise larger projects using modules and gives an introduction to writing plugins for one’s own services.
A presentation from Hashiconf 2016.
Terraform is a wonderful tool for describing infrastructure as code. It’s fast, flexible, automatically resolves dependencies, and is rapidly improving.
But in some ways, Terraform is flexible like AWS is flexible. You can do pretty much anything, but it’s also easy to shoot yourself in the foot if you aren’t careful.
In the past year, we’ve started managing thousands of resources with Terraform, allowing a lot more of the dev team to change the underlying infrastructure. During that time, we’ve learned a lot about how to set up our terraform modules so that they are easy to manage and reuse.
This talk will cover how we manage tfstate, separate environments, specific module definitions, and how use terraform to boot new services in production. I’ll also discuss the challenges we’re currently facing, and how we plan to attack them going forward.
Listen up, developers. You are not special. Your infrastructure is not a beautiful and unique snowflake. You have the same tech debt as everyone else. This is a talk about a better way to build and manage infrastructure: Terraform Modules. It goes over how to build infrastructure as code, package that code into reusable modules, design clean and flexible APIs for those modules, write automated tests for the modules, and combine multiple modules into an end-to-end techs tack in minutes.
You can find the video here: https://www.youtube.com/watch?v=LVgP63BkhKQ
A comprehensive walkthrough of how to manage infrastructure-as-code using Terraform. This presentation includes an introduction to Terraform, a discussion of how to manage Terraform state, how to use Terraform modules, an overview of best practices (e.g. isolation, versioning, loops, if-statements), and a list of gotchas to look out for.
For a written and more in-depth version of this presentation, check out the "Comprehensive Guide to Terraform" blog post series: https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca
This beginning terraform workshop will teach you how to safely create and provision Infrastructure as Code (IAC) using Hashicorp Terraform in an AWS environment. In this class you will learn how to setup and install terraform. You will also be given a walkthrough of Terraform fundamentals. You will be lead through the process of deploying a single server, deploying a cluster and setting up a load balancer. You will also learn how to author Terraform Modules, work with Route53 and how to manage DNS.
Requirements. You will need to have an AWS account set up already with Terraform v0.9.3 installed. You will also need to have git install to download the workshop material.
You can find more informaiton on how to install terraform here: https://www.terraform.io/intro/getting-started/install.html. You can sign up for an AWS account here: https://aws.amazon.com/account/
https://github.com/jasonvance/terraform-introduction
Slides for my talk at the London Perl Workshop in Nov 2013, featuring the Devel::SizeMe perl module.
See also the screencast at https://archive.org/details/Perl-Memory-Profiling-LPW2013
Infrastructure-as-Code (IaC) Using Terraform (Advanced Edition)Adin Ermie
In this new presentation, we will cover advanced Terraform topics (full-on DevOps). We will compare the deployment of Terraform using Azure DevOps, GitHub/GitHub Actions, and Terraform Cloud. We wrap everything up with some key takeaway learning resources in your Terraform learning adventure.
NOTE: A recording of this presenting is available here: https://www.youtube.com/watch?v=fJ8_ZbOIdto&t=5574s
The why, what and how to leverage Terraform to manage Cloud resources safely.
Experience feedback from adoption by Leboncoin DataEngineering team.
In these slides you will find introduction material for beginners and advanced use cases you will quickly be facing when working within a team and with enterprise constraints.
Presentación empleada en el primer MeetUp AWS del grupo de usuarios de Valencia.
Infraestructura como código empleando Terraform. Se muestra las principales características de esta tecnología que nos permite ser más ágiles y rápidos desplegando nuestras plataformas en AWS.
Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto Docker, Inc.
Terraform is a tool for building and safely iterating on infrastructure, while Consul provides service discovery, monitoring and orchestration. In this talk we discuss using Terraform and Consul together to build a Docker-based Service Oriented Architecture at scale. We use Consul to provide the runtime control plane for the datacenter, and Terraform is used to modify the underlying infrastructure to allow for elastic scalability.
OSDC 2015: Mitchell Hashimoto | Automating the Modern Datacenter, Development...NETWAYS
Physical, virtual, containers. Public cloud, private cloud, hybrid cloud. IaaS, PaaS, SaaS. These are the choices that we're faced with when architecting a datacenter of today. And the choice is not one or the other; it is often a combination of many of these. How do we remain in control of our datacenters? How do we deploy and configure software, manage change across disparate systems, and enforce policy/security? How do we do this in a way that operations engineers and developers alike can rejoice in the processes and workflow?
In this talk, I will discuss the problems faced by the modern datacenter, and how a set of open source tools including Vagrant, Packer, Consul, and Terraform can be used to tame the rising complexity curve and provide solutions for these problems.
Kamailioworld 2018 - Modular and test driven SIP Routing with LuaSebastian Damm
Kamailio Routing logic written in native Kamailio language or using a single monolithic file is neither easily maintainable nor testable. This talk shows a way to maintain a unit tested Kamailio routing configuration, reducing the fear of having your customers discover your bugs.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Hivelance Technology
Cryptocurrency trading bots are computer programs designed to automate buying, selling, and managing cryptocurrency transactions. These bots utilize advanced algorithms and machine learning techniques to analyze market data, identify trading opportunities, and execute trades on behalf of their users. By automating the decision-making process, crypto trading bots can react to market changes faster than human traders
Hivelance, a leading provider of cryptocurrency trading bot development services, stands out as the premier choice for crypto traders and developers. Hivelance boasts a team of seasoned cryptocurrency experts and software engineers who deeply understand the crypto market and the latest trends in automated trading, Hivelance leverages the latest technologies and tools in the industry, including advanced AI and machine learning algorithms, to create highly efficient and adaptable crypto trading bots
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
2. zzz
Presenter – Zane Williamson
• @sepulworld - Github
• @zane_williamson – Twitter
• Sr. DevOps at Trulia – Data Eng. Team
2
3. zzz
Terraform at Trulia
1. 100+ applied Terraform states in action
2. We have been using Terraform since the 0.6 days
3. Trulia Innovation Week Project: Terraform State of Awareness
(https://github.com/sepulworld/tfsoa - telemetry data on TF states)
4. Trulia Innovation Week Project: tfmod-generator
(https://github.com/sepulworld/tfmod-generator - scaffolding
generator for Terraform modules that are Git sourced)
3
4. zzz
Overview
1. What is Immutablish infrastructure? How is it different from
Immutable?
2. How Trulia uses Terraform and consul-template
3. What to consider when going down this route, not a ‘silverbullet’
4
6. zzz
Immutable example on AWS
• No changes allowed to running
environment
• Changes performed by adding
new immutable instances and
removing old ones (Create before
Destroy)
• Not very flexible, but offers a
strictly controlled build and release
solution
• Configuration management only
used when AMI is created
6
Packer + Terraform Continuous Integration
Infrastructure Time Line
1. Start with live autoscale group running EC2 instances
from AMI-x
2.
Autoscale Group
(AMI-x)
Autoscale Group
(AMI-y)
Autoscale Group
(AMI-x)TerraformAMI-yPacker/Jenkins
Autoscale Group
(AMI-x)
Autoscale Group
(AMI-y)
4. End with new Autoscale group running AMI-y
3. Destroy Previous ASG
7. zzz
Immutablish example on AWS
• Example done with AWS
• SolrCloud cluster (basically a multi-node EC2 instance deployment)
• Requirements:
• Consul server cluster
• Zookeeper service registered with Consul
7
8. zzz
Immutablish example on AWS
• Changes allowed through pre-defined config files controlled by
consul-template
• More flexible than immutable environment, but contains more
operational complexity
• Will use user-data templates to bootstrap instances with configs
8
10. zzz
Terraform Toolbox
• Terraform Consul provider:
• Publish consul keys that can be used by consul-template to generate
dynamic configuration files
• Terraform Template provider:
• Used to generate unique user-data.sh bootstrap scripts for EC2 hosts
10
11. zzz
Terraform + Consul keys
• Terraform code to control
unique host specific key/value
entries on Consul cluster
• Each SolrCloud instance that
comes online to join this Consul
server cluster will have access
to key/values for consul-
template to leverage
11
--- snip ---
provider "consul" {
address = "${var.consul_endpoint}:8500"
datacenter = "${var.consul_datacenter}"
}
resource "consul_keys" "solr_config" {
count = "${var.solr_count}"
key {
path = "service/solr_conf/solr${count.index + 1}-$
{var.team}-${var.service}-
${var.environment}.${var.domain}/LOG4J_PROPS"
value = "${var.log4j_props}"
}
key {
path = "service/solr_conf/solr${count.index + 1}-
${var.team}-${var.service}-
${var.environment}.${var.domain}/SOLR_HOME"
value = "${var.solr_home}"
}
--- snip ---
12. zzz
Terraform template provider
• Template_file to take in variable
inputs and pass to a template file
• Template_cloudinit_config will then
be used to renderthe user-data.sh
as a text/x-shellscript
12
13. zzz13
Terraform template providers in action
Template_File resource counted out to the
number of SolrCloud systems required.
Pass in user defined variables to be used
in user-data.sh.
Render the templates. The second part is an optional
user defined script that can be appended.
Use the rendered templates for user_data bootstrap.
1
data "template_file" "script" {
count = "${var.solr_count}"
template = "${file("${path.module}/remote_scripts/user-
data.sh.tpl")}"
vars {
environment = "${var.environment}"
team = "${var.team}"
domain = "${var.domain}"
service = "${var.service}"
product = "${var.product}"
solr_count = "${count.index + 1}"
consul_server_tag_key = "${var.consul_server_tag_key}"
consul_server_tag_value = "${var.consul_server_tag_value}"
consul_datacenter = "${var.consul_datacenter}"
consul_agent_version = "${var.consul_agent_version}"
consul_template_version = "${var.consul_template_version}"
install_consul_agent = "${var.install_consul_agent}"
install_consul_template = "${var.install_consul_template}"
install_solr_version = "${var.install_solr_version}" } }
data "template_cloudinit_config" "userdata" {
count = "${var.solr_count}"
gzip = true base64_encode = true
part { content_type = "text/x-shellscript"
content = "${element(data.template_file.script.*.rendered, count.index)}" }
}
resource "aws_instance" "solrcloud" {
count = "${var.solr_count}"
user_data = "${element(data.template_cloudinit_config.userdata.*.rendered,
count.index)}"
root_block_device {
volume_type = "${var.root_block_device_volume_type}"
volume_size = "${var.root_block_device_volume_size}"
delete_on_termination = "${var.root_block_device_delete_on_termination}" }
---snip---
}
2
3
14. zzz14
Terraform to plug data into consul for SolrCloud consul-template
to use
--- snip ---
provider "consul" {
address =
"${var.consul_endpoint}:8500"
datacenter =
"${var.consul_datacenter}"
}
resource "consul_keys"
"solr_config" { datacenter =
"${var.consul_datacenter}" count =
"${var.solr_count}"
key {
path =
"service/solr_conf/solr${count.index +
1}-${var.team}-${var.service}-
${var.environment}.${var.domain}/LO
G4J_PROPS" value =
"${var.log4j_props}"
}
--- snip ---
# /etc/default/solr.init.sh.ctmpl
# Set the ZooKeeper connection string if using an external
ZooKeeper(s) # e.g. host1:2181,host2:2181/chroot
ZK_HOST="{{range service "zookeeper"}}{{.Address}}:2181,{{end}}"
# Set hostname to match system level we control
# Variables here are Terraform template interpolation,
SOLR_HOST="solr${solr_count}-${team}-${product}-
${environment}.${domain}"
# Generate rest of configuration based upon key/values found in consul
# Variables here are Terraform template interpolation,
{{range tree "service/solr_conf/solr${solr_count}-${team}-${product}-
${environment}.${domain}"}} {{.Key}}="{{.Value}}" {{end}}
# /var/solr/log4j.
properties.ctmpl
# Generate log4j configuration
from
#z Variables here are Terraform
template interpolation,
{{range tree
"service/solr/log4j/solr${solr_
count}-${team}-${product}-
${environment}.${domain}"}}
{{.Key}}="{{.Value}}"
{{end}}
Consul server cluster SolrCloud Cluster
15. zzz
Consul-template (the tool to make it Immutablish)
• “This project provides a convenient way to populate values
from Consul into the file system using the consul-template daemon”
• Light weight daemon
• Leverage the ”range” function in consul template
15
16. zzz
Consul-template in action
• Consul-template process that
manages /etc/default/solr.init.sh
• Discover ‘zookeeper’ hosts to
work with (required for
SolrCloud clustering)
• Utilize consul-templates ‘range
tree’ to discover key/values to
use in configuration
• Action upon change is to reload
SolrCloud service
16
# /etc/default/solr.init.sh.ctmpl
# Set the ZooKeeper connection string
if using an external
ZooKeeper(s) # e.g. host1:2181,host2:2181/chroot
ZK_HOST="{{range service
"zookeeper"}}{{.Address}}:2181,{{end}}"
# Set hostname to match system level we control
# Variables here are Terraform template
interpolation, SOLR_HOST="solr${solr_count}-
${team}-${product}-${environment}.${domain}"
# Generate rest of configuration based upon
key/values found in consul
# Variables here are Terraform template
interpolation,
{{range tree
"service/solr_conf/solr${solr_count}-${team}-
${product}-${environment}.${domain}"}}
{{.Key}}="{{.Value}}" {{end}}
SolrCloud Cluster
17. zzz
Consul-template in action, cont.
• Another pre-defined configuration file that we
want to make dynamic is
/var/solr/log4j.properties.ctmpl
• Leverage the ‘range tree’ again to generate
key/value entries in configuration
• Action upon change is to reload SolrCloud
service
17
# /var/solr/log4j.properties.ctmpl
# Generate log4j configuration from
#z Variables here are Terraform template
interpolation,
{{range tree
"service/solr/log4j/solr${solr_count}-
${team}-${product}-
${environment}.${domain}"}}
{{.Key}}="{{.Value}}"
{{end}}
SolrCloud Cluster
18. zzz
Consul-template: daemon configuration
• Leverage ‘consul lock’ to ensure
you restart only system at a time
18
consul {
address = "127.0.0.1:8500"
retry {
enabled = true
backoff = "10s"
}
}
reload_signal = "SIGHUP”
template {
source = “/etc/default/solr.init.sh.ctmpl”
destination = “/etc/default/solr.init.sh”
command = "consul lock lock/solr_restart systemctl restart solr"
command_timeout = "220s"
}
19. zzz
Immutablish post provision…
• Post provision you have the ability now to update consul key/values
to make configuration changes on the fly
• Consul-template daemon is configured to restart service if a change
is made
• Terraform is used to enforce key/values in Consul for specified
service configuration
19
21. zzz
What to consider when designing for Immutablish
1. Number of dynamic files to manage
• Ideally this is 1-3. If your service needs more than 3 files managed it will get convoluted.
2. What will manage the template files?
• Terraform Template provider coupled with user-data on AWS is a good option
• Continuous management could be done via Puppet or Chef
3. Is your service able to survive losing instances, and having new
ones come back online?
• This doesn’t work with all stateful services. We found it to work ok with services that store their
cluster state in a system like Zookeeper.
21
22. zzz
Challenges
1. Hard to test and iterate over template or key/value changes
• Would be great to have a local (Vagrant perhaps) test environment to try out consul-template changes and/or
consul key/value updates
2. If you have complex configuration file formats
• Building a template may get hairy if you have many different layers of values in the
configuration
3. Increased operational complexity vs a micro-service that is Immutable
• If you want your microservice and/or a capable stateful service like SolrCloud in this talk to be Immutablish,
this will increase operational complexity.
22
23. zzz
Key takeaways
1. Provides a way to simplify application configurations across multiple
environments, via templating
2. Terraform can help enforce configuration changes via its built in consul and
template providers
3. Excellent for state-less services, but could be challenging for stateful
services
23
