The last few years have been a period of tremendous growth for Reddit. Process, tooling, and culture have all had to adapt to an organization that has tripled in size and ambition. Greg Taylor discusses Reddit's evolution and explains how one of the world’s busiest sites develops, deploys, and operates services at significant scale.
Presented at OSCON 2018 in Portland, Oregon
Kubernetes at Reddit: An Origin Story - KubeCon NA 2018Gregory Taylor
This session takes the audience through the motivations, the story, and the results of rebasing our infrastructure on Kubernetes. Today, Kubernetes forms the core of our internal Infrastructure "product", which is a paint-by-numbers experience that engineers of all backgrounds make use of.
Presented at KubeCon North America in 2018
Helm at reddit: from local dev, staging, to productionGregory Taylor
How Reddit uses Helm in local dev, staging, and production. An overview of the primary pieces (Helm and Docker repos, CI), supporting tooling, and some best practices we've identified.
Recording: https://www.youtube.com/watch?v=7Qxuo9W5SlY
Watch the recording here: https://youtu.be/0KmqEp4VxSQ
Welcome Helm users! CNCF Flux has a best-in-class way to use Helm according to GitOps principles. For you, that means improved security, reliability, and velocity - no more being on the pager on the weekends or having painful troubleshooting or rollback when things go wrong. Built on Kubernetes controller-runtime, Flux’s Helm Controller is an example of a mature software agent that uses Helm’s SDK to full effect.
Flux’s biggest addition to Helm is a structured declaration layer for your releases that automatically gets reconciled to your cluster based on your configured rules:
⭐️ The Helm client commands let you imperatively do things
⭐️ Flux Helm Custom Resources let you declare what you want the Helm SDK to do automatically
During this session, Scott Rigby, Developer Experience Engineer at Weaveworks and Flux & Helm Maintainer, will take you on a tour of Flux’s Helm Controller, share the additional benefits Flux adds to Helm and then walk through a live demo of how to manage helm releases using Flux.
If you want to follow along with Scott’s demo, here are a couple of resources to help you prepare ahead of time:
📄 Flux for Helm Users Docs: https://fluxcd.io/docs/use-cases/helm/
📄 Flux Guide: Manage Helm Releases: https://fluxcd.io/docs/guides/helmreleases/
Speaker Bio:
Scott is a Brooklyn based interdisciplinary artist and Developer Advocate at Weaveworks. He co-founded the Basekamp art and research group in 1998 and the massively collaborative Plausible Artworlds international network. In technology he enjoys helping develop open source software that anyone can use, most recently projects in the cloud native landscape including co-maintaining Helm and Flux. In daily decisions, large or small, he tries to help make the world a better place for everyone.
OpenStack is a open source software for creating private and public clouds that coordinated collection of software from a few dozen related projects. This presentation give you an introduction about OpenStack and how OpenStack can help you in DevOps culture.
DevOps Meetup at AIS Tower 2 on February 10, 2017
Piotr Perzyna discusses orchestration and where it is heading. Orchestration has evolved from a task to a process, with benefits including configuration management and automation. Many companies are now using orchestration software to deploy containers in public clouds, with options like Google Kubernetes Engine and Amazon ECS for Kubernetes making entry level. Further information can be found through documentation, Slack communities, and meetups.
This document provides an overview of using GitLab for continuous integration and continuous delivery (CI/CD) processes. It begins with definitions of CI, CD, and when they should be configured. It then discusses GitLab's capabilities for the DevOps lifecycle and its advantages as a single application for collaboration across teams. The document outlines basic CI/CD concepts in GitLab like the YAML configuration file, pipelines, jobs, stages, and runners. It concludes with suggestions for real-life project settings like defining stages, variables, templates, environments, dependencies, and examples of build, deployment, and integration jobs.
DevOps is the future and next step for developer that need to learn. This session will explain why DevOps is important. The concept of DevOps and related technology and tools. Then how to start DevOps
Guest Speaker at IT@KMITL on March 20, 2019
Accelerate your business and reduce cost with OpenStackOpsta
OpenStack is a open source software for creating private and public clouds that coordinated collection of software from a few dozen related projects. This presentation give you an introduction about OpenStack and how OpenStack can help your business move faster and reduce cost.
CloudTalk #17 at AIA Tower on March 16, 2017
Kubernetes at Reddit: An Origin Story - KubeCon NA 2018Gregory Taylor
This session takes the audience through the motivations, the story, and the results of rebasing our infrastructure on Kubernetes. Today, Kubernetes forms the core of our internal Infrastructure "product", which is a paint-by-numbers experience that engineers of all backgrounds make use of.
Presented at KubeCon North America in 2018
Helm at reddit: from local dev, staging, to productionGregory Taylor
How Reddit uses Helm in local dev, staging, and production. An overview of the primary pieces (Helm and Docker repos, CI), supporting tooling, and some best practices we've identified.
Recording: https://www.youtube.com/watch?v=7Qxuo9W5SlY
Watch the recording here: https://youtu.be/0KmqEp4VxSQ
Welcome Helm users! CNCF Flux has a best-in-class way to use Helm according to GitOps principles. For you, that means improved security, reliability, and velocity - no more being on the pager on the weekends or having painful troubleshooting or rollback when things go wrong. Built on Kubernetes controller-runtime, Flux’s Helm Controller is an example of a mature software agent that uses Helm’s SDK to full effect.
Flux’s biggest addition to Helm is a structured declaration layer for your releases that automatically gets reconciled to your cluster based on your configured rules:
⭐️ The Helm client commands let you imperatively do things
⭐️ Flux Helm Custom Resources let you declare what you want the Helm SDK to do automatically
During this session, Scott Rigby, Developer Experience Engineer at Weaveworks and Flux & Helm Maintainer, will take you on a tour of Flux’s Helm Controller, share the additional benefits Flux adds to Helm and then walk through a live demo of how to manage helm releases using Flux.
If you want to follow along with Scott’s demo, here are a couple of resources to help you prepare ahead of time:
📄 Flux for Helm Users Docs: https://fluxcd.io/docs/use-cases/helm/
📄 Flux Guide: Manage Helm Releases: https://fluxcd.io/docs/guides/helmreleases/
Speaker Bio:
Scott is a Brooklyn based interdisciplinary artist and Developer Advocate at Weaveworks. He co-founded the Basekamp art and research group in 1998 and the massively collaborative Plausible Artworlds international network. In technology he enjoys helping develop open source software that anyone can use, most recently projects in the cloud native landscape including co-maintaining Helm and Flux. In daily decisions, large or small, he tries to help make the world a better place for everyone.
OpenStack is a open source software for creating private and public clouds that coordinated collection of software from a few dozen related projects. This presentation give you an introduction about OpenStack and how OpenStack can help you in DevOps culture.
DevOps Meetup at AIS Tower 2 on February 10, 2017
Piotr Perzyna discusses orchestration and where it is heading. Orchestration has evolved from a task to a process, with benefits including configuration management and automation. Many companies are now using orchestration software to deploy containers in public clouds, with options like Google Kubernetes Engine and Amazon ECS for Kubernetes making entry level. Further information can be found through documentation, Slack communities, and meetups.
This document provides an overview of using GitLab for continuous integration and continuous delivery (CI/CD) processes. It begins with definitions of CI, CD, and when they should be configured. It then discusses GitLab's capabilities for the DevOps lifecycle and its advantages as a single application for collaboration across teams. The document outlines basic CI/CD concepts in GitLab like the YAML configuration file, pipelines, jobs, stages, and runners. It concludes with suggestions for real-life project settings like defining stages, variables, templates, environments, dependencies, and examples of build, deployment, and integration jobs.
DevOps is the future and next step for developer that need to learn. This session will explain why DevOps is important. The concept of DevOps and related technology and tools. Then how to start DevOps
Guest Speaker at IT@KMITL on March 20, 2019
Accelerate your business and reduce cost with OpenStackOpsta
OpenStack is a open source software for creating private and public clouds that coordinated collection of software from a few dozen related projects. This presentation give you an introduction about OpenStack and how OpenStack can help your business move faster and reduce cost.
CloudTalk #17 at AIA Tower on March 16, 2017
This document provides an overview and agenda for a presentation about Calico, applications profiling, and BPF observability. The presentation covers Calico networking and security, what applications are, how BPF and eBPF can provide monitoring and observability capabilities like iostat and top traditionally have, and a demo of these techniques. It encourages connecting on social media and lists additional resources for troubleshooting and learning more about Calico, eBPF, and related tools.
DevOps is the future and next step for developer that need to learn. This session will explain why DevOps is important. The concept of DevOps and related technology and tools. Then how to start DevOps
Guest Speaker at ICT Mahidol on December 24, 2018
In DevOps world, the responsibility of monitoring is not only for the infrastructure team but also for developers as well. In this session, developers will know how to help and make monitor better with their coding to define monitoring.
Coding Defines Monitoring for Code Mania 100 : Coding Defines Anything at Faculty of Engineering, Chulalongkorn University on November 6, 2016
The document provides an overview of GitOps and Flux. It introduces GitOps as an operating model that utilizes a version controlled system like Git as a single source of truth for continuous delivery of cloud native applications like Kubernetes. It describes Flux as a GitOps tool that automates the deployment of Kubernetes workloads by automatically syncing apps and infrastructure from a Git repository to a Kubernetes cluster. The document highlights key benefits and capabilities of Flux such as working with any Kubernetes cluster and tools, multi-tenancy support, notifications, and an active community.
This document summarizes a presentation about deploying applications on Kubernetes with GitOps. The presentation covers GitOps workflows and tools like FluxCD and ArgoCD for managing Helm charts from Git repositories. It also discusses integrating continuous integration pipelines with ArgoCD and provides best practices for areas like secret management, scaling, and microservices. The presenter concludes by taking questions and inviting interested parties to join their company.
Google Cloud Platform (GCP) allows developers to build and deploy applications at scale. GCP provides infrastructure like virtual machines and containers to deploy applications without hardware limitations. It also offers services for continuous integration/delivery (CI/CD) pipelines, monitoring, error handling, and machine learning/artificial intelligence to add capabilities to applications. Completing a 30 day training on GCP can help engineers become more dynamic by learning how to use GCP's full suite of tools and services to build real-world applications.
Kubernetes: Increasing velocity without sacrificing qualityAdam Schepis
My talk from the Kubernetes Boston Meetup on Feb 20, 2018. I talked about how CloudHealth is using Kubernetes both in production and in development to increase velocity without sacrificing quality.
The document discusses Intuit's transition to using canary releases in Kubernetes instead of a separate performance environment. It describes how Intuit collects metrics during canary releases to detect performance issues before fully deploying to production. The canary analysis model measures pod resource usage, JVM metrics, and application metrics to compute a score. Intuit aims to refine the model and scale the canary release process by integrating with tools like Argo Rollouts, Prometheus, and a service mesh.
Integracia security do ci cd pipelinesJuraj Hantak
The document discusses integrating security into continuous integration and continuous delivery (CI/CD) pipelines. It covers topics like vulnerability management, application security testing methodologies, and integrating security checks into the phases of a CI/CD pipeline. The presenter demonstrates some security tools and their limitations. He emphasizes the importance of automating security, handling vulnerabilities like defects, and rolling out security testing incrementally to avoid disruption.
CI:CD in Lightspeed with kubernetes and argo cdBilly Yuen
Enterprises have benefited greatly from the elastic scalability and multi-region availability by moving to AWS, but the fundamental deployment model remains the same.
At Intuit, we have adopted k8s as our new saas platform and re-invented our CI/CD pipeline to take full advantage of k8s. In this presentation, we will discuss our journey from Spinnaker to Argo CD.
1. Reduce CI/CD time from 60 minutes to 10 minutes.
2. Reduce production release (or rollback) from 10 minutes to 2 minutes.
3. Enable concurrent deployment using spinnaker and argo cd as HA/DR to safely adopt the new platform with no downtime.
4. Be compatible with the existing application monitoring toolset.
This document discusses dockerizing an e-learning application called Avendoo. It provides an overview of Avendoo and why Docker was used. Key points include:
1. Avendoo is an application for managing and publishing e-trainings that was developed externally and customized for Haufe Akademie.
2. Docker was implemented to containerize and deploy Avendoo across different environments like development, testing, staging and production in a standardized way.
3. The Docker implementation uses a factory pattern with Dockerfiles, images and volumes to build, test and deploy Avendoo and its dependencies to new virtual machines created with Docker Machine.
The document discusses various deployment strategies for updating applications on Kubernetes including recreate, rolling update, blue/green, canary, and A/B testing deployments. For each strategy, it provides a brief overview of how the strategy works, examples of the Kubernetes configuration, advantages and disadvantages. It also includes hands-on sections and bonus questions to help understand how to implement each strategy in practice and when each one may be most appropriate.
What's New in GitLab and Software Development TrendsNoa Harel
Slides we delivered on a GitLab Meetup (in Israel)
- Trends in Software Development
- GitLab Overview
- GitLab achievements on 2017
- What's New in GitLab versions 10.3, 10.2 and 10.1
- GitLab Road-map for 2018
Knative is an open source project that provides serverless capabilities and eventing on Kubernetes. It includes Knative Serving for deploying and managing serverless applications and services, and Knative Eventing for enabling event-driven architectures. Knative aims to simplify continuous delivery for developers by abstracting away many of the complexities of Kubernetes and providing out-of-the-box capabilities for traffic management, auto-scaling, and event-driven workflows. The presentation covered the key components of Knative Serving and Eventing and how they can help enable continuous delivery of cloud native applications on Kubernetes.
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a ProCodefresh
>>Watch the full presentation, demo, and live Q&A here: https://codefresh.io/saucelabs-lp
The traditional "staging" model is getting in the way of adequately testing our applications. With the power of Kubernetes, Helm, Saucelabs, and Codefresh we can dramatically speed up and improve the quality of integration testing using short-lived environments.
Join Kubernaut Dan Garfield as he shows you how to:
- Create Automated short-lived environments for every commit/branch,
- run integration tests in a scalable way, and
- use Helm, Kubernetes, Codefresh, and Saucelabs integrations
Sign up for a FREE Codefresh account today: https://codefresh.io/codefresh-signup/
What's New for GitLab CI/CD February 2020Noa Harel
The document summarizes new features and enhancements to GitLab's CI/CD capabilities between November 2019 and January 2020. Key updates include associating milestones with releases, adding an environments dashboard to pipelines, enabling squash and merge for merge trains, and collecting evidence for releases. Planned highlights for February 2020 focus on package management, release management, and automated testing capabilities.
This document outlines lessons learned from building cloud native Java applications in Kubernetes. It introduces Activiti Cloud, an open source business automation platform, and discusses several relevant technologies and patterns for building cloud native applications including Spring Cloud, Docker, Kubernetes, Jenkins X, and microservices. The document then presents a scenario for implementing a cloud native application and outlines an agenda to do some hands-on work.
The Power of GitOps with Flux & GitOps ToolkitWeaveworks
GitOps Days Community Special
Watch the video here: https://youtu.be/0v5bjysXTL8
New to GitOps or been a long-time Flux user?
We'll walk you through the benefits of GitOps and then demo it in action with a sneak peak into the next gen Flux and GitOps Toolkit!
* Automation!
* Visibility!
* Reconciliation!
* Powerful use of Prometheus and Grafana!
* GitOps for Helm!
For Flux users, Flux v1 is decoupled into Flux v2 and GitOps Toolkit. We'll demo how this decoupling gives you more control over how you can do GitOps and with fewer steps!
Join Leigh Capili and Tamao Nakahara as they show you GitOps in action with Flux and GitOps Toolkit.
Note to our Flux community that Flux v2 and the GitOps Toolkit is in development and Flux v1 is in maintenance mode. These talks and upcoming guides will give you the most up-to-date info and steps to migrate once we reach feature parity and start the migration process. We are dedicated to the smoothest experience possible for our Flux community, so please join us if you'd like early access and to give us feedback for the migration process.
We are really excited by the improvements and want to take this opportunity to show you what the GitOps Toolkit is all about, walk you through the guides and get your feedback!
For more info, see https://toolkit.fluxcd.io/.
Here's our latest blog post on Flux v2 and GitOps Toolkit updates: https://www.weave.works/blog/the-road-to-flux-v2-october-update
This document summarizes GitOps and the benefits of using GitOps for continuous delivery and deployment. It discusses how GitOps allows for simplified continuous delivery through using Git as a single source of truth, which can enhance productivity and experience while also increasing stability. Rollbacks are easy if issues arise by reverting commits in Git. Additional benefits include reliability, consistency, security, and auditability of changes. The document also provides an overview of ArgoCD, an open source GitOps tool for continuous delivery on Kubernetes, and its architecture.
This document discusses Spring Boot and Spring Cloud applications running on Kubernetes and Pivotal Cloud Foundry. It introduces the speaker and covers topics like different platforms, choosing between Kubernetes and PCF, and developer experience differences. PCF is highlighted as providing lower development complexity while Kubernetes offers more flexibility and control. The benefits of both platforms for running Spring applications are discussed.
Ernest Mueller, Karthik Gaekwad, and James Wickett, the Agile Admins (http://theagileadmin.com) delivered this presentation on what's hot in DevOps in 2015 for the BrightTALK Summit. The video is online at https://www.brighttalk.com/webcast/5742/154715
In Data Engineer’s Lunch #68, Will Angel, Technical Product Manager at Caribou Financial, will provide an introduction to DevOps practices and tooling including testing, deployment automation, logging, monitoring, and DevOps principles. Additionally, we will discuss some of the ways that DevOps for data engineering is different from conventional application development.
Accompanying Blog: Coming Soon!
Accompanying YouTube: https://youtu.be/eBtrOv_qLHQ
Sign Up For Our Newsletter: http://eepurl.com/grdMkn
Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday:
https://www.meetup.com/Data-Wranglers-DC/events/
Cassandra.Link:
https://cassandra.link/
Follow Us and Reach Us At:
Anant:
https://www.anant.us/
Awesome Cassandra:
https://github.com/Anant/awesome-cassandra
Email:
solutions@anant.us
LinkedIn:
https://www.linkedin.com/company/anant/
Twitter:
https://twitter.com/anantcorp
Eventbrite:
https://www.eventbrite.com/o/anant-1072927283
Facebook:
https://www.facebook.com/AnantCorp/
Join The Anant Team:
https://www.careers.anant.us
This document provides an overview and agenda for a presentation about Calico, applications profiling, and BPF observability. The presentation covers Calico networking and security, what applications are, how BPF and eBPF can provide monitoring and observability capabilities like iostat and top traditionally have, and a demo of these techniques. It encourages connecting on social media and lists additional resources for troubleshooting and learning more about Calico, eBPF, and related tools.
DevOps is the future and next step for developer that need to learn. This session will explain why DevOps is important. The concept of DevOps and related technology and tools. Then how to start DevOps
Guest Speaker at ICT Mahidol on December 24, 2018
In DevOps world, the responsibility of monitoring is not only for the infrastructure team but also for developers as well. In this session, developers will know how to help and make monitor better with their coding to define monitoring.
Coding Defines Monitoring for Code Mania 100 : Coding Defines Anything at Faculty of Engineering, Chulalongkorn University on November 6, 2016
The document provides an overview of GitOps and Flux. It introduces GitOps as an operating model that utilizes a version controlled system like Git as a single source of truth for continuous delivery of cloud native applications like Kubernetes. It describes Flux as a GitOps tool that automates the deployment of Kubernetes workloads by automatically syncing apps and infrastructure from a Git repository to a Kubernetes cluster. The document highlights key benefits and capabilities of Flux such as working with any Kubernetes cluster and tools, multi-tenancy support, notifications, and an active community.
This document summarizes a presentation about deploying applications on Kubernetes with GitOps. The presentation covers GitOps workflows and tools like FluxCD and ArgoCD for managing Helm charts from Git repositories. It also discusses integrating continuous integration pipelines with ArgoCD and provides best practices for areas like secret management, scaling, and microservices. The presenter concludes by taking questions and inviting interested parties to join their company.
Google Cloud Platform (GCP) allows developers to build and deploy applications at scale. GCP provides infrastructure like virtual machines and containers to deploy applications without hardware limitations. It also offers services for continuous integration/delivery (CI/CD) pipelines, monitoring, error handling, and machine learning/artificial intelligence to add capabilities to applications. Completing a 30 day training on GCP can help engineers become more dynamic by learning how to use GCP's full suite of tools and services to build real-world applications.
Kubernetes: Increasing velocity without sacrificing qualityAdam Schepis
My talk from the Kubernetes Boston Meetup on Feb 20, 2018. I talked about how CloudHealth is using Kubernetes both in production and in development to increase velocity without sacrificing quality.
The document discusses Intuit's transition to using canary releases in Kubernetes instead of a separate performance environment. It describes how Intuit collects metrics during canary releases to detect performance issues before fully deploying to production. The canary analysis model measures pod resource usage, JVM metrics, and application metrics to compute a score. Intuit aims to refine the model and scale the canary release process by integrating with tools like Argo Rollouts, Prometheus, and a service mesh.
Integracia security do ci cd pipelinesJuraj Hantak
The document discusses integrating security into continuous integration and continuous delivery (CI/CD) pipelines. It covers topics like vulnerability management, application security testing methodologies, and integrating security checks into the phases of a CI/CD pipeline. The presenter demonstrates some security tools and their limitations. He emphasizes the importance of automating security, handling vulnerabilities like defects, and rolling out security testing incrementally to avoid disruption.
CI:CD in Lightspeed with kubernetes and argo cdBilly Yuen
Enterprises have benefited greatly from the elastic scalability and multi-region availability by moving to AWS, but the fundamental deployment model remains the same.
At Intuit, we have adopted k8s as our new saas platform and re-invented our CI/CD pipeline to take full advantage of k8s. In this presentation, we will discuss our journey from Spinnaker to Argo CD.
1. Reduce CI/CD time from 60 minutes to 10 minutes.
2. Reduce production release (or rollback) from 10 minutes to 2 minutes.
3. Enable concurrent deployment using spinnaker and argo cd as HA/DR to safely adopt the new platform with no downtime.
4. Be compatible with the existing application monitoring toolset.
This document discusses dockerizing an e-learning application called Avendoo. It provides an overview of Avendoo and why Docker was used. Key points include:
1. Avendoo is an application for managing and publishing e-trainings that was developed externally and customized for Haufe Akademie.
2. Docker was implemented to containerize and deploy Avendoo across different environments like development, testing, staging and production in a standardized way.
3. The Docker implementation uses a factory pattern with Dockerfiles, images and volumes to build, test and deploy Avendoo and its dependencies to new virtual machines created with Docker Machine.
The document discusses various deployment strategies for updating applications on Kubernetes including recreate, rolling update, blue/green, canary, and A/B testing deployments. For each strategy, it provides a brief overview of how the strategy works, examples of the Kubernetes configuration, advantages and disadvantages. It also includes hands-on sections and bonus questions to help understand how to implement each strategy in practice and when each one may be most appropriate.
What's New in GitLab and Software Development TrendsNoa Harel
Slides we delivered on a GitLab Meetup (in Israel)
- Trends in Software Development
- GitLab Overview
- GitLab achievements on 2017
- What's New in GitLab versions 10.3, 10.2 and 10.1
- GitLab Road-map for 2018
Knative is an open source project that provides serverless capabilities and eventing on Kubernetes. It includes Knative Serving for deploying and managing serverless applications and services, and Knative Eventing for enabling event-driven architectures. Knative aims to simplify continuous delivery for developers by abstracting away many of the complexities of Kubernetes and providing out-of-the-box capabilities for traffic management, auto-scaling, and event-driven workflows. The presentation covered the key components of Knative Serving and Eventing and how they can help enable continuous delivery of cloud native applications on Kubernetes.
Skip Staging! Test Docker, Helm, and Kubernetes Apps like a ProCodefresh
>>Watch the full presentation, demo, and live Q&A here: https://codefresh.io/saucelabs-lp
The traditional "staging" model is getting in the way of adequately testing our applications. With the power of Kubernetes, Helm, Saucelabs, and Codefresh we can dramatically speed up and improve the quality of integration testing using short-lived environments.
Join Kubernaut Dan Garfield as he shows you how to:
- Create Automated short-lived environments for every commit/branch,
- run integration tests in a scalable way, and
- use Helm, Kubernetes, Codefresh, and Saucelabs integrations
Sign up for a FREE Codefresh account today: https://codefresh.io/codefresh-signup/
What's New for GitLab CI/CD February 2020Noa Harel
The document summarizes new features and enhancements to GitLab's CI/CD capabilities between November 2019 and January 2020. Key updates include associating milestones with releases, adding an environments dashboard to pipelines, enabling squash and merge for merge trains, and collecting evidence for releases. Planned highlights for February 2020 focus on package management, release management, and automated testing capabilities.
This document outlines lessons learned from building cloud native Java applications in Kubernetes. It introduces Activiti Cloud, an open source business automation platform, and discusses several relevant technologies and patterns for building cloud native applications including Spring Cloud, Docker, Kubernetes, Jenkins X, and microservices. The document then presents a scenario for implementing a cloud native application and outlines an agenda to do some hands-on work.
The Power of GitOps with Flux & GitOps ToolkitWeaveworks
GitOps Days Community Special
Watch the video here: https://youtu.be/0v5bjysXTL8
New to GitOps or been a long-time Flux user?
We'll walk you through the benefits of GitOps and then demo it in action with a sneak peak into the next gen Flux and GitOps Toolkit!
* Automation!
* Visibility!
* Reconciliation!
* Powerful use of Prometheus and Grafana!
* GitOps for Helm!
For Flux users, Flux v1 is decoupled into Flux v2 and GitOps Toolkit. We'll demo how this decoupling gives you more control over how you can do GitOps and with fewer steps!
Join Leigh Capili and Tamao Nakahara as they show you GitOps in action with Flux and GitOps Toolkit.
Note to our Flux community that Flux v2 and the GitOps Toolkit is in development and Flux v1 is in maintenance mode. These talks and upcoming guides will give you the most up-to-date info and steps to migrate once we reach feature parity and start the migration process. We are dedicated to the smoothest experience possible for our Flux community, so please join us if you'd like early access and to give us feedback for the migration process.
We are really excited by the improvements and want to take this opportunity to show you what the GitOps Toolkit is all about, walk you through the guides and get your feedback!
For more info, see https://toolkit.fluxcd.io/.
Here's our latest blog post on Flux v2 and GitOps Toolkit updates: https://www.weave.works/blog/the-road-to-flux-v2-october-update
This document summarizes GitOps and the benefits of using GitOps for continuous delivery and deployment. It discusses how GitOps allows for simplified continuous delivery through using Git as a single source of truth, which can enhance productivity and experience while also increasing stability. Rollbacks are easy if issues arise by reverting commits in Git. Additional benefits include reliability, consistency, security, and auditability of changes. The document also provides an overview of ArgoCD, an open source GitOps tool for continuous delivery on Kubernetes, and its architecture.
This document discusses Spring Boot and Spring Cloud applications running on Kubernetes and Pivotal Cloud Foundry. It introduces the speaker and covers topics like different platforms, choosing between Kubernetes and PCF, and developer experience differences. PCF is highlighted as providing lower development complexity while Kubernetes offers more flexibility and control. The benefits of both platforms for running Spring applications are discussed.
Ernest Mueller, Karthik Gaekwad, and James Wickett, the Agile Admins (http://theagileadmin.com) delivered this presentation on what's hot in DevOps in 2015 for the BrightTALK Summit. The video is online at https://www.brighttalk.com/webcast/5742/154715
In Data Engineer’s Lunch #68, Will Angel, Technical Product Manager at Caribou Financial, will provide an introduction to DevOps practices and tooling including testing, deployment automation, logging, monitoring, and DevOps principles. Additionally, we will discuss some of the ways that DevOps for data engineering is different from conventional application development.
Accompanying Blog: Coming Soon!
Accompanying YouTube: https://youtu.be/eBtrOv_qLHQ
Sign Up For Our Newsletter: http://eepurl.com/grdMkn
Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday:
https://www.meetup.com/Data-Wranglers-DC/events/
Cassandra.Link:
https://cassandra.link/
Follow Us and Reach Us At:
Anant:
https://www.anant.us/
Awesome Cassandra:
https://github.com/Anant/awesome-cassandra
Email:
solutions@anant.us
LinkedIn:
https://www.linkedin.com/company/anant/
Twitter:
https://twitter.com/anantcorp
Eventbrite:
https://www.eventbrite.com/o/anant-1072927283
Facebook:
https://www.facebook.com/AnantCorp/
Join The Anant Team:
https://www.careers.anant.us
AWS Community Day: From Monolith to Microservices - What Could Go Wrong?Phuong Mai Nguyen
Almost every tech organisation right from start-ups to unimaginably big ones have had monolithic applications in the past and have moved on to nimbler approaches like microservices, making use of powerful cloud technologies. But not every organisation has made this move yet, with most of them still in analysing phase.
If you are part of this or interested in exploring how major players in the industry have managed to convert monoliths to microservices, join us in the talk to get an in-depth knowledge about things that could go wrong and how to make the right choices using AWS services. On top of practical techniques and real-life case studies, we will also be exploring agile methodologies and discuss if microservices are the right choice for your field of work.
This document discusses the journey to cloud computing and cloud native applications. It covers evolving from on-premise servers and monolithic applications to distributed architectures like microservices, containers, and serverless functions. The key steps are assessing applications to determine readiness, prioritizing workloads based on business value, and establishing centers of excellence to help teams migrate applications incrementally through pilots while learning from others' experiences. The goal is to maximize cloud advantages like elastic scaling and continuous delivery while navigating technical challenges along the path to cloud native.
Data Science in Production: Technologies That Drive Adoption of Data Science ...Nir Yungster
Critical to a data science team’s ability to drive impact is its effectiveness in incorporating its solutions into new or existing products. When collaborating with other engineering teams, and especially when solutions must operate at scale, technological choices can be critical factors in determining what type of outcome you'll have. We walk through strategies and specific technologies - Airflow, Docker, Kubernetes - that can help promote successful collaboration between data science and engineering.
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...Demi Ben-Ari
Kubernetes (K8s) is a tool for managing containerized applications across multiple servers. It allows deploying and managing containerized applications without relying on virtual machines. Kubernetes can schedule containers across a cluster of nodes, provide basic health checking and restart policies, load balancing, storage orchestration and more. Some key Kubernetes concepts include pods, deployments, services, replication controllers and volumes. Kubernetes is well suited for microservices architectures as it helps manage the scaling and networking needs of distributed applications.
ProtoTech Solutions have expertise is 3D software development. We have experience in Graphics technologies (HOOPS,OpenGL,RedSDK), Geometry kernel (ACIS, Parasolid) and Interoperability solutions (HOOPS Exchange, InterOp, RealDWG, Teigha). Our development expertise includes for, Onshape, Fusion 360, cloud technologies (REST, PHP etc.) and User Interface.
[APIdays Singapore 2019] Managing the API lifecycle with Open Source Technolo...WSO2
This document discusses the benefits of using open source software to manage API lifecycles. It notes that digital transformation requires integrating new technologies rapidly, which open source allows through wider collaboration and input. Open source ensures better security, transparency, and extensibility. It also leads to higher quality code through more eyeballs and passionate developers. Open source APIs are also more cost effective and support corporate social responsibility goals. The document cites WSO2 as an example of an open source API management vendor that contributes significantly to many open source projects.
The document discusses microservices and the benefits of developing using a microservices architecture. It describes the author's experience working on both large monolithic systems and microservice architectures. Some key benefits of microservices mentioned include improved knowledge sharing, faster progress and adoption of new technologies, easier elimination of technical debt, and increased accountability. The author argues that developing with microservices can achieve scalable high productivity, but certain factors like doing agile well and defining roles appropriately are required to realize the benefits.
Efficient platform engineering with Microk8s & gopaddle.pdfVinothini Raju
This document discusses the evolution of DevOps and platform engineering challenges. It introduces gopaddle as a low-code DevSecOps platform for Kubernetes that can provide automated multi-cloud provisioning, issue identification, and app release automation. Using MicroK8s and the gopaddle Lite addon allows setting up a platform in 15-20 minutes, improving efficiency and reducing time to develop and deploy microservices compared to traditional methods.
Becoming the Docker Champion: Bringing Docker Back to WorkDocker, Inc.
This document provides tips for becoming a Docker champion by building an advocate team within an organization. It discusses assessing needs and scoping a Docker project. Common hurdles for getting different teams like app development, IT operations, and security on board are outlined. Tips from other customers are provided for each team. The document then discusses how Docker can help with issues like faster development and deployment, increased server density, reduced maintenance time, and security. Specific products in Docker Enterprise Edition are highlighted for providing a common platform and reducing risks. Metrics for measuring before and after a Docker implementation are suggested.
From prototype to production - The journey of re-designing SmartUp.ioMáté Lang
Talk about the joureny of small tech team re-designing SmartUp.io from scratch, and the technical paths from MVP to Production.
High level overview of architecture and tech stack decisions, best-practices and culture.
What's new in the latest source{d} releases!source{d}
We recently announce source{d} 0.11, 0.12 and 0.13, two releases with lots of new features and performance improvements. From windows support, to port management, C# language support and new SQL querying, there is a lot for you to get excited about. We also discussed why you should care about Engineering Observability and what are some of the top use cases for source{d} in enterprises.
Platform Engineering is the practice of building and operating a common platform as a product for technology teams.
In this session, we will talk about why and when we need a platform. How to build Platform Engineering and demo.
Jirayut Nimsaeng
Founder & CEO
Opsta (Thailand) Co., Ltd.
Youtube Record: https://youtu.be/brBZYbNbnAo
Dev Mountain Tech Festival 2022 @ Khaoyai
March 19, 2022
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2Gtedjh.
Szczepan Faber talks about two different use cases of implementing continuous delivery at scale: LinkedIn and Mockito. Yet the challenges, benefits and impact on the engineering culture are very similar. Filmed at qconsf.com.
Szczepan Faber is a Tech Lead for LinkedIn Development Tools, responsible for developer productivity at LinkedIn. Previously, he was core engineer of Gradle 1.x and 2.x. and instructed numerous classes on build automation. He created Mockito framework in 2007, currently estimated user base of 2M, and has been giving classes on automated testing since.
Excalibur: best practices for virtual desktop operations leveraging Citrix Di...Citrix
The document discusses upcoming enhancements to Citrix Director and EdgeSight products. It highlights how the products are being integrated to provide a unified monitoring console for XenDesktop environments. Key updates include performance management features in EdgeSight for long-term trend analysis, and network inspection capabilities for troubleshooting connection issues. Real-world examples are provided on how Citrix internal IT uses the products to monitor their environment.
Until recently, technology was seen as a cost centre. Software was built and operated throughout different silos, with different people, culture, tools and reward systems. The software lifecycle was neglected, creating a legacy of fragile applications.
The world has changed. We are now living in a software-driven economy, disrupting traditional businesses and changing the world’s system of capitalism as we have known it for centuries.
In today's economy, engineering effectiveness is imperative to prevent business atrophy and death. This requires an energetic response.
This talk explores how to evolve engineering effectiveness in a value-centric approach, helping you move towards having autonomous teams able to promote small and frequent changes, aligned to your unique context, objectives and concerns.
Technology and Digital Platform | 2019 partner summitAndrew Kumar
Technology: Andrew Kumar will share a refresher of our technology standards, documentation while highlighting what is changing in 2019 in the reference architecture and starter kits.
Digital Platform: Andrew Kumar will follow tech and design updates with a refresher on why the digital platform matters, what exists in the digital platform, what is being worked on, and what is coming next as we co-create value, save team member effort, and improve speed to market with investments in the digital platform.
Similar to Develop, deploy, and operate services at reddit scale oscon 2018 (20)
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Zilliz
Join us to introduce Milvus Lite, a vector database that can run on notebooks and laptops, share the same API with Milvus, and integrate with every popular GenAI framework. This webinar is perfect for developers seeking easy-to-use, well-integrated vector databases for their GenAI apps.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Speck&Tech
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
19. Growing pains: Automated testing
● Problem: Unit/integration testing as an afterthought
○ “We know we should do better, buut….”
○ We needed CI for mere mortals
● Solution: Cultural shift
○ Git hooks are not enough!
○ Testing, linting, integration, oh my!
○ Green master branches
● https://drone.io/
20. Growing pains: Something to build on
● Problem: Artisanal, hand-crafted services
● Solution: Reddit service framework
○ Named “baseplate”
○ Consistency between services
○ Batteries-included
● https://baseplate.readthedocs.io
21. Hey! That was pretty cool. Let’s do it again!
Service
Monolith
Service
23. Once more, with feeling!
Service
Service
Monolith
Service
Service
24. Growing pains: Staging/integration woes
● Problem: Inadequate staging solution for SOA
● Outlandish idea: Give Kubernetes a shot
○ Let’s see what we can MVP in 3 weeks!
○ Pile of manifests and bash scripts
○ MVP very positively received
● https://kubernetes.io/
26. Growing pains: Infra team as a bottleneck
● Problem: Development is too dependent on Infra
○ Service provisioning
○ Ongoing operation
○ Debugging and performance work
● Short-term “solution”: Train and deputize
infrastructure-oriented teams
○ Allows for more self-sufficiency
○ Results in accelerated dev velocity
27. One size fits all some
Not all teams want to operate the full stack for their service
Don’t make me
infrastructure!
My toaster
runs Docker
30. Service ownership!
A service owner is empowered to:
● Dev and test their service in a prod-like env
● Do most of the work to get to production
● Own the health of their service
● Diagnose issues
39. The user/operator contract
Product Users (Service Owners)
● Learn some Kubernetes basics
● Deploy and operate own services
Product Operators (Reddit Infrastructure)
● Keep the Kubernetes clusters running
● Provision AWS resources
● Support and advise Product Users
40. Batteries-included
Provide as much as we can out-of-the-box:
● Test, build, and deploy pipelines
● Observability
● Service discovery
● Baseline security
41. Paint-by-numbers
Enabling service ownership for all backgrounds:
● Limit learning expectations
● You want to do X? Here’s a guide for that
● Training and extensive documentation
● Auto-generate all the things
An engineer should not require deep infra
experience in order to be productive!
42. Consistency throughout
As a developer takes an application through its development cycle, the
operating environment should be as consistent as possible
Local dev Staging Production
43. Deploy with confidence
A developer should have tools and processes that will help them build
confidence as work approaches production
Code Review Unit Testing Integration Testing
Some
44. Guardrails and Safeties
Things that prevent or minimize damage
● Resource limits
● Throttling
● Network policy and Access controls
● Scanning for common mistakes
● Docker Image policies
45. Reddit Infrastructure Product Core Tenets
● Limit the surface area
● The user/operator contract
● Batteries-included
● Paint-by-numbers
● Consistency throughout
● Deploy with confidence
● Guardrails and Safeties
48. Infra team: From Operators to Enablers
Infra provisions all infrastructure Infra provides infra as a product
Infra deploys new services Service owners deploy new services
Infra operates most services Service owners operate services
Infra is a blocking dependency Infra is an adviser and enabler
I’m Greg Taylor, and I’m an engineering manager within our Infrastructure department.
Today I’ll be sharing the story of how we’ve scaled our infrastructure, our process, and our culture as we’ve grown one of the most active sites on the internet
What is Reddit?
But first: What is Reddit?
Depending on who you ask, you’ll get all kinds of answers:
“The Front Page of the Internet”, in a nod to its origins as a content aggregator
Where the cat memes live
A place for lively discourse that is always civil and serious
Today we tend to define Reddit as a social network whose focus lies on user-operated communities called subreddits.
These subreddits cover topics such as news, politics, sports, gaming, technology, music, and entertainment.
To make this more concrete, let’s run through some of my current favorites
r/CrappyDesign
Welcome to CrappyDesign, self-described as “where the comic sans and lens flare flow unfiltered”
As you attempt to read this slide, you’ll probably get an idea of what this subreddit is about.
Cupholders that don’t hold cups, interesting error messages, cringeworthy images, these are all the province of CrappyDesign.
r/theocho
Next up is The Ocho, dedicated to spreading knowledge of seldom seen and obscure sports.
You are seeing the crowd-pleasing “Trampoline Roman candle battle”.
And while not all of the sports covered by this sub are as explosive, they are every bit as unusual.
As a sidenote, please make good decisions, friends!
r/news
But Reddit offers far more than just fun and memes. For many, it’s the place to go to discuss happenings around the world.
In most cases, you’ll be able to find discussion of current events moments after they happen.
We’re showcasing /r/news on this slide, but there are many more news-oriented communities.
You can use subreddits like these to stay in touch with what’s going on in your city, your state, your country, or the wider world.
r/daddit
As my shameless Dad jokes probably gave away, I’m a father. A relatively new one at that.
If I’m telling the truth, I have no idea what I’m doing.
And unfortunately, I’ve been unable to find the user manual for my little guy thus far.
But r/daddit is the next best thing.
You’ll find other dads in here asking questions, sharing tips, telling stories, celebrating milestones, and comforting one another during the more difficult times.
Everyone gets something different from a community like this, but I will say that daddit has comforted, informed, and assisted me in my ongoing pursuit of one of those “World’s Best Dad” coffee mugs.
Before we move on, I’ll also take a moment to plug a few other similar communities: r/mommit, r/beyondthebump, and r/parenting.
r/IAMA
Reddit is also great for Q&A-style interactions
In r/IAMA, noteworthy folks stop by to conduct AMAs, or “Ask Me Anything” sessions
The participants range from politicians, athletes, entertainers, musicians, successful business people, philanthropists, and even the second person to walk on the moon.
You won’t find a place on the internet where you have more direct access to interesting individuals like these.
Now that we’ve covered a few subreddits, let’s get a sense for the tremendous scale that Reddit operates at.
Reddit by the numbers
We’re in the top 10 most trafficked sites in the world.
Our monthly active users number over 330 million
To help put that into perspective, the population of the US is about 326 million.
We’ve got over 138 thousand active communities (or subreddits).
These communities produce over 12 million posts per month.
Those 12 million posts generate about 2 billion votes per month.
Now, given the numbers you see here, I want you all to take a moment to guess the size of the engineering organization that operates and develops something like this.
We’ll show you the actual numbers on the next slide in a moment.
Rapid Recent Growth
If you guessed about 185 engineers, you’d be correct.
An interesting fact about Reddit is that we’ve been a tiny engineering org for most of our thirteen years. At least relative to our traffic.
You can see the general trend on the right, with engineering in dark orange.
We started 2016 with about 25 engineers. This was a record-high for us at the time. We thought we were hot stuff.
Today we clock in at over 185 engineers, which is more than a 7X increase over a period of less than three years.
And while this growth has been exciting to be a part of, it hasn’t been without challenge.
I’m here today to share the story of the growth and evolution of our team, our culture, our process, and our stack.
So hop in your choice of time machine and fasten your seatbelts.
We’re going back to 2016.
Redding Engineering in 2016
And here we are, Reddit Engineering as it was in early 2016.
We numbered about 25 engineers, with about three teams writing code in some form.
Pictured in the center, you’ll see our lovable monolith surrounded by its maintainers.
2016 - The Infrastructure Team
One of the few teams to rise from the primordial soup was the Infrastructure team.
There were about five of us on the team.
At the time, we handled a mixture of operations and some backend development on the monolith.
Since our infrastructure was relatively static and we weren’t spawning new services, the infrastructure team handled all provisioning and configuration of systems.
We also handled most non-trivial debugging, since we had enough access to poke into the most dusty and forgotten of corners of the codebase.
This arrangement served us well for over a decade.
2016 - The Stack
Here’s what our stack looked like at the time.
There’s nothing too crazy or noteworthy here, which was a positive for us.
This stack helped us run one of the most active sites in the world with a very small infrastructure team.
Mid 2016 - Rapid Growth
As we got to mid 2016, we started growing. Rapidly.
Reddit’s ambitions had expanded, necessitating the growth of the engineering org.
Pictured here is one of our new employee onboarding sessions in 2016.
Well, not really. But this is what it felt like.
Diminishing returns
As our ranks swelled, we began having difficulties in scaling our development of the monolith.
While there are plenty of examples of larger teams developing working together on a monolith, our particular monolith was fragile and full of sharp edges that had accumulated over many years.
It was difficult to make changes without unexpected breakages. Sort of like the development equivalent of the butterfly effect, but with more things exploding.
As a result of this, our productivity was not scaling as well as we’d like as the org grew.
Determining the path forward
So we had a spirited gathering to decide a path forward.
Our goal was to set a course that would allow many engineering teams to efficiently develop and operate alongside one another.
We considered a number of options, running the gamut from rehabilitating the monolith to a complete rewrite.
In the meantime, we had a roadmap full of exciting things that needed to be built.
Regardless of our chosen approach, we couldn’t afford to halt all development.
After much discussion and gnashing of teeth, we decided to...
Service-oriented Architecture
Start picking the monolith apart into a service-oriented architecture, or an SOA.
Previously, as a smaller org the additional complexity of an SOA was unattractive to us.
But as a larger (and still rapidly growing) org with a sprawling and fragile monolith, we saw this as the best path forward.
We thought that this would give us a better separation of concerns, and were ready and equipped to deal with the tradeoffs.
But I will say: If you have a monolith that your team is productively developing, go home and give it a big hug. Tell it you love it, warts and all.
r/IAMA
With the decision made, our first service came to be in early to mid 2016.
< Hit next >
As we brought it online alongside the monolith, trumpets sounded, fireworks took flight, a choir angels sang.
It was a beautiful moment.
Growing pains: Automated testing
But, this exercise drove home the point that we needed to improve our story for automated testing before we went much further.
This was something we always had wanted to improve, but it never made its way up “the list”.
There were three or four different attempts to more deeply incorporate automated testing over the company’s history, but it never took root.
With many new services on the horizon, we knew that had to change.
So we researched options and decided to incorporate Drone CI, which we found to be easy to use and powerful enough.
We started doing more linting, testing, and endeavored not to break master branches.
Over time, automated testing became an expectation rather than a nicety.
Growing pains: Something to build on
The other thing that became quickly apparent is the need for a foundation to build services on.
Rather than copy pasta’ing new services from one another, we’d develop a Reddit service framework.
We named it “baseplate”, and still use it to this day.
By building our services on baseplate, we get a measure of consistency between services. They are configured the same way, they expose the same or similar ports, they have the same async event loop, they all handle RPC clients and servers the same. We’ll revisit why this is important later on.
We’ve also baked in a bunch of our patterns and best practices so the other teams don’t have to re-invent the wheel.
For example, you get logging, instrumentation, tracing, secrets fetching, and a bunch of other things out of the box.
The other interesting bit is that we develop baseplate in the open on GitHub.
While this is a very opinionated building block, we make it available with the hopes that it helps or inspires someone out there.
Hey! That was pretty cool. Let’s do it again!
Back to mid 2016.
Fresh off of our success with our first service, it was time to stand a second up using baseplate, our new services framework.
< Hit Next >
We didn’t bother with the fireworks or choir of angels this time, but this went pretty smoothly as well.
Growing pains: Artisanal Infrastructure
However, the realization was setting in that our artisanal infrastructure was going to be a liability in a service-oriented world.
By artisanal, I mean going into the AWS Console and point-and-clicking your way through things.
This seems so obvious to us now, but keep in mind that Reddit predates CloudFormation, Terraform, and a host of other similar technologies.
We resolved to shift towards an Infrastructure-as-code mentality.
That means instead of clicking around in the AWS Web Console, we’d define and manage our infrastructure with in a declarative fashion.
It also meant we could use our normal code review flows and policies.
The idea of being able to define reusable modules was also very attractive.
After evaluating the options, we went with Terraform.
I’ll be candid with you, this was a tough change for us. It was and still is difficult to pull existing infrastructure into Terraform.
Is anyone here fighting with this sort of thing right now?
Added to that, we didn’t go into this with a set of commonly accepted best practices in hand.
Discussions with other users made it evident that everyone was doing things slightly differently at the time.
So we started chipping away as best as we could. We made tons of mistakes, battled with HCL at times, and even accidentally exploded a few things.
But we got better over time, and today consider Terraform a key piece of our stack.
Once more, with feeling!
Which brings us to late 2016.
We’ve got a decent testing story, a service framework, and we’re starting to manage infrastructure with code.
It was time to start pumping these new services out as more teams and engineers flowed into the company.
< Hit Next >
But as we deployed our third and fourth services, our lack of a good staging story started to hurt.
Growing pains: Staging/integration woes
The issue here was that we didn’t have a great way to develop and test new services against one another.
The monolith had a very rigid, opinionated staging system that had done the job for many years but it wasn’t something that we wanted to stamp out for our services.
And this Kubernetes thing had really started to take off at the time, and we also had some folks that had previous exposure to it.
We were also struggling to manage our background jobs at the time, and saw Kubernetes as potentially relevant there.
So we resolved to see what kind of staging system we could come up with in the span of about three weeks.
By this point in late 2016, Helm was still very young and many (or most of us) were managing clusters with bash scripts and static manifests.
Which is exactly what we did. We wrote a CLI that wrapped kubectl to deploy Docker images that our CI system built.
By the third week of our effort, developers were using the new staging system.
It ended up being very well-received, and we resolved to continue developing it.
While our staging system looks much different today, it was wonderful to be able to address a pressing need so quickly!
Reddit Engineering in 2017
As 2017 rolls around, we have a growing tapestry of services, we’ve got over 60 engineers, and have quite a few different teams writing code.
Patterns were beginning to form in Terraform land, baseplate continues to evolve, and we were settling into this brave new service-oriented world.
But we on the Infrastructure team had reached a threshold.
Growing pains: Infra team as a bottleneck
As mentioned earlier, the Infrastructure team had historically been the arbiter of production.
It had served us well for a decade, but the rest of the organization had recently grown far faster than us.
We had became a bottleneck. Service owners had to wait for us to stand up and configure new infrastructure for them.
Also, we handled most of the incident response and production debugging since we had the most permissive access.
This led to frustration all around:
The Infrastructure team was discouraged by the hefty backlog of teams waiting on us.
The other teams were [rightly] not pleased with having to wait long periods of time for their services to be stood up.
Seeking some short-term relief, we trained a few of the more infrastructure-oriented teams to work with our our systems and operations tooling.
These teams were able to block less on Infra, while reusing some of our Terraform and Puppet building blocks.
This helped them move faster and took some load off of us.
Though, it didn’t come close to fully solving our Infra-as-a-bottleneck issue.
One size fits some
The reality is that not all teams want to operate the full stack for their service.
There’s an entire spectrum of infra comfort to contend with.
When faced with the prospect of doing some infrastructure work, there are some teams will chortle with glee and tell you about the time they did a stage 1 Gentoo install on their refrigerator.
Others will will get up, scream loudly, and run away, never to be seen again.
You want me to do WHAT?!?
I’d argue that at most organizations, it’s not reasonable to expect deep infrastructure experience on every team.
I think we are at our best when our development teams can focus more directly on the problems that they have been tasked to solve, instead of doing that plus wading into the deep, murky waters of infrastructure.
And to be honest, our stack is built from some pretty ho-hum stuff for someone who spends 40 hours a week in operations land.
But this is a lot to bite off for others who don’t have that much contact with this side of things.
What do we really really want?
But let’s step back and look at these issues from a higher level perspective:
Our fellow engineering teams are:
frustrated by having to wait on us
feeling like they aren’t in control of their service’s lifecycle
unable to debug some issues in production themselves
If we had to distill all of this down into a very specific request, they are desiring...
Service Ownership!
But what does that mean?
We’ve established that not every team wants to manage its own infrastructure.
But most of our teams do want to own the full lifecycle for their service.
They want to be empowered to jump into development quickly, then graduate to a production-like test environment where the finishing touches can be applied.
After that, they want to take their service to production without much dependence on other teams.
Once in production, they want to be able to instrument, monitor, and iterate on their service, again, without blockage.
When problems arise, they want to be able to start by trying to diagnose the issues on their own.
These are all very reasonable expectations, and it is mutually beneficial for all of us to figure out how more responsibility can lie with the service owner.
But there are a few challenges to overcome...
Service ownership challenges: Learning curve
No matter how smoothly paved or neatly groomed a path we present for our service owners, the days of throwing code over the wall for an operations team are over.
They may not need to get a deep understanding of their infrastructure, but they’ll need to learn some basics about whatever contact surfaces we present to them.
Service ownership challenges: Responsibility
Depending on the team, the response to this newfound notion of service ownership will range from “this is awesome” to “can you just do it for me?”
In our case, we are there to help, but do expect the service owner to make an attempt to work towards more self-sufficiency.
And nine times out of ten, the service owner will see the value in being able to accomplish more and faster.
Buuuut...
Service ownership challenges: Mistakes are going to happen… often!
There will be explosions. Probably at least three. Maybe even five.
But instead of getting caught up on what could happen, we can look at these happy little accidents as the learning opportunities that they are.
And since the service owner is the first to get paged, they are highly motivated to get things sorted!
Early 2018 - How to enable service ownership?
We’ve spent time with the other engineering teams trying to get a feel for what they want.
We’ve worked with them to define a notion of service ownership.
We’ve also established that it’s not reasonable for most teams to become Infrastructure engineers in addition to their primary responsibilities.
So how can we limit the learning curve while empowering teams?
Reddit Infrastructure as a Product
We have to change our thinking a bit.
Instead of expecting each team to moonlight as operators and sys admins, we needed to put together a cohesive Infrastructure Product.
A Reddit Engineer should be able to take this thing off of the shelf, read and follow the instructions, and have their service running without blockage or significant fuss.
Selecting the right foundation
So we started looking at ways to re-package and more humanely offer our tools and processes.
There are even companies out there with a similar stack that managed just that very feat.
But ultimately, we felt like there may be a better way to go.
Perhaps we could continue using most of what we already had in Infrastructure land, but present some more singular “API” or contract to the users.
You’ll recall that I had mentioned a Kubernetes-based staging system earlier on, all the way back in late 2016.
We had, at this point, over a year of success in using Kubernetes for this and a few other miscellaneous purposes.
It had served us well, though we didn’t have a well-defined idea of how it would best serve us longer term.
To move in the direction of Kubernetes would mean a huge paradigm shift in how we developed, deployed, and operated our services.
This was spooky to us, in that we’d be forced to take our bumps and bruises all over again with a new stack, when we had one that was working for us.
< Hit Next >
But as we sketched and experimented, we came to believe that this was a great foundation for us to build on.
We felt that it’d be the singular, cohesive platform from which we could build our product.
So we steeled our nerves and started on the journey to enabling service ownership for all.
How to build an Infrastructure Product
But how do we actually build an Infrastructure Product?
It’s not enough to toss Kubernetes at the other engineering teams, wish them well, then run away.
Our goal of offering a cohesive product means that we need to build for ease of use, consistency, the right level of flexibility, and safety.
In thinking through our problems and our ideal future, we arrived at a set of core tenets that have directed our development.
Let’s walk through what we have and how we’ve satisfied each.
Limit the surface area
Our first core tenet is the idea that we need to limit the total technical surface area that our service owners will be expected to interact with.
In getting back to this notion of a packaged Product, we have to get away from expecting users to “build something from these hundred pieces”. Instead, we need to say “here’s one or two things you can use to develop and operate your service”.
By limiting the set of distinct technologies that the user needs to touch, we lessen context switching and lower the learning curve.
So instead of requiring knowledge of everything you see before you…
< Hit Next >
We decided to focus our users on Kubernetes as that single contact point.
And by “Kubernetes”, we mean the barest of essentials: Pods and Services.
We support this with plenty of tooling, training, and documentation.
By not having to learn our entire systems stack, users are able to focus more on their services.
But this kind of philosophy is only so valuable unless we make responsibilities and expectations explicit.
The user/operator contract
To do that, we defined a “contract” that codifies the user/operator relationship.
We define the “product user” as the service owners, and the “product operator” as the Infrastructure team.
Product users are expected to learn the barest of basics about Kubernetes, and they’re expected to develop, deploy, and operate their own services.
By extension, incidents in production involving a service, result in the owner getting paged.
Product operators are responsible for provisioning, maintaining, and scaling the underlying Kubernetes clusters.
If there are incidents with Kubernetes Clusters or supporting infrastructure, the Infrastructure team gets paged.
If a service requires an S3 bucket, a cache, or a database, the Product operators are well-equipped to provision or operate those.
In determining a dividing line for responsibilities, we allow each party to focus on the thing that they are best equipped to handle.
Batteries-included
For our user and operator contract to work, we have to provide as much as we can out-of-the-box.
This goes back to focusing each party on the tasks that they are best equipped to handle.
For example: by providing starter test, build, and deploy pipelines, we avoid the need for our users to become Release Engineers.
By baking in a set of default metrics, dashboards, and alerts, we provide an immediately observable foundation for the service owner to start with.
The same holds true for logging. This isn’t something our users should have to wire up themselves.
If a service interacts with another service, the owner shouldn’t have to learn the intricacies of our service mesh.
If secrets or credentials are involved, our user should be able to quickly and easily set those up in our secrets store.
We are able to include all of these things out of the box due to the consistency baseplate gives us. We’re able to make a bunch of assumptions about how a Reddit service operates, and can make our tooling more simple as a result.
But getting back to the main takeaway in all of this:
Your users should get all of these things just by virtue of standing their services up using your product.
If they have to rig up all of these things themselves, you don’t have an infrastructure product. You have a pile of technology.
Paint-by-numbers
As we discussed earlier, teams differ in their ability and willingness to handle infrastructure work.
Some are all about it, others want nothing to do with it.
Though, we have to be able to support both sides of this spectrum.
That means not requiring a deep understanding of the infrastructure.
Instead of expecting users to take their lumps by slogging through upstream documentation, poring over blog posts, and rooting around in forums, we provide a library of paint-by-numbers guides to cover common tasks and problems.
For example, “here’s how to set and retrieve secrets”. “Here’s how to have your application assume a specific IAM role”. “Here’s how to deploy an experimental release that handles a fraction of production traffic”.
We supplement these guides with training and extensive documentation.
If you don’t have excellent docs, training, and support, you do not have an infrastructure product!
We also must auto-generate or template everything that we can for the users.
For example, instead of asking your users to write Kubernetes manifests or Helm Charts, we can generate something that will get them most of the way there, save for some minor adjustments.
In summary, let’s read the last sentence on this slide together. OK? Ready?
Consistency throughout
Ah, look at this. More consistency!
Supporting the previous three tenets, we have to provide consistency throughout the development cycle.
For us, this means Kubernetes in local dev, Kubernetes in our lab and staging environments, and Kubernetes in production.
As far as how this works, we reuse the same Helm Charts and Docker images through all three phases of the lifecycle.
If you’d like to learn more about how this works, see our 2018 Helm Summit presentation on this very topic.
The tldr is that by pairing our Helm Charts with phase-specific values, we can account for the few differences there are at each step along the way.
The result of this is that:
Our developers don’t need to learn to use three different systems.
They build familiarity and competency with our offering even in local development.
When it’s time to go to production, they have a decent understanding of the basics.
And finally, we prevent many cases where something works locally but not in production.
Deploy with some confidence
Everything that we have discussed thus far must be offered with tools and processes that help our users develop and deploy with confidence. Otherwise we’re just throwing crap over the wall, hoping that it won’t make a mess.
For example, we have an org-wide code review policy. We require at least one other person review all commits and designs.
It can be frustrating to have to factor in additional time for review, but you’ll be collaboratively improving the quality of your contributions.
This is also a great way to learn from others.
In addition to code review, we need a robust unit and integration testing story.
We need to be able to test more isolated chunks of code...
And we should also be able to test larger systems (or even services) against one another.
To tie this all together, by the time our changes are in line to deploy to production:
They will have been ran through linters, automated unit tests, and integration tests
They will have been reviewed by at least one other colleague
They will have been deployed to a staging environment
These tools and processes give us more confidence that our deploy will be a success.
Guardrails and Safeties
With all of that said: despite our best efforts, mistakes will happen in production.
When the worse inevitably happens, it’s important to have Guardrails and safeties: things that will limit the damage.
For example, resource limits on CPU and memory can reduce the likelihood of resource starvation.
On the network side, a service mesh can be used to set service-to-service rate limits.
We’ll want to tightly scope IAM and RBAC permissions on applications and users to avoid compromise or unexpected fires.
And by virtue of running all of this on Kubernetes, we end up with a platform that makes it easy to use the API to scan for common mistakes.
For example, if someone somehow gets through our CI checks and fires up a Service of type LoadBalancer that hasn’t been approved, perhaps we alert our Security team.
Going a step further, Docker Image policies can whitelist or blacklist the acceptable images or image repositories that can be used with your clusters.
Having these guardrails and safeties is a critical part of being able to trust the integrity of our offering.
Reddit Infrastructure Product Core Tenets
We’ve covered a lot of ground, so let’s review before I end with some parting words of wisdom:
Limit the surface area
The user/operator contract
Batteries-included
Paint-by-numbers
Consistency throughout
Deploy with confidence
Guardrails and Safeties
What does all of this buy us?
And this all sounds cool, right? But what change did all of our work bring about at Reddit?
Infra team: From Operators to Enablers
Our Infrastructure Product has helped us… transform.
Let’s compare and contrast before and after.
Infra team: From Operators to Enablers
Before: Infra team provisions all infrastructure
After: Infra provides infra as a product
Before: Infra deploys new services
After: Service owners deploy new services
Before: Infra operates most services
After: Service owners operate their services
Before: Infra is a blocking dependency on production work
After: Infra is an adviser and enabler for other teams to do production work
Organizational scalability
Reddit has seen rapid growth over the last few years, and we’re likely to see more in the future.
By building a coherent infrastructure product, we’ve packaged and distributed the Infrastructure team’s expertise for the other engineering teams to use.
In doing so, we’re scaling ourselves enough to be able to handle the next few hundred engineers to come.
Closing remarks
I presented quite a bit of higher level content in this talk.
I’m going to hang around in here for a bit, or out in the hall of they kick us out.
Please feel free to stop by and ask questions.
Also feel free to contact me on Reddit as /u/gctaylor, or on Twitter as the same name.
Jobs plug
Also, I’d be remiss if I didn’t mention that we’re hiring across all sorts of teams and disciplines.
Speaker info and resources
Here are my details, along with some resources.
I’ll take a moment to plug the kubernetes subreddit, which is approaching 10k subscribers.
You can also see some other interesting content on the technology section of our blog at redditblog.com.