Everything as code

TECHNOLOGY
About Us
SERHAN GÜNEY
● Full-Stack Developer with 8 years of experience
● Currently working in Hepsiburada Marketplace
team as Senior Full-Stack Developer
https://www.linkedin.com/in/serhanguney/
serhangny@gmail.com
https://gitlab.com/serhan
ERDEM ERBAŞ
● Full-Stack developer with 2 years of experience
● Currently working in Hepsiburada Marketplace
team as Full-Stack Developer
https://www.linkedin.com/in/erdemerbas/
erdemerbas12699@gmail.com

TECHNOLOGY
Disclaimer:
In this presentation, We are going to try
to give you an overall picture.
E

TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ Test Driven Everything
EE

TECHNOLOGY
Agenda
❖ Introduction <= You are here now
➢ Why Everything As Code
➢ Automation
➢ Gitops
➢ Code Review
➢ Testing
➢ Immutability
➢ Disposability
❖ ADRs
E

TECHNOLOGY
The idea behind the Everything as Code
concept is that infrastructure codified in a
declarative specification and treated like
application code such that they follow the same
software development lifecycle practices.
E

TECHNOLOGY Why Everything As Code
❖ Automation
❖ GitOps
❖ Code Review
❖ Testing
❖ Immutability
❖ Disposability
E

TECHNOLOGY Automation
❖ Higher production rates
❖ Increased productivity
❖ More efficient use of materials
❖ Better product quality
E

TECHNOLOGY GitOps
❖ Git as the source of truth
❖ Versioned CI/CD on top of declarative infrastructure
❖ Immutable deployment
❖ Stable and reproducible rollbacks
E

TECHNOLOGY Code Review
❖ Coding standards compliance
❖ Team cohesion
❖ Teaching and sharing knowledge
❖ Higher code quality
E

TECHNOLOGY Testing
❖ Testing verifies that the system meets the different requirements including,
functional, performance, reliability, security, usability and so on.
❖ Testing validates that the system being developed is what the user needs
E

TECHNOLOGY Immutability
❖ Build components to an exact set of specifications
❖ If a change to a specification is required, then a whole new set is
provisioned based on the updated requirements, and the previous is taken
out of service
❖ It is always possible to go back to the desired state
E

TECHNOLOGY Disposability
❖ Don’t Fix it, Throw it Away!
❖ It can be compared and contrasted with an application's disposability, one
of the characteristics of the 12 factor app
E

TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning <= You are here now
➢ What is workstation provisioning
➢ Onboarding
➢ Windows
➢ Linux
➢ Benefits
❖ ADRs
S

TECHNOLOGY Onboarding
❖ Preconfigured workstation
❖ Instead of following a guideline just press start
❖ Configuration drift
❖ Time effective
❖ Before/After
S

TECHNOLOGY
Workstation Provisioning
❖Windows
➢ Puppet
➢ Choco
➢ IDE
➢ Tools
E

TECHNOLOGY
Puppet
➢ Puppet is a desired state based configuration management automation
tool.
➢ Includes its own declarative language to describe system configuration
➢ Puppet supports all powershell commands(Powershell DSC)
E

TECHNOLOGY
Windows Provisioning
❖ Package
➢ Use chocolatey for default
package provider
➢ Taking advantage of powershell
desired state configuration
➢ Use specific versions(to avoid
breaking changes)
E

TECHNOLOGY
Windows Provisioning
❖ Configuration
➢ Configure installed
packages
➢ Every developer has
the same config
➢ Provides
immutability
E

TECHNOLOGY
Workstation Provisioning
❖Linux
➢ Image Bakery
➢ Vagrant
➢ Multiple Boxes
➢ Image Bakery vs CM
S

TECHNOLOGY
Image Bakery
❖ Developer Box
➢ Specialized ubuntu images are created using
Packer
➢ Minimized image
■ https://github.com/chef/bento
S

TECHNOLOGY
Vagrant
❖Vagrant orchestrates developer box provisioning
❖Virtualbox is the preferred provider
❖Ansible is the preferred provisioner
S

TECHNOLOGY
Ansible
❖Like Puppet, Ansible is another CM tool
❖Agent-less architecture
➢ everything is done by using SSH
S

TECHNOLOGY
ShowCase
❖ Image Bakery
❖ CM
➢ Vagrant
➢ Ansible
S

TECHNOLOGY
Multiple Boxes
❖ Differences of boxes are as it is in the image
➢ Seperate box for each business vertical
❖ In Common they all have
➢ required docker containers to ease local testing and debugging
➢ Git config to specify user
➢ ssh config to ease connection to build,qa and prod envs
E

TECHNOLOGY
Image Bakery vs CM
❖ Immutables like ansible should be in image bakery.
❖ Mutables like docker version should be configured by CM
E

TECHNOLOGY
Benefits
❖ Onboarding
❖ Dev/Prod parity
➢ Keep development, staging, and production as
similar as possible
❖ Testable infrastructure
E

TECHNOLOGY
Agenda
❖ Introduction
❖ Infrastructure As Code <= You are here now
➢ Build Infra
➢ Environment Infra
❖ Pipelines As Code
❖ ADRs
❖ TDE
E

TECHNOLOGY
Infrastructure as Code
❖ Both On Premise
❖ Everything is a Docker Container
❖ Nginx as reverse-proxy
❖ No app-specific published ports
➢ Internal DNS
➢ Taking advantage of docker swarm mode SDN
E

TECHNOLOGY
Build - Environment Infra
❖ Separation of Concerns
❖ Build generates artifacts - Environment runs artifacts
❖ Reduces complexity
❖ Different networks and docker swarms
➢ Build swarm,
➢ Environment swarms(eg. qa, prod)
S

TECHNOLOGY
Agenda
➢ Build Infra <= You are here now
■ Bootstrap
■ Gitlab
■ GoCD
■ Docker Registry (Harbour)
■ Nexus
➢ Environment Infra
❖ ADRs
❖ TDE
S

TECHNOLOGY
Build Infra
❖ Where everything gets built
❖ Artifacts and Images are pushed to appropriate registries
➢ eg. docker registry, nexus
❖ Automated provisioning of build infra via bootstrap.sh
➢ cloud or on premise
❖ Uses Docker swarm mode for orchestration
S

TECHNOLOGY
Bootstrap
❖ Bootstrapping is only necessary when the build infrastructure is
provisioned for the first time
❖ Bootstrapping will install barebone build infra services
➢ Gitlab(with groups and project)
➢ GoCD
➢ Docker Registry , etc
❖ Just invoke the `bootstrap.sh` and chill
❖ Can be disposible `dispose.sh` and cry
S

TECHNOLOGY
Bootstrap
❖ bootstrap.sh creates base-images required during Bootstrap
➢ eg. alpine, curl, golang
❖ Solves dependencies with health checks
❖ Applicable even to devbox
➢ for testing purposes
❖ Two different configurations
➢ One for barebone bootstrap image
➢ Specialized one for the environment
S

TECHNOLOGY
Gitlab
❖ Our preferred VCS
❖ Holds all our code and configuration
❖ Initialization steps are done by bootstrapper
❖ Configuration
E

TECHNOLOGY
Gitlab
❖ Showcase
➢ Gitlab omnibus conf
➢ user authorization
➢ slack integration
E

TECHNOLOGY
GoCD
❖ Our preferred CI/CD tool
❖ Pull based
❖ Agent based
➢ Capability based task scheduling
➢ Only docker capability(Only one agent to rule them all)
S

TECHNOLOGY
GoCD
❖ Base Build Infra pipelines are created by bootstrapper
❖ Updates itself via docker volume
❖ Slack Integration
❖ Pipeline as Code
❖ User Authentication
S

TECHNOLOGY
Docker Registry
❖ Holds all Docker images
❖ Initially used private docker registry
❖ Recently Harbor is used as container registry
➢ Registry authorization
E

TECHNOLOGY
Nexus
❖ Stores component type artifacts
❖ Keeping it internal helps you overcome external network issues
❖ Vagrant Box
➢ Raw repository
➢ Index.json (hyperlink)
❖ Apt package plug-in
E

TECHNOLOGY
Agenda
❖ Introduction
➢ Build Infra
➢ Environment Infra <= You are here now
■ Nginx
■ Consul
■ ElasticSearch
■ RabbitMQ
❖ ADRs
❖ TDE
E

TECHNOLOGY
Environment Infra
❖ Where Build Infra artifacts get to run
❖ Artifacts and Images are pulled from appropriate registries
➢ eg. docker registry, nexus
❖ Build once, run anywhere
➢ Configures image artifacts wrt environments
❖ Uses swarm as container scheduler
➢ 260 containers in total
➢ Will be replaced by k8s w/ istio
❖ Deployments are managed by constraints
E

TECHNOLOGY
Nginx
❖ High-performance HTTP server and reverse proxy
❖ Deploys on all nodes
❖ Creates all external overlay network definitions on deployment
❖ Configured via nginx.conf
➢ regexp to match wildcards
S

TECHNOLOGY
❖ SHOWCASE
➢ nginx.conf
➢ wildcards
➢ docker file
S

TECHNOLOGY
Consul
❖ Tool for service discovery and configuration
❖ Clustered
❖ Used for dynamic service configuration data and feature toggles
❖ Service discovery as RabbitMQ cluster peer discovery mechanisms
➢ RabbitMQ Cluster with HAProxy and Consul @Hepsiburada Tech
Blog
S

TECHNOLOGY
ElasticSearch
❖ Open-source search engine built on top of Lucene, a full-text search-
engine library.
❖ You know, for Search…
E

TECHNOLOGY
RabbitMQ
❖ The most widely deployed open source message broker.
❖ Uses consul based rabbitmq clustering
❖ Migrating RabbitMQ in a high traffic setup @Hepsiburada Tech Blog
❖ Manages plugins via enabled plugin configuration
❖ Can be configured via env variables or rabbitmq.conf
E

TECHNOLOGY
Environment Infra Also Includes
❖ Redis HA
❖ MongoDB
❖ Varnish
❖ ELK
❖ Prometheus
❖ Grafana
❖ Alert Manager
❖ Exporters
❖ Jaeger
Observability as Code
E

TECHNOLOGY
Agenda
❖ Introduction
❖ Infrastructure as Code
❖ Pipelines as Code <= You are here now
➢ Definition of PaC
➢ Template Oriented Pipelines
➢ Examples
➢ Configuration
❖ ADRs
❖ TDE
S

TECHNOLOGY
Pipelines As Code
❖ Pipeline config is stored in VCS either with your application code or in
a separate repository
❖ Pipeline config repos are referenced at GoCD
❖ Application repos may contain .gocd.yml file
S

TECHNOLOGY
Template-based Pipelines
❖ Templates are stored at GoCD
❖ Docker eases templating
➢ same templates for all EaC,
➢ e.g. Golang API, React App, .Net Core API, ...
❖ Enables DSL-like pipeline definitions
➢ build-and-push,
➢ retag-and-push,
➢ auto-deploy,
➢ manual-deploy,
➢ deploy-prod, …
S

TECHNOLOGY
Template-based Pipelines
S

TECHNOLOGY
Showcase
❖ .gocd.yml
E

TECHNOLOGY
Agenda
❖ ADRs
❖ Observability As Code <= You are here now
➢ ELK
➢ Grafana
➢ Prometheus
➢ Tracing
❖ ADRs
❖ TDE
S

TECHNOLOGY
Observability As Code
❖ Where Build and Environment Infra are observed
❖ Resides with Environment Infra Code
❖ Three pillars of observability
➢ Logging
➢ Metrics
➢ Tracing
❖ Alerts
S

TECHNOLOGY
Logging
❖ ELK
❖ Logs shipped to ELK
➢ App logging
➢ Infra logging
❖ Watcher
❖ GELF for dockerized apps
❖ FileBeats for legacy apps
❖ Logstash pipeline
➢ grok
➢ gelf
S

TECHNOLOGY
ELK
❖ ElasticSearch
➢ Curator
➢ Watcher
➢ Slack Integration
❖ Logstash
❖ Kibana
E

TECHNOLOGY
Watcher
❖ Cluster Health alerts
❖ Application alerts
E

TECHNOLOGY
ELK
❖ ShowCase
➢ Watcher alerts
➢ Healthchecks
➢ Curator
E

TECHNOLOGY
Metrics
❖ Prometheus
❖ Exporters
❖ Alerts
❖ Monitors
S

TECHNOLOGY
Metric Monitoring Stack
S

TECHNOLOGY
Prometheus
❖ Holds all infra and app metrics
❖ Uses metrics exposed via exporters
❖ Constraints and docker volume utilized to persist data
❖ Generic and Environment based rules
❖ Slack Integration
➢ Alert Manager
S

TECHNOLOGY
Prometheus
❖ ShowCase
➢ Scrape conf
➢ Prometheus Rules
➢ Alert Manager
E

TECHNOLOGY
Exporters
❖ Export metrics in prometheus style
❖ Exporters used:
➢ Node Exporter
➢ cAdvisor
➢ Rabbitmq Exporter
➢ Consul Exporter
➢ Varnish Exporter
➢ Redis Exporter
E

TECHNOLOGY
Grafana
❖ Visualize prometheus data
❖ Provisioning
➢ Datasource
➢ Dashboard
➢ Env based provisioning
➢ V4 vs V5
E

TECHNOLOGY
Tracing
❖ Jaeger
❖ Open tracing
E

TECHNOLOGY
Agenda
❖ ADRs <= You are here now
❖ TDE
E

TECHNOLOGY
ADRs
❖ An architectural decision (AD) is a software design choice that addresses
a significant requirement.
❖ MarkDown syntax is used for documents.
❖ ADR cli utilities
E

TECHNOLOGY
Agenda
❖ ADRs
❖ TDE <= You are here now
S

TECHNOLOGY
Test Driven Everything
❖ Since Everything is code
➢ Everything should be tested
S

TECHNOLOGY
➢ Windows
■ ServerSpec
➢ Linux
■ Goss
❖ IaC
➢ Varnish
❖ Test Suite
➢ Ginkgo/Gomega
➢ Mountebank
■ Gobank - https://github.com/durmaze/gobank
S

TECHNOLOGY
❖ ShowCase
➢ goss
➢ serverspec
S

TECHNOLOGY
Future Roadmap
❖ Machine Provisioning
➢ Packer based images used in build/env infra
➢ SysAdmins manually provisions using this image
➢ Will be replaced by Terraform
❖ K8s w/ Istio migration
➢ Will be done after migrating the remaining legacy apps
❖ Secret Management

Everything as code

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Everything as code

Similar to Everything as code (20)

Recently uploaded

Recently uploaded (20)

Everything as code