SlideShare a Scribd company logo
DevOps Braga #12:
Infrastructural challenges of a
fast-pace startup
Gustavo Balbino | Rui Matos - DevOps Team
Agenda
About Us
Storage Issues
Rethinking our Infrastructure
Load balancing Web Applications
Containers
Primeira Gráfica Online Portuguesa, líderes Ibéricos e no Brasil
• Founded in 2013
• 200 Employees
• Offices in 3 cities
• Braga
• Torres Vedras
• Lisboa
• Looking for +150 employees by the end 2019
• 21 countries
North America
Brazil
Europe
*GB
IE
FR
DE
IT
PL
CZ
NL
AT
SE
DK
BE
CH
NO
FI
3 Stores
Brazil MexicoIberia
Each Store is (almost) independent in terms of infrastructure, virtual machines, database, etc…
2019 Market Expansion
Iberia Europe
Mexico North America
100% Cloud - Microsoft
Azure
Some numbers
40
VMs
19
DBs
90
Disks
Production Resources
Storage Issues
2 Big Size VMs >= D8s v3
Azure Load Balancer
Windows VMs running IIS, SOFS e S2D
Scalable but at what cost?
Production Environment MX Aug/2018
Windows Server 2012 R2 Scale-Out File Server
• A set of clustered file servers that make
up a transparent failover file server
cluster
• SOFS is a fantastic way to deploy small
and affordable clustered storage
• The Concept of Storage Spaces
Direct (S2D)
• Fault Tolerance
• Performance
• Scaling Out and In
Rethinking our infrastructure
One role, one machine
HAProxy Load Balancer
Smaller size VMs
Horizontal & vertical scalability
Divide to conquer
2 Smaller Virtual Machines
Faster Asynchronous replication every 30 seconds
Dedicated NICs for Replication
Scalable
SMB File Server with Storage Replica
Load Balancing Web Applications
Azure Load Balancer
Load balancing methods
What is HAProxy
• HAProxy (High Availability Proxy) , is a popular open source software TCP/HTTP
Load Balancer and proxying solution which can be run on Linux, Solaris and
FreeBSD.
• Its most common use is to improve the performance and reliability of a server
environment by distributing the workload across multiple servers (Web,
applications, databases).
• Some happy users;
 GitHub
 Reddit
 Instagram
 Stack Overflow
 Twitter
 Tumblr
 Vimeo
 YouPorn
Why HAProxy?
• Absolutely free and and widely supported
• Supports SPDY (obsolet) & HTTP/2
• Supports Traffic Encryption & SSL Termination
• Let's Encrypt ACME client integration
• Access to Integrated Server Monitoring Dashboard
• Several Load Balancing Algorithms/Configurations
• Easy to form a Active/Passive cluster configuration
• “Runtime API” for automation/integration
• Personalized error page
Active/Passive Health Check in a private
cloud
How to implement it in Azure?
PUBLIC IP ADDRESS AZURE LOAD BALANCER 2 VIRTUAL MACHINES
HAProxy & Keepalived Installation
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get install -y software-properties-common
sudo apt-get install -y haproxy keepalived
sudo apt-get install socat
Inbound NAT rules LB Azure
Inbound Custom port NAT for SSH - because you won’t have an addressable IP Address to operate your machines
Building an Active/Passive HAProxy in
Azure
Set during LB creation
Where you’ll declare HAProxy VM NIC
Simple check over port 80
HTTP & HTTPS
Before Keepalived MagicEnabling non local Virtual IP binding
sudo nano /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind=1
fs.file-max = 10000000
fs.nr_open = 10000000
net.ipv4.tcp_mem = 786432 1697152 1945728
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216
net.ipv4.ip_local_port_range = 1024 65000
sudo sysctl -p
Keepalived configuration MASTERglobal_defs {global_defs {
enable_script_security
script_user root
}
enable_script_security
script_user root
}
/etc/keepalived/keepalived.conf
vrrp_script chk_appsvc {
script /usr/local/sbin/keepalived-check.sh
interval 1
fall 2
rise 2
}
vrrp_instance VIP_1 {
advert_int 1
interface eth0
authentication {
auth_type PASS
auth_pass S0meTrickP4ss0rdY0uCantR3m3mb3R
}
virtual_router_id 51
virtual_ipaddress {
111.222.333.444
}
track_script {
chk_appsvc
}
notify /usr/local/sbin/keepalived-action.sh
notify_stop "/usr/local/sbin/keepalived-action.sh INSTANCE VIP_1 STOP"
state MASTER
priority 101
unicast_src_ip 10.255.1.51
unicast_peer {
10.255.1.52
}
}
Keepalived configuration SLAVEglobal_defs {global_defs {
enable_script_security
script_user root
}
enable_script_security
script_user root
}
/etc/keepalived/keepalived.conf
vrrp_script chk_appsvc {
script /usr/local/sbin/keepalived-check.sh
interval 1
fall 2
rise 2
}
vrrp_instance VIP_1 {
advert_int 1
interface eth0
authentication {
auth_type PASS
auth_pass S0meTrickP4ss0rdY0uCantR3m3mb3R
}
virtual_router_id 51
virtual_ipaddress {
111.222.333.444
}
track_script {
chk_appsvc
}
notify /usr/local/sbin/keepalived-action.sh
notify_stop "/usr/local/sbin/keepalived-action.sh INSTANCE VIP_1 STOP"
state SLAVE
priority 100
unicast_src_ip 10.255.1.52
unicast_peer {
10.255.1.51
}
}
Keepalived check script
#!/bin/bash
URL="http://localhost"
if [[ `curl -s -o/dev/null --connect-timeout 0.5 $URL; echo $?` -ne 0 ]]; then
exit 1
else
exit 0
fi
/usr/local/sbin/keepalived-check.sh
#!/bin/bash
TYPE=$1
NAME=$2
STATE=$3
modify_probe_status_http() {
STATUS=$1
LB_PROBE_PORT=80
LB_PROBE_DEV=eth0
if [[ $STATUS == "down" ]]; then
# Add firewall rule to block LB probe port
/sbin/iptables -A INPUT -p tcp --dport $LB_PROBE_PORT -j REJECT -i $LB_PROBE_DEV
elif [[ $STATUS == "up" ]]; then
# Remove all entries to block LB probe port
RC=0
while [[ $RC -eq 0 ]]; do
RC=`/sbin/iptables -D INPUT -p tcp --dport $LB_PROBE_PORT -j REJECT -i $LB_PROBE_DEV 2>/dev/null; echo $?`
done
else
echo "Unknown probe status"
fi
}
if [[ "$NAME" == "VIP_1" ]]; then
case $STATE in
"MASTER") modify_probe_status_http up
exit 0
;;
"BACKUP"|"STOP") modify_probe_status_http down
exit 0
;;
"FAULT") modify_probe_status_http down
exit 0
;;
*) echo "unknown state"
exit 1
;;
esac
else
echo "Nothing to do"
exit 0
fi
HAPROXY is DOWN
/sbin/iptables -A INPUT -p tcp --dport 80 -j REJECT -i eth0
HAPROXY is UP
/sbin/iptables -D INPUT -p tcp --dport 80 -j REJECT -i eth0
Keepalived action script
/usr/local/sbin/keepalived-action.sh
Keepalived in Action/Demo
Master HAProxy Slave HAProxy
Because Azure LB can reach port 80, IP is delivered to VM
In what ways can HAProxy help you?
• Protect you from DDoS;
• Prevent SQL Injection and abnormal user behavior;
• Segregate traffic, by URL, domain, user agent, IP ranges and many, many more;
• URL Redirect and request rewrites;
• Cache server for small objects;
• Centralized logging
• Compression
Containers
Production Microservices
Linux hosts
Docker containers – 4 already!
Scale ready infrastructure
One infrastructure per region
Deployment infrastructure
Where we are
Team City is our CI – It builds the software
Octopus is hosted as a tentacle on each machine, and deploys the software
A total of 50 steps on the build process and 66 on the deployment phase
The complete deployment process could take about 2 hours right now
New pipeline – Microservices oriented
Jenkins is just an orchestrator
Docker pipeline will build the container image and push to the registry
Ansible will deploy the container in the Linux hosts via SSH
New pipeline – Microservices oriented
A real DevOps environment
Run by the Teams!
Autonomous CI/CD environment
Q & A
gustavo.balbino@360imprimir.pt | https://www.linkedin.com/in/gustavo-lima-969890a9/
rui.matos@360imprimir.pt | https://www.linkedin.com/in/rui-matos/

More Related Content

What's hot

Trevor McDonald - Nagios XI Under The Hood
Trevor McDonald  - Nagios XI Under The HoodTrevor McDonald  - Nagios XI Under The Hood
Trevor McDonald - Nagios XI Under The Hood
Nagios
 
Prometheus: infrastructure and application monitoring in kubernetes cluster
Prometheus: infrastructure and application monitoring in kubernetes clusterPrometheus: infrastructure and application monitoring in kubernetes cluster
Prometheus: infrastructure and application monitoring in kubernetes cluster
Lohika_Odessa_TechTalks
 
Extending functionality in nginx, with modules!
Extending functionality in nginx, with modules!Extending functionality in nginx, with modules!
Extending functionality in nginx, with modules!
Trygve Vea
 
Architectural caching patterns for kubernetes
Architectural caching patterns for kubernetesArchitectural caching patterns for kubernetes
Architectural caching patterns for kubernetes
Rafał Leszko
 
NGINX: Basics & Best Practices - EMEA Broadcast
NGINX: Basics & Best Practices - EMEA BroadcastNGINX: Basics & Best Practices - EMEA Broadcast
NGINX: Basics & Best Practices - EMEA Broadcast
NGINX, Inc.
 
High Availability Content Caching with NGINX
High Availability Content Caching with NGINXHigh Availability Content Caching with NGINX
High Availability Content Caching with NGINX
NGINX, Inc.
 
Nginx - Tips and Tricks.
Nginx - Tips and Tricks.Nginx - Tips and Tricks.
Nginx - Tips and Tricks.Harish S
 
Pragmatic Monolith-First, easy to decompose, clean architecture
Pragmatic Monolith-First, easy to decompose, clean architecturePragmatic Monolith-First, easy to decompose, clean architecture
Pragmatic Monolith-First, easy to decompose, clean architecture
Piotr Pelczar
 
Jesse Olson - Nagios Log Server Architecture Overview
Jesse Olson - Nagios Log Server Architecture OverviewJesse Olson - Nagios Log Server Architecture Overview
Jesse Olson - Nagios Log Server Architecture Overview
Nagios
 
NGINX: High Performance Load Balancing
NGINX: High Performance Load BalancingNGINX: High Performance Load Balancing
NGINX: High Performance Load Balancing
NGINX, Inc.
 
Nginx
NginxNginx
Altitude SF 2017: Advanced VCL: Shielding and Clustering
Altitude SF 2017: Advanced VCL: Shielding and ClusteringAltitude SF 2017: Advanced VCL: Shielding and Clustering
Altitude SF 2017: Advanced VCL: Shielding and Clustering
Fastly
 
Nginx A High Performance Load Balancer, Web Server & Reverse Proxy
Nginx A High Performance Load Balancer, Web Server & Reverse ProxyNginx A High Performance Load Balancer, Web Server & Reverse Proxy
Nginx A High Performance Load Balancer, Web Server & Reverse Proxy
Amit Aggarwal
 
Server monitoring using grafana and prometheus
Server monitoring using grafana and prometheusServer monitoring using grafana and prometheus
Server monitoring using grafana and prometheus
Celine George
 
What’s New in NGINX Plus R16? – EMEA
What’s New in NGINX Plus R16? – EMEAWhat’s New in NGINX Plus R16? – EMEA
What’s New in NGINX Plus R16? – EMEA
NGINX, Inc.
 
Nginx internals
Nginx internalsNginx internals
Nginx internalsliqiang xu
 
What's New in NGINX Plus R12?
What's New in NGINX Plus R12? What's New in NGINX Plus R12?
What's New in NGINX Plus R12?
NGINX, Inc.
 
NGINX: HTTP/2 Server Push and gRPC
NGINX: HTTP/2 Server Push and gRPCNGINX: HTTP/2 Server Push and gRPC
NGINX: HTTP/2 Server Push and gRPC
NGINX, Inc.
 
A user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management toolsA user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management tools
SaltStack
 
NGINX High-performance Caching
NGINX High-performance CachingNGINX High-performance Caching
NGINX High-performance Caching
NGINX, Inc.
 

What's hot (20)

Trevor McDonald - Nagios XI Under The Hood
Trevor McDonald  - Nagios XI Under The HoodTrevor McDonald  - Nagios XI Under The Hood
Trevor McDonald - Nagios XI Under The Hood
 
Prometheus: infrastructure and application monitoring in kubernetes cluster
Prometheus: infrastructure and application monitoring in kubernetes clusterPrometheus: infrastructure and application monitoring in kubernetes cluster
Prometheus: infrastructure and application monitoring in kubernetes cluster
 
Extending functionality in nginx, with modules!
Extending functionality in nginx, with modules!Extending functionality in nginx, with modules!
Extending functionality in nginx, with modules!
 
Architectural caching patterns for kubernetes
Architectural caching patterns for kubernetesArchitectural caching patterns for kubernetes
Architectural caching patterns for kubernetes
 
NGINX: Basics & Best Practices - EMEA Broadcast
NGINX: Basics & Best Practices - EMEA BroadcastNGINX: Basics & Best Practices - EMEA Broadcast
NGINX: Basics & Best Practices - EMEA Broadcast
 
High Availability Content Caching with NGINX
High Availability Content Caching with NGINXHigh Availability Content Caching with NGINX
High Availability Content Caching with NGINX
 
Nginx - Tips and Tricks.
Nginx - Tips and Tricks.Nginx - Tips and Tricks.
Nginx - Tips and Tricks.
 
Pragmatic Monolith-First, easy to decompose, clean architecture
Pragmatic Monolith-First, easy to decompose, clean architecturePragmatic Monolith-First, easy to decompose, clean architecture
Pragmatic Monolith-First, easy to decompose, clean architecture
 
Jesse Olson - Nagios Log Server Architecture Overview
Jesse Olson - Nagios Log Server Architecture OverviewJesse Olson - Nagios Log Server Architecture Overview
Jesse Olson - Nagios Log Server Architecture Overview
 
NGINX: High Performance Load Balancing
NGINX: High Performance Load BalancingNGINX: High Performance Load Balancing
NGINX: High Performance Load Balancing
 
Nginx
NginxNginx
Nginx
 
Altitude SF 2017: Advanced VCL: Shielding and Clustering
Altitude SF 2017: Advanced VCL: Shielding and ClusteringAltitude SF 2017: Advanced VCL: Shielding and Clustering
Altitude SF 2017: Advanced VCL: Shielding and Clustering
 
Nginx A High Performance Load Balancer, Web Server & Reverse Proxy
Nginx A High Performance Load Balancer, Web Server & Reverse ProxyNginx A High Performance Load Balancer, Web Server & Reverse Proxy
Nginx A High Performance Load Balancer, Web Server & Reverse Proxy
 
Server monitoring using grafana and prometheus
Server monitoring using grafana and prometheusServer monitoring using grafana and prometheus
Server monitoring using grafana and prometheus
 
What’s New in NGINX Plus R16? – EMEA
What’s New in NGINX Plus R16? – EMEAWhat’s New in NGINX Plus R16? – EMEA
What’s New in NGINX Plus R16? – EMEA
 
Nginx internals
Nginx internalsNginx internals
Nginx internals
 
What's New in NGINX Plus R12?
What's New in NGINX Plus R12? What's New in NGINX Plus R12?
What's New in NGINX Plus R12?
 
NGINX: HTTP/2 Server Push and gRPC
NGINX: HTTP/2 Server Push and gRPCNGINX: HTTP/2 Server Push and gRPC
NGINX: HTTP/2 Server Push and gRPC
 
A user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management toolsA user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management tools
 
NGINX High-performance Caching
NGINX High-performance CachingNGINX High-performance Caching
NGINX High-performance Caching
 

Similar to Infrastructural challenges of a fast-pace startup

Adding serverless to legacy applications
Adding serverless to legacy applicationsAdding serverless to legacy applications
Adding serverless to legacy applications
brettflorio
 
Introducing Gridiron Security and Compliance Management Platform and Enclave ...
Introducing Gridiron Security and Compliance Management Platform and Enclave ...Introducing Gridiron Security and Compliance Management Platform and Enclave ...
Introducing Gridiron Security and Compliance Management Platform and Enclave ...
Aptible
 
MesosCon - Be a microservices hero
MesosCon - Be a microservices heroMesosCon - Be a microservices hero
MesosCon - Be a microservices hero
Dragos Dascalita Haut
 
Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014
Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014
Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014
Rackspace Academy
 
Start tracking your ruby infrastructure
Start tracking your ruby infrastructureStart tracking your ruby infrastructure
Start tracking your ruby infrastructure
Sergiy Kukunin
 
DevOps in PHP environment
DevOps in PHP environment DevOps in PHP environment
DevOps in PHP environment
Evaldo Felipe
 
AutoScaling and Drupal
AutoScaling and DrupalAutoScaling and Drupal
AutoScaling and Drupal
Promet Source
 
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
Anna Klepacka
 
Making Spinnaker Go @ Stitch Fix
Making Spinnaker Go @ Stitch FixMaking Spinnaker Go @ Stitch Fix
Making Spinnaker Go @ Stitch Fix
Diana Tkachenko
 
HP Helion European Webinar Series ,Webinar #3
HP Helion European Webinar Series ,Webinar #3 HP Helion European Webinar Series ,Webinar #3
HP Helion European Webinar Series ,Webinar #3
BeMyApp
 
Self Created Load Balancer for MTA on AWS
Self Created Load Balancer for MTA on AWSSelf Created Load Balancer for MTA on AWS
Self Created Load Balancer for MTA on AWSsharu1204
 
Introduction to Cloud Foundry #JJUG
Introduction to Cloud Foundry #JJUGIntroduction to Cloud Foundry #JJUG
Introduction to Cloud Foundry #JJUG
Toshiaki Maki
 
RoR Workshop - Web applications hacking - Ruby on Rails example
RoR Workshop - Web applications hacking - Ruby on Rails exampleRoR Workshop - Web applications hacking - Ruby on Rails example
RoR Workshop - Web applications hacking - Ruby on Rails example
Railwaymen
 
Setting Up a TIG Stack for Your Testing
Setting Up a TIG Stack for Your TestingSetting Up a TIG Stack for Your Testing
Setting Up a TIG Stack for Your Testing
Jet Liu
 
Python in the serverless era (PyCon 2017)
Python in the serverless era (PyCon 2017)Python in the serverless era (PyCon 2017)
Python in the serverless era (PyCon 2017)
Benny Bauer
 
Postgres the hardway
Postgres the hardwayPostgres the hardway
Postgres the hardway
Dave Pitts
 
Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)
Yan Cui
 
Docker In Bank Unrated
Docker In Bank UnratedDocker In Bank Unrated
Docker In Bank Unrated
Aleksandr Tarasov
 
Making Sense out of Amazon ECS
Making Sense out of Amazon ECSMaking Sense out of Amazon ECS
Making Sense out of Amazon ECS
WhiteHedge Technologies Inc.
 
Pluggable Databases: What they will break and why you should use them anyway!
Pluggable Databases: What they will break and why you should use them anyway!Pluggable Databases: What they will break and why you should use them anyway!
Pluggable Databases: What they will break and why you should use them anyway!
Guatemala User Group
 

Similar to Infrastructural challenges of a fast-pace startup (20)

Adding serverless to legacy applications
Adding serverless to legacy applicationsAdding serverless to legacy applications
Adding serverless to legacy applications
 
Introducing Gridiron Security and Compliance Management Platform and Enclave ...
Introducing Gridiron Security and Compliance Management Platform and Enclave ...Introducing Gridiron Security and Compliance Management Platform and Enclave ...
Introducing Gridiron Security and Compliance Management Platform and Enclave ...
 
MesosCon - Be a microservices hero
MesosCon - Be a microservices heroMesosCon - Be a microservices hero
MesosCon - Be a microservices hero
 
Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014
Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014
Software as a Service workshop / Unlocked: the Hybrid Cloud 12th May 2014
 
Start tracking your ruby infrastructure
Start tracking your ruby infrastructureStart tracking your ruby infrastructure
Start tracking your ruby infrastructure
 
DevOps in PHP environment
DevOps in PHP environment DevOps in PHP environment
DevOps in PHP environment
 
AutoScaling and Drupal
AutoScaling and DrupalAutoScaling and Drupal
AutoScaling and Drupal
 
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
Workshop KrakYourNet2016 - Web applications hacking Ruby on Rails example
 
Making Spinnaker Go @ Stitch Fix
Making Spinnaker Go @ Stitch FixMaking Spinnaker Go @ Stitch Fix
Making Spinnaker Go @ Stitch Fix
 
HP Helion European Webinar Series ,Webinar #3
HP Helion European Webinar Series ,Webinar #3 HP Helion European Webinar Series ,Webinar #3
HP Helion European Webinar Series ,Webinar #3
 
Self Created Load Balancer for MTA on AWS
Self Created Load Balancer for MTA on AWSSelf Created Load Balancer for MTA on AWS
Self Created Load Balancer for MTA on AWS
 
Introduction to Cloud Foundry #JJUG
Introduction to Cloud Foundry #JJUGIntroduction to Cloud Foundry #JJUG
Introduction to Cloud Foundry #JJUG
 
RoR Workshop - Web applications hacking - Ruby on Rails example
RoR Workshop - Web applications hacking - Ruby on Rails exampleRoR Workshop - Web applications hacking - Ruby on Rails example
RoR Workshop - Web applications hacking - Ruby on Rails example
 
Setting Up a TIG Stack for Your Testing
Setting Up a TIG Stack for Your TestingSetting Up a TIG Stack for Your Testing
Setting Up a TIG Stack for Your Testing
 
Python in the serverless era (PyCon 2017)
Python in the serverless era (PyCon 2017)Python in the serverless era (PyCon 2017)
Python in the serverless era (PyCon 2017)
 
Postgres the hardway
Postgres the hardwayPostgres the hardway
Postgres the hardway
 
Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)
 
Docker In Bank Unrated
Docker In Bank UnratedDocker In Bank Unrated
Docker In Bank Unrated
 
Making Sense out of Amazon ECS
Making Sense out of Amazon ECSMaking Sense out of Amazon ECS
Making Sense out of Amazon ECS
 
Pluggable Databases: What they will break and why you should use them anyway!
Pluggable Databases: What they will break and why you should use them anyway!Pluggable Databases: What they will break and why you should use them anyway!
Pluggable Databases: What they will break and why you should use them anyway!
 

More from DevOps Braga

DevOps Braga #11: Docker Anatomy
DevOps Braga #11: Docker AnatomyDevOps Braga #11: Docker Anatomy
DevOps Braga #11: Docker Anatomy
DevOps Braga
 
DevOps Braga #9: Introdução ao Terraform
DevOps Braga #9:  Introdução ao TerraformDevOps Braga #9:  Introdução ao Terraform
DevOps Braga #9: Introdução ao Terraform
DevOps Braga
 
DevOps Braga #4: Infrastructure as Code: Impulsionar DevOps
DevOps Braga #4: Infrastructure as Code: Impulsionar DevOpsDevOps Braga #4: Infrastructure as Code: Impulsionar DevOps
DevOps Braga #4: Infrastructure as Code: Impulsionar DevOps
DevOps Braga
 
DevOps Braga #7: Salt: Configuration Management
DevOps Braga #7: Salt: Configuration ManagementDevOps Braga #7: Salt: Configuration Management
DevOps Braga #7: Salt: Configuration Management
DevOps Braga
 
DevOps Braga #3: Admin rights, everyone gets Admin rights!
DevOps Braga #3: Admin rights, everyone gets Admin rights!DevOps Braga #3: Admin rights, everyone gets Admin rights!
DevOps Braga #3: Admin rights, everyone gets Admin rights!
DevOps Braga
 
DevOps Braga #6
DevOps Braga #6DevOps Braga #6
DevOps Braga #6
DevOps Braga
 
DevOps Braga #5
DevOps Braga #5DevOps Braga #5
DevOps Braga #5
DevOps Braga
 

More from DevOps Braga (7)

DevOps Braga #11: Docker Anatomy
DevOps Braga #11: Docker AnatomyDevOps Braga #11: Docker Anatomy
DevOps Braga #11: Docker Anatomy
 
DevOps Braga #9: Introdução ao Terraform
DevOps Braga #9:  Introdução ao TerraformDevOps Braga #9:  Introdução ao Terraform
DevOps Braga #9: Introdução ao Terraform
 
DevOps Braga #4: Infrastructure as Code: Impulsionar DevOps
DevOps Braga #4: Infrastructure as Code: Impulsionar DevOpsDevOps Braga #4: Infrastructure as Code: Impulsionar DevOps
DevOps Braga #4: Infrastructure as Code: Impulsionar DevOps
 
DevOps Braga #7: Salt: Configuration Management
DevOps Braga #7: Salt: Configuration ManagementDevOps Braga #7: Salt: Configuration Management
DevOps Braga #7: Salt: Configuration Management
 
DevOps Braga #3: Admin rights, everyone gets Admin rights!
DevOps Braga #3: Admin rights, everyone gets Admin rights!DevOps Braga #3: Admin rights, everyone gets Admin rights!
DevOps Braga #3: Admin rights, everyone gets Admin rights!
 
DevOps Braga #6
DevOps Braga #6DevOps Braga #6
DevOps Braga #6
 
DevOps Braga #5
DevOps Braga #5DevOps Braga #5
DevOps Braga #5
 

Recently uploaded

Latest trends in computer networking.pptx
Latest trends in computer networking.pptxLatest trends in computer networking.pptx
Latest trends in computer networking.pptx
JungkooksNonexistent
 
Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines
Multi-cluster Kubernetes Networking- Patterns, Projects and GuidelinesMulti-cluster Kubernetes Networking- Patterns, Projects and Guidelines
Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines
Sanjeev Rampal
 
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
3ipehhoa
 
guildmasters guide to ravnica Dungeons & Dragons 5...
guildmasters guide to ravnica Dungeons & Dragons 5...guildmasters guide to ravnica Dungeons & Dragons 5...
guildmasters guide to ravnica Dungeons & Dragons 5...
Rogerio Filho
 
BASIC C++ lecture NOTE C++ lecture 3.pptx
BASIC C++ lecture NOTE C++ lecture 3.pptxBASIC C++ lecture NOTE C++ lecture 3.pptx
BASIC C++ lecture NOTE C++ lecture 3.pptx
natyesu
 
This 7-second Brain Wave Ritual Attracts Money To You.!
This 7-second Brain Wave Ritual Attracts Money To You.!This 7-second Brain Wave Ritual Attracts Money To You.!
This 7-second Brain Wave Ritual Attracts Money To You.!
nirahealhty
 
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
3ipehhoa
 
Output determination SAP S4 HANA SAP SD CC
Output determination SAP S4 HANA SAP SD CCOutput determination SAP S4 HANA SAP SD CC
Output determination SAP S4 HANA SAP SD CC
ShahulHameed54211
 
1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样
1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样
1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样
3ipehhoa
 
ER(Entity Relationship) Diagram for online shopping - TAE
ER(Entity Relationship) Diagram for online shopping - TAEER(Entity Relationship) Diagram for online shopping - TAE
ER(Entity Relationship) Diagram for online shopping - TAE
Himani415946
 
History+of+E-commerce+Development+in+China-www.cfye-commerce.shop
History+of+E-commerce+Development+in+China-www.cfye-commerce.shopHistory+of+E-commerce+Development+in+China-www.cfye-commerce.shop
History+of+E-commerce+Development+in+China-www.cfye-commerce.shop
laozhuseo02
 
How to Use Contact Form 7 Like a Pro.pptx
How to Use Contact Form 7 Like a Pro.pptxHow to Use Contact Form 7 Like a Pro.pptx
How to Use Contact Form 7 Like a Pro.pptx
Gal Baras
 
1.Wireless Communication System_Wireless communication is a broad term that i...
1.Wireless Communication System_Wireless communication is a broad term that i...1.Wireless Communication System_Wireless communication is a broad term that i...
1.Wireless Communication System_Wireless communication is a broad term that i...
JeyaPerumal1
 
The+Prospects+of+E-Commerce+in+China.pptx
The+Prospects+of+E-Commerce+in+China.pptxThe+Prospects+of+E-Commerce+in+China.pptx
The+Prospects+of+E-Commerce+in+China.pptx
laozhuseo02
 
Living-in-IT-era-Module-7-Imaging-and-Design-for-Social-Impact.pptx
Living-in-IT-era-Module-7-Imaging-and-Design-for-Social-Impact.pptxLiving-in-IT-era-Module-7-Imaging-and-Design-for-Social-Impact.pptx
Living-in-IT-era-Module-7-Imaging-and-Design-for-Social-Impact.pptx
TristanJasperRamos
 
test test test test testtest test testtest test testtest test testtest test ...
test test  test test testtest test testtest test testtest test testtest test ...test test  test test testtest test testtest test testtest test testtest test ...
test test test test testtest test testtest test testtest test testtest test ...
Arif0071
 

Recently uploaded (16)

Latest trends in computer networking.pptx
Latest trends in computer networking.pptxLatest trends in computer networking.pptx
Latest trends in computer networking.pptx
 
Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines
Multi-cluster Kubernetes Networking- Patterns, Projects and GuidelinesMulti-cluster Kubernetes Networking- Patterns, Projects and Guidelines
Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines
 
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样
 
guildmasters guide to ravnica Dungeons & Dragons 5...
guildmasters guide to ravnica Dungeons & Dragons 5...guildmasters guide to ravnica Dungeons & Dragons 5...
guildmasters guide to ravnica Dungeons & Dragons 5...
 
BASIC C++ lecture NOTE C++ lecture 3.pptx
BASIC C++ lecture NOTE C++ lecture 3.pptxBASIC C++ lecture NOTE C++ lecture 3.pptx
BASIC C++ lecture NOTE C++ lecture 3.pptx
 
This 7-second Brain Wave Ritual Attracts Money To You.!
This 7-second Brain Wave Ritual Attracts Money To You.!This 7-second Brain Wave Ritual Attracts Money To You.!
This 7-second Brain Wave Ritual Attracts Money To You.!
 
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样
 
Output determination SAP S4 HANA SAP SD CC
Output determination SAP S4 HANA SAP SD CCOutput determination SAP S4 HANA SAP SD CC
Output determination SAP S4 HANA SAP SD CC
 
1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样
1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样
1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样
 
ER(Entity Relationship) Diagram for online shopping - TAE
ER(Entity Relationship) Diagram for online shopping - TAEER(Entity Relationship) Diagram for online shopping - TAE
ER(Entity Relationship) Diagram for online shopping - TAE
 
History+of+E-commerce+Development+in+China-www.cfye-commerce.shop
History+of+E-commerce+Development+in+China-www.cfye-commerce.shopHistory+of+E-commerce+Development+in+China-www.cfye-commerce.shop
History+of+E-commerce+Development+in+China-www.cfye-commerce.shop
 
How to Use Contact Form 7 Like a Pro.pptx
How to Use Contact Form 7 Like a Pro.pptxHow to Use Contact Form 7 Like a Pro.pptx
How to Use Contact Form 7 Like a Pro.pptx
 
1.Wireless Communication System_Wireless communication is a broad term that i...
1.Wireless Communication System_Wireless communication is a broad term that i...1.Wireless Communication System_Wireless communication is a broad term that i...
1.Wireless Communication System_Wireless communication is a broad term that i...
 
The+Prospects+of+E-Commerce+in+China.pptx
The+Prospects+of+E-Commerce+in+China.pptxThe+Prospects+of+E-Commerce+in+China.pptx
The+Prospects+of+E-Commerce+in+China.pptx
 
Living-in-IT-era-Module-7-Imaging-and-Design-for-Social-Impact.pptx
Living-in-IT-era-Module-7-Imaging-and-Design-for-Social-Impact.pptxLiving-in-IT-era-Module-7-Imaging-and-Design-for-Social-Impact.pptx
Living-in-IT-era-Module-7-Imaging-and-Design-for-Social-Impact.pptx
 
test test test test testtest test testtest test testtest test testtest test ...
test test  test test testtest test testtest test testtest test testtest test ...test test  test test testtest test testtest test testtest test testtest test ...
test test test test testtest test testtest test testtest test testtest test ...
 

Infrastructural challenges of a fast-pace startup

  • 1. DevOps Braga #12: Infrastructural challenges of a fast-pace startup Gustavo Balbino | Rui Matos - DevOps Team
  • 2. Agenda About Us Storage Issues Rethinking our Infrastructure Load balancing Web Applications Containers
  • 3. Primeira Gráfica Online Portuguesa, líderes Ibéricos e no Brasil • Founded in 2013 • 200 Employees • Offices in 3 cities • Braga • Torres Vedras • Lisboa • Looking for +150 employees by the end 2019 • 21 countries North America Brazil Europe *GB IE FR DE IT PL CZ NL AT SE DK BE CH NO FI
  • 4. 3 Stores Brazil MexicoIberia Each Store is (almost) independent in terms of infrastructure, virtual machines, database, etc…
  • 5. 2019 Market Expansion Iberia Europe Mexico North America
  • 6. 100% Cloud - Microsoft Azure
  • 10. 2 Big Size VMs >= D8s v3 Azure Load Balancer Windows VMs running IIS, SOFS e S2D Scalable but at what cost? Production Environment MX Aug/2018
  • 11. Windows Server 2012 R2 Scale-Out File Server • A set of clustered file servers that make up a transparent failover file server cluster • SOFS is a fantastic way to deploy small and affordable clustered storage • The Concept of Storage Spaces Direct (S2D) • Fault Tolerance • Performance • Scaling Out and In
  • 13. One role, one machine HAProxy Load Balancer Smaller size VMs Horizontal & vertical scalability Divide to conquer
  • 14. 2 Smaller Virtual Machines Faster Asynchronous replication every 30 seconds Dedicated NICs for Replication Scalable SMB File Server with Storage Replica
  • 15. Load Balancing Web Applications
  • 18. What is HAProxy • HAProxy (High Availability Proxy) , is a popular open source software TCP/HTTP Load Balancer and proxying solution which can be run on Linux, Solaris and FreeBSD. • Its most common use is to improve the performance and reliability of a server environment by distributing the workload across multiple servers (Web, applications, databases). • Some happy users;  GitHub  Reddit  Instagram  Stack Overflow  Twitter  Tumblr  Vimeo  YouPorn
  • 19. Why HAProxy? • Absolutely free and and widely supported • Supports SPDY (obsolet) & HTTP/2 • Supports Traffic Encryption & SSL Termination • Let's Encrypt ACME client integration • Access to Integrated Server Monitoring Dashboard • Several Load Balancing Algorithms/Configurations • Easy to form a Active/Passive cluster configuration • “Runtime API” for automation/integration • Personalized error page
  • 20. Active/Passive Health Check in a private cloud
  • 21. How to implement it in Azure? PUBLIC IP ADDRESS AZURE LOAD BALANCER 2 VIRTUAL MACHINES
  • 22. HAProxy & Keepalived Installation sudo apt-get update sudo apt-get -y upgrade sudo apt-get install -y software-properties-common sudo apt-get install -y haproxy keepalived sudo apt-get install socat
  • 23. Inbound NAT rules LB Azure Inbound Custom port NAT for SSH - because you won’t have an addressable IP Address to operate your machines
  • 24. Building an Active/Passive HAProxy in Azure Set during LB creation Where you’ll declare HAProxy VM NIC Simple check over port 80 HTTP & HTTPS
  • 25. Before Keepalived MagicEnabling non local Virtual IP binding sudo nano /etc/sysctl.conf net.ipv4.ip_nonlocal_bind=1 fs.file-max = 10000000 fs.nr_open = 10000000 net.ipv4.tcp_mem = 786432 1697152 1945728 net.ipv4.tcp_rmem = 4096 4096 16777216 net.ipv4.tcp_wmem = 4096 4096 16777216 net.ipv4.ip_local_port_range = 1024 65000 sudo sysctl -p
  • 26. Keepalived configuration MASTERglobal_defs {global_defs { enable_script_security script_user root } enable_script_security script_user root } /etc/keepalived/keepalived.conf vrrp_script chk_appsvc { script /usr/local/sbin/keepalived-check.sh interval 1 fall 2 rise 2 } vrrp_instance VIP_1 { advert_int 1 interface eth0 authentication { auth_type PASS auth_pass S0meTrickP4ss0rdY0uCantR3m3mb3R } virtual_router_id 51 virtual_ipaddress { 111.222.333.444 } track_script { chk_appsvc } notify /usr/local/sbin/keepalived-action.sh notify_stop "/usr/local/sbin/keepalived-action.sh INSTANCE VIP_1 STOP" state MASTER priority 101 unicast_src_ip 10.255.1.51 unicast_peer { 10.255.1.52 } }
  • 27. Keepalived configuration SLAVEglobal_defs {global_defs { enable_script_security script_user root } enable_script_security script_user root } /etc/keepalived/keepalived.conf vrrp_script chk_appsvc { script /usr/local/sbin/keepalived-check.sh interval 1 fall 2 rise 2 } vrrp_instance VIP_1 { advert_int 1 interface eth0 authentication { auth_type PASS auth_pass S0meTrickP4ss0rdY0uCantR3m3mb3R } virtual_router_id 51 virtual_ipaddress { 111.222.333.444 } track_script { chk_appsvc } notify /usr/local/sbin/keepalived-action.sh notify_stop "/usr/local/sbin/keepalived-action.sh INSTANCE VIP_1 STOP" state SLAVE priority 100 unicast_src_ip 10.255.1.52 unicast_peer { 10.255.1.51 } }
  • 28. Keepalived check script #!/bin/bash URL="http://localhost" if [[ `curl -s -o/dev/null --connect-timeout 0.5 $URL; echo $?` -ne 0 ]]; then exit 1 else exit 0 fi /usr/local/sbin/keepalived-check.sh
  • 29. #!/bin/bash TYPE=$1 NAME=$2 STATE=$3 modify_probe_status_http() { STATUS=$1 LB_PROBE_PORT=80 LB_PROBE_DEV=eth0 if [[ $STATUS == "down" ]]; then # Add firewall rule to block LB probe port /sbin/iptables -A INPUT -p tcp --dport $LB_PROBE_PORT -j REJECT -i $LB_PROBE_DEV elif [[ $STATUS == "up" ]]; then # Remove all entries to block LB probe port RC=0 while [[ $RC -eq 0 ]]; do RC=`/sbin/iptables -D INPUT -p tcp --dport $LB_PROBE_PORT -j REJECT -i $LB_PROBE_DEV 2>/dev/null; echo $?` done else echo "Unknown probe status" fi } if [[ "$NAME" == "VIP_1" ]]; then case $STATE in "MASTER") modify_probe_status_http up exit 0 ;; "BACKUP"|"STOP") modify_probe_status_http down exit 0 ;; "FAULT") modify_probe_status_http down exit 0 ;; *) echo "unknown state" exit 1 ;; esac else echo "Nothing to do" exit 0 fi HAPROXY is DOWN /sbin/iptables -A INPUT -p tcp --dport 80 -j REJECT -i eth0 HAPROXY is UP /sbin/iptables -D INPUT -p tcp --dport 80 -j REJECT -i eth0 Keepalived action script /usr/local/sbin/keepalived-action.sh
  • 30. Keepalived in Action/Demo Master HAProxy Slave HAProxy Because Azure LB can reach port 80, IP is delivered to VM
  • 31. In what ways can HAProxy help you? • Protect you from DDoS; • Prevent SQL Injection and abnormal user behavior; • Segregate traffic, by URL, domain, user agent, IP ranges and many, many more; • URL Redirect and request rewrites; • Cache server for small objects; • Centralized logging • Compression
  • 33. Production Microservices Linux hosts Docker containers – 4 already! Scale ready infrastructure One infrastructure per region
  • 35. Where we are Team City is our CI – It builds the software Octopus is hosted as a tentacle on each machine, and deploys the software A total of 50 steps on the build process and 66 on the deployment phase The complete deployment process could take about 2 hours right now
  • 36. New pipeline – Microservices oriented Jenkins is just an orchestrator Docker pipeline will build the container image and push to the registry Ansible will deploy the container in the Linux hosts via SSH
  • 37. New pipeline – Microservices oriented A real DevOps environment Run by the Teams! Autonomous CI/CD environment
  • 38. Q & A gustavo.balbino@360imprimir.pt | https://www.linkedin.com/in/gustavo-lima-969890a9/ rui.matos@360imprimir.pt | https://www.linkedin.com/in/rui-matos/