SlideShare a Scribd company logo
Autobots @
realestate.com.au
          Automate
   ALL THE THINGS
   (or at least the things that really matter)
Who am I?
        Geoffrey Giesemann     @ggiesemann   @geekle




Technical Lead @ realestate.
                    com.au



Some newcomers mistake me
          for a sysadmin :(
What's the problem?
(and what the hell am I talking about?)
WTF is going on?!?




                 I hope my wife still
                   recognises me...




(deployments are *hard*)
Why so hard?
(just the usual suspects)
How do we fix this?
  (and who do we blame?)
deploybot
with_down_in_load_balancer(server) do
  with_down_in_nagios(server) do
    puppet(server)
    raise "borked!" unless server.working?
  end
end
schemabot
What's worked well?
Service Discovery!
$ grep -r 'aa01' nagios/ | wc -l
      17
$ grep -r 'aa01' puppet/ | wc -l
       4
$ grep -r 'aa01' deploybot/ | wc -l
       0
> show lb vserver agentadmin-prod
     agentadmin-prod (125.56.204.120:80) - HTTP     Type: ADDRESS State: UP
     ... blah blah ...
Bound Service Groups:
1)   Group Name: agentadmin
          1) agentadmin (192.168.25.1: 80) - HTTP State: UP Weight: 1
          Persistence Cookie Value : my_random_str=9999
          2) agentadmin (192.168.25.2: 80) - HTTP State: UP Weight: 1
          Persistence Cookie Value : my_random_str=9999
         ... etc etc ...


> show servicegroup agentadmin
     agentadmin - HTTP
     State: ENABLED     Monitor Threshold : 0
     Max Conn: 0      Max Req: 0     Max Bandwidth: 0 kbits
     Monitor Name: http-diagnositic-warmup      State: ENABLED   Weight: 1
     1) 192.168.25.1:80    State: UP Server Name: 192.168.25.1
        Probes: 131205     Failed [Total: 2525 Current: 0]
        Last response: Success - HTTP response code 200 received.
     2) 192.168.25.2:80    State: UP Server Name: 192.168.25.2
        Probes: 131322     Failed [Total: 2428 Current: 0]
        Last response: Success - HTTP response code 200 received.
Standardised Monitoring!
$ curl http://aa01/diagnostic/status/nagios
OK - the application is functioning
correctly

$ curl http://ar01/diagnostic/status/nagios
OK - the application is functioning
correctly
● custom app health checks
  ○ https://github.com/tribune/is_it_working
  ○ https://github.com/blythedunham/health_monitor

● monitor requests in munin
  ○ https://github.com/pka/rack-monitor
Better Error Handling!
● Don't bother trying to handle errors - just
  make sure you can recover from them
  quickly!

● As long as you have some app servers alive
  things will work out!
FIN
Thanks!
●   http://markerguru.deviantart.com/
●   http://www.quickmeme.com/
●   http://www.doingitwrong.com/
●   Everyone who was part of @rea_autobots

More Related Content

What's hot

Selenium web driver useful commands
Selenium web driver useful commandsSelenium web driver useful commands
Selenium web driver useful commands
KadarkaraiSelvam
 
Serenity Now
Serenity NowSerenity Now
Serenity Now
Daniel MacTough
 
Server architecture
Server architectureServer architecture
Server architecture
Masters Academy
 
Websocket on Rails
Websocket on RailsWebsocket on Rails
Websocket on Rails
Jeroen Rosenberg
 
Ak12 adding shelf
Ak12 adding shelfAk12 adding shelf
Ak12 adding shelf
Accenture
 
Activation limit&flowlimit&maxjobs
Activation limit&flowlimit&maxjobsActivation limit&flowlimit&maxjobs
Activation limit&flowlimit&maxjobs
Nirmala Devi
 
Alfresco : Implementing Workflow
Alfresco : Implementing WorkflowAlfresco : Implementing Workflow
Alfresco : Implementing Workflow
Wildan Maulana
 

What's hot (7)

Selenium web driver useful commands
Selenium web driver useful commandsSelenium web driver useful commands
Selenium web driver useful commands
 
Serenity Now
Serenity NowSerenity Now
Serenity Now
 
Server architecture
Server architectureServer architecture
Server architecture
 
Websocket on Rails
Websocket on RailsWebsocket on Rails
Websocket on Rails
 
Ak12 adding shelf
Ak12 adding shelfAk12 adding shelf
Ak12 adding shelf
 
Activation limit&flowlimit&maxjobs
Activation limit&flowlimit&maxjobsActivation limit&flowlimit&maxjobs
Activation limit&flowlimit&maxjobs
 
Alfresco : Implementing Workflow
Alfresco : Implementing WorkflowAlfresco : Implementing Workflow
Alfresco : Implementing Workflow
 

Viewers also liked

Nike
NikeNike
Nike
juninho85
 
Chef, Nginx e Capistrano: Como usar!
Chef, Nginx e Capistrano: Como usar!Chef, Nginx e Capistrano: Como usar!
Chef, Nginx e Capistrano: Como usar!
Felipe Navas
 
2 sam. characters
2 sam. characters2 sam. characters
2 sam. characters
dnassen15
 
Karostas Cietums
Karostas CietumsKarostas Cietums
Karostas Cietums
42025675
 
DevOps e PHP
DevOps e PHPDevOps e PHP
DevOps e PHP
Elton Minetto
 
My adszens
My adszensMy adszens
My adszens
mfb26
 
Apresentando o OpsWorks - Bemobi
Apresentando o OpsWorks - BemobiApresentando o OpsWorks - Bemobi
Apresentando o OpsWorks - Bemobi
Ricardo Martins ☁
 
Continuous Deployment e DevOps na Nuvem
Continuous Deployment e DevOps na NuvemContinuous Deployment e DevOps na Nuvem
Continuous Deployment e DevOps na Nuvem
Amazon Web Services LATAM
 

Viewers also liked (9)

Nike
NikeNike
Nike
 
Chef, Nginx e Capistrano: Como usar!
Chef, Nginx e Capistrano: Como usar!Chef, Nginx e Capistrano: Como usar!
Chef, Nginx e Capistrano: Como usar!
 
2 sam. characters
2 sam. characters2 sam. characters
2 sam. characters
 
Instan jamu
Instan jamuInstan jamu
Instan jamu
 
Karostas Cietums
Karostas CietumsKarostas Cietums
Karostas Cietums
 
DevOps e PHP
DevOps e PHPDevOps e PHP
DevOps e PHP
 
My adszens
My adszensMy adszens
My adszens
 
Apresentando o OpsWorks - Bemobi
Apresentando o OpsWorks - BemobiApresentando o OpsWorks - Bemobi
Apresentando o OpsWorks - Bemobi
 
Continuous Deployment e DevOps na Nuvem
Continuous Deployment e DevOps na NuvemContinuous Deployment e DevOps na Nuvem
Continuous Deployment e DevOps na Nuvem
 

Similar to Autobots @ REA

Static Code Analysis PHP[tek] 2023
Static Code Analysis PHP[tek] 2023Static Code Analysis PHP[tek] 2023
Static Code Analysis PHP[tek] 2023
Scott Keck-Warren
 
DSR Testing (Part 2)
DSR Testing (Part 2)DSR Testing (Part 2)
DSR Testing (Part 2)
Steve Upton
 
Overcoming (organizational) scalability issues in your Prometheus ecosystem
Overcoming (organizational) scalability issues in your Prometheus ecosystemOvercoming (organizational) scalability issues in your Prometheus ecosystem
Overcoming (organizational) scalability issues in your Prometheus ecosystem
QAware GmbH
 
DevOps(3) : Ansible - (MOSG)
DevOps(3) : Ansible - (MOSG)DevOps(3) : Ansible - (MOSG)
DevOps(3) : Ansible - (MOSG)
Soshi Nemoto
 
Overcoming scalability issues in your prometheus ecosystem
Overcoming scalability issues in your prometheus ecosystemOvercoming scalability issues in your prometheus ecosystem
Overcoming scalability issues in your prometheus ecosystem
Nebulaworks
 
MongoDB World 2019: Becoming an Ops Manager Backup Superhero!
MongoDB World 2019: Becoming an Ops Manager Backup Superhero!MongoDB World 2019: Becoming an Ops Manager Backup Superhero!
MongoDB World 2019: Becoming an Ops Manager Backup Superhero!
MongoDB
 
DDoS: practical survival
DDoS: practical survivalDDoS: practical survival
DDoS: practical survival
Positive Hack Days
 
Developing apps using Perl
Developing apps using PerlDeveloping apps using Perl
Developing apps using Perl
Anatoly Sharifulin
 
Using SLOs for Continuous Performance Optimizations of Your k8s Workloads
Using SLOs for Continuous Performance Optimizations of Your k8s WorkloadsUsing SLOs for Continuous Performance Optimizations of Your k8s Workloads
Using SLOs for Continuous Performance Optimizations of Your k8s Workloads
ScyllaDB
 
WTF is Sensu and Monitoring
WTF is Sensu and MonitoringWTF is Sensu and Monitoring
WTF is Sensu and Monitoring
Toby Jackson
 
Zabbix Smart problem detection - FISL 2015 workshop
Zabbix Smart problem detection - FISL 2015 workshopZabbix Smart problem detection - FISL 2015 workshop
Zabbix Smart problem detection - FISL 2015 workshop
Zabbix
 
Hacking Web Performance
Hacking Web Performance Hacking Web Performance
Hacking Web Performance
Maximiliano Firtman
 
PowerPoint Presentation
PowerPoint PresentationPowerPoint Presentation
PowerPoint Presentation
webhostingguy
 
PowerPoint Presentation
PowerPoint PresentationPowerPoint Presentation
PowerPoint Presentation
webhostingguy
 
Serverless in production, an experience report (London DevOps)
Serverless in production, an experience report (London DevOps)Serverless in production, an experience report (London DevOps)
Serverless in production, an experience report (London DevOps)
Yan Cui
 
OSMC 2010 | NSClient++ - what's new? And what's coming! by Michael Medin
OSMC 2010 |  NSClient++ - what's new? And what's coming! by Michael MedinOSMC 2010 |  NSClient++ - what's new? And what's coming! by Michael Medin
OSMC 2010 | NSClient++ - what's new? And what's coming! by Michael Medin
NETWAYS
 
Windows Attacks AT is the new black
Windows Attacks   AT is the new blackWindows Attacks   AT is the new black
Windows Attacks AT is the new black
Rob Fuller
 
Windows attacks - AT is the new black
Windows attacks - AT is the new blackWindows attacks - AT is the new black
Windows attacks - AT is the new black
Chris Gates
 
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
Nagios
 
WebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan WintermeyerWebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan Wintermeyer
Elixir Club
 

Similar to Autobots @ REA (20)

Static Code Analysis PHP[tek] 2023
Static Code Analysis PHP[tek] 2023Static Code Analysis PHP[tek] 2023
Static Code Analysis PHP[tek] 2023
 
DSR Testing (Part 2)
DSR Testing (Part 2)DSR Testing (Part 2)
DSR Testing (Part 2)
 
Overcoming (organizational) scalability issues in your Prometheus ecosystem
Overcoming (organizational) scalability issues in your Prometheus ecosystemOvercoming (organizational) scalability issues in your Prometheus ecosystem
Overcoming (organizational) scalability issues in your Prometheus ecosystem
 
DevOps(3) : Ansible - (MOSG)
DevOps(3) : Ansible - (MOSG)DevOps(3) : Ansible - (MOSG)
DevOps(3) : Ansible - (MOSG)
 
Overcoming scalability issues in your prometheus ecosystem
Overcoming scalability issues in your prometheus ecosystemOvercoming scalability issues in your prometheus ecosystem
Overcoming scalability issues in your prometheus ecosystem
 
MongoDB World 2019: Becoming an Ops Manager Backup Superhero!
MongoDB World 2019: Becoming an Ops Manager Backup Superhero!MongoDB World 2019: Becoming an Ops Manager Backup Superhero!
MongoDB World 2019: Becoming an Ops Manager Backup Superhero!
 
DDoS: practical survival
DDoS: practical survivalDDoS: practical survival
DDoS: practical survival
 
Developing apps using Perl
Developing apps using PerlDeveloping apps using Perl
Developing apps using Perl
 
Using SLOs for Continuous Performance Optimizations of Your k8s Workloads
Using SLOs for Continuous Performance Optimizations of Your k8s WorkloadsUsing SLOs for Continuous Performance Optimizations of Your k8s Workloads
Using SLOs for Continuous Performance Optimizations of Your k8s Workloads
 
WTF is Sensu and Monitoring
WTF is Sensu and MonitoringWTF is Sensu and Monitoring
WTF is Sensu and Monitoring
 
Zabbix Smart problem detection - FISL 2015 workshop
Zabbix Smart problem detection - FISL 2015 workshopZabbix Smart problem detection - FISL 2015 workshop
Zabbix Smart problem detection - FISL 2015 workshop
 
Hacking Web Performance
Hacking Web Performance Hacking Web Performance
Hacking Web Performance
 
PowerPoint Presentation
PowerPoint PresentationPowerPoint Presentation
PowerPoint Presentation
 
PowerPoint Presentation
PowerPoint PresentationPowerPoint Presentation
PowerPoint Presentation
 
Serverless in production, an experience report (London DevOps)
Serverless in production, an experience report (London DevOps)Serverless in production, an experience report (London DevOps)
Serverless in production, an experience report (London DevOps)
 
OSMC 2010 | NSClient++ - what's new? And what's coming! by Michael Medin
OSMC 2010 |  NSClient++ - what's new? And what's coming! by Michael MedinOSMC 2010 |  NSClient++ - what's new? And what's coming! by Michael Medin
OSMC 2010 | NSClient++ - what's new? And what's coming! by Michael Medin
 
Windows Attacks AT is the new black
Windows Attacks   AT is the new blackWindows Attacks   AT is the new black
Windows Attacks AT is the new black
 
Windows attacks - AT is the new black
Windows attacks - AT is the new blackWindows attacks - AT is the new black
Windows attacks - AT is the new black
 
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
Nagios Conference 2011 - Nathan Vonnahme - Integrating Nagios With Test Drive...
 
WebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan WintermeyerWebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan Wintermeyer
 

Recently uploaded

Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
christinelarrosa
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Vadym Kazulkin
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
saastr
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
DanBrown980551
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
LizaNolte
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 

Recently uploaded (20)

Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 

Autobots @ REA

  • 1. Autobots @ realestate.com.au Automate ALL THE THINGS (or at least the things that really matter)
  • 2. Who am I? Geoffrey Giesemann @ggiesemann @geekle Technical Lead @ realestate. com.au Some newcomers mistake me for a sysadmin :(
  • 3. What's the problem? (and what the hell am I talking about?)
  • 4. WTF is going on?!? I hope my wife still recognises me... (deployments are *hard*)
  • 5. Why so hard? (just the usual suspects)
  • 6.
  • 7.
  • 8.
  • 9. How do we fix this? (and who do we blame?)
  • 11.
  • 12. with_down_in_load_balancer(server) do with_down_in_nagios(server) do puppet(server) raise "borked!" unless server.working? end end
  • 13.
  • 14.
  • 16.
  • 17.
  • 20. $ grep -r 'aa01' nagios/ | wc -l 17 $ grep -r 'aa01' puppet/ | wc -l 4 $ grep -r 'aa01' deploybot/ | wc -l 0
  • 21.
  • 22. > show lb vserver agentadmin-prod agentadmin-prod (125.56.204.120:80) - HTTP Type: ADDRESS State: UP ... blah blah ... Bound Service Groups: 1) Group Name: agentadmin 1) agentadmin (192.168.25.1: 80) - HTTP State: UP Weight: 1 Persistence Cookie Value : my_random_str=9999 2) agentadmin (192.168.25.2: 80) - HTTP State: UP Weight: 1 Persistence Cookie Value : my_random_str=9999 ... etc etc ... > show servicegroup agentadmin agentadmin - HTTP State: ENABLED Monitor Threshold : 0 Max Conn: 0 Max Req: 0 Max Bandwidth: 0 kbits Monitor Name: http-diagnositic-warmup State: ENABLED Weight: 1 1) 192.168.25.1:80 State: UP Server Name: 192.168.25.1 Probes: 131205 Failed [Total: 2525 Current: 0] Last response: Success - HTTP response code 200 received. 2) 192.168.25.2:80 State: UP Server Name: 192.168.25.2 Probes: 131322 Failed [Total: 2428 Current: 0] Last response: Success - HTTP response code 200 received.
  • 24.
  • 25. $ curl http://aa01/diagnostic/status/nagios OK - the application is functioning correctly $ curl http://ar01/diagnostic/status/nagios OK - the application is functioning correctly
  • 26. ● custom app health checks ○ https://github.com/tribune/is_it_working ○ https://github.com/blythedunham/health_monitor ● monitor requests in munin ○ https://github.com/pka/rack-monitor
  • 28.
  • 29. ● Don't bother trying to handle errors - just make sure you can recover from them quickly! ● As long as you have some app servers alive things will work out!
  • 30. FIN
  • 31. Thanks! ● http://markerguru.deviantart.com/ ● http://www.quickmeme.com/ ● http://www.doingitwrong.com/ ● Everyone who was part of @rea_autobots