Dev Ops:
the next generation
Warning!
meme and star trek heavy
Who is this TREKKIE
Barney Hanlon
Technical Team Leader for Inviqa
@shrikeh
We’re hiring!
Dev ops
The story so far
(Yes I know that’s Star Wars)
Sysadmins

Developers

DevOps
“Get the code on the server”
FTP

CVS

Subversion

Git
What about all this
OTHER stuff?
•

Setting up Virtual hosts

•

User management

•

SSH access

•

File permissions

•

L...
Developers need to know
more than how to push to
Github.
FIVE STAR DevOps
The Five Stars of DevOps
The Five Stars of DevOps
•

Monitoring
The Five Stars of DevOps
•

Monitoring

•

Security
The Five Stars of DevOps
•

Monitoring

•

Security

•

Performance
The Five Stars of DevOps
•

Monitoring

•

Security

•

Performance

•

Automation
The Five Stars of DevOps
•

Monitoring

•

Security

•

Performance

•

Automation

•

Scaleability
Am I doing something in my
application that is done better by
the infrastructure or an external
service?
Probably.
Got root?
Common reasons to 

“just log onto the server quickly”
•

server logs in /var/log require privileges to
tail

•

setting p...
Monitoring
Monitoring
•

Log EVERYTHING

•

Drupal default visitor logging is heavy

•

Should you be writing to the database to
log ...
NO.
Monitoring
•

Logging is only one part of monitoring

•

Send your Web logs to a remote service

•

Set error_log to syslo...
Logging Services
•

SplunkStorm

•

Loggly

•

Logentries

•

Papertrailapp
HOSTING YOUR OWN LOGGING
•

Splunk

•

GrayLog2

•

Sensu

•

Munin

•

Raven
Other monitoring
•

Nagios

•

Pingdom

•

New Relic

•

Piwik/Google Analytics
Profiling
•

Don’t be afraid to turn XHProf on in live
occasionally

•

Regularly check your browser HAR

•

Check APC and...
Security
Where is the risk?
•

Application security

•

Infrastructure security

•

End user security
Repelling Unwelcome Guests
Tools to help
•

JumpCloud

•

DuoSecurity

•

Ubuntu ACL
Capturing Morpheus…

Not so bad.
Hardening SSL
Don’t bother
hardening SSL
SSL is Dead, Long Live TLS
•

No one should be using SSL any more.

•

Transport Layer Security (TLS)

•

Latest version 1...
Vulnerabilities
•

BEAST Attack

•

CRIME Attack

•

Lucky Thirteen
HTTPS without proper ciphers gives
the illusion of security while
providing none
Default SSL implementations
Nginx
ssl_protocols SSLv3 TLSv1;
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SS...
“Today, only TLS 1.2 with GCM suites offer fully
robust security. All other suites suffer from one
problem or another (e.g...
“…Because GCM suites are not yet widely
supported, most communication today is carried
out using one of the slightly flawe...
Diffie-Hellman Key
Exchange
•

Diffie-Hellman (DH) and Elliptic Curve
Diffie-Hellman (ECDH)

•

Allows Perfect Forward Sec...
server {
…

!

add_header Strict-Transport-Security “max-age=31536000; includeSubDomains";
ssl_session_cache shared:SSL:10...
Test your strength!

https://www.ssllabs.com/ssltest/
“I don’t mind that the site is
slow, at least my data is safe.”
–No user ever
Performance
SPDY
SPDY
•

Draft HTTP 2.0

•

Allows multiplexing a single connection

•

Requires HTTPS

•

Do you need HTTP at all?
PageSpeed
•

Library for Apache and Nginx

•

Automatic minification of JavaScript, CSS
and HTML

•

On-the-fly optimisati...
The “SPDY Sandwich”

Nginx

Varnish

Nginx

PHP-FPM
The “SPDY Sandwich”

Nginx

Varnish

Responsible for:
• SPDY / SSL Termination
• Serving static assets
• Gzipping
• Pagesp...
The “SPDY Sandwich”
Responsible for:
• Caching dynamic pages
• Cookie normalisation
Nginx

Varnish

Nginx

PHP-FPM
The “SPDY Sandwich”

Nginx

Varnish

Nginx

PHP-FPM

Responsible for:
• Serving dynamic pages
• Generic Pagespeed optimisa...
The “SPDY Sandwich”
Responsible for:
• PHP interpreter
Nginx

Varnish

Nginx

PHP-FPM
Cookies
Cross Site Request Forgery
(CSRF)
•

OWASP recommendation

•

Requires a token in the form and a session
token

•

Breaks ...
Am I doing something in my
application that is done better by
the infrastructure or an external
service?
OpenResty
•

Nginx bundle

•

Has modules for connecting to Redis,
Drizzle, memcached and many more

•

Has Lua to allow p...
Time for a
Simplified Example
<?php
!

namespace InviqaDrupalCampAccess;
!

class OpenRestyTokenGenerator implements
CsrfTokenGeneratorInterface
{
priva...
<?php
namespace InviqaDrupalCampForm;
!

use DrupalCoreFormFormBuilder as CoreFormBuilder;
!

class FormBuilder extends Co...
FIXING CSRF WITH LUA
X-CSRF-Tokenize: “[[CSRF Here]]”

OpenResty

Redis

Varnish

Nginx

PHP-FPM
FIXING CSRF WITH LUA
Header cached in Varnish

OpenResty

Redis

Varnish

Nginx

PHP-FPM
FIXING CSRF WITH LUA
•
•
•

OpenResty

Redis

Parses response (regex)!
Finds token placeholder!
Replaces with real token

...
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!

header_filter_by_lua!
'!
if ngx.var.upstream_http_x_cs...
FIXING CSRF WITH LUA

OpenResty

Varnish

•
Redis

•

Nginx

PHP-FPM

Stores HMAC as value in Redis with

random key “csrf...
!
! ! -- we are about to mess around with the content of the page!
! ! -- so we need to clear this as it will be wrong!
! ...
FIXING CSRF WITH LUA
Check token on way back in

OpenResty

Redis

Varnish

Nginx

PHP-FPM
location @backend {!
! ! # You can't set variables in nginx dynamically, !
# so set this up as empty first!
! ! set $csrf_...
Full gist:
https://gist.github.com/shrikeh/4722427
Automation
Tooling
•

Bash scripts (!)

•

Chef/Puppet (retro)

•

Ansible!
Ansible
•

Requires no agent!

•

Pure SSH

•

Modules

•

YAML-based configuration

•

Playbooks (and playbooks of playbo...
Ansible PLAYBOOK
!
- name: ensure SSH key exists
digital_ocean: >
state=present
command=ssh
name=case
- name: ensure dropl...
Ansible

ansible-playbook base.yml -vvv -i "hosts/production"
Configuring Your Application
<?php
# /sites/default/settings.php
...
$databases['default']['default'] = array(
'driver'
=> 'mysql',
'database' => 'drup...
“A litmus test for whether an app has all
config correctly factored out of the code is
whether the codebase could be made ...
“The twelve-factor app stores config in
environment variables (often shortened to
env vars or env). Env vars are easy to
c...
Put your variables in PHP-FPM
/etc/php/fpm/pools/live.conf

env[db_name]
env[db_host]
env[db_user]
env[db_pass]
env[db_pre...
<?php
# /sites/default/settings.php
...
$databases['default']['default'] = array(
'driver'
=> 'mysql',
'database' => geten...
Provisioning
•

Idempotent deployments

•

Provision every environment the same way

•

Resist the urge to do something ma...
Docker
Docker - AN OVERVIEW
•

Lightweight Linux
Container

•

Portable environment

•

Install all your PECL
dependencies into a...
Problems
Problems
!

•

Still heavily in development, no “right way”
yet

•

Hard to set up syslog inside a container

•

Runs as r...
That’s being fixed though
We’re Done!
With thanks to
Paramount Pictures
and
startrek.wikia.com
for not suing me
Questions
Thank You!
My first official talk!
•

Special thanks to Lorna Mitchell, Ian
Barber and Rowan Merewood for all the
coaching

•

All fe...
Next Generation DevOps in Drupal: DrupalCamp London 2014
Next Generation DevOps in Drupal: DrupalCamp London 2014
Next Generation DevOps in Drupal: DrupalCamp London 2014
Next Generation DevOps in Drupal: DrupalCamp London 2014
Next Generation DevOps in Drupal: DrupalCamp London 2014
Next Generation DevOps in Drupal: DrupalCamp London 2014
Next Generation DevOps in Drupal: DrupalCamp London 2014
Next Generation DevOps in Drupal: DrupalCamp London 2014
Next Generation DevOps in Drupal: DrupalCamp London 2014
Next Generation DevOps in Drupal: DrupalCamp London 2014
Upcoming SlideShare
Loading in …5
×

Next Generation DevOps in Drupal: DrupalCamp London 2014

2,508 views
2,358 views

Published on

In this talk, Barney will be discussing and demonstrating how to:
- Use nginx, Varnish and Apache together in a "SPDY sandwich" to support HTTP 2.0
- Setting up SSL properly to mitigate against attack vectors
- Performance improvements with mod_pagespeed and nginx
- Deploying Drupal sites with Docker containers

Barney is a Technical Team Leader at Inviqa, a Drupal Association member and writes for Techportal on using technologies to improve website performance. He first started using PHP professionally in 2003, and has over seventeen years experience in software development. He is an advocate of Scrum methodology and has an interest in performance optimization, researching and speaking on various techniques to improve user experience through faster load times.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,508
On SlideShare
0
From Embeds
0
Number of Embeds
42
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Next Generation DevOps in Drupal: DrupalCamp London 2014

  1. 1. Dev Ops: the next generation
  2. 2. Warning! meme and star trek heavy
  3. 3. Who is this TREKKIE Barney Hanlon Technical Team Leader for Inviqa @shrikeh
  4. 4. We’re hiring!
  5. 5. Dev ops The story so far (Yes I know that’s Star Wars)
  6. 6. Sysadmins Developers DevOps
  7. 7. “Get the code on the server”
  8. 8. FTP CVS Subversion Git
  9. 9. What about all this OTHER stuff? • Setting up Virtual hosts • User management • SSH access • File permissions • Log rotation • Firewall rules • Staging servers • Patching • Build pipelines • Concurrency • SSL certificates • Failover • Minification • Alerting
  10. 10. Developers need to know more than how to push to Github.
  11. 11. FIVE STAR DevOps
  12. 12. The Five Stars of DevOps
  13. 13. The Five Stars of DevOps • Monitoring
  14. 14. The Five Stars of DevOps • Monitoring • Security
  15. 15. The Five Stars of DevOps • Monitoring • Security • Performance
  16. 16. The Five Stars of DevOps • Monitoring • Security • Performance • Automation
  17. 17. The Five Stars of DevOps • Monitoring • Security • Performance • Automation • Scaleability
  18. 18. Am I doing something in my application that is done better by the infrastructure or an external service?
  19. 19. Probably.
  20. 20. Got root?
  21. 21. Common reasons to 
 “just log onto the server quickly” • server logs in /var/log require privileges to tail • setting permissions on directories • Processes require restarting
  22. 22. Monitoring
  23. 23. Monitoring • Log EVERYTHING • Drupal default visitor logging is heavy • Should you be writing to the database to log visits?
  24. 24. NO.
  25. 25. Monitoring • Logging is only one part of monitoring • Send your Web logs to a remote service • Set error_log to syslog in php.ini
  26. 26. Logging Services • SplunkStorm • Loggly • Logentries • Papertrailapp
  27. 27. HOSTING YOUR OWN LOGGING • Splunk • GrayLog2 • Sensu • Munin • Raven
  28. 28. Other monitoring • Nagios • Pingdom • New Relic • Piwik/Google Analytics
  29. 29. Profiling • Don’t be afraid to turn XHProf on in live occasionally • Regularly check your browser HAR • Check APC and other caches for smells
  30. 30. Security
  31. 31. Where is the risk? • Application security • Infrastructure security • End user security
  32. 32. Repelling Unwelcome Guests
  33. 33. Tools to help • JumpCloud • DuoSecurity • Ubuntu ACL
  34. 34. Capturing Morpheus… Not so bad.
  35. 35. Hardening SSL
  36. 36. Don’t bother hardening SSL
  37. 37. SSL is Dead, Long Live TLS • No one should be using SSL any more. • Transport Layer Security (TLS) • Latest version 1.2
  38. 38. Vulnerabilities • BEAST Attack • CRIME Attack • Lucky Thirteen
  39. 39. HTTPS without proper ciphers gives the illusion of security while providing none
  40. 40. Default SSL implementations Nginx ssl_protocols SSLv3 TLSv1; ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP; Open to Lucky Thirteen attack Supporting SSLv3 is only required for IE6 Apache 2 SSLProtocol all SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5
  41. 41. “Today, only TLS 1.2 with GCM suites offer fully robust security. All other suites suffer from one problem or another (e.g, RC4, Lucky 13, BEAST), but most are difficult to exploit in practice…” –Ivan Ristic, Qualys
  42. 42. “…Because GCM suites are not yet widely supported, most communication today is carried out using one of the slightly flawed cipher suites. It is not possible to do better if you're running a public web site.” –Ivan Ristic, Qualys
  43. 43. Diffie-Hellman Key Exchange • Diffie-Hellman (DH) and Elliptic Curve Diffie-Hellman (ECDH) • Allows Perfect Forward Secrecy • Slow :(
  44. 44. server { … ! add_header Strict-Transport-Security “max-age=31536000; includeSubDomains"; ssl_session_cache shared:SSL:10m; ssl_prefer_server_ciphers on; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH +3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS; ssl_ecdh_curve secp521r1; } https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
  45. 45. Test your strength! https://www.ssllabs.com/ssltest/
  46. 46. “I don’t mind that the site is slow, at least my data is safe.” –No user ever
  47. 47. Performance
  48. 48. SPDY
  49. 49. SPDY • Draft HTTP 2.0 • Allows multiplexing a single connection • Requires HTTPS • Do you need HTTP at all?
  50. 50. PageSpeed • Library for Apache and Nginx • Automatic minification of JavaScript, CSS and HTML • On-the-fly optimisations based on chosen filters
  51. 51. The “SPDY Sandwich” Nginx Varnish Nginx PHP-FPM
  52. 52. The “SPDY Sandwich” Nginx Varnish Responsible for: • SPDY / SSL Termination • Serving static assets • Gzipping • Pagespeed is user agent-aware Nginx PHP-FPM
  53. 53. The “SPDY Sandwich” Responsible for: • Caching dynamic pages • Cookie normalisation Nginx Varnish Nginx PHP-FPM
  54. 54. The “SPDY Sandwich” Nginx Varnish Nginx PHP-FPM Responsible for: • Serving dynamic pages • Generic Pagespeed optimisations
  55. 55. The “SPDY Sandwich” Responsible for: • PHP interpreter Nginx Varnish Nginx PHP-FPM
  56. 56. Cookies
  57. 57. Cross Site Request Forgery (CSRF) • OWASP recommendation • Requires a token in the form and a session token • Breaks most reverse proxies without configuration
  58. 58. Am I doing something in my application that is done better by the infrastructure or an external service?
  59. 59. OpenResty • Nginx bundle • Has modules for connecting to Redis, Drizzle, memcached and many more • Has Lua to allow pre and post processing on requests and responses
  60. 60. Time for a Simplified Example
  61. 61. <?php ! namespace InviqaDrupalCampAccess; ! class OpenRestyTokenGenerator implements CsrfTokenGeneratorInterface { private $token; ! public function __construct($csrfToken) { $this->token = $csrfToken; } ! public function get($value = '') { return $this->token; } }
  62. 62. <?php namespace InviqaDrupalCampForm; ! use DrupalCoreFormFormBuilder as CoreFormBuilder; ! class FormBuilder extends CoreFormBuilder { public function __construct( ModuleHandlerInterface $module_handler, KeyValueExpirableFactoryInterface $key_value_expirable_factory, EventDispatcherInterface $event_dispatcher, UrlGeneratorInterface $url_generator, TranslationInterface $translation_manager, CsrfTokenGeneratorInterface $csrf_token = NULL, HttpKernel $http_kernel = NULL ) { …
  63. 63. FIXING CSRF WITH LUA X-CSRF-Tokenize: “[[CSRF Here]]” OpenResty Redis Varnish Nginx PHP-FPM
  64. 64. FIXING CSRF WITH LUA Header cached in Varnish OpenResty Redis Varnish Nginx PHP-FPM
  65. 65. FIXING CSRF WITH LUA • • • OpenResty Redis Parses response (regex)! Finds token placeholder! Replaces with real token Varnish Nginx PHP-FPM
  66. 66. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! header_filter_by_lua! '! if ngx.var.upstream_http_x_csrf_tokenize then! ! -- the backend requested a CSRF token be set! ! local csrf_cookie_token = nil! ! if ngx.var.cookie_csrf then! ! ! -- they have a cookie, just re-use it! ! ! local csrf_cookie_token = ngx.var.cookie_csrf! ! end! ! ! local resty_random = require "resty.random"! ! ! local str = require "resty.string"! ! ! ! ! ! ! ! ! if not csrf_cookie_token then! ! -- no valid csrf cookie found, let us make one! ! ! ! local cookie_random = resty_random.bytes(16,true)! ! ! ! ! ! ! ! ! ! ! ! ! ! while cookie_random == nil do! ! -- attempt to generate 16 bytes of! ! -- cryptographically strong (enough) random data! ! cookie_random = resty_random.bytes(16,true)! end!
  67. 67. FIXING CSRF WITH LUA OpenResty Varnish • Redis • Nginx PHP-FPM Stores HMAC as value in Redis with
 random key “csrf_” Generates cookie with Redis key as
 value
  68. 68. ! ! ! -- we are about to mess around with the content of the page! ! ! -- so we need to clear this as it will be wrong! ! ! ngx.header.Content_Length = ""! ! ! ! ! ! -- set the Cookie for the CSRF token! ! ! ngx.header.Set_Cookie = "csrf=" .. ngx.var.csrf_cookie_token! ! ! ngx.header.tokenize = ngx.var.upstream_http_x_csrf_tokenize! ! ! ! -- now generate one for the form token! ! ! while form_random == nil do! ! ! form_random = resty_random.bytes(16,true)! ! ! ! end! ! ! ! ngx.var.csrf_form_token = str.to_hex(form_random)! ! ! ! local redis = require "redis"! ! ! local client = redis.connect("127.0.0.1", 6379)! ! ! ! client:set("csrf_" .. ngx.var.csrf_cookie_token, ngx.var.csrf_form_token)! ! ! end!
  69. 69. FIXING CSRF WITH LUA Check token on way back in OpenResty Redis Varnish Nginx PHP-FPM
  70. 70. location @backend {! ! ! # You can't set variables in nginx dynamically, ! # so set this up as empty first! ! ! set $csrf_validate "";! ! ! access_by_lua ! ! ! '! if ngx.req.get_method() == "POST" then! ! -- set up forbidden as default! ! ngx.var.csrf_validate = ngx.HTTP_FORBIDDEN! ! if ngx.var.cookie_csrf then! ! ! ! ! local res = ngx.location.capture("/validate-csrf")! ! ! ! ! if ngx.HTTP_OK == res.status then! ! ! ! ! ngx.req.read_body()! ! ! ! local args = ngx.req.get_post_args()! ! ! ! ! local posted_token = tostring(args["csrf"])! ! ! ! ! if posted_token == res.body then! ! ! ! ! ! ngx.var.csrf_validate = ngx.HTTP_OK! ! ! ! ! end! ! ! ! end! ! ! ! end! ! end! ! ';
  71. 71. Full gist: https://gist.github.com/shrikeh/4722427
  72. 72. Automation
  73. 73. Tooling • Bash scripts (!) • Chef/Puppet (retro) • Ansible!
  74. 74. Ansible • Requires no agent! • Pure SSH • Modules • YAML-based configuration • Playbooks (and playbooks of playbooks)
  75. 75. Ansible PLAYBOOK ! - name: ensure SSH key exists digital_ocean: > state=present command=ssh name=case - name: ensure droplet exists digital_ocean: > state=present ssh_key_ids=57705 name={{ inventory_hostname }} size_id=66 region_id=4 image_id=1505699 wait_timeout=500 private_networking=yes virtio=yes wait=yes unique_name=yes wait_timeout=500 register: launched - debug: msg="IP is {{ launched.droplet.id }}" - debug: msg="IP is {{ launched.droplet.ip_address }}"
  76. 76. Ansible ansible-playbook base.yml -vvv -i "hosts/production"
  77. 77. Configuring Your Application
  78. 78. <?php # /sites/default/settings.php ... $databases['default']['default'] = array( 'driver' => 'mysql', 'database' => 'drupal', 'username' => 'testuser', 'password' => '123secure', 'host' => 'localhost', 'prefix' => '', );
  79. 79. “A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials” –The Twelve Factor App
  80. 80. “The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code” –The Twelve Factor App
  81. 81. Put your variables in PHP-FPM /etc/php/fpm/pools/live.conf env[db_name] env[db_host] env[db_user] env[db_pass] env[db_prefix] = = = = = drupal_live 192.168.0.2 liveuser verysecurepass drupalcamp_
  82. 82. <?php # /sites/default/settings.php ... $databases['default']['default'] = array( 'driver' => 'mysql', 'database' => getenv('db_name'), 'username' => getenv('db_user'), 'password' => getenv('db_pass'), 'host' => getenv('db_host'), 'prefix' => getenv('db_prefix'), );
  83. 83. Provisioning • Idempotent deployments • Provision every environment the same way • Resist the urge to do something manually • Get into a workflow of automation
  84. 84. Docker
  85. 85. Docker - AN OVERVIEW • Lightweight Linux Container • Portable environment • Install all your PECL dependencies into a container • Ship it
  86. 86. Problems
  87. 87. Problems ! • Still heavily in development, no “right way” yet • Hard to set up syslog inside a container • Runs as root on the box
  88. 88. That’s being fixed though
  89. 89. We’re Done!
  90. 90. With thanks to Paramount Pictures and startrek.wikia.com for not suing me
  91. 91. Questions
  92. 92. Thank You!
  93. 93. My first official talk! • Special thanks to Lorna Mitchell, Ian Barber and Rowan Merewood for all the coaching • All feedback welcome!

×