The document discusses challenges with distributed applications including maintaining consistent configurations, updating code everywhere, and locating errors. It recommends setting up a centralized syslog server to collect logs from all servers and applications. This allows errors to be tracked even as servers and containers change. It also suggests automating build, deployment and configuration processes to reduce human errors and ensure consistency across environments. Tools recommended for this include Ansible, Puppet, Vagrant and tracking code changes.

1. Failing at Scale
Chris Tankersley
@dragonmantank
Pacific Northwest PHP, September 2016
Pacific Northwest PHP, September 2016 1

2. I think something is wrong
Pacific Northwest PHP, September 2016 2

3. Problems with Distributed Applications
• Making sure that configurations are consistent
• Making sure that your code gets updated everywhere
• Actually finding errors
Pacific Northwest PHP, September 2016 3

4. I know this is a bug, but where?
Pacific Northwest PHP, September 2016 4

5. You Can’t Rely on Physical Files
• Servers/Containers Disappear
• How do you know where something happened?
Pacific Northwest PHP, September 2016 5

6. syslog
Pacific Northwest PHP, September 2016 6
Your Server
PHP Application
Monolog
syslog
Centralized syslog Server
syslog

7. Set up the Server
# /etc/init.d/syslog.conf, or something similiar
# Make sure the following is set
SYSLOGD="-r -m0"
Pacific Northwest PHP, September 2016 7

8. Set up the server
# /etc/syslog.conf, or something similiar
# Set up the log file locations
*.emerg /var/log/emergency.log
*.alert /var/log/alert.log
# Or lump them together
*.emerg /var/log/application.log
*.alert /var/log/application.log
Pacific Northwest PHP, September 2016 8

9. Set up the clients
# /etc/syslog.conf, or something similiar
# Send all log types of all log severities to a
# single server
*.* @<hostname|IP Address>
Pacific Northwest PHP, September 2016 9

10. Logging SaaS
• Same idea as a remote syslog
• Usually installs some log watching software
• Pretty interface for searching/looking at logs
Pacific Northwest PHP, September 2016 10

11. Loggly
Pacific Northwest PHP, September 2016 11

12. ELK Stack
• ElasticSearch, LogStash, Kibana
• Can run locally or through a SaaS service like logz.io
Pacific Northwest PHP, September 2016 12

13. Logz.io
Pacific Northwest PHP, September 2016 13

14. But I swear I fixed that bug!
Pacific Northwest PHP, September 2016 14

15. Don’t Trust Humans
We Suck
Pacific Northwest PHP, September 2016 15

16. Anything you do by hand, Automate
Pacific Northwest PHP, September 2016 16

17. Automating the Build Process
• Composer
• Grunt/Gulp/npm/asset pipelines
• Putting proper config files in place
Pacific Northwest PHP, September 2016 17

18. Track Everything
Pacific Northwest PHP, September 2016 18

19. Automating Code Pushes
Pacific Northwest PHP, September 2016 19

20. I’m pretty sure we have PHP… 5.6
installed?
Pacific Northwest PHP, September 2016 20

21. Remember, Don’t Trust Humans!
Pacific Northwest PHP, September 2016 21

22. Ansible
• Self contained and can be run from anywhere
• Very easy to get set up and running
• Lots of “playbooks”
Pacific Northwest PHP, September 2016 22

23. Puppet
• Master-Agent Setup
• Constantly checks to make sure machines are compliant
• You might be using it already!
Pacific Northwest PHP, September 2016 23

24. Vagrant Cookbook
Pacific Northwest PHP, September 2016 24

25. Thank You!
• https://github.com/dragonmantank
• Author of “Docker for Developers”
• https://leanpub.com/dockerfordevs
• http://ctankersley.com
• chris@ctankersley.com
• @dragonmantank
• https://joind.in/talk/64ad5
Pacific Northwest PHP, September 2016 25