5 Bare Minimum Things a Web startup CTO MUST worry about Indus Khaitan http://khaitan.org [email_address] Twitter: 1ndus *Not affiliated to any software vendors mentioned in this preso … and implement few basic things to have a good night’s sleep! 18 slides

What are these? Security Availability & Monitoring Application Errors Backup Source Control (in order of decreasing priority)

Security

Availability & Monitoring

Application Errors

Backup

Source Control

Security Threats Your website taken over Your database taken over Your server taken over (Distributed) Denial of Service

Your website taken over

Your database taken over

Your server taken over

(Distributed) Denial of Service

Prevention of Security Threats Keep your stack up-to-date. Patch. Establish security-aware coding practice Know your Logs! Install open source packages for preventive/reactive treatments Get a hardware firewall (if you are popular and have money) … Subscribe to Securityfocus alerts

Keep your stack up-to-date. Patch.

Establish security-aware coding practice

Know your Logs!

Install open source packages for preventive/reactive treatments

Get a hardware firewall (if you are popular and have money)

… Subscribe to Securityfocus alerts

Simple TODO List for You Use logwatch and monitor your logs Make your Database access local (specific IPs only) Secure your sshd Password-less login, non-default port, no root login Use denyhosts to block dictionary SSH attacks (iptables/netfilter is a good bet, I haven’t tried it) Close all ports except SSH, HTTP/HTTPS Use nmap to see what “hackers” see!

Use logwatch and monitor your logs

Make your Database access local (specific IPs only)

Secure your sshd

Password-less login, non-default port, no root login

Use denyhosts to block dictionary SSH attacks (iptables/netfilter is a good bet, I haven’t tried it)

Close all ports except SSH, HTTP/HTTPS

Use nmap to see what “hackers” see!

A log snapshot of SSH attack Didn't receive an ident from these IPs: 114.200.199.144: 1 Time(s) Illegal users from: 114.200.199.144: 6 times alias/password: 1 time office/password: 1 time recruit/password: 1 time sales/password: 1 time samba/password: 1 time staff/password: 1 time Failed logins from: 211.60.15.30: 1 time root/password: 1 time 219.137.24.12: 1 time root/password: 1 time

Didn't receive an ident from these IPs:

114.200.199.144: 1 Time(s)

Illegal users from:

114.200.199.144: 6 times

alias/password: 1 time

office/password: 1 time

recruit/password: 1 time

sales/password: 1 time

samba/password: 1 time

staff/password: 1 time

Failed logins from:

211.60.15.30: 1 time

root/password: 1 time

219.137.24.12: 1 time

root/password: 1 time

Availability & Monitoring Website, Database, SMTP, DNS were down (now up!) Poor site performance Application, Network, or hosting provider? CPU, Disk, IO, Memory, Network Interface Server down != website down. Put a load balancer

Website, Database, SMTP, DNS were down (now up!)

Poor site performance

Application, Network, or hosting provider?

CPU, Disk, IO, Memory, Network Interface

Server down != website down. Put a load balancer

Monitoring – External sample

Monitoring: Internal System Level Monitoring with Nagios

Simple TODO List for You Do some basic external monitoring Zoho does url/5minutes at $4/month...cheap! Get Nagios for system monitoring Use Load Balancer to prevent single server failure HTTP, Load Balanced database reads

Do some basic external monitoring

Zoho does url/5minutes at $4/month...cheap!

Get Nagios for system monitoring

Use Load Balancer to prevent single server failure

HTTP, Load Balanced database reads

Application Errors Bad Code function validate($key) { global $weblog ; if (empty($key)) { $errorlog->error( "Error : In function validate site key"); return FALSE; }else{ return TRUE; } } Leads to this in phperror log [13-Feb-2009 09:41:32] PHP Fatal error: Call to a member function error() on a non-object in /home/padmin/public_html/util/functions.php on line 4

Bad Code

function validate($key) {

global $weblog ;

if (empty($key)) {

$errorlog->error( "Error : In function validate site key");

return FALSE;

}else{

return TRUE;

}

}

Leads to this in phperror log

[13-Feb-2009 09:41:32] PHP Fatal error: Call to a member function error() on a non-object in /home/padmin/public_html/util/functions.php on line 4

Application Errors Simple WARNINGS/FATALs lead to bigger problems eg. INSERT failed because of duplicate key (was always inserting 0 for the parameter!) Apache error_log may show wrong configuration Database logs may show a crash (and auto-recovery!)

Simple WARNINGS/FATALs lead to bigger problems

eg. INSERT failed because of duplicate key (was always inserting 0 for the parameter!)

Apache error_log may show wrong configuration

Database logs may show a crash (and auto-recovery!)

Simple TODO for You Use a logger like log4j/log4PHP Modify the handler to send a real-time email of a desired error level Look for Database Error logs, Apache error logs – They will tell you a story! Borrow from Security: Use logwatch package Review your own application codebase

Use a logger like log4j/log4PHP

Modify the handler to send a real-time email of a desired error level

Look for Database Error logs, Apache error logs – They will tell you a story!

Borrow from Security: Use logwatch package

Review your own application codebase

Backup Backup before disaster strikes Database backups Do a dry run of recovery at least once Ensure consistent, online backups Backup your production directories

Backup before disaster strikes

Database backups

Do a dry run of recovery at least once

Ensure consistent, online backups

Backup your production directories

Simple TODO For You (mysql) Use a slave for a consistent backup. No slave? Then Lock the master before dumping Take a backup tar of production Preferably backed up every week, and just before a deployment and just after a deployment Use S3 to store the files remotely

(mysql) Use a slave for a consistent backup. No slave? Then Lock the master before dumping

Take a backup tar of production

Preferably backed up every week, and just before a deployment and just after a deployment

Use S3 to store the files remotely

Source Control: Simple TODO For You Use SVN Use hosted… DevGuard..$7/month..cheap! Few Developers? Can’t do Linux? No money? Use a local SVN server on Windows. Woorrks! But back-it-up!!! Have a prod. deployment strategy From SVN, DON’T deploy directly on Prod., use a separate instance and then scp/rsync over

Use SVN

Use hosted… DevGuard..$7/month..cheap!

Few Developers? Can’t do Linux? No money? Use a local SVN server on Windows. Woorrks! But back-it-up!!!

Have a prod. deployment strategy

From SVN, DON’T deploy directly on Prod., use a separate instance and then scp/rsync over

Summary Know Your Logs! Be Security aware Lock your SSH. Close Open Ports Do some basic external monitoring Backup your Database & prod directory onto a remote location Use SVN

Know Your Logs!

Be Security aware

Lock your SSH. Close Open Ports

Do some basic external monitoring

Backup your Database & prod directory onto a remote location

Use SVN

Sample Advanced Topics & Thanks! Incremental backups, snapshots Monitoring Apache Processes, Apache IO, Database connections, Load, Query/sec Using SSH Tunneling Virtual Private & Public LANs VPN

Incremental backups, snapshots

Monitoring Apache Processes, Apache IO, Database connections, Load, Query/sec

Using SSH Tunneling

Virtual Private & Public LANs

VPN