Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

5 Bare Minimum Things A Web Startup CTO Must Worry About

So you have started-it-up and now you are getting good traffic — Thousands of users, etc. etc.
Do you know script kiddies are scanning your website using simple dictionary attacks on SSH ports? Do you know that once in a while there is a Fatal application Error in your PHP log (which may point to bigger problem)? Do you know that the backup you are taking is actually not gonna restore your DB? Do you know that every night at 12 one of the servers has a CPU spike?
It’s a good idea to catch some of the serious problems early on and deploy tools to proactively assess them. In this session we will discuss some very basic things, as a CTO you MUST worry about and proactively solve problems around them.
These are (in the order of decreasing priority):
1. Security
2. Monitoring/Availability/Load (External/System level)
3. Application errors
4. Backup
5. Source control

  • Login to see the comments

5 Bare Minimum Things A Web Startup CTO Must Worry About

  1. 1. 5 Bare Minimum Things a Web startup CTO MUST worry about Indus Khaitan [email_address] Twitter: 1ndus *Not affiliated to any software vendors mentioned in this preso … and implement few basic things to have a good night’s sleep! 18 slides
  2. 2. What are these? <ul><li>Security </li></ul><ul><li>Availability & Monitoring </li></ul><ul><li>Application Errors </li></ul><ul><li>Backup </li></ul><ul><li>Source Control </li></ul>(in order of decreasing priority)
  3. 3. Security Threats <ul><li>Your website taken over </li></ul><ul><li>Your database taken over </li></ul><ul><li>Your server taken over </li></ul><ul><li>(Distributed) Denial of Service </li></ul>
  4. 4. Prevention of Security Threats <ul><li>Keep your stack up-to-date. Patch. </li></ul><ul><li>Establish security-aware coding practice </li></ul><ul><li>Know your Logs! </li></ul><ul><li>Install open source packages for preventive/reactive treatments </li></ul><ul><li>Get a hardware firewall (if you are popular and have money) </li></ul><ul><li>… Subscribe to Securityfocus alerts </li></ul>
  5. 5. Simple TODO List for You <ul><li>Use logwatch and monitor your logs </li></ul><ul><li>Make your Database access local (specific IPs only) </li></ul><ul><li>Secure your sshd </li></ul><ul><ul><li>Password-less login, non-default port, no root login </li></ul></ul><ul><li>Use denyhosts to block dictionary SSH attacks (iptables/netfilter is a good bet, I haven’t tried it) </li></ul><ul><li>Close all ports except SSH, HTTP/HTTPS </li></ul><ul><ul><li>Use nmap to see what “hackers” see! </li></ul></ul>
  6. 6. A log snapshot of SSH attack <ul><li>Didn't receive an ident from these IPs: </li></ul><ul><li> 1 Time(s) </li></ul><ul><li>Illegal users from: </li></ul><ul><li> 6 times </li></ul><ul><li>alias/password: 1 time </li></ul><ul><li>office/password: 1 time </li></ul><ul><li>recruit/password: 1 time </li></ul><ul><li>sales/password: 1 time </li></ul><ul><li>samba/password: 1 time </li></ul><ul><li>staff/password: 1 time </li></ul><ul><li>Failed logins from: </li></ul><ul><li> 1 time </li></ul><ul><li>root/password: 1 time </li></ul><ul><li> 1 time </li></ul><ul><li>root/password: 1 time </li></ul>
  7. 7. Availability & Monitoring <ul><li>Website, Database, SMTP, DNS were down (now up!) </li></ul><ul><li>Poor site performance </li></ul><ul><ul><li>Application, Network, or hosting provider? </li></ul></ul><ul><li>CPU, Disk, IO, Memory, Network Interface </li></ul><ul><li>Server down != website down. Put a load balancer </li></ul>
  8. 8. Monitoring – External sample
  9. 9. Monitoring: Internal System Level Monitoring with Nagios
  10. 10. Simple TODO List for You <ul><li>Do some basic external monitoring </li></ul><ul><ul><li>Zoho does url/5minutes at $4/! </li></ul></ul><ul><li>Get Nagios for system monitoring </li></ul><ul><li>Use Load Balancer to prevent single server failure </li></ul><ul><ul><li>HTTP, Load Balanced database reads </li></ul></ul>
  11. 11. Application Errors <ul><li>Bad Code </li></ul><ul><ul><li>function validate($key) { </li></ul></ul><ul><ul><li>global $weblog ; </li></ul></ul><ul><ul><li>if (empty($key)) { </li></ul></ul><ul><ul><li>$errorlog->error( &quot;Error : In function validate site key&quot;); </li></ul></ul><ul><ul><li>return FALSE; </li></ul></ul><ul><ul><li>}else{ </li></ul></ul><ul><ul><li>return TRUE; </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>Leads to this in phperror log </li></ul><ul><li>[13-Feb-2009 09:41:32] PHP Fatal error: Call to a member function error() on a non-object in /home/padmin/public_html/util/functions.php on line 4 </li></ul>
  12. 12. Application Errors <ul><li>Simple WARNINGS/FATALs lead to bigger problems </li></ul><ul><ul><li>eg. INSERT failed because of duplicate key (was always inserting 0 for the parameter!) </li></ul></ul><ul><li>Apache error_log may show wrong configuration </li></ul><ul><li>Database logs may show a crash (and auto-recovery!) </li></ul>
  13. 13. Simple TODO for You <ul><li>Use a logger like log4j/log4PHP </li></ul><ul><ul><li>Modify the handler to send a real-time email of a desired error level </li></ul></ul><ul><li>Look for Database Error logs, Apache error logs – They will tell you a story! </li></ul><ul><li>Borrow from Security: Use logwatch package </li></ul><ul><li>Review your own application codebase </li></ul>
  14. 14. Backup <ul><li>Backup before disaster strikes </li></ul><ul><li>Database backups </li></ul><ul><ul><li>Do a dry run of recovery at least once </li></ul></ul><ul><ul><li>Ensure consistent, online backups </li></ul></ul><ul><li>Backup your production directories </li></ul>
  15. 15. Simple TODO For You <ul><li>(mysql) Use a slave for a consistent backup. No slave? Then Lock the master before dumping </li></ul><ul><li>Take a backup tar of production </li></ul><ul><ul><li>Preferably backed up every week, and just before a deployment and just after a deployment </li></ul></ul><ul><li>Use S3 to store the files remotely </li></ul>
  16. 16. Source Control: Simple TODO For You <ul><li>Use SVN </li></ul><ul><ul><li>Use hosted… DevGuard..$7/! </li></ul></ul><ul><li>Few Developers? Can’t do Linux? No money? Use a local SVN server on Windows. Woorrks! But back-it-up!!! </li></ul><ul><li>Have a prod. deployment strategy </li></ul><ul><ul><li>From SVN, DON’T deploy directly on Prod., use a separate instance and then scp/rsync over </li></ul></ul>
  17. 17. Summary <ul><li>Know Your Logs! </li></ul><ul><li>Be Security aware </li></ul><ul><ul><li>Lock your SSH. Close Open Ports </li></ul></ul><ul><li>Do some basic external monitoring </li></ul><ul><li>Backup your Database & prod directory onto a remote location </li></ul><ul><li>Use SVN </li></ul>
  18. 18. Sample Advanced Topics & Thanks! <ul><li>Incremental backups, snapshots </li></ul><ul><li>Monitoring Apache Processes, Apache IO, Database connections, Load, Query/sec </li></ul><ul><li>Using SSH Tunneling </li></ul><ul><li>Virtual Private & Public LANs </li></ul><ul><li>VPN </li></ul>