What Do You Need to Know About
                Creating and Running a Scalable
                Web Site but Were Afraid to...
Learn some of the techniques we
 used to create and build
 zembly.com




           2009 CommunityOne WEST Conference | s...
Agenda

Introduction
Development process
Architecture & deployment
Availability & Performance




                  2009 C...
Agenda

Introduction
Development process
Architecture & deployment
Availability & Performance




                  2009 C...
Introduction - zembly.com

 Browser based social application development
environment (wiki for applications)
•Content Mana...
Introduction – software supporting zembly


 Development Tools
•NetBeans
•Subversion
•kenai.com
•Hudson
•Automated Testing...
Introduction – we are zembly

  Engineering Team – Chris
•Platform
•Girish
•user interaction
  Site engineering - Girish
•...
Agenda

Introduction
Development process
Architecture & deployment
Availability & Performance




                  2009 C...
Development process - overview

 Agile development - 3 week sprints
 Everyone can nominate work for the upcoming sprint
 P...
Development process - planning

 Strategic direction
 Engineering discussions before nominations
 Tasks must have
•Clear d...
Development process - code

 Pair programming
 Continuous Build
•compiler errors
•unit test failures
•JSLint – for java sc...
Development process – tests

  Deploy successful build for functional and automated UI
tests
•functional tests (blackbox t...
Development Process – code lines

  3 code lines
•trunk – must pass all automated tests
•staging – created from specific t...
Development process - experience

 No nominations
 Nominations without details
 Growing backlog
 Looooooonnnnnngggg planni...
Agenda

Introduction
Development process
Architecture & deployment
Availability & Performance




                  2009 C...
w
                    ik




                                                        ET
                                ze...
zembly database setup
                                        Master DB
Local                               All writes go ...
Architecture - experiences

  Perpetual architecture
•start simple and grow organic
•development    and maintenance inline...
Agenda

Introduction
Development process
Architecture & deployment
Availability & Performance




                  2009 C...
w
                    ik




                                                        ET
                                ze...
w
                    ik




                                                        ET
                                ze...
w
                             ik




                                                                 ET
                ...
w
                    ik




                                                        ET
                                ze...
w
                     ik




                                                         ET
                                ...
Architecture & Deployment - deployment

  Rolling deployments
•automated homegrown script
•turn   off monitoring
•no user ...
Agenda

Introduction
Development process
Architecture & deployment
Availability & Performance




                  2009 C...
Availability – experiences

 differences between staging and production
 hardware failure
 unmonitored service
 late night...
Availability - failover

  Failover
•reduce the affect on users
•unknown SLA -- have a workaround ready (hopefully        ...
Availability - backups

 Backup, backup and double backup
•users are forgiving about site outages, but not data loss
•cons...
Availability – monitoring

 Monitor everything
•appserver, DB, firewall, apache, even monitor the monitor!
•mysql monitor
...
Availability – problem prevention

  Change review
•avoid late night changes
•Trust,   but verify
•document changes -- pay...
Agenda

Introduction
Development process
Architecture & deployment
Availability & Performance




                  2009 C...
Performance – “run forest run”...fast

 Monitor your site traffic for performance numbers
 Read from slave DB always
 Memc...
What Do You Need to Know About
                   Creating and Running a Scalable Web
                   Site but Were Afr...
Upcoming SlideShare
Loading in …5
×

Scalable Website C12009 1

1,474 views

Published on

What Do You Need to Know About Creating and Running a Scalable Web Site but Were Afraid to Ask?

Published in: Technology, Design
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,474
On SlideShare
0
From Embeds
0
Number of Embeds
124
Actions
Shares
0
Downloads
79
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Scalable Website C12009 1

  1. 1. What Do You Need to Know About Creating and Running a Scalable Web Site but Were Afraid to Ask? Session Code: S304347 Chris Webster chris.webster@sun.com Girish Balachandran girish.balachandran@sun.com
  2. 2. Learn some of the techniques we used to create and build zembly.com 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 2
  3. 3. Agenda Introduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 3
  4. 4. Agenda Introduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 4
  5. 5. Introduction - zembly.com Browser based social application development environment (wiki for applications) •Content Management System for executing code •Executable API repository •many outgoing HTTP web API calls •15% writes vs. 85% reads Audience •App Developers •zembly App User – variable (one app has 30K+) •API Provider – Major players (Flickr, Twitter) but extensible and expanding Concept > Stealth > Private Beta > Open Beta launch •only constant is change 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 5
  6. 6. Introduction – software supporting zembly Development Tools •NetBeans •Subversion •kenai.com •Hudson •Automated Testing (JUnit, Selinium) Infrastructure •Solaris •Glassfish (Java EE 5, JPA) •MySQL + Enterprise Monitor •Apache •Memcached •AdventNET – Application Manager •zenoss 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 6
  7. 7. Introduction – we are zembly Engineering Team – Chris •Platform •Girish •user interaction Site engineering - Girish •Chris •monitoring, performance, DBA, etc. •Operations •data center operators - external •system admin - external 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 7
  8. 8. Agenda Introduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 8
  9. 9. Development process - overview Agile development - 3 week sprints Everyone can nominate work for the upcoming sprint Planning meeting selects sprint deliverables Engineers complete work Build vetting Weekly production deployment 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 9
  10. 10. Development process - planning Strategic direction Engineering discussions before nominations Tasks must have •Clear description, time estimate, and dependencies •Account for demos, blogging, bugs, tests No features added during planning Load balancing for people Retrospectives are important 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 10
  11. 11. Development process - code Pair programming Continuous Build •compiler errors •unit test failures •JSLint – for java script errors •archive successful builds 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 11
  12. 12. Development process – tests Deploy successful build for functional and automated UI tests •functional tests (blackbox testing) •UI tests: JUnit and Selenium Nightly build •code coverage •FindBugs Nightly performance tests •Repeatability •needs historical data to detect trends •model production traffic 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 12
  13. 13. Development Process – code lines 3 code lines •trunk – must pass all automated tests •staging – created from specific trunk version once a week. •Deployed in clone of production •exercise it for a week •blockers fixed asap •hackathon – one hour holistic testing •week keeps extending till this codeline is perfected. •production – vetted production builds •future P1 fixes go to production branch and pushed to production fast 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 13
  14. 14. Development process - experience No nominations Nominations without details Growing backlog Looooooonnnnnngggg planning meetings Adding work in the middle of the sprint No accounting for bug fixing, demos, tests ... Regressions where we don't have tests Urgent fixes needed in production Users finding problems before monitoring 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 14
  15. 15. Agenda Introduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 15
  16. 16. w ik ET ze i /b N m lo y. bl g/ bl y. fo m C ru ze O m M Load Balancer zembly.COM zembly.NET forum Static Static wiki Content Content blog z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic Site Site Execution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read Slave Read Slave Read Slave Database 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 16
  17. 17. zembly database setup Master DB Local All writes go here Local HD HD RAID RAID Primary Replication Standby Master Master Local Local HD HD n io at lic ep R Replication 1 Backup/10mins MySQL Dump &/ ZFS snapshot Backup Slave Thumper Thumper Tape ZFS Slave DB Slave DB Slave DB Slave DB 1 2 3 n Local Local Local Local RAID1 RAID1 RAID1 RAID1 HD HD HD HD 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 17
  18. 18. Architecture - experiences Perpetual architecture •start simple and grow organic •development and maintenance inline with site traffic •monitor site traffic always •keep game plan for the next level ready Have exact same setup in staging •could be a subset of production •experimentation, debug, test, etc. •changes run in staging for at least 2weeks before moving to production Monitoring built in to each layer of the stack 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 18
  19. 19. Agenda Introduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 19
  20. 20. w ik ET ze i /b N m lo y. bl g/ bl y. fo m C ru ze O m M Load Balancer zembly.COM zembly.NET forum Static Static wiki Content Content blog z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic Site Site Execution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read Slave Read Slave Read Slave Database 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 20
  21. 21. w ik ET ze i /b N m lo y. bl g/ bl y. fo m C ru ze O m M Load Balancer zembly.COM zembly.NET forum Static Static wiki Content Content blog z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic Site Site Execution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read Slave Read Slave Read Slave Database 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 21
  22. 22. w ik ET ze i /b N m lo y. bl g/ bl y. fo m C ru ze O m M Load Balancer zembly.COM zembly.NET forum Static Static wiki Content Content blog de pl oy ! z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic Site Site Execution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read Slave Read Slave Read Slave Database 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 22
  23. 23. w ik ET ze i /b N m lo y. bl g/ bl y. fo m C ru ze O m M Load Balancer zembly.COM zembly.NET forum Static Static wiki Content Content blog z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic Site Site Execution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read Slave Read Slave Read Slave Database 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 23
  24. 24. w ik ET ze i /b N m lo y. bl g/ bl y. fo m C ru ze O m M Load Balancer zembly.COM zembly.NET forum Static Static wiki Content Content blog de pl oy ! z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic Site Site Execution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read Slave Read Slave Read Slave Database 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 24
  25. 25. Architecture & Deployment - deployment Rolling deployments •automated homegrown script •turn off monitoring •no user impact during deployment •firewall block of port •localhost URL still accessible •deployment verified thru localhost URL •versioning of webserver static content •special care for db structural updates •changes are versioned •existing table definition changes need more than 1 deployment •reason? •Glassfish restarts takes the longest of the deployment time •parallelize deployment after first server 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 25
  26. 26. Agenda Introduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 26
  27. 27. Availability – experiences differences between staging and production hardware failure unmonitored service late night changes human errors data center problems using services without SLA lack of documentation lack of backup (human, infrastructure) 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 27
  28. 28. Availability - failover Failover •reduce the affect on users •unknown SLA -- have a workaround ready (hopefully automated) •redundancy everywhere (including humans)! •helps also horizontal scaling •document and test the doc •must be solid and dependable •gives you a good night sleep! •get professional help •mysql support 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 28
  29. 29. Availability - backups Backup, backup and double backup •users are forgiving about site outages, but not data loss •consider data loss from software defects & human errors •DB - mysql dump, mysql ZFS snapshot •htdocs - filesystem replication •ZFS snapshot & replication •rsync works just fine also •be aware of your restoration time •mysql dumps are slower but gives you consistent data always •watch out for mysql settings that might prolong crash recovery •innodb_log_file_size •practice your maintenance mode 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 29
  30. 30. Availability – monitoring Monitor everything •appserver, DB, firewall, apache, even monitor the monitor! •mysql monitor •homegrown scripts •log file monitors •Adventnet Use datacenter monitoring – if available •advantages – call home for issues •first level support for hardware should be 24x7 live and online •human to trouble shoot your problem as soon as it occurs 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 30
  31. 31. Availability – problem prevention Change review •avoid late night changes •Trust, but verify •document changes -- pay it forward •build up an failure proof recovery manual Watch your traffic •collect all sorts of stats •to start with, don't worry about coherency of data Monitor & penalize users for malicious activities Keep all infrastructure software security patches current •specifically the web facing ones – wiki, blog, forum, etc Run a security audit on your site 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 31
  32. 32. Agenda Introduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 32
  33. 33. Performance – “run forest run”...fast Monitor your site traffic for performance numbers Read from slave DB always Memcached is your friend DB query optimizations takes you really far Start with best practice configurations -- adjust systematically based on live data Know your OS! Know your network! Have an automated performance test suite Expert help [blog, support, ...] 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 33
  34. 34. What Do You Need to Know About Creating and Running a Scalable Web Site but Were Afraid to Ask? Session Code: S304347 Chris Webster chris.webster@sun.com Girish Balachandran girish.balachandran@sun.com

×