Case Study: Digging into the Technology Behind the Development of Digg

Slide 2: What is digg?

Slide 3: The idea - Oct. 04  Slashdot - User created content  Del.icio.us - Bookmark / Share / Save  Friendster - Social hookups, large groups, no task

Slide 4: $2,000 to launch  Developed a project spec - elance  Open source technologies - LAMP  Basic utilitarian design  $99 a mo. hosting

Slide 5: Feature Decisions  Stay away from me too features  Simple and rewarding - one click (digg, bury, #1)  Used “ajax” where it made sense  Tools to connect to other sites (blog this, javascript widget)  Experiment - spy, “cloud view”, visualizations (stack, swarm)

Slide 6: The “digg effect”

Slide 7: Case Study: From Startup to Enterprise

Slide 8: One Server (Dec 2004-Mar 2005).  Monitoring is “someone is working on the site most of the time.”  Standard LAMP architecture.  Growth -> optimizing queries -> denormalizing.

Slide 9: Growth  Paris Hilton  Silverorange/Digg 2.0  Word of mouth, minimal advertising  Steady growth with occasional insanity

Slide 10: Seed Funding.  Small Number of Servers (May 2004-Sep 2004) - web server, mysql master, mysql slave.  Ad-hoc monitoring.  1 Dedicated operations person.  Silverorange (Digg gets a professional design).  Growth continues to outstrip hardware, though now more constrained by developer resources.

Slide 11: Series A.  $100k on Servers. Spares!  Everyone in the same place  Operations Department (currently 4 people).  Hire senior developers.

Slide 12: Description of Digg Architecture  LAMP + memcached.  Rough estimate of lines of code (historical trend). I.e. Digg 1.0 = 5k lines of code. Digg 2.0 = 50k. Digg 3.0 = 200k (will fill in actual numbers).  Diagram (something like timeless’s “shards” diagram).

Slide 13: Monitoring (Nagios/Cacti). A couple of screenshots - Cacti Graphs.

Slide 14: Development Process. Subversion, Bugzilla, Wiki. Trac.1 Developer --> 3 --> 7. Talk a little about managing that.

Slide 15: Office. Pictures? Moved in May 2006, already outgrowing.

Slide 16: Stats  1M+ Registered users  10M+ Pages per day  1.2M+ Daily unique visitors  100+ Linux boxes (break this down)

Slide 17: New Features/Announcements

Slide 18: Future Plans  Party!!!