YELLOWPAGES.COM: Behind the Curtain
John Straw
AT&T Interactive

What is YELLOWPAGES.COM?
Part of AT&T
s
A local search website, serving
s
23 million unique visitors / month
q
2 million searches / day
q
More than 48 million requests / day
q
More than 1500 requests / sec
q
30 Mbit/sec (200 Mbit/sec from Akamai)
q
Entirely Ruby on Rails since June 2007
s

How we were
Website and API applications
s
written in Java
Website in application server
q
API in Tomcat
q
Search code split between
s
application and search layer

What was bad
Problems with architecture
s
Separate search implementations in each application
q
Session-heavy website application design hard to scale horizontally
q
Pointless web accelerator layer
q
Problems with application server platform
s
Technologically stagnant
q
No usable session migration features
q
Hard to do SEO
q

And also ...
Lots of code written by consultants 2004-2005
s
Fundamental design problems
s
Code extended largely by copy-and-modify since 11/2005 (to
s
around 125K lines)
No test code
s
New features hard to implement
s

The Big Rewrite
Several projects combined to become the big rewrite
s
Replacement of Java application server
q
Redesign of site look-and-feel
q
Many other wish-list projects, some of which were difficult to
q
accomplish with existing application
Project conception to completion: one year
s
Development took about four months
s
Project phases
s
7/2006 - 12/2006: Thinking, early preparation
q
12/2006: Rough architecture determination, kick-off
q
1/2007 - 3/1/2007: Technology research and prototypes, business
q
rules exploration, UI design concepts
3/1/2007 - 6/28/2007: Site development and launch
q

Requirements for a new site architecture
1. Absolute control of urls
Maximize SEO crawl-ability
q
2. No sessions: HTTP is stateless
Anything other than cookies is just self-delusion
q
Staying stateless makes scaling easier
q
3. Be agile: write less code
Development must be faster
q
4. Develop easy-to-leverage core business services
Eliminate current duplicated code
q
Must be able to build other sites or applications with common
q
search, personalization and business review logic
Service-oriented architecture, with web tier utilizing common
q
service tier

Rewrite team
Cross-functional team of about 20 people
s
Assemble stakeholders, not requirements
q
Working closely together
s
Whole team sat together for entire project
q
Lunch provided several days per week
q
Team celebrations held for milestones
q
Core development team deliberately small
s
Four skilled developers can accomplish a lot
q
Cost of communication low
q
Low management overhead
q

Picking the platform
Web tier:
s
Rails or Django
q
Utilizing common services for
q
search, personalization, and
ratings
Service tier:
s
Java application
q
Probably EJB3 in JBoss
q
Exposing a REST API and
q
returning JSON-serialized data
Started writing prototypes ...
s

One

Two

Three

And finally ...

Why Rails?
Considered Java frameworks didn't provide enough control of
s
url structure
Web tier choice became Rails vs. Django
s
Rails best web tier choice due to
s
Better automated testing integration
q
More platform maturity
q
Clearer path (for us!) to C if necessary for performance
q
Developer comfort and experience
q
Team decided to go with Rails top-to-bottom
s
Evaluation of EJB3 didn't show real advantages over Ruby or
q
Python for our application
Reasons for choosing Rails for web tier applied equally to service
q
tier
Advantage of having uniform implementation environment
q

Separate or combined?

Other considerations
How many servers?
s
How many mongrels per server?
s
How much memory for memcached?
s

Production configuration
Acquired 25 machines of identical configuration for each data
s
center
Performance testing to size out each tier, and determine how
s
many mongrels
4 GB of memory on each service-tier machine set aside for
s
memcached
Used 2 machines in each data center for database servers
s

Performance goals
Sub-second average home page load time
s
Sub 4-second average search results page load time
s
Handle traffic without dying
s

Performance optimizations
mongrel_handlers in service tier application
s
C library for parsing search cluster responses
s
Erubis rather than erb in web tier
s

Site at launch

Database performance issues
Machines inadequate to handle search load for ratings look-up
s
Additional caching added
q
Oracle doesn't like lots of connections
s
Use of named connections made this problem even worse
s
Additional memory required for database servers
q
All database look-up code moved to service tier
q
Changed to a single database connection
q

Page performance issues
Slow page performance caused more by asset download times
s
than speed of web framework
Worked through the Yahoo! performance guidelines
s
Minified and combined CSS and Javascript with
s
asset_packager
Moved image serving to Akamai edge cache
s
Apache slow serving analytics tags -- moved to nginx for web
s
tier
Started using some fragment caching
s

Slow requests, etc.
Slow requests in the web tier caused mongrel queueing
s
Developed qrp (http://qrp.rubyforge.org/)
q
Allows you to establish a backup pool where requests get parked
q
until a mongrel is available
Experimented with different malloc implementations
s
Started using a custom MRI build -- ypc_ruby
s
Started using a slightly-customized Mongrel
s

Overall performance
Performance at launch was generally acceptable
s
After web server & hosting changes performance better than
s
previous site
Extensive use of caching and elimination of obsolete queries
s
lowered load on search cluster compared to previous site
Over a year later, we need to do more profiling
s
Traffic has more than doubled since launch
q
Hardware evolution has invalidated original profiling
q
We now want sub 2-second search result pages
q

What else?
New applications on Rails
s
Server-side component of native iPhone application
q
Working on moving current Java API application
q
Other internal applications
q
Ruby but not Rails
s
Exploratory port of service tier to merb
q
Supporting development of Waves
q
Data-source ETL
q
Listing match-and-merge
q