The WebCo Way:
O que fazemos &
Como fazemos
Manoel Lemos [CTO & Founder]
Campus Party 2009 - Desenvolvimento
São Paulo, SP, Brasil

Acompanhe e participe
#cparty
http://live.blogblogs.com.br/

We’re 40 people
today!

BlogBlogs Status
Business Numbers Traffic Numbers
115.000 users 7.5M PageViews / month
200.000 blogs 3.5M Uniques / month
2.500.000 posts Top 5000 Alexa Global
250 users / day Top 120 Alexa Brasil
150 blogs / day 50M Widget
Impressions / month
7.000 posts / day

Scaling is a process
complicated and painful!
Its impossible to predict with accuracy
There are hidden bottlenecks
Sometimes you need to step back
Sometimes you need to re-factor everything
There are no “silver-bullets”
Complexity will grow with you

Growth never happens in a
single dimension
Traffic : more users, more transactions, more downloads...
Size : database, memory, bandwidth...
Complexity : algorithms, architecture, infrastructure...
Organization : team, communication, processes...
Business : models, partners, suppliers...
Mostly the issues grow faster than linearly!!!

Today
1.7MM
daily impr.

httpd
mongrel
mysql

httpd
mongrel
mongrel
mongrel
mysql

httpd
mongrel
mongrel
mongrel
memcached
mysql

httpd
mongrel
mongrel
mongrel
memcached
mysql

httpd
mongrel
mongrel
mongrel
memcached
mysql

load balancing
httpd httpd
mongrel mongrel
mongrel mongrel
mongrel mongrel
memcached memcached
mysql

load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing 1 (web)
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql (master)
mysql (slave)

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql mysql (master)
mysql (slave)

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql mysql (master)
mysql (slave)

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql mysql (master)
mysql (slave)

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
memcached memcached
memcached memcached

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
memcached memcached
memcached memcached

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
memcached memcached Robot Robot
memcached memcached Robot Robot

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
Indexer Indexer
memcached memcached Robot Robot
(Search) (Search)
memcached memcached Robot Robot

load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
Indexer Indexer
memcached memcached Robot Robot
(Search) (Search)
httpd httpd
mongrel mongrel
mongrel mongrel
memcached memcached Robot Robot
mongrel mongrel
mongrel mongrel
mongrel mongrel
load balancing 3 (API)

load balancing 1 (web) load balancing 2 (widgets)
mongrels mongrels mongrels mongrels mongrels mongrels
mysql mysql (master)
mysql (slave)
Indexer Indexer
memcached memcached Robot Robot
(Search) (Search)
memcached memcached Robot Robot mongrels mongrels
load balancing 3 (API)

load balancing
mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels
Robot
Robot
Indexer
memcached memcached
(Search)
mysql mysql
(master) (master)
Indexer
memcached memcached
(Search)

load balancing
mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels
Robot
Robot
Indexer
memcached memcached
(Search)
mysql mysql
(master) (master)
Indexer
memcached memcached
(Search)

load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Robot Robot
Robot Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Robot Robot
Robot Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Eureca!! Ideas...
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Single Sign On
Shared Resources
Robot Robot
Integrated Search
and more...
Robot Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Robot Robot
Robot Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
#howto ?
Robot Robot
Robot Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA
Credentials
(PHP)
ActiveDNA (REST)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA
Credentials Profile
(PHP) (PHP)
ActiveDNA (REST)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA DNA
Credentials Profile Avatars
(PHP) (PHP) (PHP)
ActiveDNA (REST)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph
(PHP) (PHP) (PHP) (C++) (Python)
ActiveDNA (REST)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
humm...
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
PHP... ActiveDNA (REST)
WTF??!! load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
Credentials
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
(PHP)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
humm...
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
PHP... ActiveDNA (REST)
WTF??!! load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)

Active DNA
The problem : time-to-market & development time
The solution : ready-for-use SWAT DNA components
The layer : Active DNA, a lightweight, RESTful, SWAT
DNA client API translating REST calls into fully realized
Ruby objects
Pros
Complete abstraction of the component’s complexity.
It’s just Ruby, very small learning curve for devs.

Active DNA
Application
REST API
SWAT DNA Component

Active DNA
Application
REST API
SWAT DNA Component

Active DNA
Application
REST Client
REST API
SWAT DNA Component

Active DNA
Application
Query Runner
REST Client
REST API
SWAT DNA Component

Active DNA
Application
Query Runner Query Translator
REST Client
REST API
SWAT DNA Component

Active DNA
Application
Component Layer
Query Runner Query Translator
REST Client
REST API
SWAT DNA Component

class ActiveDNA::ProfileEntry < ActiveDNA::Base
uses :profile
key :key, :suffix => :entry
depends_on :profile_id
attributes :type, :group, :level, :value => :json, :is_unique => :boolean
validates_presence_of :value, :message => \"The value must be informed\"
validates_presence_of :profile_id, :message => \"Profile must be informed\"
validates_presence_of :group, :message => \"The group must be informed\"
validates_inclusion_of :type,
:in => %w(boolean email float general guid integer url),
:message => \"The type must be informed\"
end

class RegistrationController < ApplicationController
def create_credential(profile_id)
returning credential = ActiveDNA::Credential.new do
credential.credential_identifier = self.identifier
credential.credential_type = self.identifier_type ||
IDENTIFIER_TYPE_EMAIL
credential.credential_status = \"enabled\"
credential.user_id = Profile.find(:first, :conditions => { :profile_id
=> profile_id }).user_id
credential.password = self.password if self.identifier_type ==
IDENTIFIER_TYPE_EMAIL
credential.save!
end
end
end

Technical Learnings
GIT
Distributed version control that works
No branches on SVN to dozens of branches on Git
Fine-grained control of revisions
Fewer conflicts
A branch per story

Technical Learnings

Technical Learnings
Multiple Environments
Local ➙ Sandbox
Integration (QA)
Staging ➙ Production
All referenced in Git branches
~$ sudo cap deploy

Technical Learnings
master
rc
stable

Technical Learnings
story-1
king
local
master
rc
stable

Technical Learnings
story-1 story-2
king slave
local sandbox
master
rc
stable

Technical Learnings
story-1 story-2
king slave
local sandbox
master
qa
rc
stable

Technical Learnings
story-1 story-2
king slave
local sandbox
master
qa
rc
staging
stable production

Test all the f**king time!!!
TDD/BDD is a muscle, you need to exercise it
Without coverage, there is no understanding
Knowledge socialization
Continuous integration

Continuous Integration
Dedicated integration environment with automated
integration and tests execution with the most up to
dated code base.
Automated Reporting
Test Cases Results
Tests Coverage

Tests, tests & more tests

Tests, tests & more tests

Tests, tests & more tests

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Continuous Integration

Looking Forward
Queues are the new black : starling & others
Asynchronous is the newer black : workling & others
Erlang + Cloud = massive continuous computing space
Distributed objects : Distributed search
Predictive caching
Algorithms & Computer Science : latent semantics,
vector space model, etc

Three Lines on Scrum
Fastest way to go to nowhere
Scrum without a present Product Owner
Slowest way to go to somewhere
Scrum without a dedicated Scrum Master
Embrace it fully with the best team.

Valeu!!! ;-)
mlemos@webcointernet.com
http://brasigo.com.br/
http://blog.brasigo.com.br/
http://blogblogs.com.br/
http://blog.blogblogs.com.br/

Estamos contratando novos Jedis:
Envie seu CV para
vagas@blogblogs.com.br
ou deixe seus dados no
Lounge da Abril Digital