WebCo Way CParty 2009

The WebCo Way: O que fazemos & Como fazemos Manoel Lemos [CTO & Founder] Campus Party 2009 - Desenvolvimento São Paulo, SP, Brasil

Acompanhe e participe #cparty http://live.blogblogs.com.br/

We’re 40 people today!

BlogBlogs Status Business Numbers Trafﬁc Numbers 115.000 users 7.5M PageViews / month 200.000 blogs 3.5M Uniques / month 2.500.000 posts Top 5000 Alexa Global 250 users / day Top 120 Alexa Brasil 150 blogs / day 50M Widget Impressions / month 7.000 posts / day

Scaling is a process complicated and painful! Its impossible to predict with accuracy There are hidden bottlenecks Sometimes you need to step back Sometimes you need to re-factor everything There are no “silver-bullets” Complexity will grow with you

Growth never happens in a single dimension Trafﬁc : more users, more transactions, more downloads... Size : database, memory, bandwidth... Complexity : algorithms, architecture, infrastructure... Organization : team, communication, processes... Business : models, partners, suppliers... Mostly the issues grow faster than linearly!!!

Today 1.7MM daily impr.

httpd mongrel mysql

httpd mongrel mongrel mongrel mysql

httpd mongrel mongrel mongrel memcached mysql

load balancing httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached mysql

load balancing httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached mysql

load balancing httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached mysql

load balancing 1 (web) httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached mysql

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached memcached memcached memcached mysql

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached memcached memcached memcached mysql (master) mysql (slave)

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached memcached memcached memcached mysql mysql (master) mysql (slave)

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) memcached memcached memcached memcached

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) memcached memcached memcached memcached

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) memcached memcached Robot Robot memcached memcached Robot Robot

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) Indexer Indexer memcached memcached Robot Robot (Search) (Search) memcached memcached Robot Robot

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) Indexer Indexer memcached memcached Robot Robot (Search) (Search) httpd httpd mongrel mongrel mongrel mongrel memcached memcached Robot Robot mongrel mongrel mongrel mongrel mongrel mongrel load balancing 3 (API)

load balancing 1 (web) load balancing 2 (widgets) mongrels mongrels mongrels mongrels mongrels mongrels mysql mysql (master) mysql (slave) Indexer Indexer memcached memcached Robot Robot (Search) (Search) memcached memcached Robot Robot mongrels mongrels load balancing 3 (API)

load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Robot Robot Indexer memcached memcached (Search) mysql mysql (master) (master) Indexer memcached memcached (Search)

load balancing load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Robot Robot Robot Robot Indexer Indexer memcached memcached memcached memcached (Search) (Search) mysql mysql mysql mysql (master) (master) (master) (master) Indexer Indexer memcached memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Eureca!! Ideas... mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Single Sign On Shared Resources Robot Robot Integrated Search and more... Robot Robot Indexer Indexer memcached memcached memcached memcached (Search) (Search) mysql mysql mysql mysql (master) (master) (master) (master) Indexer Indexer memcached memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Robot Robot Robot Robot Indexer Indexer memcached memcached memcached memcached (Search) (Search) mysql mysql mysql mysql (master) (master) (master) (master) Indexer Indexer memcached memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels #howto ? Robot Robot Robot Robot Indexer Indexer memcached memcached memcached memcached (Search) (Search) mysql mysql mysql mysql (master) (master) (master) (master) Indexer Indexer memcached memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA Credentials (PHP) ActiveDNA (REST) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA DNA Credentials Proﬁle (PHP) (PHP) ActiveDNA (REST) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA DNA DNA Credentials Proﬁle Avatars (PHP) (PHP) (PHP) ActiveDNA (REST) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA DNA DNA DNA DNA Credentials Proﬁle Avatars Scoring SocialGraph (PHP) (PHP) (PHP) (C++) (Python) ActiveDNA (REST) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA Credentials Proﬁle Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging (PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang) ActiveDNA (REST) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) load balancing DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA Credentials Proﬁle Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging (PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang) ActiveDNA (REST) load balancing Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels humm... mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels PHP... ActiveDNA (REST) WTF??!! load balancing DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA Credentials Proﬁle Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging Credentials (PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang) (PHP) ActiveDNA (REST) load balancing Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels humm... mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels PHP... ActiveDNA (REST) WTF??!! load balancing DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA Credentials Proﬁle Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging (PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang) ActiveDNA (REST) load balancing Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer Robot Robot memcached memcached memcached (Search) (Search)

Active DNA The problem : time-to-market & development time The solution : ready-for-use SWAT DNA components The layer : Active DNA, a lightweight, RESTful, SWAT DNA client API translating REST calls into fully realized Ruby objects Pros Complete abstraction of the component’s complexity. It’s just Ruby, very small learning curve for devs.

Active DNA Application REST API SWAT DNA Component

Active DNA Application REST Client REST API SWAT DNA Component

Active DNA Application Query Runner REST Client REST API SWAT DNA Component

Active DNA Application Query Runner Query Translator REST Client REST API SWAT DNA Component

Active DNA Application Component Layer Query Runner Query Translator REST Client REST API SWAT DNA Component

class ActiveDNA::ProfileEntry < ActiveDNA::Base uses :profile key :key, :suffix => :entry depends_on :profile_id attributes :type, :group, :level, :value => :json, :is_unique => :boolean validates_presence_of :value, :message => \"The value must be informed\" validates_presence_of :profile_id, :message => \"Profile must be informed\" validates_presence_of :group, :message => \"The group must be informed\" validates_inclusion_of :type, :in => %w(boolean email float general guid integer url), :message => \"The type must be informed\" end

class RegistrationController < ApplicationController def create_credential(profile_id) returning credential = ActiveDNA::Credential.new do credential.credential_identifier = self.identifier credential.credential_type = self.identifier_type || IDENTIFIER_TYPE_EMAIL credential.credential_status = \"enabled\" credential.user_id = Profile.find(:first, :conditions => { :profile_id => profile_id }).user_id credential.password = self.password if self.identifier_type == IDENTIFIER_TYPE_EMAIL credential.save! end end end

Technical Learnings GIT Distributed version control that works No branches on SVN to dozens of branches on Git Fine-grained control of revisions Fewer conﬂicts A branch per story

Technical Learnings

Technical Learnings Multiple Environments Local ➙ Sandbox Integration (QA) Staging ➙ Production All referenced in Git branches ~$ sudo cap deploy

Technical Learnings master rc stable

Technical Learnings story-1 king local master rc stable

Technical Learnings story-1 story-2 king slave local sandbox master rc stable

Technical Learnings story-1 story-2 king slave local sandbox master qa rc stable

Technical Learnings story-1 story-2 king slave local sandbox master qa rc staging stable production

Test all the f**king time!!! TDD/BDD is a muscle, you need to exercise it Without coverage, there is no understanding Knowledge socialization Continuous integration

Continuous Integration Dedicated integration environment with automated integration and tests execution with the most up to dated code base. Automated Reporting Test Cases Results Tests Coverage

Tests, tests & more tests

Continuous Integration

Looking Forward Queues are the new black : starling & others Asynchronous is the newer black : workling & others Erlang + Cloud = massive continuous computing space Distributed objects : Distributed search Predictive caching Algorithms & Computer Science : latent semantics, vector space model, etc

Three Lines on Scrum Fastest way to go to nowhere Scrum without a present Product Owner Slowest way to go to somewhere Scrum without a dedicated Scrum Master Embrace it fully with the best team.

Valeu!!! ;-) mlemos@webcointernet.com http://brasigo.com.br/ http://blog.brasigo.com.br/ http://blogblogs.com.br/ http://blog.blogblogs.com.br/

Estamos contratando novos Jedis: Envie seu CV para vagas@blogblogs.com.br ou deixe seus dados no Lounge da Abril Digital

WebCo Way CParty 2009

1 comments

Notes on slide 1

6 Favorites