Rails Summit 2008 : Histórias de Guerra - Enfrentando problemas reais em aplicações Web 2.0

Uma história de guerra: Enfrentando problemas reais em aplicações Web 2.0 Manoel Lemos Ronaldo Ferraz Nando Vieira Rails Summit Latin America 2008 São Paulo, Brasil, 16.Outubro

1983

1983 1984

1983 1993 1984

1983 1993 1984 1994

1983 1993 1984 1994 1997

1983 1993 1999 1984 1994 1997

1983 1993 1999 1984 1994 1997 2000

1983 1993 1999 2006 1984 1994 1997 2000

Abr.06

Nov.06 Abr.06

Fev.07 Nov.06 Abr.06

Fev.08 Fev.07 Nov.06 Abr.06

Ago.08 Fev.08 Fev.07 Nov.06 Abr.06

BlogBlogs Status Business Numbers Trafﬁc Numbers 110.000 usuários 5.2M PageViews 200.000 blogs 2.6M Uniques 2.000.000 posts Top 4000 Alexa Global 250 usuários / dia Top 120 Alexa Brasil 150 blogs / dia Top 102 IBOPE Brasil 5.000 posts / dia 50M Widget Impressions

Crescer é um processo complicado e doloroso! É impossível de se prever com precisão Existem gargalos escondidos Às vezes você precisa voltar atrás Às vezes você precisa mudar tudo Não existem “silver-bullets” A complexidade vai crescer com você

Crescimento real nunca acontece em um só eixo! Tráfego : mais usuários, transações, imagens... Tamanho : banco, memória, cache... Complexidade : algoritmos, arquitetura, infraestrutura... Organização : equipe, comunicação, processos... Negócio : modelos, parceiros, fornecedores... Na maioria dos casos os problemas crescem mais que linearmente!!!

Hoje 1.5 Milhões por dia

httpd mongrel mysql

httpd mongrel mongrel mongrel mysql

httpd mongrel mongrel mongrel memcached mysql

load balancing httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached mysql

load balancing httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached mysql

load balancing httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached mysql

load balancing 1 (web) httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached mysql

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached memcached memcached memcached mysql

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached memcached memcached memcached mysql (master) mysql (slave)

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel memcached memcached memcached memcached memcached memcached mysql mysql (master) mysql (slave)

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) memcached memcached memcached memcached

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) memcached memcached memcached memcached

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) memcached memcached Robot Robot memcached memcached Robot Robot

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) Indexer Indexer memcached memcached Robot Robot (Search) (Search) memcached memcached Robot Robot

load balancing 1 (web) load balancing 2 (widgets) httpd httpd httpd httpd httpd httpd mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mongrel mysql mysql (master) mysql (slave) Indexer Indexer memcached memcached Robot Robot (Search) (Search) httpd httpd mongrel mongrel mongrel mongrel memcached memcached Robot Robot mongrel mongrel mongrel mongrel mongrel mongrel load balancing 3 (API)

load balancing 1 (web) load balancing 2 (widgets) mongrels mongrels mongrels mongrels mongrels mongrels mysql mysql (master) mysql (slave) Indexer Indexer memcached memcached Robot Robot (Search) (Search) memcached memcached Robot Robot mongrels mongrels load balancing 3 (API)

load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Robot Robot Indexer memcached memcached (Search) mysql mysql (master) (master) Indexer memcached memcached (Search)

load balancing load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Robot Robot Robot Robot Indexer Indexer memcached memcached memcached memcached (Search) (Search) mysql mysql mysql mysql (master) (master) (master) (master) Indexer Indexer memcached memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Eureca...Idéias... mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Single Sign On Shared Resources Robot Robot Integrated Search Robot E=MC2 Robot Indexer Indexer memcached memcached memcached memcached (Search) (Search) mysql mysql mysql mysql (master) (master) (master) (master) Indexer Indexer memcached memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Robot Robot Robot Robot Indexer Indexer memcached memcached memcached memcached (Search) (Search) mysql mysql mysql mysql (master) (master) (master) (master) Indexer Indexer memcached memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels E ai, #comofaz ? Robot Robot Robot Robot Indexer Indexer memcached memcached memcached memcached (Search) (Search) mysql mysql mysql mysql (master) (master) (master) (master) Indexer Indexer memcached memcached memcached memcached (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA Credentials (PHP) ActiveDNA (REST) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA DNA Credentials Proﬁle (PHP) (PHP) ActiveDNA (REST) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA DNA DNA Credentials Proﬁle Avatars (PHP) (PHP) (PHP) ActiveDNA (REST) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA DNA DNA DNA DNA Credentials Proﬁle Avatars Scoring SocialGraph (PHP) (PHP) (PHP) (C++) (Python) ActiveDNA (REST) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA Credentials Proﬁle Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging (PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang) ActiveDNA (REST) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels ActiveDNA (REST) load balancing DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA Credentials Proﬁle Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging (PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang) ActiveDNA (REST) load balancing Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels humm... mongrels mongrels mongrels mongrels mongrels mongrels mongrels PHP... ActiveDNA (REST) WTF??!! load balancing DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA Credentials Credentials Proﬁle Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging (PHP) (PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang) ActiveDNA (REST) load balancing Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

load balancing load balancing mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels mongrels humm... mongrels mongrels mongrels mongrels mongrels mongrels mongrels PHP... ActiveDNA (REST) WTF??!! load balancing DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA Credentials Proﬁle Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging (PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang) ActiveDNA (REST) load balancing Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search) mysql mysql mysql (master) (slave) (slave) Indexer Indexer memcached memcached memcached Robot Robot (Search) (Search)

Active DNA Problema: Várias features necessárias (Proﬁle, SSO, Avatar, etc) e pouco tempo Serendipity: Companhia irmã com vários componentes REST Resultado: Active DNA, uma biblioteca REST ﬂexível mais cache e serviços

Active DNA Por que não Active Resource Interface ﬁxa Somente Rails sem overrides Pouca integração com external services

Busca Ferret: FAIL Sphinx: Muito limitado Lucene / Solr: Java Xapian: Interessante Mas, e se precisarmos mudar?

Busca Solução: Sapien Cliente / encapsulamento de servidor de busca Driver-based, naturalmente distribuído Drivers podem ser mudados e o índice refeito em um passo Brasigo Query Language

API Controle de Acesso HTTP Auth (email & senha) OAuth para suportar usuários de OpenID e outros Restrições de uso Frameworks mais leves Sinatra Thin

Dia-a-dia (SCRUM) Scrum Várias equipes Coordenação e priorização Rotacionamento de equipes Socialização de conhecimento Cultura compartilhada

Dia-a-dia (Git) Git Vários repositórios internos, todos Git Branches por estória Repositório sempre releasable

Dia-a-dia (Sharing) Cultura de bibliotecas Plugins Gems / Gem server interno Funcionalidade compartilhada via API

Dia-a-dia (Testes) Test all the f***ing time Sem cobertura, não há conhecimento Socialização de conhecimento Integração contínua

Continuous Integration Dedicated integration environment with automated integration and tests execution with the most up to dated code base. Automated Reporting Test Cases Results Tests Coverage

Tests, tests & more tests

Continuous Integration

Venha trabalhar com a gente!!! vagas@blogblogs.com.br

Obrigado! mlemos@webcointernet.com rferraz@webcointernet.com fvieira@webcointernet.com http://brasigo.com.br/ http://blog.brasigo.com.br/ http://blogblogs.com.br/ http://blog.blogblogs.com.br/

Rails Summit 2008 : Histórias de Guerra - Enfrentando problemas reais em aplicações Web 2.0

0 comments

Notes on slide 1

3 Favorites