Scalability at GROU.PS<br />EmreSokullu<br />
Disclaimer<br />We’re not fully there yet<br />We hire: jobs@groups-inc.com<br />
Challenges @ GROU.PS<br />3M unique visitors per month<br />120M page views<br />1PB assets to be served every month<br />...
What is GROU.PS ?<br />
Distributed Architecture<br />25+ servers, S3 cloud, EdgeCast CDN<br />4 cores + <br />All Linux: Red Hat<br />Some Debian...
Amazon Technologies<br />S3<br />CloudFront<br />EC2 (elastic IP and persistent storage)<br />SimpleDB<br />Queue technolo...
Amazon Technologies<br />Downside: <br />Not so cheap<br />Bad database performance<br />
Serving Content?<br />Use MogileFS<br />Distributed file serving<br />Use CDN<br />hot content served off from local serve...
Our typical sysctl additions<br />net.ipv4.tcp_syncookies = 1<br />net.ipv4.tcp_synack_retries = 2<br />## Emre edited<br ...
MySQL<br />Load off via memcache<br />$memcache-&gt;set(“group_by_name.jtpd”, 1122, false, 0);<br />$memcache-&gt;set(“hom...
MySQL<br />Replication easy<br />Split Reads<br />What about writes?<br />That’s where sharding comes to play<br />Vertica...
MySQL<br />Runs poorly on multi-cores<br />query_cache_size = 0 # on master<br />query_cache_type = 0 # on master<br />thr...
MySQL Query Optimization<br />INDEX  group, user<br />WHERE group = ? AND user = ?<br />Not WHERE user = ? AND group = ?<b...
MySQL Query Optimization<br />SHOW PROCESSLIST<br />Maatkit, mk-query-digest<br />Percona builds<br />
NOSQL<br />Voldemort, Linkedin<br />Cassandra, Facebook<br />Tokyo Cabinet, mixi<br />
Logging<br />Database logging is not the solution<br />File system is expensive too<br />A legal necessity<br />
Logging<br />Solution:<br />Scribe & Thrift<br />By Facebook<br />Eventually consistent<br />
Nginx & libevent<br />
Nginx & libevent<br />Handles 10000 connections<br />5gbit/s<br />Rambler<br />Wordpress<br />Grou.ps<br />
Postfix<br />Run multiple instances<br />Spam Clusters<br />
Monitoring<br />Munin + monit<br />Other alternatives:<br />Cacti<br />Nagios<br />Hyperic – vmware<br />
PHP<br />
More to come on my blog<br />http://emresokullu.com<br />More fine tuning tips<br />Become a member of my community<br />L...
Upcoming SlideShare
Loading in...5
×

Scalability at GROU.PS

4,092

Published on

How does GROU.PS scale to serving 1PB of assets each month. memcache, nginx, gearman, tornado, libevent, kqueue, epoll, mysql, sharding, replication, memcached, tokyo cabinet

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,092
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
48
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Scalability at GROU.PS

  1. 1. Scalability at GROU.PS<br />EmreSokullu<br />
  2. 2. Disclaimer<br />We’re not fully there yet<br />We hire: jobs@groups-inc.com<br />
  3. 3. Challenges @ GROU.PS<br />3M unique visitors per month<br />120M page views<br />1PB assets to be served every month<br />Video,Photos, Files<br />Support for 5Gbit/s<br />Very dynamic pages:<br />With social networks; p(u,t) = HTML<br />p(g,u,t) = HTML -&gt; WHERE group_id = ? AND … <br />
  4. 4. What is GROU.PS ?<br />
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9. Distributed Architecture<br />25+ servers, S3 cloud, EdgeCast CDN<br />4 cores + <br />All Linux: Red Hat<br />Some Debian, Ubuntu, CentOS<br />
  10. 10. Amazon Technologies<br />S3<br />CloudFront<br />EC2 (elastic IP and persistent storage)<br />SimpleDB<br />Queue technologies, distributed hadoop and more…<br />
  11. 11. Amazon Technologies<br />Downside: <br />Not so cheap<br />Bad database performance<br />
  12. 12. Serving Content?<br />Use MogileFS<br />Distributed file serving<br />Use CDN<br />hot content served off from local servers<br />Sysctl tunings needed!<br />
  13. 13. Our typical sysctl additions<br />net.ipv4.tcp_syncookies = 1<br />net.ipv4.tcp_synack_retries = 2<br />## Emre edited<br /># http://www.oracle-base.com/articles/11g/OracleDB11gR1InstallationOnFedora8.php<br />kernel.shmall = 2097152<br />kernel.shmmax = 2147483648<br />kernel.shmmni = 4096<br /># semaphores: semmsl, semmns, semopm, semmni<br />kernel.sem = 250 32000 100 128<br />net.ipv4.ip_local_port_range = 1024 65000<br />net.core.rmem_default=4194304<br />#net.core.rmem_max=4194304<br />net.core.wmem_default=262144<br />#net.core.wmem_max=262144<br />fs.file-max=5049800<br />vm.swappiness=10<br />## Emre edited<br /># from http://forums.softlayer.com/showthread.php?t=3252<br />net.ipv4.tcp_rmem = 4096 87380 8388608<br />net.ipv4.tcp_wmem = 4096 87380 8388608<br />net.core.rmem_max = 8388608<br />net.core.wmem_max = 8388608<br />net.core.netdev_max_backlog = 5000<br />net.ipv4.tcp_window_scaling = 1<br />net.ipv4.ip_nonlocal_bind=1<br /># http://rackerhacker.com/2007/08/24/apache-no-space-left-on-device-couldnt-create-accept-lock/<br />kernel.msgmni = 1024<br />kernel.sem = 250 256000 32 1024<br />net.ipv4.ip_conntrack_max = 524288<br />net.ipv4.netfilter.ip_conntrack_max = 524288<br />
  14. 14. MySQL<br />Load off via memcache<br />$memcache-&gt;set(“group_by_name.jtpd”, 1122, false, 0);<br />$memcache-&gt;set(“home_module_html.1122”,…, true, 30);<br />function getGroupID($group_name) { global $memcache; if( !isset($memcache) || ($res=($memcache-&gt;get(“group_by_name.{$group_name}”)))===false ) { // get it from mysql and memcache } else { return $res; // serve from memcache }}<br />
  15. 15. MySQL<br />Replication easy<br />Split Reads<br />What about writes?<br />That’s where sharding comes to play<br />Vertical Sharding<br />Horizontal Sharding<br />MMM<br />
  16. 16. MySQL<br />Runs poorly on multi-cores<br />query_cache_size = 0 # on master<br />query_cache_type = 0 # on master<br />thread_concurrency = 8 # total cores<br />max_connections = 750 # shouldn’t exceed that<br />innodb_buffer_pool_size = 10G # a little less than the total amount<br />
  17. 17. MySQL Query Optimization<br />INDEX group, user<br />WHERE group = ? AND user = ?<br />Not WHERE user = ? AND group = ?<br />B-tree<br />
  18. 18. MySQL Query Optimization<br />SHOW PROCESSLIST<br />Maatkit, mk-query-digest<br />Percona builds<br />
  19. 19. NOSQL<br />Voldemort, Linkedin<br />Cassandra, Facebook<br />Tokyo Cabinet, mixi<br />
  20. 20. Logging<br />Database logging is not the solution<br />File system is expensive too<br />A legal necessity<br />
  21. 21. Logging<br />Solution:<br />Scribe & Thrift<br />By Facebook<br />Eventually consistent<br />
  22. 22. Nginx & libevent<br />
  23. 23. Nginx & libevent<br />Handles 10000 connections<br />5gbit/s<br />Rambler<br />Wordpress<br />Grou.ps<br />
  24. 24. Postfix<br />Run multiple instances<br />Spam Clusters<br />
  25. 25. Monitoring<br />Munin + monit<br />Other alternatives:<br />Cacti<br />Nagios<br />Hyperic – vmware<br />
  26. 26. PHP<br />
  27. 27. More to come on my blog<br />http://emresokullu.com<br />More fine tuning tips<br />Become a member of my community<br />Love grou.ps ;)<br />Convert to PHP<br />We’re hiring: jobs@groups-inc.com<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×