微博架构与平台安全

@TimYang

0 ~ 50,000,000 3

1 PHP MySQL Apache Linux

• / • 1 / insert / select

• LAMP • MySQL , MyISAM • MPSS (Multi-Port Single Server)

• • • • •

• • • key-value (NoSQL) •

• • • MemcacheQ • stats queue

• InnoDB • PHP libmemcached memcache •

• • “ ” • • • MySQL •

• • • CDN • •

• Web • • API • • •

“Break large complex systems down into many services... google.com search touches 100s of services (ads, web search, books, news, spelling correction...)” - Jeff Dean, Google Fellow

→ →

• • feed cache • •

• DB • • IDC

• 50,000,000 • 3,000 / • 3,689,713 11 10

• • • •

Web cache

• • • • 100 10 • 1000 100 •

CPU IO

“CPU L1 L2 L3 ” - @ ecug 2010 Erlang

cache Inbox (home)

cache Inbox (home) Outbox (Myfeed)

cache Inbox (home) Outbox (Myfeed) Content

• • • Amazon S3: 99.9% • Amazon EC2: 99.95% • Facebook: n/a

• • • admission control... • , 7x24 • ,

• curl / • / access log • non-200 / / exceptions •

“Many services are written to alert operations on failure and to depend upon human intervention for recovery, about 20% of the time they will make mistakes. Designing for automation.” - James Hamilton,VP of Amazon

• • • • •

“System administration at Google usually have 1 week of "on call" duty, and the other 5 weeks are spent making improvements to make the on call portion more optimized, automated, and trouble- free” - Tom Limoncelli @ Everything Sysadmin Lumeta Corporation

• • IDC •

• • • •

• Master / Slave • master • Multi-Master • • • Paxos

Multi-Master Web

Yahoo! Message Broker

“We use YMB for replication for 2 reasons. 1.YMB ensure msgs are not lost before they are applied to the db. 2.YMB is designed for wide-area replication.This isolates individual PNUTS clusters from dealing with update between regions” PNUTS:Yahoo!’s Hosted Data Serving Platform

• API • Rest API • •

• (Stream API) • • • •

• 1 •

Weibo Data Msg MQ processor Queue Push DB push deamon Other Stream Server consumer

• • KISS Keep It Simple and Stupid • • batch insert • cursor read

Remote cache (Social Graph, nicks) Local Cache {uid:nick} Push {uid: Followers} data Receiver Social Stream Graph Service Stream Engine buffer resume continuation Weibo Jetty Continuation Servlet http://server/req?uid=1&since_id=2 Stream apps

• Stream Buffer • •

• • • •

Freq Realtime Post processing Content Ofﬂine Access log log analysis Activity log Abnormal Data Multi time monitor dimension data Weibo anti SPAM System

apps API requests • Auth url mapping • AppKey Auth Rate limit & Permission • OAuth Service method • • Output formatter(xml,json) Service

Join us! @TimYang

http://timyang.net	9989
http://www.alibuybuy.com	703
http://www.cnblogs.com	608
http://slj.me	506
http://www.phpit.cn	243
http://static.slidesharecdn.com	87
http://www.20ju.com	73
http://blog.bypat.com	72
http://www.php163.com	69
http://www.ourlinux.net	69
http://reader.youdao.com	67
http://blogread.cn	65
http://yunti.sinaapp.com	60
http://xianguo.com	53
http://irockman.me	49
http://blog.haohtml.com	46
http://www.zhuaxia.com	40
http://www.techgig.com	31
http://woshao.com	29
http://www.ililing.net	28
http://cache.baidu.com	25
http://zhuaxia.com	25
http://www.dcshi.com	19
http://blog.bookcold.me	14
http://www.cqumzh.cn	14
http://irockman.u4.30wan.com	13
http://www.blogread.cn	12
http://www.onlydoo.com	9
http://y.jmeye.com	8
http://tangpb.wordpress.com	8
http://www.w3ccollege.org	7
http://www.codesky.net	7
http://210.51.190.51:8002	7
http://3.alibuybuy.sinaapp.com	6
http://10.1.2.100	6
http://www.mojibar.com	5
http://feeds.feedburner.com	5
http://mailreader.163.com	3
http://www.zzfx.com	3
http://210.51.190.51	3
http://wiki.umlife.net	3
http://translate.googleusercontent.com	3
http://www.chuule.com	3
http://www.qicun.cn	3
http://www.yolao.com	3
http://r.mail.qq.com	2
http://old.xianguo.com	2
http://web.365legou.com	2
http://twittertim.es	2
http://twitter.com	2
http://www.lifeyun.com	2

微博架构与平台安全

by Tim Y on Nov 22, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

64 Embeds 13,127

Statistics

微博架构与平台安全 — Presentation Transcript