High Performance Weibo QCon Beijing 2011

Build High Performance Weibo System @TimYang

• • 140 • •

• QCon 2010• WDC 2010

Agenda3 42 1

Part 1

MySQL••

• hash•• user_timeline index

•• • • 5•

MySQL + cache••• cache • •

NoSQL“Databases are specializing – the “onesize ﬁts all” approach no longer applies.” MongoDB

NoSQL?• • Redis • MongoDB• • Cassandra

• MongoDB• Redis• HBase• Cassandra

Redis -• • snapshot - • vm - • diskstore - • aof -

Redis -• string: key value redisObject 16 bytes/item• list: 40 bytes / item• hash: zipmap(<64)• set/sorted set

Redis - Replication• rdb•

Redis -

Redis -•• failover• list/set (optional)

• MySQL• NoSQL MySQL cache

Part 2

→ →

Web cache cache•

“Web json cache json ( ) 2~5K , xml 10k , protobuf 500 ”

RDBMS→Key value (JSON)→Protocol buffers(binary)

JSON• DB• Cache• Message Queue• API

PB• Numeric: varint, from 1 byte•• Java, C++, Python...

•• cache • “192.168.0.1” → “0xc0a80000” → varint

Benchmark Text Text( http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking)

“We would like to provide public APIs thataccept protocol buffers as well as XML, bothbecause it is more efﬁcient and because werejust going to convert that XML to protocolbuffers on our end anyway.” - Google

•••

“ ”

• LAMP• 1ms•

• MQ stat• MQ Processor stat

?“1 ”

• Timeline•• • cache

1.2.3.4. cache master5. replication

• 5•

• 5• •

• 5• • •

• RAM is the new disk•

“Percona Server now both SQL and NOSQLHandlerSocket,100 rps 12cores/24threads and380GB of RAM, mysqlRedis NoSQL ”

@jackbillow “ 1. PK lookup 2. cache ”@kobe “ innodb ”@TimYang “InnoDB Adaptive Hash Indexeshash ”

• NoSQL • binlog Redis • jbinlog https://github.com/tangﬂ/jbinlog

Part 3

• • • spam••

•• GZIP•

Part 4

“@ Zheng:“ ” //@ :@bian " " http://t.cn/h0k4r (49)| (15)” 2 14 10:36 (22) | (10)

••

• • • •

• • • • cache•

••

Timeline Memcached MySQL Redis

•• RAM•

Q&A@TimYang

http://timyang.net	31543
http://www.mysqlops.com	2686
http://www.cnblogs.com	690
http://blog.caiwangqin.com	229
http://www.zhuaxia.com	115
http://reader.youdao.com	70
http://xianguo.com	67
http://yunti.sinaapp.com	63
http://static.slidesharecdn.com	63
http://yananguo.gotoip4.com	62
http://zhuaxia.com	55
http://blog.bypat.com	53
http://cache.baidu.com	22
http://flavors.me	22
http://www.chinatech.us	21
http://www.php163.com	20
http://www.zhangxiaona.net	19
http://www.dcshi.com	15
http://www.cqumzh.cn	14
http://www.mysqlops.com.	11
http://twitter.com	10
http://www.w3ccollege.org	9
http://translate.googleusercontent.com	8
https://broadvision.clearvale.com	6
http://webcache.googleusercontent.com	5
http://www.mysqlops.net	4
https://twitter.com	3
http://www.techgig.com	3
http://us-w1.rockmelt.com	3
http://paper.li	3
http://210.51.190.51:8002	3
http://chinaonrails.com	3
http://timyang.net.	2
http://old.xianguo.com	2
http://www.mysqlops.com HTTP	1
http://mysqlops.net	1
http://web-01.mysqlops.com	1
http://timyang.net} {1103846940\|\|\|pingback	1
http://localhost	1
http://x.wt	1
http://corponline.org	1
http://timyang.net} {887824539\|\|\|pingback	1
http://blog.tianya.cn	1
http://timyang.net} {846154348\|\|\|pingback	1
http://www.google.com	1
http://127.0.0.1	1
http://timyang.net} {1197229261\|\|\|pingback	1
http://mailreader.163.com	1
http://timyang.net} {1411794643\|\|\|pingback	1
http://timyang.net} {1469799522\|\|\|pingback	1
http://timyang.net} {1630763543\|\|\|pingback	1

High Performance Weibo QCon Beijing 2011

by Tim Y on Apr 10, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

56 Embeds 35,926

Statistics

High Performance Weibo QCon Beijing 2011 — Presentation Transcript