High Performance Weibo QCon Beijing 2011

61,792
-1

Published on

Sina Weibo is the most popular Microblogging platform in China. It has more than 100 million user and tens of millions of daily updates. This slide explains the performance challenges in Weibo platform.

Published in: Technology
3 Comments
166 Likes
Statistics
Notes
No Downloads
Views
Total Views
61,792
On Slideshare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
1,988
Comments
3
Likes
166
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • 细化索引拆分\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • 比较图表\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • json及性能对比\n
  • \n
  • \n
  • \n
  • 相同空间,5倍容量\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • 增加图表等接口成果\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • 迅速定位\n
  • \n
  • \n
  • \n
  • \n
  • High Performance Weibo QCon Beijing 2011

    1. 1. Build High Performance Weibo System @TimYang
    2. 2. • • 140 • •
    3. 3. • QCon 2010• WDC 2010
    4. 4. Agenda3 42 1
    5. 5. Part 1
    6. 6. ( )
    7. 7. MySQL••
    8. 8. • hash•• user_timeline index
    9. 9. •• • • 5•
    10. 10. MySQL + cache••• cache • •
    11. 11. NoSQL“Databases are specializing – the “onesize fits all” approach no longer applies.” MongoDB
    12. 12. NoSQL?• • Redis • MongoDB• • Cassandra
    13. 13. • MongoDB• Redis• HBase• Cassandra
    14. 14. Redis -• • snapshot - • vm - • diskstore - • aof -
    15. 15. Redis -• string: key value redisObject 16 bytes/item• list: 40 bytes / item• hash: zipmap(<64)• set/sorted set
    16. 16. Redis - Replication• rdb•
    17. 17. Redis -
    18. 18. Redis -•• failover• list/set (optional)
    19. 19. • MySQL• NoSQL MySQL cache
    20. 20. Part 2
    21. 21. ( )
    22. 22. → →
    23. 23. Web cache cache•
    24. 24. “Web json cache json ( ) 2~5K , xml 10k , protobuf 500 ”
    25. 25. RDBMS→Key value (JSON)→Protocol buffers(binary)
    26. 26. JSON• DB• Cache• Message Queue• API
    27. 27. PB• Numeric: varint, from 1 byte•• Java, C++, Python...
    28. 28. •• cache • “192.168.0.1” → “0xc0a80000” → varint
    29. 29. Benchmark Text Text( http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking)
    30. 30. “We would like to provide public APIs thataccept protocol buffers as well as XML, bothbecause it is more efficient and because werejust going to convert that XML to protocolbuffers on our end anyway.” - Google
    31. 31. •••
    32. 32. “ ”
    33. 33. • LAMP• 1ms•
    34. 34. • MQ stat• MQ Processor stat
    35. 35. ?“1 ”
    36. 36. • Timeline•• • cache
    37. 37. 1.2.3.4. cache master5. replication
    38. 38. • 5
    39. 39. • 5•
    40. 40. • 5• •
    41. 41. • 5• • •
    42. 42. • RAM is the new disk•
    43. 43. “Percona Server now both SQL and NOSQLHandlerSocket,100 rps 12cores/24threads and380GB of RAM, mysqlRedis NoSQL ”
    44. 44. @jackbillow “ 1. PK lookup 2. cache ”@kobe “ innodb ”@TimYang “InnoDB Adaptive Hash Indexeshash ”
    45. 45. • NoSQL • binlog Redis • jbinlog https://github.com/tangfl/jbinlog
    46. 46. Part 3
    47. 47. app
    48. 48. • • • spam••
    49. 49. Text
    50. 50. •• GZIP•
    51. 51. Part 4
    52. 52. “@ Zheng:“ ” //@ :@bian " " http://t.cn/h0k4r (49)| (15)” 2 14 10:36 (22) | (10)
    53. 53. ••
    54. 54. • • • •
    55. 55. • • • • cache•
    56. 56. ••
    57. 57. Timeline Memcached MySQL Redis
    58. 58. •• RAM•
    59. 59. Q&A@TimYang
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×