Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

High Performance Weibo QCon Beijing 2011

64,351 views

Published on

Sina Weibo is the most popular Microblogging platform in China. It has more than 100 million user and tens of millions of daily updates. This slide explains the performance challenges in Weibo platform.

Published in: Technology

High Performance Weibo QCon Beijing 2011

  1. 1. Build High Performance Weibo System @TimYang
  2. 2. • • 140 • •
  3. 3. • QCon 2010• WDC 2010
  4. 4. Agenda3 42 1
  5. 5. Part 1
  6. 6. ( )
  7. 7. MySQL••
  8. 8. • hash•• user_timeline index
  9. 9. •• • • 5•
  10. 10. MySQL + cache••• cache • •
  11. 11. NoSQL“Databases are specializing – the “onesize fits all” approach no longer applies.” MongoDB
  12. 12. NoSQL?• • Redis • MongoDB• • Cassandra
  13. 13. • MongoDB• Redis• HBase• Cassandra
  14. 14. Redis -• • snapshot - • vm - • diskstore - • aof -
  15. 15. Redis -• string: key value redisObject 16 bytes/item• list: 40 bytes / item• hash: zipmap(<64)• set/sorted set
  16. 16. Redis - Replication• rdb•
  17. 17. Redis -
  18. 18. Redis -•• failover• list/set (optional)
  19. 19. • MySQL• NoSQL MySQL cache
  20. 20. Part 2
  21. 21. ( )
  22. 22. → →
  23. 23. Web cache cache•
  24. 24. “Web json cache json ( ) 2~5K , xml 10k , protobuf 500 ”
  25. 25. RDBMS→Key value (JSON)→Protocol buffers(binary)
  26. 26. JSON• DB• Cache• Message Queue• API
  27. 27. PB• Numeric: varint, from 1 byte•• Java, C++, Python...
  28. 28. •• cache • “192.168.0.1” → “0xc0a80000” → varint
  29. 29. Benchmark Text Text( http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking)
  30. 30. “We would like to provide public APIs thataccept protocol buffers as well as XML, bothbecause it is more efficient and because werejust going to convert that XML to protocolbuffers on our end anyway.” - Google
  31. 31. •••
  32. 32. “ ”
  33. 33. • LAMP• 1ms•
  34. 34. • MQ stat• MQ Processor stat
  35. 35. ?“1 ”
  36. 36. • Timeline•• • cache
  37. 37. 1.2.3.4. cache master5. replication
  38. 38. • 5
  39. 39. • 5•
  40. 40. • 5• •
  41. 41. • 5• • •
  42. 42. • RAM is the new disk•
  43. 43. “Percona Server now both SQL and NOSQLHandlerSocket,100 rps 12cores/24threads and380GB of RAM, mysqlRedis NoSQL ”
  44. 44. @jackbillow “ 1. PK lookup 2. cache ”@kobe “ innodb ”@TimYang “InnoDB Adaptive Hash Indexeshash ”
  45. 45. • NoSQL • binlog Redis • jbinlog https://github.com/tangfl/jbinlog
  46. 46. Part 3
  47. 47. app
  48. 48. • • • spam••
  49. 49. Text
  50. 50. •• GZIP•
  51. 51. Part 4
  52. 52. “@ Zheng:“ ” //@ :@bian " " http://t.cn/h0k4r (49)| (15)” 2 14 10:36 (22) | (10)
  53. 53. ••
  54. 54. • • • •
  55. 55. • • • • cache•
  56. 56. ••
  57. 57. Timeline Memcached MySQL Redis
  58. 58. •• RAM•
  59. 59. Q&A@TimYang

×