Your SlideShare is downloading. ×
0
MyFOX技术第二讲<br />扩容与数据迁移<br />朋春<br />@我是aleafs<br />
回顾<br />
关于MyFOX<br /><ul><li>分布式MySQL集群中间层
数据离线批量写入
透明的数据查询代理
集群管理</li></li></ul><li>节点结构<br />MyFOX<br />路由表<br />8冷节点(MySQL)<br />30天无访问的冷数据<br />新增热数据<br />7.2k SATA盘,1T * 12,raid10...
路由表<br />…,cid=2<br />KEY: thedate=20110811,cid=1<br />…<br />node<br />path<br />modtime<br />hittime<br />…<br />dbname....
装完数据<br />UPDATE ... SETmodtime = NOW() WHERE ...<br />modtime:<br />每条路由对应的分片的数据更新时间<br />
路由访问<br />SELECT node, path, modtimeFROM ... WHERE ...<br />modtime:<br />if (force_load ||!cache || modtime > cache.savet...
同时…<br />if (rand(1,100) <= 10) {<br />}<br />UPDATE ... SEThittime= NOW()WHERE ...<br />Why we use “rand” here?<br />NOT ...
决策<br />SELECT node, path FROM ... <br />WHEREhittime < ’30天前 ’ <br />	ANDisarchive = 0<br />
@letonlife:<br />“…搬运工伤不起”<br />
工作队列<br />SELECT ... FROMtask_queque<br />WHEREtask_flag = FLAG_WAITANDtrytimes < 3<br />ORDER BY priority ASC, <br />tryt...
实施<br />scp (*.MYI, *.MYD)<br />MySQL<br />MySQL<br />MySQL<br />MySQL<br />MySQL<br />MySQL<br />LOAD<br />SELECT<br />脚本...
Federated<br />“The FEDERATED storage engine is available beginning with MySQL 5.0.3. It is a storage engine that accesses...
CREATE TABLE dbname.t_2_fed (<br />	...<br />) ENGINE = FEDERATED <br />CONNECTION=‘mysql://user:pass@172.1.1.1/dbname/t_2...
真相总是在背后<br />如何选择一台“最优”的机器作为源表来复制呢?<br />dbname.t_2<br />dbname.t_2<br />
例如:<br />var task = {<br />	“path”  : “dim_category_arch_0.t_92d_1”,<br />	“from”  : “1,2,3,4,5,6,7,8”,<br />	“save”  : “1...
对于 172.001.003.017而言,fed表指向谁?<br /><ul><li> 172.001.001.001
 172.002.001.001
 172.001.003.002
 172.002.021.002</li></li></ul><li>就近原则<br /><ul><li>快速
低碳
稳定
安全
同机房
同机柜
Upcoming SlideShare
Loading in...5
×

My fox 扩容与数据迁移

1,781

Published on

淘宝数据产品的分布式MySQL集群的扩容原理,怎么让40台mysql互联互通,进行数据的随时迁移。

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,781
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
66
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "My fox 扩容与数据迁移"

  1. 1. MyFOX技术第二讲<br />扩容与数据迁移<br />朋春<br />@我是aleafs<br />
  2. 2. 回顾<br />
  3. 3. 关于MyFOX<br /><ul><li>分布式MySQL集群中间层
  4. 4. 数据离线批量写入
  5. 5. 透明的数据查询代理
  6. 6. 集群管理</li></li></ul><li>节点结构<br />MyFOX<br />路由表<br />8冷节点(MySQL)<br />30天无访问的冷数据<br />新增热数据<br />7.2k SATA盘,1T * 12,raid10<br />内存:24G<br />成本:1.6W / T<br />12热节点(MySQL)<br />15k SAS盘,300G * 12,raid10<br />内存:24G<br />成本:4.5W / T<br />
  7. 7. 路由表<br />…,cid=2<br />KEY: thedate=20110811,cid=1<br />…<br />node<br />path<br />modtime<br />hittime<br />…<br />dbname.t_1<br />dbname.t_2<br />dbname.t_2<br />节点2<br />节点1<br />
  8. 8. 装完数据<br />UPDATE ... SETmodtime = NOW() WHERE ...<br />modtime:<br />每条路由对应的分片的数据更新时间<br />
  9. 9. 路由访问<br />SELECT node, path, modtimeFROM ... WHERE ...<br />modtime:<br />if (force_load ||!cache || modtime > cache.savetime) {<br /> // load data from mysql<br />}<br />
  10. 10. 同时…<br />if (rand(1,100) <= 10) {<br />}<br />UPDATE ... SEThittime= NOW()WHERE ...<br />Why we use “rand” here?<br />NOT ONLY...<br />BUT ALSO...<br />
  11. 11. 决策<br />SELECT node, path FROM ... <br />WHEREhittime < ’30天前 ’ <br /> ANDisarchive = 0<br />
  12. 12. @letonlife:<br />“…搬运工伤不起”<br />
  13. 13. 工作队列<br />SELECT ... FROMtask_queque<br />WHEREtask_flag = FLAG_WAITANDtrytimes < 3<br />ORDER BY priority ASC, <br />trytimesASC,<br />ABS(task_position – AGENT_POSITION) ASC,<br />autokidASC<br />
  14. 14. 实施<br />scp (*.MYI, *.MYD)<br />MySQL<br />MySQL<br />MySQL<br />MySQL<br />MySQL<br />MySQL<br />LOAD<br />SELECT<br />脚本部署在DB上,有安全隐患<br />APP<br />APP<br />两次落地,两次网络传输<br />
  15. 15. Federated<br />“The FEDERATED storage engine is available beginning with MySQL 5.0.3. It is a storage engine that accesses data in tables of remote databases rather than in local tables.”<br />http://dev.mysql.com/doc/refman/5.0/en/federated-storage-engine.html<br />
  16. 16. CREATE TABLE dbname.t_2_fed (<br /> ...<br />) ENGINE = FEDERATED <br />CONNECTION=‘mysql://user:pass@172.1.1.1/dbname/t_2’<br />dbname.t_2_fed<br />dbname.t_2<br />dbname.t_2<br />172.1.1.1<br />172.1.1.2<br />INSERT INTO dbname.t_2 SELECT * FROM dbname.t_2_fed<br />
  17. 17. 真相总是在背后<br />如何选择一台“最优”的机器作为源表来复制呢?<br />dbname.t_2<br />dbname.t_2<br />
  18. 18. 例如:<br />var task = {<br /> “path” : “dim_category_arch_0.t_92d_1”,<br /> “from” : “1,2,3,4,5,6,7,8”,<br /> “save” : “17,18,19,20”,<br />}<br />
  19. 19. 对于 172.001.003.017而言,fed表指向谁?<br /><ul><li> 172.001.001.001
  20. 20. 172.002.001.001
  21. 21. 172.001.003.002
  22. 22. 172.002.021.002</li></li></ul><li>就近原则<br /><ul><li>快速
  23. 23. 低碳
  24. 24. 稳定
  25. 25. 安全
  26. 26. 同机房
  27. 27. 同机柜
  28. 28. 同交换机</li></ul>IP地址“体现”物理部署<br />
  29. 29. 一致性检查<br /><ul><li>COUNT
  30. 30. XOR</li></li></ul><li>修改“指针”<br />node=9,path=dbname.t_2<br />node=2,path=dbname.t_2<br />
  31. 31. 卸磨杀驴<br />/* 数据搬完第二天:*/<br />DROP TABLE IF EXISTS dbname.t_2/* ON NODE 2 */<br />Why NOT “right now”?<br />
  32. 32. 小结<br />
  33. 33. 从16到40<br /><ul><li>Federated
  34. 34. 就近原则</li></li></ul><li>2000台机器的时候<br />MyFOX<br />...<br />route<br />route<br />route<br />route<br />MySQL<br />MySQL<br />MySQL<br />MySQL<br />...<br />chunk server<br />chunk server<br />
  35. 35. 干掉“节点”<br /><ul><li>Mysql承担chunk server
  36. 36. 每个chunk跨机房三份冗余
  37. 37. 如果宕机,立即复制
  38. 38. …</li></li></ul><li>预告<br />第三讲:数据压缩<br />
  39. 39. Thanks<br />@我是aleafs<br />pengchun@taobao.com<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×