Your SlideShare is downloading. ×
0
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Alexander Ignatyev "MapReduce infrastructure"
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Alexander Ignatyev "MapReduce infrastructure"

1,262

Published on

Семинар «Использование современных информационных технологий для решения современных задач физики частиц» в московском офисе Яндекса, 3 июля 2012 …

Семинар «Использование современных информационных технологий для решения современных задач физики частиц» в московском офисе Яндекса, 3 июля 2012

Александр Игнатьев, разработчик MapReduce, Yandex

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,262
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1.   MapReduce  Alexander  Ignatyev   Yandex  
  • 2. Classic  MapReduce  
  • 3. Yandex  MapReduce  Search  Quality  Team:  •  5  clusters  •  2,000  hosts  •  10  PB  data  •  3  TB  new  data  a  day  (only  user  logs)  •  100  users  •  2,000,000  tables    
  • 4. Structure  of  MapReduce  cluster    Hosts  specificaNon:  •  6  x  8  GB  RAM  •  2  x  6-­‐cores  Xeon  CPU    •  4  x  2  GB  HDD  •  1  Gb  Ethernet  
  • 5. Yandex  MapReduce   Tables  •  Table  consists  of  a  number  of  records  •  Record  is  a  key,  subkey  and  value  tuple  •  Table  consists  of  a  number  of  chunks  •  Size  of  chunk  is  126  MB  •  Each  chunks  has  several  replicas  (usually  3)  
  • 6. Sklad  •  Minimal  overhead  costs  file  system  for   MapReduce  •  Great  name:  Storehouse    
  • 7. netliba  •  Tolerant  algorithm  of  congesNon  control   traffic  of  network  allows  us  to  increase   available  bandwidth  of  network.  •  UDP-­‐based  •  Reliable  transmission  •  Support  IPv6  
  • 8. mr_apps   UNIX-­‐like  toolset  MapReduce  u*l   Descrip*on  mr_cat   cat  -­‐  merge  tables  mr_cp   cp  –  copy  tables  mr_diff   diff  –  compare  tables  mr_du   du  –  display  disk  usage  staNsNcs  mr_grep   grep  –  display  records  matching  a  pa`ern  mr_head   head  –  print  top  records  mr_ls   ls  –  print  list  of  tables  mr_mv   mv  –  move  tables  mr_wc   wc  –  print  number  of  keys  or  records  mr_hist   print  keys  distribuNon  
  • 9. Simplicity  of  Yandex  MapReduce  •  Simple  maintenance:   Run  Simple  Cluster  #  mapreduce  –runserver  8013  –h`p  8080  #  mapreduce  –runhost  127.0.0.1:8013  #  mapreduce  –runhost  127.0.0.1:8013  •  Simple  programming  
  • 10. Yandex  MapReduce   Map  OperaNon  
  • 11. Yandex  MapReduce   Reduce  OperaNon  
  • 12. Yandex  MapReduce  Reduce  Without  SorNng  OperaNon  
  • 13. Yandex  MapReduce   Sort  OperaNon  
  • 14. Yandex.Tables  (YT)   New  GeneraNon  of  MapReduce  •  Tables  have  flexible  structure:   –  Custom  columns   –  Composite  keys   –  Column  selecNon  read  tables  •  Triple  masters:  no  single  point  of  failure  •  New  tools  for  monitoring  •  New  API  
  • 15. Thank  you  for  your  a`enNon!     QuesNons?  

×