Kingso profile

1,673 views
1,620 views

Published on

Latex Beamer Hadoop

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,673
On SlideShare
0
From Embeds
0
Number of Embeds
868
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Kingso profile

  1. 1. Kingso Profile henshaoOctober 28, 2011 henshao Kingso Profile
  2. 2. Agenda henshao Kingso Profile
  3. 3. Agenda What is Profile? henshao Kingso Profile
  4. 4. Agenda What is Profile? How to build Profile? henshao Kingso Profile
  5. 5. Agenda What is Profile? How to build Profile? Let’s talk Kbuild henshao Kingso Profile
  6. 6. group henshao Kingso Profile
  7. 7. groupsegment henshao Kingso Profile
  8. 8. groupsegmentencode henshao Kingso Profile
  9. 9. group group profile group group segment segment profile detail 3500w profile 12GB $ ls | grep group profile group 0.seg 0 profile group 1.seg 0 profile group 2.seg 0 henshao Kingso Profile
  10. 10. segment group segment (1<<20) doc segment $ ls | grep seg profile group 0.seg 0 profile group 0.seg 1 profile group 0.seg 2 henshao Kingso Profile
  11. 11. encode provcity mlr feature prop vid group 6GB $ ls | grep encode cat id path.encode idx cat id path.encode cnt henshao Kingso Profile
  12. 12. henshao Kingso Profile
  13. 13. How to build Profile? : xml v3 segment segment offset henshao Kingso Profile
  14. 14. Profile
  15. 15. bitrecord 32 bits profile 32group varint 4 int32 t 1 byte varint henshao Kingso Profile
  16. 16. Kingso Profile doc doc isearch 1500w profile 9.5GB kingso 3500w 12GB cat id path 2GB DocAccessor profile henshao Kingso Profile
  17. 17. buildXML 80 henshao Kingso Profile
  18. 18. Let’s talk Kbuild ! henshao Kingso Profile
  19. 19. use Hadoop builder Hadoop xml builder merge merge index merge profile merge detail merge Hadoop streaming Hadoop streaming Hadoop Hadoop mapred.cache.files henshao Kingso Profile
  20. 20. task 1.6GB*18 7index 15GB 4profile 19GB 9 28GB 9task 21GB 6 henshao Kingso Profile
  21. 21. Hadoop Happy Hadoop cpu henshao Kingso Profile
  22. 22. mapred.map.tasks.speculative.execution=false data node tarindex tar rmhadoop fs -cat index.tar | tar xf - -C outputtar -c index | hadoop fs -put - index.target/put index profile detaildetail index profile job henshao Kingso Profile
  23. 23. Job 720 map slot Job Job 20 tasknode henshao Kingso Profile
  24. 24. The end Thank you! henshao Kingso Profile

×