Open Network Live - Hack Day Report

1,591 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,591
On SlideShare
0
From Embeds
0
Number of Embeds
71
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Open Network Live - Hack Day Report

  1. 1. Chirp Hack Day Report ) wada@garage.co.jp @Koichi
  2. 2. •  –  wada@garage.co.jp –  @Koichi •  –  http://twinavi.jp
  3. 3. Chirp •  4/14, 15 in San Francisco •  –  Chirp –  Hack Day
  4. 4. Chirp
  5. 5. 1 -Conference
  6. 6. 1 - Hack Day Start
  7. 7. 1 - Ignite
  8. 8. 1 - Coding time
  9. 9. 2 -
  10. 10. -Sessions
  11. 11. 2 -Lunch, Meet The founders
  12. 12. •  2 •  • 
  13. 13. Hack Day
  14. 14. •  • 
  15. 15. •  –  –  •  SPAM –  DM •  – 
  16. 16. (2) •  –  Tweet Display Guidlines •  http://media.twitter.com/14/tweet-display-guidelines –  Terms Of Service •  http://twitter.com/tos •  •  Awesome –  !
  17. 17. (3) •  API Cache •  OAuth Key
  18. 18. •  •  at slideshare - #chirppolicy –  We Have Faith in (Most of) You: How Twitter Crafts Policies to Allow Good Apps to Thrive –  http://www.slideshare.net/delbius/ chirppolicy
  19. 19. Twitter 7TB/ Chirp GB
  20. 20. Challenge •  •  & • 
  21. 21. •  syslog-ng • 
  22. 22. •  Scribe –  Facebook –  Thrift – 
  23. 23. Scribe •  –  •  –  •  HDFS
  24. 24. •  7TB/ •  80MB/s •  24.3 •  • 
  25. 25. •  Hadoop –  –  MapReduce –  –  Y! 4000 –  1TB 62
  26. 26. •  MySQL –  : COUNT, GROUP –  : JOIN •  Hadoop –  5 –  – 
  27. 27. •  Java •  –  MapReduce – 
  28. 28. •  Pig –  –  SQL –  –  1
  29. 29. Pig sample users = load ‘users.csv’ as (username: charaarray, age: int); users_1825 = filter users by age >= 18 and age <=25; pages = load ‘pages.csv’ as (username: chararay, url: chararray) joined = join users_1825 by username, pages by username; grouped = group joined by url; summed = foreach grouped generate group as url, COUNT (joined) AS views; sorted = order summed by views desc; top_5 = limit sorted 5; store top_t into ‘top_5_sites.csv’
  30. 30. Java 5%
  31. 31. •  –  Scribe –  Hadoop –  Pig •  at slideshare - #chirpdata –  Analyzing Big Data at Twitter –  http://www.slideshare.net/kevinweil/big-data-at- twitter-chirp-2010
  32. 32. •  •  • 

×