Inside Buzztter

4,177 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,177
On SlideShare
0
From Embeds
0
Number of Embeds
2,010
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Inside Buzztter

  1. 1. RubyConf 2009 2009-11-19Inside buzztter設樂 洋爾 (Yoji Shidara)yoji.shidara@enishi-tech.com
  2. 2. My name isYoji SHIDARA.
  3. 3. Enishi Tech Inc.Small Company.Only 2 staffs.
  4. 4. Small Office.(CEO’s flat)
  5. 5. I’m from Sapporo! @darashi (me)Sapporo 1.4hrs 9.5hrsTokyo San Fransisco!
  6. 6. Sapporo is abeautiful provincialcity of Japan.
  7. 7. Delicious foods arewaiting for you!
  8. 8. Do you knowbuzztter.com?
  9. 9. RubyConf 2009 2009-11-19 LTOutside buzztter設樂 洋爾 (Yoji Shidara)yoji.shidara@enishi-tech.com
  10. 10. http://buzztter.com
  11. 11. http://buzztter.com buzz-phrases (auto update)Full-text Search Results (auto update)
  12. 12. Servers are in my room! next to my bed
  13. 13. We crawls:1,932,823 tweets/day (Japanese) 607,749 tweets/day (English) (We’re now focusing on Japanese users due to API limits) We have: 1,336,444 PVs / Month
  14. 14. Server 1: Phenom 9950 Memory 8GB SSD 80GB + HDD 80GBServer 2: PhenomII x4 955 Memory 8GB HDD 500GB
  15. 15. @buzztter
  16. 16. Bot for English wasbanned because it tweeted spam as hot :-<
  17. 17. Buzztter:
  18. 18. buzz-phrase detection +real time full text search
  19. 19. ... like official“trending topics”
  20. 20. buzztter releasedbefore the official one launched!
  21. 21. buzztter supports English and Japanese!
  22. 22. 1-2 minutes forbuzz analysis par a language.
  23. 23. HOT notification bot is available.
  24. 24. RubyConf 2009 2009-11-19 LTInside buzztter設樂 洋爾 (Yoji Shidara)yoji.shidara@enishi-tech.com
  25. 25. System Diagram:Rough Sketch Web Notification Frontend module crawler Fulltext Analyzer module Search module BuzzDAS: Buzz Data Analysis System
  26. 26. System Diagram: Crawler Typhoeus Twitter crawler EventMachine RabbitMQ MQ crawl crawl scheduling controller libtextcatUsers DB langugage language, guesserPostgreSQL post frequency
  27. 27. notifier pagecachecrawler bot varnish Twitter4R Web Frontend net/irc Rails MQ Web Service API Sinatra memcached groonga fulltext importer search analyzer engineBuzzDAS: Buzz Data Analysis System
  28. 28. analyzer tokenizer phrase extractor detects change of phrase occurrence Reference Recent Index Indexkeeps post in 24 hours keeps recent 1 hour
  29. 29. Try http://buzztter.com and giveme your feedback please!If you Interested in thesekeywords:groonga, AMQP, RabbitMQ, Typhoeus,EventMachine, PostgreSQL, libtextcat,Sinatra, Rails, Twitter4R, net/irc,memcached, PrefixSpan ,MeCab, TF-IDF, ... or our BuzzDAS Engine, please contact me! My name is Yoji Shidara.

×