Buzztterの裏側とその周辺技術

•

18 likes•4,274 views

Yoji Shidara

Technology

w
w TFtgt DFtgt TFref DFref

w TFtgt
DFtgt

w TFref
DFref

>> t = Time.parse(quot;2007-11-3quot;)
=> Sat Nov 03 00:00:00 +0900 2007

>> Status.count(:conditions=>[quot;created_at
BETWEEN ? AND ?quot;, t, t.tomorrow])
=> 125626

Tue Nov 06 15:17:40 +0900 2007 - received 8 / 20, 5793 tuples
Tue Nov 06 15:17:45 +0900 2007 - received 10 / 20, 5794 tuples
Tue Nov 06 15:17:51 +0900 2007 - received 10 / 20, 5798 tuples
Tue Nov 06 15:17:55 +0900 2007 - received 4 / 20, 5797 tuples
Tue Nov 06 15:18:00 +0900 2007 - received 5 / 20, 5797 tuples
Tue Nov 06 15:18:05 +0900 2007 - received 11 / 20, 5797 tuples
Tue Nov 06 15:18:12 +0900 2007 - received 8 / 20, 5802 tuples
Tue Nov 06 15:18:16 +0900 2007 - received 9 / 20, 5807 tuples
Tue Nov 06 15:18:21 +0900 2007 - received 8 / 20, 5809 tuples
Tue Nov 06 15:18:25 +0900 2007 - received 12 / 20, 5810 tuples
Tue Nov 06 15:18:30 +0900 2007 - received 10 / 20, 5812 tuples
Tue Nov 06 15:18:35 +0900 2007 - received 13 / 20, 5817 tuples
Tue Nov 06 15:18:40 +0900 2007 - received 3 / 20, 5811 tuples
Tue Nov 06 15:18:45 +0900 2007 - received 5 / 20, 5811 tuples
Tue Nov 06 15:18:50 +0900 2007 - received 15 / 20, 5820 tuples
Tue Nov 06 15:18:55 +0900 2007 - received 14 / 20, 5826 tuples
Tue Nov 06 15:19:01 +0900 2007 - received 3 / 20, 5823 tuples
Tue Nov 06 15:19:08 +0900 2007 - received 8 / 20, 5814 tuples
Tue Nov 06 15:19:12 +0900 2007 - received 8 / 20, 5822 tuples
Tue Nov 06 15:19:18 +0900 2007 - received 10 / 20, 5818 tuples

i j

i, j
j
Ci,j = P (tk−1 |tk )P (tk+1 |tk )
k=i

Ci,j < 0.75
i..j

$count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM statuses WHERE #{IGNORE_COND} AND language = ? AND (created_at BETWEEN ? AND ?) AND text @@ ?quot;, language, t.ago(ago), t, add_pragma(word)]$

2007-11-06 13:19:45 ANALYZER-ng(22499) begin for japanese-utf8
2007-11-06 13:19:46 ANALYZER-ng(22499) extracted 3120 sentences
2007-11-06 13:20:12 ANALYZER-ng(22499) 6006 keywords extracted from 3120 sentences
2007-11-06 13:20:12 ANALYZER-ng(22499) deleting stopwords ...
2007-11-06 13:20:19 ANALYZER-ng(22499) odd terms removed (5902 terms)
2007-11-06 13:20:19 ANALYZER-ng(22499) ignore case (5895 terms)
2007-11-06 13:20:19 ANALYZER-ng(22499) trivial terms are removed (1796 terms)
2007-11-06 13:21:38 ANALYZER-ng(22499) occurrence calculated (72.738133 s)
2007-11-06 13:23:35 ANALYZER-ng(22499) modified DDFs calculated
2007-11-06 13:23:35 ANALYZER-ng(22499) scores calculated (1563 terms)
2007-11-06 13:23:40 ANALYZER-ng(22499) redundant terms removed (1151 terms)
2007-11-06 13:23:42 ANALYZER-ng(22499) end for japanese-utf8 (237.531316 s)

2007-11-06 13:23:42 ANALYZER-ng(22499) begin for english
2007-11-06 13:23:43 ANALYZER-ng(22499) extracted 6181 sentences
2007-11-06 13:24:20 ANALYZER-ng(22499) 10168 keywords extracted from 6181 sentences
2007-11-06 13:24:20 ANALYZER-ng(22499) deleting stopwords ...
2007-11-06 13:24:33 ANALYZER-ng(22499) odd terms removed (9808 terms)
2007-11-06 13:24:33 ANALYZER-ng(22499) ignore case (9444 terms)
2007-11-06 13:24:33 ANALYZER-ng(22499) trivial terms are removed (2738 terms)
2007-11-06 13:26:18 ANALYZER-ng(22499) occurrence calculated (96.306258 s)
2007-11-06 13:27:59 ANALYZER-ng(22499) modified DDFs calculated
2007-11-06 13:27:59 ANALYZER-ng(22499) scores calculated (2109 terms)
2007-11-06 13:28:10 ANALYZER-ng(22499) redundant terms removed (1643 terms)
2007-11-06 13:28:13 ANALYZER-ng(22499) end for english (270.044345 s)

Recently uploaded

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren

Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Next-generation AAM aircraft unveiled by Supernal, S-A2Hyundai Motor Group

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

Key Features Of Token Development (1).pptxLBM Solutions

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard

#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh

How to convert PDF to text with Nanonetsnaman860154

Vulnerability_Management_GRC_by Sohang Sengupta.pptxnull - The Open Security Community

Artificial intelligence in the post-deep learning eraDeakin University

Snow Chain-Integrated Tire for a Safe Drive on Winter RoadsHyundai Motor Group

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

Recently uploaded (20)

Injustice - Developers Among Us (SciFiDevCon 2024)

Advanced Test Driven-Development @ php[tek] 2024

Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Next-generation AAM aircraft unveiled by Supernal, S-A2

08448380779 Call Girls In Civil Lines Women Seeking Men

Key Features Of Token Development (1).pptx

Human Factors of XR: Using Human Factors to Design XR Systems

Benefits Of Flutter Compared To Other Frameworks

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Maximizing Board Effectiveness 2024 Webinar.pptx

#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi

How to convert PDF to text with Nanonets

Vulnerability_Management_GRC_by Sohang Sengupta.pptx

Artificial intelligence in the post-deep learning era

Snow Chain-Integrated Tire for a Safe Drive on Winter Roads

Presentation on how to chat with PDF using ChatGPT code interpreter

Buzztterの裏側とその周辺技術

8. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref

10.

11.

12. >> t = Time.parse(quot;2007-11-3quot;) => Sat Nov 03 00:00:00 +0900 2007 >> Status.count(:conditions=>[quot;created_at BETWEEN ? AND ?quot;, t, t.tomorrow]) => 125626

13.

14.

15.

16.

17. Tue Nov 06 15:17:40 +0900 2007 - received 8 / 20, 5793 tuples Tue Nov 06 15:17:45 +0900 2007 - received 10 / 20, 5794 tuples Tue Nov 06 15:17:51 +0900 2007 - received 10 / 20, 5798 tuples Tue Nov 06 15:17:55 +0900 2007 - received 4 / 20, 5797 tuples Tue Nov 06 15:18:00 +0900 2007 - received 5 / 20, 5797 tuples Tue Nov 06 15:18:05 +0900 2007 - received 11 / 20, 5797 tuples Tue Nov 06 15:18:12 +0900 2007 - received 8 / 20, 5802 tuples Tue Nov 06 15:18:16 +0900 2007 - received 9 / 20, 5807 tuples Tue Nov 06 15:18:21 +0900 2007 - received 8 / 20, 5809 tuples Tue Nov 06 15:18:25 +0900 2007 - received 12 / 20, 5810 tuples Tue Nov 06 15:18:30 +0900 2007 - received 10 / 20, 5812 tuples Tue Nov 06 15:18:35 +0900 2007 - received 13 / 20, 5817 tuples Tue Nov 06 15:18:40 +0900 2007 - received 3 / 20, 5811 tuples Tue Nov 06 15:18:45 +0900 2007 - received 5 / 20, 5811 tuples Tue Nov 06 15:18:50 +0900 2007 - received 15 / 20, 5820 tuples Tue Nov 06 15:18:55 +0900 2007 - received 14 / 20, 5826 tuples Tue Nov 06 15:19:01 +0900 2007 - received 3 / 20, 5823 tuples Tue Nov 06 15:19:08 +0900 2007 - received 8 / 20, 5814 tuples Tue Nov 06 15:19:12 +0900 2007 - received 8 / 20, 5822 tuples Tue Nov 06 15:19:18 +0900 2007 - received 10 / 20, 5818 tuples

18.

19.

20. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref

21. k

22.

23.

24. i j i, j j Ci,j = P (tk−1 |tk )P (tk+1 |tk ) k=i Ci,j < 0.75 i..j

25.

26.

27. count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM statuses WHERE #{IGNORE_COND} AND language = ? AND (created_at BETWEEN ? AND ?) AND text @@ ?quot;, language, t.ago(ago), t, add_pragma(word)]

28. 2007-11-06 13:19:45 ANALYZER-ng(22499) begin for japanese-utf8 2007-11-06 13:19:46 ANALYZER-ng(22499) extracted 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) 6006 keywords extracted from 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:20:19 ANALYZER-ng(22499) odd terms removed (5902 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) ignore case (5895 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) trivial terms are removed (1796 terms) 2007-11-06 13:21:38 ANALYZER-ng(22499) occurrence calculated (72.738133 s) 2007-11-06 13:23:35 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:23:35 ANALYZER-ng(22499) scores calculated (1563 terms) 2007-11-06 13:23:40 ANALYZER-ng(22499) redundant terms removed (1151 terms) 2007-11-06 13:23:42 ANALYZER-ng(22499) end for japanese-utf8 (237.531316 s) 2007-11-06 13:23:42 ANALYZER-ng(22499) begin for english 2007-11-06 13:23:43 ANALYZER-ng(22499) extracted 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) 10168 keywords extracted from 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:24:33 ANALYZER-ng(22499) odd terms removed (9808 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) ignore case (9444 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) trivial terms are removed (2738 terms) 2007-11-06 13:26:18 ANALYZER-ng(22499) occurrence calculated (96.306258 s) 2007-11-06 13:27:59 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:27:59 ANALYZER-ng(22499) scores calculated (2109 terms) 2007-11-06 13:28:10 ANALYZER-ng(22499) redundant terms removed (1643 terms) 2007-11-06 13:28:13 ANALYZER-ng(22499) end for english (270.044345 s)

Buzztterの裏側とその周辺技術

Recommended

Recommended

More Related Content

More from Yoji Shidara

More from Yoji Shidara (12)

Recently uploaded

Recently uploaded (20)

Buzztterの裏側とその周辺技術