Buzztterの裏側とその周辺技術

•

18 likes•4,274 views

Yoji Shidara

Technology

w
w TFtgt DFtgt TFref DFref

w TFtgt
DFtgt

w TFref
DFref

>> t = Time.parse(quot;2007-11-3quot;)
=> Sat Nov 03 00:00:00 +0900 2007

>> Status.count(:conditions=>[quot;created_at
BETWEEN ? AND ?quot;, t, t.tomorrow])
=> 125626

Tue Nov 06 15:17:40 +0900 2007 - received 8 / 20, 5793 tuples
Tue Nov 06 15:17:45 +0900 2007 - received 10 / 20, 5794 tuples
Tue Nov 06 15:17:51 +0900 2007 - received 10 / 20, 5798 tuples
Tue Nov 06 15:17:55 +0900 2007 - received 4 / 20, 5797 tuples
Tue Nov 06 15:18:00 +0900 2007 - received 5 / 20, 5797 tuples
Tue Nov 06 15:18:05 +0900 2007 - received 11 / 20, 5797 tuples
Tue Nov 06 15:18:12 +0900 2007 - received 8 / 20, 5802 tuples
Tue Nov 06 15:18:16 +0900 2007 - received 9 / 20, 5807 tuples
Tue Nov 06 15:18:21 +0900 2007 - received 8 / 20, 5809 tuples
Tue Nov 06 15:18:25 +0900 2007 - received 12 / 20, 5810 tuples
Tue Nov 06 15:18:30 +0900 2007 - received 10 / 20, 5812 tuples
Tue Nov 06 15:18:35 +0900 2007 - received 13 / 20, 5817 tuples
Tue Nov 06 15:18:40 +0900 2007 - received 3 / 20, 5811 tuples
Tue Nov 06 15:18:45 +0900 2007 - received 5 / 20, 5811 tuples
Tue Nov 06 15:18:50 +0900 2007 - received 15 / 20, 5820 tuples
Tue Nov 06 15:18:55 +0900 2007 - received 14 / 20, 5826 tuples
Tue Nov 06 15:19:01 +0900 2007 - received 3 / 20, 5823 tuples
Tue Nov 06 15:19:08 +0900 2007 - received 8 / 20, 5814 tuples
Tue Nov 06 15:19:12 +0900 2007 - received 8 / 20, 5822 tuples
Tue Nov 06 15:19:18 +0900 2007 - received 10 / 20, 5818 tuples

i j

i, j
j
Ci,j = P (tk−1 |tk )P (tk+1 |tk )
k=i

Ci,j < 0.75
i..j

$count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM statuses WHERE #{IGNORE_COND} AND language = ? AND (created_at BETWEEN ? AND ?) AND text @@ ?quot;, language, t.ago(ago), t, add_pragma(word)]$

2007-11-06 13:19:45 ANALYZER-ng(22499) begin for japanese-utf8
2007-11-06 13:19:46 ANALYZER-ng(22499) extracted 3120 sentences
2007-11-06 13:20:12 ANALYZER-ng(22499) 6006 keywords extracted from 3120 sentences
2007-11-06 13:20:12 ANALYZER-ng(22499) deleting stopwords ...
2007-11-06 13:20:19 ANALYZER-ng(22499) odd terms removed (5902 terms)
2007-11-06 13:20:19 ANALYZER-ng(22499) ignore case (5895 terms)
2007-11-06 13:20:19 ANALYZER-ng(22499) trivial terms are removed (1796 terms)
2007-11-06 13:21:38 ANALYZER-ng(22499) occurrence calculated (72.738133 s)
2007-11-06 13:23:35 ANALYZER-ng(22499) modified DDFs calculated
2007-11-06 13:23:35 ANALYZER-ng(22499) scores calculated (1563 terms)
2007-11-06 13:23:40 ANALYZER-ng(22499) redundant terms removed (1151 terms)
2007-11-06 13:23:42 ANALYZER-ng(22499) end for japanese-utf8 (237.531316 s)

2007-11-06 13:23:42 ANALYZER-ng(22499) begin for english
2007-11-06 13:23:43 ANALYZER-ng(22499) extracted 6181 sentences
2007-11-06 13:24:20 ANALYZER-ng(22499) 10168 keywords extracted from 6181 sentences
2007-11-06 13:24:20 ANALYZER-ng(22499) deleting stopwords ...
2007-11-06 13:24:33 ANALYZER-ng(22499) odd terms removed (9808 terms)
2007-11-06 13:24:33 ANALYZER-ng(22499) ignore case (9444 terms)
2007-11-06 13:24:33 ANALYZER-ng(22499) trivial terms are removed (2738 terms)
2007-11-06 13:26:18 ANALYZER-ng(22499) occurrence calculated (96.306258 s)
2007-11-06 13:27:59 ANALYZER-ng(22499) modified DDFs calculated
2007-11-06 13:27:59 ANALYZER-ng(22499) scores calculated (2109 terms)
2007-11-06 13:28:10 ANALYZER-ng(22499) redundant terms removed (1643 terms)
2007-11-06 13:28:13 ANALYZER-ng(22499) end for english (270.044345 s)

Recently uploaded

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxnull - The Open Security Community

Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

SIEMENS: RAPUNZEL – A Tale About Knowledge GraphNeo4j

Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC

Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm

"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays

Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

Understanding the Laravel MVC ArchitecturePixlogix Infotech

AI as an Interface for Commercial BuildingsMemoori

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren

Install Stable Diffusion in windows machinePadma Pradeep

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106

DMCC Future of Trade Web3 - Special EditionDubai Multi Commodity Centre

The transition to renewables in India.pdfCompetition Advisory Services (India) LLP

Recently uploaded (20)

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx

Designing IA for AI - Information Architecture Conference 2024

Scanning the Internet for External Cloud Exposures via SSL Certs

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

SIEMENS: RAPUNZEL – A Tale About Knowledge Graph

Breaking the Kubernetes Kill Chain: Host Path Mount

Streamlining Python Development: A Guide to a Modern Project Setup

"Federated learning: out of reach no matter how close",Oleksandr Lapshyn

Unblocking The Main Thread Solving ANRs and Frozen Frames

Human Factors of XR: Using Human Factors to Design XR Systems

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Understanding the Laravel MVC Architecture

AI as an Interface for Commercial Buildings

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...

Injustice - Developers Among Us (SciFiDevCon 2024)

SQL Database Design For Developers at php[tek] 2024

Install Stable Diffusion in windows machine

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics

DMCC Future of Trade Web3 - Special Edition

The transition to renewables in India.pdf

Buzztterの裏側とその周辺技術

8. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref

10.

11.

12. >> t = Time.parse(quot;2007-11-3quot;) => Sat Nov 03 00:00:00 +0900 2007 >> Status.count(:conditions=>[quot;created_at BETWEEN ? AND ?quot;, t, t.tomorrow]) => 125626

13.

14.

15.

16.

17. Tue Nov 06 15:17:40 +0900 2007 - received 8 / 20, 5793 tuples Tue Nov 06 15:17:45 +0900 2007 - received 10 / 20, 5794 tuples Tue Nov 06 15:17:51 +0900 2007 - received 10 / 20, 5798 tuples Tue Nov 06 15:17:55 +0900 2007 - received 4 / 20, 5797 tuples Tue Nov 06 15:18:00 +0900 2007 - received 5 / 20, 5797 tuples Tue Nov 06 15:18:05 +0900 2007 - received 11 / 20, 5797 tuples Tue Nov 06 15:18:12 +0900 2007 - received 8 / 20, 5802 tuples Tue Nov 06 15:18:16 +0900 2007 - received 9 / 20, 5807 tuples Tue Nov 06 15:18:21 +0900 2007 - received 8 / 20, 5809 tuples Tue Nov 06 15:18:25 +0900 2007 - received 12 / 20, 5810 tuples Tue Nov 06 15:18:30 +0900 2007 - received 10 / 20, 5812 tuples Tue Nov 06 15:18:35 +0900 2007 - received 13 / 20, 5817 tuples Tue Nov 06 15:18:40 +0900 2007 - received 3 / 20, 5811 tuples Tue Nov 06 15:18:45 +0900 2007 - received 5 / 20, 5811 tuples Tue Nov 06 15:18:50 +0900 2007 - received 15 / 20, 5820 tuples Tue Nov 06 15:18:55 +0900 2007 - received 14 / 20, 5826 tuples Tue Nov 06 15:19:01 +0900 2007 - received 3 / 20, 5823 tuples Tue Nov 06 15:19:08 +0900 2007 - received 8 / 20, 5814 tuples Tue Nov 06 15:19:12 +0900 2007 - received 8 / 20, 5822 tuples Tue Nov 06 15:19:18 +0900 2007 - received 10 / 20, 5818 tuples

18.

19.

20. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref

21. k

22.

23.

24. i j i, j j Ci,j = P (tk−1 |tk )P (tk+1 |tk ) k=i Ci,j < 0.75 i..j

25.

26.

27. count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM statuses WHERE #{IGNORE_COND} AND language = ? AND (created_at BETWEEN ? AND ?) AND text @@ ?quot;, language, t.ago(ago), t, add_pragma(word)]

28. 2007-11-06 13:19:45 ANALYZER-ng(22499) begin for japanese-utf8 2007-11-06 13:19:46 ANALYZER-ng(22499) extracted 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) 6006 keywords extracted from 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:20:19 ANALYZER-ng(22499) odd terms removed (5902 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) ignore case (5895 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) trivial terms are removed (1796 terms) 2007-11-06 13:21:38 ANALYZER-ng(22499) occurrence calculated (72.738133 s) 2007-11-06 13:23:35 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:23:35 ANALYZER-ng(22499) scores calculated (1563 terms) 2007-11-06 13:23:40 ANALYZER-ng(22499) redundant terms removed (1151 terms) 2007-11-06 13:23:42 ANALYZER-ng(22499) end for japanese-utf8 (237.531316 s) 2007-11-06 13:23:42 ANALYZER-ng(22499) begin for english 2007-11-06 13:23:43 ANALYZER-ng(22499) extracted 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) 10168 keywords extracted from 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:24:33 ANALYZER-ng(22499) odd terms removed (9808 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) ignore case (9444 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) trivial terms are removed (2738 terms) 2007-11-06 13:26:18 ANALYZER-ng(22499) occurrence calculated (96.306258 s) 2007-11-06 13:27:59 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:27:59 ANALYZER-ng(22499) scores calculated (2109 terms) 2007-11-06 13:28:10 ANALYZER-ng(22499) redundant terms removed (1643 terms) 2007-11-06 13:28:13 ANALYZER-ng(22499) end for english (270.044345 s)

Buzztterの裏側とその周辺技術

Recommended

Recommended

More Related Content

More from Yoji Shidara

More from Yoji Shidara (12)

Recently uploaded

Recently uploaded (20)

Buzztterの裏側とその周辺技術