Buzztterの裏側とその周辺技術

Yoji Shidara
Yoji ShidaraCTO at Enishi Tech Inc.
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
w
w       TFtgt   DFtgt    TFref   DFref


        w               TFtgt
    DFtgt

        w               TFref
    DFref
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
>> t = Time.parse(quot;2007-11-3quot;)
=> Sat Nov 03 00:00:00 +0900 2007

>> Status.count(:conditions=>[quot;created_at
BETWEEN ? AND ?quot;, t, t.tomorrow])
=> 125626
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Tue   Nov   06   15:17:40   +0900   2007   -   received    8   /   20,   5793   tuples
Tue   Nov   06   15:17:45   +0900   2007   -   received   10   /   20,   5794   tuples
Tue   Nov   06   15:17:51   +0900   2007   -   received   10   /   20,   5798   tuples
Tue   Nov   06   15:17:55   +0900   2007   -   received    4   /   20,   5797   tuples
Tue   Nov   06   15:18:00   +0900   2007   -   received    5   /   20,   5797   tuples
Tue   Nov   06   15:18:05   +0900   2007   -   received   11   /   20,   5797   tuples
Tue   Nov   06   15:18:12   +0900   2007   -   received    8   /   20,   5802   tuples
Tue   Nov   06   15:18:16   +0900   2007   -   received    9   /   20,   5807   tuples
Tue   Nov   06   15:18:21   +0900   2007   -   received    8   /   20,   5809   tuples
Tue   Nov   06   15:18:25   +0900   2007   -   received   12   /   20,   5810   tuples
Tue   Nov   06   15:18:30   +0900   2007   -   received   10   /   20,   5812   tuples
Tue   Nov   06   15:18:35   +0900   2007   -   received   13   /   20,   5817   tuples
Tue   Nov   06   15:18:40   +0900   2007   -   received    3   /   20,   5811   tuples
Tue   Nov   06   15:18:45   +0900   2007   -   received    5   /   20,   5811   tuples
Tue   Nov   06   15:18:50   +0900   2007   -   received   15   /   20,   5820   tuples
Tue   Nov   06   15:18:55   +0900   2007   -   received   14   /   20,   5826   tuples
Tue   Nov   06   15:19:01   +0900   2007   -   received    3   /   20,   5823   tuples
Tue   Nov   06   15:19:08   +0900   2007   -   received    8   /   20,   5814   tuples
Tue   Nov   06   15:19:12   +0900   2007   -   received    8   /   20,   5822   tuples
Tue   Nov   06   15:19:18   +0900   2007   -   received   10   /   20,   5818   tuples
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
w
w       TFtgt   DFtgt    TFref   DFref


        w               TFtgt
    DFtgt

        w               TFref
    DFref
k
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
i                           j


i, j
                 j
       Ci,j =         P (tk−1 |tk )P (tk+1 |tk )
                k=i

Ci,j < 0.75
                                                   i..j
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM
statuses WHERE #{IGNORE_COND} AND language = ? AND
(created_at BETWEEN ? AND ?) AND text @@ ?quot;,
language, t.ago(ago), t, add_pragma(word)]
2007-11-06   13:19:45   ANALYZER-ng(22499)   begin for japanese-utf8
2007-11-06   13:19:46   ANALYZER-ng(22499)   extracted 3120 sentences
2007-11-06   13:20:12   ANALYZER-ng(22499)   6006 keywords extracted from 3120 sentences
2007-11-06   13:20:12   ANALYZER-ng(22499)   deleting stopwords ...
2007-11-06   13:20:19   ANALYZER-ng(22499)   odd terms removed (5902 terms)
2007-11-06   13:20:19   ANALYZER-ng(22499)   ignore case (5895 terms)
2007-11-06   13:20:19   ANALYZER-ng(22499)   trivial terms are removed (1796 terms)
2007-11-06   13:21:38   ANALYZER-ng(22499)   occurrence calculated (72.738133 s)
2007-11-06   13:23:35   ANALYZER-ng(22499)   modified DDFs calculated
2007-11-06   13:23:35   ANALYZER-ng(22499)   scores calculated (1563 terms)
2007-11-06   13:23:40   ANALYZER-ng(22499)   redundant terms removed (1151 terms)
2007-11-06   13:23:42   ANALYZER-ng(22499)   end for japanese-utf8 (237.531316 s)

2007-11-06   13:23:42   ANALYZER-ng(22499)   begin for english
2007-11-06   13:23:43   ANALYZER-ng(22499)   extracted 6181 sentences
2007-11-06   13:24:20   ANALYZER-ng(22499)   10168 keywords extracted from 6181 sentences
2007-11-06   13:24:20   ANALYZER-ng(22499)   deleting stopwords ...
2007-11-06   13:24:33   ANALYZER-ng(22499)   odd terms removed (9808 terms)
2007-11-06   13:24:33   ANALYZER-ng(22499)   ignore case (9444 terms)
2007-11-06   13:24:33   ANALYZER-ng(22499)   trivial terms are removed (2738 terms)
2007-11-06   13:26:18   ANALYZER-ng(22499)   occurrence calculated (96.306258 s)
2007-11-06   13:27:59   ANALYZER-ng(22499)   modified DDFs calculated
2007-11-06   13:27:59   ANALYZER-ng(22499)   scores calculated (2109 terms)
2007-11-06   13:28:10   ANALYZER-ng(22499)   redundant terms removed (1643 terms)
2007-11-06   13:28:13   ANALYZER-ng(22499)   end for english (270.044345 s)
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
Buzztterの裏側とその周辺技術
1 of 70

Recommended

Åbning Phdskole Mette Thunø by
Åbning Phdskole Mette ThunøÅbning Phdskole Mette Thunø
Åbning Phdskole Mette Thunøhcseidelin.hum.ku.dk
845 views8 slides
Twitter4Rでつくるゆるふわ愛されTwitter bot by
Twitter4Rでつくるゆるふわ愛されTwitter botTwitter4Rでつくるゆるふわ愛されTwitter bot
Twitter4Rでつくるゆるふわ愛されTwitter botYoji Shidara
1.6K views16 slides
Sinatraで鼻歌混じりのWeb開発 @OSC2009-Do by
Sinatraで鼻歌混じりのWeb開発 @OSC2009-DoSinatraで鼻歌混じりのWeb開発 @OSC2009-Do
Sinatraで鼻歌混じりのWeb開発 @OSC2009-DoYoji Shidara
1.7K views78 slides
Den Gode Phdansøgning Peter Stray Jørgensen by
Den Gode Phdansøgning Peter Stray JørgensenDen Gode Phdansøgning Peter Stray Jørgensen
Den Gode Phdansøgning Peter Stray Jørgensenhcseidelin.hum.ku.dk
3.6K views9 slides
Erhvervsphd Morten Bovbjerg by
Erhvervsphd Morten BovbjergErhvervsphd Morten Bovbjerg
Erhvervsphd Morten Bovbjerghcseidelin.hum.ku.dk
1K views15 slides
Sinatraで鼻歌まじりのWeb開発 by
Sinatraで鼻歌まじりのWeb開発Sinatraで鼻歌まじりのWeb開発
Sinatraで鼻歌まじりのWeb開発Yoji Shidara
10.8K views73 slides

More Related Content

More from Yoji Shidara

絵文字Ruby: From Sapporo.rb with Love for Emoji. by
絵文字Ruby: From Sapporo.rb with Love for Emoji.絵文字Ruby: From Sapporo.rb with Love for Emoji.
絵文字Ruby: From Sapporo.rb with Love for Emoji.Yoji Shidara
1.8K views34 slides
Jpmobile: Who I Wanna Be And Who I Am by
Jpmobile: Who I Wanna Be And Who I AmJpmobile: Who I Wanna Be And Who I Am
Jpmobile: Who I Wanna Be And Who I AmYoji Shidara
1.7K views42 slides
Building Static Website With Github And Jekyll by
Building Static Website With Github And JekyllBuilding Static Website With Github And Jekyll
Building Static Website With Github And JekyllYoji Shidara
1.8K views55 slides
From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets... by
From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets...From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets...
From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets...Yoji Shidara
10.2K views70 slides
The Way We Are Working On Our Website @とちぎRuby会議02 by
The Way We Are Working On Our Website @とちぎRuby会議02The Way We Are Working On Our Website @とちぎRuby会議02
The Way We Are Working On Our Website @とちぎRuby会議02Yoji Shidara
1.7K views48 slides
SAPICAの利用履歴を可視化する by
SAPICAの利用履歴を可視化するSAPICAの利用履歴を可視化する
SAPICAの利用履歴を可視化するYoji Shidara
2.2K views13 slides

More from Yoji Shidara(12)

絵文字Ruby: From Sapporo.rb with Love for Emoji. by Yoji Shidara
絵文字Ruby: From Sapporo.rb with Love for Emoji.絵文字Ruby: From Sapporo.rb with Love for Emoji.
絵文字Ruby: From Sapporo.rb with Love for Emoji.
Yoji Shidara1.8K views
Jpmobile: Who I Wanna Be And Who I Am by Yoji Shidara
Jpmobile: Who I Wanna Be And Who I AmJpmobile: Who I Wanna Be And Who I Am
Jpmobile: Who I Wanna Be And Who I Am
Yoji Shidara1.7K views
Building Static Website With Github And Jekyll by Yoji Shidara
Building Static Website With Github And JekyllBuilding Static Website With Github And Jekyll
Building Static Website With Github And Jekyll
Yoji Shidara1.8K views
From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets... by Yoji Shidara
From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets...From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets...
From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets...
Yoji Shidara10.2K views
The Way We Are Working On Our Website @とちぎRuby会議02 by Yoji Shidara
The Way We Are Working On Our Website @とちぎRuby会議02The Way We Are Working On Our Website @とちぎRuby会議02
The Way We Are Working On Our Website @とちぎRuby会議02
Yoji Shidara1.7K views
SAPICAの利用履歴を可視化する by Yoji Shidara
SAPICAの利用履歴を可視化するSAPICAの利用履歴を可視化する
SAPICAの利用履歴を可視化する
Yoji Shidara2.2K views
Ruby on Rails でつくるアタシ好みの愛され Web サービス by Yoji Shidara
Ruby on Rails でつくるアタシ好みの愛され Web サービスRuby on Rails でつくるアタシ好みの愛され Web サービス
Ruby on Rails でつくるアタシ好みの愛され Web サービス
Yoji Shidara11.7K views
RubyKaigi2008弾丸レポート / ガラパゴスに線路を敷こう by Yoji Shidara
RubyKaigi2008弾丸レポート / ガラパゴスに線路を敷こうRubyKaigi2008弾丸レポート / ガラパゴスに線路を敷こう
RubyKaigi2008弾丸レポート / ガラパゴスに線路を敷こう
Yoji Shidara1.4K views
ガラパゴスに線路を敷こう: 携帯電話用RailsプラグインJpmobile by Yoji Shidara
ガラパゴスに線路を敷こう: 携帯電話用RailsプラグインJpmobileガラパゴスに線路を敷こう: 携帯電話用RailsプラグインJpmobile
ガラパゴスに線路を敷こう: 携帯電話用RailsプラグインJpmobile
Yoji Shidara2.1K views
Twitter分散クロールの野望 by Yoji Shidara
Twitter分散クロールの野望Twitter分散クロールの野望
Twitter分散クロールの野望
Yoji Shidara2.4K views
Pluginが広げるRailsの魅力 by Yoji Shidara
Pluginが広げるRailsの魅力Pluginが広げるRailsの魅力
Pluginが広げるRailsの魅力
Yoji Shidara2.1K views
Rubyistからみたsoupcurry.info by Yoji Shidara
Rubyistからみたsoupcurry.infoRubyistからみたsoupcurry.info
Rubyistからみたsoupcurry.info
Yoji Shidara1.4K views

Recently uploaded

Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ... by
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...ShapeBlue
79 views17 slides
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti... by
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...ShapeBlue
98 views29 slides
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O... by
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...ShapeBlue
88 views13 slides
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ... by
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...ShapeBlue
123 views28 slides
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha... by
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...ShapeBlue
138 views18 slides
Why and How CloudStack at weSystems - Stephan Bienek - weSystems by
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsWhy and How CloudStack at weSystems - Stephan Bienek - weSystems
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsShapeBlue
197 views13 slides

Recently uploaded(20)

Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ... by ShapeBlue
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
ShapeBlue79 views
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti... by ShapeBlue
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
ShapeBlue98 views
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O... by ShapeBlue
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
ShapeBlue88 views
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ... by ShapeBlue
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
ShapeBlue123 views
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha... by ShapeBlue
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
ShapeBlue138 views
Why and How CloudStack at weSystems - Stephan Bienek - weSystems by ShapeBlue
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsWhy and How CloudStack at weSystems - Stephan Bienek - weSystems
Why and How CloudStack at weSystems - Stephan Bienek - weSystems
ShapeBlue197 views
The Power of Heat Decarbonisation Plans in the Built Environment by IES VE
The Power of Heat Decarbonisation Plans in the Built EnvironmentThe Power of Heat Decarbonisation Plans in the Built Environment
The Power of Heat Decarbonisation Plans in the Built Environment
IES VE69 views
Business Analyst Series 2023 - Week 4 Session 7 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 7Business Analyst Series 2023 -  Week 4 Session 7
Business Analyst Series 2023 - Week 4 Session 7
DianaGray10126 views
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ... by ShapeBlue
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
ShapeBlue144 views
Extending KVM Host HA for Non-NFS Storage - Alex Ivanov - StorPool by ShapeBlue
Extending KVM Host HA for Non-NFS Storage -  Alex Ivanov - StorPoolExtending KVM Host HA for Non-NFS Storage -  Alex Ivanov - StorPool
Extending KVM Host HA for Non-NFS Storage - Alex Ivanov - StorPool
ShapeBlue84 views
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue by ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlueWhat’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
ShapeBlue222 views
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by James Anderson
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
James Anderson156 views
State of the Union - Rohit Yadav - Apache CloudStack by ShapeBlue
State of the Union - Rohit Yadav - Apache CloudStackState of the Union - Rohit Yadav - Apache CloudStack
State of the Union - Rohit Yadav - Apache CloudStack
ShapeBlue253 views
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P... by ShapeBlue
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...
Developments to CloudStack’s SDN ecosystem: Integration with VMWare NSX 4 - P...
ShapeBlue154 views
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava... by ShapeBlue
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
ShapeBlue101 views
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue by ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueVNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
ShapeBlue163 views
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f... by TrustArc
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc160 views

Buzztterの裏側とその周辺技術

  • 8. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref
  • 12. >> t = Time.parse(quot;2007-11-3quot;) => Sat Nov 03 00:00:00 +0900 2007 >> Status.count(:conditions=>[quot;created_at BETWEEN ? AND ?quot;, t, t.tomorrow]) => 125626
  • 17. Tue Nov 06 15:17:40 +0900 2007 - received 8 / 20, 5793 tuples Tue Nov 06 15:17:45 +0900 2007 - received 10 / 20, 5794 tuples Tue Nov 06 15:17:51 +0900 2007 - received 10 / 20, 5798 tuples Tue Nov 06 15:17:55 +0900 2007 - received 4 / 20, 5797 tuples Tue Nov 06 15:18:00 +0900 2007 - received 5 / 20, 5797 tuples Tue Nov 06 15:18:05 +0900 2007 - received 11 / 20, 5797 tuples Tue Nov 06 15:18:12 +0900 2007 - received 8 / 20, 5802 tuples Tue Nov 06 15:18:16 +0900 2007 - received 9 / 20, 5807 tuples Tue Nov 06 15:18:21 +0900 2007 - received 8 / 20, 5809 tuples Tue Nov 06 15:18:25 +0900 2007 - received 12 / 20, 5810 tuples Tue Nov 06 15:18:30 +0900 2007 - received 10 / 20, 5812 tuples Tue Nov 06 15:18:35 +0900 2007 - received 13 / 20, 5817 tuples Tue Nov 06 15:18:40 +0900 2007 - received 3 / 20, 5811 tuples Tue Nov 06 15:18:45 +0900 2007 - received 5 / 20, 5811 tuples Tue Nov 06 15:18:50 +0900 2007 - received 15 / 20, 5820 tuples Tue Nov 06 15:18:55 +0900 2007 - received 14 / 20, 5826 tuples Tue Nov 06 15:19:01 +0900 2007 - received 3 / 20, 5823 tuples Tue Nov 06 15:19:08 +0900 2007 - received 8 / 20, 5814 tuples Tue Nov 06 15:19:12 +0900 2007 - received 8 / 20, 5822 tuples Tue Nov 06 15:19:18 +0900 2007 - received 10 / 20, 5818 tuples
  • 20. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref
  • 21. k
  • 24. i j i, j j Ci,j = P (tk−1 |tk )P (tk+1 |tk ) k=i Ci,j < 0.75 i..j
  • 27. count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM statuses WHERE #{IGNORE_COND} AND language = ? AND (created_at BETWEEN ? AND ?) AND text @@ ?quot;, language, t.ago(ago), t, add_pragma(word)]
  • 28. 2007-11-06 13:19:45 ANALYZER-ng(22499) begin for japanese-utf8 2007-11-06 13:19:46 ANALYZER-ng(22499) extracted 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) 6006 keywords extracted from 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:20:19 ANALYZER-ng(22499) odd terms removed (5902 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) ignore case (5895 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) trivial terms are removed (1796 terms) 2007-11-06 13:21:38 ANALYZER-ng(22499) occurrence calculated (72.738133 s) 2007-11-06 13:23:35 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:23:35 ANALYZER-ng(22499) scores calculated (1563 terms) 2007-11-06 13:23:40 ANALYZER-ng(22499) redundant terms removed (1151 terms) 2007-11-06 13:23:42 ANALYZER-ng(22499) end for japanese-utf8 (237.531316 s) 2007-11-06 13:23:42 ANALYZER-ng(22499) begin for english 2007-11-06 13:23:43 ANALYZER-ng(22499) extracted 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) 10168 keywords extracted from 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:24:33 ANALYZER-ng(22499) odd terms removed (9808 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) ignore case (9444 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) trivial terms are removed (2738 terms) 2007-11-06 13:26:18 ANALYZER-ng(22499) occurrence calculated (96.306258 s) 2007-11-06 13:27:59 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:27:59 ANALYZER-ng(22499) scores calculated (2109 terms) 2007-11-06 13:28:10 ANALYZER-ng(22499) redundant terms removed (1643 terms) 2007-11-06 13:28:13 ANALYZER-ng(22499) end for english (270.044345 s)