CPANTS: Kwalitative website and its tools
Upcoming SlideShare
Loading in...5
×
 

CPANTS: Kwalitative website and its tools

on

  • 840 views

CPANTS talk at YAPC::EU 2012

CPANTS talk at YAPC::EU 2012

Statistics

Views

Total Views
840
Views on SlideShare
839
Embed Views
1

Actions

Likes
0
Downloads
1
Comments
0

1 Embed 1

https://si0.twimg.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

CPANTS: Kwalitative website and its tools CPANTS: Kwalitative website and its tools Presentation Transcript

  • CPANTSKwalitative website and its tools Kenichi Ishigaki (charsbar) @YAPC::EU 2012 August 22, 2012
  • Kenichi Ishigaki (charsbar)From Shibuya.pm, Tokyo, Japan.
  • Freelancer - Perl programmer- Writer/Translator
  • Around 40 CPAN distributions
  • DBD::SQLite
  • Acme::CPANAuthors
  • We have been enjoying theCPANTS game since 2005.
  • 輝け!全日本最強 CPAN Author 決定選手権 by Koichi Taniguchihttp://blog.livedoor.jp/nipotan/archives/16108466.html
  • He picked upJapanese authors by eye.
  • Our names are easy to find.
  • There were notso many authors. - Total: ~4000 - Japanese: ~50
  • YAPC::Asia increased the number of Japanese authors.
  • YAPC::Asia / Japanese authors 2006 (Mar) 98 2007 (Apr) 154 2008 (May) 191 2009 (Sep) 228 2010 (Oct) 255 2011 (Oct) 270
  • Neededsomething to pick up Japanese authors more easily.
  • Thats why Icreated a list ofJapanese authors and a script to maintain it.
  • Ive been reporting theJapanese top 10 authors since 2008.
  • Ive been adding something new every year.
  • 2008: sum of the kwalitee scores per author
  • 2009: authors who releasedmost in the year
  • 2010: authors/population ratio
  • 2011: launched awebsite (finally)acme.cpanauthors.org
  • It had one big problem.
  • No data.
  • The officialCPANTS site had been down for some time.
  • I needed to set up mine.
  • I created aprivate repositoryand put everything into it.
  • Merged recentcommits from domms repository.
  • Added a few columns.
  • TweakedCatalyst/DBIC stuff.
  • It worked.
  • Warnings were left.
  • I needed to find some tuits to remove them.
  • Perl QAHackathon
  • Warnings were removed.
  • Ported some ofthe changes I didlocally to daxims repository.
  • Showed a new acme.cpanauthors.orgfeaturing CPANTS info.
  • Unfortunately,the porting took too much time.
  • I didnt mergethe changes backto my repository.
  • OSDC.TW
  • I finally merged the changes.
  • Got severalreports thatCPANTS was broken.
  • What brokeCPANTS was a small change.
  • "modules" : [ { "file" : "lib/Path/Extended.pm", "in_basedir" : 0, "in_lib" : 1, "module" : "Path::Extended", "uses" : { "Sub::Install" : 1, "strict" : 1, "warnings" : 1 } }]
  • I dont think this change is bad.
  • Module::CPANTS:: ProcessCPAN shouldnt have died by this.
  • It should have had tests.
  • Is should have run faster.
  • It should havebeen easier to fix analysis.
  • Enough issues for a summer.
  • What should we do?
  • - We need tests.- we need to find test cases.- we need to do it many times.
  • Making it runfaster is thefirst priority.
  • I wrote abarebone scriptto store data in parallel.
  • JSONcreate table if not exists analysis ( id integer primary key autoincrement, path text unique, distv text, author text, json text, duration integer);
  • Raw SQLstatements
  • Parallel::ForkManager
  • SQLite queue
  • Beware a race conditionmy ($id) = $dbh->selectrow_array(" SELECT id FROM queue WHERE status = 0 LIMIT = 1");$dbh->do(" UPDATE queue SET status = 1 WHERE id = ?", undef, $id);
  • sqlite_update_hookmy $id;my $dbh->sqlite_update_hook(sub { (undef, undef, undef, $id) = @_;});
  • $dbh->do(" UPDATE queue SET status = 1, WHERE id IN ( SELECT id FROM queue WHERE status = 0 LIMIT 1 )");
  • Archive::Any::Lite
  • Archive::Any::Plugin::Bzip2
  • WorePAN- Bundling is bad- We need a specificversion- Derived from OrePAN
  • use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, files => [qw( I/IS/ISHIGAKI/WorePAN-0.01.tar.gz )], use_backpan => 1, no_network => 0, cleanup => 1,);
  • use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, files => [qw( I/IS/ISHIGAKI/WorePAN-0.01.tar.gz )], local_mirror => /home/ishigaki/minicpan/, no_network => 1, cleanup => 1,);
  • use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, dists => { Catalyst-Runtime => 5.9, DBIx-Class => 0, }, cleanup => 1,);
  • Bonus featuresmy $worepan = WorePAN->new( root => path/to/a/CPAN/mirror/, cleanup => 0,);my $authors = $worepan->authors;my $modules = $worepan->modules;my $file = $worepan->files;my $dists = $worepan->latest_distributions;
  • $worepan->add_files(qw{ /path/to/a/local/distribution-0.01.tar.gz});$worepan->update_indices;
  • Now we haveenough tools.
  • Processing time is significantly decreased.
  • Whats next?
  • ::Site refactoring
  • Im preparing the data now.
  • Creating moredatabases/tables.
  • Merginginformation fromexternal sources.- CPAN indices- CPAN uploads database
  • Calculating scores on prerequisite modules.
  • It will be thisyears somethingnew in my annual report.
  • And then, Illmove on to fixing the metrics.
  • Some of them are badly broken."versions" : { "lib/Data/Phrasebook.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Debug.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Generic.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader/Base.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader/Text.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Plain.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/SQL.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/SQL/Query.pm" : "use vars qw($VERSION);¥n"},
  • Error is not a stash."error" : { "easily_repackageable" : "easily_repackageable_by_fedora", "easily_repackageable_by_fedora" : "fits_fedora_license", "metayml_conforms_spec_current" : [ "1.4", "Expected a map structure from data string or file. [Validation: 1.4]" ], "metayml_conforms_to_known_spec" : [ "1.0", "Expected a map structure from data string or file. [Validation: 1.0]" ], "no_pod_errors" : " home cpants tmp analyze 11442 8001be43fb65..."}
  • Should haveinitialize/finalize phases.Module::CPANTS::Kwalitee::Distros doesnt clean up after mirrored Debian CPANTS file https://rt.cpan.org/Ticket/Display.html?id=51514
  • There are much more to do.- JSON API for metacpan.org and so on.- Email Reporting like CPAN Testers- Evaluate new Kwalitee indicators- New metrics like portable filename- Blog about recent tendency- More comprehensive tests- Analysis per perl version/architecture- Cover Perl::Critic, CPAN::Critic::Module::Abstract- 35 RT tickets and several github isses
  • Resources github.com/charsbar/www-cpants github.com/charsbar/worepangithub.com/daxim/Module-CPANTS-Analyse
  • Questions?
  • Thank you