CPANTS: Kwalitative website and its tools

CPANTSKwalitative website and its tools Kenichi Ishigaki (charsbar) @YAPC::EU 2012 August 22, 2012

Kenichi Ishigaki (charsbar)From Shibuya.pm, Tokyo, Japan.

Freelancer - Perl programmer- Writer/Translator

Around 40 CPAN distributions

DBD::SQLite

Acme::CPANAuthors

We have been enjoying theCPANTS game since 2005.

輝け！全日本最強 CPAN Author 決定選手権 by Koichi Taniguchihttp://blog.livedoor.jp/nipotan/archives/16108466.html

He picked upJapanese authors by eye.

Our names are easy to find.

There were notso many authors. - Total: ~4000 - Japanese: ~50

YAPC::Asia increased the number of Japanese authors.

YAPC::Asia / Japanese authors 2006 (Mar) 98 2007 (Apr) 154 2008 (May) 191 2009 (Sep) 228 2010 (Oct) 255 2011 (Oct) 270

Neededsomething to pick up Japanese authors more easily.

Thats why Icreated a list ofJapanese authors and a script to maintain it.

Ive been reporting theJapanese top 10 authors since 2008.

Ive been adding something new every year.

2008: sum of the kwalitee scores per author

2009: authors who releasedmost in the year

2010: authors/population ratio

2011: launched awebsite (finally)acme.cpanauthors.org

It had one big problem.

No data.

The officialCPANTS site had been down for some time.

I needed to set up mine.

I created aprivate repositoryand put everything into it.

Merged recentcommits from domms repository.

Added a few columns.

TweakedCatalyst/DBIC stuff.

It worked.

Warnings were left.

I needed to find some tuits to remove them.

Perl QAHackathon

Warnings were removed.

Ported some ofthe changes I didlocally to daxims repository.

Showed a new acme.cpanauthors.orgfeaturing CPANTS info.

Unfortunately,the porting took too much time.

I didnt mergethe changes backto my repository.

OSDC.TW

I finally merged the changes.

Got severalreports thatCPANTS was broken.

What brokeCPANTS was a small change.

"modules" : [ { "file" : "lib/Path/Extended.pm", "in_basedir" : 0, "in_lib" : 1, "module" : "Path::Extended", "uses" : { "Sub::Install" : 1, "strict" : 1, "warnings" : 1 } }]

I dont think this change is bad.

Module::CPANTS:: ProcessCPAN shouldnt have died by this.

It should have had tests.

Is should have run faster.

It should havebeen easier to fix analysis.

Enough issues for a summer.

What should we do?

- We need tests.- we need to find test cases.- we need to do it many times.

Making it runfaster is thefirst priority.

I wrote abarebone scriptto store data in parallel.

JSONcreate table if not exists analysis ( id integer primary key autoincrement, path text unique, distv text, author text, json text, duration integer);

Raw SQLstatements

Parallel::ForkManager

SQLite queue

Beware a race conditionmy ($id) = $dbh->selectrow_array(" SELECT id FROM queue WHERE status = 0 LIMIT = 1");$dbh->do(" UPDATE queue SET status = 1 WHERE id = ?", undef, $id);

sqlite_update_hookmy $id;my $dbh->sqlite_update_hook(sub { (undef, undef, undef, $id) = @_;});

$dbh->do(" UPDATE queue SET status = 1, WHERE id IN ( SELECT id FROM queue WHERE status = 0 LIMIT 1 )");

Archive::Any::Lite

Archive::Any::Plugin::Bzip2

WorePAN- Bundling is bad- We need a specificversion- Derived from OrePAN

use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, files => [qw( I/IS/ISHIGAKI/WorePAN-0.01.tar.gz )], use_backpan => 1, no_network => 0, cleanup => 1,);

use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, files => [qw( I/IS/ISHIGAKI/WorePAN-0.01.tar.gz )], local_mirror => /home/ishigaki/minicpan/, no_network => 1, cleanup => 1,);

use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, dists => { Catalyst-Runtime => 5.9, DBIx-Class => 0, }, cleanup => 1,);

Bonus featuresmy $worepan = WorePAN->new( root => path/to/a/CPAN/mirror/, cleanup => 0,);my $authors = $worepan->authors;my $modules = $worepan->modules;my $file = $worepan->files;my $dists = $worepan->latest_distributions;

$worepan->add_files(qw{ /path/to/a/local/distribution-0.01.tar.gz});$worepan->update_indices;

Now we haveenough tools.

Processing time is significantly decreased.

Whats next?

::Site refactoring

Im preparing the data now.

Creating moredatabases/tables.

Merginginformation fromexternal sources.- CPAN indices- CPAN uploads database

Calculating scores on prerequisite modules.

It will be thisyears somethingnew in my annual report.

And then, Illmove on to fixing the metrics.

Some of them are badly broken."versions" : { "lib/Data/Phrasebook.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Debug.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Generic.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader/Base.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader/Text.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Plain.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/SQL.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/SQL/Query.pm" : "use vars qw($VERSION);¥n"},

Error is not a stash."error" : { "easily_repackageable" : "easily_repackageable_by_fedora", "easily_repackageable_by_fedora" : "fits_fedora_license", "metayml_conforms_spec_current" : [ "1.4", "Expected a map structure from data string or file. [Validation: 1.4]" ], "metayml_conforms_to_known_spec" : [ "1.0", "Expected a map structure from data string or file. [Validation: 1.0]" ], "no_pod_errors" : " home cpants tmp analyze 11442 8001be43fb65..."}

Should haveinitialize/finalize phases.Module::CPANTS::Kwalitee::Distros doesnt clean up after mirrored Debian CPANTS file https://rt.cpan.org/Ticket/Display.html?id=51514

There are much more to do.- JSON API for metacpan.org and so on.- Email Reporting like CPAN Testers- Evaluate new Kwalitee indicators- New metrics like portable filename- Blog about recent tendency- More comprehensive tests- Analysis per perl version/architecture- Cover Perl::Critic, CPAN::Critic::Module::Abstract- 35 RT tickets and several github isses

Resources github.com/charsbar/www-cpants github.com/charsbar/worepangithub.com/daxim/Module-CPANTS-Analyse

Questions?

Thank you

CPANTS: Kwalitative website and its tools

by charsbar on Aug 22, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics