• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
CPANTS: Kwalitative website and its tools
 

CPANTS: Kwalitative website and its tools

on

  • 797 views

CPANTS talk at YAPC::EU 2012

CPANTS talk at YAPC::EU 2012

Statistics

Views

Total Views
797
Views on SlideShare
796
Embed Views
1

Actions

Likes
0
Downloads
1
Comments
0

1 Embed 1

https://si0.twimg.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    CPANTS: Kwalitative website and its tools CPANTS: Kwalitative website and its tools Presentation Transcript

    • CPANTSKwalitative website and its tools Kenichi Ishigaki (charsbar) @YAPC::EU 2012 August 22, 2012
    • Kenichi Ishigaki (charsbar)From Shibuya.pm, Tokyo, Japan.
    • Freelancer - Perl programmer- Writer/Translator
    • Around 40 CPAN distributions
    • DBD::SQLite
    • Acme::CPANAuthors
    • We have been enjoying theCPANTS game since 2005.
    • 輝け!全日本最強 CPAN Author 決定選手権 by Koichi Taniguchihttp://blog.livedoor.jp/nipotan/archives/16108466.html
    • He picked upJapanese authors by eye.
    • Our names are easy to find.
    • There were notso many authors. - Total: ~4000 - Japanese: ~50
    • YAPC::Asia increased the number of Japanese authors.
    • YAPC::Asia / Japanese authors 2006 (Mar) 98 2007 (Apr) 154 2008 (May) 191 2009 (Sep) 228 2010 (Oct) 255 2011 (Oct) 270
    • Neededsomething to pick up Japanese authors more easily.
    • Thats why Icreated a list ofJapanese authors and a script to maintain it.
    • Ive been reporting theJapanese top 10 authors since 2008.
    • Ive been adding something new every year.
    • 2008: sum of the kwalitee scores per author
    • 2009: authors who releasedmost in the year
    • 2010: authors/population ratio
    • 2011: launched awebsite (finally)acme.cpanauthors.org
    • It had one big problem.
    • No data.
    • The officialCPANTS site had been down for some time.
    • I needed to set up mine.
    • I created aprivate repositoryand put everything into it.
    • Merged recentcommits from domms repository.
    • Added a few columns.
    • TweakedCatalyst/DBIC stuff.
    • It worked.
    • Warnings were left.
    • I needed to find some tuits to remove them.
    • Perl QAHackathon
    • Warnings were removed.
    • Ported some ofthe changes I didlocally to daxims repository.
    • Showed a new acme.cpanauthors.orgfeaturing CPANTS info.
    • Unfortunately,the porting took too much time.
    • I didnt mergethe changes backto my repository.
    • OSDC.TW
    • I finally merged the changes.
    • Got severalreports thatCPANTS was broken.
    • What brokeCPANTS was a small change.
    • "modules" : [ { "file" : "lib/Path/Extended.pm", "in_basedir" : 0, "in_lib" : 1, "module" : "Path::Extended", "uses" : { "Sub::Install" : 1, "strict" : 1, "warnings" : 1 } }]
    • I dont think this change is bad.
    • Module::CPANTS:: ProcessCPAN shouldnt have died by this.
    • It should have had tests.
    • Is should have run faster.
    • It should havebeen easier to fix analysis.
    • Enough issues for a summer.
    • What should we do?
    • - We need tests.- we need to find test cases.- we need to do it many times.
    • Making it runfaster is thefirst priority.
    • I wrote abarebone scriptto store data in parallel.
    • JSONcreate table if not exists analysis ( id integer primary key autoincrement, path text unique, distv text, author text, json text, duration integer);
    • Raw SQLstatements
    • Parallel::ForkManager
    • SQLite queue
    • Beware a race conditionmy ($id) = $dbh->selectrow_array(" SELECT id FROM queue WHERE status = 0 LIMIT = 1");$dbh->do(" UPDATE queue SET status = 1 WHERE id = ?", undef, $id);
    • sqlite_update_hookmy $id;my $dbh->sqlite_update_hook(sub { (undef, undef, undef, $id) = @_;});
    • $dbh->do(" UPDATE queue SET status = 1, WHERE id IN ( SELECT id FROM queue WHERE status = 0 LIMIT 1 )");
    • Archive::Any::Lite
    • Archive::Any::Plugin::Bzip2
    • WorePAN- Bundling is bad- We need a specificversion- Derived from OrePAN
    • use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, files => [qw( I/IS/ISHIGAKI/WorePAN-0.01.tar.gz )], use_backpan => 1, no_network => 0, cleanup => 1,);
    • use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, files => [qw( I/IS/ISHIGAKI/WorePAN-0.01.tar.gz )], local_mirror => /home/ishigaki/minicpan/, no_network => 1, cleanup => 1,);
    • use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, dists => { Catalyst-Runtime => 5.9, DBIx-Class => 0, }, cleanup => 1,);
    • Bonus featuresmy $worepan = WorePAN->new( root => path/to/a/CPAN/mirror/, cleanup => 0,);my $authors = $worepan->authors;my $modules = $worepan->modules;my $file = $worepan->files;my $dists = $worepan->latest_distributions;
    • $worepan->add_files(qw{ /path/to/a/local/distribution-0.01.tar.gz});$worepan->update_indices;
    • Now we haveenough tools.
    • Processing time is significantly decreased.
    • Whats next?
    • ::Site refactoring
    • Im preparing the data now.
    • Creating moredatabases/tables.
    • Merginginformation fromexternal sources.- CPAN indices- CPAN uploads database
    • Calculating scores on prerequisite modules.
    • It will be thisyears somethingnew in my annual report.
    • And then, Illmove on to fixing the metrics.
    • Some of them are badly broken."versions" : { "lib/Data/Phrasebook.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Debug.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Generic.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader/Base.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader/Text.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Plain.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/SQL.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/SQL/Query.pm" : "use vars qw($VERSION);¥n"},
    • Error is not a stash."error" : { "easily_repackageable" : "easily_repackageable_by_fedora", "easily_repackageable_by_fedora" : "fits_fedora_license", "metayml_conforms_spec_current" : [ "1.4", "Expected a map structure from data string or file. [Validation: 1.4]" ], "metayml_conforms_to_known_spec" : [ "1.0", "Expected a map structure from data string or file. [Validation: 1.0]" ], "no_pod_errors" : " home cpants tmp analyze 11442 8001be43fb65..."}
    • Should haveinitialize/finalize phases.Module::CPANTS::Kwalitee::Distros doesnt clean up after mirrored Debian CPANTS file https://rt.cpan.org/Ticket/Display.html?id=51514
    • There are much more to do.- JSON API for metacpan.org and so on.- Email Reporting like CPAN Testers- Evaluate new Kwalitee indicators- New metrics like portable filename- Blog about recent tendency- More comprehensive tests- Analysis per perl version/architecture- Cover Perl::Critic, CPAN::Critic::Module::Abstract- 35 RT tickets and several github isses
    • Resources github.com/charsbar/www-cpants github.com/charsbar/worepangithub.com/daxim/Module-CPANTS-Analyse
    • Questions?
    • Thank you