CPANTS: Kwalitative website and its tools
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

CPANTS: Kwalitative website and its tools

on

  • 876 views

CPANTS talk at YAPC::EU 2012

CPANTS talk at YAPC::EU 2012

Statistics

Views

Total Views
876
Views on SlideShare
875
Embed Views
1

Actions

Likes
0
Downloads
1
Comments
0

1 Embed 1

https://si0.twimg.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

CPANTS: Kwalitative website and its tools Presentation Transcript

  • 1. CPANTSKwalitative website and its tools Kenichi Ishigaki (charsbar) @YAPC::EU 2012 August 22, 2012
  • 2. Kenichi Ishigaki (charsbar)From Shibuya.pm, Tokyo, Japan.
  • 3. Freelancer - Perl programmer- Writer/Translator
  • 4. Around 40 CPAN distributions
  • 5. DBD::SQLite
  • 6. Acme::CPANAuthors
  • 7. We have been enjoying theCPANTS game since 2005.
  • 8. 輝け!全日本最強 CPAN Author 決定選手権 by Koichi Taniguchihttp://blog.livedoor.jp/nipotan/archives/16108466.html
  • 9. He picked upJapanese authors by eye.
  • 10. Our names are easy to find.
  • 11. There were notso many authors. - Total: ~4000 - Japanese: ~50
  • 12. YAPC::Asia increased the number of Japanese authors.
  • 13. YAPC::Asia / Japanese authors 2006 (Mar) 98 2007 (Apr) 154 2008 (May) 191 2009 (Sep) 228 2010 (Oct) 255 2011 (Oct) 270
  • 14. Neededsomething to pick up Japanese authors more easily.
  • 15. Thats why Icreated a list ofJapanese authors and a script to maintain it.
  • 16. Ive been reporting theJapanese top 10 authors since 2008.
  • 17. Ive been adding something new every year.
  • 18. 2008: sum of the kwalitee scores per author
  • 19. 2009: authors who releasedmost in the year
  • 20. 2010: authors/population ratio
  • 21. 2011: launched awebsite (finally)acme.cpanauthors.org
  • 22. It had one big problem.
  • 23. No data.
  • 24. The officialCPANTS site had been down for some time.
  • 25. I needed to set up mine.
  • 26. I created aprivate repositoryand put everything into it.
  • 27. Merged recentcommits from domms repository.
  • 28. Added a few columns.
  • 29. TweakedCatalyst/DBIC stuff.
  • 30. It worked.
  • 31. Warnings were left.
  • 32. I needed to find some tuits to remove them.
  • 33. Perl QAHackathon
  • 34. Warnings were removed.
  • 35. Ported some ofthe changes I didlocally to daxims repository.
  • 36. Showed a new acme.cpanauthors.orgfeaturing CPANTS info.
  • 37. Unfortunately,the porting took too much time.
  • 38. I didnt mergethe changes backto my repository.
  • 39. OSDC.TW
  • 40. I finally merged the changes.
  • 41. Got severalreports thatCPANTS was broken.
  • 42. What brokeCPANTS was a small change.
  • 43. "modules" : [ { "file" : "lib/Path/Extended.pm", "in_basedir" : 0, "in_lib" : 1, "module" : "Path::Extended", "uses" : { "Sub::Install" : 1, "strict" : 1, "warnings" : 1 } }]
  • 44. I dont think this change is bad.
  • 45. Module::CPANTS:: ProcessCPAN shouldnt have died by this.
  • 46. It should have had tests.
  • 47. Is should have run faster.
  • 48. It should havebeen easier to fix analysis.
  • 49. Enough issues for a summer.
  • 50. What should we do?
  • 51. - We need tests.- we need to find test cases.- we need to do it many times.
  • 52. Making it runfaster is thefirst priority.
  • 53. I wrote abarebone scriptto store data in parallel.
  • 54. JSONcreate table if not exists analysis ( id integer primary key autoincrement, path text unique, distv text, author text, json text, duration integer);
  • 55. Raw SQLstatements
  • 56. Parallel::ForkManager
  • 57. SQLite queue
  • 58. Beware a race conditionmy ($id) = $dbh->selectrow_array(" SELECT id FROM queue WHERE status = 0 LIMIT = 1");$dbh->do(" UPDATE queue SET status = 1 WHERE id = ?", undef, $id);
  • 59. sqlite_update_hookmy $id;my $dbh->sqlite_update_hook(sub { (undef, undef, undef, $id) = @_;});
  • 60. $dbh->do(" UPDATE queue SET status = 1, WHERE id IN ( SELECT id FROM queue WHERE status = 0 LIMIT 1 )");
  • 61. Archive::Any::Lite
  • 62. Archive::Any::Plugin::Bzip2
  • 63. WorePAN- Bundling is bad- We need a specificversion- Derived from OrePAN
  • 64. use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, files => [qw( I/IS/ISHIGAKI/WorePAN-0.01.tar.gz )], use_backpan => 1, no_network => 0, cleanup => 1,);
  • 65. use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, files => [qw( I/IS/ISHIGAKI/WorePAN-0.01.tar.gz )], local_mirror => /home/ishigaki/minicpan/, no_network => 1, cleanup => 1,);
  • 66. use WorePAN;my $worepan = WorePAN->new( root => path/to/a/directory/, dists => { Catalyst-Runtime => 5.9, DBIx-Class => 0, }, cleanup => 1,);
  • 67. Bonus featuresmy $worepan = WorePAN->new( root => path/to/a/CPAN/mirror/, cleanup => 0,);my $authors = $worepan->authors;my $modules = $worepan->modules;my $file = $worepan->files;my $dists = $worepan->latest_distributions;
  • 68. $worepan->add_files(qw{ /path/to/a/local/distribution-0.01.tar.gz});$worepan->update_indices;
  • 69. Now we haveenough tools.
  • 70. Processing time is significantly decreased.
  • 71. Whats next?
  • 72. ::Site refactoring
  • 73. Im preparing the data now.
  • 74. Creating moredatabases/tables.
  • 75. Merginginformation fromexternal sources.- CPAN indices- CPAN uploads database
  • 76. Calculating scores on prerequisite modules.
  • 77. It will be thisyears somethingnew in my annual report.
  • 78. And then, Illmove on to fixing the metrics.
  • 79. Some of them are badly broken."versions" : { "lib/Data/Phrasebook.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Debug.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Generic.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader/Base.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Loader/Text.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/Plain.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/SQL.pm" : "use vars qw($VERSION);¥n", "lib/Data/Phrasebook/SQL/Query.pm" : "use vars qw($VERSION);¥n"},
  • 80. Error is not a stash."error" : { "easily_repackageable" : "easily_repackageable_by_fedora", "easily_repackageable_by_fedora" : "fits_fedora_license", "metayml_conforms_spec_current" : [ "1.4", "Expected a map structure from data string or file. [Validation: 1.4]" ], "metayml_conforms_to_known_spec" : [ "1.0", "Expected a map structure from data string or file. [Validation: 1.0]" ], "no_pod_errors" : " home cpants tmp analyze 11442 8001be43fb65..."}
  • 81. Should haveinitialize/finalize phases.Module::CPANTS::Kwalitee::Distros doesnt clean up after mirrored Debian CPANTS file https://rt.cpan.org/Ticket/Display.html?id=51514
  • 82. There are much more to do.- JSON API for metacpan.org and so on.- Email Reporting like CPAN Testers- Evaluate new Kwalitee indicators- New metrics like portable filename- Blog about recent tendency- More comprehensive tests- Analysis per perl version/architecture- Cover Perl::Critic, CPAN::Critic::Module::Abstract- 35 RT tickets and several github isses
  • 83. Resources github.com/charsbar/www-cpants github.com/charsbar/worepangithub.com/daxim/Module-CPANTS-Analyse
  • 84. Questions?
  • 85. Thank you