Ab(Using) the MetaCPAN API for Fun and Profit v2013

2,047 views

Published on

This talk builds on the previous Ab(Using) the MetaCPAN API talk from 2012. There are links to comprehensive examples and there is a more in-depth look at the available endpoints.

Published in: Technology, Art & Photos
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,047
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Ab(Using) the MetaCPAN API for Fun and Profit v2013

  1. 1. (Ab)Using theMetaCPAN API forFun and Profit v2013Olaf Alders (OALDERS)@wundercounterThursday, 13 June, 13
  2. 2. Architecture• Built on ElasticSearch• Uses Catalyst as a thin wrapperThursday, 13 June, 13
  3. 3. Objectives• Rough introduction to the API• Some ElasticSearch pointersThursday, 13 June, 13
  4. 4. ElasticSearch• ElasticSearch (ES) has indexes andtypes• In RDBMS terms?• index = database• type = table• OK to think of it that wayThursday, 13 June, 13
  5. 5. Let’s look at types• Every type has a correspondingendpointThursday, 13 June, 13
  6. 6. Thursday, 13 June, 13
  7. 7. Thursday, 13 June, 13
  8. 8. Base URL• http://api.metacpan.org/v0Thursday, 13 June, 13
  9. 9. Endpoints• Convenience endpoints• ElasticSearch endpoints• There *is* some overlapThursday, 13 June, 13
  10. 10. Think of it this way• Every type has a correspondingendpoint• Not every endpoint has acorresponding typeThursday, 13 June, 13
  11. 11. Convenience EndpointsThursday, 13 June, 13
  12. 12. Convenience Endpoints• /author/DOY• /distribution/Moose• /release/Moose• /module/Moose• /pod/Moose• /search/autocomplete?q=Moose• /search/reverse_dependencies/MooseThursday, 13 June, 13
  13. 13. Versioned ConvenienceEndpoints• /release/DOY/Moose-2.0001• /module/DOY/Moose-2.0001/lib/Moose.pm• /pod/DOY/Moose-2.0001/lib/Moose.pm• /search/reverse_dependencies/DOY/Moose-2.0001Thursday, 13 June, 13
  14. 14. Special Cases• Some convenience URLs don’t exposeElasticSearch’s DSLThursday, 13 June, 13
  15. 15. /pod• GET your /pod• Don’t send JSON in your request• Don’t expect JSON in your responseThursday, 13 June, 13
  16. 16. Set content-type via param or header• /pod/Moose?content-type=text/html (default)• /pod/Moose?content-type=text/plain• /pod/Moose?content-type=text/x-pod• /pod/Moose?content-type=text/x-markdownThursday, 13 June, 13
  17. 17. /pod the easy wayuse MetaCPAN::API;my $mcpan = MetaCPAN::API->new;my $html_pod = $mcpan->pod(module => Carton,content-type => text/plain);Text(MetaCPAN::API::Tiny has the same syntax)Thursday, 13 June, 13
  18. 18. Thursday, 13 June, 13
  19. 19. Thursday, 13 June, 13
  20. 20. Autocomplete Made Easy• /v0/search/autocomplete?q=HTTP• GET this endpoint• No JSON in• Expect JSON outThursday, 13 June, 13
  21. 21. Thursday, 13 June, 13
  22. 22. Alfred 2 Autocompletionhttps://github.com/2shortplanks/alfred-metacpanThursday, 13 June, 13
  23. 23. The (real) EndpointsThursday, 13 June, 13
  24. 24. The (real) Endpoints• /authorThursday, 13 June, 13
  25. 25. The (real) Endpoints• /author• /distributionThursday, 13 June, 13
  26. 26. The (real) Endpoints• /author• /distribution• /favoriteThursday, 13 June, 13
  27. 27. The (real) Endpoints• /author• /distribution• /favorite• /fileThursday, 13 June, 13
  28. 28. The (real) Endpoints• /author• /distribution• /favorite• /file• /ratingThursday, 13 June, 13
  29. 29. The (real) Endpoints• /author• /distribution• /favorite• /file• /rating• /releaseThursday, 13 June, 13
  30. 30. Why is /module missing?• A module is a kind of file• file.module.name = $module_nameThursday, 13 June, 13
  31. 31. One special endpoint• https://api.metacpan.org/v0/user• You need to be logged in• Note the SSL ^^^Thursday, 13 June, 13
  32. 32. Exploring Endpoints• Documentation is weak• You may need to dig around a bitThursday, 13 June, 13
  33. 33. MetaCPAN ExplorerThursday, 13 June, 13
  34. 34. Thursday, 13 June, 13
  35. 35. Thursday, 13 June, 13
  36. 36. Thursday, 13 June, 13
  37. 37. Dig into the API code• https://github.com/CPAN-API/cpan-api/blob/master/lib/MetaCPAN/Document/Author.pmThursday, 13 June, 13
  38. 38. Good Habits• Make your life easierThursday, 13 June, 13
  39. 39. Enable Compression• use WWW::Mechanize::Gzip• use WWW::Mechanize::Cached::GZip• Or set the appropriate request headerThursday, 13 June, 13
  40. 40. POST POST POST• Beyond the most trivial of queries,GET is a PITA.• POSTing will make your life easierThursday, 13 June, 13
  41. 41. Are you asking for too much?• There is an upper limit of 5,000 onquery size• If you need more data points, use thescrolling APIThursday, 13 June, 13
  42. 42. Scrolling• The scrolling API allows you to iterateover an arbitrary number of results• Be aware that when you scroll, yourdocs will come back unsorted• Let ElasticSearch.pm take care of thedetailsThursday, 13 June, 13
  43. 43. query vs filter context• generally you just want filters• use a query if you need to sort yourresults by relevance• https://metacpan.org/module/DRTECH/ElasticSearch-SearchBuilder-0.18/lib/ElasticSearch/SearchBuilder.pm#QUERY-FILTER-CONTEXTThursday, 13 June, 13
  44. 44. Let’s Get StartedThursday, 13 June, 13
  45. 45. Let’s Get Started• We want to fetch some dataThursday, 13 June, 13
  46. 46. Let’s Get Started• We want to fetch some data• TIMTOWDIThursday, 13 June, 13
  47. 47. Code Samples• https://github.com/cpan-api/metacpan-examplesThursday, 13 June, 13
  48. 48. In closing...Thursday, 13 June, 13
  49. 49. Hack on *MetaCPAN*• Download a pre-configured VM• Requires VirtualBox + vagrant• https://github.com/cpan-api/metacpan-developerThursday, 13 June, 13
  50. 50. Production• Before you go live...Thursday, 13 June, 13
  51. 51. Rule #1• Be politeThursday, 13 June, 13
  52. 52. Rule #2• There isn’t actually a rule #2. Just bepolite.Thursday, 13 June, 13
  53. 53. Identify Yourself• Not mandatory, but encouraged• Do this when you’re in production• https://github.com/CPAN-API/cpan-api/wiki/API-ConsumersThursday, 13 June, 13
  54. 54. Getting Help• #metacpan on irc.perl.org• https://metacpan.org/about/resourcesThursday, 13 June, 13
  55. 55. Resources• https://github.com/CPAN-API/cpan-api/wiki/Beta-API-docs• http://www.slideshare.net/clintongormley/terms-of-endearment-the-elasticsearch-query-dsl-explainedThursday, 13 June, 13
  56. 56. Bonus Slides• These apps all use the MetaCPAN APIThursday, 13 June, 13
  57. 57. iCPAN - iPhoneThursday, 13 June, 13
  58. 58. iCPAN - iPadThursday, 13 June, 13
  59. 59. AndroidThursday, 13 June, 13
  60. 60. Thursday, 13 June, 13
  61. 61. Thursday, 13 June, 13
  62. 62. Thursday, 13 June, 13
  63. 63. Thursday, 13 June, 13
  64. 64. Thursday, 13 June, 13
  65. 65. Thursday, 13 June, 13
  66. 66. Thursday, 13 June, 13
  67. 67. Thursday, 13 June, 13
  68. 68. Thursday, 13 June, 13

×