Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

No Hugging, No Learning

629 views

Published on

My attempt to create wundercharts.com without actually learning anything new.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

No Hugging, No Learning

  1. 1. No Hugging, No Learning Olaf Alders Toronto Perl Mongers Aug 25, 2016 @olafalders (Twitter) oalders (GitHub) OALDERS (PAUSE) https://github.com/oalders/no-hugging-no-learning
  2. 2. The Problem Building an app that can track and chart (almost) anything that is available via 3rd party APIs No screen scraping No ToS violations Doing this in my limited spare time
  3. 3. The Solution Build an app based solely on what I already know: "No hugging, no learning" Of the three programmer virtues (laziness, impatience and hubris), I would let laziness be my guiding light I tried to self-impose a real lack of intellectual curiosity
  4. 4. The Evolution
  5. 5. The App Before Some command line scripts as a proof of concept This became a Dancer (1) app run via Plack
  6. 6. Pros Dancer apps are easy to get up and running Cons I didn't really love the Dancer (1) syntax Dancer2 was still quite new and the plugin support wasn't there yet
  7. 7. After I moved to a Mojolicous Lite app run via Plack I then transitioned to a full Mojo app Run via morbo in development Run via hypnotoad in production
  8. 8. morbo By default watches your lesystem for changes and then reloads your app
  9. 9. hypnotoad Zero downtime restarts You don't need to provide your own init script If you don't care about zero downtime or just want the prefork server, you have that option perl script/myapp prefork same con g args as morbo/daemon
  10. 10. Did I learn? I partly became familiar with Mojo via my day job, but I did have to do a fair bit of learning on my own. I was not familiar with Hypnotoad , but luckily that was a pretty easy transition. The docs are really good.
  11. 11. Authentication
  12. 12. Before Mozilla's Persona
  13. 13. Pros Pretty easy to con gure All anyone really needs is an email address Cons Issues getting it to work of ine No widespread adoption Felt like the rst 80% was easy but the last 20% was eating into my time
  14. 14. After OAuth via any of the available integrations I needed this anyway in order to get user data, so it keeps things simpler to have Persona out of the mix With Mozilla no longer 100% behind Persona, it becomes a less attractive solution most non-technical users won't be familiar with it at this point anyway
  15. 15. SSL Everywhere Looked brie y into Let's Encrypt via Ansible Ended up spending USD 10 for a RapidSSL cert
  16. 16. Relational Database
  17. 17. Before Application data stored in MySQL database
  18. 18. Pros I'm very familiar with MySQL I prefer many MySQL tools, like the mysql command line interface and phpMyAdmin MySQL replication is dead easy to con gure Cons "Foreign keys that point to the renamed table are not automatically updated. In such cases, you must drop and re-create the foreign keys in order for them to function properly."
  19. 19. After 3 Postgres databases Minion Application data Test suite xtures
  20. 20. Did I learn? Since we use Postgres at $work , I didn't have to learn a whole lot to make the switch. The database does a litle bit more than the bare minimum -- I'm not taking advantage of all that Postgres has to offer No jsonb columns just yet
  21. 21. Job Management
  22. 22. Before cron scripts
  23. 23. After Minion
  24. 24. Did I learn? Had basically zero knowledge of Minion implementation But, there's not much you need to learn in order to get up and running Minimized a lot of really convoluted cron job logic This probably saved me time in the long run
  25. 25. Minion How To You don't need to have a full-blown Mojo app to use Minion Just create a bare bones app to get started Use this with your Catalyst, Dancer, [insert framework here] application Let's look at how MetaCPAN uses Minion
  26. 26. Create an App https://github.com/metacpan/metacpan- api/blob/master/lib/MetaCPAN/Queue.pm Sets up a tiny Mojo app Adds Minion tasks in the startup() method You can see how we trap warnings and log failed jobs in the _gen_index_task_sub() method
  27. 27. Optionally Create Your Minion Backend Elsewhere We've abstracted it out to https://github.com/metacpan/metacpan- api/blob/master/lib/MetaCPAN/Queue/Helper.pm We do this so that we can use an SQLite database when running under the test harness and a Postgres database in all other cases Using SQLite for testing makes our Travis con guration much easier This will also allow us more easily to run tests in parallel
  28. 28. Start Up Your Queue In development, you can create a 4-5 line script to start up your queue via morbo https://github.com/metacpan/metacpan- api/blob/master/bin/queue.pl In production we use Daemon::Control to manage starting and stopping a daemon https://github.com/metacpan/metacpan- puppet/blob/master/modules/minion_queue/te mplates/init.pl.erb
  29. 29. Add Tasks A task is essentially an anonymous sub which you can later refer to by name A job is an instance of a task $minion->add_task( add_things => sub { # $job is a Minion::Job object my ($job, $first, $second) = @_; $job->finish($first + $second); });
  30. 30. More Complete Example $minion->add_task( add_things => sub { my ($job, $first, $second) = @_; $job->finish({ message => 'Great!', total => $first + $second, }); }); my $id = $minion->enqueue(add_things => [1, 1]); my $result = $minion->job($id)->info->{result}; $result is now { message => 'Great!', total => 2 }
  31. 31. Storing Your Job Result $job->finish; $job->finish('Fantastic!'); $job->finish({ foo => 'bar' }); Later, you can nd this arbitrary data in $job->result
  32. 32. Populate Your Queue enqueue() means "add job to queue" $app->minion->enqueue( my_task_name => ['foo', 'bar'] ); $self->minion->enqueue( track_service_resource => [{ service_resource_id => $service_resource->id }] => { priority => 1 }, ); priority ranges from 0-X, where X is a positive integer jobs with priority 10 will run before jobs with priority 9 etc
  33. 33. enqueue() options attempts => 10 (# of times to attempt job) delay => 3600 (delay this job for 3600 seconds) parents => [$id1, $id2] (One or more existing jobs this job depends on, and that need to have transitioned to the state finished before it can be processed) priority => 3 (see previous slide) queue => 'dexter' (some arbitrary name, defaults to default )
  34. 34. Minion::Job https://metacpan.org/pod/Minion::Job You can set up events to be red on failed and finished states Get information via $job->info $job->is_finished $job->remove Much, much more
  35. 35. Inspect Your Queue On the MetaCPAN Vagrant box: $ vagrant ssh $ cd /home/vagrant/metacpan-api/ $ ./bin/run bin/queue.pl minion job -s { "active_jobs" => 0, "active_workers" => 0, "delayed_jobs" => 0, "failed_jobs" => 0, "finished_jobs" => 0, "inactive_jobs" => 0, "inactive_workers" => 1 }
  36. 36. Test Your Queue https://github.com/metacpan/metacpan- api/blob/master/t/queue.t use MetaCPAN::Queue; use Test::More; use Test::RequiresInternet ( 'cpan.metacpan.org' => 443 ); my $app = MetaCPAN::Queue->new; my $release = 'https://cpan.metacpan.org/authors/id/O/OA/OALDERS $app->minion->enqueue( index_release => [ '--latest', $release ] ); $app->minion->perform_jobs; done_testing();
  37. 37. Advanced: Behaviour on Events https://github.com/jberger/Minion- Noti er/blob/master/lib/Minion/Noti er.pm#L39- L56
  38. 38. sub setup_worker { my $self = shift; my $dequeue = sub { my ($worker, $job) = @_; my $id = $job->id; $self->transport->send($id, 'dequeue'); $job->on(finished => sub { $self->transport->send($id, $job->on(failed => sub { $self->transport->send($id, }; $self->minion->on(worker => sub { my ($minion, $worker) = @_; $worker->on(dequeue => $dequeue); }); return $self }
  39. 39. Code Context The preceding code is called like this: https://github.com/jberger/Minion- Noti er/blob/master/lib/Mojolicious/Plugin/Minion/ Noti er.pm#L37
  40. 40. Pros We can replace a failure-prone, forking, long running process with a series of jobs If a job fails in an unexpected way, it doesn't prevent all the other jobs from running We can check the status of all jobs at any point to gauge how things are progressing Jobs which fail can be retried at intervals We can start an arbitrary number of worker apps Using Postgres replication MetaCPAN can now start X workers on 3 different boxes, which gives us greater scaling when needed
  41. 41. Cons Minion doesn't come with a handy daemon handler like hypnotoad , but you can set this stuff all up yourself pretty easily There aren't yet a lot of tools to visualize what's going on with the queue, but again this can be done pretty easily if you need it.
  42. 42. Database Migrations
  43. 43. Before Database schema managed by Database::Migrator
  44. 44. Pros Database::Migrator is what we use at $work It's pretty simple In addition to SQL, you can also run Perl scripts as part of a migration Cons Database::Migrator needs to be subclassed, which means you have to provide a fair bit of functionality just to get up and running
  45. 45. After Application and xture databases managed via App::Sqitch Minion has its own schema deployment logic
  46. 46. Pros sqitch handles rollbacks as well as deployment Optionally can use custom assertions to ensure that deployments were successful Cons There's a lot more going on here, so you probably have to do a bit of reading before just jumping in I had to use the CLI rather than the modules because of some weirdness between Moo and Moose when I tried to use the modules directly
  47. 47. Did I learn? Yes, I did have to learn sqitch from scratch The author (David Wheeler) was extremely helpful I was able to get it working with my xtures
  48. 48. See Also Mojo::Pg::Migrations Mojo::mysql::Migrations Mojo::SQLite::Migrations
  49. 49. Automation of Deployment
  50. 50. Before The plan was to use Puppet
  51. 51. Pros I was already familiar with Puppet Cons I was already familiar with Puppet Even simple things seemed hard Tired of dealing with hard to debug certi cate issues There may have been obvious and easy xes to my problems, but I couldn't easily nd them I'm not saying that Puppet isn't a wonderful tool, but it felt like a bigger hammer than I needed
  52. 52. After Ansible
  53. 53. Pros Excellent integration with Vagrant No need for bootstrapping on the target deployment machines It's easily installed via Homebrew on OS X Has good handling of git submodule User contributed plugins are trivial to install
  54. 54. Cons I actually don't have a lot of complaints Moving from v1 to v2 addressed the few issues that I ran into
  55. 55. Ansible in Vagrant le config.vm.provision "ansible" do |ansible| ansible.verbose = "v" ansible.playbook = "ansible/site.yml" ansible.sudo = true ansible.groups = { "development" => ["default"], "webservers" => ["default"], } end
  56. 56. Installing Packages --- - gather_facts: no hosts: webservers tasks: - apt: 'pkg={{item}} state=installed' name: Install misc apt packages with_items: - curl - ntp - tree - unzip - apt: update_cache=yes cache_valid_time=3600 name: Run the equivalent of "apt-get update" - apt: upgrade=dist autoremove=true name: Update all packages to the latest version
  57. 57. Did I learn? Yes, I had to learn all about Ansible , since I had never used it before. In the meantime $work has also switched from puppet to ansible , so it turned out to be knowledge that I can apply there as well. Keeping in mind how many hours I've spent battling Puppet on past projects, I think learning about Ansible was actually the better, faster choice.
  58. 58. Predictions Which Did Not Change
  59. 59. JS-Driven Charts Highcharts turned out to be an excellent choice Excellent docs Responsive support I'm also quite happy with the product itself
  60. 60. Twitter Bootstrap Still a solid choice for a CSS framework
  61. 61. Travis CI Private repository support is not within my budget (starts at USD 129/month) Open Source option works excellently for https://github.com/oalders/wundercharts-plugin
  62. 62. Carton for Dependency Pinning This suits my needs ne at this point https://metacpan.org/pod/Carmel is still listed as experimental
  63. 63. Vagrant Initially I just ran the app inside OS X directly I realized pretty quickly that I wanted to have a sandbox I've even got a Vagrantfile for the plugin system https://github.com/oalders/wundercharts- plugin/blob/master/Vagrant le So, anyone who wants to contribute can have a fully operational, sandboxed system with all of the modules installed within a matter of minutes
  64. 64. Code::TidyAll I use this to tidy as much code as possible It's enforced via a Git pre-commit hook so that I don't have to think about it
  65. 65. LWP::ConsoleLogger I use this a lot when debugging interactions with 3rd party APIs This only really works when you can get at the UA which a module is using I wish more module authors would expose their user agents The Mojo equivalent is setting MOJO_USERAGENT_DEBUG=1
  66. 66. DBIx::Class Use foreign keys and the schema dumper automatically sets up relationships for you Use this trick to enable perltidy to tidy your schema les: https://metacpan.org/pod/release/ILMARI/DBIx- Class-Schema-Loader- 0.07045/lib/DBIx/Class/Schema/Loader/Base.pm # lter_generated_code
  67. 67. Mojolicious::Plugin::AssetPack Compress and convert css, less, sass, javascript and coffeescript les In development mode you get the individual les in your template (for easier debugging)
  68. 68. Strict Constructors MooseX::StrictConstructor MooX::StrictConstructor MetaCPAN::Moose
  69. 69. etc nginx Ubuntu 14.04.5 LTS Perl v5.18.2
  70. 70. Things I Hadn't Planned on Using
  71. 71. Wercker I didn't know this existed I'm not good about running my test suite after every change, so I realized I had to automate this I didn't want to run my own Jenkins/TeamCity/etc service I've seen how much con guration can go into these things and I didn't have the time to mess with it Free, concurrent builds for private repositories
  72. 72. Wercker (cont'd) I didn't want to give it access to all of my private repositories I created a BitBucket account (free private repositories) Added a new remote to my repository "bitbucket" Every time I push to the "bitbucket" remote the tests are run and I get an email about success or failure Protip: if weird errors happen, sometimes you have to clear your Wercker cache
  73. 73. wrapbootstrap.com I hadn't even thought about the UI when I started this project I bought the "Inspina" theme for USD 18 This gave me solid options for both logged in and logged out UI Was trivial to implement and immediately made my terrible design much better Since I was already using Bootstrap there weren't a lot of changes to be made
  74. 74. Date Range Picker http://www.daterangepicker.com Easy to use and makes it easy to use sensible defaults when selecting dates or ranges of dates
  75. 75. Open Source That Came Out of This Mojolicious::Plugin::Web::Auth::Site::Spotify Mojolicious::Plugin::Web::Auth::Site::Runkeeper WebService::HealthGraph Code::TidyAll::Plugin::YAML WWW::Spotify (co-maint) Patch to WebService::Instagram I open sourced all of my plugins
  76. 76. WunderCharts Plugins https://github.com/oalders/wundercharts-plugin 100% open source I'm in a position to integrate user contributed patches without open sourcing the entire code base
  77. 77. For example, if we want to integrate FitBit Need to create Mojolicious::Plugin::Web::Auth::Site::FitBit Need to send a pull request for WunderCharts::Plugin::FitBit
  78. 78. Third Party Integrations I didn't have a rm idea of how many I wanted Got a mixture of OAuth, OAuth2 and no auth
  79. 79. OAuth Twitter OAuth 2 Facebook Instagram Spotify GitHub No authentication required Hacker News
  80. 80. Don't Count Your Chickens Instagram apps (unlike the other sites) have a formal review process I didn't realize until after my integration was nished that my use case may not qualify I did get accepted at the end of the day, but it's a bit of a coin toss You need to show them the integration via a screen cast before you get approval, so it's possible to do the heavy lifting and still be denied
  81. 81. Success? Among other things, I ended up having to learn (or learn more about) the following technologies: Ansible BitBucket Date Range Picker hypnotoad Minion Mojo Mojolicious::Plugin::AssetPack
  82. 82. Persona sqitch Wercker [also various 3rd party APIs for web services]
  83. 83. Thanks! To Joel Berger for proofreading the Mojo bits of an earlier version of this talk All remaining (or since introduced) errors are my own
  84. 84. The "Finished" Product https://wundercharts.com

×