Beijing Perl Workshop 2008 Hiveminder Secret Sauce

2,028 views
1,927 views

Published on

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,028
On SlideShare
0
From Embeds
0
Number of Embeds
27
Actions
Shares
0
Downloads
32
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Beijing Perl Workshop 2008 Hiveminder Secret Sauce

  1. 1. Everything but the Secret Sauce Modules, Magic and Monstrosities created for
  2. 2. Jesse Vincent Best Practical
  3. 3. We make Hiveminder
  4. 4. It’s not opensource
  5. 5. We built lots of cool stuff for Hiveminder
  6. 6. We opensource everything we can
  7. 7. Today’s Agenda Ten tools and techniques to help you: Find bugs faster Build web apps Ship software Get input from users Own the Inbox
  8. 8. Find bugs faster
  9. 9. TAP-Harness-Remote Run your tests over there.
  10. 10. The Problem
  11. 11. Hiveminder has lots of tests
  12. 12. That's a problem?
  13. 13. Yes.
  14. 14. 30 minutes to run tests on my Macbook
  15. 15. (It took the same on a Macbook Pro)
  16. 16. What to do?
  17. 17. TAP::Harness 3.0
  18. 18. I have two cores!
  19. 19. TAP::Harness::Parallel
  20. 20. prove -j 5 --fork
  21. 21. 22 minutes
  22. 22. Better!
  23. 23. Not good.
  24. 24. Run tests on a server!
  25. 25. rsync -rvp . server: ssh server quot;prove -j 9 tquot;
  26. 26. Better.
  27. 27. Annoying to run.
  28. 28. TAP::Harness::Remote
  29. 29. Magic Remote Testing
  30. 30. # ~/.remote_test --- local: /path/to/local/testing/root/ host: - remote1.testing.host.example.com - remote2.testing.host.example.com root: /where/to/place/local/root/on/remote/ user: username master: 1 perl: /usr/bin/perl ssh: /usr/bin/ssh ssh_args: - -x - -S - '~/.ssh/master-%r@%h:%p'
  31. 31. prove --harness TAP::Harness::Remote t/*.t
  32. 32. 16.5 minutes
  33. 33. prove --harness TAP::Harness::Remote -j 5 t/*.t
  34. 34. Magic Parallel Remote Testing
  35. 35. 12 minutes!
  36. 36. Not good enough for TDD
  37. 37. What I need is... • Fast Servers • Lots of Fast Servers • Right Now • Not Very Often
  38. 38. A use for cloud computing!
  39. 39. TAP::Harness::Remote::EC2
  40. 40. 4 hosts. -j 5
  41. 41. 89 seconds
  42. 42. WIN
  43. 43. EC2 Caveats Default Perl builds are really bad (“XL” instances are great, though) Machine Setup/Teardown is ~manual The billing structure is crazy
  44. 44. What's next for ::Harness::Remote::* ? More polish on EC2 tools Better EC2 watchdogs More widespread usage
  45. 45. Carp::REPL It's not quite a debugger
  46. 46. How many of you use the Perl debugger?
  47. 47. How many of you know how to use the perl debugger?
  48. 48. I don’t, either.
  49. 49. So, what do I use?
  50. 50. 31b:~ jesse$ perl /tmp/contrived.pl Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 20. 31b:~ jesse$ vim /tmp/contrived.pl 18.sub c { 19. my $to_call = shift; 20. $to_call = quot;does_quot;.$to_call; 21. no strict refs; 22. $to_call->(); 23.}
  51. 51. 31b:~ jesse$ perl /tmp/contrived.pl Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 22. 31b:~ jesse$ vim /tmp/contrived.pl 18.sub c { 19. my $to_call = shift; 20. $to_call = quot;does_quot;.$to_call; 21. no strict refs; 22. use Carp; Carp::cluck(quot;$to_callquot;); 23. $to_call->(); 24.}
  52. 52. Ick. A Print Statement.
  53. 53. Anyone else here use print statement debugging?
  54. 54. (Oh god, I hope I'm not alone)
  55. 55. I want a simple debugger that's easier than print statements
  56. 56. sartak created Carp::REPL
  57. 57. A lightweight, modern, interactive debugger
  58. 58. Built on Devel::REPL
  59. 59. Carp::REPL Hooks into $SIG{__DIE__}
  60. 60. $SIG{__WARN__} hooks are optional
  61. 61. 31b:~ jesse$ perl -MCarp::REPL /tmp/contrived.pl Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 22. 0: Carp::REPL::repl called at /tmp/contrived.pl:22. 1: main::c called at /tmp/contrived.pl:15. 2: main::b called at /tmp/contrived.pl:9. 3: main::a called at /tmp/contrived.pl:27. Now at /tmp/contrived.pl:22 (frame 0). $
  62. 62. That's not a shell prompt
  63. 63. 31b:~ jesse$ perl -MCarp::REPL /tmp/contrived.pl Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 22. 0: Carp::REPL::repl called at /tmp/contrived.pl:22. 1: main::c called at /tmp/contrived.pl:15. 2: main::b called at /tmp/contrived.pl:9. 3: main::a called at /tmp/contrived.pl:27. Now at /tmp/contrived.pl:22 (frame 0). $ warn $0 /tmp/contrived.pl at (eval 135) line 15, <FIN> line 1.
  64. 64. It's Perl, but it's not just Perl
  65. 65. :t - Stack trace :u - Up one stack frame :d - Down one stack frame :e - Print this frame's lexical environment :q - Quit this REPL $_e - ARRAYREF of this frame's @_
  66. 66. $ :t 0: Carp::REPL::repl called at /tmp/contrived.pl:22. 1: main::c called at /tmp/contrived.pl:15. 2: main::b called at /tmp/contrived.pl:9. 3: main::a called at /tmp/contrived.pl:27. $
  67. 67. $ :e $Env = { quot;$to_callquot; => do { my $v = 'does_not_exist' } }; $ :u $ :u Now at /tmp/contrived.pl:9 (frame 2). $ :e $Env = { quot;$to_callquot; => do { my $v = 'exist' } }; $ :u Now at /tmp/contrived.pl:27 (frame 3). $ :e $Env = { quot;$to_callquot; => do { my $v = '' } }; $ :d Now at /tmp/contrived.pl:9 (frame 2). $ :e $Env = { quot;$to_callquot; => do { my $v = 'exist' } }; $ warn join(quot;, quot;,@$_a); exist at (eval 138) line 18, <FIN> line 4.
  68. 68. Why is my code 'warn'ing? Maybe it's in an eval. Or in a CPAN library. No problem! perl -MCarp::REPL=warn
  69. 69. 31b:~ jesse$ perl -MCarp::REPL=warn /tmp/contrived.pl Use of uninitialized value in concatenation (.) or string at /tmp/contrived.pl line 8. 0: Carp::REPL::repl called at /tmp/contrived.pl:8. 1: main::a called at /tmp/contrived.pl:27. Now at /tmp/contrived.pl:8 (frame 0). $ :e $Env = { quot;$to_callquot; => do { my $v = undef } }; $ :q Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 22, <FIN> line 2. 0: Carp::REPL::repl called at /tmp/contrived.pl:22. 1: main::c called at /tmp/contrived.pl:15. 2: main::b called at /tmp/contrived.pl:9. 3: main::a called at /tmp/contrived.pl:27. Now at /tmp/contrived.pl:22 (frame 0). $
  70. 70. Inserting breakpoints You know your 'print' statements? Turn them into breakpoints. Two ways: s/print/warn/ and -MCarp::REPL=warn Carp::REPL::repl
  71. 71. 18.sub c { 19. my $to_call = shift; 20. no strict 'refs'; 21. Carp::REPL::repl(); #Gimme a breakpoint here! 22. $to_call = quot;does_quot;.$to_call; 23. $to_call->(); 24.}
  72. 72. What's next for Carp::REPL? I have no idea. It's simple. It works.
  73. 73. Build web apps
  74. 74. Template::Declare A Pure Perl Templating Engine
  75. 75. Perl has two big templating camps
  76. 76. Template::Toolkit A DSL for templates It's NOT Perl Why should I learn a new language for templates?
  77. 77. HTML::Mason There's Perl inside.
  78. 78. HTML::Mason <mason><is><still> <full><of> <angle><brackets>
  79. 79. HTML::Mason </brackets></angle> </of></full> </still></is></mason>
  80. 80. I'm Just A Simple Perl Hacker
  81. 81. I needed something more Perlish
  82. 82. template '/pages/mypage.html' => sub { html { head {}; body { h1 {'Hey, this is text'}; } } };
  83. 83. template choices => page { h1 { 'My Choices' } dl { my $choices = Doxory::Model::ChoiceCollection->new; $choices->limit( column => 'asked_by', value => Jifty->web->current_user->id, ); while (my $c = $choices->next) { dt { $c->name, ' (', $c->asked_by->name, ')' } dd { b { $c->a } em { 'vs' } b { $c->b } } } } };
  84. 84. But! Content! Templates! Design! Code! OMGWTFBBQ!? THAT'S WRONG!
  85. 85. The person who told you it's wrong was lying to you.
  86. 86. We're Perl hackers
  87. 87. Why are we writing our templates in another language?
  88. 88. This is not 1997
  89. 89. It’s 2008
  90. 90. People use CSS for design now
  91. 91. Programmers still have to make templates
  92. 92. Templates run like CODE
  93. 93. Because they ARE code
  94. 94. Let's use our CODING tools to work with them
  95. 95. How about Refactoring?
  96. 96. Have you ever tried to refactor HTML?
  97. 97. template 'mypage.html' => page { h1 { 'Two choices' }; div { attr { class => 'item' }; h2 { 'Item 1'}; }; div { attr { class => 'item' }; h2 { 'Item 2'}; }; };
  98. 98. template 'mypage.html' => page { h1 { 'Two choices' }; for (quot;Item 1quot;, quot;Item 2quot;) { item($_); } }; sub item { my $content = shift; div { attr { class => 'item' }; h2 {$content}; }; }
  99. 99. We can refactor templates!
  100. 100. Our HTML is magically valid
  101. 101. (Syntax errors are... Syntax Errors)
  102. 102. Object Oriented Templates
  103. 103. Subclassing It just works We're using Perl's symbol table (Template::Declare's show() passes the class for you)
  104. 104. Mixins alias Some::Template::Library under '/path'; That's it.
  105. 105. Closures
  106. 106. Tags as Closures HTML tags take blocks of content Our tag methods take blocks of Perl They return closures when you want them They run and output their content when you want them to
  107. 107. sub h1 (&;$) { my $code = shift; ... if (defined wantarray) { return sub { ...closure around $code...}; } else { # Actually do our work, run $code and # return the output return $code->(); } }
  108. 108. What's next for Template::Declare? HTML Attribute Validation Compile to .js
  109. 109. CSS::Squish A compiler for Cascading Style Sheets
  110. 110. The Problem Many small CSS Files Easy to work with Painful to serve
  111. 111. The Problem One big CSS file Painful to work with Easy to serve
  112. 112. How CSS works main.css: @import quot;yui-fonts.cssquot;; @import quot;base.cssquot;; @import quot;layout.cssquot;; @import quot;nav.cssquot;; @import quot;forms.cssquot;; @import quot;boxes.cssquot;; @import quot;login.cssquot;; ...
  113. 113. How CSS works • Browser downloads CSS stylesheets listed in HTML • Browser processes each stylesheet for @include directives • Browser fetches each stylesheet found • This is slow.
  114. 114. How CSS works To make CSS faster, you can tell browsers to cache stylesheets ...then you have trouble if you want to update your styles Some people add datestamps to the stylesheet names: /css/mypagestyle.css?1210362379
  115. 115. How CSS::Squish works • It pretends to be a browser. • You give it the path to a stylesheet. • It returns the fully computed stylesheet the browser would get eventually. • You serve that file out to the user.
  116. 116. How we use CSS::Squish Instead of: <link rel=quot;stylesheetquot; type=quot;text/cssquot; href=quot;main.css”/> We have this: <link rel=quot;stylesheetquot; type=quot;text/cssquot; href=quot;/ css/squished/ 4a50a85044a6ba727f7aa0683ac21f7e.css”/>
  117. 117. How we use CSS::Squish If we have a cached copy of the squished CSS, name the ‘file’ as an MD5 of the content Tell browsers to cache it forever When users load a page, they will always fetch the squished CSS if it’s changed. By magic
  118. 118. What’s next for CSS::Squish Refactor the MD5 hash bits from Jifty to CSS::Squish What else do you want?
  119. 119. Ship software
  120. 120. App::ChangeLogger Gorgeous changelogs
  121. 121. We create a lot of software
  122. 122. (145 Projects in bps-public svn)
  123. 123. We’re not so good at doing releases
  124. 124. Lots of manual work
  125. 125. ShipIt can help with most of it
  126. 126. What about Changelog?
  127. 127. svn log svn://svn.foo.com/MyProj > Changelog
  128. 128. Ship it!
  129. 129. Ew. No. Ick. Wrong. Bad.
  130. 130. App::ChangeLogger
  131. 131. Two commands • sort-changelog • generate-changelog
  132. 132. Ok, three commands: • svn log --xml svn://svn.bestpractical.com/ App-Changelogger > changelog.raw.xml • ./bin/sort-changelog changelog.xml changelog.out.xml • bin/generate-changelog --generate changelog.out.xml Changelog
  133. 133. What's next for App::Changelogger? Use it on itself. (Release it) Store changelog metadata in repository Support for Git, Hg Easier customization
  134. 134. Shipwright Build. Ship.
  135. 135. The Problem CPAN
  136. 136. The Problem Everything that is not CPAN
  137. 137. We make RT
  138. 138. RT has Many Dependencies
  139. 139. Apache::DBI Digest::MD5 HTML::Scrubber Mail::Mailer Test::Warn Apache::Request Digest::base HTML::TokeParser Module::Refresh Text::ParseWords Apache::Session File::Find HTML::TreeBuilder Module::Versions::Report Text::Quoted CGI::Cookie File::Glob HTTP::Request::Common Net::SMTP Text::Template CGI::Fast File::ShareDir HTTP::Server::Simple PerlIO::eol Text::WikiFormat CGI::SpeedyCGI File::Spec HTTP::Server::Simple::Mason Pod::Usage Text::Wrapper CSS::Squish File::Temp IPC::Run Regexp::Common Time::HiRes Cache::Simple::TimedExpiry GD::Graph IPC::Run::SafeHandles Scalar::Util Time::ParseDate Calendar::Simple GD::Text LWP::UserAgent String::ShellQuote Tree::Simple Class::ReturnValue Getopt::Long Locale::Maketext Term::ReadKey UNIVERSAL::require DBD::Oracle GnuPG::Interface Locale::Maketext::Fuzzy Term::ReadLine WWW::Mechanize DBD::Pg HTML::Entities Locale::Maketext::Lexicon Test::Builder XML::RSS DBD::SQLite HTML::Form Log::Dispatch Test::Deep XML::Simple DBD::mysql HTML::FormatText Log::Dispatch::Perl Test::Expect DBIx::SearchBuilder HTML::Mason MIME::Entity Test::HTTP::Server::Simple Data::ICal HTML::RewriteAttributes MIME::Types Test::WWW::Mechanize
  140. 140. 116 CPAN distributions once you do the recursion
  141. 141. And then there are the external libraries and programs
  142. 142. • perl • readline • expat • zlib • fontconfig • gnupg • freetype • mysql • libgd • postgresql • libjpeg • libpng
  143. 143. Plagger beats RT. (It has 135 CPAN deps)
  144. 144. RT should be easy to install
  145. 145. All software should be easy to install
  146. 146. What Shipwright does Tracks all dependencies in version control (As far down as you want. I skip libc.)
  147. 147. What Shipwright does Computes a full dependency order No more recursive CPAN installs
  148. 148. What Shipwright does Keeps all packages in version control No more “that site is down”
  149. 149. What Shipwright does Automates multi-package builds (No manual intervention)
  150. 150. What Shipwright does Makes installed packages relocatable (Wraps all scripts and binaries)
  151. 151. What Shipwright does Makes historical builds reproducible (No more “Oh. that API changed in 2.0”)
  152. 152. Shipwright and CPAN CPAN-friendly (Not CPAN-only) CPAN dist import Automatic CPAN dependency resolution Full build ordering No more CPAN dependency hell.
  153. 153. Shipwright and C Native support for autoconfiscated packages No support for magic C->C dependency resolutions
  154. 154. Shipwright and Ruby Not Ruby friendly :( (not Ruby unfriendly) No Gem support yet Hackathon this weekend?
  155. 155. What you can ship A platform-neutral source distribution Platform-specific binary distributions
  156. 156. Making a Shipwright Vessel svnadmin create file:///Users/jesse/shiprepo shipwright create --repository svn:file:///Users/jesse/shiprepo/myproject shipwright import --repository svn:file:///Users/jesse/shiprepo/myproject cpan:Plagger Go get a cup of coffee (Shipwright will chase all 135 dependencies)
  157. 157. Building a Shipwright Vessel svn co file:///Users/jesse/shiprepo/myproject cd myproject ./bin/shipwright-builder (Possibly more coffee) tar czvf myproject.tgz /path/to/build
  158. 158. Using a Shipwright Vessel • tar xzvf myproject.tgz c /usr/local/ myproject • ln -s /usr/local/myproject/bin/myprogram /usr/local/bin
  159. 159. What's next for Shipwright? Build tool usability improvements Ruby support?
  160. 160. Get input from users
  161. 161. The Feedback Box Let users make suggestions everywhere
  162. 162. mailto: links?
  163. 163. getsatisfaction.com?
  164. 164. I give up.
  165. 165. Don't make your users give up.
  166. 166. Feedback is important!
  167. 167. Feedback can be easy!
  168. 168. If you make it easy for users to give you feedback...
  169. 169. You can capture lots of good information!
  170. 170. From: Jesse Vincent / Feedback <comment-373709-destygrypuju@tasks.hiveminder.com> Date: Mon, 12 May 2008 12:31:49 -0700 Subject: Up for grabs: I'd really like Hiveminder to do all my work for me! (#EFYG) Jesse Vincent <jesse@bestpractical.com> created a task and put it up for grabs: http://task.hm/EFYG I'd really like Hiveminder to do all my work for me! -- Private debugging information: Is a Pro user! HTTP_ACCEPT: application/xml, text/xml, */* HTTP_ACCEPT_ENCODING: gzip, deflate HTTP_ACCEPT_LANGUAGE: en-us HTTP_CONNECTION: keep-alive HTTP_COOKIE: JIFTY_SID_HIVEMINDER=b58e123456719fd411bb9f3a8123458f; HTTP_HOST: hiveminder.com HTTP_REFERER: http://hiveminder.com/todo/ HTTP_USER_AGENT: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_2; en-us) AppleWebKit/526.5+ (KHTML, like Gecko) Version/3.1.1 Safari/525.18 HTTP_X_REQUESTED_WITH: XMLHttpRequest REMOTE_ADDR: 75.147.59.54 REMOTE_PORT: 35412 REQUEST_METHOD: POST REQUEST_URI: /todo/
  171. 171. Collect lots of feedback
  172. 172. Reply to lots of feedback
  173. 173. Conversations make users happy
  174. 174. Happy = Loyal
  175. 175. Own the inbox
  176. 176. Net::IMAP::Server An IMAP Server in Pure Perl
  177. 177. Why?
  178. 178. Offline Access
  179. 179. iPhone Access
  180. 180. Desktop Access
  181. 181. Really, we just like things that are sick and wrong.
  182. 182. Now Hiveminder users can take their tasks with them
  183. 183. Update your tasks while offline
  184. 184. What sort of mailbox do you use with that?
  185. 185. How about Perl?
  186. 186. Net::IMAP::Server
  187. 187. Complete RFC 3501 Implementation
  188. 188. Write your own backends
  189. 189. Let’s write one now
  190. 190. How about a Spam server?
  191. 191. package Demo::IMAP::Auth; use base 'Net::IMAP::Server::DefaultAuth'; sub auth_plain { my ($self, $user, $pass ) = @_; # XXX DO AUTH CHECK return 1; }
  192. 192. package Demo::IMAP::Model; use base 'Net::IMAP::Server::DefaultModel'; sub init { my $self = shift; $self->root(Demo::IMAP::Mailbox->new()); $self->root->add_child( name => quot;INBOXquot;, is_inbox => 1); }
  193. 193. package Demo::IMAP::Mailbox; use base qw/Net::IMAP::Server::Mailbox/; my $msg = Net::IMAP::Server::Message->new(<<'EOM'); From: jesse@example.com To: user@example.com Subject: This is a test message! Hello. I am executive assistant to the director of Bear Stearns, a failed investment Bank. I have access to USD5,000,000. ... EOM sub load_data { my $self = shift; $self->add_message($msg); }
  194. 194. #!/usr/bin/perl use Net::IMAP::Server; Net::IMAP::Server->new( auth_class => quot;Demo::IMAP::Authquot;, model_class => quot;Demo::IMAP::Modelquot;, user => 'nobody', port => 143, ssl_port => 993 )->run();
  195. 195. It runs! Ship it! (Just add SSL certificates)
  196. 196. with.hm Assign tasks by email
  197. 197. We wanted to make it easy to work with Hiveminder by email.
  198. 198. We had a few ideas...
  199. 199. Cc:
  200. 200. Too clumsy
  201. 201. Bcc:
  202. 202. Too likely to break
  203. 203. Commands in the body
  204. 204. Too ugly
  205. 205. hm.
  206. 206. What could we do?
  207. 207. DNS Hacks! Everybody loves DNS hacks
  208. 208. .hm
  209. 209. Hiveminder!
  210. 210. .hm
  211. 211. Heard and McDonald Islands!
  212. 212. with.hm
  213. 213. Send your email with Hiveminder
  214. 214. Wildcard MX Record!
  215. 215. *.with.hm ➡ mail.hiveminder.com
  216. 216. An Example: From: jesse@bestpractical.com To: miyagawa@bulknews.net.with.hm Subject: Could you release a new version of plagger?
  217. 217. My mail server looks up bulknews.net.with.hm
  218. 218. “Oh. That's handled by mail.hiveminder.com”
  219. 219. mail.hiveminder.com unwraps the address.
  220. 220. Hiveminder creates a task for miyagawa@bulknews.net
  221. 221. Hiveminder emails miyagawa@bulknews.net
  222. 222. Preventing Spam
  223. 223. miyagawa@bulknews.net.my- secret.with.hm
  224. 224. I bet you can come up with even scarier uses of wildcard DNS.
  225. 225. Thank you!

×