Hiveminder - Everything but the Secret Sauce

Slide 1: Everything but the Secret Sauce Contains No Jifty! Modules, Magic and Monstrosities created for

Slide 2: Jesse Vincent Best Practical

Slide 3: We make Hiveminder Hiveminder

Slide 4: It’s not opensource

Slide 5: We built lots of cool stuff for Hiveminder Hiveminder

Slide 6: We opensource everything we can

Slide 7: Today’s Agenda Ten tools and techniques to help you: Find bugs faster Build web apps Ship software Get input from users Own the Inbox

Slide 8: Find bugs faster

Slide 9: TAP-Harness-Remote Run your tests over there.

Slide 10: The Problem

Slide 11: Hiveminder has lots of tests Hiverminder

Slide 12: That's a problem?

Slide 13: Yes.

Slide 14: 30 minutes to run tests on my Macbook Macbook 30

Slide 15: (It took the same on a Macbook Pro) (Macbook Pro )

Slide 16: What to do?

Slide 17: TAP::Harness 3.0

Slide 18: I have two cores! 2 !

Slide 19: TAP::Harness::Parallel

Slide 20: prove -j 5 --fork

Slide 21: 22 minutes prove -j 5 22

Slide 22: Better! !

Slide 23: Not good.

Slide 24: Run tests on a server! !

Slide 25: rsync -rvp . server: ssh server \"prove -j 9 t\"

Slide 26: Better.

Slide 27: Annoying to run.

Slide 28: TAP::Harness::Remote

Slide 29: Magic Remote Testing

Slide 30: # ~/.remote_test --- local: /path/to/local/testing/root/ host: - remote1.testing.host.example.com - remote2.testing.host.example.com root: /where/to/place/local/root/on/remote/ user: username master: 1 perl: /usr/bin/perl ssh: /usr/bin/ssh ssh_args: - -x - -S - '~/.ssh/master-%r@%h:%p'

Slide 31: prove --harness TAP::Harness::Remote t/*.t

Slide 32: 16.5 minutes 16.5 !

Slide 33: prove --harness TAP::Harness::Remote -j 5 t/*.t

Slide 34: Magic Parallel Remote Testing

Slide 35: 12 minutes! 12 !

Slide 36: Not good enough for TDD

Slide 37: What I need is... • Fast Servers ( ) • Lots of Fast Servers ( ) • Right Now ( ) • Not Very Often ( )

Slide 38: A use for cloud computing! !

Slide 39: TAP::Harness::Remote::EC2

Slide 40: 4 hosts. -j 5

Slide 41: 89 seconds 89

Slide 42: Yatta!

Slide 43: Yatta!

Slide 44: EC2 Caveats Default Perl builds are really bad Perl Machines are mostly underpowered (“XL” instances are great, though) XL Machine Setup/Teardown is ~manual EC2

Slide 45: What's next for ::Harness::Remote::* ? More polish on EC2 tools EC2 Better EC2 instance control EC2 More widespread usage

Slide 46: Carp::REPL It's not quite a debugger

Slide 47: How many of you use the Perl debugger? Perl

Slide 48: How many of you know how to use the perl debugger?

Slide 49: I don’t, either.

Slide 50: So, what do I use?

Slide 51: 31b:~ jesse$ perl /tmp/contrived.pl Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 20. 31b:~ jesse$ vim /tmp/contrived.pl 18.sub c { 19. my $to_call = shift; 20. $to_call = \"does_\".$to_call; 21. no strict refs; 22. $to_call->(); 23.}

Slide 52: 31b:~ jesse$ perl /tmp/contrived.pl Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 22. 31b:~ jesse$ vim /tmp/contrived.pl 18.sub c { 19. my $to_call = shift; 20. $to_call = \"does_\".$to_call; 21. no strict refs; 22. use Carp; Carp::cluck(\"$to_call\"); 23. $to_call->(); 24.}

Slide 53: Ick. A Print Statement. Print

Slide 54: Anyone else here use print statement debugging? print

Slide 55: (Oh god, I hope I'm not alone)

Slide 56: Acme, you wrote Devel::ebug... Leon Brocard Devel::ebug …

Slide 57: Do you use it?

Slide 58: I want a simple debugger that's easier than print statements print

Slide 59: sartak created Carp::REPL Shawn Moore Carp::REPL

Slide 60: A lightweight, modern, interactive debugger

Slide 61: Built on Devel::REPL Devel::REPL

Slide 62: Carp::REPL Hooks into $SIG{__DIE__} Carp::REPL $SIG{__DIE__}

Slide 63: $SIG{__WARN__} hooks are optional $SIG{__WARN__}

Slide 64: 31b:~ jesse$ perl -MCarp::REPL /tmp/contrived.pl Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 22. 0: Carp::REPL::repl called at /tmp/contrived.pl:22. 1: main::c called at /tmp/contrived.pl:15. 2: main::b called at /tmp/contrived.pl:9. 3: main::a called at /tmp/contrived.pl:27. Now at /tmp/contrived.pl:22 (frame 0). $

Slide 65: That's not a shell prompt

Slide 66: 31b:~ jesse$ perl -MCarp::REPL /tmp/contrived.pl Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 22. 0: Carp::REPL::repl called at /tmp/contrived.pl:22. 1: main::c called at /tmp/contrived.pl:15. 2: main::b called at /tmp/contrived.pl:9. 3: main::a called at /tmp/contrived.pl:27. Now at /tmp/contrived.pl:22 (frame 0). $ warn $0 /tmp/contrived.pl at (eval 135) line 15, <FIN> line 1.

Slide 67: It's Perl, but it's not just Perl Perl Perl

Slide 68: :t - Stack trace :u - Up one stack frame :d - Down one stack frame :e - Print this frame's lexical environment :q - Quit this REPL $_e - ARRAYREF of this frame's @_

Slide 69: $ :t 0: Carp::REPL::repl called at /tmp/contrived.pl:22. 1: main::c called at /tmp/contrived.pl:15. 2: main::b called at /tmp/contrived.pl:9. 3: main::a called at /tmp/contrived.pl:27. $

Slide 70: $ :e $Env = { \"\\$to_call\" => \\do { my $v = 'does_not_exist' } }; $ :u $ :u Now at /tmp/contrived.pl:9 (frame 2). $ :e $Env = { \"\\$to_call\" => \\do { my $v = 'exist' } }; $ :u Now at /tmp/contrived.pl:27 (frame 3). $ :e $Env = { \"\\$to_call\" => \\do { my $v = '' } }; $ :d Now at /tmp/contrived.pl:9 (frame 2). $ :e $Env = { \"\\$to_call\" => \\do { my $v = 'exist' } }; $ warn join(\", \",@$_a); exist at (eval 138) line 18, <FIN> line 4.

Slide 71: Why is my code 'warn'ing? Maybe it's in an eval. eval Or in a CPAN library. CPAN No problem! perl -MCarp::REPL=warn

Slide 72: 31b:~ jesse$ perl -MCarp::REPL=warn /tmp/contrived.pl Use of uninitialized value in concatenation (.) or string at /tmp/contrived.pl line 8. 0: Carp::REPL::repl called at /tmp/contrived.pl:8. 1: main::a called at /tmp/contrived.pl:27. Now at /tmp/contrived.pl:8 (frame 0). $ :e $Env = { \"\\$to_call\" => \\do { my $v = undef } }; $ :q Undefined subroutine &main::does_not_exist called at /tmp/contrived.pl line 22, <FIN> line 2. 0: Carp::REPL::repl called at /tmp/contrived.pl:22. 1: main::c called at /tmp/contrived.pl:15. 2: main::b called at /tmp/contrived.pl:9. 3: main::a called at /tmp/contrived.pl:27. Now at /tmp/contrived.pl:22 (frame 0). $

Slide 73: Inserting breakpoints You know your 'print' statements? print Turn them into breakpoints. Two ways: 2 : s/print/warn/ and -MCarp::REPL=warn Carp::REPL::repl

Slide 74: 18.sub c { 19. my $to_call = shift; 20. no strict 'refs'; 21. Carp::REPL::repl(); #Gimme a breakpoint here! 22. $to_call = \"does_\".$to_call; 23. $to_call->(); 24.}

Slide 75: What's next for Carp::REPL? I have no idea. It's simple. It works.

Slide 76: Build web apps

Slide 77: Template::Declare A Pure Perl Templating Engine 2007 Talk Encore Presentation!

Slide 78: Perl has two big templating camps Perl 2

Slide 79: Template::Toolkit A DSL for templates DSL It's NOT Perl Why should I learn a new language for templates?

Slide 80: HTML::Mason There's Perl inside. Perl

Slide 81: HTML::Mason <mason><is><still> <full><of> <angle><brackets> < >< >< >< >< >< >

Slide 82: HTML::Mason </brackets></angle> </of></full> </still></is></mason> </ ></ ></ ></ ></ ></ >

Slide 83: I'm Just A Simple Perl Hacker Perl

Slide 84: I needed something more Perlish Perl

Slide 85: template '/pages/mypage.html' => sub { html { head {}; body { h1 {'Hey, this is text'}; } } };

Slide 86: template choices => page { h1 { 'My Choices' } dl { my $choices = Doxory::Model::ChoiceCollection->new; $choices->limit( column => 'asked_by', value => Jifty->web->current_user->id, ); while (my $c = $choices->next) { dt { $c->name, ' (', $c->asked_by->name, ')' } dd { b { $c->a } em { 'vs' } b { $c->b } } } } };

Slide 87: But! Content! Templates! ! ! Design! Code! ! ! OMGWTFBBQ!? !? THAT'S WRONG! !

Slide 88: The person who told you it's wrong was lying to you.

Slide 89: We're Perl hackers Perl

Slide 90: Why are we writing our templates in another language?

Slide 91: This is not 1997 1997

Slide 92: It’s 2008 2008

Slide 93: People use CSS for design now CSS

Slide 94: Programmers still have to make templates

Slide 95: Templates run like CODE

Slide 96: Because they ARE code

Slide 97: Let's use our CODING tools to work with them

Slide 98: How about Refactoring?

Slide 99: Have you ever tried to refactor HTML? HTML

Slide 100: template 'mypage.html' => page { h1 { 'Two choices' }; div { attr { class => 'item' }; h2 { 'Item 1'}; }; div { attr { class => 'item' }; h2 { 'Item 2'}; }; };

Slide 101: template 'mypage.html' => page { h1 { 'Two choices' }; for (\"Item 1\", \"Item 2\") { item($_); } }; sub item { my $content = shift; div { attr { class => 'item' }; h2 {$content}; }; }

Slide 102: We can refactor templates! !

Slide 103: Our HTML is magically valid HTML

Slide 104: (Syntax errors are... Syntax Errors) …

Slide 105: Object Oriented Templates

Slide 106: Subclassing It just works We're using Perl's symbol table Perl (Template::Declare's show() passes the class for you) Template::Declare show()

Slide 107: Mixins alias Some::Template::Library under '/path'; That's it.

Slide 108: Closures

Slide 109: Tags as Closures HTML tags take blocks of content HTML Our tag methods take blocks of Perl Perl They return closures when you want them They run and output their content when you want them to

Slide 110: sub h1 (&;$) { my $code = shift; ... if (defined wantarray) { return sub { ...closure around $code...}; } else { # Actually do our work, run $code and # return the output return $code->(); } }

Slide 111: What's next for Template::Declare? HTML Attribute Validation HTML Compile to .js js

Slide 112: CSS::Squish A compiler for Cascading Style Sheets CSS

Slide 113: The Problem Many small CSS Files CSS Easy to work with Painful to serve

Slide 114: The Problem One big CSS file CSS Painful to work with Easy to serve

Slide 115: How CSS works main.css: @import \"yui-fonts.css\"; @import \"base.css\"; @import \"layout.css\"; @import \"nav.css\"; @import \"forms.css\"; @import \"boxes.css\"; @import \"login.css\"; ... CSS

Slide 116: How CSS works • Browser downloads CSS stylesheets listed in HTML • Browser processes each stylesheet for @include directives • Browser fetches each stylesheet found • This is slow. CSS::Squish

Slide 117: How CSS works To make CSS faster, you can tell browsers to cache stylesheets CSS ...then you have trouble if you want to update your styles Some people add datestamps to the stylesheet names: /css/mypagestyle.css?1210362379 CSS::Squish

Slide 118: How CSS::Squish works It pretends to be a browser. You give it the path to a stylesheet. It returns the fully computed stylesheet the browser would get eventually. You serve that file out to the user. CSS::Squish

Slide 119: How we use CSS::Squish Instead of: <link rel=\"stylesheet\" type=\"text/css\" href=\"main.css”/> We have this: <link rel=\"stylesheet\" type=\"text/css\" href=\"/ css/squished/ 4a50a85044a6ba727f7aa0683ac21f7e.css”/> CSS::Squish

Slide 120: How we use CSS::Squish If we have a cached copy of the squished CSS, name the ‘file’ as an MD5 of the content CSS MD5 Tell browsers to cache it forever When users load a page, they will always fetch the squished CSS if it’s changed. By magic CSS CSS::Squish

Slide 121: What’s next for CSS::Squish Refactor the MD5 hash bits from Jifty to CSS::Squish MD5 Jifty CSS::Squish What else do you want?

Slide 122: Ship software

Slide 123: App::ChangeLogger Gorgeous changelogs

Slide 124: We create a lot of software

Slide 125: (145 Projects in bps-public svn) 145

Slide 126: We’re not so good at doing releases

Slide 127: Lots of manual work

Slide 128: ShipIt can help with most of it ShipIt

Slide 129: What about Changelog?

Slide 130: svn log svn://svn.foo.com/MyProj > Changelog

Slide 131: Ship it! ShipIt

Slide 132: Ew. No. Ick. Wrong. Bad.

Slide 133: App::ChangeLogger

Slide 134: Two commands • sort-changelog • generate-changelog 2

Slide 135: Ok, three commands: • svn log --xml svn://svn.bestpractical.com/ App-Changelogger > changelog.raw.xml • ./bin/sort-changelog changelog.xml changelog.out.xml • bin/generate-changelog --generate changelog.out.xml Changelog 3

Slide 139: What's next for App::Changelogger? Use it on itself. (Release it) Store changelog metadata in repository Support for Git, Hg Git Hg Easier customization

Slide 140: Shipwright Build. Ship.

Slide 141: The Problem CPAN

Slide 142: The Problem Everything that is not CPAN : CPAN

Slide 143: We make RT RT

Slide 144: RT has Many Dependencies RT

Slide 145: Apache::DBI Digest::MD5 HTML::Scrubber Mail::Mailer Test::Warn Apache::Request Digest::base HTML::TokeParser Module::Refresh Text::ParseWords Apache::Session File::Find HTML::TreeBuilder Module::Versions::Report Text::Quoted CGI::Cookie File::Glob HTTP::Request::Common Net::SMTP Text::Template CGI::Fast File::ShareDir HTTP::Server::Simple PerlIO::eol Text::WikiFormat CGI::SpeedyCGI File::Spec HTTP::Server::Simple::Mason Pod::Usage Text::Wrapper CSS::Squish File::Temp IPC::Run Regexp::Common Time::HiRes Cache::Simple::TimedExpiry GD::Graph IPC::Run::SafeHandles Scalar::Util Time::ParseDate Calendar::Simple GD::Text LWP::UserAgent String::ShellQuote Tree::Simple Class::ReturnValue Getopt::Long Locale::Maketext Term::ReadKey UNIVERSAL::require DBD::Oracle GnuPG::Interface Locale::Maketext::Fuzzy Term::ReadLine WWW::Mechanize DBD::Pg HTML::Entities Locale::Maketext::Lexicon Test::Builder XML::RSS DBD::SQLite HTML::Form Log::Dispatch Test::Deep XML::Simple DBD::mysql HTML::FormatText Log::Dispatch::Perl Test::Expect DBIx::SearchBuilder HTML::Mason MIME::Entity Test::HTTP::Server::Simple Data::ICal HTML::RewriteAttributes MIME::Types Test::WWW::Mechanize

Slide 146: That’s just the non-core CPAN dependencies. Without all their dependencies.

Slide 147: 116 CPAN distributions once you do the recursion 116

Slide 148: And then there are the external libraries and programs

Slide 149: • perl • readline • expat • zlib • fontconfig • gnupg • freetype • mysql • libgd • postgresql • libjpeg • libpng

Slide 150: Plagger beats RT. (It has 135 CPAN deps) Plagger (Plagger 135 )

Slide 151: RT is hard to install RT

Slide 152: RT should be easy to install

Slide 153: All software should be easy to install

Slide 154: What Shipwright does Tracks all dependencies in version control (As far down as you want. I skip libc.) libc Shipwright

Slide 155: What Shipwright does Computes a full dependency order No more recursive CPAN installs CPAN Shipwright

Slide 156: What Shipwright does Keeps all packages in version control No more “that site is down” Shipwright

Slide 157: What Shipwright does Automates multi-package builds (No manual intervention) Shipwright

Slide 158: What Shipwright does Makes installed packages relocatable (Wraps all scripts and binaries) Shipwright

Slide 159: What Shipwright does Makes historical builds reproducible (No more “Oh. that API changed in 2.0”) API 2.0 Shipwright

Slide 160: Shipwright and CPAN CPAN-friendly CPAN (Not CPAN-only) CPAN CPAN dist import CPAN Automatic CPAN dependency resolution Full build ordering No more CPAN dependency hell. Shipwright CPAN

Slide 161: Shipwright and C Native support for autoconfiscated packages No support for magic C->C dependency resolut