• Save
Ball Of Mud Yapc 2008
Upcoming SlideShare
Loading in...5
×
 

Ball Of Mud Yapc 2008

on

  • 4,849 views

Or, what to do when your organically grown Perl application grows too big for its (and your) own good.

Or, what to do when your organically grown Perl application grows too big for its (and your) own good.

Statistics

Views

Total Views
4,849
Views on SlideShare
4,829
Embed Views
20

Actions

Likes
3
Downloads
0
Comments
0

4 Embeds 20

http://paper.li 9
http://www.slideshare.net 7
http://115.112.206.131 2
http://www.techgig.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Ball Of Mud Yapc 2008 Ball Of Mud Yapc 2008 Presentation Transcript

  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Handling the Ball of Mud Or, what to do when your organically grown Perl application grows too big for its (and your) own good phred@redhotpenguin.com Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 1
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Who the heck am I? • Hacking Perl for about 8 years, avid mod_perl hacker • Mechanical and Computer Engineering background • Worked for startups as well as big corporations • I've created balls of mud • I've worked on balls of mud Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 2
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud What is a Ball of Mud? • A system that has no distinguishable architecture • http://en.wikipedia.org/wiki/Big_ball_of_mud • Common in large, rapidly developed applications • Cost, Quality, Time - pick two (or Fast, Good, and Cheap) • Cost (scalar(@engineers)) is fixed • Time is usually fixed (needs to be done right now!) • So Quality must give • Systems are complex in terms of business logic, not necessarily in LoC Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 3
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud The Startup • Startups are great places for creating balls of mud • Your development team (two to three engineers if you are lucky) can try out new ideas fast • You have to adapt to rapidly changing business requirements • If you don't adapt, your competitors will destroy you, or you will run out of money • Once ideas start sticking, the application starts to generate revenue, great for the business Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 4
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud The Tools Group of an Organization • Usually one experienced Perl developer and a number of other technical specialists • You're the expert in Perl • But you're not the only one writing the code • When there's a problem though, you're the one that gets called • You spend a lot of time maintaining code Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 5
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Evolution of the Ball of Mud • Development is driven by organizational needs • Not always enough information or time to make perfect decisions • Everything has a purpose at some point in the codebase lifetime • The Perfect is the Enemy of the Good Enough • When a piece of code is written that gets the job done, it is usually shipped to production Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 6
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Rapid, Scalable Prototyping • Perl allows product management, a technical specialist, or a programmer with an idea to write a low risk proof of concept • Rarely do you have time to architect a perfect solution • One of Perl's greatest strengths is that you can generate prototypes very quickly • Another strength is those prototypes can be optimized to perform without needing to be replaced by a C program Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 7
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Hey look, it works great! • So you've got a large codebase • It’s ugly, but it works, and it's Mission Critical • And forward development is expected to continue at a steady pace, or increase • "Why do we need to do this refactoring thing, everything works just fine?" • An analogy here is that your code needs maintenance like cars need oil changes, or airplanes need regular maintenance Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 8
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Five Alarm Codebase • The maintenance cost of the app is a visible concern (software fire drills) • When something breaks it needs to be fixed fast • The test suite doesn't pass completely anymore (if one exists) • And you have to also focus on scaling the system since you are experiencing success • You look at the code you wrote six months ago and shake your head Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 9
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud High Maintenance • Congratulations, you now have a ball of mud! • The good news is that it generates revenue, usually a lot of revenue • Or it makes your team much needed by the organization • The bad news is that it's higher maintenance than Paris Hilton • The other bad news is that when new developers touch it, they break it (you break it occasionally too) Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 10
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Here Comes Downtime! • How much does downtime cost you? • You have dozens or hundreds of servers running this code • You push out a code change, and something goes wrong • Assuming a 100 person company with operational costs roughly one million dollars per month • So one hour of downtime costs 1.000.000/30/8, about $4.200 • The real cost is much higher though in terms of morale and lasting impact Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 11
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Getting Your Hands Dirty • Now YOU have to work on this thing • Test the code before you touch it • No test for the use case? Write one! • "It's too hard to test!" (everyone has a unique part of their app that makes testing difficult) • Just step up and make an attempt at testing it Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 12
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Stay within the Lines • Don't re-architect yet • Follow existing design patterns • Hard to test user facing functionality, easy to test a module API • Big problems are hard to solve, small problems are easy, so look at your system as lots of small problems instead of one big one • Write lots of small tests as opposed to a few big tests Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 13
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Frameworks are not Silver Bullets... • "Let's port everything to a framework!” • Be careful of this approach, it is more difficult than you think • Business logic can run deeper than is obvious in the code • Popular approach with managers who don’t see the technical depth • Only a good approach when you have unbelievable test coverage • Ensure that framework fits your application design patterns Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 14
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Refactoring is your Friend • Paying off technical debt • Refactoring can be your single biggest ally • Management asks "What business value do we gain here?" • The value proposition is not so much what you gain but what you lose - maintenance costs drop Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 15
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud The N-thousand line program • awstats.pl • Very difficult to figure out if it is doing the right thing • Hard to automate, but not impossible • You can't just move all the subroutines into modules without breaking something • So setup scaffolding around the program Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 16
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud The N-thousand line program use Test::More tests => 3; my $class = 'program'; require_ok("$class.pl"); import $class qw( addition should_die ); no strict 'refs'; ${"main::var"} = 5; # override globals my $method = 'addition'; cmp_ok($method->(1,1), '==', 2, '1+1 returns 2'); $method = 'should_die'; eval { $method->() }; like($@, qr/oops/i, 'should_die() died'); Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 17
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud The N-thousand line program • With a test in place you can port the subroutines into modules, but watch out for globals! • program.pl should refactor down to a command line interface that wraps a module • Look at qpsmtpd for a good example (http://wiki.qpsmtpd.org) use strict; use Getopts::Long; # process arguments into %args use My::Module; my $runner = My::Module->new(%args); $runner->run(); Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 18
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud The test suite that runs forever • As the codebase grows fast, you try to keep up with testing • More tests equals more time to run the test suite before checkins • At some point, you hit a piece of code that's hard to test so you write a bad test • Or you make a small change and don't run the entire test suite • The test suite is a problem now, instead of a solution Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 19
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Testing large systems is difficult • The core problem is too big to handle well • Break it up into small manageable pieces • CPAN is a great example of this approach • Trying to fix a really big testing suite while continuing forward feature development is tough • The CPAN approach has proven scalable ( more than 10k modules at last count) Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 20
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud MyPAN • Identify parts of your code that are loosely coupled • My::Log and My::Config are good starting points • 'h2xs -X' is your friend, start simple • If you can split your one big app into a few medium sized apps, the test suites for each become much more manageable • Identify entrenched design patterns in your code and create modules from those Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 21
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud MyPAN Most large applications contain the following components in some form: • One or more web based applications (mod_perl, etc.) • Standalone job processing daemons • A database abstraction layer • Libraries which are extensions to C code In the early days of the application, hard to see the lines between different sections because you are focused on features Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 22
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud MyPAN • Use your OS's packaging system to deploy code releases • ppm, rpm, ports, ebuilds, etc... • Check out Ovid on CPAN for rpm generation • Creating your spec file, ebuild, etc. is part of development • Most shops use a version control system for deployment • That works for a few servers, but not for a dozens or hundreds • No need to push out a new release with 100k lines of code for a change to one job processing daemon Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 23
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud My own Ball of Mud • Started out with one codebase named SL 1. Web application 2. Special purpose standalone app 3. Reporting daemons 4. cron based utilities 5. external program interfaces • Quickly became too big to manage, too tightly coupled • Intertwined tests made deployment and development difficult Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 24
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud My own Ball of Mud 1. SL => SL && SL::Config 2. && SL::Model 3. && SL::App 4. && SL::Cache 5. && RHP::Timer 6. && SL::RateLimit • Currently 14 modules • New modules from scratch or emerge from SL • Unit tests are easy to maintain and always pass before release • More maintenance work, but quality is worth it • Happy customers don’t experience broken software Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 25
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Code Compliance • Code reviews don't always work • You are the Perl expert, so be humble • What's easy for you might be hard for others • Focus on stabilizing the code base through test writing and you'll win people over • Bring Perl::Tidy and Perl::Critic into the codebase slowly, starting with your own code Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 26
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Code Narcissism • Narcissus • (när-sĭs'əs) pronunciation • n. Greek Mythology. • A young man who pined away in love for his own image in a pool of water and was transformed into the flower that bears his name. • “The Narcissism of Small Code Differences” • http://weblog.raganwald.com/2008/05/narcissism-of-small- code-differences.html • Be detail oriented, but don’t anthropomorphize your code Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 27
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Tools for Success • Version control - use a system that preserves code change history across file moves (CVS is not one of them) • Persistence testing environment - the worst failures occur going from QA to production, not dev to QA • Coffee, beer, food, cigarettes - foster good working relationships with other team members Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 28
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Tools::Bugs • Use one and only one bug tracking system • Use a system designed for software development • Watch out for non-technical managers offering substitutes • Watch out for managers requiring multiple systems for the same issues ( doubles your workload ) Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 29
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Tools::Bugs::Bugzilla • Mature and time tested • Ugly, but can be skinned • Kitchen sink UI approach, but workflow built for bug tracking • Third party tools for source code management integration • Designed for software development, not general issue tracking • Written in Perl Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 30
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Tools::Bugs::Trac • Another great bug tracker • Less mature than Bugzilla, but widely adopted • Not near as ugly as Bugzilla • SCM integration • Popular with developers Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 31
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Tools::Bugs::RT • A helpdesk and issue tracking system • Not a software bug tracking system • Can be used as a bug tracking system with customization • Used by CPAN • Has been known to be slow and hard to use • Good command line interfaces • Written in Perl Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 32
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Tools::Bugs::Basecamp • Project management hosted application • Not a software bug tracking system • Managers love it! Programmers aren’t so fond • Easy to deploy • Use a real bug tracking system instead • Avoid “doubling up” with a bug tracker, twice as much work Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 33
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Tools::Bugs::FogBugz • A bug tracking / project management system • SCM integration • Don’t look at the source or you will go blind • Goes offline when the licensing server goes down • Not extensible • Managers love it - but you should avoid it Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 34
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Tools::Testing::Smolder • Must have for multiple developer systems • Uses Buildbot to find bugs before you do • Continuous testing ensures that the tests pass when you svn update • Written in Perl - see it in action at the Smolder talk Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 35
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Tools::Testing::Smolder • Small agile teams need sharp tools • Small teams should run their own infrastructure • Managers may ask “Why can’t we use the smoker the 100 dev Java team is using?” • “Boss, would you make sushi with a butter knife?” • “You mean the one that crashes twice a day?” • “Because we need it setup before next year” Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 36
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Emotional Attachments • Programming teams are complex social dynamic systems • It's natural for code authors to become emotionally attached to their code • How do you explain to your coworker who has a Ph.D. in Astrophysics that you had to add 'use strict;' to his code when you fixed a bug the week he was out? • Better yet, how to you get him to add strictures to his code in the future? Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 37
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Egos • Promote ego-less programming practices • Lead by example, not by persuasion • If someone writes some code that looks dodgy, write a test for it rather than touching their code right away • When you find a bug, send the list an email with a diff, and an example of the test passing under your change Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 38
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud No-Win Scenarios 1. You add a feature and it works great 2. It exposes a previous bug 3. Guess who gets the blame for it? 1. You save the world when the system melts down on New Years Eve at 11 pm 2. Your S.O. is not pleased that you got pulled away to work 3. When you get back into work, your solution that solved the emergency is criticized, and you are told to be more professional in such situations Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 39
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Co-Workers • Authors of the ball of mud • Usually smart and good coders (but don’t always use strict;) • “Don’t touch that piece of code!” • Sometimes they are right, sometimes they are wrong, just like everybody else • Approach with respect, but don’t tolerate being bullied Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 40
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Managers • If they have never been a coder, be conscious of that • Usually have their friends they want to hire • Always have an agenda of their own (as should you) • Remember this person is your boss • Maintain a professional relationship Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 41
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Technical Managers • Have been coders, but got promoted into management • “Here is this problem I need you to solve. You should solve it this way” • Now you have two problems to solve • Can be a great resource in many situations • Good ones can make a world class organization Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 42
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Conclusion: Don’t Worry • Don’t try to change the world in a day • Don’t get fired over whitespace • Do be a positive contributor • Lose some battles, win the war • Admit to your mistakes • use strict; Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 43
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Credits • YAPC::NA 2008 Staff • Those who had to suffer through the balls of mud I created • Those who created balls of mud which I suffered through and gained wisdom • The great programmers I've had a chance to work with and who have shown me what writing good code is all about • Red Hot Penguin Consulting LLC and Silver Lining Networks Inc. for sponsorship of this talk Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 44
  • YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud Shameless Plug Need mod_perl / Perl / PostgreSQL consulting? fred@redhotpenguin.com just another mod_perl hacker Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008 Monday, June 16, 2008 45