1. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Handling the Ball of Mud
Or, what to do when your organically grown Perl
application grows too big for its (and your) own good
phred@redhotpenguin.com
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 1
2. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Who the heck am I?
• Hacking Perl for about 8 years, avid mod_perl hacker
• Mechanical and Computer Engineering background
• Worked for startups as well as big corporations
• I've created balls of mud
• I've worked on balls of mud
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 2
3. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
What is a Ball of Mud?
• A system that has no distinguishable architecture
• http://en.wikipedia.org/wiki/Big_ball_of_mud
• Common in large, rapidly developed applications
• Cost, Quality, Time - pick two (or Fast, Good, and Cheap)
• Cost (scalar(@engineers)) is fixed
• Time is usually fixed (needs to be done right now!)
• So Quality must give
• Systems are complex in terms of business logic, not
necessarily in LoC
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 3
4. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
The Startup
• Startups are great places for creating balls of mud
• Your development team (two to three engineers if you are
lucky) can try out new ideas fast
• You have to adapt to rapidly changing business requirements
• If you don't adapt, your competitors will destroy you, or
you will run out of money
• Once ideas start sticking, the application starts to generate
revenue, great for the business
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 4
5. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
The Tools Group of an
Organization
• Usually one experienced Perl developer and a number of other
technical specialists
• You're the expert in Perl
• But you're not the only one writing the code
• When there's a problem though, you're the one that gets called
• You spend a lot of time maintaining code
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 5
6. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Evolution of the
Ball of Mud
• Development is driven by organizational needs
• Not always enough information or time to
make perfect decisions
• Everything has a purpose at some point in the
codebase lifetime
• The Perfect is the Enemy of the Good Enough
• When a piece of code is written that gets the
job done, it is usually shipped to production
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 6
7. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Rapid, Scalable
Prototyping
• Perl allows product management, a technical specialist, or a
programmer with an idea to write a low risk proof of concept
• Rarely do you have time to architect a perfect solution
• One of Perl's greatest strengths is that you can generate
prototypes very quickly
• Another strength is those prototypes can be optimized to
perform without needing to be replaced by a C program
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 7
8. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Hey look, it
works great!
• So you've got a large codebase
• It’s ugly, but it works, and it's Mission Critical
• And forward development is expected to continue at a
steady pace, or increase
• "Why do we need to do this refactoring thing, everything
works just fine?"
• An analogy here is that your code needs maintenance like
cars need oil changes, or airplanes need regular maintenance
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 8
9. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Five Alarm
Codebase
• The maintenance cost of the app is a visible
concern (software fire drills)
• When something breaks it needs to be fixed fast
• The test suite doesn't pass completely anymore
(if one exists)
• And you have to also focus on scaling the system
since you are experiencing success
• You look at the code you wrote six months ago
and shake your head
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 9
10. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
High
Maintenance
• Congratulations, you now have a ball of mud!
• The good news is that it generates revenue,
usually a lot of revenue
• Or it makes your team much needed by the
organization
• The bad news is that it's higher maintenance
than Paris Hilton
• The other bad news is that when new
developers touch it, they break it (you break it
occasionally too)
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 10
11. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Here Comes
Downtime!
• How much does downtime cost you?
• You have dozens or hundreds of servers running this code
• You push out a code change, and something goes wrong
• Assuming a 100 person company with operational costs
roughly one million dollars per month
• So one hour of downtime costs 1.000.000/30/8, about
$4.200
• The real cost is much higher though in terms of morale and
lasting impact
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 11
12. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Getting Your
Hands Dirty
• Now YOU have to work on this thing
• Test the code before you touch it
• No test for the use case? Write one!
• "It's too hard to test!" (everyone has a unique part
of their app that makes testing difficult)
• Just step up and make an attempt at testing it
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 12
13. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Stay within
the Lines
• Don't re-architect yet
• Follow existing design patterns
• Hard to test user facing functionality, easy to test a
module API
• Big problems are hard to solve, small problems are easy,
so look at your system as lots of small problems instead of
one big one
• Write lots of small tests as opposed to a few big tests
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 13
14. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Frameworks are
not Silver Bullets...
• "Let's port everything to a framework!”
• Be careful of this approach, it is more difficult than you think
• Business logic can run deeper than is obvious in the code
• Popular approach with managers who don’t see the technical depth
• Only a good approach when you have unbelievable test coverage
• Ensure that framework fits your application design patterns
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 14
15. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Refactoring is
your Friend
• Paying off technical debt
• Refactoring can be your single biggest ally
• Management asks "What business value do we gain here?"
• The value proposition is not so much what you gain but what
you lose - maintenance costs drop
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 15
16. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
The N-thousand
line program
• awstats.pl
• Very difficult to figure out if it is doing
the right thing
• Hard to automate, but not impossible
• You can't just move all the subroutines
into modules without breaking
something
• So setup scaffolding around the
program
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 16
17. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
The N-thousand line program
use Test::More tests => 3;
my $class = 'program';
require_ok("$class.pl");
import $class qw( addition should_die );
no strict 'refs';
${"main::var"} = 5; # override globals
my $method = 'addition';
cmp_ok($method->(1,1), '==', 2, '1+1 returns 2');
$method = 'should_die';
eval { $method->() };
like($@, qr/oops/i, 'should_die() died');
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 17
18. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
The N-thousand line program
• With a test in place you can port the subroutines into modules,
but watch out for globals!
• program.pl should refactor down to a command line interface
that wraps a module
• Look at qpsmtpd for a good example (http://wiki.qpsmtpd.org)
use strict;
use Getopts::Long;
# process arguments into %args
use My::Module;
my $runner = My::Module->new(%args);
$runner->run();
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 18
19. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
The test suite
that runs forever
• As the codebase grows fast, you try to keep up with testing
• More tests equals more time to run the test suite before checkins
• At some point, you hit a piece of code that's hard to test so you
write a bad test
• Or you make a small change and don't run the entire test suite
• The test suite is a problem now, instead of a solution
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 19
20. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Testing large
systems is difficult
• The core problem is too big to handle well
• Break it up into small manageable pieces
• CPAN is a great example of this approach
• Trying to fix a really big testing suite while
continuing forward feature development is tough
• The CPAN approach has proven scalable
( more than 10k modules at last count)
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 20
21. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
MyPAN
• Identify parts of your code that are loosely coupled
• My::Log and My::Config are good starting points
• 'h2xs -X' is your friend, start simple
• If you can split your one big app into a few medium sized
apps, the test suites for each become much more manageable
• Identify entrenched design patterns in your code and
create modules from those
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 21
22. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
MyPAN
Most large applications contain the following components in
some form:
• One or more web based applications (mod_perl, etc.)
• Standalone job processing daemons
• A database abstraction layer
• Libraries which are extensions to C code
In the early days of the application, hard to see the lines
between different sections because you are focused on features
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 22
23. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
MyPAN
• Use your OS's packaging system to deploy code releases
• ppm, rpm, ports, ebuilds, etc...
• Check out Ovid on CPAN for rpm generation
• Creating your spec file, ebuild, etc. is part of development
• Most shops use a version control system for deployment
• That works for a few servers, but not for a dozens or hundreds
• No need to push out a new release with 100k lines of code for a
change to one job processing daemon
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 23
24. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
My own Ball of Mud
• Started out with one codebase named SL
1. Web application
2. Special purpose standalone app
3. Reporting daemons
4. cron based utilities
5. external program interfaces
• Quickly became too big to manage, too tightly coupled
• Intertwined tests made deployment and development difficult
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 24
25. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
My own Ball of Mud
1. SL => SL && SL::Config
2. && SL::Model
3. && SL::App
4. && SL::Cache
5. && RHP::Timer
6. && SL::RateLimit
• Currently 14 modules
• New modules from scratch or emerge from SL
• Unit tests are easy to maintain and always pass before release
• More maintenance work, but quality is worth it
• Happy customers don’t experience broken software
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 25
26. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Code Compliance
• Code reviews don't always work
• You are the Perl expert, so be humble
• What's easy for you might be hard for others
• Focus on stabilizing the code base through
test writing and you'll win people over
• Bring Perl::Tidy and Perl::Critic into the
codebase slowly, starting with your own code
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 26
27. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Code Narcissism
• Narcissus
• (när-sĭs'əs) pronunciation
• n. Greek Mythology.
• A young man who pined away in love for his own image in a
pool of water and was transformed into the flower that bears
his name.
• “The Narcissism of Small Code Differences”
• http://weblog.raganwald.com/2008/05/narcissism-of-small-
code-differences.html
• Be detail oriented, but don’t anthropomorphize your code
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 27
28. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Tools for Success
• Version control - use a system that preserves code
change history across file moves (CVS is not one of them)
• Persistence testing environment - the worst failures
occur going from QA to production, not dev to QA
• Coffee, beer, food, cigarettes - foster good working
relationships with other team members
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 28
29. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Tools::Bugs
• Use one and only one bug tracking system
• Use a system designed for software development
• Watch out for non-technical managers offering substitutes
• Watch out for managers requiring multiple systems for the
same issues ( doubles your workload )
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 29
30. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Tools::Bugs::Bugzilla
• Mature and time tested
• Ugly, but can be skinned
• Kitchen sink UI approach, but workflow built for bug tracking
• Third party tools for source code management integration
• Designed for software development, not general issue tracking
• Written in Perl
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 30
31. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Tools::Bugs::Trac
• Another great bug tracker
• Less mature than Bugzilla, but widely adopted
• Not near as ugly as Bugzilla
• SCM integration
• Popular with developers
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 31
32. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Tools::Bugs::RT
• A helpdesk and issue tracking system
• Not a software bug tracking system
• Can be used as a bug tracking system with customization
• Used by CPAN
• Has been known to be slow and hard to use
• Good command line interfaces
• Written in Perl
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 32
33. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Tools::Bugs::Basecamp
• Project management hosted application
• Not a software bug tracking system
• Managers love it! Programmers aren’t so fond
• Easy to deploy
• Use a real bug tracking system instead
• Avoid “doubling up” with a bug tracker, twice as much work
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 33
34. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Tools::Bugs::FogBugz
• A bug tracking / project management system
• SCM integration
• Don’t look at the source or you will go blind
• Goes offline when the licensing server goes down
• Not extensible
• Managers love it - but you should avoid it
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 34
35. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Tools::Testing::Smolder
• Must have for multiple developer systems
• Uses Buildbot to find bugs before you do
• Continuous testing ensures that the tests pass
when you svn update
• Written in Perl - see it in action at the Smolder
talk
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 35
36. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Tools::Testing::Smolder
• Small agile teams need sharp tools
• Small teams should run their own infrastructure
• Managers may ask “Why can’t we use the
smoker the 100 dev Java team is using?”
• “Boss, would you make sushi with a butter
knife?”
• “You mean the one that crashes twice a day?”
• “Because we need it setup before next year”
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 36
37. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Emotional
Attachments
• Programming teams are complex social dynamic systems
• It's natural for code authors to become emotionally
attached to their code
• How do you explain to your coworker who has a Ph.D. in
Astrophysics that you had to add 'use strict;' to his code
when you fixed a bug the week he was out?
• Better yet, how to you get him to add strictures to his
code in the future?
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 37
38. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Egos
• Promote ego-less programming
practices
• Lead by example, not by persuasion
• If someone writes some code that looks
dodgy, write a test for it rather than
touching their code right away
• When you find a bug, send the list an
email with a diff, and an example of the
test passing under your change
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 38
39. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
No-Win
Scenarios
1. You add a feature and it works great
2. It exposes a previous bug
3. Guess who gets the blame for it?
1. You save the world when the system melts down on New
Years Eve at 11 pm
2. Your S.O. is not pleased that you got pulled away to work
3. When you get back into work, your solution that solved the
emergency is criticized, and you are told to be more professional
in such situations
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 39
40. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Co-Workers
• Authors of the ball of mud
• Usually smart and good coders (but don’t always use strict;)
• “Don’t touch that piece of code!”
• Sometimes they are right, sometimes they are wrong, just
like everybody else
• Approach with respect, but don’t tolerate being bullied
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 40
41. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Managers
• If they have never been a coder, be conscious of that
• Usually have their friends they want to hire
• Always have an agenda of their own (as should you)
• Remember this person is your boss
• Maintain a professional relationship
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 41
42. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Technical
Managers
• Have been coders, but got promoted into management
• “Here is this problem I need you to solve. You should solve
it this way”
• Now you have two problems to solve
• Can be a great resource in many situations
• Good ones can make a world class organization
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 42
43. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Conclusion:
Don’t Worry
• Don’t try to change the world in a day
• Don’t get fired over whitespace
• Do be a positive contributor
• Lose some battles, win the war
• Admit to your mistakes
• use strict;
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 43
44. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Credits
• YAPC::NA 2008 Staff
• Those who had to suffer through the balls of mud I created
• Those who created balls of mud which I suffered through and
gained wisdom
• The great programmers I've had a chance to work with and who
have shown me what writing good code is all about
• Red Hot Penguin Consulting LLC and Silver Lining Networks
Inc. for sponsorship of this talk
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 44
45. YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud
Shameless Plug
Need mod_perl / Perl / PostgreSQL consulting?
fred@redhotpenguin.com
just another mod_perl hacker
Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 45