In the desert, without a camel
•Download as PPTX, PDF•
0 likes•821 views
This is the tale of a DevOps journey through a wasteland where Perl is a third-class citizen. In the Hadoop world, anything that doesn't run on the JVM is not exactly, um... welcome. And while I love where I work these days, Perl is looked down upon, and often considered a tool of "last resort"... However, despite the title of this talk, I have managed to sneak in quite a lot of Perl by means of a solid DevOps ethos and a solid philosophy of systems management from bare metal up through end-user applications. Turns out, Perl really *is* the best tool for the job in a *lot* of the situations I deal with from day-to-day. In this talk you will learn about the fundamentals of my approach, along with details of the tools and techniques I have employed in the administration of the data warehouse infrastructure of the world's largest travel-review company. You will also learn how to make it *much* easier to use Perl and CPAN code in places other people would consider it infeasible, and fall back to more "universal" tools like awk and bash.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to In the desert, without a camel
Similar to In the desert, without a camel (20)
Recently uploaded
Recently uploaded (20)
In the desert, without a camel
- 1. In the Desert Without a Camel A devops journey, with very little Perl Steve Scaffidi stephen@scaffidi.net irc nick: hercynium YAPC::NA June 8, 2015 Salt Lake City, Utah
- 2. The person who approved this talk has been sacked.
- 3. "It's not got much Perl in it!"
- 4. ☺
- 5. "Senior Software Engineer" Data Warehouse Team at TripAdvisor My job title...
- 6. What people think I do:
- 7. What I actually spend most of my time doing:
- 8. The Best Tool for the Job
- 9. The Best Tool for the Job Not Always
- 10. The Best Tool for the Job Not Always ...but sometimes it is.
- 11. Get on with it!
- 12. Y U NO PERL?
- 13. redhat/centos
- 14. Solutions?
- 16. curl -L http://install.perlbrew.pl | bash source ~/perl5/perlbrew/etc/bashrc perlbrew install perl-5.22.0 perlbrew use perl-5.22.0 Insanely Simple:
- 17. perlbrew install-cpanm For your convenience:
- 18. perlbrew install perl-5.12.5 perlbrew use perl-5.12.5 Go ahead, have another!
- 19. perlbrew use 5.22 cpan Moose # wait a little bit perlbrew use 5.12.5 cpan Moose # wait a bit longer cpan Dist::Zilla # go out for lunch
- 20. Share with others: export PERLBREW_ROOT=/opt/perlbrew export PERLBREW_HOME=/tmp/.perlbrew # put perlbrew perls under /opt to # share with multiple users curl -L http://install.perlbrew.pl | bash
- 21. cpanm
- 22. fu-redhat
- 23. curl -L https://cpanmin.us | perl - App::cpanminus
- 24. WARNING local::lib + perlbrew = pain
- 25. cpanminus++
- 26. carton
- 27. # On your development environment > cat cpanfile requires 'Plack', '0.9980'; requires 'Starman', '0.2000'; > carton install > git add cpanfile cpanfile.snapshot > git commit -m "add Plack and Starman" # Other developer's machine, # or on a deployment box > carton install > carton exec starman -p 8080 myapp.psgi (blatantly copied from carton's perldoc)
- 28. Chicken & Egg?
- 29. cpanm to the rescue!
- 31. "Please sir, I want some more"
- 32. fatpack
- 33. # "myscript" is using cpan deps that need to # be installed by the user first... fatpack pack myscript.pl > myscript.packed.pl # "myscript.packed" can just be run by anybody chmod +x myscript.packed.pl ./myscript.packed.pl
- 34. use the cpan to develop, but don't make your users have to
- 36. mst++
- 37. PAR
- 38. Caveat: complex, but very powerful. But can pack XS modules and even perl itself into a single-file executable!!!!!!
- 39. audreyt++
- 44. fpm -t rpm -s cpan --cpan-perl-lib-path /opt/perl-5.16.1/site_perl/5.16.1 Some::Module
- 45. Works with lots of languages!
- 46. Segue
- 47. NoPerl
- 48. Bow to the JVM
- 51. Y U NO PERL?
- 52. No Habla Perl
- 53. "DevOps"
- 54. "DevOps"
- 55. Systems Administration unless ( Ops::Engineer->can("write_code") ) { warn "Keep your filthy hands off my servers"; }
- 56. OutOfMemoryError
- 57. Thank You!
Editor's Notes
- Good morning, and thanks for coming! So, since this is a Perl conference, and you're all presumably here to learn about Perl, and I'll bet some of you actually use Perl at work, I want to be right up-front with you: This talk is going to be rather light on Perl. I know, I know... it's shocking!
- I apologize in advance for any disappointment and lay the blame firmly on the conference organizers for approving my talk in the first place. But in all seriousness, this talk is going to be light on Perl and a bit heavier on ops/devops, big data, and how I decide what the right tool for the job is in various situations.
- But despite that, I'm gonna talk about how I've managed to work some Perl into a place where almost nobody else knows any Perl, and everybody's opinion of it is based on what they've read on HackerNews.
- So, allow myself to introduce... myself. Blah Blah, tripadvisor, ops/sysadmin/etc...
- I've been working for TripAdvisor for the last three years on the "Data Warehouse" team. We manage a Hadoop-based infrastructure for a variety of business-critical functions, and keeping up with the company's needs and the ever-growing volume of data is really a lot like...
- ...herding elephants. And because Hadoop is ridiculously big and complicated, it turns out that pulling this off requires a whole lot of discipline, knowledge and...
- Well, some days are a lot more difficult than others...
- But, in my day-to-day work, I'm a systems administrator, and I build a whole lot of tools to automate things, collect info, and help other people use the complex systems my team manages. And guess what?
- Perl is not always the best tool for the job.
- But sometimes it is, and really, it's a judgement call.
- But we'll talk about that later. Let's get to the perl.
- There are lots of perfectly valid reasons people avoid using Perl, but they're all things that can be resolved. Just off the top of my head - no C compiler available for XS-based CPAN modules, worry about breakages in CPAN dependencies, not wanting to require users to have to even use CPAN or not having access to the public CPAN over the internet.
- And finally, running on a redhat-based system with all the issues that comes with. You know - ancient versions of perl, missing core modules from the base install, and other kinds of fuckery that causes all sorts of stupid problems.
- But every one of these issues can be resolved! Let's start with the simplest fixes.
- Perlbrew is made of rainbows and awesome-sauce. It contains ninja monkeys that fly out your screen and make you a superhero in mere minutes. What is it? It's a tool that can download and compile whatever version of perl you want on your local system, and you can switch between multiple builds of perl with a simple command. It's insanely easy to use, and insanely useful.
- So, once this is done, what you've got is a brand-spanking new Perl 5.22, installed under your home directory, with it's bin dir added to your $PATH. Furthermore, if you now run the cpan command, the modules you install will be installed using and in that installation of perl.
- One extra step I heartily recommend: install the "special" perlbrew cpanminus which makes it available for all perlbrew-installed perls, without having to first install it with the standard cpan client. I'll talk a little bit more about cpanminus later.
- With these commands, you've just downloaded, compiled, installed and "activated" a fresh build of Perl 5.12.5... It's got its own lib dirs and whatnot, and so if you use the cpan command to install some modules after running the "perlbrew use" command, they will be build and installed with and in that perl's lib dirs.
- So, wanna make sure the latest version of Moose still works on multiple versions of Perl? Simple!
- Of course, it's not always the right solution to have each user install their own builds of perl in their home directories. If you want to put brewed perls in a place where multiple users can get to them, you can do that! And by the way, putting builds of perl under /opt is something I'll say more about shortly.
- So, you can learn a whole lot more about perlbrew on its site, and on IRC, and in it's docs on the CPAN. Let's talk about cpanminus. Cpanminus is an alternative to perl's longtime-default cpan command, and it brings some *really* nice things to the table.
- First of all, on a minimal redhat install, if you're using a version that even *has* perl installed any more, guess what's missing from the system perl? The CPAN command. *facepalm* But, not all is lost.
- If, for some reason, you can't use perlbrew, or get your sysadmin to install a newer, unfucked version of perl somewhere like under /opt or /usr/local, You can install cpanminus into your home directory with a one-liner, just like perlbrew. What cpanminus does is it bootstraps itself using local::lib, and that's important to know for this reason.
- This is one of the reasons that perlbrew has its own special cpanm install. cpanminus uses local::lib internally to be able to install stuff under your home dir. And perlbrew uses it, too. And they can end up messing on each other because the default local::lib that cpanm creates is for whichever perl you're running at the moment, while perlbrew uses one for each version of perl. What can happen is you end up mixing libs intended for different versions of perl into the same lib dir and if that happens, you're gonna have a bad time.
- But despite that gotcha, local::lib is really useful stuff and has been a part of my standard toolkit, both with and without cpanminus. The bottom line is that cpanminus has a lot of advantages over the standard cpan client that make it worthwhile. Being able to do a one-liner install, and being contained in a single file, you can take run it from anywhere and with a little care use it in conjunction with perlbrew, local::lib, and lots of other things, like...
- carton! If perlbrew is ninja-powered awesome sauce, then carton is rainbow unicorns delivering magically delicious vegan bacon directly into your mouth on-demand. Think about that for a moment. Mmmmmmm....
- Carton allows you to specify your exact module dependencies in a plain text file, including the versions you need. It can then bundle all the dist tarballs into a directory with your project, then you can send it all to your users who no longer need to have to install the deps from the cpan, and risk getting the wrong versions or having no internet access or whatever else could go wrong, like running redhat and not having a cpan client...
- But, wait! Doesn't the end user need carton installed in order to install and run your carton-powered code?
- Not at all! Turns out, cpanminus recently gained the ability to work with carton-packaged code! Before, you'd have to instruct the users to first install carton into a local lib, and run it from there, but now you can just drop a copy of cpanm into your project's source tree and tell them to run that.
- Seriously, this is awesome, and the author of both cpanminus and carton is the freakin' man. Tatsuhiko Miyagawa is just an incredible hacker and ironically, as I wrote this slide during the lightning talks, somebody asked the room to give him props because he's just that good.
- So, ready to move on? Good. Let's talk about one more amazing powertool, which is used by cpanm and I use it all the time...
- If you positively, absolutely, must kill ever mother-loving dependency in the room, accept no substitutes. What is does is simple to describe - it figures out all the pure-perl dependencies in your program, and packs them and your program into a single file that your users can just execute.
- And it's also simple to use. Really. just. like. that. If you look at cpanminus' source code on github, you'll see that is uses quite a few cpan dependencies. But if you take a look at the cpanm script that you run on your system, it's all in one file, thanks to fatpack.
- In short it lets you do this.
- There's this one caveat, though, and it's something that can be a deal-breaker for some things, but it's just a decision you have to make. Fatpack is dead simple to use, and the value it delivers keeps it right at the top of my toolbox.
- By the way, more props must be given, as the original author of both local::lib and App::fatpacker is Matt S. Trout, who you may also know for countless other pieces of awesome code out there. He's here at YAPC, and if you meet him, make sure to buy him a beer or four.
- Since I mentioned fatpack, I think I have to mention par. I have to say, I didn't make any other slides on using it because I wrote them this morning with no wifi, and I have to look at the docs every time I use it, but it can do things fatpack simply can't. By the way, if you look for this on the CPAN, you want PAR in all caps. I'm not sure what the other ones are.
- The most basic usage isn't difficult at all, but since it's capable of packing compiled binary modules and even perl itself, there's a lot of complexity under the hood, and some of that leaks out into the docs and options, but when you need a single executable and fatpacker won't do, par is there to save the day. Also, it can do some absolutely amazing things, like packing compiled code for multiple platforms into the same binary...
- And IIRC, this was originally written by audrey tang, who is one of the most amazing hackers ever, and one of the nicest people you'll ever meet. If I might gush for a moment, to be that brilliant and that nice, and write such amazing code, she's one of my heroes.
- So, there's a bunch of different ways to run different versions of perl on a given system, and allow users to consistently build your programs with the same dependencies that you developed with, and even eliminate the need for any installation whatsoever, but sometimes, what you really want is to create RPMs or .deb packages. First, let me tell you how I create rpms to install newer versions of perl on the system that won't interfere with the system's default perl.
- Talk about HQT a little, about how some users were on CentOS 5 with ancient perl that Mojolicious didn't run on, and that they can get this from github, along with instructions on how to use it.
- For cpan modules, or even perl modules you've writen in-house there are a bunch of ways you can build OS packages, but the one I've fallen in love with is called "fpm"
- FPM stands for "Fine" package management if you're being polite, but officially it's "Effing" And it supports building packages of perl modules really, really well.
- And like magic, you will have an RPM of that module dist that will install to the perl that is installed under /opt/perl-5.16.1
- And it works with more than perl. It does ruby, python and can ever make an RPM from the contents of a directory.
- So if there's time, let me talk about systems admin for a bit.
- Over my career, in order to work effectively with others, I have become a "NoPerl" hacker - Not Only Perl. I use lots of languages from day to day, everything - shell, awk, SQL, ruby, java, python... and I choose what I'm working with based on several different factors.
- TripAdvisor is almost entirely a Java shop. The site itself runs on on Java and Scala, and almost every piece of software in the Hadoop ecosystem is written in Java or some other language that runs on the JVM.
- Now, Perl is a fantastic tool for systems administration, but over the years, python, and ruby have both gotten better and gained a whole lot of popularity, and fewer and fewer people I work with know any perl, so I don't use it very much these days.
- But that's the site, right? What about actual systems administration? Java kinda sucks for that, right? And it does. Scripting is what sysadmins want and need, and instead of Perl, we usually use python, ruby, and of course shell scripts.
- So, why no Perl? It's a great glue language! It's pre-installed on every system out there! It's fast, flexible, dynamic, it's got the CPAN! Everybody knows at least a little perl, right? There's almost 400 people right in this building who all know Perl! Extrapolate that out and that that's like 17 billion people in the world who know perl!
- But we all know that's both not true, and not entirely relevant. Over the last 5-6 years, I've discovered fewer and fewer people who know Perl well enough to use it effectively, and more and more who can do Python and Ruby. In the last couple years, I've met people who are otherwise great programmers, but they don't know any perl whatsoever, and have never even written a one-liner.
- So, I want to know - what does "DevOps" mean to you? Anybody?
- Personally, I think it's just a buzzword, and it's simply got too many different meanings to different people. It's kind of mess. But personally, I prefer the one where it means making ops part of the development process, and treating ops work as development work.
- So here's my not-so-humble opinion. If you work as a sysadmin and you can't write basic scripts, use version control, and at least partially decipher the code the software you use is written in, you need to hit the books, because a competent ops engineer *has* to be able to do these things to manage anything more complicated than a LAMP stack.
- So, I ran out of time to make more slides, but I can talk for hours...