Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle

Full-Stack & Full Circle
What The Heck Happens in an HTTP Request-Response Cycle

Carina C. Zona
http://cczona.com/
@cczona

Conﬁdent Coding
Microsoft San Francisco
October 20, 2012

@cczona

Tuesday, October 23, 12
When Estelle said she wanted to do a day of teaching "everything else" about web development, I knew right away that the HTTP request-response
cycle is something I just _had_ to talk about.
It's the topic that always gets skipped. "Welcome to web development! Let's start at HTML, CSS, and JavaScript!"
We pass it right by. And we shouldn't.

Hypertext Transfer Protocol (HTTP)
is the core technology of the web

@cczona

Hypertext Transfer Protocol (HTTP) is THE CORE technology of the web.
Yes, more essential than even HTML. Assets on the web can be HTML, but don't have to be. We download software, graphics,
PDFs, Word docs, Flash...none of these is HTML. Nor is XML, or JSON. CSS, or JavaScript.
Data ﬂowing across the web relies either on HTTP, or its secure counterpart HTTPS.
The web relies on the HTTP request-response cycle.

?

?


As a developer, you wants to take an interest in the cycle. It's where your work gets assembled. That assembly
can be slow, or it can be fast. _Which_? Well, that depends on how the request-response cycle is managed.

As a developer, you want to _understand_ what the cycle passes through so that you can arrange it to do whatever
you want.

Frontend

Backend


Web development is usually regarded as two parts:
Frontend
Backend

Client side

Server side


Sometimes we refer instead to
Client side
Server side

Frontend & Backend refer to infrastructure
Client side & server side refer to programming languages

@cczona


It's ﬁne, we do it all the time. But technically imprecise: [slide]

Client side
(JavaScript)

Server side
(Python, Ruby, PHP,
Perl, Java, etc.)


So the cycle refers to the communication passing through our frontend & backend infrastructure. Later we'll look
at how programming languages ﬁt into this picture.


Let's add a User to this mental picture, too. Because someone's got to make the request, right? And someone's got
to be interested in what response it gets. So now we've got at least 3 potential components. They even
correspond pretty nicely to a common team conﬁguration: a user experience designer, a frontend developer,
a backend developer.

Web
site/app


Frontend and backend are different parts of a cohesive whole, the website. It's the user and site that are in
relationship with each other, want something of each other, desiring to communicate with each other.

The frontend and backend _together_ form the stack of infrastructure for the site to do that.

Full stack


That's what we mean by "full stack". The combination of frontend and backend infrastructure, plus the
technologies that allow them to interact.

HTTP/1.1 200 OK

response

request

GET /something


Voila. The request-response cycle just refers to passing a request message through the
stack, and the stack delivering a response to that.

Request

@cczona


URL

@cczona


There's an important piece missing from that diagram so far, right? URL.

What is a URL anyway? Many people compare it to a street address. A series of progressive
detail that narrow it down to an exact location.

Anatomy of a URL

@cczona


http://cczona.com/
<scheme>://<host>/

URL Basics
@cczona


http://cczona.com/
<scheme>://<host>/

At minimum, a URL needs two parts:

http://cczona.com/
<scheme>://<host>/

URL Basics
@cczona


Scheme - Browsers have lately been hiding the scheme from you. Don't be misled by that.
Scheme is _mandatory_. It is an instruction, specifying which transfer method to use.

http://cczona.com/
<scheme>://<host>/

URL Basics
@cczona


Host - This part I _would_ compare to an address. It speciﬁes where the host server is. We'll
expand on that later.

http://cczona.com/blog/2012/10/20/conﬁdent-coding
<scheme>://<host>/< path >

URL Common
@cczona


It usually also has at least one more part:
Path - the name of a resource being requested

http://cczona.com/blog/2012/10/20/conﬁdent-coding
<scheme>://<host>/< path >
HOW WHERE WHAT

URL Common
@cczona


Consider these three parts the "How, Where, and What".

http://carinazona:opensesame@cczona.com:3000/
blog;year=2012?
search&term=Conﬁdent&month=October#session2

<scheme>://<user>:<password>@<host>:<port>/
<path>;<params >?
<query>#fragment

URL All the parts!
@cczona


There are 6 optional additional parts available.

I know, crazy, right?

Port
Query
Fragment

URL Less Common
@cczona


Port - is where the host server is expected to be listening for an HTTP request
Query - is one or more values intended to be used by an application
Fragment - is the name of some part within the requested resource.

Username
Password
Parameters

URL Rarely used for HTTP
@cczona


User-Agent

@cczona


A request is submitted via a User-Agent.

User-Agent examples
Browser
Crawlers/spiders/robots
Screen readers
Programs
Scripts

@cczona


Browsers - are the most common by far. But many other user-agent types exist as well.

Crawlers/spiders/robots - automated site browser. Some examples include search engine indexers
(Googlebot), archivers (Archive.org aka "The Internet Archive"), testing tools (curl-loader), monitoring
services (uptime)

User-Agent examples
Browser
Screen readers
Programs
Scripts

@cczona


Screenreaders - assistive devices that read content aloud

Programs - such as cURL & Wget

User-Agent examples
Browser
Screen readers
Programs
Scripts

@cczona


Scripts - these can be written in any language capable of using the HTTP scheme. Which is pretty much any
common one you can imagine. Perl, PHP, Ruby, and Python have HTTP libraries that make it easy to submit
requests and accept responses.

Method

@cczona


The User-Agent submits the request using one of the deﬁned HTTP methods.

HTTP Methods
GET
POST
HEAD
PUT
DELETE
OPTIONS
TRACE

@cczona

GET - is the most common method. The request is communicated primarily via the URL itself

HTTP Methods
GET
POST
HEAD
PUT
DELETE
OPTIONS
TRACE

@cczona

POST - is also common, typically used via an HTML form action. The request is communicated via the URL and the form's data is
communicated via the request's headers. GETs have a length limit, so POST is a handy alternative when your form might receive a whole
lot of data.

HTTP Methods
GET
POST
HEAD
PUT
DELETE
OPTIONS
TRACE

@cczona

HEAD - the request is communicated the same as a GET, so via URL. But HEAD signals that the response should only contain headers,
not body. Whereas the response to a GET is both.

PUT, DELETE, OPTIONS & TRACE have not been widely adopted.

Headers

@cczona


I've been mentioning headers. The obvious question is: _what the heck is a header_?

Headers are metadata

@cczona


A header is metadata about the request or response.

Headers Firefox Web Developer Tools
@cczona


Browsers' developer tools can display headers. Let's take a look at these.
Notice the structure: each line is one key/value pair. The line starts with a key, then a colon,
then the value.

Headers Chrome Developer Tools
@cczona


Headers Safari Developer Tools
@cczona


Request headers
Include information about:

• request
• user's environment
• user-agent's capabilities
• user-agent's identity*

@cczona

Request headers originate at the user-agent. Its headers include information about:
request
user's environment
user-agent's capabilities
user-agent's identity*

Request headers
Include information about:

• request
• user's environment
• user-agent's capabilities
• user-agent's identity*

@cczona

There's a huge caveat here. The "user-agent" header frequently is a lie. There's a long story as to why. Never, ever
depend on the user-agent header to be accurate.

Domain Name Server (DNS)

@cczona


DNS Distributed map of domains to IPs
@cczona


When a request is submitted by the user, the user-agent ﬁrst has to determine where to
submit it. The URL's host part can be a domain name or an IP address. But the user-agent
actually can only submit to an IP address. This is where DNS comes in. A domain name
service (DNS) is a directory of domains and their IP addresses.

Nameservers
For example:

NS1.Dreamhost.com
NS2.Dreamhost.com
NS3.Dreamhost.com
NS4.Dreamhost.com

@cczona


If you've ever registered a domain name, you may have noticed that the domain was either
conﬁgured with a default set of nameservers, or you had to provide the set of nameservers
for your web hosting provider.

For example (slide)

Nameservers
For example:

NS1.dreamhost.com
NS2.dreamhost.com
NS3.dreamhost.com
NS4.dreamhost.com

@cczona


Nameservers are a secondary directory necessary for DNS resolution. They're responsible for
keeping track of IP addresses where each of _your domain's_ services are located.

Host Server

@cczona


Host Servers
@cczona


The host server is not the web server. Rather, it's the hardware and operating system that
house the web server, and potentially houses other infrastructure used by the site. So for
example: OSX, Linux, Unix, Windows Server, or Novel NetWare.

HTTP/1.1 200 OK

response

request

GET /something


The host server passes the request off to the web server.

HTTP/1.1 200 OK

response

request

GET /something


Web Server

@cczona


Web Server
@cczona


A web server is software that's capable of taking an HTTP request as input, and of delivering
an HTTP response as output.
The most popular web servers are Apache, Microsoft IIS, and "engine X" (nginx).
Let's take a look at the web server's log of our request.

108.75.73.197 - - [20/Oct/2012:13:58:09 +0000]
"GET /blog/?search&term=Conﬁdent&month=October
HTTP/1.1" 200 11633 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML,
like Gecko) Chrome/22.0.1229.94 Safari/537.4"

Web Server Request Log
@cczona


Looks pretty much like gibberish, right? Ah, but wait....

108.75.73.197 - - [20/Oct/2012:13:58:09 +0000]
"GET /blog/?search&term=Conﬁdent&month=October
HTTP/1.1" 200 11633 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML,
like Gecko) Chrome/22.0.1229.94 Safari/537.4"

Web Server Request Log
@cczona


Now see how many parts you can name already. What's the request method? The IP address?
The path? And query string? Any thoughts on what that purple one is?

Web Server Redirect
@cczona


The web server interprets the request & determines how it should be handled.
It looks at the request path and checks whether it matches any redirection directives.
These may be temporary (e.g. "offline for maintenance"), or they may be permanent (e.g.
"blog" is now "old-blog").

Web Server Redirect
@cczona


In lines 1 & 2, the web server gracefully handles ﬁles that have moved.
In lines 4 & 5, it uses a regular expression to match any ﬁles that match the pattern.

Web Server Redirect
@cczona


Maybe you've seen something similar to this WordPress panel. It looks like some cool fancy
feature. Naw. It's just using a web server redirect. Regular expressions are awesome.

Web Server Redirect (example)
@cczona


Here's the redirect. Every page and post of your WordPress blog is actually one script ﬁle +
an HTTP redirect. That's it. It only look like inﬁnite pages. Why do this? Flexibility, while
enforcing consistency. You could change every URL without breaking a single link. Nice.

Web Server Redirect (example)
@cczona


Once a web server has ﬁnished doing redirects, it checks whether the new path matches a
corresponding path in the web server's directory hierarchy. If it doesn't match, or there's a
problem reading in the content, error time!

Web Server 404 Page Not Found
@cczona


Aww, sad panda.
Well, sad Octocat.

Web Server 500 Internal Server Error
@cczona


Uh-oh. Headache Puppy says badness happened.

Response

@cczona


When it does match, now the magic happens.

HTML
@cczona


The web server reads in whatever content the ﬁle provides. Text, html, image, output from a
script...whatever.

HTTP/1.1 200 OK

response

request

GET /something


Passing the request down to a script gives us opportunity to generate some (or all) content
dynamically. Often this involves using values passed in via the URL and/or POSTed data.

Script
@cczona


This example script generates output, starting with a response header (line 1), then declaring
the end of headers (line 3), then outputting response body (starting at line 5).

HTTP/1.1 200 OK

response

request

GET /something


A script may also want to call on other services, such as a database.

Example Response Response Message
@cczona


Once the web server receives output suitable for delivery in an HTTP response, it send it on
its way.

Response headers
Mostly describe:
• Content
• How user-agent should handle it
First line is HTTP version and final status

HTTP/1.1 200 OK

@cczona


The response likewise has headers. Most of these describe the content and how it should be
handled by the user-agent.
Notice that the first line of the response states the final status of the request. If things went
well, that's "200 OK".

Example Response Response Message
@cczona


The user agent receives the response, and interprets it. In this case, the header had told the
user-agent to interpret our response's body as HTML.

Back to the Frontend

@cczona


Now it's the user-agent's turn to process what it's received. It parses the content, to
determine how to render it for the user. Cycle complete!

HTTP/1.1 200 OK

response

request

GET /something


Now it can determine what _other_ assets are needed. The user-agent ﬁres off requests for
those as well. JavaScript, CSS, images, etc. A bunch more cycles!

HTTP/1.1 200 OK

response

request

GET /something


When the user-agent has received everything it needs, its ﬁnal steps are to layout the
content, and render it for the user.

Done.
@cczona


Voila.

Optimizations

@cczona


So far we've seen that a pretty simple set of web infrastructure. But much more technology
can optionally be involved.

Optimizing

AJAX Miniﬁcation
Caching Sprites
Expires headers Database indexes, operations,
Compression joins
Proxy server Partials
Content delivery network (CDN) REST/APIs

@cczona


You may have noticed that there's a lot of potential here to make a site seem sluggish. So
what we learn from the HTTP request-response cycle is that there are many opportunities to
optimize for fast performance.

Stay tuned...

@cczona


Luckily, Estelle will be talking next about a bunch of methods for optimizing performance!

Thank you!

Any questions?

@cczona


Any questions for me?

Credits

• Networking icons by http://icons.iconarchive.com

• Illustrations from "HTTP: The Deﬁnitive Guide"
by David Gourley & Brian Totty

• Header screenshots from Safari Developer Tools, Chrome Developer Tools, and
Firefox Web Developer Tools

• DNS diagram from HowStuﬀWorks.com

• 404 Page Not Found from http://www.github.com & htttp://mozilla.org

• 500 Internal Server from http://www.tumblr.com/tagged/internal-server-error

@cczona


Resources

• "HTTP: The Deﬁnitive Guide" by David Gourley & Brian Totty (O'Reilly)

• http://dev.opera.com/articles/view/http-basic-introduction/

• http://dev.opera.com/articles/view/http-lets-get-it-on/

• http://dev.opera.com/articles/view/http-response-codes/

• http://www.w3.org/community/webed/wiki/How_does_the_Internet_work

• http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol

• http://en.wikipedia.org/wiki/Representational_state_transfer

• http://en.wikipedia.org/wiki/URI_scheme

• http://en.wikipedia.org/wiki/Ajax_(programming)

@cczona


Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle

Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle

Recommended

More Related Content

What's hot

What's hot(20)

Viewers also liked

Viewers also liked(20)

Similar to Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle

Similar to Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle(20)

More from Carina C. Zona

More from Carina C. Zona(11)

Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle