Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle


Published on

Presented at Confident Coding III, San Francisco, CA. October 20, 2012.

Flying in from 10,000 foot view (“Hey, browser, show me this”, “Okay, here it is”), we’ll take thoughtful overview of the HTTP request/response cycle. Its essence is simply a series of questions & answers, accumulating portions of content to be gracefully assembled for the user.

We’ll hone in on some key players amidst the “full stack” of communications, with special attention to how an understanding of the HTTP lifecycle endows any developer or designer with the power to optimize for performance, cost, and UX

Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle

  1. 1. Full-Stack & Full Circle What The Heck Happens in an HTTP Request-Response Cycle Carina C. Zona http://cczona.com/ @cczona Confident Coding Microsoft San Francisco October 20, 2012 @cczonaTuesday, October 23, 12When Estelle said she wanted to do a day of teaching "everything else" about web development, I knew right away that the HTTP request-responsecycle is something I just _had_ to talk about.Its the topic that always gets skipped. "Welcome to web development! Lets start at HTML, CSS, and JavaScript!"We pass it right by. And we shouldnt.
  2. 2. Hypertext Transfer Protocol (HTTP) is the core technology of the web @cczonaTuesday, October 23, 12Hypertext Transfer Protocol (HTTP) is THE CORE technology of the web.Yes, more essential than even HTML. Assets on the web can be HTML, but dont have to be. We download software, graphics,PDFs, Word docs, Flash...none of these is HTML. Nor is XML, or JSON. CSS, or JavaScript.Data flowing across the web relies either on HTTP, or its secure counterpart HTTPS.The web relies on the HTTP request-response cycle.
  3. 3. ? ?Tuesday, October 23, 12As a developer, you wants to take an interest in the cycle. Its where your work gets assembled. That assemblycan be slow, or it can be fast. _Which_? Well, that depends on how the request-response cycle is managed.As a developer, you want to _understand_ what the cycle passes through so that you can arrange it to do whateveryou want.
  4. 4. Frontend BackendTuesday, October 23, 12Web development is usually regarded as two parts: Frontend Backend
  5. 5. Client side Server sideTuesday, October 23, 12Sometimes we refer instead to Client side Server side
  6. 6. Frontend & Backend refer to infrastructure Client side & server side refer to programming languages @cczonaTuesday, October 23, 12Its fine, we do it all the time. But technically imprecise: [slide]
  7. 7. Client side (JavaScript) Server side (Python, Ruby, PHP, Perl, Java, etc.)Tuesday, October 23, 12So the cycle refers to the communication passing through our frontend & backend infrastructure. Later well lookat how programming languages fit into this picture.
  8. 8. Tuesday, October 23, 12Lets add a User to this mental picture, too. Because someones got to make the request, right? And someones gotto be interested in what response it gets. So now weve got at least 3 potential components. They evencorrespond pretty nicely to a common team configuration: a user experience designer, a frontend developer,a backend developer.
  9. 9. Web site/appTuesday, October 23, 12Frontend and backend are different parts of a cohesive whole, the website. Its the user and site that are inrelationship with each other, want something of each other, desiring to communicate with each other.The frontend and backend _together_ form the stack of infrastructure for the site to do that.
  10. 10. Full stackTuesday, October 23, 12Thats what we mean by "full stack". The combination of frontend and backend infrastructure, plus thetechnologies that allow them to interact.
  11. 11. HTTP/1.1 200 OK response request GET /somethingTuesday, October 23, 12Voila. The request-response cycle just refers to passing a request message through thestack, and the stack delivering a response to that.
  12. 12. Request @cczonaTuesday, October 23, 12
  13. 13. URL @cczonaTuesday, October 23, 12Theres an important piece missing from that diagram so far, right? URL.What is a URL anyway? Many people compare it to a street address. A series of progressivedetail that narrow it down to an exact location.
  14. 14. Anatomy of a URL @cczonaTuesday, October 23, 12
  15. 15. http://cczona.com/ <scheme>://<host>/ URL Basics @cczonaTuesday, October 23, 12 http://cczona.com/ <scheme>://<host>/At minimum, a URL needs two parts:
  16. 16. http://cczona.com/ <scheme>://<host>/ URL Basics @cczonaTuesday, October 23, 12Scheme - Browsers have lately been hiding the scheme from you. Dont be misled by that.Scheme is _mandatory_. It is an instruction, specifying which transfer method to use.
  17. 17. http://cczona.com/ <scheme>://<host>/ URL Basics @cczonaTuesday, October 23, 12Host - This part I _would_ compare to an address. It specifies where the host server is. Wellexpand on that later.
  18. 18. http://cczona.com/blog/2012/10/20/confident-coding <scheme>://<host>/< path > URL Common @cczonaTuesday, October 23, 12It usually also has at least one more part: Path - the name of a resource being requested
  19. 19. http://cczona.com/blog/2012/10/20/confident-coding <scheme>://<host>/< path > HOW WHERE WHAT URL Common @cczonaTuesday, October 23, 12Consider these three parts the "How, Where, and What".
  20. 20. http://carinazona:opensesame@cczona.com:3000/ blog;year=2012? search&term=Confident&month=October#session2 <scheme>://<user>:<password>@<host>:<port>/ <path>;<params >? <query>#fragment URL All the parts! @cczonaTuesday, October 23, 12There are 6 optional additional parts available.I know, crazy, right?
  21. 21. Port Query Fragment URL Less Common @cczonaTuesday, October 23, 12Port - is where the host server is expected to be listening for an HTTP requestQuery - is one or more values intended to be used by an applicationFragment - is the name of some part within the requested resource.
  22. 22. Username Password Parameters URL Rarely used for HTTP @cczonaTuesday, October 23, 12
  23. 23. User-Agent @cczonaTuesday, October 23, 12A request is submitted via a User-Agent.
  24. 24. User-Agent examples Browser Crawlers/spiders/robots Screen readers Programs Scripts @cczonaTuesday, October 23, 12Browsers - are the most common by far. But many other user-agent types exist as well.Crawlers/spiders/robots - automated site browser. Some examples include search engine indexers(Googlebot), archivers (Archive.org aka "The Internet Archive"), testing tools (curl-loader), monitoringservices (uptime)
  25. 25. User-Agent examples Browser Crawlers/spiders/robots Screen readers Programs Scripts @cczonaTuesday, October 23, 12Screenreaders - assistive devices that read content aloudPrograms - such as cURL & Wget
  26. 26. User-Agent examples Browser Crawlers/spiders/robots Screen readers Programs Scripts @cczonaTuesday, October 23, 12Scripts - these can be written in any language capable of using the HTTP scheme. Which is pretty much anycommon one you can imagine. Perl, PHP, Ruby, and Python have HTTP libraries that make it easy to submitrequests and accept responses.
  27. 27. Method @cczonaTuesday, October 23, 12The User-Agent submits the request using one of the defined HTTP methods.
  28. 28. HTTP Methods GET POST HEAD PUT DELETE OPTIONS TRACE @cczonaTuesday, October 23, 12GET - is the most common method. The request is communicated primarily via the URL itself
  29. 29. HTTP Methods GET POST HEAD PUT DELETE OPTIONS TRACE @cczonaTuesday, October 23, 12POST - is also common, typically used via an HTML form action. The request is communicated via the URL and the forms data iscommunicated via the requests headers. GETs have a length limit, so POST is a handy alternative when your form might receive a wholelot of data.
  30. 30. HTTP Methods GET POST HEAD PUT DELETE OPTIONS TRACE @cczonaTuesday, October 23, 12HEAD - the request is communicated the same as a GET, so via URL. But HEAD signals that the response should only contain headers,not body. Whereas the response to a GET is both.PUT, DELETE, OPTIONS & TRACE have not been widely adopted.
  31. 31. Headers @cczonaTuesday, October 23, 12Ive been mentioning headers. The obvious question is: _what the heck is a header_?
  32. 32. Headers are metadata @cczonaTuesday, October 23, 12A header is metadata about the request or response.
  33. 33. Headers Firefox Web Developer Tools @cczonaTuesday, October 23, 12Browsers developer tools can display headers. Lets take a look at these.Notice the structure: each line is one key/value pair. The line starts with a key, then a colon,then the value.
  34. 34. Headers Chrome Developer Tools @cczonaTuesday, October 23, 12
  35. 35. Headers Safari Developer Tools @cczonaTuesday, October 23, 12
  36. 36. Request headers Include information about: • request • users environment • user-agents capabilities • user-agents identity* @cczonaTuesday, October 23, 12Request headers originate at the user-agent. Its headers include information about: request users environment user-agents capabilities user-agents identity*
  37. 37. Request headers Include information about: • request • users environment • user-agents capabilities • user-agents identity* @cczonaTuesday, October 23, 12Theres a huge caveat here. The "user-agent" header frequently is a lie. Theres a long story as to why. Never, everdepend on the user-agent header to be accurate.
  38. 38. Domain Name Server (DNS) @cczonaTuesday, October 23, 12
  39. 39. DNS Distributed map of domains to IPs @cczonaTuesday, October 23, 12When a request is submitted by the user, the user-agent first has to determine where tosubmit it. The URLs host part can be a domain name or an IP address. But the user-agentactually can only submit to an IP address. This is where DNS comes in. A domain nameservice (DNS) is a directory of domains and their IP addresses.
  40. 40. Nameservers For example: NS1.Dreamhost.com NS2.Dreamhost.com NS3.Dreamhost.com NS4.Dreamhost.com @cczonaTuesday, October 23, 12If youve ever registered a domain name, you may have noticed that the domain was eitherconfigured with a default set of nameservers, or you had to provide the set of nameserversfor your web hosting provider.For example (slide)
  41. 41. Nameservers For example: NS1.dreamhost.com NS2.dreamhost.com NS3.dreamhost.com NS4.dreamhost.com @cczonaTuesday, October 23, 12Nameservers are a secondary directory necessary for DNS resolution. Theyre responsible forkeeping track of IP addresses where each of _your domains_ services are located.
  42. 42. Host Server @cczonaTuesday, October 23, 12
  43. 43. Host Servers @cczonaTuesday, October 23, 12The host server is not the web server. Rather, its the hardware and operating system thathouse the web server, and potentially houses other infrastructure used by the site. So forexample: OSX, Linux, Unix, Windows Server, or Novel NetWare.
  44. 44. HTTP/1.1 200 OK response request GET /somethingTuesday, October 23, 12The host server passes the request off to the web server.
  45. 45. HTTP/1.1 200 OK response request GET /somethingTuesday, October 23, 12
  46. 46. Web Server @cczonaTuesday, October 23, 12
  47. 47. Web Server @cczonaTuesday, October 23, 12A web server is software thats capable of taking an HTTP request as input, and of deliveringan HTTP response as output.The most popular web servers are Apache, Microsoft IIS, and "engine X" (nginx).Lets take a look at the web servers log of our request.
  48. 48. - - [20/Oct/2012:13:58:09 +0000] "GET /blog/?search&term=Confident&month=October HTTP/1.1" 200 11633 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4" Web Server Request Log @cczonaTuesday, October 23, 12Looks pretty much like gibberish, right? Ah, but wait....
  49. 49. - - [20/Oct/2012:13:58:09 +0000] "GET /blog/?search&term=Confident&month=October HTTP/1.1" 200 11633 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4" Web Server Request Log @cczonaTuesday, October 23, 12Now see how many parts you can name already. Whats the request method? The IP address?The path? And query string? Any thoughts on what that purple one is?
  50. 50. Web Server Redirect @cczonaTuesday, October 23, 12The web server interprets the request & determines how it should be handled.It looks at the request path and checks whether it matches any redirection directives.These may be temporary (e.g. "offline for maintenance"), or they may be permanent (e.g."blog" is now "old-blog").
  51. 51. Web Server Redirect @cczonaTuesday, October 23, 12In lines 1 & 2, the web server gracefully handles files that have moved.In lines 4 & 5, it uses a regular expression to match any files that match the pattern.
  52. 52. Web Server Redirect @cczonaTuesday, October 23, 12Maybe youve seen something similar to this WordPress panel. It looks like some cool fancyfeature. Naw. Its just using a web server redirect. Regular expressions are awesome.
  53. 53. Web Server Redirect (example) @cczonaTuesday, October 23, 12Heres the redirect. Every page and post of your WordPress blog is actually one script file +an HTTP redirect. Thats it. It only look like infinite pages. Why do this? Flexibility, whileenforcing consistency. You could change every URL without breaking a single link. Nice.
  54. 54. Web Server Redirect (example) @cczonaTuesday, October 23, 12Once a web server has finished doing redirects, it checks whether the new path matches acorresponding path in the web servers directory hierarchy. If it doesnt match, or theres aproblem reading in the content, error time!
  55. 55. Web Server 404 Page Not Found @cczonaTuesday, October 23, 12Aww, sad panda.Well, sad Octocat.
  56. 56. Web Server 500 Internal Server Error @cczonaTuesday, October 23, 12Uh-oh. Headache Puppy says badness happened.
  57. 57. Response @cczonaTuesday, October 23, 12When it does match, now the magic happens.
  58. 58. HTML @cczonaTuesday, October 23, 12The web server reads in whatever content the file provides. Text, html, image, output from ascript...whatever.
  59. 59. HTTP/1.1 200 OK response request GET /somethingTuesday, October 23, 12Passing the request down to a script gives us opportunity to generate some (or all) contentdynamically. Often this involves using values passed in via the URL and/or POSTed data.
  60. 60. Script @cczonaTuesday, October 23, 12This example script generates output, starting with a response header (line 1), then declaringthe end of headers (line 3), then outputting response body (starting at line 5).
  61. 61. HTTP/1.1 200 OK response request GET /somethingTuesday, October 23, 12A script may also want to call on other services, such as a database.
  62. 62. Example Response Response Message @cczonaTuesday, October 23, 12Once the web server receives output suitable for delivery in an HTTP response, it send it onits way.
  63. 63. Response headers Mostly describe: • Content • How user-agent should handle it First line is HTTP version and final status HTTP/1.1 200 OK @cczonaTuesday, October 23, 12The response likewise has headers. Most of these describe the content and how it should behandled by the user-agent.Notice that the first line of the response states the final status of the request. If things wentwell, thats "200 OK".
  64. 64. Example Response Response Message @cczonaTuesday, October 23, 12The user agent receives the response, and interprets it. In this case, the header had told theuser-agent to interpret our responses body as HTML.
  65. 65. Back to the Frontend @cczonaTuesday, October 23, 12Now its the user-agents turn to process what its received. It parses the content, todetermine how to render it for the user. Cycle complete!
  66. 66. HTTP/1.1 200 OK response request GET /somethingTuesday, October 23, 12Now it can determine what _other_ assets are needed. The user-agent fires off requests forthose as well. JavaScript, CSS, images, etc. A bunch more cycles!
  67. 67. HTTP/1.1 200 OK response request GET /somethingTuesday, October 23, 12When the user-agent has received everything it needs, its final steps are to layout thecontent, and render it for the user.
  68. 68. Done. @cczonaTuesday, October 23, 12Voila.
  69. 69. Optimizations @cczonaTuesday, October 23, 12So far weve seen that a pretty simple set of web infrastructure. But much more technologycan optionally be involved.
  70. 70. Optimizing AJAX Minification Caching Sprites Expires headers Database indexes, operations, Compression joins Proxy server Partials Content delivery network (CDN) REST/APIs @cczonaTuesday, October 23, 12You may have noticed that theres a lot of potential here to make a site seem sluggish. Sowhat we learn from the HTTP request-response cycle is that there are many opportunities tooptimize for fast performance.
  71. 71. Stay tuned... @cczonaTuesday, October 23, 12Luckily, Estelle will be talking next about a bunch of methods for optimizing performance!
  72. 72. Thank you! Any questions? @cczonaTuesday, October 23, 12Any questions for me?
  73. 73. Credits • Networking icons by http://icons.iconarchive.com • Illustrations from "HTTP: The Definitive Guide" by David Gourley & Brian Totty • Header screenshots from Safari Developer Tools, Chrome Developer Tools, and Firefox Web Developer Tools • DNS diagram from HowStuffWorks.com • 404 Page Not Found from http://www.github.com & htttp://mozilla.org • 500 Internal Server from http://www.tumblr.com/tagged/internal-server-error @cczonaTuesday, October 23, 12
  74. 74. Resources • "HTTP: The Definitive Guide" by David Gourley & Brian Totty (OReilly) • http://dev.opera.com/articles/view/http-basic-introduction/ • http://dev.opera.com/articles/view/http-lets-get-it-on/ • http://dev.opera.com/articles/view/http-response-codes/ • http://www.w3.org/community/webed/wiki/How_does_the_Internet_work • http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol • http://en.wikipedia.org/wiki/Representational_state_transfer • http://en.wikipedia.org/wiki/URI_scheme • http://en.wikipedia.org/wiki/Ajax_(programming) @cczonaTuesday, October 23, 12