Download to read offline
![Chapter 1
Introduction
Electronic commerce (e-commerce) dates back to late 1970s when the first electronic
data interchange (EDI) systems were introduced to facilitate the exchange of data be-
tween companies and institutions. The initial types of data exchange were limited
to invoices and purchase orders and required significant financial investment in the
computer networks and hardware. The first EDI-based e-commerce networks were
later augmented with other forms of e-commerce such as automated teller machines
(ATM) and telephone banking in the 1980s and subsequently enterprise resource plan-
ning, data mining and data warehouse systems in the early 1990s.
In late 1990s and early 2000s, the term e-commerce started to include consumer-
based shopping transactions over the World Wide Web (WWW). These transactions
typically include a combination of browsing of a website’s catalog and its related con-
tent in the form of HTML, image files and marketing videos followed by the presenta-
tion of e-shopping cart and check out transactions supported by secure payment au-
thorization, generally over the HTTPs protocol. The United States Census Bureau [62]
defines several important concepts related to this topic. For example, it defines e-
commerce as “any transaction completed over a computer-mediated network that in-
volves the transfer of ownership or rights to use goods or services.” This definition
does not differentiate between monetary and non-monetary transactions as well zero
price and paid transactions. The concept of e-business infrastructure is also defined as
“the share of total economic infrastructure to support electronic business and conduct
electronic commerce”. The examples of infrastructure thus include the computers,
routers, satellites, network channels, system and application software, website host-
ing and programmers.
1.1 Problem statement
The goal of this thesis is to evaluate the impact of a Content Delivery Network on
scalability and performance of a content-rich and highly transactional website that
utilizes the Java Enterprise Edition (EE) platform. Since its inception in 1996 the web-
1](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-9-320.jpg)
![CHAPTER 1. INTRODUCTION 3
Table 1.2: Annual e-commerce customer satisfaction survey results
Satisfaction Level 2005 2004 2003 2002 2002 vs 2005 change
Very Satisfied 40% 37% 40% 37% 3%
Somewhat Satisfied 24% 24% 23% 22% 2%
Neutral 31% 32% 30% 33% -3%
Somewhat Dissatisfied 4% 5% 5% 5% -1%
Very Dissatisfied 2% 3% 3% 3% -1%
that run within browsers to enable rich consumer experience. The number of objects
on HTML pages is the key indicator of the number of connections that will need to be
opened between the client (browser) and the Web server.
While e-commerce still represents only a minor fraction of total retail sales, it is
quickly becoming a very important component of any major retailer’s presence in the
marketplace. One study of European consumers shows increase in online participa-
tion from 42 percent to 56 percent between 2003 and 2006. Forrester Research also re-
ports that in 2006, on average, 80 percent of consumers reported researching products
online with approximately 55 percent reporting either an online or offline purchase in
the preceding three months. Nielsen Research [50] confirms this growth trend, but its
research also points out that consumer satisfaction from online purchases has experi-
enced only a slight growth as shown in the Table 1.2.
Current level of consumer satisfaction underscores the need to continuously im-
prove customer experience on the Web, especially with regards to website perfor-
mance. More than half of online shoppers cite response time as a key influence on
their shopping behavior. Consumers have come to expect fast, uninterrupted and
relevant shopping experience as well as scalability during major holidays and contin-
uous availability of services. Content delivery has become a critical component of the
online shopping experience and Content Delivery Networks (CDNs) were established
with the primary purpose of addressing issues with performance and scalability.
1.3 Significance and expected contributions
Current model of content delivery on the Internet can be classified as centralized in
a sense that a majority of the websites use the origin servers at their hosting environ-
ments to serve content. User requests have to travel through several networks and
encounter a myriad of bottlenecks including Peering, First Mile, Last Mile and Backbone
- all of which are inherent to the way Internet works [2,3,5]. CDNs have emerged as
collections of servers located in close proximity to the Internet backbone, and thus to
the end user, with the purpose of reducing the number of “hops” between network](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-11-320.jpg)
![CHAPTER 1. INTRODUCTION 5
ships between content delivery and server-side Java EE components of the site’s ar-
chitecture. This granularity is necessary due to inherent inter-dependencies between
processing tiers when queuing networks are involved.
Figure 1.1: Queue network for a typical n-tier e-commerce system
Typical performance model of an e-commerce site could be represented by the
queue network in Figure 1.4 according to [44]. According to this model, the bottle-
necks in the database (DB) server tier could result in requests queuing up at the Web
server tier and vice versa. Also, performance improvements in any of the tiers will
most likely result in either a positive or negative impact on other tiers.
1.5 Evaluation criteria
Our evaluation framework will include consumer-centric and system-centric metrics.
Consumer-centric metrics focus on response times as observed by the users of the site.
We will implement an appliance-based monitoring solution that captures HTTP and
HTTPS traffic at the URI stem leave and provides a high-level view of application re-
sponse times. It will give us benchmarks for key function points of our e-commerce
solution, including browsing of Category and Product Detail pages as well as Shop-
ping Cart, Billing and Order Confirmation components. The tool will provide us with
measurements in terms of network, client and server time. We will capture measure-
ments from before and after the CDN implementation.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-13-320.jpg)
![Chapter 2
Background and Previous Work
Bass et al. [15] give a generic definition of performance as responsiveness of the system
characterized by the time required to respond to events. The level of performance is
often described by the number of user or system transactions per unit of time. In e-
commerce, this concept encompasses several measures of throughput in terms of Web
page response times and number of page views for a given period of time. It can
also be expressed in terms of responsiveness and throughput capacity of key server-
side Java EE components such as Servlets and Enterprise Java Beans. Each site has a
unique traffic profile, but a generic traffic pattern can be represented by the Customer
Behavior Model (CBM) state transition graph in Figure 2 as proposed by Menasce et
al. in [45].
The CBM graph identifies several key states in the browsing and shopping expe-
rience. The numbers for each state transition represent its probability. Hence, there
is a 30 percent chance of a customer selecting the Search page from the Browse page.
Performance of a website can thus be characterized by the response times for each of
these states under a particular level of volume of user site visits. A visit consists of
multiple requests to different components of the browsing and shopping experience.
For the purpose of this study, we will further breakdown the performance of each
behavior point into three components: network time, client time and server time. The
network time is the amount time that a request and response pairs spend traveling to
and from the load balancers. The client time represents the amount of time it takes the
Web browser to issue an HTTP request and render the response in the form of HTML
and other objects. The server time is the amount of time each request spends on the
Web, application and database servers.
Scalability represents the ability for a given site to handle its projected traffic vol-
ume as the number of requests for the critical points grows while maintaining its cur-
rent hardware footprint. The latter is particularly important as most websites find it
difficult to significantly change their footprint during a critical business period. For
example, the subject website experiences up to four-fold increases in page views and
a 15-fold increase in bandwidth usage during some of its critical business periods.
8](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-16-320.jpg)
![CHAPTER 2. BACKGROUND AND PREVIOUS WORK 9
Figure 2 represents the subject site’s transactional bursts during key shopping holi-
days. Brebner et al. [22] provide several benchmark-based insights into the potential
scalability of J2EE environments.
2.1 Architectural overview of subject site
A typical e-commerce website can be represented in a series of views that depict its
functional components, software modules and hardware infrastructure as proposed
by [20, 42]. For the purpose of this study we will analyze a website that has existed
since 1996 and was recently (2003) re-architected using the IBM Websphere Commerce
software components using the Java Enterprise Edition (Java EE) platform. The site’s
business process view is represented in Figure 2.1. The business process view high-
lights several key functional components of the subject website, namely the catalog,
marketing and order management features. The functional view is supported by the
hardware and software components as shown in Figure 2.1 and Figure 2.1.
2.2 Website content
In the realm of e-commerce, content refers to any information made available for con-
sumers to download. This includes catalog data, HTML pages, images, audio files,
video files in Flash or other formats and even software downloads. Efficient man-
agement and deployment of content-related assets have thus become mission critical
activities for many organizations that depend on the Internet as a sales channel. E-
commerce websites rely on three types of assets to support the selling function: cat-
alog data assets, content assets and application assets. Catalog assets are used to define
products with characteristics such as price or product name. Content assets represent
editorial content such as marketing information, product reviews and promotional
information. Application assets include software components created by software de-
velopers that are generally managed separately from the other two types of assets.
Content can also be classified as dynamic or static. Dynamic content is generated upon
consumer’s request and it may be unique for each request. It is typically assembled
on the application server with involvement of the DBMS. Static content is represented
by HTML, client-side scripts (JavaScript), images and other multimedia assets that do
not change from request to request. This type of content generally resides on Web
server(s) and is served up from there. Figure 2.2 represents a typical approach to
management and publishing of static and dynamic content. It is important to note
that generally the static content will be published to a file system and dynamic con-
tent will be made available via server side components with support of a DBMS. The
fundamental concepts of content management and publishing are covered in [19].
Even if we were to achieve infinite scalability and performance at the hosting fa-
cility we would still have a dependency on the Internet as a whole for delivery of](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-17-320.jpg)
![CHAPTER 2. BACKGROUND AND PREVIOUS WORK 10
content to the browsers. The Internet could not function efficiently unless the indi-
vidual networks that make its backbone can also communicate with one another. The
process of exchanging data between two networks is called peering and it takes place
between two routers also known as peering points located at the edge of each net-
work. The two networks use an exterior gateway protocol called the Border Gateway
Protocol (BGP) to send traffic between them. The interior gateway protocol (OSPF
and RIP) is used to route traffic within a network. Hence, the peering points and rout-
ing protocols connect different networks into a unified infrastructure that allows the
Internet to function efficiently. Internet is made up of many different networks that
communicate with one another using the IP protocol. Individual networks consist of
a variety of routers, switch-es and fiber in order to move data packets within the net-
work. The structure of “network of networks” results in several potential scalability
and performance bottlenecks. They are generally classified as: first mile, peering points,
backbone and last mile. Figure 2.2 represents the types and location of the bottlenecks as
described by [3].
The First Mile bottleneck refers to the limitations in the website’s connectivity to
the Internet via its Internet Service Provider (ISP). In order to reach desired scalability
it needs to continuously expand its connectivity to the ISP. The ISP, in turn, must also
expand its capacity in order to meet its customers’ scalability requirements. Peering
points also represent potential bottlenecks as large networks are not economically mo-
tivated to scale up the number of peering points with the networks of their competi-
tors, especially since a significant portion of the traffic handled by the peering points
is transit traffic with packets originating on other networks. This lack of competitive
and financial motivation over time has resulted in a limited number of peering points
across major networks. The Backbone Problem refers to the fact that the ISPs’ router
capacity has historically not kept up with growth of traffic demands. Finally, the Last
Mile problem reflects the limited capacity of a typical user’s connection to their ISP.
It’s important to note that just solving one of the above bottlenecks, such as the Last
Mile, by increasing the reach of broadband connectivity at home will not automat-
ically address the other limitations. These need to be treated as separate problems
that, if addressed, would help solve the problem as a whole.
2.3 Performance considerations and tactics
Most websites measure their performance and scalability using a combination of user
and system-centric metrics. Page response times represent the user-centric metric
while the CPU, memory and disk utilization of the web, application and database
servers illustrate the system dimension of performance. High consumer expectations
have led many highly-transactional websites to implement a variety of technology
strategies to achieve a combination of high performance and scalability [10]. Typical
approaches include scaling both horizontally and vertically. Horizontal scalability en-](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-18-320.jpg)
![CHAPTER 2. BACKGROUND AND PREVIOUS WORK 11
tails increasing the number of computing nodes at each layer while vertical scalability
focuses on increasing computing capacity of the existing nodes. These tactics address
the bottlenecks at each computing node and can have significant impact on what we
defined as server time. However, they generally have minimal impact on the network
and client times.
Caching has also been used as a tactic to achieve better performance. It can refer to
caching of static content on the Web servers or caching dynamic content in memory of
the application server. We will refer to this concept as centralized caching. Centralized
refers to caching mechanism(s) that are implemented at the hosting facility and are
not geographically dispersed. The key limitation to the caching tactic is the overall
size of the computing environment and the incremental cost of adding new hardware
to support growth.
2.4 Previous work related to WWW environments
Web server behavior has been studied extensively since the early days of the In-
ternet [11, 13, 48] . Network considerations have also been given a lot of attention
with much research in the area of HTTP and TCP - the core protocols of the Inter-
net [12, 41, 51]. Workload characterization should be a key component of any perfor-
mance or scalability study regardless of the tactics used to achieve the two qualities of
service in an e-commerce environment. Early studies [11] in this space attempted to
generalize workload classifications in terms of Web server characteristics. Pitkow [53]
summarizes a group of traffic characterization studies on the World Wide Web until
1998. Subsequently, numerous characterization studies of Web server workloads were
conducted, including [9,10].
Menasce et al. [45] were first to extend workload modeling studies to include e-
commerce business transactions and not just Web server traffic. They presented a
technique to model workloads on the Web using the Customer Behavior Model Graph
profiles. We will use it to establish performance baseline and criteria to measure im-
provement in performance that we are expecting from implementing the CDN tactic.
Performance modeling of n-tier systems and the associated queuing networks was
also covered in [32]. Crovella et al. [27] conclude that e-commerce workloads tend
to vary very dynamically and exhibit short-term fluctuations. We agree with this as-
sessment based on several years’ worth of traffic measurements. Urgaonkar et al. [61]
represents some of the most recent work in performance modeling of multitier Inter-
net systems based on a network of queues that represent application tiers.
One of the key conclusions from the workload studies so far is that the popularity
of content elements on the Web follows a Zipf-like distribution [23]. Zipf distribution
supports the need for caching - considering that 25-40 percent of pieces of content
account for 70 percent requests. This concentration of access to key resources on the
site, such as Flash videos in the case of the system under study, strongly suggests that](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-19-320.jpg)
![CHAPTER 2. BACKGROUND AND PREVIOUS WORK 12
any meaningful level of caching should help performance as long as there exists a
cache replacement policy that minimizes the unavailability of the key resources. Bent
et al. [16] have determined that much of the workload of some of the busiest websites
remains uncacheable.
Perhaps the largest shortcoming of work to-date is the lack of analysis of correla-
tions between de-centralized content delivery tactics such as a CDN and performance
characteristics in the web, application server and database tiers. The existing body
of research exhibits a tendency towards evaluating individual tiers in the context of
content delivery instead of studying the relationships and interdependencies between
them. This is not surprising since the first implementations of websites focused on
serving static content. Our work builds on the existing research by augmenting the
queueing models with the new de-centralized content delivery tier and studying the
impact of such implementation across tiers.
We will need to use some sort of a benchmark in order to determine performance
gains from deployment of a CDN. Benchmarking of e-commerce applications has been
addressed by the Transaction Processing Council’s TPC-W (obsolete as of 2005) and
TPC-App specification [57]. Many websites have used TPC-W as a benchmark for
simulating the load on an e-commerce site [43]. We have the advantage of dealing
with a live website, so we do not need to use the TPC specification for load gener-
ation. However, we will refer to its performance metrics and response time capture
requirements in order to assess improvement.
Performance and scalability of Web-based environments have generated a lot of
interest within the research community. These environments usually involve static
requests for documents and dynamic requests for data or content generated by the
combination of application and database servers [36]. Iyengar’s paper also points out
that serving dynamic content can be orders of magnitude more CPU-intensive than
serving static content. Andreolini et al. [8] conclude that there is a need for differ-
ent levels of measurement granularity in order to identify bottlenecks. The levels are
classified into system-level, node level, resource level and component level metrics.
This study further classifies hardware resource utilizations into CPU, disk and net-
work utilization types. We also use classification in of bottlenecks could of client, net-
work and server constraints. Client bottlenecks are related to the ability to efficiently
render downloaded content within the browser. The diversity of personal comput-
ing environments, from Personal Digital Assistants to full-blown personal computers
contributes to a large variance in client-side performance.
Network bottlenecks result primarily from network congestion at the origin net-
work or the Wide Area Network [14]. The server-based bottlenecks result from inter-
tier interactions and associated CPU, I/O and memory utilization costs. Menasce et
al. [44] discuss the system bottlenecks and tactics to address them such as replicating
Web server queues into a cluster of Web servers in order better balance the load. We
will utilize the existing research around performance bottlenecks to identify metrics
that will help us prove the impact of the CDN tactic. The recurring theme in perfor-](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-20-320.jpg)
![CHAPTER 2. BACKGROUND AND PREVIOUS WORK 13
mance and scalability research is the continuous need offload as much of repetitive
processing from the websites as possible. For example, Challenger et al. [24] deter-
mined that the cost of generating a cacheable response is 500 times higher from the
CPU utilization standpoint than serving it from a cache.
Caching has been a part of many performance evaluations and recommendations
for improvement of the World Wide Web. Challenger et al. developed an approach
for consistently caching dynamic Web data that became a critical component of the
1998 Olympic Winter Games Website [25]. Performance results from implementing
a Web proxy cache as well as the impact of various cache replacement strategies are
also presented in [9]. Additionally, Arlit et al. present a number of workload char-
acteristics and their impact on effectiveness of caching. Rodriguez et al. [55] discuss
different types of Web caches including hierarchical and distributed and conclude that
geographic dispersion of content reduces the expected network distance to access a
document as well as band-width usage. Their results confirm the value proposition
of caching in general, but they stop short of recommending a decentralized caching
approach such as a CDN. Nahum’s paper [49] is one of the first efforts to consider
Wide Area Network effects in website performance studies. Wide Area Network’s
implications on performance are also discussed in [8]. A very detailed overview of
proxy-based caching with emphasis on dynamically generated content is also pre-
sented in [1]. They do not characterize the types of workload and do not compare the
response times with and without CDN as well resource utilization on the Web servers
and how a CDN impacts it.
The case for geographic distribution of content has been made by the research
community for quite some time [28, 52]. Perhaps the most comprehensive review of
CDN-related concepts is presented in [63]. We found Verma’s summary of the po-
tential applicability of a CDN to be particularly relevant. According to his work, the
application is a good CDN candidate if
• it has a high ratio of reads compared to writes
• client access patterns tend to access some set of objects more frequently
• limited windows of inconsistent data are acceptable
• data updates occur relatively slowly
Our analysis of the subject site shows consistency with the first three items, but
we have to concede that the data updates have to occur quickly in order to support
ever-changing content on the site. Nonetheless, the tradeoffs are well worth further
investigation of CDN’s applicability and the potential performance benefits. What we
found lacking in Verma’s work was a comprehensive analysis of sample websites that
have implemented a CDN and benefited from it. We had to look no further than [17]
to get a first glimpse of potential real-life benefits of CDNs using DNS redirection and
URL rewriting. This study was conducted on home pages of several hundred most](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-21-320.jpg)
![CHAPTER 2. BACKGROUND AND PREVIOUS WORK 14
popular websites as of November and December of 2000. One of the key findings was
that the networks performed significantly better than the origin sites even with the
overhead of DNS redirection. Other studies [39], confirm the CDN-related reduction
of download response times even with the DNS redirection tradeoff. Finally, Su et
al. [60] present a set of measurements for the Akamai delivery network along with
a detailed explanation of its architecture. This is the first study we are aware of that
presents data to support the network path selection efficiency gains that result from
implementing a CDN.
The key missing component in the research to-date is the impact of reduced down-
load times on e-commerce transactions as a whole from client, server, network and
consumer performance perspective. Also, due to proprietary nature of surrogate net-
works and e-commerce websites, there has been little more than anecdotal evidence
of quality attribute improvements for production environments. Our intent is to close
this gap with data that speaks to the impact of a CDN across the enterprise processing
tiers of a production e-commerce site.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-22-320.jpg)
![Chapter 3
Proposed Solution
Our solution will consist of several components that will need to be deployed, con-
figured and tested at our hosting facility. We will also need to make configuration
changes in our network, namely the DNS server, in order to re-route certain portions
of our website’s traffic to the chosen CDN. The high-level implementation steps can
be outlined as follows:
1. Select a CDN provider
2. Configure solution components
3. Implement network and application monitoring infrastructure
4. Test implementation
Web traffic re-routing almost always involves DNS - one of the core subsystems
of the Internet. We present a short overview of key DNS concepts as described in
[7,46,47]. Figure 3 depicts a typical series of steps in resolution of a URL to an IP ad-
dress for www.example.com. DNS is essentially a distributed database that follows
the client-server architecture. Adequate performance of DNS is achieved through
replication and caching. The server side portion of a request is handled by programs
called name servers. They contain information about a portion of the global database
and are capable of forwarding requests to other authoritative servers if necessary. The
information is made available to the client-side software components called resolvers.
A typical domain name on the Internet consists of two or more parts separated by
dots such as my.yahoo.com. Top-level domain (TLD) represents the rightmost por-
tion, .com in our case, while the subdomain(s) are represented by the labels to the left
of the top-level domain.
In our example, my.yahoo.com is a subdomain of yahoo.com, which in turn
belongs to the .com top-level domain. Finally, the hostname refers to a domain name
that has one or more IP addresses associated with it. Each domain or subdomain has
an authoritative server associated with it. It contains and publishes information about
22](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-30-320.jpg)
![CHAPTER 3. PROPOSED SOLUTION 23
Figure 3.1: How DNS works
the domain and any other domains it encapsulates. Root nameservers reside at the top
of the DNS hierarchy and they are queried first to resolve TLD. Caching and emphtime-
to-live (TTL) are very important concepts in DNS and, as we will later discover, in
CDN implementations [38,56]. IP mappings obtained from DNS can be stored in the
local resolver for a period of time as defined in TTL. This greatly reduces the load on
the DNS servers.
Figure 3 shows the usage of local cache in the resolution process. Our DNS config-
uration follows the configuration steps from [35] for the AIX 5.3 environment using
the BIND protocol. We have created a set of configuration files that will define the
name server mapping and resolvers and then we will start the appropriate processes
(daemons) on the server to activate DNS resolutions.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-31-320.jpg)
![CHAPTER 3. PROPOSED SOLUTION 24
Figure 3.2: Caching in DNS
3.1 CDN selection
The main purpose of a CDN is to direct consumer requests for objects to a server at
the optimal Internet location relative to the consumer’s location. The key components
of a CDN architecture are described in [37]. They are defined as: overlay network for-
mation, client request redirection, content routing and last-mile content delivery. The two
most common techniques employed by the networks are DNS redirection and URL
rewriting. Wills et al. [17] provide a good overview of these techniques and their effec-
tiveness. The DNS redirection technique utilizes a series of DNS resolutions based on
several factors such as server availability and network conditions with the purpose
of identifying the most suitable server. This identification step is usually encapsu-
lated into a series of algorithms that are proprietary to each network’s implementa-
tion. The end result is a DNS response with the IP address to the content server. The
response includes a time-to-live value that is usually limited to less than a minute (in
the case of Akamai it is 20 seconds). The TTL has to be set to a relatively low value
because the network conditions and server availability change constantly and quick
IP re-mapping is key.
The DNS redirection technique can facilitate either a full- or partial-site delivery.
Will full-site delivery, all requests to the origin server are directed using DNS to a
CDN server. If the CDN server can’t fulfill the request it simply routes it back to the
origin server. Several networks, including Adero and NetCaching, employ this deliv-
ery model. The main shortcoming of this model is the additional routing overhead
of wasted DNS requests that could have been handled by the origin server to begin
with. With partial-site content delivery, on the other hand, the origin site modifies the
URLs for certain objects or object directory locations to be resolved by the CDN’s DNS
server. This approach seems to be well suited for our website due to its combination of
static digital assets and dynamically generated server-side presentation components.
URL rewriting is another potential solution for server lookups. With this tech-
nique, the origin server continuously rewrites the URL links for dynamically gener-
ated pages in order to redirect them to the appropriate CDN server. The DNS func-](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-32-320.jpg)
![CHAPTER 3. PROPOSED SOLUTION 25
Figure 3.3: URL rewrite process
tionality remains on the origin site with this approach. When a page is requested by
the user it will be served from the origin server. However, before it is served, all of the
embedded links will be rewritten to point to the CDN’s DNS. Figure 3.1 shows a typi-
cal rewrite approach. The main drawback to the URL rewrite approach from the mea-
surement standpoint is the fact that the rewrites usually take place at the Web server
tier. Hence, the rewrite steps would inevitably introduce additional background noise
to our performance measurements. Therefore, we decided to avoid this approach for
the purpose of our study.
The latest list of available CDN implementations is maintained in [29]. At the time
of writing of this thesis we counted 18 different networks on Davison’s website. It
is not our primary purpose to evaluate tradeoffs between the various networks and
their implementations. The choice we made does not reflect a belief in superiority of
one network over others - it is merely a reflection of the need to get our experimental
test bed up and running as quickly as possible within boundaries imposed on us by
our existing hosting facility. For our implementation, we settled on partial-site, DNS
redirection-based CDN implementation using the Akamai delivery network.
3.2 Solution configuration
There exist several public sources that describe the DNS redirection mechanism im-
plemented by Akamai. We relied on [21, 40, 60] for more in-depth view of the inner
workings of this CDN. Akamai’s website and the available white papers [3] provide
the steps for configuring our network environment to redirect partial-site traffic. We
will start by explaining how Akamai works and then move on to our configuration
steps. Figure 3.2 depicts the DNS redirection process employed by this CDN.
In essence, Akamai performs a highly complex translation of a customer’s domain
to the IP address of the most suitable edge server. First, the Web browser requests](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-33-320.jpg)
![CHAPTER 3. PROPOSED SOLUTION 26
an HTML object. In order to accommodate this request, the local DNS resolver has
to translate the domain name into an IP address. The resolver issues a query to the
customer’s DNS server which in turn forwards the request to the Akamai network.
This is enabled via a configuration of a canonical name record (CNAME) in the origin
site’s DNS name server. The CNAME triggers the request redirection to the CDN.
Next, a hierarchy of Akamai servers responds to the request using the requestor’s
IP address, the name of the CDN customer, and the name of the requested content
as seeds for its DNS resolution. The CDN name resolution step is perhaps the most
critical in this sequence of events. Configuration of the Akamai CDN is described
in [4]. The steps for our deployment can be summarized as follows:
1. Create origin hostname
2. Activate Akamai edge hostname
3. Activate content delivery configuration
4. Point website to Akamai network
In our case, this process begins with configuration of a CNAME in our DNS name
server. A CNAME record maps an alias or nickname to the real name which may lie
outside the current zone. Typical format of a CNAME entry is as follows:
name ttl class rr canonical name
www IN CNAME joe.example.com.
We need to set up an origin server hostname that will resolve to our content server.
This server will be used by Akamai edge servers to retrieve our content, so it can be
made available to all of the nodes in the CDN. The naming convention for the origin
server is:
origin-<website>
where “website” refers to the is the hostname for our content that will be delivered
from Akamai. Our website stores all of its static content in the generic images folder,
so we will define the following origin server name:
origin-images.example.com for images.example.com
Next, we will create a DNS record for our origin server hostname on our authoritative
name server. We will use the CNAME record type for this step.
origin-www.example.com IN CNAME loadbalancer.example.com
We are now pointing our website to the Akamai network. An edge hostname will need
to be activated on an Akamai domain for our website using the CDN’s configuration
console. It will resolve to the Akamai network. For example, www.example.com
would have to point to www.example.edgesuite.net and www.example.edgesuite.net
would in turn resolve to individual servers on the Akamai network since it owns the
edgesuite.net domain. The remaining configuration steps need to be performed
in the configuration console of Akamai and they are covered in-depth in [4].](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-34-320.jpg)
![CHAPTER 3. PROPOSED SOLUTION 27
3.3 Monitoring infrastructure configuration
We will utilize three different types of monitoring solutions in the course of our per-
formance experiments. Monitoring infrastructure will help us obtain measurements
from three different perspectives: consumer, application and CDN. We need this full
view in order identify correlations between the CDN implementation and the key
performance characteristics in terms of system and user response times. The CDN
monitoring will aid the bandwidth scalability improvement assessment.
3.3.1 Performance measurement tools
The consumer perspective will be measured using a real-time user monitoring (RTUM)
service. RTUM services [18] provide details of user interactions with a website by
reporting on user application successes and failures. We will define a series of trans-
actions with sequences of browsing, clicks, object downloads and data input. This is
functionally equivalent to a consumer interacting with our website via a Web browser.
Transactions will be configured with our service provider and deployed into its net-
work of six different geographically dispersed origin locations within the U.S. with ac-
cess speeds ranging from dial-up to broadband. We will configure two different trans-
actions with multiple steps within them. The two types of transactions are necessary
because different paths through our website tend to have different content delivery
characteristics in terms of number of assets delivered and the number of server-side
components triggered. Our service provides very detailed HTML object-level report-
ing on transaction steps which will include going to the home page, catalog browsing,
product selection, adding an item to a shopping cart and checking out. The RTUM
time measurements will break the download time into slices consisting of DNS lookup
time, Web server connect time, content time and SSL time. Following are the definitions
for the key metrics used by our RTUM:
DNS look-up The process of calling a DNS server to lookup and convert a hostname
to an IP address. For instance, to convert www.foo.com to 10.0.0.1
Connect time The time it takes to connect to a Web server (or CDN edge server in our
case) across a network from a client browser or an RTUM agent
Secure sockets layer time The time it takes to create an SSL TCP/IP connection with
a website.
First byte time The time between the completion of the TCP connection with the des-
tination server that will provide the displayed page’s HTML, graphic, or other
component and the reception of the first packet (also known as first byte) for that
object.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-35-320.jpg)
![CHAPTER 3. PROPOSED SOLUTION 28
Content download time The time in seconds that measures the actual time to deliver
content (images, HTML, or other objects) from the Web server to the browser.
The application perspective will be captured using an appliance based application
monitoring solution. The network location of this appliance is depicted in Figure 3.3.1.
We will configure “watchpoints” using the appliance’s configuration tool to capture
server-side response times of Java EE components corresponding to the transaction
steps defined in the RTUM service. The appliance uses passive traffic analysis to cap-
ture actual transactions from the RTUM within our hosting environment and mea-
sures performance and availability of our e-commerce application as a whole. The im-
portant difference between this and other approaches is that our appliance does not
generate any traffic and the only performance overhead it introduces is reading the
copy of traffic from the network connection. The data is assembled into requests for
objects, pages and user sessions. Performance metrics include host, SSL and redirect
times. This solution also measures server errors or prematurely terminated connec-
tions due to increase in traffic. Figure 3.3.1 depicts the measurement timeline for a
sample request that would be captured by our appliance [26].
The appliance solution groups latency into the following six categories and defines
them as follows:
Host time This is the combined time the Web, application, and database servers take
to process a request. Host time is a key measure to assess performance impli-
cations of implementing a CDN on performance of our Java EE components
(servlet, EJBs, etc.). It can be very short in the case of a static image or long
in cases of long reports and complex server-side transactions such as adding a
list of items to the shopping basket.
Network time This is the time spent traveling across intervening networks. Once the
server has prepared its response, host time is over and network time begins. A
small object might be delivered quickly; a large one might take a long time. This
time is highly dependent on the type of consumer’s connection. Low-bandwidth
connections will result in higher network times and vice versa with broadband
connections. Our monitoring appliance also records additional information on
packet loss, out-of-order delivery, and round-trip time to help with this diagno-
sis.
SSL time The appliance will record the time spent negotiating the encryption of en-
crypted transactions. This portion of the SSL time represents the server-side
latency elements of the handshake versus the client-side SSL time captured by
the RTUM.
Redirect time This is the time the site spends sending a request on to other pages. In
some applications, a request for a page results in a redirect that usually points
elsewhere. This delay is recorded as redirect time.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-36-320.jpg)
![CHAPTER 3. PROPOSED SOLUTION 29
Idle time When a browser is retrieving a page, but there is no activity between ob-
jects on the same page, the HTTP interaction is defined as “idle”. This measure-
ment is key to understanding the amount of time spent processing client-side
scripts such as JavaScript. When there is inactivity in the middle of rendering
the page within the browser, our appliance will measure it as idle time.
End-to-end This is the total time for the object or page, from the moment the first
packet of a request is seen until the browser acknowledges delivery of the last
packet.
In order to quantify the benefits from implementing a CDN we will need to com-
bine server-side measurements taken by our application monitoring appliance with
client-side RTUM service data. The combined response timeline for the two monitor-
ing components is represented in Figure 3.3.1. We will capture measurements for all
elements of the above timeline and compare their pre- and post-CDN response times.
The individual latency components can also be summarized using the following two
equations:
Tee = Tdns + Tnetwork + Thost + Tsslclient + Tsslserver + Tidle + Thttpredirect (3.1)
where
Tnetwork = Tconnect + Tfirstbyte + Tdownload (3.2)
We will use them to assess the CDN’s impact on the latency as a whole in terms of
end-to-end and network times.
3.3.2 Scalability assessment framework and tools
Duboc et al. [30] argue that scalability of a system should be seen in the context of
its requirements. Scalability is often associated with performance and the terms have
at times been used interchangeably. However, performance is just one of the dimen-
sions used to represent scalability. We particularly agree with the authors’ assessment
that scalability is implicit on the relationship between cause and effect as observed
by a stakeholder. In our case, we define scalability as a relationship between the im-
plementation of a CDN on the following key dependent variables for our site and its
infrastructure: page views, CPU utilization, memory utilization, and bandwidth.
Based on the claims made by our CDN solution provider, we expect it to offload
a significant portion of connections to our Web server tier. We also expect a signifi-
cant decrease in CPU, memory and input/output (I/O) utilization of the Web tier that
should result in higher website capacity in terms of page views - the key dependent
variable of our scalability assessment. The third component where we should be able](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-37-320.jpg)
![CHAPTER 3. PROPOSED SOLUTION 30
to observe a significant improvement is the bandwidth usage of our environment at
the hosting facility since a significant portion of it will be served from the edge servers.
We will use a combination of monitoring reports available with our CDN and our in-
ternal server usage measurements to determine these scalability characteristics in the
context of traffic bursts our site experiences during the February peak week.
The internal server measurements will be taken using the vmstat UNIX util-
ity [33]. We will set the appropriate flags in vmstat in order to enable capturing of
utilization statistics for CPU and memory usage in the Production (Prod) environment
during the most traffic intensive time interval for our website - February 11th through
February 15th. The Prod environment consists of multiple servers in the Web and ap-
plication tiers and one server in the database tier. Typical operational reports for our
website include CPU and memory usage-related charts whose data is gathered via
vmstat. The examples of the reports are represented in Figure 3.3.2 and Figure 3.3.2.
The CPU utilization chart simply depicts the usage of all available CPU resources for
a given server. Servers may have multiple CPUs installed, so data is averaged out
across all of them for a given node. The data represents both kernel and any other
processes, so it gives a full picture for this particular resource.
Physical memory is a finite resource on any system. The UNIX memory handler
manages the memory allocations. The kernel is responsible for freeing up physical
memory of idle processes by saving it to disk until it is needed again. Paging and
swapping are used to accomplish this task. Paging refers to writing portions, termed
pages, of a process’ memory to disk. Swapping refers to writing the entire process, not
just part, to disk. Page-out represents the event of writing pages to disk, while page-in
is defined as retrieving memory data from disk. Page-ins are common and under nor-
mal circumstances are not a cause for concern. However, if page-ins become excessive
the kernel can reach a point where it’s actually spending more time managing paging
activity than running the applications, and system performance suffers. This state is
referred to as thrashing. Our memory measurements have focused on identifying ex-
cessive memory utilization in terms of paging characteristics as shown in Figure 3.3.2.
We will compare the pre-Akamai resource utilization for 2006 and post-Akamai
resource utilization for 2007 for this period of time. Figure 3.3.2 represents the traffic
patterns for the years 2005, 2006, and 2007 and it speaks to the type of traffic burstiness
our site experiences during the holiday. Even though year 2005 is not included in our
resource measurements, we decided to show it on the chart to illustrate the traffic
pattern during the peak week over the last several years. It’s important to note that
sudden drops in traffic for years 2005 and 2006 represent site outages due to Web
servers getting overloaded with incoming requests.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-38-320.jpg)
![Chapter 4
Quality of Service Evaluation
We have devised two different ways to measure performance and scalability impacts
resulting from implementing a CDN on our website. The need for two separate stud-
ies resulted from the fact that performance observations tend to be impacted by the
amount of load on the system under study. On the other hand, we need sufficient load
to assess scalability improvements. These two goals are mutually exclusive, so we de-
cided to take measurements in two different runtime environments. In order to have
a more controlled environment for performance measurements, we decided to obtain
response time experiments in our performance evaluation (PerfEval) environments.
It represents a scaled down replica of our production (Prod) environment based on
the rPerf benchmark measurement [31]. We determined that from the resource capac-
ity perspective our PerfEval environment represents approximately 17 percent of our
Prod capacity. We will discuss the details of the experiments in subsequent sections.
In order to assess scalability, we need to generate significant load on the produc-
tion environment and compare the results before and after we implement the CDN
configuration. A typical symptom of our scalability problems during key holidays is
that our Web servers get overloaded with HTTP connections from the browsers and
the CPU utilization spikes to above 90 percent when the Web servers stop respond-
ing to the incoming requests. Another key symptom of our scalability challenges is
related to the amount of bandwidth that has to be provided by our hosting facility.
Therefore, our scalability factors can be represented by three key benchmarks: CPU
and memory utilization, page views, and bandwidth. We decided to capture the measure-
ments during the most load-intensive holiday in February before and after the CDN
implementation in the full-scale Production environment.
4.1 Performance experiments
In order to fully represent the impact of the CDN on response times, we needed to de-
ploy our measurement transactions at geographically dispersed locations. Our RTUM
38](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-46-320.jpg)
![CHAPTER 4. QUALITY OF SERVICE EVALUATION 41
Performance results for each servlet will be aggregated by the appliance’s report-
ing components. A typical performance report generated by the appliance is repre-
sented by Figure 4.1.2. The key highlights of the performance report are the number of
requests to a given watchpoint along with their host, network and end-to-end times.
4.1.3 Bandwidth utilization monitoring
Our pre-CDN bandwidth utilization was captured using a combination of network
measurement tools available at the hosting facility. With the implementation of a
CDN, we also gained the ability to generate reports for bandwidth utilization coming
through the CDN. HTTP request traffic data includes the header size and any protocol
overhead. The HTTP response data includes object bytes plus overhead bytes along
with origin traffic (ingress), midgress traffic and edge traffic (or egress, all response
traffic from edge servers to end users [3]. Ingress traffic represents all response traf-
fic from our origin server to the Edge Servers, while the midgress includes responses
between Edge Servers. We will monitor several data elements in order to assess band-
width efficiency gains. They include:
Edge traffic, in requests per second The total number of requests for the content provider
(CP) and time period selected. This data includes all hits with all response codes
from the edge server to the end user.
Origin bandwidth, in Megabits per second The total bandwidth usage to our origin
servers for the CP code(s) and time period selected. This occurs when the Aka-
mai Edge Server cache does not have the requested content, and therefore the
edge server had to request it from the origin. This data includes the object bytes
plus overhead bytes, all ingress traffic and all HTTP response codes
Origin traffic, in hits per second The total number of hits on the origin for the CP
code(s) and time period selected. The peak number of hits per second and the
total origin hits are highlighted in the graph. The breakdown of response codes
is shown with different colors. This data includes all response codes served from
the origin to the edge server.
Origin bandwidth offload, in Megabits per second The origin offload view draws a
picture of the bandwidth reduction Akamai provides for our origin by showing
an aggregate view of site traffic from the edge compared to the traffic going to
our origin.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-49-320.jpg)
![CHAPTER 5. ANALYSIS 45
Figure 5.2: Operating system run queue: February of 2006 and 2007
The outages we experienced in previous years in the Web tier prevented a high
percentage of requests from even being forwarded to the application tier. With the
elimination of the Web tier bottleneck, we experienced higher CPU utilization in the
application tier and the database responsible for the shop transactions. This was to be
expected since a larger number of requests usually translates into higher CPU utiliza-
tion. However, when we accounted for the two times the amount of traffic on peak
day of the 14th between 2006 and 2007, we concluded that we still gained significant
scalability in the application tier. The two-percent increase of CPU utilization well
offsets the amount of the additional traffic the tier handled successfully. We suspect
that some of the efficiencies gained in the application tier were caused by the appli-
cation tuning activities that were undertaken prior to the holiday, similar to the steps
described in [58], but based on the results of the load tests prior and after the tuning
activities, we concluded that the contribution was minimal and most, if not all, of the
scalability gains could be attributed to the elimination of the Web tier bottleneck with
the CDN.
Perhaps the most unexpected result was uncovered in the database tier. One of the](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-53-320.jpg)
![CHAPTER 5. ANALYSIS 46
databases experienced 80 percent average CPU utilization decrease. We decided to
investigate it further and we discovered the main contributor to be the database tun-
ing activities performed by the engineering team prior to the peak week. Therefore,
we deemed the data invalid for analysis purposes and we concluded that additional
efficiencies could not be attributed to the CDN implementation.
Figure 5.1.1 further supports our conclusions for the Web and application tiers.
Generally speaking, the fewer number of threads waiting for execution in the run
queue results in lower CPU utilization. As in our CPU results, the most significant
impact on the run queues was on the peak traffic day - 2/14. The slight increase in
some cases between 2006 and 2007 is negligible from the CPU perspective as long as
the number oscillates around one or two threads. According to [34] and [6], blocking
I/O contributes to high run queue thread waits. The implications of the high cost of
I/O-related context switching are also discussed in [59]. We conclude that the blocking
I/O was the main culprit behind high CPU utilization prior to the CDN implemen-
tation because requests made directly to the Web servers resulted in significant I/O
waits for long-running large-object connections which, in turn, resulted in high I/O
blocking in the application tier. The offloading of a large number of HTTP connections
to the CDN has had a significant impact on the CPU utilization.
5.1.2 Memory utilization
We used three different measures to gauge the impact of the CDN on memory utiliza-
tion. The same level of memory resources was maintained throughout 2006 and 2007.
We observed the largest differences in memory paging activity. Retrievals of memory
pages from the paging space instead of RAM are considerably slower and constant
paging activity can cause system performance degradation. The most pronounced
impact on memory paging activity was observed on the peak day with average mem-
ory page-ins per second decreasing from 25 to almost zero and page-outs decreasing
from approximately 60 to one in the Web tier as shown in Figure 5.1.2 and Figure 5.1.2.
Interestingly enough, the paging activity increased on all other days during the peak
week, but the increase was not significant enough to cause performance problems or
outages.
Figure 5.1.2 represents the memory scanned-to-freed ratios. Overcommitting of
memory results in high scanned-to-freed ratios. It’s difficult to establish the level of
this ratio that will definitely result in memory being constrained to the point of having
an impact on performance. Overall, the ratios were neutral across all tiers and did
not seem to have a visible impact on scalability in either 2006 or 2007. All evidence
is pointing to the excessive paging activity as the main culprit of memory and CPU
utilization spikes in 2006. The impact of the CDN is probably indirect in the case of
memory utilization. The most plausible conclusion is that the lower number of I/O
waits due to offloading of HTTP connections results in a significant decrease of calls
from the kernel scheduler to the paging system which, in turn, alleviates the memory](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-54-320.jpg)
![CHAPTER 5. ANALYSIS 47
Figure 5.3: Memory page-ins, February of 2006 and 2007
constraints [54].
5.1.3 Bandwidth
Bandwidth costs and availability are key considerations for any highly transactional
e-commerce environment. Hosting facilities usually impose limits on bandwidth us-
age in order to allow many customers to share the connectivity to the Internet from
the hosting facility. Increasing bandwidth availability by a significant amount entails
significant costs during peaks due to market conditions. There are also significant
costs associated with deployment of network infrastructure to support bandwidth
increases. Therefore, bandwidth offloading represents a significant benefit for CDN
implementations.
The bandwidth impact in our environment between 2006 and 2007 peak weeks
is depicted in Figure 5.1.3. We determined that low bandwidth utilization in 2006
was caused by the refusal of a large number of requests in the Web tier and the sub-
sequent outages. However, even if we were to allow all of the requests we would](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-55-320.jpg)
![Bibliography
[1] A.Datta, K. Dutta, H. Thomas, D. Vandermeer, and A. Ramamritham. Proxy-
based acceleration of dynamically generated content on the world wide web: An
approach and implementation. ACM Transactions on Database Systems, 29(2):403–
443, 2004.
[2] V. Aggarwal, A. Feldmann, and C. Scheideler. Can isps and p2p users cooper-
ate for improved performance? SIGCOMM Computer Communications Review,
37(3):29–40, 2007.
[3] Akamai. www.akamai.com, 2007.
[4] Akamai. Akamai http content delivery configuration guide. https://dl.
akamai.com/customers/HTTPCD/HTTPCD_Activation_Guide.pdf,
2007.
[5] A. Akella, S. Seshan, and A. Shaikh. An empirical evaluation of wide-area inter-
net bottlenecks. In IMC ’03: Proceedings of the 3rd ACM SIGCOMM conference on
Internet measurement, pages 101–114, New York, NY, USA, 2003. ACM.
[6] J. Akella and D. Siewiorek. Modeling and measurement of the impact of in-
put/output on system performance. In ISCA ’91: Proceedings of the 18th annual
international symposium on Computer architecture, pages 390–399, New York, NY,
USA, 1991. ACM.
[7] P. Albitz and C. Liu. DNS and BIND. O’Reilly, Sebastopol, CA, 2001.
[8] M. Andreolini, M. Colajanni, R. Lancellotti, and F. Mazzoni. Fine grain perfor-
mance evaluation of e-commerce sites. SIGMETRICS Performance Evaluation Re-
view, 32(3):14–23, 2004.
[9] M. Arlitt, L. Cherkasova, J. Dilley, R. Friedrich, and T. Jin. Evaluating content
management techniques for web proxy caches. SIGMETRICS Performance Evalu-
ation Review, 27(4):3–11, 2000.
[10] M. Arlitt, D. Krishnamurthy, and J. Rolia. Characterizing the scalability of a large
web-based shopping system. ACM Transactions on Internet Technologies, 1(1):44–
69, 2001.
59](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-67-320.jpg)
![BIBLIOGRAPHY 60
[11] M. Arlitt and C. Williamson. Internet web servers: workload characterization
and performance implications. IEEE/ACM Transactions on Networking, 5(5):631–
645, 1997.
[12] H. Balakrishnan, V. Padmanabhan, S. Seshan, M. Stemm, and R. Katz. Tcp be-
havior of a busy internet server: analysis and improvements. INFOCOM ’98.
Seventeenth Annual Joint Conference of the IEEE Computer and Communications Soci-
eties. Proceedings. IEEE, 1:252–262, 29 Mar-2 Apr 1998.
[13] P. Barford and M. Crovella. A performance evaluation of hyper text transfer pro-
tocols. In SIGMETRICS ’99: Proceedings of the 1999 ACM SIGMETRICS interna-
tional conference on Measurement and modeling of computer systems, pages 188–197,
New York, NY, USA, 1999. ACM.
[14] Paul Barford and Mark Crovella. Measuring web performance in the wide area.
SIGMETRICS Performance Evaluation Review, 27(2):37–48, 1999.
[15] L. Bass, P. Clements, and R. Kazman. Software Architecture in Practice. Addison
Wesley, 1998.
[16] L. Bent, M. Rabinovich, G. M. Voelker, and Z. Xiao. Characterization of a large
web site population with implications for content delivery. In WWW ’04: Pro-
ceedings of the 13th international conference on World Wide Web, pages 522–533, New
York, NY, USA, 2004. ACM.
[17] B.Krishnamurthy, C. Wills, and Y. Zhang. On the use and performance of content
distribution networks. In IMW ’01: Proceedings of the 1st ACM SIGCOMM Work-
shop on Internet Measurement, pages 169–182, New York, NY, USA, 2001. ACM.
[18] B. Boardman. Real time monitoring products. http://www.
networkcomputing.com/showArticle.jhtml?queryText=
&articleID=175803253&pgno=4, 2006.
[19] B. Boiko. Content Management Bible. Wiley, 2nd edition, 2004.
[20] G. Booch, J. Rumbaugh, and I. Jacobson. The Unified Modeling Language user guide.
Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, USA, 1999.
[21] C. Bornstein, T. Canfield, G. Miller, and S. Rao. Optimal route selection in a
content delivery network. United States Patent 7,274,658, September 2007.
[22] P. Brebner and J. Gosper. How scalable is j2ee technology? SIGSOFT Software
Engineering Notes, 28(3):4–4, 2003.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-68-320.jpg)
![BIBLIOGRAPHY 61
[23] L. Breslau, C. Pei, L. Fan; G. Phillips, and S. Shenker. Web caching and zipf-like
distributions: evidence and implications. INFOCOM ’99. Eighteenth Annual Joint
Conference of the IEEE Computer and Communications Societies. Proceedings. IEEE,
1:126–134 vol.1, 21-25 Mar 1999.
[24] J. Challenger, P. Dantzig, and A. Iyengar. A scalable and highly available system
for serving dynamic data at frequently accessed web sites. In Supercomputing ’98:
Proceedings of the 1998 ACM/IEEE conference on Supercomputing (CDROM), pages
1–30, Washington, DC, USA, 1998. IEEE Computer Society.
[25] J. Challenger, P. Dantzig, A. Iyengar, M. Squillante, and L. Zhang. Efficiently
serving dynamic data at highly accessed web sites. IEEE/ACM Transactions on
Networking, 12(2):233–246, 2004.
[26] Coradiant. www.coradiant.com, 2007.
[27] M. Crovella and A. Bestavros. Self-similarity in world wide web traffic: evi-
dence and possible causes. In SIGMETRICS ’96: Proceedings of the 1996 ACM SIG-
METRICS international conference on Measurement and modeling of computer systems,
pages 160–169, New York, NY, USA, 1996. ACM.
[28] P. Danzig, R. Hall, and M. Schwartz. A case for caching file objects inside internet-
works. In SIGCOMM ’93: Conference proceedings on Communications architectures,
protocols and applications, pages 239–248, New York, NY, USA, 1993. ACM.
[29] B. Davison. Content delivery and distribution services. http://www.
web-caching.com/cdns.html, 2007.
[30] L. Duboc, D. Rosenblum, and T. Wicks. A framework for modelling and analysis
of software systems scalability. In ICSE ’06: Proceeding of the 28th international con-
ference on Software engineering, pages 949–952, New York, NY, USA, 2006. ACM.
[31] T. Farwell. It’s all relative: A guide to ibm’s rperf benchmark. http:
//www.ibmsystemsmag.com/opensystems/junejuly05/features/
6657p1.aspx, June 2005.
[32] G. Ferrari, P. Ezhilchelvan, and I. Mitrani. Performance modeling and evaluation
of e-business systems. In ANSS ’06: Proceedings of the 39th annual Symposium on
Simulation, pages 135–142, Washington, DC, USA, 2006. IEEE Computer Society.
[33] IBM. Vmstat Command Reference. IBM, 2006.
[34] IBM. CPU monitoring and tuning. IBM, 2007.
[35] IBM. Ibm aix 5.3 technical information. http://publib.boulder.ibm.com/
infocenter/systems/index.jsp?topic=/com.ibm.aix.howtos/doc/
howto/HT_commadmn_configure_dns.htm&tocNode=int_2049, 2007.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-69-320.jpg)
![BIBLIOGRAPHY 62
[36] A. Iyengar, E. Nahum, A. Shaikh, and R. Tewari. Communication Systems: The State
of the Art, chapter Enhancing Web Performance, pages 95–127. Kluwer Academic
Publishers, 2002.
[37] N. Jian, D. Tsang, I. Yeung, and H. Xiaojun. Hierarchical content routing in large-
scale multimedia content delivery network. Communications, 2003. ICC ’03. IEEE
International Conference on, 2:854–859, May 2003.
[38] J. Jung, E. Sit, H. Balakrishnan, and R. Morris. Dns performance and the effec-
tiveness of caching. In IMW ’01: Proceedings of the 1st ACM SIGCOMM Workshop
on Internet Measurement, pages 153–167, New York, NY, USA, 2001. ACM.
[39] M. Koletsou and G. Voelker. The medusa proxy: A tool for explor-
ing user-perceived web performance. http://citeseer.ist.psu.edu/
koletsou01medusa.html, 2001.
[40] F. Leighton and D. Lewin. Html delivery from edge-of-network servers in a con-
tent delivery network (cdn). United States Patent 7,293,093, November 2007.
[41] B. Mah. An empirical model of http network traffic. INFOCOM ’97. Sixteenth
Annual Joint Conference of the IEEE Computer and Communications Societies. Driving
the Information Revolution, 00:592, 1997.
[42] N. Medvidovic, D. Rosenblum, D. Redmiles, and J. Robbins. Modeling software
architectures in the unified modeling language. ACM Transactions on Software
Engineering Methodology, 11(1):2–57, 2002.
[43] D. Menasce. Tpc-w: a benchmark for e-commerce. Internet Computing, IEEE,
6(3):83–87, May/Jun 2002.
[44] D. Menasce, V. Almeida, and L. Dowdy. Performance by Design. Prentice Hall,
2004.
[45] D. Menasc´e, D. Barbar´a, and R. Dodge. Preserving qos of e-commerce sites
through self-tuning: a performance model approach. In EC ’01: Proceedings of the
3rd ACM conference on Electronic Commerce, pages 224–234, New York, NY, USA,
2001. ACM.
[46] P. Mockapetris. Ietf rfc 1034 - concepts and facilities of dns. http://www.ietf.
org/rfc/rfc1034.txt, 1987.
[47] P. Mockapetris. Ietf rfc 1035 - implementation and specification of dns. http:
//www.ietf.org/rfc/rfc1035.txt, 1987.
[48] J. Mogul. Network behavior of busy web server and its clients. DEC Western
Research Laboratory, Palo Alto, CA, 1995.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-70-320.jpg)
![BIBLIOGRAPHY 63
[49] E. Nahum, M. Rosu, S. Seshan, and J. Almeida. The effects of wide-area condi-
tions on www server performance. In SIGMETRICS ’01: Proceedings of the 2001
ACM SIGMETRICS international conference on Measurement and modeling of com-
puter systems, pages 257–267, New York, NY, USA, 2001. ACM.
[50] Nielsen. Sixth annual holiday e-spending report. http://www.
symboldomains.com/nn_ratingreport.pdf, 2006.
[51] H. Frystyk Nielsen, J. Gettys, A. Baird-Smith, E. Prud’hommeaux, H. Lie, and
C. Lilley. Network performance effects of http/1.1, css1, and png. In SIGCOMM
’97: Proceedings of the ACM SIGCOMM ’97 conference on Applications, technologies,
architectures, and protocols for computer communication, pages 155–166, New York,
NY, USA, 1997. ACM.
[52] V. Pai, M. Aron, G. Banga, M. Svendsen, P. Druschel, W. Zwaenepoel, and
E. Nahum. Locality-aware request distribution in cluster-based network servers.
In ASPLOS-VIII: Proceedings of the eighth international conference on Architectural
support for programming languages and operating systems, pages 205–216, New York,
NY, USA, 1998. ACM.
[53] J. Pitkow. World Wide Web, volume Volume 2, Numbers 1-2 of Computer Science,
chapter Summary of WWW characterizations, pages 3–13. Springer Netherlands,
June, 1999.
[54] J. Quarterman, A. Silberschatz, and J. Peterson. 4.2bsd and 4.3bsd as examples of
the unix system. ACM Computing Surveys, 17(4):379–418, 1985.
[55] P. Rodriguez, C. Spanner, and E. Biersack. Analysis of web caching architec-
tures: hierarchical and distributed caching. Networking, IEEE/ACM Transactions
on, 9(4):404–418, Aug 2001.
[56] A. Shaikh, R. Tewari, and M. Agrawal. On the effectiveness of dns-based server
selection. INFOCOM 2001. Twentieth Annual Joint Conference of the IEEE Computer
and Communications Societies. Proceedings. IEEE, 3:1801–1810 vol.3, 2001.
[57] W. Smith. Tpc-w: Benchmarking an ecommerce solution. http://tpc.org/
tpcw/TPC-W_Wh.pdf, 2000.
[58] M. Sopitkamol and D. Menasc´e. A method for evaluating the impact of software
configuration parameters on e-commerce sites. In WOSP ’05: Proceedings of the
5th international workshop on Software and performance, pages 53–64, New York,
NY, USA, 2005. ACM.
[59] J. Sterbenz and J. Touch. High-Speed Networking. John Wiley and Sons, 2001.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-71-320.jpg)
![BIBLIOGRAPHY 64
[60] A. Su, D. Choffnes, A. Kuzmanovic, and F. Bustamante. Drafting behind akamai
(travelocity-based detouring). In SIGCOMM ’06: Proceedings of the 2006 conference
on Applications, technologies, architectures, and protocols for computer communications,
pages 435–446, New York, NY, USA, 2006. ACM.
[61] B. Urgaonkar, G. Pacifici, P. Shenoy, M. Spreitzer, and A. Tantawi. Analytic mod-
eling of multitier internet applications. ACM Transactions on the Web, 1(1):2, 2007.
[62] USCB. E-stats - measuring the state of electronic economy. www.census.gov/
estats, 2007.
[63] D. Verma. Content Distribution Networks. Willey, 2002.](https://image.slidesharecdn.com/c2a743d5-98c3-4eb7-84fb-943e851be6ca-160418134932/85/Rzepnicki_thesis-72-320.jpg)
Rzepnicki_thesis
- 1. Evaluating the Impactof Content Delivery Networks on the Performance and Scalability of Content-Rich and Highly Transactional Java EE-based Websites Witold Rzepnicki Submitted to the Department of Electrical Engineering and Computer Science and the Faculty of the Graduate School of the University of Kansas in partial fulfillment of the requirements for the degree of Master’s of Science Committee Members: Hossein Saiedian, Ph.D. Professor and Thesis Adviser Arvin Agah, Ph.D., Associate Professor Prasad Kulkarni, Ph.D. Associate Professor Date Defended: February ??, 2008
- 2.
- 3. iii Abstract N-tier e-commerce environmentspresent unique performance and scalability chal- lenges under highly variable transactional loads. This thesis investigates Content De- livery Networks (CDNs) as a tactic to manage transactional peaks and valleys in a “bursty” environment that has suffered from content delivery-related performance, scalability and, ultimately, availability problems during key selling seasons. We estab- lished monitoring infrastructure that consists of client (browser) and server-side per- formance monitors as well as server and network resource utilization measurements in order to gauge the full impact of our chosen CDN on response times, bandwidth utilization and CPU utilization. We ran a series of controlled experiments to establish performance and scalabil- ity gains at the Java EE component level and HTML object level of granularity. Our results suggest a 30 percent improvement in page response times, a four-fold improve- ment in bandwidth utilization and Web server CPU utilization reduction of almost 90 percent. At the same time, we were able to sustain twice the amount of Web traffic with zero outages and the exact same hardware footprint year-over-year. Our conclu- sions are applicable to many other n-tier e-commerce environments that suffer from content delivery issues.
- 4.
- 5. Contents Acceptance ii Abstract iii Acknowledgementsiv 1 Introduction 1 1.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Justification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Significance and expected contributions . . . . . . . . . . . . . . . . . . . 3 1.4 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.5 Evaluation criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.6 Thesis organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Background and Previous Work 8 2.1 Architectural overview of subject site . . . . . . . . . . . . . . . . . . . . 9 2.2 Website content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Performance considerations and tactics . . . . . . . . . . . . . . . . . . . 10 2.4 Previous work related to WWW environments . . . . . . . . . . . . . . . 11 3 Proposed Solution 22 3.1 CDN selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2 Solution configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3 Monitoring infrastructure configuration . . . . . . . . . . . . . . . . . . . 27 3.3.1 Performance measurement tools . . . . . . . . . . . . . . . . . . . 27 3.3.2 Scalability assessment framework and tools . . . . . . . . . . . . 29 4 Quality of Service Evaluation 38 4.1 Performance experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.1.1 Transaction types . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.1.2 Server-side transaction capture . . . . . . . . . . . . . . . . . . . . 40 4.1.3 Bandwidth utilization monitoring . . . . . . . . . . . . . . . . . . 41 v
- 6. CONTENTS vi 5 Analysis43 5.1 Scalability and availability observations . . . . . . . . . . . . . . . . . . . 44 5.1.1 CPU utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 5.1.2 Memory utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.1.3 Bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 5.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 5.2.1 Consumer Perspective . . . . . . . . . . . . . . . . . . . . . . . . . 50 5.2.2 Server-side Perspective . . . . . . . . . . . . . . . . . . . . . . . . 52 6 Conclusions and Further Research 56 6.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 6.2 Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Bibliography 59
- 7. List of Figures 1.1Queue network for a typical n-tier e-commerce system . . . . . . . . . . 5 2.1 Sample customer behavior model graph . . . . . . . . . . . . . . . . . . . 15 2.2 Subject site page views by month, 2004-2007 . . . . . . . . . . . . . . . . 16 2.3 Logical view of subject site . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.4 Infrastructure diagram of system under study . . . . . . . . . . . . . . . 18 2.5 Websphere Commerce interaction diagram . . . . . . . . . . . . . . . . . 19 2.6 Typical approach for content management and publishing . . . . . . . . 20 2.7 Location of Internet bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . 21 3.1 How DNS works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2 Caching in DNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.3 URL rewrite process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.4 How Akamai works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.5 Application monitoring appliance network location . . . . . . . . . . . . 32 3.6 Application monitoring appliance measurement timeline . . . . . . . . . 33 3.7 Content delivery measurement combined timeline . . . . . . . . . . . . . 34 3.8 Sample CPU utilization report . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.9 Memory page-in report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.10 Hourly page views for 2/11-2/15, 2005-2007 . . . . . . . . . . . . . . . . 37 4.1 Sample server-side performance report . . . . . . . . . . . . . . . . . . . 42 5.1 CPU utilization, February of 2006 and 2007 . . . . . . . . . . . . . . . . . 44 5.2 Operating system run queue: February of 2006 and 2007 . . . . . . . . . 45 5.3 Memory page-ins, February of 2006 and 2007 . . . . . . . . . . . . . . . . 47 5.4 Memory page-outs, February of 2006 and 2007 . . . . . . . . . . . . . . . 48 5.5 Memory scan rate, February of 2006 and 2007 . . . . . . . . . . . . . . . . 49 5.6 Peak bandwidth comparison, February of 2006 and 2007 . . . . . . . . . 50 5.7 Akamai bandwidth offload, February of 2007 . . . . . . . . . . . . . . . . 51 5.8 RTUM shop transaction response times in seconds . . . . . . . . . . . . . 52 5.9 RTUM browse transaction response times in seconds . . . . . . . . . . . 54 vii
- 8. List of Tables 1.1Subject site’s content characteristics, 1998-2007 . . . . . . . . . . . . . . . 2 1.2 Annual e-commerce customer satisfaction survey results . . . . . . . . . 3 4.1 Performance test nodes by ISP and location . . . . . . . . . . . . . . . . . 39 4.2 RTUM transaction characteristics . . . . . . . . . . . . . . . . . . . . . . . 40 5.1 RTUM browse transaction performance . . . . . . . . . . . . . . . . . . . 53 5.2 RTUM shop transaction performance . . . . . . . . . . . . . . . . . . . . . 53 5.3 Server-side performance in milliseconds . . . . . . . . . . . . . . . . . . . 53 5.4 TCP packet counts at hosting environment . . . . . . . . . . . . . . . . . 55 viii
- 9. Chapter 1 Introduction Electronic commerce(e-commerce) dates back to late 1970s when the first electronic data interchange (EDI) systems were introduced to facilitate the exchange of data be- tween companies and institutions. The initial types of data exchange were limited to invoices and purchase orders and required significant financial investment in the computer networks and hardware. The first EDI-based e-commerce networks were later augmented with other forms of e-commerce such as automated teller machines (ATM) and telephone banking in the 1980s and subsequently enterprise resource plan- ning, data mining and data warehouse systems in the early 1990s. In late 1990s and early 2000s, the term e-commerce started to include consumer- based shopping transactions over the World Wide Web (WWW). These transactions typically include a combination of browsing of a website’s catalog and its related con- tent in the form of HTML, image files and marketing videos followed by the presenta- tion of e-shopping cart and check out transactions supported by secure payment au- thorization, generally over the HTTPs protocol. The United States Census Bureau [62] defines several important concepts related to this topic. For example, it defines e- commerce as “any transaction completed over a computer-mediated network that in- volves the transfer of ownership or rights to use goods or services.” This definition does not differentiate between monetary and non-monetary transactions as well zero price and paid transactions. The concept of e-business infrastructure is also defined as “the share of total economic infrastructure to support electronic business and conduct electronic commerce”. The examples of infrastructure thus include the computers, routers, satellites, network channels, system and application software, website host- ing and programmers. 1.1 Problem statement The goal of this thesis is to evaluate the impact of a Content Delivery Network on scalability and performance of a content-rich and highly transactional website that utilizes the Java Enterprise Edition (EE) platform. Since its inception in 1996 the web- 1
- 10. CHAPTER 1. INTRODUCTION2 Table 1.1: Subject site’s content characteristics, 1998-2007 Content Type Size 1998 2004 2007 Documents 0 KB 28 KB 34 KB Images 36 KB 42 KB 96 KB Scripts 0 KB 2 KB 20 KB StyleSheets 0 KB 0 KB 62 KB Total 36 KB 72 KB 181 KB site that comprises the experimental test bed of this thesis has dealt with significant scalability and performance challenges around major holidays. These challenges can be classified as transactional and content delivery-related. Over the last several years the subject site has been able to tune its system infrastructure to achieve acceptable trans- actional performance in the application server and database server tiers. However, it has struggled with Web server content delivery. As a matter of fact, it has not been able to service all of the requests during its most content-intensive peak week in February for the last several years. This is mostly due to the popularity of the free customiz- able Flash videos available on the website. Web servers responsible for delivering the Flash videos frequently get overloaded with long-running connections that result in high CPU utilization in the Web tier and a domino effect on the processing queues in the application server tier. We have decided to investigate the tactic of implementing a Content Delivery Network to help address the issues with content delivery not just for this particular peak week, but for the website in general. 1.2 Justification Text-only and informational websites from mid to late 1990s have transformed into rich and highly interactive environments that offer not only physical goods, but also services and digital assets such as MP3 files (e.g., itunes.com) for download. Ad- ditional features such as wish lists, consumer product reviews and message boards have also been introduced in order to aid consumers in objective product selection. Some sites even offer interactive chat-based product selection advice. This conver- gence of services and products as well as content presentation and interaction has re- sulted in much larger Web pages and increased diversity of e-commerce transactions. We used web.archive.org website as a source for a comparison of the subject web- site’s home page characteristics for years 1998, 2004 and 2007 in Table 1.2. The number of downloadable page elements has increased from 15 in 1998 to 28 in 2007 and the overall size of the home page has increased from 36 KB to 181 KB. Another intriguing aspect of the subject site’s growth is increased reliance on scripts
- 11. CHAPTER 1. INTRODUCTION3 Table 1.2: Annual e-commerce customer satisfaction survey results Satisfaction Level 2005 2004 2003 2002 2002 vs 2005 change Very Satisfied 40% 37% 40% 37% 3% Somewhat Satisfied 24% 24% 23% 22% 2% Neutral 31% 32% 30% 33% -3% Somewhat Dissatisfied 4% 5% 5% 5% -1% Very Dissatisfied 2% 3% 3% 3% -1% that run within browsers to enable rich consumer experience. The number of objects on HTML pages is the key indicator of the number of connections that will need to be opened between the client (browser) and the Web server. While e-commerce still represents only a minor fraction of total retail sales, it is quickly becoming a very important component of any major retailer’s presence in the marketplace. One study of European consumers shows increase in online participa- tion from 42 percent to 56 percent between 2003 and 2006. Forrester Research also re- ports that in 2006, on average, 80 percent of consumers reported researching products online with approximately 55 percent reporting either an online or offline purchase in the preceding three months. Nielsen Research [50] confirms this growth trend, but its research also points out that consumer satisfaction from online purchases has experi- enced only a slight growth as shown in the Table 1.2. Current level of consumer satisfaction underscores the need to continuously im- prove customer experience on the Web, especially with regards to website perfor- mance. More than half of online shoppers cite response time as a key influence on their shopping behavior. Consumers have come to expect fast, uninterrupted and relevant shopping experience as well as scalability during major holidays and contin- uous availability of services. Content delivery has become a critical component of the online shopping experience and Content Delivery Networks (CDNs) were established with the primary purpose of addressing issues with performance and scalability. 1.3 Significance and expected contributions Current model of content delivery on the Internet can be classified as centralized in a sense that a majority of the websites use the origin servers at their hosting environ- ments to serve content. User requests have to travel through several networks and encounter a myriad of bottlenecks including Peering, First Mile, Last Mile and Backbone - all of which are inherent to the way Internet works [2,3,5]. CDNs have emerged as collections of servers located in close proximity to the Internet backbone, and thus to the end user, with the purpose of reducing the number of “hops” between network
- 12. CHAPTER 1. INTRODUCTION4 nodes. This overlay network approach allows shorter routing paths and more efficient DNS routing that helps alleviate all but the Last Mile bottleneck. CDN companies claim that by running thousands of servers in hundreds of networks in geograph- ically dispersed locations they can deliver a significant performance and scalability boost to e-commerce websites. We will review the merits of this claim in our study of the subject site. We will also review the impact of the potential network benefits realized from implementation of a CDN on all processing tiers of a Java EE-based e- commerce environment. Additionally, we will evaluate tradeoffs that may result from deploying this solution. Based on our analysis of the subject site’s page profile, we concluded that the av- erage number of objects (images, cascading style sheets, JavaScript and Flash files) is approximately 40 items with the size of between 1 KB for a simple image to 400 KB for a Flash movie. While the load of 100 million page views per month probably would not pose a problem for the Web servers for 1 KB items, the story is quite different for the 400 KB items since they will take up a much longer connection time. Also, each hosting facility imposes limits associated with amount of network bandwidth avail- able to each customer. It is not unusual for the subject site to reach this limit during the key holidays. 1.4 Methodology Many studies to-date have focused on evaluating the network effects of implementing a CDN on the Internet Service Providers in terms of bandwidth efficiency and network traffic patterns. In these cases, the terms performance and scalability refer to the ability of the network to sustain “bursty” traffic patterns within reasonable response times during peaks with minimal changes in the hardware footprint. The methodologies used to measure the impact have revolved around implementing a network monitor that captured traffic patterns and characteristics of objects passing through the net- work. To our knowledge, there has been no research to-date that established a methodol- ogy to measure the correlations between performance improvements in content deliv- ery and individual tiers of n-tier e-commerce environments. We assert that in order to identify meaningful conclusions in such research the methodologies must encompass considerations of all tiers. Our approach is to conduct a more thorough case study of one particular highly transactional e-commerce environment with the purpose of identifying conclusions that could be applicable to a variety of n-tier platforms. In order to improve validity of our conclusions, we decided to analyze and char- acterize the workload for the subject site for one year prior to implementing the pro- posed solution and one year after. We will use a combination of network and ap- plication monitoring technologies for measurements. Our methodology will also en- compass software-level component measurements that will aid in analysis of relation-
- 13. CHAPTER 1. INTRODUCTION5 ships between content delivery and server-side Java EE components of the site’s ar- chitecture. This granularity is necessary due to inherent inter-dependencies between processing tiers when queuing networks are involved. Figure 1.1: Queue network for a typical n-tier e-commerce system Typical performance model of an e-commerce site could be represented by the queue network in Figure 1.4 according to [44]. According to this model, the bottle- necks in the database (DB) server tier could result in requests queuing up at the Web server tier and vice versa. Also, performance improvements in any of the tiers will most likely result in either a positive or negative impact on other tiers. 1.5 Evaluation criteria Our evaluation framework will include consumer-centric and system-centric metrics. Consumer-centric metrics focus on response times as observed by the users of the site. We will implement an appliance-based monitoring solution that captures HTTP and HTTPS traffic at the URI stem leave and provides a high-level view of application re- sponse times. It will give us benchmarks for key function points of our e-commerce solution, including browsing of Category and Product Detail pages as well as Shop- ping Cart, Billing and Order Confirmation components. The tool will provide us with measurements in terms of network, client and server time. We will capture measure- ments from before and after the CDN implementation.
- 14. CHAPTER 1. INTRODUCTION6 The existing website traffic monitoring solution will be utilized to capture traffic characteristics in terms of typical e-commerce measures such as page views, visits and orders. This data will enhance our application monitoring with aggregated load metrics. System metrics, especially the correlation between system resource utilization (CPU, memory and I/O) and content delivery offloading, will also need to be studied in order to measure benefits from implementing a CDN in terms of hardware footprint reduction and bandwidth cost efficiency. We will use a set of typical UNIX server utilities to measure CPU, memory and network interface utilization for servers in all three tiers. We expect that making our content available via a network of geographically dis- persed servers will have a significant impact on bandwidth utilization. Most CDNs provide a monitoring console that allows customers to measure the origin server of- fload effect using several metrics. We will use the console to measure the size and types of website objects that most benefit from this solution and to quantify savings in terms of bandwidth and the number of connections to the Web servers that were avoided. Our expectation is that the CDN will offload a significant amount of band- width from the network of our hosting environment. Current limitation imposed on our subject site by the hosting facility is 1 Gbps (gigabit per second). The evaluation would not be complete without tradeoff analysis of our tactic. Cur- rent research points to manageability and the speed of content propagation as key concerns when implementing a CDN. In this case, manageability refers to the abil- ity to deploy configuration and content changes into the CDN. We will measure the amount of time necessary to have content changes reflected in the overlay network from the origin server as well the amount of time needed to deploy configuration changes. 1.6 Thesis organization The thesis will be organized as follows: • Chapter 1: Introduction - The background and justification for the research presented in this thesis. • Chapter 2: Background and Previous Work - The architectural background of the subject site and related work in measuring and characterization of perfor- mance and scalability of WWW environments. • Chapter 3: Proposed Solution - The selection and implementation of a Content Delivery Network and performance monitoring tools. • Chapter 4: Quality of Service Evaluation: Performance and Scalability - The impact of the CDN on the subject site and its processing tiers.
- 15. CHAPTER 1. INTRODUCTION7 • Chapter 5: Analysis - Validation of quantitative and qualitative results of the implementation • Chapter 6: Conclusions and Further Research - The conclusions resulting from this research and proposed areas for further investigations
- 16. Chapter 2 Background andPrevious Work Bass et al. [15] give a generic definition of performance as responsiveness of the system characterized by the time required to respond to events. The level of performance is often described by the number of user or system transactions per unit of time. In e- commerce, this concept encompasses several measures of throughput in terms of Web page response times and number of page views for a given period of time. It can also be expressed in terms of responsiveness and throughput capacity of key server- side Java EE components such as Servlets and Enterprise Java Beans. Each site has a unique traffic profile, but a generic traffic pattern can be represented by the Customer Behavior Model (CBM) state transition graph in Figure 2 as proposed by Menasce et al. in [45]. The CBM graph identifies several key states in the browsing and shopping expe- rience. The numbers for each state transition represent its probability. Hence, there is a 30 percent chance of a customer selecting the Search page from the Browse page. Performance of a website can thus be characterized by the response times for each of these states under a particular level of volume of user site visits. A visit consists of multiple requests to different components of the browsing and shopping experience. For the purpose of this study, we will further breakdown the performance of each behavior point into three components: network time, client time and server time. The network time is the amount time that a request and response pairs spend traveling to and from the load balancers. The client time represents the amount of time it takes the Web browser to issue an HTTP request and render the response in the form of HTML and other objects. The server time is the amount of time each request spends on the Web, application and database servers. Scalability represents the ability for a given site to handle its projected traffic vol- ume as the number of requests for the critical points grows while maintaining its cur- rent hardware footprint. The latter is particularly important as most websites find it difficult to significantly change their footprint during a critical business period. For example, the subject website experiences up to four-fold increases in page views and a 15-fold increase in bandwidth usage during some of its critical business periods. 8
- 17. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 9 Figure 2 represents the subject site’s transactional bursts during key shopping holi- days. Brebner et al. [22] provide several benchmark-based insights into the potential scalability of J2EE environments. 2.1 Architectural overview of subject site A typical e-commerce website can be represented in a series of views that depict its functional components, software modules and hardware infrastructure as proposed by [20, 42]. For the purpose of this study we will analyze a website that has existed since 1996 and was recently (2003) re-architected using the IBM Websphere Commerce software components using the Java Enterprise Edition (Java EE) platform. The site’s business process view is represented in Figure 2.1. The business process view high- lights several key functional components of the subject website, namely the catalog, marketing and order management features. The functional view is supported by the hardware and software components as shown in Figure 2.1 and Figure 2.1. 2.2 Website content In the realm of e-commerce, content refers to any information made available for con- sumers to download. This includes catalog data, HTML pages, images, audio files, video files in Flash or other formats and even software downloads. Efficient man- agement and deployment of content-related assets have thus become mission critical activities for many organizations that depend on the Internet as a sales channel. E- commerce websites rely on three types of assets to support the selling function: cat- alog data assets, content assets and application assets. Catalog assets are used to define products with characteristics such as price or product name. Content assets represent editorial content such as marketing information, product reviews and promotional information. Application assets include software components created by software de- velopers that are generally managed separately from the other two types of assets. Content can also be classified as dynamic or static. Dynamic content is generated upon consumer’s request and it may be unique for each request. It is typically assembled on the application server with involvement of the DBMS. Static content is represented by HTML, client-side scripts (JavaScript), images and other multimedia assets that do not change from request to request. This type of content generally resides on Web server(s) and is served up from there. Figure 2.2 represents a typical approach to management and publishing of static and dynamic content. It is important to note that generally the static content will be published to a file system and dynamic con- tent will be made available via server side components with support of a DBMS. The fundamental concepts of content management and publishing are covered in [19]. Even if we were to achieve infinite scalability and performance at the hosting fa- cility we would still have a dependency on the Internet as a whole for delivery of
- 18. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 10 content to the browsers. The Internet could not function efficiently unless the indi- vidual networks that make its backbone can also communicate with one another. The process of exchanging data between two networks is called peering and it takes place between two routers also known as peering points located at the edge of each net- work. The two networks use an exterior gateway protocol called the Border Gateway Protocol (BGP) to send traffic between them. The interior gateway protocol (OSPF and RIP) is used to route traffic within a network. Hence, the peering points and rout- ing protocols connect different networks into a unified infrastructure that allows the Internet to function efficiently. Internet is made up of many different networks that communicate with one another using the IP protocol. Individual networks consist of a variety of routers, switch-es and fiber in order to move data packets within the net- work. The structure of “network of networks” results in several potential scalability and performance bottlenecks. They are generally classified as: first mile, peering points, backbone and last mile. Figure 2.2 represents the types and location of the bottlenecks as described by [3]. The First Mile bottleneck refers to the limitations in the website’s connectivity to the Internet via its Internet Service Provider (ISP). In order to reach desired scalability it needs to continuously expand its connectivity to the ISP. The ISP, in turn, must also expand its capacity in order to meet its customers’ scalability requirements. Peering points also represent potential bottlenecks as large networks are not economically mo- tivated to scale up the number of peering points with the networks of their competi- tors, especially since a significant portion of the traffic handled by the peering points is transit traffic with packets originating on other networks. This lack of competitive and financial motivation over time has resulted in a limited number of peering points across major networks. The Backbone Problem refers to the fact that the ISPs’ router capacity has historically not kept up with growth of traffic demands. Finally, the Last Mile problem reflects the limited capacity of a typical user’s connection to their ISP. It’s important to note that just solving one of the above bottlenecks, such as the Last Mile, by increasing the reach of broadband connectivity at home will not automat- ically address the other limitations. These need to be treated as separate problems that, if addressed, would help solve the problem as a whole. 2.3 Performance considerations and tactics Most websites measure their performance and scalability using a combination of user and system-centric metrics. Page response times represent the user-centric metric while the CPU, memory and disk utilization of the web, application and database servers illustrate the system dimension of performance. High consumer expectations have led many highly-transactional websites to implement a variety of technology strategies to achieve a combination of high performance and scalability [10]. Typical approaches include scaling both horizontally and vertically. Horizontal scalability en-
- 19. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 11 tails increasing the number of computing nodes at each layer while vertical scalability focuses on increasing computing capacity of the existing nodes. These tactics address the bottlenecks at each computing node and can have significant impact on what we defined as server time. However, they generally have minimal impact on the network and client times. Caching has also been used as a tactic to achieve better performance. It can refer to caching of static content on the Web servers or caching dynamic content in memory of the application server. We will refer to this concept as centralized caching. Centralized refers to caching mechanism(s) that are implemented at the hosting facility and are not geographically dispersed. The key limitation to the caching tactic is the overall size of the computing environment and the incremental cost of adding new hardware to support growth. 2.4 Previous work related to WWW environments Web server behavior has been studied extensively since the early days of the In- ternet [11, 13, 48] . Network considerations have also been given a lot of attention with much research in the area of HTTP and TCP - the core protocols of the Inter- net [12, 41, 51]. Workload characterization should be a key component of any perfor- mance or scalability study regardless of the tactics used to achieve the two qualities of service in an e-commerce environment. Early studies [11] in this space attempted to generalize workload classifications in terms of Web server characteristics. Pitkow [53] summarizes a group of traffic characterization studies on the World Wide Web until 1998. Subsequently, numerous characterization studies of Web server workloads were conducted, including [9,10]. Menasce et al. [45] were first to extend workload modeling studies to include e- commerce business transactions and not just Web server traffic. They presented a technique to model workloads on the Web using the Customer Behavior Model Graph profiles. We will use it to establish performance baseline and criteria to measure im- provement in performance that we are expecting from implementing the CDN tactic. Performance modeling of n-tier systems and the associated queuing networks was also covered in [32]. Crovella et al. [27] conclude that e-commerce workloads tend to vary very dynamically and exhibit short-term fluctuations. We agree with this as- sessment based on several years’ worth of traffic measurements. Urgaonkar et al. [61] represents some of the most recent work in performance modeling of multitier Inter- net systems based on a network of queues that represent application tiers. One of the key conclusions from the workload studies so far is that the popularity of content elements on the Web follows a Zipf-like distribution [23]. Zipf distribution supports the need for caching - considering that 25-40 percent of pieces of content account for 70 percent requests. This concentration of access to key resources on the site, such as Flash videos in the case of the system under study, strongly suggests that
- 20. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 12 any meaningful level of caching should help performance as long as there exists a cache replacement policy that minimizes the unavailability of the key resources. Bent et al. [16] have determined that much of the workload of some of the busiest websites remains uncacheable. Perhaps the largest shortcoming of work to-date is the lack of analysis of correla- tions between de-centralized content delivery tactics such as a CDN and performance characteristics in the web, application server and database tiers. The existing body of research exhibits a tendency towards evaluating individual tiers in the context of content delivery instead of studying the relationships and interdependencies between them. This is not surprising since the first implementations of websites focused on serving static content. Our work builds on the existing research by augmenting the queueing models with the new de-centralized content delivery tier and studying the impact of such implementation across tiers. We will need to use some sort of a benchmark in order to determine performance gains from deployment of a CDN. Benchmarking of e-commerce applications has been addressed by the Transaction Processing Council’s TPC-W (obsolete as of 2005) and TPC-App specification [57]. Many websites have used TPC-W as a benchmark for simulating the load on an e-commerce site [43]. We have the advantage of dealing with a live website, so we do not need to use the TPC specification for load gener- ation. However, we will refer to its performance metrics and response time capture requirements in order to assess improvement. Performance and scalability of Web-based environments have generated a lot of interest within the research community. These environments usually involve static requests for documents and dynamic requests for data or content generated by the combination of application and database servers [36]. Iyengar’s paper also points out that serving dynamic content can be orders of magnitude more CPU-intensive than serving static content. Andreolini et al. [8] conclude that there is a need for differ- ent levels of measurement granularity in order to identify bottlenecks. The levels are classified into system-level, node level, resource level and component level metrics. This study further classifies hardware resource utilizations into CPU, disk and net- work utilization types. We also use classification in of bottlenecks could of client, net- work and server constraints. Client bottlenecks are related to the ability to efficiently render downloaded content within the browser. The diversity of personal comput- ing environments, from Personal Digital Assistants to full-blown personal computers contributes to a large variance in client-side performance. Network bottlenecks result primarily from network congestion at the origin net- work or the Wide Area Network [14]. The server-based bottlenecks result from inter- tier interactions and associated CPU, I/O and memory utilization costs. Menasce et al. [44] discuss the system bottlenecks and tactics to address them such as replicating Web server queues into a cluster of Web servers in order better balance the load. We will utilize the existing research around performance bottlenecks to identify metrics that will help us prove the impact of the CDN tactic. The recurring theme in perfor-
- 21. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 13 mance and scalability research is the continuous need offload as much of repetitive processing from the websites as possible. For example, Challenger et al. [24] deter- mined that the cost of generating a cacheable response is 500 times higher from the CPU utilization standpoint than serving it from a cache. Caching has been a part of many performance evaluations and recommendations for improvement of the World Wide Web. Challenger et al. developed an approach for consistently caching dynamic Web data that became a critical component of the 1998 Olympic Winter Games Website [25]. Performance results from implementing a Web proxy cache as well as the impact of various cache replacement strategies are also presented in [9]. Additionally, Arlit et al. present a number of workload char- acteristics and their impact on effectiveness of caching. Rodriguez et al. [55] discuss different types of Web caches including hierarchical and distributed and conclude that geographic dispersion of content reduces the expected network distance to access a document as well as band-width usage. Their results confirm the value proposition of caching in general, but they stop short of recommending a decentralized caching approach such as a CDN. Nahum’s paper [49] is one of the first efforts to consider Wide Area Network effects in website performance studies. Wide Area Network’s implications on performance are also discussed in [8]. A very detailed overview of proxy-based caching with emphasis on dynamically generated content is also pre- sented in [1]. They do not characterize the types of workload and do not compare the response times with and without CDN as well resource utilization on the Web servers and how a CDN impacts it. The case for geographic distribution of content has been made by the research community for quite some time [28, 52]. Perhaps the most comprehensive review of CDN-related concepts is presented in [63]. We found Verma’s summary of the po- tential applicability of a CDN to be particularly relevant. According to his work, the application is a good CDN candidate if • it has a high ratio of reads compared to writes • client access patterns tend to access some set of objects more frequently • limited windows of inconsistent data are acceptable • data updates occur relatively slowly Our analysis of the subject site shows consistency with the first three items, but we have to concede that the data updates have to occur quickly in order to support ever-changing content on the site. Nonetheless, the tradeoffs are well worth further investigation of CDN’s applicability and the potential performance benefits. What we found lacking in Verma’s work was a comprehensive analysis of sample websites that have implemented a CDN and benefited from it. We had to look no further than [17] to get a first glimpse of potential real-life benefits of CDNs using DNS redirection and URL rewriting. This study was conducted on home pages of several hundred most
- 22. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 14 popular websites as of November and December of 2000. One of the key findings was that the networks performed significantly better than the origin sites even with the overhead of DNS redirection. Other studies [39], confirm the CDN-related reduction of download response times even with the DNS redirection tradeoff. Finally, Su et al. [60] present a set of measurements for the Akamai delivery network along with a detailed explanation of its architecture. This is the first study we are aware of that presents data to support the network path selection efficiency gains that result from implementing a CDN. The key missing component in the research to-date is the impact of reduced down- load times on e-commerce transactions as a whole from client, server, network and consumer performance perspective. Also, due to proprietary nature of surrogate net- works and e-commerce websites, there has been little more than anecdotal evidence of quality attribute improvements for production environments. Our intent is to close this gap with data that speaks to the impact of a CDN across the enterprise processing tiers of a production e-commerce site.
- 23. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 15 Figure 2.1: Sample customer behavior model graph
- 24. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 16 Figure 2.2: Subject site page views by month, 2004-2007
- 25. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 17 J2EE Application Server E-Commerce Engine Administration Tools Admin. Console Commerce Accelerator Organization Admin. Console Business Context Engine Policies Roles and Relationships Analytics Business Process Personalization Globalization Business Process Subsystems Member Subsystem Trading Subsystem Catalog Subsystem Marketing Subsystem Merchandising Subsystem Inventory Subsystem Order Subsystem Business Intelligence Websphere Commerce Payments Server E-Commerce Engine Database create content retrieve data manage business rules settle payments Content Management Portal Content Management CM Console Content Publishing Content Integration Utility Content Data Sourcing Scripts Data & Content create create Site Search Engine Components Consumer Search Engine Indexing Rules Indexed Objects index data & content execute search CM Database create source www.example.com view content access store features Figure 2.3: Logical view of subject site
- 26. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 18 Figure 2.4: Infrastructure diagram of system under study
- 27. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 19 Figure 2.5: Websphere Commerce interaction diagram
- 28. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 20 Figure 2.6: Typical approach for content management and publishing
- 29. CHAPTER 2. BACKGROUNDAND PREVIOUS WORK 21 Figure 2.7: Location of Internet bottlenecks
- 30. Chapter 3 Proposed Solution Oursolution will consist of several components that will need to be deployed, con- figured and tested at our hosting facility. We will also need to make configuration changes in our network, namely the DNS server, in order to re-route certain portions of our website’s traffic to the chosen CDN. The high-level implementation steps can be outlined as follows: 1. Select a CDN provider 2. Configure solution components 3. Implement network and application monitoring infrastructure 4. Test implementation Web traffic re-routing almost always involves DNS - one of the core subsystems of the Internet. We present a short overview of key DNS concepts as described in [7,46,47]. Figure 3 depicts a typical series of steps in resolution of a URL to an IP ad- dress for www.example.com. DNS is essentially a distributed database that follows the client-server architecture. Adequate performance of DNS is achieved through replication and caching. The server side portion of a request is handled by programs called name servers. They contain information about a portion of the global database and are capable of forwarding requests to other authoritative servers if necessary. The information is made available to the client-side software components called resolvers. A typical domain name on the Internet consists of two or more parts separated by dots such as my.yahoo.com. Top-level domain (TLD) represents the rightmost por- tion, .com in our case, while the subdomain(s) are represented by the labels to the left of the top-level domain. In our example, my.yahoo.com is a subdomain of yahoo.com, which in turn belongs to the .com top-level domain. Finally, the hostname refers to a domain name that has one or more IP addresses associated with it. Each domain or subdomain has an authoritative server associated with it. It contains and publishes information about 22
- 31. CHAPTER 3. PROPOSEDSOLUTION 23 Figure 3.1: How DNS works the domain and any other domains it encapsulates. Root nameservers reside at the top of the DNS hierarchy and they are queried first to resolve TLD. Caching and emphtime- to-live (TTL) are very important concepts in DNS and, as we will later discover, in CDN implementations [38,56]. IP mappings obtained from DNS can be stored in the local resolver for a period of time as defined in TTL. This greatly reduces the load on the DNS servers. Figure 3 shows the usage of local cache in the resolution process. Our DNS config- uration follows the configuration steps from [35] for the AIX 5.3 environment using the BIND protocol. We have created a set of configuration files that will define the name server mapping and resolvers and then we will start the appropriate processes (daemons) on the server to activate DNS resolutions.
- 32. CHAPTER 3. PROPOSEDSOLUTION 24 Figure 3.2: Caching in DNS 3.1 CDN selection The main purpose of a CDN is to direct consumer requests for objects to a server at the optimal Internet location relative to the consumer’s location. The key components of a CDN architecture are described in [37]. They are defined as: overlay network for- mation, client request redirection, content routing and last-mile content delivery. The two most common techniques employed by the networks are DNS redirection and URL rewriting. Wills et al. [17] provide a good overview of these techniques and their effec- tiveness. The DNS redirection technique utilizes a series of DNS resolutions based on several factors such as server availability and network conditions with the purpose of identifying the most suitable server. This identification step is usually encapsu- lated into a series of algorithms that are proprietary to each network’s implementa- tion. The end result is a DNS response with the IP address to the content server. The response includes a time-to-live value that is usually limited to less than a minute (in the case of Akamai it is 20 seconds). The TTL has to be set to a relatively low value because the network conditions and server availability change constantly and quick IP re-mapping is key. The DNS redirection technique can facilitate either a full- or partial-site delivery. Will full-site delivery, all requests to the origin server are directed using DNS to a CDN server. If the CDN server can’t fulfill the request it simply routes it back to the origin server. Several networks, including Adero and NetCaching, employ this deliv- ery model. The main shortcoming of this model is the additional routing overhead of wasted DNS requests that could have been handled by the origin server to begin with. With partial-site content delivery, on the other hand, the origin site modifies the URLs for certain objects or object directory locations to be resolved by the CDN’s DNS server. This approach seems to be well suited for our website due to its combination of static digital assets and dynamically generated server-side presentation components. URL rewriting is another potential solution for server lookups. With this tech- nique, the origin server continuously rewrites the URL links for dynamically gener- ated pages in order to redirect them to the appropriate CDN server. The DNS func-
- 33. CHAPTER 3. PROPOSEDSOLUTION 25 Figure 3.3: URL rewrite process tionality remains on the origin site with this approach. When a page is requested by the user it will be served from the origin server. However, before it is served, all of the embedded links will be rewritten to point to the CDN’s DNS. Figure 3.1 shows a typi- cal rewrite approach. The main drawback to the URL rewrite approach from the mea- surement standpoint is the fact that the rewrites usually take place at the Web server tier. Hence, the rewrite steps would inevitably introduce additional background noise to our performance measurements. Therefore, we decided to avoid this approach for the purpose of our study. The latest list of available CDN implementations is maintained in [29]. At the time of writing of this thesis we counted 18 different networks on Davison’s website. It is not our primary purpose to evaluate tradeoffs between the various networks and their implementations. The choice we made does not reflect a belief in superiority of one network over others - it is merely a reflection of the need to get our experimental test bed up and running as quickly as possible within boundaries imposed on us by our existing hosting facility. For our implementation, we settled on partial-site, DNS redirection-based CDN implementation using the Akamai delivery network. 3.2 Solution configuration There exist several public sources that describe the DNS redirection mechanism im- plemented by Akamai. We relied on [21, 40, 60] for more in-depth view of the inner workings of this CDN. Akamai’s website and the available white papers [3] provide the steps for configuring our network environment to redirect partial-site traffic. We will start by explaining how Akamai works and then move on to our configuration steps. Figure 3.2 depicts the DNS redirection process employed by this CDN. In essence, Akamai performs a highly complex translation of a customer’s domain to the IP address of the most suitable edge server. First, the Web browser requests
- 34. CHAPTER 3. PROPOSEDSOLUTION 26 an HTML object. In order to accommodate this request, the local DNS resolver has to translate the domain name into an IP address. The resolver issues a query to the customer’s DNS server which in turn forwards the request to the Akamai network. This is enabled via a configuration of a canonical name record (CNAME) in the origin site’s DNS name server. The CNAME triggers the request redirection to the CDN. Next, a hierarchy of Akamai servers responds to the request using the requestor’s IP address, the name of the CDN customer, and the name of the requested content as seeds for its DNS resolution. The CDN name resolution step is perhaps the most critical in this sequence of events. Configuration of the Akamai CDN is described in [4]. The steps for our deployment can be summarized as follows: 1. Create origin hostname 2. Activate Akamai edge hostname 3. Activate content delivery configuration 4. Point website to Akamai network In our case, this process begins with configuration of a CNAME in our DNS name server. A CNAME record maps an alias or nickname to the real name which may lie outside the current zone. Typical format of a CNAME entry is as follows: name ttl class rr canonical name www IN CNAME joe.example.com. We need to set up an origin server hostname that will resolve to our content server. This server will be used by Akamai edge servers to retrieve our content, so it can be made available to all of the nodes in the CDN. The naming convention for the origin server is: origin-<website> where “website” refers to the is the hostname for our content that will be delivered from Akamai. Our website stores all of its static content in the generic images folder, so we will define the following origin server name: origin-images.example.com for images.example.com Next, we will create a DNS record for our origin server hostname on our authoritative name server. We will use the CNAME record type for this step. origin-www.example.com IN CNAME loadbalancer.example.com We are now pointing our website to the Akamai network. An edge hostname will need to be activated on an Akamai domain for our website using the CDN’s configuration console. It will resolve to the Akamai network. For example, www.example.com would have to point to www.example.edgesuite.net and www.example.edgesuite.net would in turn resolve to individual servers on the Akamai network since it owns the edgesuite.net domain. The remaining configuration steps need to be performed in the configuration console of Akamai and they are covered in-depth in [4].
- 35. CHAPTER 3. PROPOSEDSOLUTION 27 3.3 Monitoring infrastructure configuration We will utilize three different types of monitoring solutions in the course of our per- formance experiments. Monitoring infrastructure will help us obtain measurements from three different perspectives: consumer, application and CDN. We need this full view in order identify correlations between the CDN implementation and the key performance characteristics in terms of system and user response times. The CDN monitoring will aid the bandwidth scalability improvement assessment. 3.3.1 Performance measurement tools The consumer perspective will be measured using a real-time user monitoring (RTUM) service. RTUM services [18] provide details of user interactions with a website by reporting on user application successes and failures. We will define a series of trans- actions with sequences of browsing, clicks, object downloads and data input. This is functionally equivalent to a consumer interacting with our website via a Web browser. Transactions will be configured with our service provider and deployed into its net- work of six different geographically dispersed origin locations within the U.S. with ac- cess speeds ranging from dial-up to broadband. We will configure two different trans- actions with multiple steps within them. The two types of transactions are necessary because different paths through our website tend to have different content delivery characteristics in terms of number of assets delivered and the number of server-side components triggered. Our service provides very detailed HTML object-level report- ing on transaction steps which will include going to the home page, catalog browsing, product selection, adding an item to a shopping cart and checking out. The RTUM time measurements will break the download time into slices consisting of DNS lookup time, Web server connect time, content time and SSL time. Following are the definitions for the key metrics used by our RTUM: DNS look-up The process of calling a DNS server to lookup and convert a hostname to an IP address. For instance, to convert www.foo.com to 10.0.0.1 Connect time The time it takes to connect to a Web server (or CDN edge server in our case) across a network from a client browser or an RTUM agent Secure sockets layer time The time it takes to create an SSL TCP/IP connection with a website. First byte time The time between the completion of the TCP connection with the des- tination server that will provide the displayed page’s HTML, graphic, or other component and the reception of the first packet (also known as first byte) for that object.
- 36. CHAPTER 3. PROPOSEDSOLUTION 28 Content download time The time in seconds that measures the actual time to deliver content (images, HTML, or other objects) from the Web server to the browser. The application perspective will be captured using an appliance based application monitoring solution. The network location of this appliance is depicted in Figure 3.3.1. We will configure “watchpoints” using the appliance’s configuration tool to capture server-side response times of Java EE components corresponding to the transaction steps defined in the RTUM service. The appliance uses passive traffic analysis to cap- ture actual transactions from the RTUM within our hosting environment and mea- sures performance and availability of our e-commerce application as a whole. The im- portant difference between this and other approaches is that our appliance does not generate any traffic and the only performance overhead it introduces is reading the copy of traffic from the network connection. The data is assembled into requests for objects, pages and user sessions. Performance metrics include host, SSL and redirect times. This solution also measures server errors or prematurely terminated connec- tions due to increase in traffic. Figure 3.3.1 depicts the measurement timeline for a sample request that would be captured by our appliance [26]. The appliance solution groups latency into the following six categories and defines them as follows: Host time This is the combined time the Web, application, and database servers take to process a request. Host time is a key measure to assess performance impli- cations of implementing a CDN on performance of our Java EE components (servlet, EJBs, etc.). It can be very short in the case of a static image or long in cases of long reports and complex server-side transactions such as adding a list of items to the shopping basket. Network time This is the time spent traveling across intervening networks. Once the server has prepared its response, host time is over and network time begins. A small object might be delivered quickly; a large one might take a long time. This time is highly dependent on the type of consumer’s connection. Low-bandwidth connections will result in higher network times and vice versa with broadband connections. Our monitoring appliance also records additional information on packet loss, out-of-order delivery, and round-trip time to help with this diagno- sis. SSL time The appliance will record the time spent negotiating the encryption of en- crypted transactions. This portion of the SSL time represents the server-side latency elements of the handshake versus the client-side SSL time captured by the RTUM. Redirect time This is the time the site spends sending a request on to other pages. In some applications, a request for a page results in a redirect that usually points elsewhere. This delay is recorded as redirect time.
- 37. CHAPTER 3. PROPOSEDSOLUTION 29 Idle time When a browser is retrieving a page, but there is no activity between ob- jects on the same page, the HTTP interaction is defined as “idle”. This measure- ment is key to understanding the amount of time spent processing client-side scripts such as JavaScript. When there is inactivity in the middle of rendering the page within the browser, our appliance will measure it as idle time. End-to-end This is the total time for the object or page, from the moment the first packet of a request is seen until the browser acknowledges delivery of the last packet. In order to quantify the benefits from implementing a CDN we will need to com- bine server-side measurements taken by our application monitoring appliance with client-side RTUM service data. The combined response timeline for the two monitor- ing components is represented in Figure 3.3.1. We will capture measurements for all elements of the above timeline and compare their pre- and post-CDN response times. The individual latency components can also be summarized using the following two equations: Tee = Tdns + Tnetwork + Thost + Tsslclient + Tsslserver + Tidle + Thttpredirect (3.1) where Tnetwork = Tconnect + Tfirstbyte + Tdownload (3.2) We will use them to assess the CDN’s impact on the latency as a whole in terms of end-to-end and network times. 3.3.2 Scalability assessment framework and tools Duboc et al. [30] argue that scalability of a system should be seen in the context of its requirements. Scalability is often associated with performance and the terms have at times been used interchangeably. However, performance is just one of the dimen- sions used to represent scalability. We particularly agree with the authors’ assessment that scalability is implicit on the relationship between cause and effect as observed by a stakeholder. In our case, we define scalability as a relationship between the im- plementation of a CDN on the following key dependent variables for our site and its infrastructure: page views, CPU utilization, memory utilization, and bandwidth. Based on the claims made by our CDN solution provider, we expect it to offload a significant portion of connections to our Web server tier. We also expect a signifi- cant decrease in CPU, memory and input/output (I/O) utilization of the Web tier that should result in higher website capacity in terms of page views - the key dependent variable of our scalability assessment. The third component where we should be able
- 38. CHAPTER 3. PROPOSEDSOLUTION 30 to observe a significant improvement is the bandwidth usage of our environment at the hosting facility since a significant portion of it will be served from the edge servers. We will use a combination of monitoring reports available with our CDN and our in- ternal server usage measurements to determine these scalability characteristics in the context of traffic bursts our site experiences during the February peak week. The internal server measurements will be taken using the vmstat UNIX util- ity [33]. We will set the appropriate flags in vmstat in order to enable capturing of utilization statistics for CPU and memory usage in the Production (Prod) environment during the most traffic intensive time interval for our website - February 11th through February 15th. The Prod environment consists of multiple servers in the Web and ap- plication tiers and one server in the database tier. Typical operational reports for our website include CPU and memory usage-related charts whose data is gathered via vmstat. The examples of the reports are represented in Figure 3.3.2 and Figure 3.3.2. The CPU utilization chart simply depicts the usage of all available CPU resources for a given server. Servers may have multiple CPUs installed, so data is averaged out across all of them for a given node. The data represents both kernel and any other processes, so it gives a full picture for this particular resource. Physical memory is a finite resource on any system. The UNIX memory handler manages the memory allocations. The kernel is responsible for freeing up physical memory of idle processes by saving it to disk until it is needed again. Paging and swapping are used to accomplish this task. Paging refers to writing portions, termed pages, of a process’ memory to disk. Swapping refers to writing the entire process, not just part, to disk. Page-out represents the event of writing pages to disk, while page-in is defined as retrieving memory data from disk. Page-ins are common and under nor- mal circumstances are not a cause for concern. However, if page-ins become excessive the kernel can reach a point where it’s actually spending more time managing paging activity than running the applications, and system performance suffers. This state is referred to as thrashing. Our memory measurements have focused on identifying ex- cessive memory utilization in terms of paging characteristics as shown in Figure 3.3.2. We will compare the pre-Akamai resource utilization for 2006 and post-Akamai resource utilization for 2007 for this period of time. Figure 3.3.2 represents the traffic patterns for the years 2005, 2006, and 2007 and it speaks to the type of traffic burstiness our site experiences during the holiday. Even though year 2005 is not included in our resource measurements, we decided to show it on the chart to illustrate the traffic pattern during the peak week over the last several years. It’s important to note that sudden drops in traffic for years 2005 and 2006 represent site outages due to Web servers getting overloaded with incoming requests.
- 39. CHAPTER 3. PROPOSEDSOLUTION 31 Figure 3.4: How Akamai works
- 40. CHAPTER 3. PROPOSEDSOLUTION 32 Figure 3.5: Application monitoring appliance network location
- 41. CHAPTER 3. PROPOSEDSOLUTION 33 Figure 3.6: Application monitoring appliance measurement timeline
- 42. CHAPTER 3. PROPOSEDSOLUTION 34 Figure 3.7: Content delivery measurement combined timeline
- 43. CHAPTER 3. PROPOSEDSOLUTION 35 Figure 3.8: Sample CPU utilization report
- 44. CHAPTER 3. PROPOSEDSOLUTION 36 Figure 3.9: Memory page-in report
- 45. CHAPTER 3. PROPOSEDSOLUTION 37 Figure 3.10: Hourly page views for 2/11-2/15, 2005-2007
- 46. Chapter 4 Quality ofService Evaluation We have devised two different ways to measure performance and scalability impacts resulting from implementing a CDN on our website. The need for two separate stud- ies resulted from the fact that performance observations tend to be impacted by the amount of load on the system under study. On the other hand, we need sufficient load to assess scalability improvements. These two goals are mutually exclusive, so we de- cided to take measurements in two different runtime environments. In order to have a more controlled environment for performance measurements, we decided to obtain response time experiments in our performance evaluation (PerfEval) environments. It represents a scaled down replica of our production (Prod) environment based on the rPerf benchmark measurement [31]. We determined that from the resource capac- ity perspective our PerfEval environment represents approximately 17 percent of our Prod capacity. We will discuss the details of the experiments in subsequent sections. In order to assess scalability, we need to generate significant load on the produc- tion environment and compare the results before and after we implement the CDN configuration. A typical symptom of our scalability problems during key holidays is that our Web servers get overloaded with HTTP connections from the browsers and the CPU utilization spikes to above 90 percent when the Web servers stop respond- ing to the incoming requests. Another key symptom of our scalability challenges is related to the amount of bandwidth that has to be provided by our hosting facility. Therefore, our scalability factors can be represented by three key benchmarks: CPU and memory utilization, page views, and bandwidth. We decided to capture the measure- ments during the most load-intensive holiday in February before and after the CDN implementation in the full-scale Production environment. 4.1 Performance experiments In order to fully represent the impact of the CDN on response times, we needed to de- ploy our measurement transactions at geographically dispersed locations. Our RTUM 38
- 47. CHAPTER 4. QUALITYOF SERVICE EVALUATION 39 Table 4.1: Performance test nodes by ISP and location ISP City and State Level3 Los Angeles, CA Savvis Santa Clara, CA Verizon Denver, CO MFN Washington, D.C. Internap Miami, FL Level3 Chicago, IL Sprint New York, NY service allowed us to use seven transaction nodes as described in Table 4.1. We de- cided not to use international testing locations due to additional cost and the fact that almost 80 percent of our website’s traffic originates from within the continental U.S. 4.1.1 Transaction types We recorded two types of scripts with two distinct sequences of transaction steps: one for the browse transaction profile and one for shop transaction profile. The browse transaction consists of the following steps: 1. Access the Website’s Home Page 2. Go to four different levels of Category pages 3. Access Flash video product detail page 4. Personalize Flash video 5. Sign In to the website 6. Send Flash video to recipient The shop transaction type consists of the following steps: 1. Access the Website’s Home Page 2. Go to the top Category page for one of our product types 3. Go to Product Detail page for one of the products (it remains constant through- out the experiment) 4. Select delivery options and go to the Shipping and Billing page
- 48. CHAPTER 4. QUALITYOF SERVICE EVALUATION 40 Table 4.2: RTUM transaction characteristics Transaction workload characteristics Browse Shop Number of transaction steps 9 6 Number of images retrieved 163 94 Number of scripts, HTML, CSS, Flash components 57 39 Number of server-side J2EE components accessed 12 15 Average image size 2.9 KB 2.8 KB Average size of HTML, script and Flash 4.9 KB 5.8 KB Total number of bytes retrieved per connection 250 KB 98 KB Number of web-server connections initiated from the browser 4 5 5. Submit payment information and go to Order review page The key difference between the two transaction types are the types of server-side Java EE components as well as the amount of content that gets served by the Web servers. The browse transaction focuses more on content delivery characteristics of large Flash video files while the shop transaction emphasizes server-side functionality. The combination of the two transactions should provide us with adequate coverage of content delivery and server-side performance. Table 4.1.1 summarizes the key char- acteristics of the two transaction types in terms of workload, types of items retrieved and their size as well as server-side load generated by the transactions in the form of Java EE component requests. In order to gather a representative data sample, we decided to schedule the two transactions to run uninterrupted for 48 hours with the CDN re-routing enabled and for another 48 hours without the CDN. The RTUM will initiate six transactions for each transaction type and for each node per hour. This will give us a sample of over a thousand transactions for each test’s duration. 4.1.2 Server-side transaction capture We will enable server-side performance monitoring using a series of “watchpoints” that will be defined in our application monitoring appliance. The watchpoints are typically configured at the servlet level. We identified 21 unique servlets that will be accessed from the two transaction sequences. We dedicated one full instance of the ap- pliance to monitor traffic directed to our PerfEval environment by the RTUM service. This will enable us to filter out traffic destined to our production environment and get more accurate server-side measurements. Typical configuration steps for a servlet in- clude defining the URI stem to be intercepted and any additional filtering parameters. Since our environment is isolated from other traffic, we can simply configure the URI stems to be intercepted in the configuration utility.
- 49. CHAPTER 4. QUALITYOF SERVICE EVALUATION 41 Performance results for each servlet will be aggregated by the appliance’s report- ing components. A typical performance report generated by the appliance is repre- sented by Figure 4.1.2. The key highlights of the performance report are the number of requests to a given watchpoint along with their host, network and end-to-end times. 4.1.3 Bandwidth utilization monitoring Our pre-CDN bandwidth utilization was captured using a combination of network measurement tools available at the hosting facility. With the implementation of a CDN, we also gained the ability to generate reports for bandwidth utilization coming through the CDN. HTTP request traffic data includes the header size and any protocol overhead. The HTTP response data includes object bytes plus overhead bytes along with origin traffic (ingress), midgress traffic and edge traffic (or egress, all response traffic from edge servers to end users [3]. Ingress traffic represents all response traf- fic from our origin server to the Edge Servers, while the midgress includes responses between Edge Servers. We will monitor several data elements in order to assess band- width efficiency gains. They include: Edge traffic, in requests per second The total number of requests for the content provider (CP) and time period selected. This data includes all hits with all response codes from the edge server to the end user. Origin bandwidth, in Megabits per second The total bandwidth usage to our origin servers for the CP code(s) and time period selected. This occurs when the Aka- mai Edge Server cache does not have the requested content, and therefore the edge server had to request it from the origin. This data includes the object bytes plus overhead bytes, all ingress traffic and all HTTP response codes Origin traffic, in hits per second The total number of hits on the origin for the CP code(s) and time period selected. The peak number of hits per second and the total origin hits are highlighted in the graph. The breakdown of response codes is shown with different colors. This data includes all response codes served from the origin to the edge server. Origin bandwidth offload, in Megabits per second The origin offload view draws a picture of the bandwidth reduction Akamai provides for our origin by showing an aggregate view of site traffic from the edge compared to the traffic going to our origin.
- 50. CHAPTER 4. QUALITYOF SERVICE EVALUATION 42 Figure 4.1: Sample server-side performance report
- 51. Chapter 5 Analysis This chapteris divided into sections that discuss the respective quality of service (QoS) attribute impacts resulting from the implementation of the CDN. We undertook two separate evaluations of performance and scalability qualities of service. The scala- bility evaluation focused on comparing a variety of resource utilization metrics cap- tured during the February peak transaction period for years 2006 and 2007. We antic- ipated increased throughput of transactions as well as page views while maintaining the same hardware footprint and resource utilization. Better scalability typically re- sults in better availability and the two QoS are frequently viewed as derivatives of each other. Therefore, we will include the availability observations in the scalability section. In order to assess the performance impact of the CDN, we ran a series of experi- ments for with pre- and post-CDN environments for browse and shop types of transac- tions. With the monitoring infrastructure in place, we were able to gather performance at the object level of granularity. An object in our case represents the lowest level of granularity from the browser perspective in terms of how it renders content and it corresponds to an image (JPEG, GIF or PNG), a script (JavaScript or otherwise), Flash SWF files, HTML content, style sheets and any other low-level components that a web page may consist of. We anticipated improvement in end-to-end download times for the objects. Assuming that each web page consists of tens or hundreds of them, we expected the performance improvement of each individual object to add up to a sig- nificant impact on the overall performance of the RTUM transactions. It was unclear to us how a CDN may impact server-side performance of JSPs and Servlets, especially under steady-state, low-volume conditions, so we decided to capture server-side per- formance metrics as well. 43
- 52. CHAPTER 5. ANALYSIS44 Figure 5.1: CPU utilization, February of 2006 and 2007 5.1 Scalability and availability observations 5.1.1 CPU utilization We maintained the same number of CPUs and thus the rPerf value for the Production environment throughout 2006 and 2007. However, the CPU allocation changed in the application server and Web server tiers. We decreased the available resources by 50 percent in the web tier and allocated them to the application server. It was our expectation that the CDN would allow us to change the ratios in favor of the application tier since we expected a majority of the resource utilization decrease to be visible in the web tier. We made no changes to the database tier. Figure 5.1.1 depicts CPU utilization across all tiers after accounting for resource allocation changes. As we anticipated, the Web tier experienced the most impact from the CDN - an average of 57 percent utilization decrease during the peak week and close to 60 percent on the highest traffic day. Meanwhile, the overall traffic increased almost five-fold from the previous day (see Figure 3.3.2 for peak week traffic pattern).
- 53. CHAPTER 5. ANALYSIS45 Figure 5.2: Operating system run queue: February of 2006 and 2007 The outages we experienced in previous years in the Web tier prevented a high percentage of requests from even being forwarded to the application tier. With the elimination of the Web tier bottleneck, we experienced higher CPU utilization in the application tier and the database responsible for the shop transactions. This was to be expected since a larger number of requests usually translates into higher CPU utiliza- tion. However, when we accounted for the two times the amount of traffic on peak day of the 14th between 2006 and 2007, we concluded that we still gained significant scalability in the application tier. The two-percent increase of CPU utilization well offsets the amount of the additional traffic the tier handled successfully. We suspect that some of the efficiencies gained in the application tier were caused by the appli- cation tuning activities that were undertaken prior to the holiday, similar to the steps described in [58], but based on the results of the load tests prior and after the tuning activities, we concluded that the contribution was minimal and most, if not all, of the scalability gains could be attributed to the elimination of the Web tier bottleneck with the CDN. Perhaps the most unexpected result was uncovered in the database tier. One of the
- 54. CHAPTER 5. ANALYSIS46 databases experienced 80 percent average CPU utilization decrease. We decided to investigate it further and we discovered the main contributor to be the database tun- ing activities performed by the engineering team prior to the peak week. Therefore, we deemed the data invalid for analysis purposes and we concluded that additional efficiencies could not be attributed to the CDN implementation. Figure 5.1.1 further supports our conclusions for the Web and application tiers. Generally speaking, the fewer number of threads waiting for execution in the run queue results in lower CPU utilization. As in our CPU results, the most significant impact on the run queues was on the peak traffic day - 2/14. The slight increase in some cases between 2006 and 2007 is negligible from the CPU perspective as long as the number oscillates around one or two threads. According to [34] and [6], blocking I/O contributes to high run queue thread waits. The implications of the high cost of I/O-related context switching are also discussed in [59]. We conclude that the blocking I/O was the main culprit behind high CPU utilization prior to the CDN implemen- tation because requests made directly to the Web servers resulted in significant I/O waits for long-running large-object connections which, in turn, resulted in high I/O blocking in the application tier. The offloading of a large number of HTTP connections to the CDN has had a significant impact on the CPU utilization. 5.1.2 Memory utilization We used three different measures to gauge the impact of the CDN on memory utiliza- tion. The same level of memory resources was maintained throughout 2006 and 2007. We observed the largest differences in memory paging activity. Retrievals of memory pages from the paging space instead of RAM are considerably slower and constant paging activity can cause system performance degradation. The most pronounced impact on memory paging activity was observed on the peak day with average mem- ory page-ins per second decreasing from 25 to almost zero and page-outs decreasing from approximately 60 to one in the Web tier as shown in Figure 5.1.2 and Figure 5.1.2. Interestingly enough, the paging activity increased on all other days during the peak week, but the increase was not significant enough to cause performance problems or outages. Figure 5.1.2 represents the memory scanned-to-freed ratios. Overcommitting of memory results in high scanned-to-freed ratios. It’s difficult to establish the level of this ratio that will definitely result in memory being constrained to the point of having an impact on performance. Overall, the ratios were neutral across all tiers and did not seem to have a visible impact on scalability in either 2006 or 2007. All evidence is pointing to the excessive paging activity as the main culprit of memory and CPU utilization spikes in 2006. The impact of the CDN is probably indirect in the case of memory utilization. The most plausible conclusion is that the lower number of I/O waits due to offloading of HTTP connections results in a significant decrease of calls from the kernel scheduler to the paging system which, in turn, alleviates the memory
- 55. CHAPTER 5. ANALYSIS47 Figure 5.3: Memory page-ins, February of 2006 and 2007 constraints [54]. 5.1.3 Bandwidth Bandwidth costs and availability are key considerations for any highly transactional e-commerce environment. Hosting facilities usually impose limits on bandwidth us- age in order to allow many customers to share the connectivity to the Internet from the hosting facility. Increasing bandwidth availability by a significant amount entails significant costs during peaks due to market conditions. There are also significant costs associated with deployment of network infrastructure to support bandwidth increases. Therefore, bandwidth offloading represents a significant benefit for CDN implementations. The bandwidth impact in our environment between 2006 and 2007 peak weeks is depicted in Figure 5.1.3. We determined that low bandwidth utilization in 2006 was caused by the refusal of a large number of requests in the Web tier and the sub- sequent outages. However, even if we were to allow all of the requests we would
- 56. CHAPTER 5. ANALYSIS48 Figure 5.4: Memory page-outs, February of 2006 and 2007 still encounter the bandwidth limitation imposed on our environment by the hosting provider. The impact of the CDN is two-fold. First, it allows us to fully utilize all of our bandwidth capacity at the hosting facility. Secondly, it allows us offload over 1 Gbps of bandwidth utilization to the CDN network with thousands of Web servers that represent a much more resilient and cost effective infrastructure. We conclude that the scalability impact here is likely in the range of a factor of two to four. Figure 5.1.3 provides another view into the impact of the CDN and it supports our scalability assessment from the standpoint of bandwidth usage during the peak day. It shows that we were able to sustain a five-fold spike in bandwidth usage for several hours. Our analysis of the number hits further verifies our scalability claim. A hit is defined by our CDN as a retrieval of an object from an edge server versus an origin server. Our peak edge hit rate was 16,000 per second. This number is of interest to us because we can use it to determine the approximate Web tier scalability terms of TCP connections using the following formula:
- 57. CHAPTER 5. ANALYSIS49 Figure 5.5: Memory scan rate, February of 2006 and 2007 ScaleFactortcp = HitsCDN /AvailableThreads (5.1) where Hits represents the peak number hits per second offloaded to the CDN and AvailableThreads represents the maximum possible number of threads for our Web server cluster without impacting server availability. Our Web servers’ thread capacity varies depending on the type of hardware currently in use, but we have historically estimated it at approximately 3,600 for a typical day. Therefore, ScaleFactortcp = 3.9 and, during peak, the CDN allows us to quadruple our TCP connection capacity with- out additional hardware. This number represents the best case scenario as many con- nections will be kept alive by the browsers. Nonetheless, we expect that it will still lead to a significant load decrease on the Web servers, especially since many of the connections constitute long-running Flash video downloads.
- 58. CHAPTER 5. ANALYSIS50 Figure 5.6: Peak bandwidth comparison, February of 2006 and 2007 5.2 Performance We decided to divide performance observations into two perspectives: consumer and server-side. The consumer perspective corresponds to measurements obtained from the RTUM transactions executed under steady-state with and without the CDN. The RTUM allowed us to capture object-level download times in an environment that closely emulates consumers’ experience. The server-side perspective supplements our consumer perspective with server-side pre- and post-CDN measurements obtained from the performance measurement appliance deployed in our hosting environment’s network. 5.2.1 Consumer Perspective The RTUM experiments resulted in approximately 1,200 combined shop and browse transactions for the duration of 48 hours before and after implementing the CDN. The 1,200 transactions for each time period generated over 8,700 page views and downloads of over 250,000 objects to the RTUM clients. We eliminated transactions that were incomplete due to unavailability of the environment. Figure 5.2.1 and Fig- ure 5.2.1 summarize response time measurements for each transaction type. Shop transaction response times averaged 14 seconds with the CDN and 11 seconds without the CDN resulting in 21 percent performance improvement at the transaction level.
- 59. CHAPTER 5. ANALYSIS51 Figure 5.7: Akamai bandwidth offload, February of 2007 Browse transaction response times averaged 15 seconds with CDN versus 10 seconds without it for the overall improvement of around 30 percent. We expected the browse transaction to realize slightly better benefit from the CDN due to its higher number of content assets and the usage of larger content assets. Subsequently, we decided to analyze performance data at the object level of granu- larity in order to identify the source of performance improvements and to distinguish between characteristics of server-side components such as servlets and static content. Due to very different content characteristics we expected to see significant differences between the shop and browse transactions in terms of object response times as well. Table 5.2.1 and Table 5.2.1 illustrate end-to-end response time changes by content type. Positive percentages represent performance improvement while negative percentages represent performance degradations. Both transaction types show significant perfor- mance improvements at the object level. Although the transaction and object level performance realized significant benefits from the CDN, it was not without a trade-off in terms of servlet performance. Browse and shop servlets suffered performance degradation of 5 percent and 18 percent, re- spectively. We suspect that the primary reason for the degradation are the additional
- 60. CHAPTER 5. ANALYSIS52 Figure 5.8: RTUM shop transaction response times in seconds DNS redirections that take place during edge server resolution. Overall, however, the benefits at the object level far outweigh the degradation at the servlet level and the transaction response times improved. The 30 percent improvement in browse transac- tions is particularly important to our environment during the peak week in February when majority of our transactions are browse transactions. 5.2.2 Server-side Perspective The analysis of server-side performance centered around measuring the servlet re- sponse times. However, during our measurements we uncovered additional network- related characteristics that helped us better understand the impact of the CDN not only on the server performance of our hosting environment, but also on its network infrastructure. Table 5.2.2 illustrates the server-side performance changes for the various key servlets. It’s worth noting that the measurements are represented in milliseconds and while the improvement in terms of a percentage seems very impressive at around 20 percent av-
- 61. CHAPTER 5. ANALYSIS53 Table 5.1: RTUM browse transaction performance Content Type Performance Change JavaScript 69% CSS 68% GIF 68% JPG 65% PNG 77% Servlet -5% Table 5.2: RTUM shop transaction performance Content Type Performance Change JavaScript 68% CSS 68% GIF 69% JPG 61% PNG 72% Servlet -18% Table 5.3: Server-side performance in milliseconds Servlet Name Server time without CDN Server time with CDN Chg Category 1 64.05 28.06 56% Category 2 38.89 38.62 1% Category 3 45.48 39.81 12% Category 4 56.18 33.08 41% Category 5 42.78 39.67 7% Category 6 56.26 46.51 17% Checkout 236.18 193.54 18% Ecard Display 31.46 21.75 31% Ecard Personalize 81.54 69.38 15% Home Page 50.60 20.14 60% Login 97.25 58.68 40% Marketing Spot 13.74 13.48 2% Order Review 267.64 248.73 7% Product Detail 473.52 500.06 -6% Shopping Cart 4052.16 4069.80 0%
- 62. CHAPTER 5. ANALYSIS54 Figure 5.9: RTUM browse transaction response times in seconds erage, the majority of improvement can be attributed to servlets that already perform at 100 milliseconds or less. Therefore, the improvement has very little or no impact on the overall response time for the Web pages that correspond to the servlets. On the other hand, the servlets that performed at above 100 milliseconds suffered either a slight performance degradation or insignificant improvement. We anticipated the servlet performance to experience more of an improvement, but the only logical con- clusion based on the data is that the impact of the CDN on server side performance in the application and database tiers is neutral or slightly negative. We were able to obtain data regarding packet flow through our hosting environ- ment’s network before and after the CDN implementation. The summary of the re- sults is presented in Table 5.2.2. There is a strong correlation between the amount of content associated with a servlet and the decrease in the number of packets in our network. This supports the results from the Scalability section of this chapter where we discuss the bandwidth implications of the CDN. With much of the bandwidth of- floaded to the edge network we would expect a much smaller number of TCP packets in our network.
- 63. CHAPTER 5. ANALYSIS55 Table 5.4: TCP packet counts at hosting environment Servlet Name TCP Pkt Cnt without CDN TCP Pkt Cnt with CDN Chg Category 1 758.46 74.08 90% Category 2 281.20 84.65 70% Category 3 159.78 85.36 47% Category 4 419.39 87.01 79% Category 5 621.74 95.43 85% Category 6 482.11 95.96 80% Checkout 594.68 178.45 70% Ecard Display 124.92 51.21 59% Ecard Personalize 178.09 118.57 33% Home Page 795.08 76.92 90% Login 531.43 215.71 59% Marketing Spot 16.24 27.05 -66% Order Review 350.12 178.77 49% Product Detail 347.56 221.57 36% Shopping Cart 267.09 201.32 25%
- 64. Chapter 6 Conclusions andFurther Research In this thesis, we evaluated the merits of implementing a Content Delivery Network as a strategy to improve scalability and performance of a highly transactional e-commerce website. The key differentiating factor of our research is the evaluation of impact across all tiers and processing components of our environment. We were also inter- ested in potential tradeoffs from this architectural decision - especially with regards to configuration and content management across a geographically dispersed environ- ment. Research to-date has focused primarily on network efficiencies that result from the CDN implementations. By implementing a series of server and client-side monitoring components, we were able to obtain measurements under steady transactional load and also during significant transactional bursts. We captured results from the consumer as well as application and infrastructure perspectives. Our overall conclusion is that CDN of- floading has a significant impact on overall performance and scalability of n-tier e- commerce environments such as ours. The results of our experiments lead us to the following conclusions: • CDNs have a significant impact on page response times as observed by con- sumers. We achieved a 30 percent improvement in the most utilized transaction on our website - the browse transaction. • Implementing a CDN as a scalability tactic is a much more cost-efficient and flexible way to achieve the desired results in the Web tier as opposed to hori- zontal and vertical scalability tactics because it does not require an investment in additional server and network hardware. • We conclude that the CDN resolved our content delivery bottlenecks and al- lowed us to scale to meet the peak demand without sacrificing page response times. We also achieved a 100 percent availability during the peak day with much of the success attributable to the offloading solution. 56
- 65. CHAPTER 6. CONCLUSIONSAND FURTHER RESEARCH 57 • Server-side performance impact varies throughout the tiers with the Web tier benefiting from the CDN the most. We observed a slight performance degrada- tion in the application tier and the impact on the DB tier was neutral. Servlet per- formance degradation and additional DNS redirections had minimal impact on consumer experience. This leads us to conclude that CDNs are not the panacea for all performance-related problems including poor application and database design. We observed that CDN content delivery has little or no impact on server- side application-tier caching strategies because of its primary impact on the Web tier. • We observed a five-fold increase in the efficiency of bandwidth utilization with the CDN in place. This was partly due to the ability to fully utilize our hosting center’s bandwidth and the additional bandwidth provided by Akamai. In the past, we were unable to achieve full utilization at our hosting facility due to Web tier outages. With the CDN in place, we achieved bandwidth scalability equivalent to adding another hosting facility at a fraction of the cost. • The CDN allowed us to maintain half of our previous Web server footprint while increasing the number of page views almost two fold and decreasing CPU uti- lization by almost 90 percent in the Web tier. We also saw a positive impact on memory utilization with memory page-ins and page-outs decreasing by almost 95 percent. • Maintainability should be considered as a part of tradeoff analysis during CDN selection. We observed that configuration changes can take up to two hours and content changes can take an additional 7-10 minutes. Majority of our changes are content-related, so the several-minute delay may have an impact on our ability to respond to content changes during peaks. Configuration changes do not impact availability because they do not require system re-starts or a planned outage. 6.1 Summary of Contributions Our research is the most comprehensive and architecturally significant attempt we are aware of to correlate content delivery offloading with Quality of Service factors of an e-commerce environment. We assert that the CDN impact on the network behavior is just a part of the full range of performance and scalability benefits that an e-commerce website may realize from it. This work also attempted to quantify its impact on the computing infrastructure - especially the Web server footprint. We were able to quan- tify scalability gains in terms of bandwidth and Web tier CPU and memory utilization. Additionally, we proposed an approach to measuring performance of an e-commerce website with a combination of server-side and client-side monitors in order to obtain
- 66. CHAPTER 6. CONCLUSIONSAND FURTHER RESEARCH 58 full end-to-end picture of performance. This is important because much of the re- search to-date focuses on either client or server perspective without identifying cor- relations between the two. The full performance picture is a pre-requisite to under- standing the locations of the performance bottlenecks in an environment such as ours. Finally, were able to verify the claim that CDNs enhance not only performance but also scalability and availability of e-commerce websites. Much research to-date focuses on performance alone and treats scalability as an enabler or, at times, a by- product of performance. We were able to isolate measurements that pertain to scala- bility alone and it is our assertion that this quality of service deserves as much focus as performance. 6.2 Future Research Edge content delivery is already a well-established technology and quite a few solu- tions exist in the marketplace to enable it. Edge computing has been proposed as a way to extend this model to geographically dispersed application delivery and there have been several attempts in the research community to define application replication and deployment standards related to this concept. This area deserves further research as the complexity for multi-platform geographically dispersed computing environments will far outweigh the content delivery of a limited set of static asset types as is the case with the CDNs. N-tier system quality of service investigations of e-commerce environments have also been scarce in the research community. The transactional burstiness of these environments makes them particularly relevant to performance and scalability re- search. More research into availability, reliability and security of these environments is needed to establish a full architectural view of tradeoffs and dependencies of the qualities of service. We did not evaluate the tradeoffs between the various techniques that can be used to enable CDN re-routing. We also did not consider differences between HTTP down- loading and streaming of video content using protocols such as MMS or RTSP. Edge streaming has gained popularity as an alternative to hosting streaming servers in cen- tralized e-commerce environments. The above could be areas for further research.
- 67. Bibliography [1] A.Datta, K.Dutta, H. Thomas, D. Vandermeer, and A. Ramamritham. Proxy- based acceleration of dynamically generated content on the world wide web: An approach and implementation. ACM Transactions on Database Systems, 29(2):403– 443, 2004. [2] V. Aggarwal, A. Feldmann, and C. Scheideler. Can isps and p2p users cooper- ate for improved performance? SIGCOMM Computer Communications Review, 37(3):29–40, 2007. [3] Akamai. www.akamai.com, 2007. [4] Akamai. Akamai http content delivery configuration guide. https://dl. akamai.com/customers/HTTPCD/HTTPCD_Activation_Guide.pdf, 2007. [5] A. Akella, S. Seshan, and A. Shaikh. An empirical evaluation of wide-area inter- net bottlenecks. In IMC ’03: Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement, pages 101–114, New York, NY, USA, 2003. ACM. [6] J. Akella and D. Siewiorek. Modeling and measurement of the impact of in- put/output on system performance. In ISCA ’91: Proceedings of the 18th annual international symposium on Computer architecture, pages 390–399, New York, NY, USA, 1991. ACM. [7] P. Albitz and C. Liu. DNS and BIND. O’Reilly, Sebastopol, CA, 2001. [8] M. Andreolini, M. Colajanni, R. Lancellotti, and F. Mazzoni. Fine grain perfor- mance evaluation of e-commerce sites. SIGMETRICS Performance Evaluation Re- view, 32(3):14–23, 2004. [9] M. Arlitt, L. Cherkasova, J. Dilley, R. Friedrich, and T. Jin. Evaluating content management techniques for web proxy caches. SIGMETRICS Performance Evalu- ation Review, 27(4):3–11, 2000. [10] M. Arlitt, D. Krishnamurthy, and J. Rolia. Characterizing the scalability of a large web-based shopping system. ACM Transactions on Internet Technologies, 1(1):44– 69, 2001. 59
- 68. BIBLIOGRAPHY 60 [11] M.Arlitt and C. Williamson. Internet web servers: workload characterization and performance implications. IEEE/ACM Transactions on Networking, 5(5):631– 645, 1997. [12] H. Balakrishnan, V. Padmanabhan, S. Seshan, M. Stemm, and R. Katz. Tcp be- havior of a busy internet server: analysis and improvements. INFOCOM ’98. Seventeenth Annual Joint Conference of the IEEE Computer and Communications Soci- eties. Proceedings. IEEE, 1:252–262, 29 Mar-2 Apr 1998. [13] P. Barford and M. Crovella. A performance evaluation of hyper text transfer pro- tocols. In SIGMETRICS ’99: Proceedings of the 1999 ACM SIGMETRICS interna- tional conference on Measurement and modeling of computer systems, pages 188–197, New York, NY, USA, 1999. ACM. [14] Paul Barford and Mark Crovella. Measuring web performance in the wide area. SIGMETRICS Performance Evaluation Review, 27(2):37–48, 1999. [15] L. Bass, P. Clements, and R. Kazman. Software Architecture in Practice. Addison Wesley, 1998. [16] L. Bent, M. Rabinovich, G. M. Voelker, and Z. Xiao. Characterization of a large web site population with implications for content delivery. In WWW ’04: Pro- ceedings of the 13th international conference on World Wide Web, pages 522–533, New York, NY, USA, 2004. ACM. [17] B.Krishnamurthy, C. Wills, and Y. Zhang. On the use and performance of content distribution networks. In IMW ’01: Proceedings of the 1st ACM SIGCOMM Work- shop on Internet Measurement, pages 169–182, New York, NY, USA, 2001. ACM. [18] B. Boardman. Real time monitoring products. http://www. networkcomputing.com/showArticle.jhtml?queryText= &articleID=175803253&pgno=4, 2006. [19] B. Boiko. Content Management Bible. Wiley, 2nd edition, 2004. [20] G. Booch, J. Rumbaugh, and I. Jacobson. The Unified Modeling Language user guide. Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, USA, 1999. [21] C. Bornstein, T. Canfield, G. Miller, and S. Rao. Optimal route selection in a content delivery network. United States Patent 7,274,658, September 2007. [22] P. Brebner and J. Gosper. How scalable is j2ee technology? SIGSOFT Software Engineering Notes, 28(3):4–4, 2003.
- 69. BIBLIOGRAPHY 61 [23] L.Breslau, C. Pei, L. Fan; G. Phillips, and S. Shenker. Web caching and zipf-like distributions: evidence and implications. INFOCOM ’99. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings. IEEE, 1:126–134 vol.1, 21-25 Mar 1999. [24] J. Challenger, P. Dantzig, and A. Iyengar. A scalable and highly available system for serving dynamic data at frequently accessed web sites. In Supercomputing ’98: Proceedings of the 1998 ACM/IEEE conference on Supercomputing (CDROM), pages 1–30, Washington, DC, USA, 1998. IEEE Computer Society. [25] J. Challenger, P. Dantzig, A. Iyengar, M. Squillante, and L. Zhang. Efficiently serving dynamic data at highly accessed web sites. IEEE/ACM Transactions on Networking, 12(2):233–246, 2004. [26] Coradiant. www.coradiant.com, 2007. [27] M. Crovella and A. Bestavros. Self-similarity in world wide web traffic: evi- dence and possible causes. In SIGMETRICS ’96: Proceedings of the 1996 ACM SIG- METRICS international conference on Measurement and modeling of computer systems, pages 160–169, New York, NY, USA, 1996. ACM. [28] P. Danzig, R. Hall, and M. Schwartz. A case for caching file objects inside internet- works. In SIGCOMM ’93: Conference proceedings on Communications architectures, protocols and applications, pages 239–248, New York, NY, USA, 1993. ACM. [29] B. Davison. Content delivery and distribution services. http://www. web-caching.com/cdns.html, 2007. [30] L. Duboc, D. Rosenblum, and T. Wicks. A framework for modelling and analysis of software systems scalability. In ICSE ’06: Proceeding of the 28th international con- ference on Software engineering, pages 949–952, New York, NY, USA, 2006. ACM. [31] T. Farwell. It’s all relative: A guide to ibm’s rperf benchmark. http: //www.ibmsystemsmag.com/opensystems/junejuly05/features/ 6657p1.aspx, June 2005. [32] G. Ferrari, P. Ezhilchelvan, and I. Mitrani. Performance modeling and evaluation of e-business systems. In ANSS ’06: Proceedings of the 39th annual Symposium on Simulation, pages 135–142, Washington, DC, USA, 2006. IEEE Computer Society. [33] IBM. Vmstat Command Reference. IBM, 2006. [34] IBM. CPU monitoring and tuning. IBM, 2007. [35] IBM. Ibm aix 5.3 technical information. http://publib.boulder.ibm.com/ infocenter/systems/index.jsp?topic=/com.ibm.aix.howtos/doc/ howto/HT_commadmn_configure_dns.htm&tocNode=int_2049, 2007.
- 70. BIBLIOGRAPHY 62 [36] A.Iyengar, E. Nahum, A. Shaikh, and R. Tewari. Communication Systems: The State of the Art, chapter Enhancing Web Performance, pages 95–127. Kluwer Academic Publishers, 2002. [37] N. Jian, D. Tsang, I. Yeung, and H. Xiaojun. Hierarchical content routing in large- scale multimedia content delivery network. Communications, 2003. ICC ’03. IEEE International Conference on, 2:854–859, May 2003. [38] J. Jung, E. Sit, H. Balakrishnan, and R. Morris. Dns performance and the effec- tiveness of caching. In IMW ’01: Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement, pages 153–167, New York, NY, USA, 2001. ACM. [39] M. Koletsou and G. Voelker. The medusa proxy: A tool for explor- ing user-perceived web performance. http://citeseer.ist.psu.edu/ koletsou01medusa.html, 2001. [40] F. Leighton and D. Lewin. Html delivery from edge-of-network servers in a con- tent delivery network (cdn). United States Patent 7,293,093, November 2007. [41] B. Mah. An empirical model of http network traffic. INFOCOM ’97. Sixteenth Annual Joint Conference of the IEEE Computer and Communications Societies. Driving the Information Revolution, 00:592, 1997. [42] N. Medvidovic, D. Rosenblum, D. Redmiles, and J. Robbins. Modeling software architectures in the unified modeling language. ACM Transactions on Software Engineering Methodology, 11(1):2–57, 2002. [43] D. Menasce. Tpc-w: a benchmark for e-commerce. Internet Computing, IEEE, 6(3):83–87, May/Jun 2002. [44] D. Menasce, V. Almeida, and L. Dowdy. Performance by Design. Prentice Hall, 2004. [45] D. Menasc´e, D. Barbar´a, and R. Dodge. Preserving qos of e-commerce sites through self-tuning: a performance model approach. In EC ’01: Proceedings of the 3rd ACM conference on Electronic Commerce, pages 224–234, New York, NY, USA, 2001. ACM. [46] P. Mockapetris. Ietf rfc 1034 - concepts and facilities of dns. http://www.ietf. org/rfc/rfc1034.txt, 1987. [47] P. Mockapetris. Ietf rfc 1035 - implementation and specification of dns. http: //www.ietf.org/rfc/rfc1035.txt, 1987. [48] J. Mogul. Network behavior of busy web server and its clients. DEC Western Research Laboratory, Palo Alto, CA, 1995.
- 71. BIBLIOGRAPHY 63 [49] E.Nahum, M. Rosu, S. Seshan, and J. Almeida. The effects of wide-area condi- tions on www server performance. In SIGMETRICS ’01: Proceedings of the 2001 ACM SIGMETRICS international conference on Measurement and modeling of com- puter systems, pages 257–267, New York, NY, USA, 2001. ACM. [50] Nielsen. Sixth annual holiday e-spending report. http://www. symboldomains.com/nn_ratingreport.pdf, 2006. [51] H. Frystyk Nielsen, J. Gettys, A. Baird-Smith, E. Prud’hommeaux, H. Lie, and C. Lilley. Network performance effects of http/1.1, css1, and png. In SIGCOMM ’97: Proceedings of the ACM SIGCOMM ’97 conference on Applications, technologies, architectures, and protocols for computer communication, pages 155–166, New York, NY, USA, 1997. ACM. [52] V. Pai, M. Aron, G. Banga, M. Svendsen, P. Druschel, W. Zwaenepoel, and E. Nahum. Locality-aware request distribution in cluster-based network servers. In ASPLOS-VIII: Proceedings of the eighth international conference on Architectural support for programming languages and operating systems, pages 205–216, New York, NY, USA, 1998. ACM. [53] J. Pitkow. World Wide Web, volume Volume 2, Numbers 1-2 of Computer Science, chapter Summary of WWW characterizations, pages 3–13. Springer Netherlands, June, 1999. [54] J. Quarterman, A. Silberschatz, and J. Peterson. 4.2bsd and 4.3bsd as examples of the unix system. ACM Computing Surveys, 17(4):379–418, 1985. [55] P. Rodriguez, C. Spanner, and E. Biersack. Analysis of web caching architec- tures: hierarchical and distributed caching. Networking, IEEE/ACM Transactions on, 9(4):404–418, Aug 2001. [56] A. Shaikh, R. Tewari, and M. Agrawal. On the effectiveness of dns-based server selection. INFOCOM 2001. Twentieth Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings. IEEE, 3:1801–1810 vol.3, 2001. [57] W. Smith. Tpc-w: Benchmarking an ecommerce solution. http://tpc.org/ tpcw/TPC-W_Wh.pdf, 2000. [58] M. Sopitkamol and D. Menasc´e. A method for evaluating the impact of software configuration parameters on e-commerce sites. In WOSP ’05: Proceedings of the 5th international workshop on Software and performance, pages 53–64, New York, NY, USA, 2005. ACM. [59] J. Sterbenz and J. Touch. High-Speed Networking. John Wiley and Sons, 2001.
- 72. BIBLIOGRAPHY 64 [60] A.Su, D. Choffnes, A. Kuzmanovic, and F. Bustamante. Drafting behind akamai (travelocity-based detouring). In SIGCOMM ’06: Proceedings of the 2006 conference on Applications, technologies, architectures, and protocols for computer communications, pages 435–446, New York, NY, USA, 2006. ACM. [61] B. Urgaonkar, G. Pacifici, P. Shenoy, M. Spreitzer, and A. Tantawi. Analytic mod- eling of multitier internet applications. ACM Transactions on the Web, 1(1):2, 2007. [62] USCB. E-stats - measuring the state of electronic economy. www.census.gov/ estats, 2007. [63] D. Verma. Content Distribution Networks. Willey, 2002.