Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Preparing Website for the Holiday Rush


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Preparing Website for the Holiday Rush

  1. 1. Making a Site, and Checking it Twice A Guide to Preparing a Web site for the Holiday RushI f any retailer needs proof of the importance of load testing, they need only look to the headlines around the launch of the new Apple iPhone 4. All day on June 15, Apple and AT&T’s online stores went down and out. Tech bloggers spent hours trying to place online orders, and hours more on hold with customer service. An official announcement from AT&T called it “the busiest online sales day in AT&T history.” Yet despite the problems, they managed to sell out their first day’s allotment before the day was out. Keynote Benchmark
  2. 2. 2Retailers dream of having such problems as cash lost when visitors abandon the site THE FIRST BILLION-DOLLAR ONLINE DAY?rolling out a new blockbuster and taking because of poor performance. As the One likely milestone in the upcoming 2010orders for 600,000 units in one day. But you accompanying chart shows, using actual holiday season will be the first billion-dollarcan count on one hand the number of sellers results, a traffic spike 25 percent beyond the e-commerce day. Sales broke the $900who command such fanatic loyalty and site’s optimum capacity can cost $100,000 in million mark on Green Tuesday (December 15patience. For the rest, site outages, freezes and lost revenue every hour. last year), and at least nine shopping dayscheckout failure mean lost revenue that will Further complicating matters is the exceeded $800 million.iv A major east coastlikely never be made up — and those losses huge unknown factor of mobile for 2010. snow storm kept shoppers home-bound andmount minute by minute, hour by hour, With the tremendous and ever-accelerating contributed to brisk online sales. This year,especially during the peak holiday season, proliferation of browser-equipped, app-loaded mobile, with shoppers logging on even as theywhen most retailers rack up most of their smartphones, mobile is sure to play a bigger are out in the stores, is likely to have an evenannual business. During those critical four to role than ever, and likely to exceed even the greater effect.six weeks at the end of the year, when the site most optimistic retailer’s projections. All thosebreaks, the bottom line breaks, too. smartphones searching for products, FIRST THINGS FIRST: UPDATES So: it’s July or later as you’re reading comparing prices, and making purchases Getting a site into the near-final form it willthis. The biggest, most successful online will be accessing the same back-end have for the holiday season is the first step toretailers have been prepping their sites for the databases and applications as all the PC-based making sure it will handle the demand. Newholidays for six or seven months now. The browsers, putting even more stress on features and functionality — like enhancednext tier started in June. Retailers who have e-commerce systems. product presentation, reviews, andnot started to stress test their sites could belooking at many sleepless nights come the lastmonths of the year, and potentially serioushits to their 2010 revenues.ONLINE MAKES THE BOTTOM LINEThere’s no underestimating the importance ofe-commerce to overall retail success,particularly during the crucial holiday season.Every year from 2000 through 2007, onlinesales posted double-digit growth, handilyoutpacing in-store sales.i And last year, asretail struggled to recover from its worstholiday ever (in 2008), online holiday salesgrew at four times the rate of total retailgrowth — though admittedly modest growth,at 4 percent for online sales,ii and just 1percent for total retail.iii (For the year,e-commerce sales grew 2 percent, while overallretail sales fell by nearly 3 percent, accordingto the U.S. Department of Commerce.) With online playing such a criticalrole in holiday sales, it is impossible to ignorethe real bottom-line costs in lost revenue ofsite slowdowns and outages. Site metrics and This diagram illustrates actual results of a test of a major retailer’s Web site. The golden area illustrates that the more users who log on beyond the site’s optimum capacity,sales data show the revenue value of every site the more revenue is lost. A spike of 25 percent results in losses in the neighborhood ofvisitor, and site testing can show the actual $100,000 every hour. Keynote Benchmark
  3. 3. 3personalization options — should be well make sure that any kinks are worked out prior dilemma for retailers. With more and moreunder way if not complete by now. Sports to the busy holiday season, obviously.” products approaching commodity status andfashion leader Lacoste is one retail marketer “We’re adding larger images with available at multiple online outlets, sitethat understands the importance of readying greater resolution,” Miller continues. “We’re experience becomes a key differentiator.major site revisions early. adding new features to be able to check your Retailers want to create a rich experience for “We’ve been working on a huge tax and shipping or your total costs much visitors with interactivity, dramatic productupgrade of our site and have been doing earlier in the process. We’re also adding social presentation, perhaps Flash, personalizationside-by-side testing with Keynote in order to networking tools — the ability for people to or other features to set themselves off from themeasure the current experience against the more easily share products through any of the competition. But a heavy load of features andnew experience,” Lacoste E-Commerce major social networks.” functionality can drag site performance down,Director Maryssa Miller told Benchmark. often because of third-party content, and,“Our criteria is to be better in performance BEST PRACTICE: BUILD PAGES LEAN instead of making visitors sticky, can drivethan the current existing experience, even Performance management starts with how the them to leaner, faster competitive sites.with all the new functionality. And we want to pages are built, which often presents a “Ideally, it’s best to focus on limiting your number of third-party content and domains on each page to roughly six domains at most,” says Keynote Consulting Manager Cliff Crocker. “As opposed to what we see for a lot of retailers, where there’s 20 or 30 different domains that are killing performance on their page.” Crocker also recommends keeping the number of objects on a page to 40 or 50; many retailers are packing their pages with 150 or more elements. “You have to try to limit the time that’s being spent in the browser,” Crocker says. “The more JavaScript and the more functionality a retailer adds to their page, oftentimes can create very big delays within the user’s browser.” FACTOR MOBILE IN FROM THE START The successful retailers this year will have built mobile into their strategy right from the start — not just as an afterthought to the “main” site, but side-by-side with it. Shoppers carrying smartphones are using them to check prices, locate products, find deals, look at reviews and, more and more, to make purchases. (See “Shoppers, Start YourLacoste is updating its site half a year ahead of the holiday season, and setting a performance Smartphones!” in Benchmark.) Many retailersbar even higher than last year. Bigger images and social media functionality are among the were surprised at the amount of mobile trafficenhancements being implemented. Keynote Benchmark
  4. 4. 4they got during the 2009 holiday season. And marketing team, this is where it gets a little connections, and make mobile scripts withthere will be millions more smartphones in tricky; consumer behavior is difficult to mobile line speed coming into the site.”the hands of shoppers this year. project, especially in these times of continued For some retail marketers such as “We were very surprised,” says economic turbulence. Whatever the traffic and Lacoste, the mobile and wired WebLacoste’s Miller. “And I think that’s where we sales projections, it’s up to IT and the Web experiences are tied tightly together, so thereally realized that we need to focus on team to make sure the site can handle any necessity for holistic testing is apparent.[mobile] for 2010. We already knew we unexpected surge in site visitors. Speaking about their iPhone app,needed to focus on it, but I think [the 2009 “From the technology side, you Lacoste’s Maryssa Miller says, “My favoriteseason] was just really additional proof and should always treat whatever the business tells part is that it’s actually connected to thewe knew that it needed to be a huge part of you with a grain of salt,” says Keynote Director e-commerce site. I’ve seen other apps where itour strategy for this year, and then continue to of Global Testing Services Donald Foss. “You is a very separate experience, but ours is verybe part of our strategy going forward.” may say, ‘I’m going to add 50 percent as much integrated, which I think is great Because of the inherent slowness of planning-appropriate, and then I’m going to because we’ve found that a lot of people willcellular networks and devices, mobile sites add another 30 or 50 percent as a technology save items in their shopping cart and, forneed to be even leaner and meaner than wired buffer — an insurance policy,’ and so it ends whatever reason, they don’t purchase them.Web sites. It takes some hard decision-making up roughly double, or maybe 250 percent, of This way, if they decide they want it, they canand analysis of what is essential for users what the marketing department projects.” access their shopping cart on the e-commercewhen they are browsing on the go and what it Many, if not most, retailers still fail site or directly from the app. That’s a reallytakes to satisfy them, including their need for to account for mobile in their traffic usable feature and one of the best thingsspeed. Search results can be confined to return projections, and that could be a bigger mistake about it.”four or five results, for example, instead of the than ever this year. Visitors logging on from40 or 50 that might be delivered on the wired mobile devices are calling on the same MAKE TESTING A REAL REALITY CHECKWeb. And perhaps tracking pixels are needed resources and databases as visitors coming In a very oversimplified nutshell, with loadonly on the landing page and cart page, through PC browsers, and so are adding to testing you are throwing users at the site untilinstead of every page on the site. overall site load. Making matters worse is that it breaks (which is why live-site load testing is “There are some pretty strict rules those mobile connections are extremely slow; typically done in off-hours). But to be realistic,around how many elements you want to have traffic coming in through mobile actually holds accurate, and reliable, a load test has to be faron a mobile page,” Crocker says. “Where we say the connections longer, and consumes more more sophisticated than simply simulating a40 to 50 over the wired Web, we’re looking at 8 than its share of resources. gross volume of visitors showing up, which isto 10 elements on a mobile page, just because “In planning for traffic, it’s important the essence of the often-used but imperfectof all the latency that’s encountered over the to account for the total sum of the traffic from “concurrent user” methodology.carrier networks. So step one is, you’ve got to a holistic standpoint,” Foss says. “It’s A concurrent user approach deliversbuild a dedicated mobile site. There’s no way important to make sure you’re actually testing a picture of performance that is far fromyou’re going to port what you’re putting up on the mobile site and the regular Web traffic all accurate. Basically, it measures a staticthe wired Web or pare that down. You’ve really at the same time, in the correct proportions threshold. Here’s 100 thousand users. Does itgot to build it from scratch.” and in the right demographics, to see how break? OK, add another 10 thousand. And it works. another. The methodology assumes incomingSTEP 1: ESTABLISH YOUR BASELINE AND “Early on, you take a multi-pronged users will patiently wait their turn to enter,PROJECT TRAFFIC INCLUDING MOBILE approach. You test the core Web site by itself. and that everything will be fine until they do.Knowing your site’s baseline — how much You test the mobile site by itself, and make sure But in reality, those additional users are tryingtraffic it can handle right now — is the logical they’re both tuned and working correctly. Then to log on and being turned away, and thestarting point to begin understanding how you start the holistic testing, with the full sum of experience for the users already on the sitemuch work needs to be done. Then, traffic from the correct sources that you predict. degrades, often quickly. Testing only forprojections need to be made for how much If 15 to 20 percent of user traffic is coming over concurrent users does not accurately mirrortraffic and sales are expected for the holiday mobile, then we will actually have 15 to 20 user behaviors and actual site traffic dynamics.season. Typically the purview of the sales and percent of the test traffic running mobile-type Keynote Benchmark
  5. 5. 5 methodology, which factors the peaks and BEST PRACTICES: METHODOLOGY valleys of traffic and availability based on & MODELING likely user behaviors and site slowdowns. To really understand how your site Hand-in-hand with this methodology is performance will hold up — or not — under behavior modeling, which factors, among holiday stress, and to understand what the other things, users who are familiar/ experience will be like for users, use an arrival unfamiliar with the site, their tolerance for rate methodology and factor in behavior delays, and their tenacity in sticking with the models for the many, many types of users and site until they accomplish their tasks. tasks your site will serve. “There’s a certain amount of Behavior modeling results in slowness that can be tolerated, and a certain numerous permutations (often thousands) amount that won’t be,” says Foss. “That’s combining these variables: latency tolerance. There’s also tenacity, which • Familiarity: experienced users vs. means essentially, how important is it for a newcomers person to finish the transaction they’re on? • Connection speed: super-fast FIOS vs. How likely are they to continue with it or super-slow mobile device, and everything in not?” If they’re checking out, they’re more between likely to be patient with order and credit card • Latency tolerance: patience of users with processing. But if they’re elsewhere on the slow site response site, tenacity may be significantly lower.Lacoste’s iPhone app is tied directly to theire-commerce site, so registered users canaccess their cart and make purchases fromeither place. With mobile sites typicallycalling on the same backend resources asthe wired site, it is important to test the twosimultaneously to evaluate overall load-handling capability.In reality, users are continuously coming andgoing. Those with slow connections or mobiledevices are taking longer. Some who arealready familiar with the site are in and outquickly. When the site slows down under theload, more and more users pile on, because it’staking longer for the previous users tocomplete their tasks and leave. The load isnever static. And users are not merely numbersgoing in and out. They are individuals withindividual thresholds of patience — and thatis the critical factor to know when it comes toconverting visitors to sales, and minimizing The “concurrent user” model assumes a constant volume of site traffic, which is a poorrevenue lost due to poor performance. representation of the ups and downs of actual traffic. Arrival rate methodology, on the other A more accurate representation of hand, combined with user behavior modeling, mirrors much more closely the patterns andsite load is created using an “arrival rate” volumes of peak site traffic. Keynote Benchmark
  6. 6. 6• Interaction speed: complexity of the page to variations in performance, and overcome the The holiday shopping season is the navigate, and attention level of the user danger of looking at averages. An average culmination of many hard hours of work for• Tenacity: willingness of users to stick with a page-load time of three or four seconds may the IT/Web department. And no matter how task through completion seem OK, but that kind of average could mean well things are planned, no matter how your page is loading in one second for rigorously everything is tested, there’s always “We’re not just replaying a someone in New York, but taking six or more the chance that the unexpected will happentransaction multiple times ad nauseam to try seconds for someone in Chicago. And that is and something will go wrong. So it makesto put load on the site,” Foss says of Keynote’s not likely to be acceptable. The solution is to good sense to have technical personnel onLoadPro testing. “There’s a whole lot more test from multiple, geographically dispersed hand and on call during all the criticalgoing on behind the scenes. There’s literally locations, look at the data, and address any shopping periods to handle any emergencies,hundreds or thousands of different types of local or regional bottlenecks. and to have extra computing capacity standingusers that we emulate during a single test to by just in case it’s needed.make sure we’re emulating what a real user STAFF UP AND STOCK UP — ON CAFFEINEpopulation would be.” AND CAPACITY DON’T WAIT ANOTHER MINUTE Retailers who put off load testing can look If you haven’t started load testing your siteTEST IN THE REAL WORLD — ALL OF IT forward to many sleepless nights in October yet, there’s no time to lose. “The mostTo know how your site will perform for users and early November as they try to get their successful retailers look at holiday readinessdispersed across the country or the world, sites up to speed. But even the best-in-class as an ongoing process,” says Crocker. “Theytesting must be done over the Internet, from retailers can count on a sleepless night or start in January and they test and theythe same geographic locations as your users, two — particularly Thanksgiving night — as monitor performance and they benchmarknot from behind the firewall. There’s simply they stay up to make sure all their hard work and they trend all the way through until theno way to simulate the vagaries of Internet has paid off. end of December.”backbones, third-party content feeds, CDN “One of my biggest clients spent the And December is just a few shortperformance, and signal transmission through night at headquarters on Thanksgiving night months away.the critical last mile — unless you are at the last year,” Foss says. “He’s up all night makingend of that mile, with a browser. sure everything is running perfectly, and goes With testing agents dispersed where home Black Friday morning at 9:00 a.m.”your users are, you get an accurate picture ofENDNOTESi. U.S. Department of Commerceii. comScore press release, “comScore Reports $29.1 Billion in U.S. Retail E-Commerce Spending for Full November-December Holiday Season, UP 4 Percent vs. Year Ago, 1/6/2010iii. Internet Retailer, “Total holiday retail sales rise 1%, trailing e-retail’s 4% increaseiv. op cit ............................................................................................................................................................................. © 2010 Keynote Systems, Inc. All rights reserved. The trademarks of Keynote Systems, Inc. include Keynote®, DataPulse®, CustomerScope®, Keynote CE Rankings®, Keynote Customer Experience Rankings®, Perspective®, Keynote Red Alert®, Keynote Traffic Perspective®, Keynote WebEffective®, The Internet Performance Authority®, MyKeynote®, SIGOS®, SITE®, keynote™, The Mobile & Internet Performance Authority™ and all related trademarks, trade names, logos, characters, design and trade dress are trademarks or registered trademarks of Keynote Systems, Inc. in the United States and other countries and may not be used without written permission. Keynote Benchmark