This document summarizes a project to iteratively improve the United States Mint's website over more than a year starting in 2012. The goals were to improve the customer experience and satisfaction without negatively impacting outdated core technology, prepare customers and media for larger upcoming changes, and buy time until a new e-commerce solution launched in 2014. Major pain points identified through customer research included inconsistent navigation, an outdated shopping cart, and confusing product pages. Improvements were made within constraints of separate contractors managing different sites and the fragile underlying e-commerce technology. Stakeholders were regularly involved through bi-weekly meetings to discuss design mockups and provide feedback.

1. 1
Keeping Customers Happy While Making Major Website Improvements
Chris Noonan Sturm, PMP1
1United States Mint, 801 9th St. NW, Washington D.C. 20220;
chris.noonansturm@usmint.treas.gov
ABSTRACT
With more than $300 million in numismatic Web sales annually, the United States
Mint is one of the top 100 retailers by volume in the nation. This case study describes
how Agile, iterative improvements to a federal bureau’s website with an e-commerce
component -- usmint.gov -- were made over a period of more than one year starting in
Fall 2012 to improve the customer experience. Project management challenges in-
cluded gaining buy-in from stakeholders, coordinating contractors, mitigating risks to
outdated core technology, communicating changes to the media without negative re-
action and to customers without a reduction in sales or satisfaction. Results included
positive media reaction, a dramatic reduction in customer complaints, and, most im-
portant, a 94 customer satisfaction rating from ACSI, one of the highest ratings in the
federal government and higher than many private industries.
Problem and Opportunity
The United States Mint’s website – usmint.gov – has two components that
serve two major functions: inform the public about the bureau’s mission and opera-
tions (information website), and sell numismatic products to the public (e-commerce
website). These two functions are supported by two separate website infrastructures
and contractors. The e-commerce site, which provides 78% of total numismatic sales,
suffers from an aging infrastructure and outdated customer experience, putting the
business at risk and causing many customer complaints. To resolve this, the Mint em-
barked on a long-range plan to modernize the entire e-commerce business solution,
including non-web systems, which will take at least two years.
As a short term solution, the Mint Web Team launched its Web Improvement
and Integration Project in Fall 2012 to make Agile, iterative improvements to both the
e-commerce and information websites. The goals of the project: 1) improve the front-
end customer experience, or user experience (UX) as it’s known in the field, and cus-
tomer satisfaction with the website without negatively impacting the antiquated core
technology of the e-commerce website; 2) prepare the media and customer base for
the larger website change coming in the future by introducing them to smaller, itera-
tive website changes; 3) buy time and goodwill of customers by improving the web-
site as much as possible in the short term until the new e-commerce solution launches
in late 2014.
A few words about the word “iterative” in the context of this project are nec-
essary. Website refresh projects, especially those related to design and functionality

2. 2
improvements, are often executed on a grand scale as web redesign projects with con-
crete lifespans, funding, etc. This is often termed the “once and done” or the “launch
and leave” approach. (McGovern, 2013) However, as any savvy website manager
knows, a website is never done and can never be left – it’s ever an exercise in contin-
uous quality improvement. (McGovern, 2007) The best, most effective approach to
managing and improving a website is to make changes iteratively, implementing
small improvements over time that as a whole make a substantial difference for the
customer experience. In this sense, the Web Improvement and Integration Project is
iterative in that there is no great reveal, but a series of small reveals, all of which add
up to a much better whole.
The project is also iterative, in the Agile sense, in that it regularly involves
customers in usability testing of concepts for new interface designs such as a new
online shopping cart and new product description pages. The customer story is critical
to the clarifying the goals of each improvement, to collaborating with internal cus-
tomers, to conveying development and design tasks to the contractors, and to com-
municating with media and customers. Throughout, the team is open to any modifica-
tions to requirements at any time within the product schedule. Web work, due to its
never-done nature, lends itself to an Agile approach, even if it’s not executed perfect-
ly. (McLachlan, 2013) As General Dwight D. Eisenhower, the architect of the D-Day
invasion of Europe during World War II, is often quoted to have said, “Plans are use-
less, but planning is essential.”
Analysis, Pain Points, and Constraints
The United States Mint is very robust about constantly tracking customer sat-
isfaction and integrates the customer into many facets of its numismatic business. The
Mint conducts in-person focus groups around the country, does phone and online sur-
veys, and performs usability studies – all on an annual basis. These practices provide
the Web Team with ample data to review and analyze to identify customer pain points
with the e-commerce and information websites. The team reviewed past usability
studies on previous designs, analyzed metrics and customer emails, gathered input
from the Mint’s Call Center representatives, and consulted internal stakeholders. Res-
olutions to the pain points were identified, prioritized, and a tentative schedule devel-
oped that was open to change. The sprint for each improvement was usually a month
and was not necessarily related to the previous one or the next one. The pain points
and remediation improvements covered the gamut and included issues with infor-
mation architecture, site navigation, editorial content, page design, functionality de-
sign (buttons and forms), images, search, and more.(Nielsen, 2007). The major pain
points included:
 Inconsistent and confusing site navigation: usmint.gov had three different site-
wide navigation schemes, which changed depending on whether the customer

3. 3
was on the Information site, the E-commerce site, or the Education site (for
teachers)
 Home page carousel with no user controls and deployed with Flash (not mo-
bile friendly)
 Separate site searches for information and e-commerce sites and no best bets
 Confusing and redundant product release schedule pages
 Product description pages with few images, small images, no zoom, and too
much text, and no ability to cross-sell / upsell products (you might also like)
 Confusing shopping cart pages that did not conform to industry standards
 Poor, confusing error messaging
The team faced several constraints during the project. First, usmint.gov is essen-
tially two websites managed by two contractors – one managing the e-commerce site
and one managing the information site. Implementing changes that cross both sites
required close coordination and communication across the two contractors, which
was not current practice.
Second, the existing, antiquated technology of the e-commerce website imposed
constraints on the scope of the changes. Due to the fragile nature of the underlying
technology, the Web Team had to be careful to limit improvements to the frontend of
the website -- interface, design, and editorial content, the presentation and content
layers. The improvements could not touch business rules, logic and related code. In
addition, the contractor managing the e-commerce site could not make changes “on
the fly.” While the contractor managing the information website could make changes
to that site at any time, the e-commerce contractor could only make changes once a
month during a technical build. Due to the high risk of changes impacting old code,
the contractor must conduct extensive testing before putting even the smallest chang-
es into production. The monthly build constraint, which is not typical in website man-
agement, required much advance planning.
An interesting point can be made here: It’s important for website managers to
understand the layers that comprise a website. Web managers that don’t have a good
understanding of the layers (content, presentation, code, database, infrastructure, net-
work, etc.) may believe that there are limits to what they can do to improve the cus-
tomer experience. In actuality, improvements can usually be made constantly to the
aspects that have the most impact on the customer – the front end content and design
that the customer interacts with and reacts to. The layers of a website can be defined
differently, depending on the point of view, but the point is that knowing they exist is
important. (Heilmann, 2005). Understanding the layers enables the manager to be
more proactive in working with contractors. Bring proposed improvements to them –
don’t wait for them to take the initiative – and reassure them that what you propose
won’t hurt their infrastructure.
Stakeholder Involvement and Communications

4. 4
The e-commerce website is critical to the Mint’s numismatic business, which
involves the sale of collectible coins, medals, and bullion. As such, many internal
customers have an extremely critical stake in any changes to the website’s content
and design. And any change, even for the good, brings risk. Without the internal cus-
tomers’ input, feedback, and support, the Web Improvement and Integration Project
would fail. The most critical internal customers are the Product Managers, who over-
see all aspects of a numismatic product’s lifecycle, and Customer Operations, which
is responsible for customer interactions involving commerce transactions, including
the Call Center.
To regularly involve these stakeholders, all proposed web improvements were
presented and discussed at bi-weekly web communications meetings. Design mock-
ups (or comps) of new interfaces and pages were presented for comment and feed-
back. In many cases the feedback was actionable and fed refinements to the new
screens. The process also helped internal customers become comfortable with the
concept of constant and continuous improvements to the website, a new approach in
their experience.
During these internal collaborations, it was more effective to bring a proposal
to the table to get discussion flowing rather than a blank sheet of paper asking for re-
quirements and ideas. It’s difficult to overstate how important design mock-ups of
new screens were for communicating with all involved. The mock-ups, of very high
quality and created by a web designer, enabled the team to tell the customer experi-
ence story in a powerful, easy-to-understand fashion and obtain buy-in. More im-
portant, they were critical to showing the contractors exactly what the requirements
were for the customer experience. The moral: Use images to communicate about
change whenever possible.
One of the most important sections of any e-commerce website is the shop-
ping cart. If the cart is confusing, unclear, and hard to understand when filling out
forms, a sale can be lost. Industry standards and conventions for how the shopping
cart customer experience works have evolved, and customers expect online stores to
conform to those standards. Unfortunately, the Mint’s e-commerce site’s shopping
cart did not continue evolve, so its screens were trapped in time. Because these
screens are so important, usability tests were conducted with customers to gather
feedback and input on the new shopping cart before it was launched. (Biddle, 2013)
They provided excellent feedback on problems with screen progression and editorial
content. Customers welcomed the new shopping cart design with more white space,
less clutter, more efficient use of text, product images, and industry standard ap-
proaches. (See Figures 1 and 2.)

5. 5
Figure 1: Old shopping cart screen
Figure 2: New shopping cart screen

6. 6
Public Outreach and Messaging
Changing the design and functionality of a public website is challenging.
Website managers don’t have the luxury of taking the website down for a while to
make changes. Like electricity, the site has to be available 24/7/365. So changing a
public website is like putting new tracks down in front of the train while it’s roaring
down the track with passengers aboard.
And visitors accustomed to using a site a certain way can be confused by
changes to the familiar, which can have a negative impact on customer satisfaction
and on business. You are moving the customer’s cheese. Even if it’s better cheese,
it’s different cheese. They may not like it.
To complicate matters further, the Mint’s customer base skews older – 61 per-
cent of Mint customers are 55 or older and 30 percent are 65 or older. Senior web us-
ers have their own special requirements when it comes to meeting their customer ex-
perience needs. According to Nielsen, half of seniors in a study keep a cheat sheet of
instructions for how to use the websites they visit often. (Nielsen, 2013)
Lastly, the Mint’s every activity is very closely watched and reported upon
daily by the numismatic trade press and bloggers such as Coin World, Numismatic
News, Mint News Blog, CoinNews.net, Coin Update, and more. Any change in the
Mint’s website is quickly reported and commented upon.
To mitigate these issues, the centerpiece of the communications plan was the
development of a “visual tour” for each major improvement rollout. A visual tour is
several screenshots of the new web pages with annotations calling out what is new
and explaining its value to customers. The tours educate internal stakeholders and the
Call Center, who rely on the tours explain web changes to customers. The visual tour
for each rollout is shared with Public Affairs staff in advance of the deployment so
they can send it to the trade press in advance of their print deadlines. The press then
publish the screenshots in their print publication and online. This enlists the numis-
matic media as a partner in educating customers about upcoming changes. All visual
tours are also published on the Mint’s website. (See Figure 3.)

7. 7
Figure 3: Visual tour with annotations explaining new Product Description web
page.
Results
The first substantial website improvements rolled out in September 2012 and
continued almost monthly until February 2013, when the pace slowed to every other
month or so. Customer compliments began coming in, the flow of customer com-
plaints dropped, and positive media coverage began. Coin World lauded the im-
provements in an editorial titled “Mint makes improvements that show revived cus-
tomer focus” and About.com’s numismatic guide wrote “I like these new website
changes and I hope they continue over the coming months and years. Finally, the
Mint is starting to show that it considers the customer to be ‘number one.’ ”
But the most impactful result is a sustained improvement in website customer
satisfaction scores based on the Foresee customer survey taken by visitors to us-
mint.gov over the past two years. In February 2012 the website’s ACSI score was 76.
But by the end of 2012 it was up to 80 or higher, and stayed at that level for almost
two years. Analysts say that an increase of five points, and then sustaining the im-
provement, is a significant achievement.
In February 2014, the American Customer Satisfaction Index (ACSI) an-
nounced that the Mint was among the top federal websites for customer satisfaction in
2013 with a 94 rating, saying that “the very best federal agency services continue to
rival the private sector.” This finding was based on interviews with 1,448 users cho-
sen at random and contacted by phone and email. (American Customer Satisfaction
Index, 2014).

8. 8
High customer satisfaction with the website is critical because it is deeply
connected to the Mint’s strategic goal to responsibly expand the numismatic program
and to federal requirements to use the web to communicate to the public about mis-
sion and operations. Some 78% of numismatic sales come from the catalog website,
so keeping customers satisfied and positively engaged in using the website is critical
to sustaining online sales and revenue.
The other silver lining to the project? It cost the Mint no additional outlay in
funds or task orders. All work was conducted by federal staff and contractors within
the terms of their existing contracts.
Conclusion
Websites are organic, imperfect, and absolutely essential to almost every or-
ganizations’ mission. While large web projects will always be with us, it’s a nimble
web manager who is able to make lemonade on a regular basis from lemons he or she
may have inherited, using analysis and iterative approaches to change the customer
experience for the better. She/he should not be shy about shaking up the status quo,
because a static website is often an ineffective website. With strong outreach and vis-
ual communication to internal and external customers, the website manager may very
well find audiences open to, even excited about change. Keeping the media involved
early and often can result in a powerful partner in selling the benefits of change to
customers. And thanks to the Agile principles of focus on the customer and rapid de-
velopment, results are seen in weeks rather than years.
REFERENCES
American Customer Satisfaction Index (2014). “Citizen Satisfaction for Federal Gov-
ernment Falls as Users Encounter Difficulties with Government Websites.”
http://www.theacsi.org/news-and-resources/press-releases/press-2014/press-
release-federal-government-2013
Biddle, Toby (2013). “Shopping Cart Usability: Why a simple, honest checkout pro-
cess (usually) wins.” UX Magazine
http://uxmag.com/articles/shopping-cart-usability
Heilmann, Christian (2005). “Three Separated Layers of Web Development? Think
Again.”
http://christianheilmann.com/2005/06/16/three-separated-layers-of-web-
development-think-again/
McGovern, Gerry. (2013). “Changing the Launch and Leave culture.” New Thinking
http://www.gerrymcgovern.com/new-thinking/changing-launch-and-leave-culture
McGovern, Gerry. (2007). “The Web is Messy.” New Thinking
http://www.gerrymcgovern.com/new-thinking/web-messy
McLachlan, Gordon (2013). “Applying Agile Development to Web Projects.”
http://8gramgorilla.com/applying-agile-development-to-web-projects/
Nielsen, Jakob (2007). “10 High-Profit Redesign Priorities.”
http://www.nngroup.com/articles/10-high-profit-redesign-priorities/

9. 9
Nielsen, Jakob (2013). “Seniors as Web Users.”
http://www.nngroup.com/articles/usability-for-senior-citizens/