![“There is not the slightest
inclination that [nuclear
technology] will ever be
obtainable. It would mean
that the atom would have
to be shattered at will.”](https://image.slidesharecdn.com/edgeclassedmundsspring2013-130617140424-phpapp01/75/Transforming-Edmunds-33-2048.jpg)
Uploaded byEdmunds.com, Inc.
Transforming Edmunds
Transforming Edmunds has 500 employees, 18 million monthly visitors, and 30% mobile traffic. Edmunds has used various techniques over the years to transform including Agile in 2007, Design Thinking in 2009, hackathons in 2010, and Continuous Delivery in 2011. Recently, Edmunds has used Design Thinking techniques like personas, prototyping and usability testing to better understand users and design products. Edmunds also holds hackathons to generate new product ideas from its employees and partners. Going forward, Edmunds aims to continue innovating and disrupting itself through experimentation and a focus on understanding user needs.
More Related Content
Transforming Edmunds
- 1.
- 2. by the Numbers 500Employees 18M Visitors/month 30% Mobile Traffic
- 3. Phil by theNumbers UCLA ’92, Economics 16 years in tech @Edmunds since ‘05
- 4. A Journey WithNo End Agile 2007 Design Thinking 2009 Hackathons 2010 Continuous Delivery 2011 Lean 2012 Disrupted 20?? ITIL 2005
- 5. Design Thinking @Edmunds.com
- 6.
- 7.
- 8. Bells & Whistles(solutions focused)
- 9. Users = Clickstreams(lacked empathy)
- 10. Spec Zombies (lowteam engagement)
- 11. Not Feeling It Accordingto the Myers- Briggs Type Indicator, 80% of the Edmunds product development team are thinkers versus feelers
- 12. Edmunds Used DesignThinking To Backup and Find the User
- 13. Design Thinking Toolbox Knowthe user first… • Personas • Observation/Interviews • User Scenarios • Experience Maps …then design the product • Design Studio/Ideation • Prototyping • Usability Testing
- 14.
- 15. Personas Are Not… Statistics Demographics DistributionTargets or Static
- 16.
- 17.
- 18. Best Ideas ArePrototyped First low-fi Then hi-fi
- 19.
- 20.
- 21. Usability: Homepage Widget& Ed. Content When: First Concept launched 5/8; with year dropdown and editorial content launching soon. What: Changed placement of “Find My Next Car widget”, added drop down make/model menu, and changed the landing page. Quick quotes widget has been moved to below the fold. Hypothesis: With the design change, the team seeks to increase engagement with the widget and the landing page (vin detail NCI search results page), and ultimately lead count and lead revenue. Recipe: Control Recipe: Challenger A/B Testing
- 22. Show evidence of: •Desirability • Viability • Feasibility Decide whether to: • Continue • Change course 3month roadmaps Fail Fast to Succeed Sooner
- 23.
- 24.
- 25.
- 27. Real Source ofNew Products
- 28. Speed = Experience= Growth?
- 29. 2 Days 120 Participants 19 Teams $28,500in cash awards 50 Edmunds Volunteers 4 Problem Spaces
- 31. HACKATHON 1) Inclusive 2) Bold 3)Quick 4) Inspiring
- 32.
- 33. “There is notthe slightest inclination that [nuclear technology] will ever be obtainable. It would mean that the atom would have to be shattered at will.”
- 34. Lord Humungous vs.The Pipeline • Intuition Based • Quality First • Big Bets • Failure = Death • Experiments Based • Quantity First • Small Bets • Failure = More Experiments
- 35. All Hail theKing of Testing “One accurate measurement is worth more than a thousand expert opinions.” Admiral Grace Hopper
- 36. 1300+ active developers 175new apps/month 20+ strategic partners
- 37.
- 38. Historical # ofNew Applications per Month 5/19/13 Edmunds.com - 20 40 60 80 100 120 140 160 180 200 - 200 400 600 800 1,000 1,200 1,400 1,600 1,800 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 JuneJulyAugustSeptemberOctoberNovemberDecemberJanuaryFebruaryMarchAprilMayJuneJulyAugustSeptemberOctoberNovemberDecemberJanuaryFebruaryMarchApril 20112011201120112011201120112012201220122012201220122012201220122012201220122013201320132013 New Applications in PeriodTotal # of Applications Date New Applications in Period Total # of Applications
- 39. A Journey WithNo End Agile 2007 Design Thinking 2009 Hackathons 2010 Continuous Delivery 2011 Lean 2012 Disrupted 20?? ITIL 2005
- 40. Transforming Edmunds Philip Potloff,EVP CIO ppotloff@edmunds.com @philippotloff
Editor's Notes
- #2 Good afternoon/evening everyone. Thanks for having me, and I am really looking forward to sharing a few experiences from Edmunds. After Vijay and I discussed the idea I read through your course overview and there was one statement in particular that stood out to me: It stated:“we will focus on how business leaders should proactively and continuously adapt their enterprises to be better suited for survival and success in an always- evolving landscape.“ My plan is to share the experience we have in leadership at Edmunds applying this same focus. I will leave time at the end for a discussion and I would also encourage you to ask any questions that come up while I go through this slide deck.
- #3 First a little about EdmundsFounded in 1966 as publishing company. It is a private family-owned business. The current owner purchased the publishing rights in the 80’s and the first website went live in 1995. We have 500 employees, with 400 based in Santa MonicaThere are 18 Million unique visitors each month to the Edmunds.com website, which translates to about 250 million page views per monthOver the last three years mobile device visits to our website have grown from trace to 30% of our total visits which is not an uncommon stat these days.
- #4 Phil BioAnd a little data on me:Graduated from UCLA in 1992, but if my memory serves me well I was actually rejected from UCIStarted my career in San Francisco and Silicon Valley, and it was pretty hard to be there in the late 90’s and not be involved with technology or web companies. My experience up there includes startups, consulting, search engines and enterprise software.I have been at Edmunds since 2005 and CIO since 2010.
- #5 This illustrates why I latched on to the statement in the course overview about continuously adapting your business to emerging trends. There are episodes or stages for sure, but the journey itself does not end, and as I am sure you have been studying the pace of change is accelerating. I’d like to focus our discussion on three of these approaches in particular: Design Thinking to start, Hackathons, and I’ll close it out with a dash of disruption.
- #6 I am going to start by going way back to 2009 when most of us at Edmunds had first heard the term and were trying to figure out what it meant for us.
- #7 Now, Design Thinking is somewhat of a trendy term at the moment, but in plain English it is just a user-centered design process. In fact, when we first started exploring design thinking about a year ago and brought in the actual design professionals in the company, they were like “Oh yeah this just user-centered design.” For a consumer research site like Edmunds, we figured this should be a no brainer and was probably already in our DNA.
- #8 Well, not the case. Edmunds is a phenomenal machine when it comes to churning out new features and data for our users. And that year when we first took a look at Design Thinking, we had just pulled off the most successful project in the company’s history in redesigning our entire website from the ground up, writing more than a million lines of code in the process.The problem was that when we examined our own practices against the Design Thinking approach, we were surprised to find such a large gap between the two.
- #9 We immediately found a few issues with how we were operating. Our incremental releases began to take on a bells and whistles approach. We had a small group of people really good at writing specs and coming up with new features, but they weren’t necessarily rooted in solving the users problems
- #10 Users were best known by their Clickstreams. We do a tremendous job analyzing our data, traffic patterns and creating prediction models, but there is no factor for empathizing with the users interaction with our site or the car buying process
- #11 And something that was especially troubling for me to see happen was that our very talented technology teams were just building to a spec, barely engaged in product design, and it was starting to manifest itself in higher turnover and negative employee survey results.
- #12 Raise your hand if you have ever taken a Myers Briggs Personality Test As if to reinforce some of the endemic biases we were dealing with in our product design, When we launched Design Thinking the entire 180 person product and technology group took a Myers Briggs Personality Type profile, and it was revealed that 80% of the team naturally approach problems as thinkers (all about logic and consistency) versus feelers that want to understand how something affects others
- #13 We made the decision to go with a Design Thinking approach because it was consistent with our mission and desire as company to be empathetic and solve real user problems in the automotive space.
- #14 Design Thinking approach has a set of tools intended to first empathize with the user by understanding their problems, by building personas and experience maps through research and observation of actual people. Only after we have strong understanding of these user problems do we move on to another set of tools to create ideas and test how they might solve those problems What really resonated with me is that we are not just handing over product design from a few product owners to an even smaller group of UX designers, rather Design Thinking involves the entire team as Design Thinkers One of the most valuable outcomes of this effort to date has been the engagement level by all members of the team, from introverted developers to previously peripheral test engineers. They are all involved in product design Each of these could be presentation or book themselves, so I am going to focus on our experience with two of these tools, personas and prototyping
- #15 We started Design Thinking by building personas. We interviewed larger number of people, and there was no emphasis on finding existing users of our website. We created personas based on their lifestyle, profession and relationships. We then used key characteristics to relate how personas might interface with our website. Roberto will consider a new or used car and has a high level of comfort with the buying process Each scrum team picked primary and secondary personas They used the personas to represent things like behaviors, attitudes, and goals they used this input to guide new navigation, features and design elements.
- #16 It is important to call out that personas are not statistical or demographic representations of users. They are used to empathize and build a product that delights a user. We haven’t abandoned stats or research data, they remain part of this process.
- #17 Personas are not new or exclusive to Design Thinking, and they have been used by leading company’s for at least a couple of decades in product design
- #18 I am going to skip over what many consider to be the fun or pull your hair out in frustration part of the Design Thinking approach which is the ideation or brainstorming of ideas and go straight to where the winners of this step end up, prototypes
- #19 One of the best uses of prototypes is to get feedback at the earliest possible point in the process, and the net result is that we end up writing much less code as result of the up front tuning Early on our prototypes were extremely low-fi, and as they worked their way through usability tests the teams began experiment with more sophisticated prototyping tools.
- #20 As part of Design Thinking we have created a continuous feedback loop through a few different channels. It is striking to see how many people we now have coming through our offices for the scrum teams to engage in usability testing of their offline prototypes.
- #21 We also have internal tradeshows where every team sells their wares to the rest of the company. Quite a sight to have a sysadmin sell the chairman on their team’s new prototype
- #22 And we haven’t abandoned our traditional means of input like A/B testing, as 18 Million Unique visitors a month provide the ultimate test for products that make it onto the website.
- #23 We have been accustomed to making big, long bets on new product ideas, but with Design Thinking we have started operating with 3-month roadmaps for our various product teams. That gives teams four 3-week release cycles to demonstrate progress and show If their product is desirable, viable and feasible? A decision is made every three months whether to change course, or continue. Dogs do not get the resources to continue.
- #24 We have introduced Design Thinking across all internal and external product areas and haven’t limited this practice to the consumer facing website. Business systems and Infrastructure teams have really benefited. Just a quick example, the infrastructure team had been floundering in demonstrating value from their 2-year long cloud initiative, and after using Design Thinking to find the real problems of their users (not fancy dashboards), they were able to quickly deliver the API interface that their development and test customers really wanted. At a minimum our site is now being shaped more by the visitors that use it, and we have seen our Net Promoter Score rising as we continue to focus on the tasks our consumers value most.Because we have built our brand as a trust agent for the car buyer, that experience always comes first, but using Design Thinking as a 360 opportunity, we have been able to work with Manufacturers that advertise on our site to increase the average ad unit performance by 100% and also have built 6,000+ direct dealer subscription business in about three years by better understanding the dealer perspective in designing our offering and not viewing them as just a necessary evil to extract revenue from.Disruptive ideas have a tendency to get killed pretty early and often, but with Design Thinking in play the chances of something really transformative getting prototyped and tested are definitely more likely.
- #25 I am going to transition now from a more structured research and design process to the complete opposite end of the spectrum, and talk about how we are managing what is supposed to be an unmanageable process to put hackathons to work for Edmunds.
- #26 Not going into the technical origins of Hackathons, but at most companies Hackathons began and gained popularity as structurelessnerdathons. This is also the case for how we started them at Edmunds, and some of those original pure technical hackathons produced the first edmunds website in the cloud, real-time analytics prototype and next gen image processing tool.
- #27 We have held about a dozen Hackathons or Hack Days as we call them since 2010 and we realized early on the benefits to culture, engagement and product roadmaps and began to experiment with different formats to maximize their effect. These have become creative detours for the company that not only present product teams with new twists on existing features, but also empower anyone in the company the opportunity to contribute new ideas.
- #28 Because of the interest, we began opening them up to all employees and they became wide-open democratized product development exercises. Some of these ideas had the legs to move beyond a hack and become production applications. Pomelo, which is on display here is a great example of this. Leading up to the Hack Day where this app was prototyped, our reporting and analytics development team had been in a bit of a battle with our dealer sales operations group. Sales wanted an ipad app with 50 new report templates using a very expensive piece of software. So the dev team took it upon themselves to show a different way by way of a hack days demo and they built Pomelo, which leverages an open source touch framework, is compatible with any device or platform and only presents a handful of key metrics (as opposed to the 50 being requested). They won the event and following some user testing with dealers they proved that this was the better approach, and it has become the foundation for our entire dealer reporting roadmap.
- #29 Speed HackOur most recent Hackathons have had very specific focus areas to create business value and our last hack day event, Speed Hack, was a competition to improve perceived page load time on our site. There have been a number of studies on website performance affecting revenues and the customer experience, most notably by Google and Amazon. Even hundreds of milliseconds increase in search results affects associated revenue by double digit percentages according to Google. We agreed with this hypothesis and offered $10k in prize money to the winning teams.The results of the hack were astounding and long held belief were left in the dust. Given the time, eight self-forming teams tested hypothesis and refactored code to reduce page load time by as much as 80% on our site. The irony is that we discovered that reducing page load time doesn’t necessarily lead to a better user experience and most of our tests show poorer results the faster the page loads, so we are now exploring the optimal page load time metric as opposed to focusing on the fastest time.
- #30 HackomotiveThis past February we went big and hosted our first external hackathon to reinvent the automotive shopping experience. It was an open invitation and we didn’t have enough room to accommodate everyone that wanted to participate. Designers, students, developers, dealers, auto manufacturers, marketers were all represented. Edmunds employees only operated in support roles and we created four problem spaces to seed the process along the lines of improving the research, shopping and ownership experience and from that 19 teams of mostly total strangers self-formed around their ideas.They basically had a day and a half to create their pitch, some actually coded prototypes, and then give a three minute pitch to the crowd and judging panel. slide 4: Then boldly attempted to Re-imagine an entire vertical (Hackomotive) - image: Hackomotive concepts, problem spaces, etc.such a broad range of backgrounds: designers, students, entrepreneurs, auto manufactures, dealers, even a theologianwe are having conversations with manufacturers at a whole new level. they are opening up, slowly.
- #31 I am going to show you a short piece on hackomotive and hackathons in general that the CBS evening news ran a couple of weeks ago.After video plays:We ended up giving two first prizes of $10k each plus second and third place winners. The winning ideas both focused on a matching and rating service for individual salesmen that are most aligned with your interests.What’s funny is that the auto manufacturers love this and they have been asking us how we might come in and help them run these within their own company.
- #32 A few tips for your hackathon playbook:Inclusive - don't limit to techies that can code. More people than you know want to be involved. bold – These are freeform events and the most interesting ideas aren't usually incremental ones quick - 1-2 day events seem to be just the right amount of time to produce a meaningful demo or iterate through an idea. Inspiring - The best ideas should live on, and setting up a mechanism for that to happen has become part of our playbook. We now typically have an incentive that the winning idea is assured continued testing or actual implementation in some form One additional note I would add: Use them everywhere - kick start a project with a partner or team. I am taking the top 40 technology leaders at Edmunds tomorrow for an offsite to hack leadership challenges like how to better leverage internal analytics, the future of project management and how to introduce next generation architectures.
- #33 I have to admit that the topic of disruptive change is a little more raw as it remains a clear and present struggle for us at Edmunds, and I suspect most other companies that still have a pulse.
- #34 Einstein QuoteIf you have watched CNBC recently you may have seen this for their teaser on their top 50 disruptors. The smartest, most experience or highest paid people are often the most effective not spotting new opportunities, but rather killing them before they have a chance to flourish. It is an institutional problem that is not easily overthrown or subverted.There is not the slightest indication that [nuclear energy] will ever be obtainable. It would mean that the atom would have to be shattered at will."Atom Energy Hope is Spiked By Einstein / Efforts at Loosing Vast Force is Called Fruitless," Pittsburgh Post-Gazette (29 December 1934) It was following the breakthroughs by Enrico Fermi and others did the use of nuclear power become plausible
- #35 Lord Humungous vs. the pipelineI am going to juxtapose this Hippo model with an alternative approachFirst, does anyone recognize Lord Humungous? I prefer him as a proxy for the HPPO.Even though pretty much every idea starts as intuition or insight, the Lord Humungous path probably allows bad ideas to have life a few too many steps. The alternative here is a relatively free-flowing pipeline of experiments and ideas. Ways to rapidly test big and small changes with limited exposure or risk to the businessIn early stage brainstorming, the default approach is also a total creativity killer. Focusing on the merits and quality of ideas at this stage lead to filtering, self-sensorship create the fear of being jeered and don’t produce collaborative environments when this force of personality is present and only something that seems consistent with humungous’ views get raised. An effective pipeline supports a large number of simultaneous mature product tests as well as purely directional experiments to resolve debates or conflicts of opinion.The risk profile of these two approaches is completely different. The small number of big bets gives you lottery odds, and upon failure careers are often in jeopardy. As you heard from Pedram at Facebook in the Hackomotive video, you need to have the ability to test 99 bad ideas to get to that one good one, and that is the first principle of the pipeline approach.
- #36 Slide 3: Amazon recommendation serviceAmazon is considered the leader with no equal when it comes to web testing and optimization, but there is some humble pie in their journey to that dominant position. Has anyone heard the story of how the personalized recommendations feature came to be?Greg Linden at Amazon created a prototypeto show personalized recommendations based on items in theshopping cart (2). You add an item, recommendations show up;add another item, different recommendations show up. Lindennotes that while the prototype looked promising, ―a marketingsenior vice-president was dead set against it,‖ claiming it willdistract people from checking out. Greg was ―forbidden to workon this any further.‖ Nonetheless, Greg ran a controlledexperiment, and the ―feature won by such a wide margin that nothaving it live was costing Amazon a noticeable chunk of change.With new urgency, shopping cart recommendations launched.‖And to show why Amazon is in the position it is, to their credit, they just didn’t make Greg Linden the next Lord Humungous, instead they created a testing and innovation pipeline to ensure they didn’t miss the next opportunity like this because of an expert opinion.Creating this type of pipeline is a major focus of mine at Edmunds, and Amazon is a great model for us, but while this is a great story, some view it as model for low-end disruption and growth?
- #37 Perhaps that is true, and I am going to share a little about our API program, which we hope will reveal new market disruptors As I mentioned earlier, in the very early days of the web we were a print publishing company, and put all of our pricing guides online for free with no revenue model. It was an important experiment that transformed the company. Some would say that our API program is a similar leap of faith experiment. If you’re not familiar with APIs, or Application Programming Interfaces, they are basically a common method for retrieving data, and in this case we have made made most all of the proprietary Edmunds datasets available for free The rational is three-fold:Make Edmunds the ubiquitous source of all vehicle data and contentIdentify revenue-share partnerships from existing lines of businessBuild an adjacency to new ideas as they are developed by enterprises, startups and individuals using Edmunds data
- #38 Through the API program over a thousand companies and individuals are using our data for new product development within enterprises like ebay, ADP and cobaltManufacturers including Toyota and Honda, as well as startups and individuals that are working with fresh concepts.
- #39 You can’t give a business school presentation without at least one chart, so I threw one in here at the end to illustrate how we are leveraging the capabilities and ideas of outsiders to drive innovation. We are seeing 175 new apps per month and climbing, which is way more ideas than we could possibly test. I can’t say with any confidence how Edmunds would respond to the next disruptive external threat, but rather than operating in fear of it, I hope we are using the API program to nurture and draw closer to what might become the next great partnership or line of business for Edmunds, even if it originates outside of our walls.
- #40 The lifespan of our current technology platform is only about three years at this point, and we are already designing the replacement for what we just finished building… and now business transformation seems to be approaching this pace. Again, this is a journey with no end but I can sum up the Edmunds transformation experience to date as: love your customer, hack your product and kill lord humungous.