Download as ODP, PPTX
![Thanks for listening. ご清聴、ありがとうございました。 [email_address] ← My email. http://www.kalzumeus.com ← My blog. http://www.abingo.org ← A/Bingo software (OSS) http://www.startuplessonslearned.com ← Eric Ries, very useful.](https://image.slidesharecdn.com/designmatterspresentation-100422045319-phpapp02/75/Data-Driven-Software-Design-40-2048.jpg)
Uploaded byPatrick McKenzie
Data-Driven Software Design
The document discusses using data and statistics to improve software and websites. It covers instrumentation to track user behavior, funnel analysis to identify where users drop off, and A/B testing to compare design alternatives. The goal is to supplement design with scientific process to decrease costs, uncertainty and increase satisfaction, success and profits by learning from user data.
More Related Content
![[HCM Scrum Breakfast] How to improve product quality in Scrum Team](https://cdn.slidesharecdn.com/ss_thumbnails/aavnscrumbreakfast10-151102070108-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Data-Driven Software Design
![[DSC Adria 23]Rok Piltaver Lessons Learned From 10 Years of AB Testing.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/rokpiltaverlessonslearnedfrom10yearsofabtesting-230530180353-baf610f5-thumbnail.jpg?width=640&height=640&fit=bounds)
Data-Driven Software Design
- 1. Data-driven Design MakingSoftware & Websites Better Through Use of Statistics Patrick McKenzie April 15th, 2010 Creative Commons Licensed CC-BY. Please feel free to share.
- 2. Outline Introduction Art& Science Rationale Instrumentation Funnel Analysis A/B Testing Conclusion 自己紹介 芸術と 技術 目的 動作監視 ファネル分析 A/Bテスト まとめ
- 3.
- 4. What Happened D:\Dropbox\MyDropbox\Photos\Sakura Season 2010\IMG_0348.JPG
- 5.
- 6.
- 7. Art into Science芸術から技術へ Supplement the vision and craft of designers with the logic, regimentation, and process of science. デザイナーの芸術を技術の倫理、数学、体験方式で拡張する。
- 8. Rationale 目的 DecreaseCosts Decrease Uncertainty Increase Satisfaction Increase Success Increase Profits 開発工数ダウン 不確実性ダウン ユーザ満足度アップ 成功率アップ 利益アップ
- 9. Designing With DataGet Facts to Answer The Questions: What are users doing? 何をしているか？ Which users are doing things? 誰がしているか？ Where are they doing things? 何処にしているか？ (Maybe) Why are they doing it? 何故しているか？
- 10. Types of InstrumentationAnalytics software Heatmapping Surveys Ad hoc analysis (Excel, etc) Custom software
- 11.
- 12. Landing Page Photoof problem solved goes here
- 13.
- 14. Paving The TrailsPicture of paving a path
- 15.
- 16.
- 17. Funnel Analysis ファネル分析Many applications have several steps followed by a goal. Only a percentage of users clear each step. Funnel analysis finds where folks are falling out, to concentrate efforts on improvement Viewed Landing Page Signed Up For Trial Did Work $$
- 18. Software ソフト GoogleAnalytics Free but limited Good for starter Mixpanel Cheap API Harder to use
- 19. Define Funnel SignsIn Opens Word List Saves Word List Customizes Cards Creates Cards Downloads Card
- 20. Collect Data SignsIn: 260 Opens Word List: 218 (83%) Saves Word List: 191 (87%) Customizes Cards 161 (84%) Creates Cards: 131 (81%) Downloads Card 126 (95%) Total Conversion Rate 126 / 260 = ~48.4%
- 21. Identify Areas forImprovement Signs In: 260 Opens Word List: 218 (83%) Saves Word List: 191 (87%) Customizes Cards 161 (84%) Creates Cards: 131 (81%) Downloads Card 126 (95%) Total Conversion Rate 126 / 260 = ~48.4%
- 22. Implement Improvements OpenWord List needlessly complicated: make easier. Save Button was unclear: tweak microcopy, button design. Customize Cards needlessly complicated: show less options.
- 23.
- 24.
- 25. Collect Metrics SignsIn: 194 Opens Word List: 155 (79%) Saves Word List: 151 (97%) Customizes Cards: 127 (84%) Creates Cards: 112 (88%) Downloads Card 126 (98%) Up from 83% Up from 81% Total Conversion Rate goes from 48.5% to 56.7% 16.6% lift in customer success / revenue for 1 hour of work .
- 26. http://farm1.static.flickr.com/21/33346213_e5d5842024.jpg Pretend YouOwn A Bar Salty Peanut by EuroMagic. Half of Customers Get... Half of Customers Get...
- 27. A/B Testing Dividesimilar population into two groups randomly. Treat the groups differently. Measure difference in outcomes. Scientific Method applied to design . Eliminates bias, randomness, poor test design.
- 28. Actual A/B TestHalf of Users See... Half of Users See...
- 29. Software Tools GoogleWebsite Optimizer Terrible. Do not use. Visual Website Optimizer visualwebsiteoptimizer.com WYSIWYG alternative creation. Very nice. Free for now. A/Bingo or similar Rails A/B testing library. Programmer friendly, powerful, easy to use.
- 30. Core A/B TestingLoop Identify conversion and instrument. Create alternatives. Assign users randomly to alternatives Watch behavior. Identify winning alternative. Iterate, iterate, iterate...
- 31. Landing Pages experianvasmexperianvbsm Source: whichtestwon.com
- 32.
- 33.
- 34. Signup Forms Vastcontact dealer form
- 35. New Features http://farm3.static.flickr.com/2422/3981364314_d4b30cb739.jpg60 ~ 80% of features do not change user behavior Eric Ries
- 36. You can quicklycreate bingo cards using our included wizards! Old Version
- 37.
- 38. Core A/B TestingDiscipline Accept that you do not know what is right. ( Hardest lesson. ) Test frequently, as a matter of routine. (Helps to have one-line testing.) Test changes in isolation. Use evidence from tests to inform your marketing and development plans.
- 39. Conclusion What welearned today: Importance of tracking user interaction. Funnel analysis – measuring user success and goal completion. A/B tests – improving marketing message and product design.
- 40. Thanks for listening. ご清聴、ありがとうございました。 [email_address] ← My email. http://www.kalzumeus.com ← My blog. http://www.abingo.org ← A/Bingo software (OSS) http://www.startuplessonslearned.com ← Eric Ries, very useful.
- 41. Microsoft logo: http://www.flickr.com/photos/scobleizer/2264763977/Sleeping Salaryman: http://www.flickr.com/photos/kansai/3452417503/ Edamame (Green Soybeans): http://www.flickr.com/photos/kankan/33346213/ Salty Peanuts: http://www.flickr.com/photos/euromagic/2351628831/ Facepalm Statue: http://www.flickr.com/photos/cesarastudillo/3981364314/ Photo Credits
Editor's Notes
- #2 I delivered this presentation on April 15 th , 2010 to the Design Matters group at the Apple Store in Osaka. Please feel free to re-use any part of it. Pictures are mostly from other people and licensed CC-BY as well: see last page for attributions. Non-attributed pictures are my own -- you can consider them CC-BY as well.
- #3 Interesting note: Garr Reynolds, the guy behind Presentation Zen, suggested this slide get excised later (“Outline slides are boring, cost attention, and distract from the narrative.”). Oh, what will I ever do without structure…
- #4 Back when I was in college, outsourcing was in full swing and I thought I would be unlikely to be competitive with hundreds of thousands of Indian/Chinese engineers if I only knew how to program. So I decided to pick up a language, and stumbled into Japanese. I was going to get a nice, safe job at Microsoft or some similar international company after graduation, but didn’t feel my Japanese was up to snuff, so…
- #5 I came to Ogaki (a city of 150,000 in Gifu Prefecture) in 2004 to be a technical translator for the prefectural technology incubator. I worked there for three years, doing a bit of translation, a bit of interpreting, and a bit of R&D when I had neither to keep me busy. I grew to love Ogaki so much that when my contract ended I wanted to stay a little longer, so I became…
- #6 … a Japanese salaryman , at a large company in Nagoya. (No, not THAT large company in Nagoya. One of the other ones.) I learned an awful lot about being a professional engineer, as opposed to the hobbyist with coding skills that I was on graduating university and working at the very low-stress position at the technology incubator. On the flip side, being a salaryman is sort of not conducive with work/life balance, so I was ready for a change after 2.5 years. While this was happening…
- #7 Brief introduction of my business. Bingo Card Creator: creates bingo cards for teachers, mostly for use in lessons. Started 7/2006 (sideline business), went full-time on 4/1/2010. Desktop application (Java) + web version. ~3,000 customers, 500,000 cards created in last year, millions upon millions downloaded.
- #8 Making interface/interaction design into a science, not just an art. Art is lovely, but: * Failure is too common * Uncertainty is too high * Beliefs are too unquestioned * Competent practioners are too rare Using scientific approaches means * Repeatable, scalable processes * Mathematically quantifiable certainty * Constant change with users * Any idiot (i.e. me) can do it
- #9 Business needs addressed by this presentation * Cost cutting: rather than spending lots on a one-shot, do many cheap experiments, develop what works. * Uncertainty: Acknowledge that we don't know what solution looks like, work to discover it, rather than praying we guess right. * Satisfaction: Adapt to measured preferences of users. Give them what they want, not what they ask for. * Success: Measurable improvements in task success. * Profits: All of the above means you make $$$.
- #10 High-level description of software instrumentation. Software instrumentation means tools & techniques for: * Making your software capture usage data * Getting the data out again * Analyzing it * Acting on what you learn
- #11 Quick overview of several types of instrumentation.
- #12 CrazyEgg: heat mapping software. The more people click on an interface element, the ”hotter” it appears. Lets you quickly see what users are actually focused on on your pages/applications. Technical notes: works via Javascript. Business notes: dirt cheap ($9, $19, etc monthly)
- #13 Actual example of a problem solved by CrazyEgg This is a landing page for an adwords campaign. Goal is to have customer click on Big Ol' Button to sign up for service and hopefully purchase it. Note left hand column is not active , but customers are clicking on it. Frustrated customers leave page without signing up, costing money! Note how it is very obvious what the problem is – one look at this shows it. Nice feature to have in instrumentation – the less work you have to do to get useful results, the more likely you are to actually use it.
- #14 Redone version of same page Note left side now has links, and they're being clicked on.
- #15 Paving the trails: observe what people do, then, direct your resources into making what they do better. Has great relevance for creating/extending features, creating content, etc . This story went over VERY well at the presentation: There was once a park in America created by a famous architect. When he delivered his final set of specs, the owners realized that he hadn’t included any roads in the park. They asked him how he was such a famous architect and could forget something more basic. ”Ahh, but I have not forgotten the roads – I merely do not know where to put them yet.” So they asked how he could be a famous architect and not know where to put the roads. ”Oh, I have some ideas. But then I would be laying roads just for me. So I’m going to wait a while, and watch as people walk within the park. The paths they past most often will have the grass stamped down, and trails will form, worn by each pair of passing feet. Then I will tell you where to put your roads. Pave the trails, because a road is where people want to walk. 歩くところこそが道である。 ” In software, if you want to know where you should spend your resources: pave the trails !
- #16 Standard overly complicated options dialog What do customers actually perceive as valuable here, and how can we cut it such that their experience improves?
- #17 Sample of 10,000 customers. Red numbers indicate percentage who tweaked the particular setting. As we can see, customers care about some far more than others. Note that consistently ~8% of customers tweak EVERYTHING. Power users. Identify power users automatically, give them the full version of this screen. Let normal users see only the stuff they're likely to perceive value from. Increases success tremendously (see next section on Funnels).
- #18 Explanation of funnel analysis
- #19 Software to use for Funnel Analysis I really, really like Mixpanel . See http://www.bingocardcreator.com/articles/tracking-with-mixpanel.htm
- #20 Steps folks go through in the core interaction loop in Bingo Card Creator.
- #21 Collecting data on user success at every stage in core interaction loop.
- #22 Find obvious candidates for improvement of core interaction loop.
- #23 Figure out ways to make core interaction loop better at hotspots identified for improvement. These days I’d A/B test all of these suckers, but I did not have a convenient way of doing so at the time.
- #24 The complicated version of the Customize form.
- #25 The simplified version of the Customize form – shown by default now to any user who heuristics identify as a “newbie”. (Heuristics are another topic I could do a presentation on. Ask me some time!)
- #27 Pretend you own a bar in a hip hop happening town like Osaka. Your business model is, obviously, getting patrons to order drinks. There are a lot of things you could do to improve how many drinks you sell: maybe spend more money on advertising the bar, or hiring comely waitresses to stand outside and hawk your bar to passersby, or changing the menu, or even moving locations. But these all cost money and you’re not guaranteed that any will work. And who is to say that if sales go up, it is because of the thing you changed, as opposed to seasonal factors or blind chance? So, because you’re a smart bartender, you start small: you figure “hey, maybe salty peanuts will make people drink more beer”. Peanuts are not more expensive than the edamame (that green soybean) you’re giving folks as an appetizer right now. So you have your staff flip a coin for every party that comes in. Heads, they get edamame , tails, they get salty peanuts. This gets marked discretely on their check. Then, at the end of the week, you sum up how many drinks are ordered by both groups, and apply a little math magic. Since your customers are exactly identical with the exception of their appetizer – because your coin does not play favorites – you know that if your math magic says “Wow, peanuts really work!” then peanuts really work. And if on average peanuts cause every person to order 5 drinks instead of 4, then just switching to peanuts increases your sales 25% without hiring more staff or moving to a new location. Welcome to A/B testing.
- #29 This was around Halloween. Depressingly, the Halloween-themed signup page got thumped.
- #32 Landing pages are easy to test: constant incoming traffic, can create alternatives virtually without impact to rest of site, customers are extraordinarily sensitive to small changes because they don’t yet have an investment in a relationship with you. Test on right won.
- #33 Boring signup form. Signup forms are another great place to test, because a) everybody goes through them (changes have immediate, massive business impact) and they’re so easy to screw up.
- #34 Prettier signup form. Won, by a bit.
- #35 The madlibs test. Not a great A/B test design – too much changes at once, so it is hard to isolate cause of improvement – but the right form squashed the left one. Note when I tried a similar test madlibs lost by quite a bit. Lesson: run your own tests.
- #36 A/B testing can help you tell which 80% of your engineer hours are being wasted prior to burning all of them. If the simplest possible version of the feature gets NO uptake – nobody even clicks on the screen to open it -- then you can be fairly confident that pouring more resources past the click is a waste.
- #37 Bingo Card Creator 2.51.
- #38 Bingo Card Creator 3.0. Did you see the change I labored for a month on? Nope? Good, you’re in good company with about 59,978 of the first 60,000 people to use this version. (The new features are under the Internet menu. Huge failure in giving customers what they wanted and in communicating the existence of a feature they wanted – ask me some other time.)