Improve and Prevent with Analytics Hygiene
•Download as PPTX, PDF•
1 like•418 views
The document discusses the concept of analytics hygiene, which refers to practices that improve analytics programs and prevent issues. It involves maintaining cleanliness, governance, and general prevention practices for reports, processes, and data. Good analytics hygiene includes collaborative reporting, well-defined processes, and ensuring data quality through practices like anomaly detection. Bad hygiene can result in diseased reports, stagnating processes, and rotten data. The document emphasizes continuous improvement and prevention.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (16)
Similar to Improve and Prevent with Analytics Hygiene
Similar to Improve and Prevent with Analytics Hygiene (20)
Recently uploaded
Recently uploaded (20)
Improve and Prevent with Analytics Hygiene
- 1. analytics hygiene eMetrics Chicago 2014 tbdac.com | @toddmetrics
- 2. analytics hygiene: what? tbdac.com | @toddmetrics Google, define hygiene. “conditions or practices conducive to maintaining health and preventing disease” Thanks, Google.
- 3. analytics hygiene: but what is it about? Analytics hygiene is about improvement and prevention tbdac.com | @toddmetrics
- 4. analytics hygiene: improving/preventing what? tbdac.com | @toddmetrics Good Stuff Bad Stuff
- 5. analytics hygiene: introduction tbdac.com | @toddmetrics
- 6. analytics hygiene: one more time Analytics hygiene is about improvement and prevention tbdac.com | @toddmetrics
- 7. analytics hygiene: improvement and prevention tbdac.com | @toddmetrics Good Stuff Bad Stuff • Trust in data and analyses • Spending time on actionables, improvements, prevention • Collaborative analytics environment (involvement, participation, inquiry) • … • Misinterpretations of data • Treading water • Stagnating analytics environment (ignorance, repetition, degrading in quality) • …
- 8. analytics hygiene: good and bad All analytics programs have hygiene. There’s good analytics hygiene… and there’s bad analytics hygiene. tbdac.com | @toddmetrics
- 9. analytics hygiene: improvement and prevention tbdac.com | @toddmetrics Good Stuff Bad Stuff • Trust in data and analyses • Spending time on actionables, improvements, prevention • Collaborative analytics environment (involvement, participation, inquiry) • … • Misinterpretations of data • Treading water • Stagnating analytics environment (ignorance, repetition, degrading in quality) • …
- 10. analytics hygiene: good and bad All analytics programs have hygiene. Question: How’s yours? tbdac.com | @toddmetrics
- 11. analytics hygiene: better and better All analytics programs have hygiene. Question: How’s yours? Accepted Answer 1: Better than it was. tbdac.com | @toddmetrics
- 12. analytics hygiene: better and better All analytics programs have hygiene. Question: How’s yours? Accepted Answer 2: Not as good as it could be. tbdac.com | @toddmetrics
- 13. Diseased reports Stinky processes Rotten data analytics hygiene: improvement and prevention tbdac.com | @toddmetrics Bad Stuff
- 14. analytics hygiene: the principles The principles of analytics hygiene: tbdac.com | @toddmetrics
- 15. analytics hygiene: the principles Antifragility if something breaks, fix it and make it stronger as a result. Frameworks having a plan is the best plan. Omnivision know the past and future of your program, from organization to technologies. Outsiders don’t stay in your silo. external learning. Collaboration hey, this whole thing is bigger than you. tbdac.com | @toddmetrics
- 16. analytics hygiene: hygiene? tbdac.com | @toddmetrics 1. Cleanliness from http://darkhorseanalytics.com/blog/data-looks-better-naked/
- 17. analytics hygiene: reports tbdac.com | @toddmetrics 1. Cleanliness 2. Governance 3. General hygiene… prevention.
- 18. analytics hygiene: diseased reports and general hygiene tbdac.com | @toddmetrics Prevention: Stop the diseased report from being created. Improvement: Periodic reviews of what’s out there.
- 19. analytics hygiene: collaborative reporting tbdac.com | @toddmetrics Context is king
- 20. analytics hygiene: collaborative reporting vs the opposite Learn. Distribute accountability. Accountability for… understanding the data. understanding how the data informs action. tbdac.com | @toddmetrics
- 21. analytics hygiene: collaborative reporting techniques Technique #1: Ask what’s so important about this report, anyways. (really!) example starters: “At what point would you need to be alerted to something within this report?” “What can I keep an eye on for you?” tbdac.com | @toddmetrics
- 22. analytics hygiene: collaborative reporting techniques Technique #2: If you get a response… ask (at least) two more questions! tbdac.com | @toddmetrics
- 23. analytics hygiene: reports (closing) Antifragility periodic reviews. automation/alerting (more on that later) Frameworks AARRR, gap analysis, other reporting frameworks? Omnivision technology gaps, previously perceived gaps that weren’t. Outsiders what sorts of reports do others depend on? ASK. (gain exposure to other tools / features, gain exposure to ideas, etc.) Collaboration other collaborative reporting techniques? tbdac.com | @toddmetrics
- 24. analytics hygiene: processes Signs of a stinky process: tbdac.com | @toddmetrics
- 25. analytics hygiene: processes Signs of a stinky process: 1) It doesn’t exist tbdac.com | @toddmetrics
- 26. analytics hygiene: processes Signs of a stinky process: 1) It doesn’t exist 2) It isn’t followed, or doesn’t achieve its objective 3) It adds complexity and frustration tbdac.com | @toddmetrics
- 27. analytics hygiene: optimization process management tbdac.com | @toddmetrics tim.webanalyticsdemystified.com @tgwilson
- 28. analytics hygiene: optimization process management / BugHerd tbdac.com | @toddmetrics
- 29. analytics hygiene: automation services tbdac.com | @toddmetrics
- 30. analytics hygiene: tagging processes tbdac.com | @toddmetrics
- 31. analytics hygiene: tagging processes tbdac.com | @toddmetrics 1. Data layers aren’t just about objects
- 32. analytics hygiene: tagging processes tbdac.com | @toddmetrics 1. Data layers aren’t just about objects 2. Technical expertise will always be necessary
- 33. analytics hygiene: processes / tagging tbdac.com | @toddmetrics
- 34. analytics hygiene: processes (closing) tbdac.com | @toddmetrics Antifragility periodic review (sensing a theme?) Frameworks ADAPTAL, tagging frameworks, change management Omnivision what has worked/failed? changes to consider? Outsiders what processes do others depend on, or think might help? Collaboration build together, follow together.
- 35. analytics hygiene: data tbdac.com | @toddmetrics 1. Cleanliness 2. Data Governance Universal Data Governance Principles: 1. Integrity 2. Transparency 3. Auditability 4. Accountability 5. Stewardship 6. Checks and Balances 7. Standardization 8. Change Management from the Data Governance Institute
- 36. analytics hygiene: second guessing vendor reported data tbdac.com | @toddmetrics Blasted non-humans!
- 37. analytics hygiene: data… remember this? tbdac.com | @toddmetrics Good Stuff Bad Stuff • Trust in data and analyses • Spending time on actionables, improvements, prevention • Collaborative analytics environment (involvement, participation, inquiry) • … • Misinterpretations of data • Treading water • Stagnating analytics environment (ignorance, repetition, degrading in quality) • …
- 38. analytics hygiene: rotten data vs antifragility Data antifragility: Anomaly/normality detection. No excuses! tbdac.com | @toddmetrics
- 39. analytics hygiene: rotten data tbdac.com | @toddmetrics Antifragility alerting! normality! anomalies! trends! DETECTION! Frameworks see above Omnivision why is it the way it is, and what will it take to modify? Outsiders what new types of data are relevant for others? Collaboration speak/work with those who capture, who control.
- 40. analytics hygiene: in closing… Diseased reports. Stinky processes. Rotten data. tbdac.com | @toddmetrics IMPROVE and PREVENT
- 41. analytics hygiene: thank you tbdac.com | @toddmetrics You’re good people. Source: You opened, attended, read, or otherwise engaged with something that references “analytics hygiene”. Deck will be available at slideshare.net/ToddBelcher. See remaining slides for references and supplemental content.
- 42. analytics hygiene: references & recommended reading tbdac.com | @toddmetrics
Editor's Notes
- Welcome! THANK YOU! I have planned for you about 10 or 15 minutes of background and general motivational speaking, followed by strategies and tactics bucketed in three groups: reports, processes, and data. Timer: 10-15 minutes, followed by three 7-9 minute sections
- what is analytics hygiene? It’s a two-word phrase I chose to broadly categorize some tactics and strategies intended to improve the output of an analytics program (and prevent the opposite). “Analytics hygiene” is a unifying theme for some of my content, but it has also become a new way of thinking about what I’m up to as a digital analytics manager.
- improving/preventing what? Good question! I like to think that a digital analytics managers primary objectives include improvement (increase occurrences of “good stuff”), and prevention (decrease occurrences of “bad stuff”)… which in itself really is an improvement. Improvement. It’s why analytics is here! We’ll get to more details on what the good/bad stuff might be in a minute.
- Introduction/background… why I’m here (at eMetrics)? There are many, many reasons… The tl;dr version of my professional life timeline: Freelancer / Online business owner Enterprise software / services / support in Web analytics Digital analytics manager (consultant) in financial services (primary / full-time) and others (secondary / part-time)
- Analytics hygiene: improvement and prevention There are plenty of other examples of good/bad stuff – have one? +$$$ -$$$
- Motivational speaker time. You can’t not have analytics hygiene. Might as well have good hygiene then, right? What’s good versus bad hygiene?! Good analytics hygiene is simply having a framework and setting the time aside to improve and prevent. It’s not so much about where we are, or where we were, but where we’re heading. At the program / organizational level and an individual level. Being persistent and consistent. Bad hygiene leads to more bad stuff… let’s go back for a second and talk about things that lead to good/bad stuff!
- improvement and prevention / good stuff and bad stuff Anyone have examples of what has led to “good stuff” or “bad stuff”? A few maybe: Demonstrated ability to meet goals through the use of data (e.g. cut costs, gain revenue, build user base Intentional over-sharing (or is this just good communication) Persistence and consistence in approach Good stories w/data to back them up vs Slop Analysis paralysis (which slop can lead to) Lack of process
- Now that we’ve reviewed what this analytics hygiene thing is all about, it’s time to think about our own programs. How’s your analytics hygiene? I believe there are two acceptable answers:
- Variations also accepted.
- Variations also accepted.
- Almost through the motivational speaking. Bad stuff… how do we decrease it? Fire everyone and hire new people? Our community generally accepts that people are critical. Maybe you don’t have an all-star team. (Maybe you ARE the team.) Don’t let this become an excuse. Regardless of the team at hand, you should be able to come to consensus on good and bad ways to do things. Leave a mark on the organization, instead of letting it stagnate. Be a driving force… and remember, data doesn’t drive people, people drive people. Let’s talk through a few strategies and tactics!
- As I embarked on an analytics hygiene writing adventure, I bumped into a few recurring themes… the principles of analytics hygiene.
- As we go through some strategies and tactics, these principles of analytics hygiene will be referenced repeatedly… here they are with the shortest possible description I could come up with. For longer descriptions, keep listening to me or visit my blog at tbdac.com. On to the reports/processes/data!
- Analytics hygiene: diseased reports As much flak as the word “report” itself has taken of late, reports are a part of our lives… right? It’s tough to get things done in analytics without reports. There are a couple specific areas where the word “hygiene” lends itself to reports.
- Analytics hygiene: diseased reports Then you have governance… bless it and every governance-focused person on the planet… I’m not going to really get into governance a ton, though clearly based on the similarity of some of the concepts I am a fan. It will come up a couple times throughout the presentation. When I’m thinking of analytics hygiene and reports, I’m thinking about general hygiene … as in… prevention!
- diseased reports and general hygiene What might some signs of diseased reports be? Not trusted. Not viewed. Not used. Not accurate or otherwise misleading.
- Collaborative reporting Do these charts represent disease? Maybe. Context is king. Why were they created? Who is looking at them and how are they being used? It is so easy to misinterpret data. All it takes is a chart and a person. Consider where you leave your reports and who has access to them.
- collaborative reporting vs the opposite
- Technique #1: When an informational report is requested (no clear actionables), ask what you should be looking for within the data to trigger alerts when important events happen. Whether or not you can accomplish this doesn’t even matter. Just change the question to ask what you would be looking for if you had the capability. The exercise of thinking about what is important about the data will help drive towards something actionable… although not directly. If the response you get, for example, is … “we like to see growth in our user base”, you now can take your totals and add growth metrics……..
- Know when to cut bait and move on… but, if you are getting engagement… here is technique #2: If you get a response, take it as an opportunity to ask at least two more questions. Okay. You know you are going to be dealing with growth metrics. Is there a time period, rolling or static, that would be helpful to report out on? What are the reasons for that? If the growth over that time period is at a certain high or low threshold, what would we do and who would need to know about it? What might be leading/lagging indicators of success or failure to meet our goals? Getting folks thinking about these things is good hygiene.
- Other reporting strategies and tactics that align well with the principles of analytics hygiene: Antifragility – automation. Ever been caught without required data for a report? Monitoring and alerting… can’t say enough good things about it, and will definitely get back to it. Frameworks – ARRR and QuEST are good examples of reporting frameworks. I’ve witnessed plenty of great gap analysis frameworks, where a project manager or analytics manager of some kind really hones in on what the program has versus what it needs. Omnivision – The gap analysis framework is omnivision, too… and then there’s this anecdote: preparing to move on from a dying platform (separation, parallel/triangulation) Outsiders – Get a sense of the reports that other folks depend on. It takes care of a couple things at the same time. You get exposure to other tools and feature sets. You get exposure to other ideas and ways of doing things. Collaboration – Anyone have collaborative reporting techniques? We’ll probably end up brushing past a couple more before the end of my time either way…
- On to processes. What might the signs of a stinky process be?
- #1. Yes. Process!
- As you can imagine… there is one analytics hygiene principle in particular that lends itself to the topic of stinky processes: frameworks. Sometimes you have to bend. The framework, the process, yourself… you have to bend to meet the people you have and get them to move. Anecdote: social media & tracking codes… the wild west. From a simple code not tied to anything to a unique identifier and metadata. Sometimes, all it takes is creativity and will… whether you have a coder, or you can spec something out well enough for a coder to take care of it. Another fine example of an area where process is sorely needed: optimization!
- Not sure how many of you may have had a chance to catch Tim Wilson speak on optimization process management earlier? You should have ALL been there. I’ve seen it several times and I was there. I will see it again, I hope. Process management in analytics is a beast. A beast that Tim has tamed with ADAPTAL. It’s not just about testing “optimization”, but about improvement… like… analytics hygiene. I won’t speak to it much more, because I am not Tim… but I will focus for a second on the DAP from ADAPTAL: Discover Assess Prioritize
- Watch BugHerd video & comment. BugHerd + Google SpreadSheets + Zapier
- Watch BugHerd video & comment. BugHerd + Google SpreadSheets + Zapier
- Tagging frameworks. I can’t speak to tagging frameworks. I’ve been around too long, have seen too many things. I’ve become radicalized as you will see in a minute. Regardless of tagging methodology, no doubt you’ve come across the concept of a “data layer” by now… The two things I would state on tagging before I do get into my crackpot ideas is that
- Data layers are the real deal if you’re planning on working with more than one technology. With your omnivision glasses, consider the data layer before the multiple technologies are introduced. You might be surprised at how much the data layer helps socialize analytics within the organization…
- Track everything. Here are some screenies of my radical jibber jabber. I’ve made an effort to document my feeling that the very methods many of us use today to collect Web and other digital data are flawed, even archaic. I don’t want to get all tin foil hat here in this presentation, but find me later and let’s do it up.
- Other reporting strategies and tactics that align well with the principles of analytics hygiene: Antifragility – Periodic reviews (again). Frameworks – We mentioned ADAPTAL, tagging frameworks… change management processes are near and dear to me. I am fortunate to have IT partners who take change management to heart. It can be bulky at times, but having spent plenty of time working through issues caused my a lack of change management… I just get warm and fuzzy when I know I’m being looked out for. Omnivision – The principle is a principle for a reason. For any category, you can ask what has worked/failed and consider whether it may be time to make changes. Outsiders – What processes do others depend on? What do others with experience think might help, if you’re struggling in a particular area? Anecdote: tracking EVERYTHING. No, not that everything. Everything else associated with dates. Collaboration – a processes that is built as a partnership has a better chance of survival. You also get to shirk some of the responsibility/accountability.
- There are all kinds of rotten data. Scrubbing is great, validating and checking for bad data is super important and I’ll get to that in a minute after. Data governance and analytics hygiene don’t sound all that different, do they.
- Explain the CTR tactic (measure click to view rate, click to visit rate… compare across pubishers and do business with whoever you trust the most) … very interested in view-through analysis, and what sorts of things folks are doing to reconcile in those areas. Another interesting area of money getting sucked away in digital advertising is search arbitrage, a years old issue that continues… Flash Boys analogy…
- Doesn’t a lot of this depend on “good” data to being with? Get a tool like ObservePoint or Hub`Scan to do real time monitoring if you’ve experienced pains and can swing it… for live data issue notifications… but then, through your tool or other means… investigate normality, or anomaly detection!
- http://statistics.about.com/od/Formulas/a/Standard-Normal-Distribution-Table.htm The estimator… 12 weeks of data (same day of week) for me. What about you?
- Data anti fragility... Build normality /anomalydetection. No excuses. Outsiders will help you call out organizational bad habits. Omnivision prevents you from making the same mistakes, and prepares you for the future. Putting frameworks in place will keep your data cleaner. Frameworks Data governance. Frameworks are everywhere, and involve everything from scrubbing and format/syntax consistency to internal systems of checks and balances for change management.
- Tim Wilson’s presentation and other materials: http://bit.ly/webprocess Normality reference again: http://statistics.about.com/od/Formulas/a/Standard-Normal-Distribution-Table.htm Consultancies: Web Analytics Demystified, Cardinal Path, L3, independents, etc… outsiders are good.