Measuring What Matters - Building Valuable, Effective & Efficient Reporting
•
3 likes•1,322 views
The talking points to the presentation are on the Notes tab. This presentation was given at the Salt Lake City Search Engine Marketing Digital Marketing Conference (#slcsem) in August 2015. In it, I don't talk about KPIs, or the specific metrics that you should be measuring. Instead, I focus on what I have found to be really important - Building reporting that you can use, and building relationships so your reporting has value. I also discuss how building a report is like building a product. Then I finish off with some quick tips on making your reports better.
Recommended
Recommended
More Related Content
Similar to Measuring What Matters - Building Valuable, Effective & Efficient Reporting
Similar to Measuring What Matters - Building Valuable, Effective & Efficient Reporting (20)
Recently uploaded
Recently uploaded (20)
Measuring What Matters - Building Valuable, Effective & Efficient Reporting
- 1. @RickGalan#SLCSEM @RickGalan#SLCSEM MEASURING WHAT MATTERS BUILDING VALUABLE, EFFECTIVE AND EFFICIENT REPORTING
- 2. @RickGalan#SLCSEM @RickGalan#SLCSEM #SLCSEM Rick Galán Director of Digital Marketing at Blendtec @RickGalan
- 3. @RickGalan#SLCSEM @RickGalan#SLCSEM YAY REPORTING! A truly transferable skill between roles, jobs or even careers. Your job is not what you do. Your job is the results of what you do.
- 4. @RickGalan#SLCSEM @RickGalan#SLCSEM Building Valuable Reporting Reporting For You 1
- 5. @RickGalan#SLCSEM @RickGalan#SLCSEM REPORTING FOR YOU IS IMPORTANT!! ◉ Establish your own baselines and measures that you use to monitor performance of your programs ◉ Build regular and structured reporting to review and evaluate that performance Checklist Pattern Foundation
- 11. @RickGalan#SLCSEM @RickGalan#SLCSEM The data you collect here should serve as the basis for reporting for THEM. ASK YOURSELF… What actions do I need to take to make my programs successful? How can I measure how efficient those actions are? What do I need to monitor to make sure my programs are running to plan? Automate what you can, but stay close to the data. KPIs should include measures of both scale and efficiency A FEW THINGS TO KEEP IN MIND…
- 12. @RickGalan#SLCSEM @RickGalan#SLCSEM Building Valuable Reporting Reporting For Them 2
- 13. @RickGalan#SLCSEM @RickGalan#SLCSEM REPORTING IS 100% ABOUT RELATIONSHIPS
- 14. @RickGalan#SLCSEM @RickGalan#SLCSEM HEALTHY WORKING RELATIONSHIPS ARE BASED ON 3 KEYS Mutual Benefit ◉ You have to provide value to each other ◉ Both parties need to understand that value ◉ It can’t be completely one- sided Ownership ◉ The lines between you must be clearly drawn and understood ◉ There must be trust in the other parties’ ownership of their responsibilities Process ◉ The way you work together needs to be defined ◉ Systems put in place to ensure both parties are benefitting
- 15. @RickGalan#SLCSEM @RickGalan#SLCSEM THE RELATIONSHIP BETWEEN YOU AND YOUR BOSS IS UP TO YOU TO MAINTAIN
- 16. @RickGalan#SLCSEM @RickGalan#SLCSEM MUTUAL BENEFIT How you benefit should be fairly clear. A job, pay, increased resources, etc. It’s up to you to make sure you understand what success looks like, and align to it.
- 17. @RickGalan#SLCSEM @RickGalan#SLCSEM THE METRICS YOU THINK MATTER, DON’T.
- 18. @RickGalan#SLCSEM @RickGalan#SLCSEM TACTICS TO ALIGN WITH YOUR BOSS ◉ Discuss their goals ◉ Look for the KPIs that are already measured & discussed ◉ Talk to other direct reports and managers ◉ Get examples of existing reports ◉ Look for “sacred cows” ◉ Build a theory on what they want to see, and communicate it back!
- 20. @RickGalan#SLCSEM @RickGalan#SLCSEM OWNERSHIP & TRUST COMES FROM MAKING PROMISES… AND SEEING THEM THROUGH • Gather some history on the data you’ve discussed and confirmed as important • Set a meeting to discuss the program to date, in terms of this new understanding of the metrics • Start by showing them the position they are in by emphasizing what’s going well, and how you are going to build from there – using the metrics they care about • Make promises!
- 21. @RickGalan#SLCSEM @RickGalan#SLCSEM REPORTING IS THE PROCESS BY WHICH YOU DEMONSTRATE OWNERSHIP IN TERMS OF MUTUAL BENEFIT
- 22. @RickGalan#SLCSEM @RickGalan#SLCSEM Building Effective Reporting Designing the product 3
- 23. @RickGalan#SLCSEM @RickGalan#SLCSEM BUILD REPORTS LIKE YOU ARE BUILDING A PRODUCT
- 24. @RickGalan#SLCSEM JESSE JAMES GARRETT THE ELEMENTS OF USER EXPERIENCE http://jjg.net/elements
- 25. @RickGalan#SLCSEM • What is the purpose of the report? • Who is this for? What do they care about? • What do they want? What do you want? • How will the data be used? STRATEGY • What features or content does it need? • How often does it need to be pulled? • What data sources are necessary? • What can be automated? What can’t? SCOPE • How do all the data and features fit together? • How can the data be combined to get to the right metrics? • How can this be automated? STRUCTURE • What views of the data will we need? • What timelines, trends, graphs, images, etc.? • How do these views satisfy the needs of the user? SKELETON • How can we make it look awesome? • What actions can it drive? • What visualizations would help, and how? • What context do I need to provide? SURFACE
- 26. @RickGalan#SLCSEM @RickGalan#SLCSEM AN EFFECTIVE REPORT IS ONE THAT ACCOMPLISHES GOALS
- 27. @RickGalan#SLCSEM “ @RickGalan#SLCSEM You have to make every single detail perfect, and you have to limit the number of details... if you are paying attention to the smallest things, while knowing what's important, then everything else takes care of itself. JACK DORSEY TWITTER COFOUNDER FOUNDER & CEO OF SQUARE
- 28. @RickGalan#SLCSEM @RickGalan#SLCSEM START SMALL, ITERATE AND ADD FEATURES AS YOU GO
- 29. @RickGalan#SLCSEM @RickGalan#SLCSEM ALWAYS INCLUDE COMMENTARY AND CONTEXT Never send a report without analysis. Show your value. Be prepared to answer “why?” up to 5 times on any dataset you send.
- 30. @RickGalan#SLCSEM @RickGalan#SLCSEM Building Efficient Reporting Fast and light 4
- 31. @RickGalan#SLCSEM @RickGalan#SLCSEM YOU SHOULD BE SPENDING VERY LITTLE TIME RUNNING REPORTS
- 32. @RickGalan#SLCSEM @RickGalan#SLCSEM 2 PEOPLE, 6 HOURS OF WORK, GIANT FILES, ERRORS GALORE
- 33. @RickGalan#SLCSEM @RickGalan#SLCSEM 1 PERSON, 20 MINUTES OF WORK, FORMATTED FOR EXECUTIVES
- 34. @RickGalan#SLCSEM @RickGalan#SLCSEM A COUPLE OF QUICK TIPS
- 35. @RickGalan#SLCSEM @RickGalan#SLCSEM THERE IS A RIGHT WAY AND A WRONG WAY TO BUILD A PIVOT TABLE ◉ Data should be broad on the left, more refined on the right ◉ Columns should only include items you want to “pivot” off ◉ Segments + metrics should never be a column ◉ Never include calculated metrics in your data – only raw numbers ◉ Keep your metrics separated by data source, and in the order you pull them ◉ Create the Pivot using the whole column, not specific cells
- 36. @RickGalan#SLCSEM @RickGalan#SLCSEM A FEW PIVOT TABLE OPTIONS CAN DRASTICALLY EFFECT FILE SIZE AND PRESENTATION ◉ Check the box “For error values show:” - clean up all the #DIV/0 ◉ Turn off “Autofit column widths on update” – Annoying! ◉ Uncheck “Enable show details” – prevent accidental double-click issues ◉ Uncheck “Save source data with file” – it copies your whole data set into the file ◉ Check “Refresh data when opening file” – you’ll never know the difference
- 37. @RickGalan#SLCSEM @RickGalan#SLCSEM BUILD A REFERENCE TAB SO YOU NEVER HAVE TO MANUALLY FILL IN ANYTHING A special tab in the spreadsheet to keep data such as ◉ Week / Date mapping ◉ Category Groupings (that don’t exist in the data) ◉ Budget or goal reference data ◉ Anything you’ll need more than once!
- 38. @RickGalan#SLCSEM @RickGalan#SLCSEM A FORMULA TO MASTER THAT WILL CHANGE YOUR LIFE: GETPIVOTDATA =GETPIVOTDATA(“Data I want from the pivot table”, “Top Left Corner cell of the pivot table”, “Filter 1”, Value 1, “Filter 2”, Value 2) =GETPIVOTDATA(“Impressions”, ‘PIVOT’!$A$1, “Year”, 2015, “Month”, 6) NEVER CREATE PIVOTCHARTS AGAIN – THEY ARE THE WORST
- 39. @RickGalan#SLCSEM @RickGalan#SLCSEM SPEND THE TIME TO MAKE PRETTY AND USEFUL CHARTS Spend the time to get the chart right – it’s the primary UI for your user ◉ Chart Tools > Save as Template (top left corner) ◉ Now any chart you want to build can match exactly – just choose that template as your chart type
- 40. @RickGalan#SLCSEM @RickGalan#SLCSEM THANKS! Any questions? You can find me at @RickGalan / rickgalan@gmail.com
- 41. @RickGalan#SLCSEM @RickGalan#SLCSEM CREDITS ◉ Presentation template by SlidesCarnival ◉ Imagery and Icons from SlidesCarnival & Noun Project • thenounproject.com/ahasoft/ • thenounproject.com/megan.sheehan/ • thenounproject.com/DHETTEIX/ • thenounproject.com/creativestall/ • thenounproject.com/kolito/ • thenounproject.com/MisterPixel/
Editor's Notes
- Reporting is a topic I love! Why? Because people are generally terrible at it Because reporting is one of the truly transferable skills, from role to role, job to job, or even between careers Because your job is not what you do You weren't hired to do keyword research, write ad copy, run a/b tests, etc. Your job is the results of what you do So since your job is the results, you need to be able to demonstrate those results in a way that makes it clear how you and your work benefits the organization.
- There are two types of reporting – Reporting for you, and reporting for them. Let’s start with reporting for You.
- I have been shocked to find out in the past that some folk run their programs without building reporting to keep them on track - they do it ad hoc. I believe it is critically important to establish your own baselines and measures that YOU use to monitor performance of your programs, and to build regular and structured reporting to evaluate them. This self-inflicted reporting serves three primary purposes: A checklist of actions to take A pattern of data to watch A foundation upon which other reporting is built
- In "The Checklist Manifesto" by Atul Gawande, he tells a particular story that is relevant to our discussion: In 1935, the U.S. Army held a competition to build the next-generation bomber plane. Boeing entered its Model 299 During the first test flight, the plane roared down the runway, lifted off and climbed to 300 feet, stalled, turned on one wing and crashed to the ground in a fiery explosion killing two of its five crew.
- The army had planned to order sixty-five of the aircraft but went with a competitor because of this incident, which almost sent Boeing into bankruptcy. The crash investigation revealed nothing mechanical had gone wrong, and the crash was attributed to “pilot error.” The report also said the 299 was more complex than previous aircraft. The pilot was busy with all the various new things he had to do that he forgot something routine. The army did ultimately decide to purchase a few as test planes. A few pilots decided to get together and figure out a way to make the 299 more user-friendly. Pilot training wasn't the solution – the pilot who died had been the Air Corp’s chief of flight testing and pretty much their top guy.
- It turns out, flying this new plane was too complicated to be left to the memory of any one person, even an expert. Instead, this group came up with a simple approach – they created the first pilot’s checklist.
- It included all the stuff pilots already knew to do including checking the brakes were released, that instruments were set correctly, that doors and windows were closed, and so forth. Using these simple checklists, the pilots went on to fly the Model 299 - later known as the B17 Bomber, a total of 1.8 million miles without one incident. The use of a checklist took a catastrophe and turned it into one of the most successful military planes in history. Checklists have affected other industries in the same way – surgeries without checklists are twice as likely to produce casualties. You are not rocket surgeons, you just have to design, set up and manage a program with thousands to millions of moving parts in a constantly changing landscape. No biggy. Structured and routinely run reporting is a marketers checklist. This is how we make sure we're pressing all the buttons, and those buttons still work.
- The other purpose of Reporting for you is building a pattern of data I have a bachelor's degree in Neuroscience. While that has proven to be quite a useless degree, every now and then I get to apply some interesting nuggets to my current work. One of those interesting things that I've learned is that human brain is a pattern recognition machine. We're wired to find patterns in things. There is a reason that you see faces in the bark of tree and shapes in the clouds. Running your report "checklist" should produce data and trends, which your pattern engine will be able to turn into action and insight. The trick is to make sure your checklist is surfacing the right data...
- So ask yourself "What actions do I need to take in order to make my programs successful?“ "How can I measure the efficacy of those actions?“ "What do I need to monitor to make sure my programs are running to plan" We won't spend more time on this, other than to highlight a few things to keep in mind: I'm a huge fan of automation, but not when it gets you too far from the data. No one ever looks at dashboards. Putting things together yourself ensures that you are hitting the checklist and finding the patterns. You will have to develop your own KPIs. But keep in mind that any set of KPIs needs to have a measure of both Scale and Efficiency. There are no impressions without click-through-rate or CPM, no orders without AOS or ROAS, etc. Lastly, remember that any data you collect for yourself should end up being the source of data you pull for other folks. Let's not double-up work
- So we've discussed a little bit about the importance of reporting for you. Hopefully most of you are already doing that to some degree, and not letting whimsy and random flights of fancy drive your work. In order to talk about reporting for Them, let's talk about relationships.
- Your success or failure in any organization is largely determined by the way you and your department interacts with other departments. No marketer exists in a vacuum. In order to succeed, there are other groups that have to support you. Reporting up is 100% about relationships. With your boss, your clients, your coworkers, etc. No amount of data is worth anything if you haven't built the right kind of relationship.
- In my experience, healthy working relationships are based on 3 keys: Mutual Benefit - You have to provide value to each other. It can't be a one-sided. And both parties need to understand that value. Ownership - The lines between you on who is responsible for what must be clearly drawn and understood. Process - The system through which you work together needs to be defined. Think about the teams you interact with. Maybe software developers, customer service, sales, etc. I think you'll find that for the relationships that are working well, it's easy to identify these keys. For the relationships that aren't really working, I've always found it's because one of these are out of whack.
- But let's narrow our focus a bit to those relationships that matter the most - the one between you and the people you report to (Boss, client, executives, etc.) The trick with the relationship between you and the boss or client, is that these three keys are ENTIRELY up to you to maintain. You can't count on them doing it. That's why they HAVE you.
- How YOU benefit in your relationship should be pretty clear: They pay you, control your resources, not to mentioned provide employment, health insurance, etc. Ideally, you get direction and goals from them, although I've learned not to count on that. There is an implied benefit that you provide to them too - after all, they hired you to do a job. But it's up to you to make SURE you understand what success looks like for them, and align to it. You can, and should, try simply asking them. In my experience, there are three likely scenarios that will come from this: They know exactly what they want from you, and what it means They will tell you. And it will make sense. They will ask for reporting that sounds a lot like the reporting you built or would build for yourself. This should be pretty common for mid-level managers and directors like myself, because we're still pretty close to the action They have no idea what's important They will likely ask you what you think They might have a high-level goal (grow more!) but want you to tell them how you contribute You should fear for your job in this environment. They say one thing, but actually care about another (either because they don't understand it or aren't honest with themselves) This one could be masquerading as either of the other two. Unfortunately it may come down to trial and error to figure that out. In any case, regardless of your initial impressions, you have to make sure you understand what's important to your boss.
- Because the fact is, that even though you have read ALL this detail about what the right KPIs are, and what you should and shouldn't measure - none of that matters. What matters is what matters to them. At least to start.
- So what are some tactics to align with your boss? Look for those key numbers and segments that are discussed and praised. Do they only care about new customer signups? Is it all recurring revenue? Talk to other direct reports or managers that have worked with him or her. Get example reporting that already exists Look for "sacred cows" within the organization Find the buttons to push, and the wrong ones to avoid Build a hypothesis of what measures you should report on to show progress towards their goals.
- Now communicate that hypothesis. Before you build ANYTHING for your boss, you should strive to get 100% clear on what he or she wants to see. Make sure you understand what drives them. Make SURE. You need to be clear on how that data contributes to their decision-making.
- So we've discussed making sure we understand the mutual benefit we provide each other, now we're on to Ownership. Taking ownership in this relationship really means doing your job. Building plans and executing strategies. True ownership and trust comes from making promises, and seeing them through. Sadly that's something not a lot of marketers are willing to do. There is a lot of sandbagging and hedging. Gives us all a bad name. The allocation of resources (people, budgets, etc) is based on an expectation of return - so set those expectations first so the decision is easy. Gather some of that initial data that you believe and have confirmed is important, and meet with your boss to get on the same page on where the programs have been. Don't start by telling them everything they are doing wrong - no one wants to hear their baby is ugly. Start by showing them in the position they are in by emphasizing what's going well, then describe how you are going to build from there - using those measures you believe matter to them. Make promises that align to those priorities Early in my tenure at my current job, I gave a presentation on where we were and where things were going, and went through this process. I had 17 slides on how we're moving the right direction After those 17 slides, I had 12 more that showed them what we were going to do now to move forward, and how that would affect the numbers. Some of my metrics were right, some were wrong and we adjusted. We're still adjusting. But I have clearly demonstrated my ownership of the programs I've been entrusted with, and showed what benefit I could provide. In turn, I have been able to hire 4 people, have two more open jobs, spin up programs and move forward with relative autonomy to accomplish the promises I made.
- Lastly, the process Reporting IS the process by which you demonstrate your ownership in terms of you and your boss's mutual benefit.
- Now that we've discussed how to make sure the data you are presenting is valuable, let's talk about what it takes to build a fantastic report
- I've learned that building the most effective and efficient reporting comes from treating your reports like you are building a product. I mean that literally, like a website or app.
- This is a diagram made by Jesse James Garrett, from “the Elements of User Experience.” Any product people will tell you this is one of the quintessential works on effective product and website design. The image is obviously part of a much larger body of work on building web products, but I like to use it to illustrate that reports, like websites, should be approached in layers. Garrett's model starts at the bottom with the very abstract, and through strategy, scope, structure, skeleton and surface gets more and more refined and concrete
- For reporting, we can ask slightly different questions, but the process of refinement is the same: Strategy What is the purpose of the report? Make sure you know - never build reports just to build them I've had a lot of luck with inherited reports that I've just stopped sending Who is this report for? Who is the audience? You or them? What do they care about? How will the report or data be used? That's tricky. I've found that in most cases the only one who will "use" the data is you To get more resources, show progress, build alignment, etc. Scope What features or content does it need to include? What are the KPIs we've defined? What are the actionable data points that are going to show benefit for the audience? How often does it need to be refreshed? Is this a weekly report? Monthly? What are the data sources necessary? What can be automated? What must be manually pulled? What kind of granular data do you need? Weekly? Daily? At what level should the data be presented? Campaign? Partner? Program? Structure How will it fit together How can the data be combined to get to the right metrics? How can we combine the data moving forward? Skeleton What views of the data will we need? How do these views satisfy the needs of the audience? Surface How can we make it look awesome? What actions can it drive? What visualizations would help?
- An effective report though, is one that accomplishes goals. The process of vetting all the strategy and scope for your report should enable you to define and refine those goals, and make sure that you are hitting them If the report is for you - those goals should be the checklist and pattern you need to monitor your programs If the report is for them - those goals should primarily be the demonstrating of your ownership of your program, and the benefit it has on the organization.
- Your final product should be more simple than you are likely comfortable with. Jack Dorsey, one of the Cofounders of Twitter and the founder of Square, said in an address given at Stanford in February 2011, speaking about building products "You have to make every single detail perfect, and you have to limit the number of details... if you are paying attention to the smallest things, while knowing what's important, then everything else takes care of itself.“ Focus on making sure that the data you provide is precise, understandable, and valuable to the recipient.
- Your reporting can take a terrible amount of time if you let it - so don't Start small, keep it focused on just those things that are important to THEM at first. Then add features as you go. Every report should slowly get better and more refined. Begin with a minimum viable product - just like in product design. That may mean you can't get to all the numbers you feel matter at once. Sometimes that means you will have to manually instead of automating them. Iterate!
- You should always provide context and commentary for your data. You should never send numbers without analysis. You are paid for your interpretation, not to regurgitate numbers. For any change in the data you send above, you should be able to answer "why" about 5 times to get to the crux of the issue. It's about demonstrating your ownership of these programs, and ensuring you are measuring the right things. I learned this the hard way when I had a boss who EVERY WEEK would take apart the reporting I sent. It would require hours of follow up after the report, until I learned to anticipate the questions based on what was clearly important to them.
- You should be spending VERY LITTLE of your time pulling numbers together or updating dashboards. That process should be, and absolutely can be, done in minutes. The initial creation of your report will take some time - but that's time you are spending building relationships with your managers, understanding your business, and getting to know your program.
- This is an example of a report I inherited at a previous company It took 2 people 6 hours each of manual work to gather all the numbers, and input them into the report It was extremely error prone, and couldn't even be emailed because the file got so big. The executives that received it looked at about 3 numbers on it.
- After going through this process of building a report product, and iterating a bit for a few weeks, we ended up with a report that looked like this. It took 1 person about 20 minutes to pull, and had significantly more data than the previous report. It was formatted in a way that the executives could easily find those numbers that were important to them. It had automatically generated trendlines and WoW comparison numbers.