Recommended
More Related Content
Similar to Project Deliverable 2AB Test Plan.docx
Similar to Project Deliverable 2AB Test Plan.docx (20)
More from briancrawford30935
More from briancrawford30935 (20)
Recently uploaded
Recently uploaded (20)
Project Deliverable 2AB Test Plan.docx
- 1. Project Deliverable 2: A/B Test Plan What platform are you using for the test? We are using Google Analytics as our platform for the experiment. We will be using Squarespace as our User Interface with designing the two different front pages. Squarespace offers simple control for web design, and has the capability of hiding webpages if needed.
- 2. What treatments will you test? (You may still be working with your client to figure that out, but give me one idea.) Since the main goal of the website is to guide or convince the consumers to request a quote from us by hitting the button, “request a quote”, we and the company sought best to display two versions of the front page. The original front page will have a button on the top right that says, “Request a quote”. Once the button is pressed, it will guide the users to the page where they can request the specific chiller that their company requires. The second front page will have the same purpose; however, the visual display will be much more direct. The button will have an arrow pointing at the three words: “Request a quote”, and instead of having a visual slideshow, there will be a bland front cover with the company’s logo. How will you assign treatments to customers? The Google Analytics Interface offers an experiment mode that enables us to test almost any change or variation to the website to measure its impact. Within the configuration of the experiment, we will have the original page as Variation 1, and the direct-approach page as Variation 2. We then choose to split the experiment 50/50 for all users who approach our website. Google Analytics will automatically help us divide the population randomly for testing. What response measures will you use to evaluate your treatments? We will evaluate the experiment based on the number of page views on average over the time of 3 weeks. However, this is the tricky part. Since the goal of the website is to convince the user to click on the “request a quote” button to negotiate for a chiller, we cannot measure the page views of the two front pages. Instead we will design two “request a quote” pages. Afterwards we can measure how many page views the two pages will receive; therefore, finding out which front page is more effective. How will this test help inform future decisions that the client will make?
- 3. The client’s main goal is to see how digital marketing can help drive their sales up for the company. Drake Refrigeration has always been communicating with wholesalers for clients, and thus decided to go for a more direct approach. This test will help them know which front page can drive more potential clients. The more potential clients they have, the more actual clients they will have, it’s a simple correlation. Our experiment will not only introduce them to the power of digital marketing, but will also show them the most effective way to execute. Will you have enough data to make a conclusion? Make sure you make fake data and analyze it to determine if your confidence interval will detect a difference between the two groups. According to our previous analytics over the past few months, it is clear that our website takes in an average page view of 5000 visits a month. This amount of visits, is a good indication that we will have enough data to make a conclusion. We performed a mock test with fake data by using past 19 days from our website and generating another set of data using random number generator between the minimum integer and maximum integer of the real data. Here is the data: Day Visit to Targeted Page-A Visitor(A) P(A) Visit to Target Page-B Visitor(B) P(B) 1 73 212 0.34434 29
- 7. 12 36 0.333333 29 95 0.305263 18 50 186 0.268817 43 21 2.047619 19 42 110 0.381818 12 85 0.141176 Targeted Page A= Request a Quote Page A Targeted Page B= Request a Quote Page B P(A)= P(B)= Based on these two data sets, the average of the average of P(A) is 0.318194 and P(B) has an average of 0.511177. Under these conditions, we can compute a confidence interval for the difference between the percentage of successes in group A and group B giving us the confidence interval of {0.10646014, -0.492426}. This means there is no significant difference between the two groups.
- 8. Project Deliverable 1 Client Brief Before the meeting: B1. What business is the company in? How do they make money? If they are a start-up, are they selling any product yet? Drake Refrigeration is company that sells refrigerators, mostly to other businesses. B2. Who is their target audience? Drake Refrigeration’s target audience is mostly businesses with refrigeration needs, such as, breweries, dairy farms, and hospitals. B3. What types of digital marketing are they already doing? Do they already have some owned media? Do they have some earned media? Drake Refrigeration only uses owned media. They have a website and an email list of their regular customers. They also have a Facebook and Twitter that both have very little activity or presence. B4. Do you have any initial ideas for A/B testing that might help the company? We are thinking of changing the appearance or functionality of a feature on Drake Refrigeration’s website with a goal of
- 9. increasing the amount of requests for quotes the website gets. During the Interview: Martin Squicciarini, Engineer Manager Linda Quirk, Marketing Manager C1. Who is your target audience? Or if you already have some idea of this, ask the client to confirm, e.g., I imagine that your client is primarily caterers who want to use the product at events. Is that right? For the longest time, our company has focused on contacting with contractors over the nation, one of our biggest contractors is United Refrigeration. They provide most of our customers and we communicate mostly through them. However, after hiring Edward, our company took a new direction in marketing, we wanted to contact our customers directly, thus we narrowed our audience down to 8 applications: Breweries, Dairy Farms, Food Processors, Low-Temp Facilities, Industrial Plantations, Hospitals, and Refrigeration. C2. What types of marketing are you currently doing? Make sure to ask about specific marketing channels that they might be forgetting (website, email, social, etc., based on your research). The best channels for A/B testing are owned media channels, especially email and websites, so make sure to focus on those. For each channel, try to get a sense of how many users/viewers they have, e.g., How many visits does your website get in a month? or How many emails are on your list? Since our company is mostly B2B, there wasn’t much need for social medias or digital marketing. We have over 800 email contacts that are our regular customers. However, our competitors, GD Chillers, began to develop a digital marketing campaign, thus we adapted. We created a website that leads the users to request a quote for our website. The website gets an
- 10. average of 5,767 visits per month. C3. What do you see as your biggest marketing challenges and opportunities? Listen carefully to the response. Great project ideas come out of this question. Our biggest challenge is to get to the decision makers. Our products are a huge investment for our customers, so mostly it is the CEO or “The Big Guys” who make the decision on whether they should purchase our chillers. Our biggest opportunities is when we get the chance to meet face to face with our customers and show them what our chillers do. C4. Do you have any question you've always want to know about your customers? Our number 1 question would be why they chose other chiller companies except for ours, did we lose on price, size, or feature? C5. As you know, we have to do an A/B test for our class. Do you have any specific ideas for A/B testing? Is there a specific channel you'd like us to work with? Are there alternative marketing messages that you are interested in comparing? Drake concluded that they will provide two versions of its newly updated website. The first version would consist of a new look of the new website, with the home page displaying the eight applications through a big slideshow header, on the top right will have the button, “Request a Quote”. The second version would consist a different set of cover photos, and will have the button on the top right that says, “Request for Info”. After the meeting: A1. How the company makes (or plans to make) money. For startups, summarize where they are in their development. Drake Refrigeration Inc. makes money through selling high quality, dependable refrigeration and cooling equipment product
- 11. line, including cooling and conditioning equipment used in commercial, industrial & residential, medical & pharmaceutical, food & beverage industry. The price for products is various depending the contracts because it is a B2B company. Overall, current estimates show this company has annual revenue of $50 to 100 million. A2. Target audience(s) for the company. The target audience refers to the specific market that Drake Refrigeration wants to dominate. According to the company interview, Drake refrigeration has been working with contractors who in turn talk to their consumers. However, the company has decided to focus on establishing a connection with consumers. The target audience of the company includes industrial plantations, breweries, food processors, hospitals, low temp facilities and refrigeration. The products from Drake Refrigeration are meant to cater to the needs of industrial organizations and businesses. Drake Refrigeration has extensive experience in the engineering of refrigeration and cooling solutions for a wide range of consumers. The company has worked with breweries, food processors, low temp facilities, and medical institutions for many years. The increased communication between the company and its consumers is an opportunity to focus on meeting the specific needs of each group of consumers. The exact definition of the company’s target audience is a good for the company. The definition of the target audience means the company models advertisements that communicate to their target market. Also, Drake Refrigeration can focus on conducting research on how to improve the solutions it offers to the targeted consumers. A3. Summary of current marketing. Make sure to include information about how many users they have on each marketing channel that you might use for your A/B test. Drake Refrigeration Inc. focused more on building its brand
- 12. website that allows customers and contractors to request a quote rather than using social media like others do because it is a B2B company. Also, they heavily rely on emails to keep in touch with its partners. As the Engineer Manager Marty said, there are around 5,767 visits per month in average. Although the company does not spend a lot of time and money in Marketing, the partners would put more efforts into digital marketing for promoting the products. A3. Three preliminary ideas for A/B tests. Please be sure to describe what treatments you plan to use (mock-ups are great!) and what response measure you plan to use to compare the treatments. The aim of digital marketing is to improve the effectiveness of a company’s website. Website improvement improves the online presence of the company by ensuring that consumers can get the information needed without having to navigate for too long. A/B testing describes a randomized testing process that has two variants, which are the variation and control in the experiment. Based on its current marketing, Drake Refrigeration is not heavily operating their marketing strategy digitally. Because their business is mostly B2B, the sales department work closely with industrial contractors who connect Drake Refrigeration with their clients. For example, a brewery owner, who is interested in purchasing an industrial chiller for his 15-barrel microbrewery, will contact a contractor, such as United Refrigeration. United Refrigeration will then connect Drake Refrigeration with the brewery and request specifications. Since the company has been doing well with this strategy for quite some time, Drake has not considered implementing a new digital marketing strategy. After discussing what A/B testing is and the exact direction Drake Refrigeration wants to do for its digital marketing, we have come up with three campaigns that can help secure a definitive A/B test: a variation of an email blast, two different logos, or two versions of their new website. With our help,
- 13. Drake concluded that they will provide two versions of its newly updated website. The first version would consist of a new look of the new website, with the home page displaying the eight applications through a big slideshow header, on the top right will have the button, “Request a Quote”. The second version would consist a different set of cover photos, and will have the button on the top right that says, “Request for Info”. To perform this test, we decided to use Google Analytics as our experiment tool. In Google Analytics, there is an option called experiments, which will direct us to a setup for an A/B test. The objective of this test is to see which home page generates better CTR, CPM, visits per day, views per day, and average audience size. After the setup is complete, we set the percentage of traffic to 100%, which allows all users to be part of the experiment. Google Analytics will then segment the traffic 50/50, and will randomly choose which versions to give to which users. The test will last for 3 weeks, gathering a predicting size of about 4000 audiences, depending how long we are allowed from the class. Afterwards, we generate the data and conclude which version had better results. Project Deliverable 1 Client Brief Before the meeting: MKTG 367: Data-Driven Digital Marketing Term Project Grading Rubric
- 14. Client __________________________________ Students: _____________________________________________________ _______________________ _____________________________________________________ _______________________ Background and Learning Objectives Report provides appropriate background about the client organiz ation and their marketing strategy 5 pts Report provides a clear statement of what you hoped to learn wi th the test and how that will help the client 5 pts Assessment of test strategy: Was this an appropriate platform to test on? How does the test fit into the client’s marketing strategy? Will the te st results be useful to the client? 5 pts A/B Test Design
- 15. Report provides description/screenshots/photos of treatments 5 pts Report describes planned KPIs (response variables) and how the y relate to the business objectives. When KPIs are not perfectly aligned wi th business goals, due to limitations in measurement, these mis-alignments are noted. 5 pts Report summarizes any design challenges, e.g., potential confou nds that might make it difficult to interpret the test 5 pts Assessment of test design: Will the test answer the learning obj ectives that were set out? How does the test KPI relate to the learning objec tives? 5 pts A/B Test Execution Client and timeline were managed effectively 5 pts Test groups were properly randomized and the treatments were applied as planned 5 pts KPIs were collected 5 pts
- 16. (continued on reverse) Results & Conclusions Report provides an attractive and clear visualization of the data 5 pts Analysis uses statistical tests / statement of confidence to deter mine if there is sufficient data to draw conclusions. Non-significant results are not reported. 5 pts Report gives a short statement of conclusions and recommendati ons for the business. Conclusions are supported by the data. 10 pts Report provides appropriate caveats. What are the limitations/as sumptions of your test? 5 pts Report re-assess the sample size to determine how much data w ould be needed to detect a meaningful difference. 5 pts
- 17. Lessons Learned s Report provided several tips, information about advertising plat forms, measurement, etc. that would be useful to your classmates. 20 pts Bonus Bonus for presenting earlier 2 pts Bonus for using experimental design methods or analysis techni ques that were outside the class material. 0 - 5 pts TOTAL MKTG 367: Data-Driven Digital Marketing Term Project Peer Review Client Name_____________________________ In general, I will evaluate the project as a whole and apply that grade to all the team members. However, I understand that there are occasionally situations wh ere a group member was clearly unhelpful or just flakes out. If I get a clear signal of this from t he entire team, I will adjust that
- 18. team member’s grade. Similarly, I do occasionally adjust a grad e up when I get a clear signal that someone on the team was a stand-out leader. In the table below, please list the members of your group, inclu ding yourself and describe briefly how each team member contributed to the project. Group Member Name (first names are fine) Primary Contribution (design, analysis, report, etc.) Your Evaluation (yourself) ☐below standard ☐good ☐outstanding ☐below standard ☐good ☐outstanding ☐below standard ☐good ☐outstanding ☐below standard
- 19. ☐good ☐outstanding ☐below standard ☐good ☐outstanding Below standard = Contribution was in some way clearly below e xpectations. This person either did not contribute (e.g. skipped meetings, didn’t answer emails) or actively got in the way of progress. Good = Contributed in a satisfactory way. This person found a way to contribute his/her talents to the project. Outstanding = A leader for this project. We couldn’t have done it without him/her. For any below standard or outstanding ratings, please explain w hat happened: _____________________________________________________ ________________ _____________________________________________________ ________________ Powerpoint instruction 1. My part is the first part: Background and Learning objectives(first part of rubric), make power point(2-3 slides) of this part and follow each points in rubric.(15 points)Please also write the notes for each slide, so I can read it when I am presenting... Thank you so much!
- 20. 2. www.drakechillers.com Here is the company website, you could find some information in there but not copy on the PPT. 3. I uploaded the our group’s interview of this company’s manager and some a/b test information for u, maybe you will use them. 4. You need to put these following points on slides: - the problem for the company is they don’t have digital marketing strategy, and our group create website for them, we wanted to see if the aesthetic, function,and usage of the website was helping the marketing of the website. - Yes, it is an appropriate platform to test on. Because as the Internet becomes more tangible with business, it is time for drake to evolve. The test helps drake understand what their consumer want from the website: information and lively experience or straightforward business to business.