Interactive Marketing
Upcoming SlideShare
Loading in...5
×
 

Interactive Marketing

on

  • 1,990 views

 

Statistics

Views

Total Views
1,990
Views on SlideShare
1,989
Embed Views
1

Actions

Likes
1
Downloads
30
Comments
0

1 Embed 1

http://technofella.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Interactive Marketing Interactive Marketing Document Transcript

  • THE IMPACT OF EXPERIMENTATION ON PRODUCT DEVELOPMENT IN COMPANIES INVOLVED IN INTERACTIVE MARKETING by Sergey L. Sundukovskiy H. PERRIN GARSOMBKE, Ph.D., Faculty Mentor and Chair LARRY KLEIN, Ph.D., Committee Member JOSE NIEVES, Ph.D., Committee Member William A. Reed, Ph.D., Acting Dean, School of Business & Technology A Dissertation Presented in Partial Fulfillment Of the Requirements for the Degree Doctor of Philosophy Capella University August, 2009
  • UMI Number: 3369490 Copyright 2009 by Sundukovskiy, Sergey L. All rights reserved INFORMATION TO USERS The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleed-through, substandard margins, and improper alignment can adversely affect reproduction. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. ______________________________________________________________ UMI Microform 3369490 Copyright 2009 by ProQuest LLC All rights reserved. This microform edition is protected against unauthorized copying under Title 17, United States Code. _______________________________________________________________ ProQuest LLC 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, MI 48106-1346
  • © Sergey L. Sundukovskiy, 2009 View slide
  • Abstract Since its emergence, the Internet and Internet related technologies have permeated almost all aspects of the modern life. The impact of the Internet on the day-to-day activities of its users has been quite dramatic. However, its effect on the business community has been even more profound. Besides yielding additional opportunities for existing businesses, the Internet has facilitated a new era of companies that have the Internet at the center of their business model, businesses that simply would not be able to exist without it. As such Internet-centric and increasingly traditional businesses now rely on interactive marketing as a source of revenue, differentiation and competitive advantage. Despite the obvious importance to many businesses, the efficiency and improvement of interactive marketing has largely stagnated or at least proceeds at a low pace. This study examined interactive marketing through the prism of experimentation as a way of propelling interactive marketing forward and enabling it to keep pace with technological advances. Experimentation lies at the core of product development, improvement and innovation. Active experimentation has been utilized as a product strategy in numerous business fields, but it has been largely ignored in interactive marketing. This study examined a number of experimentation models and their applicability to interactive marketing. In addition it focused on the elements of interactive marketing that are conducive to experimentation. View slide
  • Dedication I dedicate this work to my wife Galina for her commitment, dedication and unconditional love. Also, to my children: Aaron, Rebekah and Miriam for giving meaning to my life. Your love and support gave me strength to see this project through. To my parents: Valentina and Leonid, for always believing in me and forever encouraging me to see physical, emotional and intellectual limits as imaginary lines that are meant to be crossed. Last, but by no means least, to my brother: Aleksey, for seeing me the way I wish I could truly be. iii
  • Acknowledgements Special thanks to my friend Boris Droutman for introducing me to the subject of experimentation and helping me generate the ideas that made this dissertation possible. Thanks for encouraging me, listening to the things I say and hearing the things I do not say. iv
  • Table of Contents Dedication ...................................................................................................................................... iii Acknowledgements ........................................................................................................................ iv Table of Contents ............................................................................................................................ v List of Figures ............................................................................................................................... vii CHAPTER 1: INTRODUCTION ................................................................................................... 2 Introduction to the Problem......................................................................................................... 2 Internet Marketing ....................................................................................................................... 2 Interactive Marketing .................................................................................................................. 4 Product Development .................................................................................................................. 9 Background of the Study ........................................................................................................... 10 Purpose of the Study ................................................................................................................. 11 Research Questions ................................................................................................................... 12 Definition of Terms ................................................................................................................... 15 Assumptions and Limitations .................................................................................................... 17 Organization of the Remainder of the Study............................................................................. 19 CHAPTER 2: LITERATURE REVIEW ...................................................................................... 20 Experimentation Introduction ................................................................................................... 20 Experimentation Strategies ....................................................................................................... 21 Experimentation Models ........................................................................................................... 22 Interactive Marketing ................................................................................................................ 27 Experimentation ........................................................................................................................ 29 Product Innovation .................................................................................................................... 31 CHAPTER 3: METHODOLOGY ................................................................................................ 33 Description of the Methodology ............................................................................................... 33 Design of the Study ................................................................................................................... 36 Population and Sampling .......................................................................................................... 37 Measurement Strategy ............................................................................................................... 38 Instrumentation.......................................................................................................................... 39 Data Collection .......................................................................................................................... 40 Data Analysis Procedures.......................................................................................................... 41 v
  • Qualitative Data Display for Describing the Phenomenon ....................................................... 44 Qualitative Data Display for Explaining the Phenomenon ....................................................... 45 CHAPTER 4: DATA COLLECTION AND ANALYSIS............................................................ 46 Overall Response Analysis ....................................................................................................... 46 Research Objective One ............................................................................................................ 48 Research Objective Two ........................................................................................................... 53 Research Objective Three ......................................................................................................... 58 Research Objective Four ........................................................................................................... 61 Research Objective Five............................................................................................................ 63 Research Objective Six ............................................................................................................. 66 Validity and Bias ....................................................................................................................... 69 CHAPTER 5: RESULTS, CONCLUSIONS, AND RECOMMENDATIONS ........................... 71 Results ....................................................................................................................................... 71 Conclusions ............................................................................................................................... 74 Recommendations ..................................................................................................................... 75 Limitations of the Study ............................................................................................................ 76 REFERENCES ............................................................................................................................. 78 APPENDIX A. SURVEY TEMPLATE ....................................................................................... 82 APPENDIX B. CONCEPTUAL FRAMEWORK........................................................................ 88 APPENDIX C. INTERACTIVE MARKETING .......................................................................... 89 APPENDIX D. MARKETING ..................................................................................................... 90 APPENDIX E. EXPERIMENTATION CYCLE ......................................................................... 91 APPENDIX F. DATA ANALYSIS .............................................................................................. 92 vi
  • List of Figures Figure F1. Response summary...................................................................................................... 92 Figure F2. Job function breakdown .............................................................................................. 92 Figure F3. Job function and experimentation experience cross-tabulation .................................. 93 Figure F4. Experimentation impact breakdown............................................................................ 93 Figure F5. Experimentation impact and department cross-tabulation .......................................... 94 Figure F6. Experimentation impact and experimentation experience cross-tabulation ................ 94 Figure F7. Experimentation levers impact breakdown ................................................................. 95 Figure F8. Experimentation levers impact and experimentation experience cross-tabulation ..... 95 Figure F9. Experimentation levers impact and job function cross-tabulation .............................. 96 Figure F10. Experimentation conditions impact breakdown ........................................................ 96 Figure F11. Experimentation conditions impact and experimentation experience cross-tabulation ....................................................................................................................................................... 97 Figure F12. Experimentation conditions impact and job function cross-tabulation ..................... 97 Figure F13. Experimentation CPA impact and experimentation overall impact cross-tabulation 98 Figure F14. Experimentation CPI impact and experimentation overall impact cross-tabulation . 98 Figure F15. Experimentation CPA impact and experimentation overall impact cross-tabulation 99 Figure F16. Experimentation upsell impact and experimentation overall impact cross-tabulation ....................................................................................................................................................... 99 Figure F17. Experimentation CPI impact and experimentation overall impact cross-tabulation100 Figure F18. Experimentation product lifecycle impact breakdown............................................ 100 Figure F19. Experimentation impact on product lifecycle and job function cross-tabulation.... 101 Figure F20. Experimentation impact on product lifecycle and experimentation experience cross- tabulation..................................................................................................................................... 101 Figure F21. Experimentation product innovation impact breakdown ........................................ 102 Figure F22. Experimentation product innovation impact and experimentation experience cross- tabulation..................................................................................................................................... 102 Figure F23. Experimentation product innovation impact and job function cross-tabulation ..... 103 Figure F24. Experimentation product improvement impact breakdown .................................... 103 Figure F25. Experimentation product improvement impact and job function cross-tabulation . 104 Figure F26. Experimentation product improvement impact and experimentation experience cross-tabulation ........................................................................................................................... 104 Figure F27. Experimentation product deployment risk impact breakdown ............................... 105 vii
  • Figure F28. Experimentation product deployment risk impact and experimentation experience cross-tabulation ........................................................................................................................... 105 Figure F29. Experimentation product deployment risk and job function breakdown ................ 106 Figure F30. Experimentation competitive standing of the company impact breakdown ........... 106 Figure F31. Experimentation competitive standing of the company impact and job function cross-tabulation ........................................................................................................................... 107 Figure F32. Experimentation competitive standing of the company impact and experimentation experience cross-tabulation ......................................................................................................... 107 Figure F33. Experimentation revenue impact ............................................................................. 108 Figure F34. Experimentation profit impact ................................................................................ 108 Figure F35. Experimentation market size impact ....................................................................... 108 Figure F36. Experimentation market penetration impact ........................................................... 109 Figure F37. Experimentation marketing goals impact horizontalization.................................... 110 Figure F38. Experimentation marketing levers impact horizontalization .................................. 111 Figure F39. Experimentation conditions impact horizontalization............................................. 112 Figure F40. Experimentation product lifecycle impact horizontalization .................................. 113 Figure F41. Experimentation product innovation impact horizontalization ............................... 114 Figure F42. Experimentation product improvement impact horizontalization........................... 115 Figure F43. Experimentation product deployment risk impact horizontalization ...................... 116 Figure F44. Experimentation competitive standing impact horizontalization ............................ 117 viii
  • ix
  • CHAPTER 1: INTRODUCTION Introduction to the Problem Internet Marketing The introduction of the World Wide Web, enabled by the Internet, saw an explosive user growth (Newman, 2001). Consequently both online based and traditional companies with significant Internet presence started to realize the full potential of World Wide Web as a marketing medium. Since the popularization of the Internet and Internet related technologies, several new marketing paradigms have emerged that take advantage of Internet. Such emergence was not planned and occurred spontaneously as an offshoot of traditional marketing (Mark, 2003). Internet marketing was formulated as one form of non-traditional marketing. Due to early disappointments and lack of experience with a new non-traditional marketing medium Internet marketing was initially used to supplement traditional marketing efforts. According to eMarketer Research (2007), US online marketing in 2002 shrunk by 15.8%. Companies did not see Internet marketing as a separate marketing channel, with a unique set of characteristics and considered it the same as radio and television (Raman, 1996). Internet marketing was combined with traditional television and radio campaigns because at first, it was not considered capable of carrying the full weight of responsibilities. However, over time marketers began to realize that traditional marketing campaigns consisting of television and radio advertisements, and print media designed to attract users to their websites had a limited success, whereas the Internet itself proved quite promising in that regard. According to eMarketer Research (2007) online advertizing spending in 2007 had 2
  • reached $19.5 billion, projecting to reach $36.5 billion in 2011. Companies had realized that Internet marketing was not only a strong and independent marketing medium, but also had a number of advantages over traditional forms of marketing, such as television, radio and print media (Mark, 2003). In order to earn its independence and be considered comparable to one of the traditional dominant marketing channels, Internet marketing had to be evaluated against a multitude of traditional marketing problems. It had successfully answered a number of questions that were historically asked by marketing science. Internet marketing showed its potential for consumer accessibility, defining corporate identity, promoting brand awareness, enabling market segmentation and achieving market localization and internationalization (Jebakumari, 2002). In addition, Internet marketing capabilities such as personalization, interactivity and traceability were formulated among its advantages over traditional dominant medium channels such as television, radio and print media. The Internet allowed synchronous and asynchronous user behavior analysis, which in turn allowed for previously unavailable personalization capabilities (Milley, 2000). It had the potential to reach a vast number of World Wide Web users that otherwise would be either difficult to reach, or in some cases completely unreachable. According to Internet World Stats (2007), by the end of 2007 the Internet had reached 1,319,872,109 users with the largest percentage growth accounted for by countries in Middle East, Africa and Latin America. Since its inception, the Internet has developed into one of the strongest and mature Marketing channels that is segmented into several sub-channels. At the present time Internet marketing could be considered as a conglomerate marketing methodology, providing a common umbrella for a set of diverse marketing sub-channels. The following distinct Internet sub- marketing channels have now emerged: (a) search engine marketing (SEM); (b) e-mail 3
  • marketing; (c) affiliate marketing; (d) display marketing; and (e) social media marketing (See Appendix C). These Internet marketing approaches, founded upon sociological, economic and cultural marketing aspects, are now interwoven into Internet marketing fabric as a cohesive entity. At the same time each one of these Internet marketing sub-channels has a set of unique characteristics, based on its execution and marketing strategies. These characteristics distinguish it from other tangential marketing approaches (Mark, 2003). Interactive Marketing In addition to being part of Internet marketing, social media marketing, affiliate marketing, search engine marketing and e-mail marketing sub-channels have another aspect in common, their interactivity. Each one of these sub-channels relies on the user to make an interactive step directly related to the advertisement stimuli. As such these marketing sub- channels could be placed under another umbrella called “Interactive Marketing”. In its purest form Deighton (1996) defines “Interactive Marketing” as an ability of the computer based system to interact with the user for marketing based purposes. In many cases however interactive marketing consists of a multi-transactional interaction that is based on user purchase and visit history, declared or implied preferences and demographic information consisting of age, sex, income level and even race. Most of the interactive marketing accomplished through the World Wide Web is enabled by the Internet. However, it would be incorrect to consider Internet marketing as synonymous with interactive marketing (Deighton, 1996). Based on the Veen diagram (see Appendix D) not all Internet marketing is interactive and not all interactive marketing is conducted on the World Wide Web or is Internet based. For instance display marketing, one of the Internet marketing sub-channels, is not interactive in nature. In the same vain iPods, mobile devices, gaming 4
  • devices, interactive screen devices and e-books are capable of carrying a marketing message by means of wireless or hard-connected communication. Each of these devices has its own marketing specifics related to its size and interactivity, tracing power, display and communication capabilities. However, based on the Veen diagram (see Appendix D) it is evident that a great amount of overlap exists between the Internet and interactive marketing. Chulho (1998) defined the “Interactive Age” as an age of full interactivity between humans and machines and technology connected humans. Similar parallels can be drawn between the “Information Age” and “Participation Age”, coined by Schwartz (2005). Blogging and Social Networking are two recent examples of the Interaction or Participation Age that are utilized by social media marketing. At the heart of interactive marketing are the two main concepts of “Personalization” and “Customer Engagement”. The interactive marketing itself could be fully defined in terms of these concepts. More specifically, this researcher proposes an alternative definition of “Interactive Marketing” describing it as a process of customer engagement through personalization. In turn, Steinberg (2006) defines “Customer Engagement” as a process of engaging potential or existing customers in inter-client or inter-customer communication by means of the push or pull marketing model. According to Haag (2006) “Personalization” is a process of generating unique experience for an ever finer grain segment of customers. Taken to the logical extreme, the process of personalization is a process of targeting a single individual with the stimuli uniquely tailored to that individual and that individual alone. The logical conclusion of these definitions is the hypothesis that the most effective customer engagement could be achieved through granular personalization activities. The companies involved in interactive Internet marketing endeavor to achieve 5
  • personalization through data mining and information analysis. The data mining and analysis constitutes the bulk of the personalization effort in both temporal and relative importance perspectives. Ozdemiz (2002) describes personalization as an effort consisting of three major steps: (a) data gathering; (b) data analysis; (c) personalized content generation. According to comSource (2007) the data gathering portion of the personalization process undertaken by major interactive marketing companies such as Yahoo, Google and Facebook has reached a significant, earlier unseen level. According to comSource (2007), in a period of a single month in 2007 Yahoo on average collected data 2,520 times for every visitor to its site. Mark (2003) posited that Internet marketing companies collect as much data a possible even without a specific goal in mind. Since, it is not always clear what customer data would become useful in the future, all available data is collected (Mark, 2003). It usually consists of webpage visits, visit history, purchasing history, abandonment and completion history, self-declared or inferred preferences, computer information, web browser information, self-declared or inferred demographic information self-declared or inferred financial information, geographic information, age and sex information (Mark, 2003). The collected data is carefully cataloged and analyzed. The purpose of this data analysis is to determine the most appropriate marketing strategy for a particular user and other users that fit similar characteristics. Milley (2000) describes this approach as a “Data Empowered” marketing strategy. The intention of any marketing strategy in general and interactive Internet marketing in particular is to achieve a desired goal. In general interactive Internet marketing tries to achieve a goal of conversion where the user is synchronously driven to a purchasing decision. According to Macias (2000) even though conversion is certainly the dominant interactive marketing goal, other goals may include customer retention, customer acquisition, lifetime value maximization 6
  • and product up-sell. Each one of these goals could be considered a question that interactive Internet marketing is supposed to answer. Namely, interactive Internet marketing is required to provide answers to the following questions: (a) what is the most effective customer retention strategy? (b) what is the most effective to driver customer to a purchasing decision? (c) what is the most effective customer acquisition strategy? and (e) what is the most effective product up- sell strategy? By analyzing collected user data interactive marketing researchers try to answer these questions in a predictive manner. The predictive data analysis, which is at the core of the data empowered marketing strategy, is accomplished through diverse data modeling techniques. Dou (1999) described the use of the Catastrophe Theory to model online store sales. Other techniques include the Recency-Frequency-Monetary Value (RFM) model designed to assess the customer purchasing decision. However, these and other models have several fundamental problems. First of all, these models are predictive and as such carry a fair amount of statistical error due to a need to make assumptions and use guess work. Consequently, these models are subject to Simpson’s Paradox where a lack of understanding of data granularity could lead to incorrect conclusions. Secondly, the Internet is an open system where not all of the variables and their effect are clearly defined (Dou, 1999). As such confounding variables become a real issue. Thirdly, the feedback loop between data collection, data analysis and interactive Marketing changes is quite long and inefficient (Dou, 1999). In addition to the limitations mentioned above, predictive modeling strategies are fundamentally unsuitable to the set of marketing tasks related to product development and risk mitigation. The data empowered marketing strategy absolutely requires extensive amounts of data in order to come up with a predictive marketing model. However, at the stage of new 7
  • product development, such data does not exist or is very limited. As such, predictive modeling marketing strategies by definition are not suitable for new product development. In the past marketers had to rely on focus groups to gain some guidance on new and existing product development (Mark, 2003). However, the focus group method has proven to be extremely error prone due to the Survey Paradox where people say one thing and then another. According to Mark (2003), in many cases companies would also fail to generate a focus group of the required diversity. As such it was not uncommon to see focus group results being skewed to a particular group of users. In addition, the focus group approach to product development resulted in a monolithic product development driven by corporate executives. In this product development model, product ideas were generated by company managers and corporate executives. Kohavi, Longbotham, Henne and Sommerfield (2007) described this model as the “Highest Paid Person’s Opinion” (HiPPO), where the most senior member of the team comes up with the product idea or is responsible for choosing among multiple product options. When the product idea is selected, the resultant product is manufactured in the monolithic fashion and subsequently demonstrated to the focus group. One possible name for the risk profile in this type of product development is “Risk Backloading”, where the bulk of the risk is born at the end of the product development lifecycle. Consequently, HiPPO initiated and focus group validated product development is extremely deficient in both risk mitigation and consumer-product fit. The main goal of this study is to propose an alternative interactive Internet marketing product development strategy that mitigates the shortcomings of predictive data modeling and HiPPO-Focus Group product development. It addresses data modeling shortcomings related to statistical uncertainly, data feedback loop and confounding variables. In addition it recommends 8
  • a more advantageous interactive Internet marketing product development approach that places product initiative in the hands of the end-users and allows for a more advantageous risk profile related to product innovation and product development. Even though this study treats Internet and interactive marketing synonymously, it more specifically considers only the part of interactive marketing that is enabled by Word Wide Web and Internet technologies. The findings of the study should however be applicable to the interactive devices mentioned above. Product Development As mentioned earlier, the use of experimentation to investigate the problem has been described by several researchers. It is possible to consider product development as a particular type of problem and so the process of experimentation should be fully applicable to product development. According to Thomke (1997), experimentation as a method of product development has been utilized by a number of industries including Automotive, Pharmaceutical and Electronics. Furthermore von Hippel (1998) posited that experimentation facilitates product development by end-users of the product. User-driven product development allows companies to get away from the HiPPO principal, where product decisions are made by the people who are furthest removed from the product itself. Product users themselves consciously or subconsciously galvanize and propel product development forward. More importantly they do so with their actions instead of the words and in this way they eliminate the Survey Paradox. Thomke, von Hippel, and Franke (1997) argued that employing experimentation in product development allows companies to improve their competitive standing. Since products are shaped by the end-users directly, the chances of product failure are greatly decreased. The experimentation process allows controlled failures early and often when the cost of change is minimal. By employing the experimentation approach companies reverse the risk profile from 9
  • back-loading to frontloading. Depending on the nature of the product companies may utilize several experimentation modes: (a) simulation; (b) prototype; and (c) sample. Experimentation modes depend on such factors as experiment result applicability, experiment cost and experiment efficiency. According to Thomke (1997), experimentation modes constitute a continuous process optimized based on the efficiency vs. number of experimental cycles. Based on the applicability of experimentation to product development in other areas it would seem logical that the process of experimentation would be pertinent to Internet marketing and interactive marketing as well. However, very little if any empirical studies have been conducted to explore this matter. Researchers such as Li (2003), Raman (1996), Mark (2003), among others, have broached the subject from the predictive modeling point of view. Even if experimentation is mentioned it lacks continuity with interactive marketing itself and is always driven by reactive analysis. Background of the Study This study examines the applicability of experimentation to product development in the confines of Internet and interactive marketing. Since, empirical evidence of experimentation in the field of interactive marketing is sorely lacking it draws on experimentation experience from other industries and product categories. Its background is rooted in the fields of experimentation, product development, production innovation, Internet development, Internet marketing and interactive marketing. The narrative of the study is centered on the product development strategies in interactive Internet marketing. It is designed to address what can be called the Information Age Product Development Paradox, where companies that have a significant Internet eCommerce presence tend to ignore technology-centric product development strategies 10
  • and instead favor low-tech solutions. The paradox consists of a seeming contradiction between relying on the technologies as the source of survival, yet disregarding them when it comes to product development. With regards to product development, companies involved in the Internet and the interactive marketing tend to ignore product development strategies that have been developed in other fields. As such they face problems related to product risk, product improvement and product innovation. Purpose of the Study The main purpose of the study is to examine the applicability of experimentation as the product development strategy in the companies involved in interactive marketing. It examines the use of experimentation in all parts of the interactive marketing product development lifecycle: (a) idea generation; (b) idea screening; (c) concept development and testing; (d) business analysis; (e) beta testing and market testing; (f) technical implementation; and (g) commercialization. In addition this study examines the impact of experimentation on product evolution as a function of user interactivity. It aims to contribute to the experimentation body of knowledge as well as establishing an empirical base for interactive marketing experimentation practice that is quite limited at the present time. Furthermore, this study looks at the unique elements of interactive marketing experimentation. It attempts to define the majority of the exogenous and endogenous interactive marketing factors and their effect on interactive marketing goals. The secondary purpose of the study is to point out the gaps and propose improvements in the Design of Experiments Theory, as it pertains to interactive marketing. 11
  • Research Questions This study aims to answer six main research questions. What is the impact of experimentation on interactive marketing goals? What are the key experimentation levers pertaining to interactive marketing? What is the impact of experimentation on product innovation in interactive marketing? What is the impact of experimentation on interactive marketing product improvement? What is the impact of experimentation on interactive marketing product development and deployment risk? What is the impact of experimentation on the competitive standing of the companies involved in interactive marketing? Online conversion is one of the key goals of interactive marketing companies. Even though conversion is not the only goal, the majority of interactive marketing companies are striving to achieve higher conversion. Even though conversion events of interactive marketing companies are quite diverse, in essence, the conversion event is equivalent to a revenue generating event. It is paramount to determine if the use of experimentation has a positive effect on online conversion. This study examines a number of exogenous and endogenous variables pertaining to interactive marketing and analyses their impact on the interactive marketing goals. The aim of the study is not to define an exhaustive list of exogenous and endogenous factors across all interactive marketing products and environments, but rather to determine an effective subset usable across interactive marketing companies of different types. A significant number of the research questions posed by this study are devoted to the analysis of the impact of experimentation on product development lifecycle. More specifically the study aims to examine the impact of experimentation on product innovation, product improvement, product development and product deployment risk. Similar to the goals, the interactive marketing companies’ products are equally as complex and as diverse. However, it is 12
  • important to understand the fundamental benefits offered by experimentation as it relates to interactive marketing product development regardless of the product type. Lastly, this study will examine the impact of experimentation on the competitive standing of companies in the interactive marketing space. The aim of the study is to show that companies in the interactive marketing space that abstain from experimentation, in relation to product development, ignore it at their own peril and put their competitive standing and quite possibly long term survival at risk. Nature of the Study Experimentation in the interactive marketing field is a social phenomenon since it is designed to achieve the best results through social interaction. Despite the fact that the social interaction in this context occurs between man and machine, the social context of the interaction is preserved. As such the use of a qualitative design approach is consistent with social constructivism and fully justified as the research approach for this dissertation. It is would have been absolutely acceptable to employ a quantitative or mixed-method design approach since experimentation relies heavily on statistical analysis and statistical data modeling. However, the topic of this dissertation and the state of the empirical research in the experimentation field of interactive marketing are more conducive to qualitative analysis. This study is theory forming in nature and as such it is consistent with one of the strong fundamentals of qualitative analysis. Significance of the Study The experimentation approach can give companies involved in interactive marketing the means to test marketing ideas without putting the majority of its traffic at risk. In the past 13
  • marketing ideas have been tested using surveys and focus groups. However, this approach is long and inexact. It also gives rise to what can be called a “Survey Paradox”, where people say one thing and do another (Sheffrin, 1996). The process of experimentation presents qualitative and quantitative results and represents an innovative development in testing Internet marketing and interactive marketing ideas. However, very little or virtually none of the empirical research confirms positive effects of experimentation by online conversion when used as part of interactive marketing strategies. Furthermore, due to a lack of solid theoretical research companies involved in experimentation in the field of interactive and Internet marketing are forced to develop a practical base behind experimentation through trial and error. Such approach results in limited success and might lead to companies abandoning experimentation altogether. In addition companies that do successfully pursue experimentation, in the context of interactive marketing, quite limit their success due to poor alignment between the Business and Information Systems designed to conduct experimentation. As such experimentation is done haphazardly, lacks continuity and does not result in a cohesive strategy. The significance of this study is in establishing a theoretical base behind experimentation in the interactive and Internet marketing fields. One of the key implications of the study is the assertion that companies that are involved in interactive and Internet marketing, outside of the experimentation framework, negatively impact their competitive standing, increase product development risk and reduce their likelihood of success in achieving their marketing goals. Another essential contribution of the study is an outline of the implementation process in the context of the experimentation framework that allows achieving cohesion between departmental units involved in the interactive marketing experimentation. This study positions experimentation as a product development and product improvement method designed to 14
  • optimize the marketing goals of the companies involved in interactive and Internet marketing. Definition of Terms Base Flow: current web site execution path with the highest fitness value. All related flows performance is compared to the Base Flow. Challenger Flow: variation of the Base Flow designed to achieve a higher fitness value. Confirmation Flow: previous Base Flow designed to confirm its losing status to the current Base Flow. The introduction of the Confirmation Flow can be classified as confirmation testing, where results of the previous Experiment are reconfirmed. The purpose of the confirmation testing is to make sure that the previous Base Flow had lost to the current Base Flow in the head to head comparison and was not a result of a statistical error or impact of unaccounted or unknown confounding variable. Experiment: manipulation on one or more of the endogenous variables in the context of the exogenous variables with the purpose of achieving Silo Goals. The purpose of the experiment is to reject or accept the null hypothesis. Experimentation Levers: collection of exogenous variables that is intrinsic to the Experiment. Flow: single variation of the web site execution path. Typically the experiment consists of several Flows that manipulate a single variable for a side by side comparison. Multivariate Experiment: experiment consisting of manipulation of multiple endogenous variables simultaneously. Multiple Singlevariate Experiments: Multiple experiment sets affecting a single endogenous variable per set. 15
  • Phantom Flow: flow that does not cause a variation in behavior. It represents a virtual duplication of any physical flow for achieving statistical significance. Silo: area under experimentation. The Silo usually consists of a collection of endogenous variables that can be manipulated in a particular context. In typical interactive marketing Silos consist of landing web pages, up sell web pages, offer pages, etc. Silo Goal: purpose of the experimentation. In the majority of interactive marketing cases the Silo Goal is represented by conversion also known as a purchasing decision. Other Silo Goals may include link offs, up sell actions, subscriptions, content contribution, etc. In general the terms the goal of the experimentation is to achieve the highest fitness value. Singlevariate Experiment: experiment consisting of manipulation of a single endogenous variable. View: single web page alongside the web site execution path. Typically Flow consists of several Views that contain one or more endogenous variables that are being manipulated. Imposed Flow: flow that allows predictive execution of the web site path without distribution and analysis. Imposed Flow is often utilized for regulatory or testing purposes. Silo Visit: represents a visit to one of the areas under experimentation, typically represented by a collection of web pages. Site Visit: represents a visit to the web site or any other Traffic Origin of the interactive marketing company. Traffic Origin: source where web traffic originates. Traffic Origin is often represented by one of the interactive marketing channels such as email, search, display, affiliates, call center and social media. Traffic Origin is one of the key exogenous variables. Traffic Criteria: criteria that is intrinsic to the incoming web traffic, but extrinsic to the 16
  • Experiment itself. Traffic criteria are a collection of exogenous variables. Traffic Distribution: allocation of the web traffic between Experiments and Flows according to the distribution algorithm. Traffic Distribution between Experiments and Flows must add up to 100% otherwise some of the incoming traffic will fail to be distributed. Vertical: product centric collection of interactive marketing assets. View Visit: represents a display web page or any other asset of interactive marketing such as email, banner, etc. Visitor: represents a unique visitor to the web site. Assumptions and Limitations In addition to paradigmatic and methodological assumptions this study has a number of domain assumptions. First, it assumes the synonymous nature of following terms: A/B Testing, Split Testing, A/B/C Testing, Singlevariate Testing, Singlevariate Multi-Experiment Testing, and Multivariate Testing. All of these terms are aimed at describing the same phenomenon and had evolved under a common umbrella (see Appendix C). However, it needs to be noted that the term multivariate testing refers to the experimentation technique related to univariate testing strategies, yet it is not synonymous with them. However, for all intent and purposes the practice of multivariate testing as well all univariate testing techniques will be assumed to be describing a common phenomenon. In cases where the study discusses the intricacies of a particular experimentation approach the name of the approach will be explicitly noted. In the same vain such terms as Internet marketing and interactive marketing are treated synonymously and interchangeably. In cases where a distinction between these concepts needs to be made it is made explicitly. 17
  • This study also assumes that all participants of the study are not aware of their active participation in the study. This assumption is essential for illuminating participant bias. It is also assumed that all participants of the study have at least cursory knowledge of interactive and internet marketing as well as a practical understanding of experimentation as it applies to the above mentioned concepts. Besides listed domain assumptions this study has a number of paradigmatic and methodological assumptions. In some respects it follows the basic assumptions of the systems approach. It assumes objective reality that is independent of the observer. Similar to the systems approach the key aim of experimentation is to model the real world phenomenon. As such it employs inductive and deductive reasoning as well as verification as the basis for its conclusions. It assumes that the results of any given experiment may be a result of casual relationships and must be verified by further experimentation. The methodological assumptions of the study are related to phenomenological inquiry. According to Becker (1992), the researcher of the phenomenological study is viewed as a co- creator of knowledge alongside the participants of the study. Even though such researcher involvement is beneficial, in this particular case discrepancy between the participant’s and the researcher’s knowledge of the studied phenomenon is too great. As such, the level of threat to researcher bias, validity, generalization and repeatability of the study is too great. Consequently, the researcher employed a combination of transcendental and experimental phenomenological methodology that allowed the researcher to detach his experience from the experience of the participants of the study. 18
  • Organization of the Remainder of the Study The remainder of the study consists of four additional chapters. Chapter Two examines literature related to interactive and Internet marketing, product development and experimentation. The literature has been selected to describe each of the aspects of the study independently yet leading to cohesion between experimentation, interactive marketing and product development concepts. Chapter Three describes the design methodology that was chosen to conduct the study. It defends the choice of research methodology by examining the methodological fit to the research approach and the subject of the study. It also describes elements of the study including the data collection procedures, data sampling, data collection instruments and data coding. Chapter Four illustrates the findings of the study and undertakes a detailed analysis of the results of the study. It consists of data analysis and data display for describing and explaining the phenomenon of the study. Chapter Five summarizes the findings of the study and introduces an alternative hypothesis for the findings of the study, validity and bias as well as examines the trend in the findings of the study. It also proposes topics for future research and the areas of inquiry in the subject area. 19
  • CHAPTER 2: LITERATURE REVIEW Experimentation Introduction The process of experimentation in general is at the center of scientific discovery. In fact science or knowledge derived from experimentation is called “experimental science”. People have utilized experimentation since ancient times with one of the classic examples of early experimentation originating from Egypt around 2613-2589 BC. When Egyptians attempted to build a smooth-sided pyramid they engaged in the process of experimentation. They initially started building a pyramid at Meidum which collapsed due to angle acuteness. Based on the result of this failed experiment Egyptians altered the angle of the Bent-Pyramid at Dahshur more than half way through to save it from collapse, resulting in a bent shape. Subsequent smooth- sided Egyptian pyramids have utilized the correct angle from the outset. By looking at the experiment conducted by the Egyptians it is possible to extract several observations: (a) experiments often fail (sometimes they are designed to fail); (b) full scale experiments can be quite expensive; (c) conducting experiments consecutively may take a long time; and (d) conducting experiments concurrently allows effective side by side comparison. Unbeknown to them, this Egyptian experiment created the basis for the Design of Experiments theory. This early experiment and subsequent application of the experiment results contained the majority of elements founds in modern product experiments. Thomke (1997) described experimentation as one of the forms of problem solving. In short Thomke (1997) defined experimentation as a trial and error process. According to Thomke (1998), the experimentation process consists of four major steps: (1) design (design consists of 20
  • coming up with an improved solution based on the previous experience from the preceding experiment cycles); (2) build (the build step consists of modeling and constructing products to be experimented upon); (3) run (the run step of the experimentation process consists of executing the experiment in the real or simulated environment); and (4) analyze (analysis consists of data mining and data investigation collected during experiment execution). Thomke (1998), pointed out that the experimentation process changes under pressure from exogenous elements. These exogenous elements consist of uncertain requirements, environmental and technological changes. Experimentation Strategies The process of conducting experimentation is not uniform. The choice of the experimentation strategy depends on the problem it is trying to resolve. In order to understand experimentation strategies it is important to look at what biologists define as the “fitness landscape” (Beerenwinkel, Pachter & Sturmfels, 2007). Quite often the solution to a particular problem is not singular. Fitness landscape consists of all possible solutions to a particular problem. A fitness function defines the quality of the solution in relation to other solutions to the same problem. The optimal solution to the problem is thought to have the highest fitness value. A real world demonstration of the fitness landscape could be observed by looking at the simple problem. For example, the way between home and the office consists of n routes. The set of all routes constitutes a fitness landscape. Among all of the possible routes one of the routes is the “best”, where best = f(n). Since, best is a relative term it must be defined in the context of the fitness landscape. It is likely that the majority of people would consider the best route to constitute shortest time. However, it is possible that some would choose a scenic route to be the best. It is also possible to have multiple optimal solutions where the best route changes based on exogenous elements such as time of day, day of the week and weather conditions. In that case, 21
  • the best = df(n) / dt, taking the time of the day into consideration. Thomke (1998) defines three experimentation strategies: (a) parallel experimentation; (b) serial experimentation with minimal learning; and (c) serial experimentation with learning. It is possible to demonstrate these experimentation strategies by continuing with the “driving from home to office” problem. To find out the route that yields the shortest driving time between the home and the office and is the quickest way would require employing a parallel experimentation strategy. However, it is impossible to accomplish this using a single driver, since it requires taking all possible routes simultaneously. The driver of the car would need help from his friends. If the driver of the car was determined to find the shortest route himself/herself, he/she could employ serial experimentation with minimal learning or experimentation with learning. The serial experimentation with minimal learning would require the driver to follow a predefined plan of taking different routes until all possible routes where exhausted. After all routes have been tried, the driver would have analyze the time each has taken to determine the one with the shortest time. The serial experimentation with learning would allow the driver to avoid trying all routes by analyzing results of initial experiments and illuminating routes that certainly would not yield good results. Experimentation Models The experiments are further complicated by endogenous and the exogenous elements. It is certainly possible to affect the experimental results by changing the endogenous elements of the system (see Appendix E). For instance besides the different routes, changing drivers, car type, fuel type, and the number of drivers among many other things would possible affect the driving time. The experiments with multiple changing endogenous elements are called multivariate. The experiments with a single endogenous variable are called univariate. It is 22
  • always possible to represent multivariate experiments as a series of univariate ones, by temporarily freezing all but one variable. Ideally, in order to understand the impact of all exogenous variables each of them requires a separate experiment. In other words in order to understand the impact of the time of departure from the office on the overall travel time, the driver needs to conduct experiments by leaving the office every hour on the hour. In the same vain all weather conditions must be tested as well. Based on the analysis of the experimental models it is quite obvious that the complexity of experimentation grows exponentially based on the number of endogenous and exogenous elements, such that Number of Experiments (E) = k * (m^n)!, where k is the number of the steps in the process, m is the number endogenous variables and n is the number of exogenous variables. Based on formula above it is possible to arrive at the conclusion that the chance of guessing the option with the highest fitness value, when the number of endogenous and the exogenous elements are beyond trivial, is infinitesimally small. At the same time running experimentation of all combinations of endogenous and the exogenous factors in order to determine the best possible combination yielding the highest fitness value would constitute a full factorial experiment (Xu &Wu, 2001). However, even with a trivial number of exogenous and endogenous factors the number of resulting experiments that satisfies full factorial design is truly staggering. By applying the formula listed above a single step process with 3 endogenous factors and 3 exogenous factors would result in 1.0888 E 28 number of experiments. For all intent and purposes running full factorial experiments beyond a trivial number of factors is simply not practical. In order to resolve the conundrum of no experimentation and full factorial experimentation it is necessary to look at the nature of exogenous and endogenous factors. Not all of the variables involved in the experiment affect the outcome of the experiment equally. 23
  • Depending on the nature of the experiment it is possible to find a smaller subset of variables that have the greatest effect on the experiment. Anderson (1972) called this approach to experimentation partial factorial. There are number of statistical methods of determining what variables truly matter to the outcome of the experimentation. Li (2003) identified the following partial factorial reduction models: (a) Univariate Poisson (relies on the analysis of the variable of all of the involved variables); (b) Univariate Tobit without Log Transformation; (c) Univariate Tobit with Log Transformation; (d) Discretized Univariate Tobit with Log Transformation; (e) Discretized Univariate Tobit with Heteregeity; (f) Multivariate Count; (g) Multivariate Count with Mixture; and (f) Multivariate two-state hidden Markov Chain Tobit. Even a cursory look at the listed models allows them to be separated into two categories: (a) univariate; and (b) multivariate. Fundamentally univariate models, where ANOVA tests are applied in succession, are designed to ascertain the effect of the independent variables on the dependant variables (Biskin, 1980). In the context of the experimentation univariate models are designed to highlight the exogenous and endogenous factors that have a significant impact of the outcome. On the other hand multivariate models, where a MANOVA test is conducted, are designed to come up with sets of independent variables that have an impact on the dependant variables (Huberty, 1986). Again by taking the experimentation context into account the multivariate models are designed to highlight sets of exogenous and endogenous factors that have a significant impact on the outcome of the experimentation. According to Huberty and Moris (1989) the fundamental difference between multiple univariate ANOVA tests and multivariate MANOVA tests consist in the consideration of the effects of the independent variables on each other and their compound effect on the outcome. More specifically univariate models tend to ignore the relationship between exogenous and endogenous factors and their compounding effect, where multivariate models take this relationship into account. 24
  • In practical terms, going back to the “home to the office” driving example, univariate models consider the independent impact of the weather conditions and the time of the departure on the driving time, where multivariate models would consider these two factors in conjunction with each other. Even though interactive and Internet marketing are relatively new phenomena they have generated a fair amount of popular and scientific foment. It is even fair to say that the scientific community has been lagging behind interactive and Internet marketing practitioners who have pushed exploration boundaries. At the same time, in recent years, interactive and Internet marketing, as a subject of scientific inquiry, have seen an increased rate of exciting empirical research. These studies have focused on interactive and Internet marketing from social, physiological and physiological perspectives (Jebakumari, 2002; Milley, 2000; Macias, 2000; Liu, 2002; Newman, 2001; Mark, 2003; Bezjian-Avery, 1997; Raman, 1996). Unlike Internet and interactive marketing, experimentation is not a new phenomenon. There is abundant evidence that ancient Egyptians conducted experiments during pyramid construction. In a more recent example of experimentation James Lind, while servicing onboard the Salisbury, conducted an experiment of using citrus to cure scurvy. One of the notable differences between early experimentation efforts and the experiments conducted by Lind was the use of control and treatment groups. The results of the control group were compared with the results of treatment groups in order to confirm or reject the hypothesis of the experiment. In the early 20th century Ronald Fisher formulated a mathematical method for designing and analyzing experiments. He had introduced “factorial” as the term applicable to experiments involving several factors or variables at the time (Fisher, 1935). Fisher (1926) initially used “complex experimentation” as the term describing experimentation with multiple variables at the same 25
  • time. In more recent years researchers have focused on experimentation in the context of product development. Enlarged experimentation methods have been researched in the context of Electrical and Mechanical Engineering (Wang, 1999; Hansen, 1989; Donne, 1996). It is certainly not surprising that experimentation practice has been widely employed in industrial manufacturing since manufacturing product commitment is quite expensive and may result in significant losses and even, in some cases, impact on the long-term survival of the company. The product development lifecycle often requires experimentation to be part of the product development process. The deceptive ease of change in the Internet and interactive marketing product development has resulted in the situation where experimentation best practices found in the industrial product engineering are ignored. There is certainly a glaring lack of empirical research into experimentation in the Internet and interactive marketing fields as it pertains to product development. There are a few empirical works that have broached this subject (Dou, 1999; Li 2003; Ozdemir, 2000); however, these studies are primarily dedicated to data mining and predictive modeling rather than experimentation as a continuous practice. It is important to note that in recent years several researchers have focused on interactivity in general and interactive experimentation in particular as a key driver in product innovation (Thomke, Von Hippel & Franke, 1997; Thomke, 1998; Thomke 2001; Von Hippel 1998; Thomke; 1995). These studies have asserted that product innovation is driven by product users themselves through interaction and experimentation. Product innovation was particularly highlighted in these studies and was considered separately from the remaining phases of product development lifecycle. In large, this part of this dissertation capitalizes on the mentioned studies in the context of applying findings of above mentioned research both to Internet and interactive 26
  • marketing. Some of the mentioned research in the areas of interactive marketing, experimentation and product innovation is examined in more detail below. Interactive Marketing If the Internet timeline could be separated into three decadal stages: (a) mid 1990s – introduction stage; (b) early 2000 – development stage; and (c) late 2000 – maturity stage; then the Jebakumari (2002) study could classified as a study of the stages of Internet development. It was during this time that Internet interactivity came into strong researcher and practitioner focus. The overall purpose of the Jebakumari (2002) study was to describe interactivity in the context of Internet marketing. Lyons (2007) offered several research questions: (a) what are the nature, characteristics and components of interactivity? (b) what are the shortcomings of the traditional marketing models in context of the interactive medium? (c) how is interactivity related to comprehension? Jebakumari (2002) examined traditional marketing and its shortcomings to explain this interactive phenomenon. Both traditional and interactive Internet marketing were compared and contrasted. The conclusions reached by Jebakumari (2002) were reminiscent of a similar study conducted by Mark (2003). Jebakumari (2002) found that a number of traditional marketing techniques were inconsistent with the interactive media and did not adequately address the interactive audience. A study by Milley (2000) could be attributed to the late introductory and early development stages of Internet marketing. Miley (2002) explored, what he calls, Web-enabled consumer marketing, its intricacies and specifics. Miley (2002) tried to formulate a theoretical 27
  • model of interactive marketing on the basis of numerous case studies, presented and analyzed in succession. An additional focus of the study was related to the operational recommendations of running a consumer oriented interactive web site. Miley (2002) proposed the following research questions: (a) what is the theoretical basis of Web-enabled consumer marketing? and (b) how should the company align its operations to be congruent with the Web-enabled consumer marketing model? Miley (2002) reached the conclusion that Web-enabled consumer marketing requires analysis of behavioral user data to guide future actions and marketing decisions. He also concluded that in order to facilitate comprehensive data analysis, interactive user data must be both extensive and complete. As such, Web-enabled consumer marketing or interactive marketing companies must position their human and systems resources, as well as establish operational practices conducive to data capture. The Raman (1996) study could be attributed to the introductory period of interactive and Internet Marketing. Raman (1996) explored interactivity on the Web, at the time when it was emerging phenomenon. In particular, Raman (1996) examined the desired customer exposure to online banners. Similar to the later studies by Mark (2003) and Jebakumari (2002) that focused on the comparison and contrast between traditional and interactive marketing, Raman (1996) contrasted banner exposure in traditional and interactive marketing models. The Raman (1996) study is similar to parallel study by Bezjian-Avery (1997) which attempted to define interactive marketing and its core concepts. Raman (1996) proposed the following research questions: (a) what are the factors affecting the desired interactive exposure? and (b) how do the levels of interactive exposure affect the desired advertising outcome? Raman (1996) concluded that the dominant factor 28
  • affecting desired interactive exposure is predominantly related to interactive content richness. Additionally, Raman (1996) concluded that an interactive advertisement that speaks to the consumer on an individual level at the same time as being pertinent and engaging has a high chance of achieving the designed interactive outcome. Experimentation As mentioned earlier, the body of knowledge regarding experimentation and experiment design is heavily focused on major engineering disciplines. The empirical research on the subject of experimentation in interactive and Internet marketing is scarce and tangential. This study relies on several seminal works on Design of Experiments, data modeling and product innovation. In the area of Design of Experiments this study examined several research papers related to the Taguchi Method. Weng (2007) presented a detailed analysis of experiment optimization methods. These methods were compared on the bases of (a) global optimization; (b) discontinuous object function; (c) non-differentiable function; and (d) convergence rate. Weng (2007) found that the Taguchi Method scored extremely well in all of the compared categories. Weng (2007) gave a detailed review of the Taguchi Method itself and its benefits over other optimization methods. Weng (2007) also suggested several improvements to the Taguchi Method that can be applicable to experimental design in interactive marketing. In addition to the design of experiments in Electrical and Mechanical Engineering this dissertation is based on another tangential topic related to data mining and data modeling in interactive and internet marketing. It is important to note that it is impossible to conduct experimentation without being engaged in some form of data mining and data modeling. The experimentation is enabled by data analysis and data mining. More specifically, the 29
  • experimentation process is data analysis driven. In his essays on interactive marketing Li (2003) examined three cases of interactive marketing. In the first essay Li (2003) described the functionality of cross-selling services on an interactive banking web site. Li (2003) analyzed the behavioral reasoning behind online user actions as they pertain to the purchasing of products and services offered by the interactive banking web site. In conducting behavioral analysis Li (2003) utilized several multivariate probit models implemented by Hierarchical Bayer framework. This dissertation examined the applicability of the models proposed by Li (2003) in conducting interactive real-time online experiments. In his second essay Li (2003) analyzed the browsing behavior of users on several interactive web sites. In order to predict future browsing paths Li (2003) utilized several Poisson and discretized tobit models. These models were compared and contrasted in the context of their ability to accurately predict user browsing behavior. This dissertation utilizes the modeling technique findings presented by Li (2003). In his third essay Li (2003) analyzed purchase and conversion data from several eCommerce web sites. He used this data to build a predictive purchase model. Li (2003) concluded that his Hierarchical Bayer framework supplemented with hidden Markov model could accurately predict a path reflecting user goals ultimately leading to a potential purchase. This dissertation capitalizes on the findings of this essay during the set up and analysis of the effect of experimentation on reaching interactive marketing goals. Similar to the Li (2003) study, research by Dou (1999) utilized similar statistical analysis for modeling online sales. Dou (1999) examined the applicability of the Catastrophe Theory to modeling actual behavior and predicting potential purchasing online decisions. Dou (1999) explored what he termed the data empowered marketing strategy, where data was mined through tracking users to guide the interactive marketing decisions of the company. Even though Dou 30
  • (1999) did not mention this concept as interactive marketing experimentation by name, he hypothesized that interactive marketing data can be used to alter the interactive user experience in real time as more of the user data was collected and analyzed. Dou (1999) called this approach adaptive marketing communication, where consumer behavior is analyzed through continuous observation. Dou (1999) proposed that interactive marketing data can be modeled using the Catastrophe model. He hypothesized that Catastrophe Theory is eminently suitable for this type of analysis and predictive modeling. Dou (1999) concluded that it was indeed possible to model and ultimately predict the browsing and purchasing behavior of users on interactive marketing web sites. The significance of both the Li (2003) and Dou (1999) studies is the fact that interactive marketing data is been actively analyzed using a multitude of statistical models in the context of interactive marketing. However, it is import to note that use of the Taguchi Method for similar analysis has not been empirically researched. Additional computing paradigms for predictive data modeling such as Evolutionary Computing and Genetic Algorithms have been explored by several researchers (Ozdemir, 2002). Ozdemir (2002) argued that Evolutionary Computing offers real potential in deriving a best fitness value. As such it holds significant promise for online data modeling and interactive marketing experimentation. Product Innovation A significant portion of this study is devoted to analyzing the impact of experimentation on the product development lifecycle in the context of interactive marketing. Even though there are few empirical studies that directly deal with experimentation in interactive marketing, emphasizing the web site as an interactive marketing product, there is a significant body of 31
  • empirical work that is devoted to experimentation in the context of product development. This dissertation capitalizes on the several seminal works by Thomke and Von Hippel. Thomke (1995) hypothesized that the mode of experimentation such as prototyping and simulation has a significant impact on the economics of experimentation. More specifically, Thomke (1995) proposed that the use of simulation experimentation is more economical and therefore far more likely to be used in product development. Consequently simulation experimentation could be viewed as a product enabler and innovation driver. Thomke (1995) presented two case studies where experimentation was used in the design of new pharmaceutical drugs and integrated- circuit based systems. Thomke (1995) proposed experimentation design cycles consisting of designing, building, running and analyzing activities performed in a contiguous manner. Each successive cycle was built taking into account the findings of the previous cycle. This study hypothesized the applicability of this cycle in general, and the process in particular, to interactive marketing product development (see Appendix E). Thomke (1995) found that switching between prototyping and simulation experimentation modes significantly affected experimentation economics, resulting in a substantial reduction in design cost. In a seemingly unrelated study Von Hippel (1998) argued that product innovation should be driven by the people who would benefit from the end product of innovation, end users of the product themselves. Von Hippel (1998) described what could be called the Von Hippel paradox, where product specialists should not be primarily responsible for product innovation, but rather defer to product users as a source of ultimate innovation. Von Hippel (1998) described this paradox as a shift in locus of problem-solving. In this dissertation Von Hippel’s ideas are combined with the approach proposed by Thomke (1995), where interactive marketing product development is driven by users through interactive experimentation. 32
  • CHAPTER 3: METHODOLOGY Description of the Methodology The study assessing the impact of experimentation on interactive product development utilized a qualitative research paradigm. The choice of qualitative research methodology was related to the nature of the topic and the innate characteristics of the field of the study. Employing qualitative research methods makes the quality of the data of paramount importance. Consequently, emphasis is placed on how and under what circumstances the data is collected (Morgan & Smircich, 1980). In contrast to quantitative research methods, it is rare to see a qualitative researcher working with large quantities of data. This is the case with the current study as all analyzed data comes from a single organization. Maxwell (1992) defined the qualitative research methods as theory forming. These methods are used to generate new theories or introduce new hypotheses. Maxwell (1992) called qualitative research a paradigm that is concerned with a “breadth first” approach as opposed to a “depth first” as is the case with quantitative research. More specifically, qualitative research is preoccupied with describing a phenomenon as thoroughly as possible, and forming a theory behind it. Based on the paradigmatic characteristics provided by Maxwell (1992), the use of qualitative research methods was consistent with the goals of the study and the state of knowledge in the field of experimentation in the context of interactive marketing. This study utilized a phenomenological method as one of the research methodologies under the qualitative paradigm umbrella. The phenomenological method was first formulated by Husserl (1983). Creswell (2007) defined the phenomenological method as a description of the 33
  • meaning for several individuals of their direct experience of a concept or a phenomenon. In this particular case the phenomenon is a process of experimentation in the context of interactive marketing. Husserl (1983) described the application of this phenomenological approach as an execution of three consecutive steps. The first step consisted of adopting a phenomenological method that encouraged the researcher to infuse quantitative data with the qualitative context that allowed the data to be meaningful. The second step consisted of seeking out an instance where the phenomenon can be studied in its natural context in order to distill the essence of the phenomenon. The third and final step was described by Husserl (1983), and consisted of describing the discovered meaning of the phenomenon. The experimentation phenomenon in the context of interactive marketing sits well in the phenomenological inquiry. This researcher wanted to discover meaning behind the experimentation phenomenon in the context of interactive marketing by studying individuals who have experienced the phenomenon first hand. Even though participants of the study have experienced the phenomenon they are not necessarily aware of its meaning (Georgi, 2006). This point of view is certainly consistent with the description of the experimentation phenomenon. The participants of this study have certainly experienced the experimentation phenomenon in the context of interactive marketing, but by and large they are not aware of its meaning and its fundamental characteristics. By utilizing phenomenological tools as a bracketing, horizontalization, clustering, delimiting and imaginative variation, phenomenological tools allow the researcher to extract meaning from the experiences of the participants of the study. The phenomenological method allows the researcher of the study to quantify his/her own experience by supplementing findings of the study with his/her own observations and interpretations in the context of the experience. Creswell (2007) described this type of 34
  • phenomenological method as hermeneutical. Van Manen (1990) described the researcher of the study as one of the participants of the study. This researcher has an extensive experience with experimentation in the context of Internet and interactive marketing. The phenomenological method allows researcher understanding of his own experience while maintaining a strong relationship to the topic (van Manen, 1990). However, in order to address generalization, validation, validity and bias, the researcher must employ bracketing to distinguish his own experience from the experience of the participants of the study. As such this researcher tried to deemphasize his own experience. This researcher employed a combination of transcendental and experimental phenomenology rather than hermeneutical phenomenology, where emphasis is placed on the experience of the participants of the study and the experience of the researcher is bracketed (Moustakas, 1994). Use of experimental phenomenology allowed the researcher to focus on the practical application of the phenomenon rather than the philosophical side of it. It is important to note the distinction between other qualitative methodologies such as grounded theory or other narrative approaches and phenomenological methods. Creswell (2007) made a distinction between narrative study and phenomenological study, where the former is experienced by several study participants individually, as opposed to as a group in a latter case. Even though participants of the study were selected from several groups participating in interactive marketing experimentation, individual group representatives described an experience of the group to which they belong. According to Cresswell (2007) phenomenological methods place emphasis on the shared experience of the phenomenon. It is critical to understand experimentation in interactive marketing in the context of a particular group as well as the organization as a whole. 35
  • Design of the Study The qualitative research paradigm employs interviews as its predominant data collection instrument (Babour, 1998). When an interview is conducted in a purely qualitative manner, the researcher then takes an active participation in the interview process. In that case, the researcher is considered to be an actual instrument of the study. Babour, (1998) pointed out that participants in a study often receive major guidance from the researcher throughout the interview process. The qualitative research paradigm thus encourages researcher participation in order to reduce language ambiguity and supplement the possible lack of context associated with quantitative data collection instruments. This study however, did not employ interviews as the means of data collection. The major concerns of the study were related to credibility, validity and bias. Since the researcher of the study is employed by the company where the research is being conducted, actual or potential undue influence was a paramount concern. The researcher struggled to maintain the balance between extricating himself from the data collection process on one hand, yet maintaining the qualitative nature of the study on the other hand. In order to address validity, credibility and bias concepts as well as maintaining the qualitative nature of the study, the researcher employed a research instrument used in mixed- method research studies. More specifically this study utilized a mixed-method survey as a research instrument. Johnson and Onwuegbuzie (2004) also described a mixed-method survey that embodies both qualitative and quantitative aspects. Mixed-method surveys typically contain questions found in fixed surveys. These types of questions are referred to as close-ended, where the set of responses are limited. In addition to quantitative questions, these surveys contain corresponding sections that allow freehand expression, allowing for a qualitative context to what 36
  • otherwise would be purely quantitative data. In contrast to close-ended questions these questions are open-ended. The resultant survey consisted of 15 open-ended questions and 17 close-ended questions. In order to eliminate possible researcher influence the survey was administered over the Internet in an anonymous fashion. In addition data was collected under false pretenses. The participants of the study were not told that the data was being collected for the purposes of research due to the possibility of participant bias. The survey was positioned as providing helpful feedback on the experimentation efforts of the company in the context of interactive marketing. Population and Sampling The study included 23 human participants. The participants of the study work in the same organization as the author of the study. A particular set of participants was chosen from all of the groups involved in the interactive marketing experimentation. The technology group was excluded from study participation, since the author of the study works for the technology group and may exert undue influence on the participants of the study. The participants of the study were randomly chosen from Interactive Marketing, Business Development, User Experience, Data Analysis and Creative Design groups. The participants from each of the mentioned business units provided information relevant to the results of experimentation and its impact on various aspects of interactive marketing. They were asked to elaborate on their experiences of experimentation in the context of the interactive marketing. The participants of the study were solicited on the perceived success of experimentation relative to its goals such as improved conversion, product innovation, product improvement, risk mitigation and improved competitive standing. 37
  • Assuming that the chosen sources were both valid and credible, further research credibility and validity depended only on the researcher himself/herself. One of the ways of selecting credible and valid sources is by selecting them at random. More specifically only a single representative of each business unit was selected at random. This type of selection method helped reduce personal bias. Measurement Strategy This study surveyed interactive marketing professionals in the confines of a single company. The study participants were selected at random to represent their interactive marketing channel. The respondents of the study were asked to complete a mixed-method survey consisting of 40 questions related to their experience with experimentation in the context of interactive marketing. The research questions of the survey were designed to understand the relationship between the corresponding dependent and independent variables. Since the number of independent variables was too great they were grouped under common categories. For instance, independent variables related to experimentation such as color, font, font size, and images were grouped under a visual category. It is important to note that independent variable categories were classified as either endogenous or exogenous. The resultant survey included five categories of endogenous independent variables (see Appendix B) (a) visual; (b) functional; (c) positional; (d) informational; and (e) behavioral, as well as six categories of exogenous independent variables (see Appendix B) (a) temporal; (b) demographical; (c) seasonal; and (d) contextual. In addition to the independent variables each of the research questions had a number of dependent variables associated with it. These dependent variables were assigned as follows (see Appendix B) (a) competitive standing (revenue, market size, profit, market share, and market 38
  • segmentation); (b) interactive marketing product development lifecycle (product risk, product innovation, product improvement, product life cycle, and product targeting); and (c) interactive marketing goals (cost per acquisition, cost per impression, cost per action, upsell, and click- through rate). Each research question was directly represented in the survey in the form of several survey questions. In addition to asking participants of the study to answer research questions directly, each of the dependent variables was investigated in isolation (see Appendix A). The survey questions were formulated to draw a connection between independent variables in an amortized form and the dependent variables associated with research questions. The amortized independent variables were referred to as “Experimentation” (see Appendix A), where dependent variables were called out in exactly the same way as they were specified in the “Conceptual Framework” (see Appendix B). Even though the quantitative questions were utilized alongside qualitative questions, qualitative data was not used in drawing conclusions of the study. The point of analysis associated with the quantitative data was to ascertain consistency between qualitative and quantitative answers. The quantitative survey questions used single and multiple choice scales. The qualitative survey questions utilized a measurement strategy associated with the phenomenological method consisting of horizontalization, clustering, textual and combined description. Instrumentation The survey (see Appendix A) contained 40 questions designed to solicit information related to the experimentation efforts of the company in the context of interactive marketing. The 39
  • company where the research was conducted utilized multifaceted interactive content. More specifically the company used email, search, social, display, internet and affiliate interactive marketing approaches. The participants of the study were asked to fill out the survey relating to their experience of product development through experimentation in each of the respective interactive marketing areas. The questions of the study were crafted to meet the objectives of the study. The close-ended questions of the study were not used in the final analysis of the study, but rather they were designed to ensure consistency of a corresponding open-ended question as well as to guide the user to stay in the confines of the intended question. The survey was designed and implemented using online survey software and conducted over the Internet. The participants of the study were invited by the CEO of the company to complete the survey via. The email contained the link to the online survey as well as an explanation of the purpose of the survey, ensuring that participation in the study was voluntary and anonymous. The survey contained five major sections: (a) introduction (questions related to overall experimentation experience); (b) interactive marketing goals (questions related to the impact of experimentation process on the goals of various interactive marketing channels); (c) interactive marketing product (questions related to the impact of the experimentation process on the various aspects of product development lifecycle of various interactive marketing channels); (d) competitive standing (questions related to the impact of experimentation process on the various key competitive indicators); and (e) sustaining effects (questions related to the sustaining effects of experimentation in the context of interactive marketing) Data Collection The data was collected via the SurveyMonkey.com web site. The initial survey was pre- 40
  • tested and modified according to the feedback from the pilot group and the mentors of the study. The pilot group consisted of three members chosen from a pool of potential participants. The participants of the study were given a week to complete the survey with multiple reminders sent two days and on the day before the survey expiration period. All questions in the study were designated as mandatory and the only two ways to exit the survey were to either to complete it or abandon it. If the survey was abandoned in order to proceed with the survey at the future date the participant of the study had to start the survey over again. According to the SurveyMonkey.com statistics none of the surveys were abandoned and the effective survey completion was at 100%. Due to employing the survey online participant anonymity was preserved. After all surveys were completed the survey results were downloaded onto the researcher’s computer and analyzed. At all times the survey results were protected from inadvertent or intentional disclosure. The surveys were conducted online over secure protocol and access to the survey results was username and password protected. When the results of the survey were downloaded to the researcher’s computer access to the computer itself was username and password protected as well. Data Analysis Procedures The data analysis procedures roughly consisted of the steps outlined by Creswell (1998) with slight adaptation for the needs of this study. These steps consisted of (a) horizontalization and bracketing; (b) clusters of meaning; (c) textual description; (d) composite description. It is important to note that the usual phenomenological step of transcription was omitted since the data was collected via an open-ended survey administered over the Internet. As such the data transcription consisted of downloading the results of the completed surveys. The data analysis 41
  • was conducted on two occasions. The initial data analysis consisted of interpretation of the surveys submitted by the “pilot” group, comprised of a small population sample. The subsequent data analysis was conducted during the analysis phase of the actual study. Horizontalization According to Moustakas (1994), the process of horizontalization consists of defining common themes and significant statements from the responses of the participants. The researcher read responses several times while trying to comprehend and interpret their meaning. The qualitative data collected was correlated with similarly intentioned quantitative answers. For instance, quantitative survey question, “Please select experimentation lever types that you feel are most instrumental in achieving interactive marketing goals? (a) visual; (b) functional; (d) behavioral; (e) informational” was paired with the qualitative question, “Please describe in your own words what experimentation levels were most useful in achieving interactive marketing goals?” All discrepancies were recorded and analyzed. These discrepancies were resolved in the final version of the survey and modifications were based on the analysis of the pilot sample. During the second reread path significant statements and common themes were underlined and extracted. All similar statements were grouped under a common umbrella and labeled accordingly. The process of horizontalization was performed several times. The resultant groups were compared and analyzed for discrepancies. Clusters of Meaning According to Van Manen (1990), the process of clustering consists of extracting meaning from the grouped quotes. The process of clustering capitalized on the previous horizontalization 42
  • step. The responses grouped under common labels were examined for the common clumps of meaning. In order to make this process simpler, groups were attributed to the corresponding research questions. Since some of the survey responses could have been attributed to the one or more research questions it was imperative to keep them organized. In some cases however quotes were attributed to multiple emerging themes. At the same time some of the quotes contained several themes or meanings at once. Textual Description The textual description step of the phenomenological method described by Alvesson and Sköldberg (2000) consists of the reflection of the participant experience written by the researcher. The textual description process logically followed a clustering step, where clusters of meanings were extracted and recorded. After analyzing clusters of meaning, this researcher tried to come up with the textual narrative of what survey participants were trying to convey. The resultant narrative contained a number of expressions used by the study participants themselves as well as the meaning phrases derived in the earlier step. Prior to coming up with the textual description the researcher of the study prepared a written description of his own thoughts and feelings about the experimentation phenomenon in the context of Internet and interactive marketing. By utilizing bracketing techniques the researcher tried to separate his experience from the experience of the participants of the study. Even though it was difficult for the researcher to extricate himself from the studied phenomenon, a process requiring a transcendental phenomenological method, a written description prior to the textual description step, raised his awareness of bias and validity. 43
  • Composite Description According to Giorgi (1985), composite description is designed to crystallize the meaning of the research itself. The narrative produced in the textual description step of the phenomenological method is wordy and lacks coherence. The researcher of the study tried to distill the meaning of the earlier produced narrative into a single thought that could be easily expressed. In most cases however, the composite description step of the data analysis related to phenomenological research methodology was combined with the textual description step. One of the key reasons for doing so was the fact that the qualitative responses were short and as such, represented a concise summary that required no further reduction. This is often the case with mixed-method research instrumentation, where the qualitative portion is filled out by the participants of the study rather than the researcher. The composite description section of the data analysis was used to discuss issues related to the analysis of a particular research question. . Qualitative Data Display for Describing the Phenomenon The qualitative data display is used to describe experimentation in the interactive utilized matrix display method. Since the study utilized a phenomenological design approach, the horizontalization step of extracting the most meaningful aspects of the survey answers was followed by clustering. Horizontalized data was clustered based on the business units of the participants. The clustered data was displayed using role-ordered display. Every research question was cross-tabulated with self-assessed experimentation experiences indicated by the research participants. 44
  • Qualitative Data Display for Explaining the Phenomenon The qualitative data display for explaining the experimentation phenomenon in interactive marketing utilized a matrix display method. Based on the problem type, experimentation phenomenon is ideally suited for the explanatory effects matrix display type. According to Miles and Huberman (1994), the explanatory effect matrix is used to answer questions of the following type, “Why were these outcomes achieved?” and “What caused them generally or specifically?” In the case of experimentation in interactive marketing, the study tried to prove causality between improved competitive standing, reduced operational risk, improvements along the product development lifecycle as well as improved online conversion and experimentation. The explanatory effects matrix consists of cross-tabulation between various groups that participated in the study and the research question itself. This researcher considers the explanatory effects matrix as the most useful display technique for explaining and predicting dissertation topic. It is a very good fit for experimentation into the interactive marketing phenomenon for several key reasons. First of all the matrix format allows an analysis of the experimentation practice across multiple departmental units involved in interactive marketing experimentation. However plausible, it was not expected that different departmental units were going to feel differently about the usefulness of experimentation and its positive impact on the dependant variables of the study such as competitive standing, operational risks and attainment of interactive marketing goals. Possible differences between opinions of the different departments may be explained by Proximity Principal, described by Torre and Rallet (2005). The group closest to a particular aspect of the phenomenon is more familiar with its nuances, when compared with other groups that have experienced this aspect in a tangential manner. 45
  • CHAPTER 4: DATA COLLECTION AND ANALYSIS Overall Response Analysis The survey was sent out to 23 participants over several installments. Initial responses were analyzed for consistency and clarity. In order to increase the validity and decrease bias survey links were only sent to departments that do not fall under direct supervision of the researcher. In addition survey participation was proposed under false pretenses. The participants of the study were asked to fill out a survey in the context of their job function with an express purpose stated as evaluating and improving experimentation efforts undertaken by the company. All participants were asked to participate on a voluntary basis with the strict assurances of confidence. General survey settings were as follows: (a) allow only one response per computer; (b) respondents can go back to previous pages in the survey and update existing responses until the survey is finished or until they have exited the survey. After the survey is finished, the respondent will not be able to re-enter the survey; (c) respondents can exit the survey and come back at any time, unless the survey is finished; and (d) do not display a thank you page. Based on the survey statistics, 11 out of the 23 participants completed the survey (see Figure 1) which constitutes a 47.8% overall survey uptake. Under open-field study conditions such a response rate would be considered extremely high especially given the length of the survey and its mixed nature. However, under controlled conditions consistent with an organizational study, such a response rate cannot be considered unusual. Furthermore, the response rate most likely buoyed by the belief that participants were filling out the survey in the context of their work, rather than being participants of the study. Based on the statistical analysis (see Figure 2), the study participants fell into the 46
  • following self-identified groups: (a) Interactive Marketing – either participants or 34.8%; (b) Business Development – two participants or 8.7%; (c) User Experience – seven participants or 30.4%; (d) Creative Design – three participants or 13%; and (e) Data Analysis – three participants or 13%. The self-identified groups were presented as a set of options without the possibility of adding additional groups. The group names were specifically chosen to represent job functions instead of an actual department name. One of the key reasons behind doing so was a desire to tie experimentation responses to the job function of the participants instead of an actual department, based on the fact that actual responsibilities of the participants vary greatly even in the same department and may have led to wrong conclusions during analysis. In addition, the actual job function served as a key determinant in assigning relative weight to the responses of the study participants. In other words, more weight was given to the members of Business Development group when questions were centered on the effects of experimentation on the business development function of the company. The second question of the survey asked the participants to identify their level of expertise with respect to experimentation. Based on the statistical analysis of the responses to the second question, the self-identified experimentation expertises were divided as follows: (a) Novice – eight participants or 34.8%; (b) Intermediate – 13 participants or 56.5%; and (c) Expert – two participants or 8.7%. The experience level data has been cross-tabulated with job function data (see Figure 3). The distribution of participants that identified themselves in the Novice and Intermediate categories was fairly uniform across all of the groups; however, self-identified experts belonged only to the Interactive Marketing and User Experience groups. It is difficult to draw any conclusions based on the distribution of the self-identified experts. It could most likely be explained by the scarcity of the collected data. In a much larger survey it is expected that the distribution of the self-identified experimentation experts would be uniform as well. 47
  • Research Objective One One of the main research questions was designed to ascertain the impact of experimentation on the interactive marketing goals. It was represented by the following qualitative and quantitative survey questions: (a) Please Rate the Impact of Experimentation on the Overall Interactive Marketing Goals (Cost Per Acquisition, Cost Per Impression, Cost Per Action, Upsell, and Click-Through Rate); (b) Please Describe in Your Own Words the Impact of Experimentation on the Overall Interactive Marketing Goals (Cost Per Acquisition, Cost Per Impression, Cost Per Action, Upsell, and Click-Through Rate); (c) Please Rate The Impact of Experimentation on the Cost Per Acquisition (CPA); (d) Please Describe in Your Own Words How Experimentation Has Reflected upon the Cost Per Acquisition (CPA); (e) Please Rate The Impact of Experimentation on the Cost Per Impression (CPI); (f) Please Describe in Your Own Words How Experimentation Has Reflected upon the Cost Per Impression (CPI); (g) Please Rate The Impact of Experimentation on the Cost Per Action (CPA); (h) Please Describe in Your Own Words How Experimentation Has Reflected upon the Cost Per Action (CPA); (j) Please Rate The Impact of Experimentation on the Upsell; (k) Please Describe in Your Own Words How Experimentation Has Reflected upon the Upsell; (l) Please Rate The Impact of Experimentation on the Click-Through Rate (CTR); and (m) Please Describe in Your Own Words How Experimentation Has Reflected upon the Click-Through Rate (CTR). Horizontalization The first step of qualitative data analysis consisted of the technique of horizontalization. Horizontalization consisted of grouping like-intentioned statements, underlining significant statements and checking for consistency between the qualitative and quantitative response. According to the statistical analysis of the quantitative question number three (see Figure 4), 13 48
  • or 92.9% of participants that had completed this question found that experimentation had a positive impact on their overall interactive marketing goals. Only one participant or 7.1%, determined that the impact of the experimentation had been neutral. Based on the analysis of the qualitative data (see Figure 37) it is clear that the qualitative responses were consistent with the quantitative response breakdown. In order to gain additional insights into the data the researcher produced two cross-tabulation tables. The first cross-tabulation analyzed participant responses by job function (see Figure 5), whereas the other looked at the participant experience in relation to given responses (see Figure 6). The majority of the survey participants that indicated a positive impact of the experimentation on interactive marketing self-identified as having a user experience function. The only participant that found the impact of experimentation on interactive marketing goals was neutral came from an interactive marketing group. In the same vein the majority of the survey participants that found the experimentation impact positive described their experimentation experience as intermediate. The two self-identified experts were split between positive and neutral impact. In order to gain more insight into the responses of the two self- identified experts, corresponding qualitative data was also analyzed (see Figure 36). In the qualitative portion of the answer the self-identified expert had indicated that, “Our initiation experimentation efforts had mixed results. However, as experimentation practices matured the overall impact was consistently positive”. As such it is possible to speculate that the other self- identified expert was referring to early stage efforts that could not have been characterized as successful. This supposition is further confirmed by statements contributed by self-identified intermediate experimentation users. On several occasions survey participants mentioned that experimentation efforts yielded mixed results, with overall result being positive, “Initial experimentation efforts had neutral impact on the business overall. After experimentation has been coupled with the process improvements it had a positive impact on the Interactive 49
  • Marketing in general and listed goals in particular”. Even though the majority of participants felt that experimentation had a positive impact on the marketing goals as a whole, when each particular marketing goal was itemized some participant answers exhibited inconsistency. More specifically several participants answered that experimentation had a positive impact on the Cost Per Action (CPA), one of the goals of interactive marketing. However, when asked to describe this positive effect in their-own words answered, “Not sure.” This discrepancy could be explained by the fact that these participants are not part of the group that keeps track of individual interactive marketing metrics (see Figure 13), (see Figure 14), (see Figure 15), (see Figure 16), (see Figure 17). In contrast with these responses, study participants that interacted with the marketing data on a daily basis were able to articulate in detail the impact of experimentation on individual interactive marketing goals, “Experimenting with the bidding strategy had a positive impact on the CPA”, and “CPA was reduced though experimentation with webpage content and execution workflow.” The complete extraction of the answers related to research question one was grouped and analyzed for significant statements (see Figure 37). Clusters of Meaning The second step of qualitative data analysis consisted of clustering. The process of extracting clusters of meaning consisted of grouping significant statements identified in the horizontalization step. The responses grouped under common labels were examined for common clusters of meaning. In some cases quotes were attributed to multiple emerging themes. At the same time some of the quotes contained several themes or meanings at once. The quantitative survey questions pertaining to individual marketing goals were cross tabulated with an assessment of overall experimentation impact. Based on the analysis of the cross-tabulations, the majority of the survey participants displayed consistency in responses. The participants of the study felt that experimentation had an equally consistent impact on the interactive marketing 50
  • goals, as a whole, as well as the individual interactive marketing goals in particular. In other words, if a participant felt that experimentation had a positive overall impact, he/she felt the same way when the interactive marketing goals were itemized. However, in several cases responses were inconsistent, where two participants felt that the overall impact was positive, but were not sure about the impact of experimentation on individual interactive marketing goals. This could have been considered an anomaly; however, after looking at several responses by the same participants of the study it became clear that “not sure” response was given to all itemized individual interactive marketing goals as opposed to some. The most likely explanation for the observed anomaly is due to the fact that participants of the study that gave inconsistent responses did not deal with individual interactive marketing goals directly, but rather were only familiar with it in a combined fashion. The analysis of the qualitative and quantitative data revealed several emerging themes: (a) the overall impact of experimentation on interactive marketing goals was positive; (b) initial experimentation efforts were less than successful until the company found a way to couple experimentation with a mature process; (c) the impact of experimentation was viewed in an amortized fashion, where individual marketing was not heavily emphasized. Textual Description The third step of qualitative data analysis involved textual description. Textual description consisted of the amalgamation and summation of participant responses into a cohesive narrative. In summary, the majority of the surveyed participants felt that experimentation had a positive impact on their interactive marketing goals as a whole. However, early efforts related to experimentation yielded mixed results due to the lack of an organizational process and strong experimentation experience. When an experimentation process was fully established its impact remained consistently positive. Assessment of the experimentation impact 51
  • on individual interactive marketing goals was largely consistent with the assessment of experimentation as a whole. The majority of participants felt that individual marketing goals such as Cost Per Action, Cost Per Impression, Cost Per Acquisition, Upsell and Click Through Rate had individually benefited from the experimentation effort. Composite Description The composite description step of the data analysis related to phenomenological research methodology was combined with the textual description step. One of the key reasons for doing so was the fact that the qualitative responses were short and as such, represented a concise summary that required no further reduction. This is often the case with mixed-method research instrumentation, where the qualitative portion is filled out by the participants of the study rather than the researcher. The composite description section of the data analysis was used to discuss issues related to the analysis of a particular research question. In some cases participants of the study confounded several issues related to experimentation. It is expected that some experiments yield negative results. These do not however have a negative connotation in the sense of being bad or poor. The nature of experimentation is such that some experiments result in conversion improvements, whereas others result in conversion decreases. Hence, it is important to talk about the impact of experimentation as a process rather than the impact of the individual experiment. It is however expected that the experimentation process should result in a positive outcome as a whole; otherwise it would be considered unsuccessful. 52
  • Research Objective Two Another key research question was designed to identify key experimentation levers that impact upon the interactive marketing goals. The objective was represented by the following qualitative and quantitative survey questions: (a) Please Select Experimentation Lever Types That You Feel Are Most Instrumental In Achieving Interactive Marketing Goals (Cost Per Acquisition, Cost Per Impression, Cost Per Action, Upsell, and Click-Through Rate); (b) Please Describe in Your Own Words What Experimentation Levers Were Most Useful In Achieving Interactive Marketing Goals (Cost Per Acquisition, Cost Per Impression, Cost Per Action, Upsell, and Click-Through Rate); (c) Please Select Experimentation Conditions That You Feel Had the Strongest Impact upon Interactive Marketing Goals (Cost Per Acquisition, Cost Per Impression, Cost Per Action, Upsell, and Click-Through Rate); and (d) Please Describe in Your Own Words What Experimentation Conditions Were Most Impactful upon Interactive Marketing Goals (Cost Per Acquisition, Cost Per Impression, Cost Per Action, Upsell, and Click-Through Rate). Horizontalization According to the statistical analysis of the quantitative question five (see Figure 7), 11 participants or 78.6% who completed this question found that visual experimentation levers had the strongest impact on their overall interactive marketing goals. Similarly, seven participants or 50% felt the same way about the functional experimentation levels. Lastly, three participants or 21.4% and four participants or 28.6% participants felt that behavioral and informational experimentation levers respectively had the stronger impact on the interactive marketing goals. With regards to external conditions related to experimentation, 11 participants or 71.4% felt that seasonal variations had the strongest impact on the interactive marketing goals. The feelings related to Temporal, Demographical, and Contextual external conditions were almost evenly 53
  • split, with a slight preference given to experimentation taking into account contextual conditions (see Figure 10). Based on the analysis of the qualitative data (see Figure 38) it is clear that the qualitative responses were consistent with the quantitative response breakdown. In order to gain additional insights into the data, the researcher produced two cross-tabulation tables. The first cross- tabulation analyzed participants’ responses by job function (see Figure 8), where the other looked at the participants’ experience in relation to given responses (see Figure 9). It was found that 100% of the survey participants from Creative Design, Data Analysis and User Experience agreed that Visual experimentation levels had the strongest impact on the interactive marketing goals. In addition 50% of participants from the Interactive Marketing groups identified Visual experiments levels as the most impactful. In contrast, only survey participants that identified Informational experimentation levels as most impactful belonged to Interactive Marketing and User Experience groups. This result seems consistent with the fact that Informational levels are underrepresented as the percentage of the overall experiments performed by the company. The fact that survey participants from the other groups did not identify the Information experimentation level as impactful could be attributed to their lack of awareness that such levers even exist. One of the key insights gained from the analysis of the cross-tabulation of the participant experience levels and their thoughts related to the most impactful experimentation levers was the fact that survey participants with more significant experimentation experience focused on fewer experimentation levers. In other words self-identified novice participants identified all levers as equally impactful, self-identified intermediate participants focused more on the Visual and Functional levers, and self-identified experts chose Visual and Functional experimentation levers exclusively. 54
  • Additional cross-tabulations analyzed participant responses related to experimentation by job function and experience on the basis of external conditions (see Figure 11), (see Figure 12). Members of all groups identified experimentation with seasonal variations as most impactful with the exception of the Interactive marketing group members. In contrast to the experimentation experience and the cross-tabulation of the external experimental conditions, statistical analyses exhibited an inverse behavior. 100% of self-identified novice participants uniformly felt that seasonality had a strong impact. This opinion held largely true with self- identified intermediate participants, accounting for more than 70% of responses. There was split opinion among Self-identified experts whether seasonality experimentation had the strongest impact. The complete extraction of the questions, related to research question two, was grouped and analyzed for significant statements (see Figure 38). Clusters of Meaning Based on the results of the horizontalized data it was evident that an overwhelming majority of the participants felt that experimentation with visual elements yielded the most impactful results on their interactive marketing goals. The following statement clearly expresses a prevailing opinion, “Visual experimentation more than any other factors contributed to the overall marketing goals.” In several instances visual experimentation was mentioned in conjunction with functional levers, “Experimentation with the visual and functional elements of our customer facing solutions had the most measurable effect on the interactive marketing KPIs.” It is important to note that several participants indicated that experimentation with visual elements was not only most impactful, but was the element that the company experimented with the most, “More than 80% of the experiments have been conducted using visual elements. It is not surprising that visual experimentation had the greatest impact on the listed marketing goals.” 55
  • In addition several participants of the study chose to identify all experimentation levers indicating the importance of all factors, “I believe that all the lever types mentioned above are necessary in a successful experimentation practice…” In the same vain participants’ answers related to experimentation in the context of external conditions were heavily clustered in favor of seasonal factors (see Figure 39), “Our business is highly seasonal. In addition achieving marketing goals is very dependent on the demographics of the users. Experimenting in the context of user demographics and seasonal changes had a significant impact on the listed marketing goals.” The most significant statement is the following response which ties the results of the analysis to the company specifics, “The economics of the company change dramatically on the micro and macro temporal bases. Taking these internal factors into consideration during experimentation proved to be successful.” Textual Description Analysis of the clusters of meaning resulted in an amalgamated textual description that had several prevailing themes and conclusions. The participants of the study felt that experimentation with visual experimentation levers had a very strong positive impact on the interactive marketing goals of the company. A significant number of participants also felt that in addition to functional experimentation elements, functional levers had a strong positive impact as well. One of the prevailing themes clearly identifiably in many survey responses had to do with fact that the company chose to emphasize visual experimentation elements over other possible options. At the same time several participants of the study emphasized that all experimentation levers yielded positive results and therefore all need to be considered as part of the experimentation process. With regards to experimentation with external factors, the opinions of the participants were similarly strongly centered on seasonal factors. Several responses highlighted the seasonal 56
  • nature of the business. As such, taking seasonality into account had a strong positive impact on the experimentation results. Furthermore several participants looked at seasonality in a macro context, where seasonality implied changes in the business patterns from month to month and even from quarter to quarter. However, several participants also emphasized experimentation in the context of the micro temporal level, where experimentation accounted for changing traffic patterns throughout the day or several days. Composite Description Similar to research question one, responses given by the participants of the study in the qualitative portion of research question two were short enough not to require textual description. Therefore textual description has been combined with composite description and this section showed possible anomalies and points of focus. One such point was related to causality. It was not clear if the company chose to perform the majority of the experiments with visual elements because they yielded most significant results, or experimentation with visual elements yielded superior results because the company chose to concentrate on them. There are strong arguments that could be made in support of both points. Visual elements are the easiest to experiment with and they are most abundant. At the same time, the largest improvements in the interactive marketing metrics were related to experimentation with visual elements. It is quite likely that both statements are true giving a bi-directional causality. 57
  • Research Objective Three Another key research question was designed to ascertain the impact of experimentation on product innovation. It was represented by the following qualitative and quantitative survey questions: (a) Please Rate the Impact of Experimentation on Interactive Marketing Product Innovation; and (b) Please Describe in Your Own Words How Experimentation Has Reflected upon Interactive Marketing Product Innovation. Horizontalization Based on the statistical analysis of the quantitative results related to the impact of experimentation on product innovation nine participants or 81.8% felt that experimentation had a positive impact on interactive marketing product innovation (see Figure 21). A total of two or 18.2% participants indicated that they were not sure how to answer this question. These results are strongly supported by qualitative data contributed by the participants (see Figure 41). Even though the analysis of the interactive marketing product lifecycle did not directly address product innovation while answering questions related to product lifecycle, participants of the study offered several insights that were very helpful in understanding experiment driven product innovation (see Figure 19), (see Figure 20). Eight participants or 72.7% felt that experimentation has a positive impact on product lifecycle, whereas three participants or 27.3% were not sure how to answer this question (see Figure 18). In order to gain further insights into the qualitative and quantitative answers given by the participants, several cross-tabulations were produced (see Figure 22), (see Figure 23). The first cross-tabulation correlated the participants’ views on the impact of experimentation on product innovation with their job function. The majority of the participants of the study that indicated that experimentation had a positive impact on product invocation were either directly involved in product development or had a tangential job function. Two participants that were not sure about 58
  • the impact of experimentation on product innovation belonged to the Interactive Marketing and Data Analysis groups. The second cross-tabulation correlated participants’ views on the impact of experimentation on product innovation with their experimentation experience. Two participants of the study who were not sure about the impact of experimentation on product innovation self- assessed their experience as intermediate and novice respectively. On a percentage basis, 100% of the self-identified expert participants, 87.5% of the self-identified intermediate participants and 50% of novice participants assessed the impact of the experimentation on product innovation as positive. The complete extraction of the questions, related to research question three, were grouped and analyzed for significant statements (see Figure 41). Clusters of Meaning The following answer expressed the prevailing opinion of the survey participants, “Experimentation affected product innovation in a positive manner. Several new products emerged as a result.” Some qualitative statements were more specific in identifying the exact impact of experimentation on product innovation, “Customer feedback collected through experimentation yielded several innovative ideas.” Those participants that were not sure about the impact of the experimentation on product innovation gave the following qualitative answers, “I am not sure how to answer this question” and “No specific data was collected.” The answers to the product innovation questions were cross-referenced with the questions related to the impact of experimentation on product lifecycle. Several phases of the product development lifecycle are closely related to product innovation. In particular, idea incubation is almost directly synonymous with product innovation (see Figure 40). We find support for this supposition in one answer regarding product development lifecycle, “Product development 59
  • experimentation impacted on various phases of product lifecycle, especially in the early stages of product development.” In addition to the insights gained on product innovation, we find that the same survey participants gave identical answers, “I am not sure how to answer this question” and “No specific data was collected” to the questions related to product lifecycle. Textual Description Based on the analysis of the clustered data, a compound description emerged pointing to the fact that the majority of survey participants felt that experimentation had a positive impact on product innovation. More specifically, study participants stated that interactive marketing experimentation helped product innovation by allowing the company to churn through several product ideas on an expedited basis. In addition it allowed the company to gather customer feedback before the product was fully crafted and the company had committed to the full product development. Furthermore, customer feedback either directly or tangentially pointed the company to several innovative ideas that were either not actively discussed or had previously not been considered as promising by the decision makers of the company. As such these ideas came to fruition in a true customer-driven fashion. In general the company observed a shift from opinion-driven product development to a customer-experimentation model, where opinions of the customers were given a greater weight than previously. Composite Description It is important to note that product related questions are strongly correlated with the job function of the participants. In other words participants of the study that did not focus on product development or did not perform job duties that were at least tangential to product development may have not been aware of the impact of the experimentation on product development or might have given contradictory answers. 60
  • Research Objective Four Another research question related to product development was designed to ascertain the impact of experimentation on product improvement. It was represented by the following qualitative and quantitative survey questions: (a); Please Rate The Impact of Experimentation on Interactive Marketing Product Improvement; and (b) Please Describe in Your Own Words How Experimentation Has Reflected upon Interactive Marketing Product Improvement. Horizontalization Based on the statistical analysis of the quantitative results related to the impact of experimentation on product improvement nine participants or 81.8% felt that experimentation had a positive impact on interactive marketing product innovation (see Figure 24). Only two participants or 18.2% indicated that they were not sure how to answer this question. These results are strongly supported by qualitative data contributed by the participants (see Figure 42). In order to gain further insights into the qualitative and quantitative answers given by the participants, several cross-tabulations were produced (see Figure 25), (see Figure 26). The first cross-tabulation correlated participants’ views on the impact of experimentation on product improvement with their job function. The majority of participants of the study that indicated that experimentation had a positive impact on product improvement were either directly involved in product development or had a tangential job function. Two participants that were not sure about the impact of experimentation on product improvement belonged to the Interactive Marketing and Data Analysis groups. The complete extraction of the questions, related to research question four, was grouped and analyzed for significant statements (see Figure 42). Clusters of Meaning The following answer highlighted a prevailing opinion of the survey participants, 61
  • “Experimentation affected product improvement in a positive manner. It manifested itself in the increased pace of product improvement.” Some qualitative statements were more specific in identifying the exact impact of experimentation on product improvement, “Customer feedback collected through experimentation allowed for continuous product improvement.” Those participants of the study that were not sure about the impact of experimentation on product improvement gave the following qualitative answers, “I am not sure how to answer this question” and “No specific data was collected.” Textual Description Based on the analysis of the clustered data a compound description emerged pointing to the fact that the majority of survey participants felt that experimentation had a positive impact on product improvement. More specifically study participants stated that interactive marketing experimentation helped product innovation by allowing the company to churn through several feature ideas on an expedited basis. In addition experimentation allowed the company to gather customer feedback on the newly planned features. Furthermore, customer feedback either directly or tangentially pointed the company to several innovative feature ideas. Composite Description It is important to note that the survey participants gave almost identical answers to the questions related to product innovation and product improvement. In fact some answers were seemingly cut and pasted from the product innovation section. It is possible that participants of the survey combined product innovation and product improvement concepts into one. 62
  • Research Objective Five The remaining research question related to product development was designed to ascertain the impact of experimentation on product deployment risk. It was represented by the following qualitative and quantitative survey questions: (a) Please Rate the Impact of Experimentation on Interactive Marketing Product Deployment Risk; and (b) Please Describe in Your Own Words How Experimentation Has Reflected upon Interactive Marketing Product Deployment Risk. Horizontalization Based on the statistical analysis of the quantitative results related to the impact of experimentation on product improvement, eight participants or 72.7% felt that experimentation had a positive impact on interactive marketing product innovation (see Figure 27). Only two participants or 18.2% indicated that they were not sure how to answer this question and one participant or 9.1% felt that the impact of experimentation on product deployment risk was neutral. These results are strongly supported by qualitative data contributed by the participants (see Figure 43). In order to gain further insights into the qualitative and quantitative answers given by the participants, several cross-tabulations were produced (see Figure 28), (see Figure 29). The first cross-tabulation correlated participants’ views on the impact of experimentation on product deployment risk with their job function. The majority of study participants that indicated experimentation had a positive impact on product deployment were either directly involved in product development or had a tangential job function. Two participants that were not sure about the impact of experimentation on product deployment risk belonged to Interactive Marketing and Data Analysis groups. One of the participants that indicated that the impact of the experimentation on the product deployment risk was neutral belonged to a User Experience 63
  • group. The second cross-tabulation cross-referenced participants’ answers with their self- assessed experimentation experience. The participants that felt that experimentation had a positive impact on product deployment risk were either self-assessed experts or intermediate experimentation users. The participants of the study that indicated that they were not aware of the specific impact or were not sure how to answer this question self-identified as either novice or intermediate experimentation users (see Figure 29). The complete extraction of the questions, related to research question five, was grouped and analyzed for significant statements (see Figure 43). Clusters of Meaning The following answer expressed a prevailing opinion of the survey participants, “Experimentation practice has reduced product deployment risk by allowing limited product deployment.” Some qualitative statements were more specific in identifying the exact impact of experimentation on product deployment risk, “Experimentation reduced product deployment risk by allowing easy rollback in the case of errors.” Those participants that were not sure about the impact of the experimentation on product deployment risk gave the following qualitative answers, “I am not sure how to answer this question” and “No specific data was collected.” A participant that indicated in the quantitative portion of the survey that he/she felt that impact of the experimentation on product deployment had been neutral, gave the following answer in the qualitatively paired question, “I am not aware of any product deployment risks that where mitigated.” Textual Description Based on analysis of the clustered data, compound description emerged as significant pointing to the fact that the majority of survey participants felt that experimentation had a 64
  • positive impact on product deployment risk. More specifically study participants stated that interactive marketing experimentation helped mitigate product deployment risk by allowing the company to make a partial commitment to product deployment. It allowed the company to assert a greater control over the product deployment cycle. Only partial traffic was directed to the newly deployed products. Full traffic flow was only turned on when the company felt that new product did not adversely affect key performance indicators of the company such CTR, CPA, CPI, Upsell Score and so on. If an adverse effect was discovered the company had an easy route to either further reduce the flow of traffic to the newly deployed product in order to observe it in a more controlled environment or reduce web traffic to zero, a decision that is tantamount to a product rollback. Since the control of traffic is virtually instantaneous and the amount of potential loss could be strictly controlled companies are more willing to allow investigative analysis while products are still only exposed to a limited number of users. Composite Description Similar to the other research questions, the responses given by the study participants, the qualitative portion of the research question five were short enough not to require textual description. Therefore textual description has been combined with composite description. This section highlighted possible anomalies and points of focus. It is important to note that research questions three, four and five induced almost identical responses from the survey participants. The answers to these questions were consistent among study participants. In other words, if a participant felt that experimentation had a positive impact on product improvement he/she felt the same way about product innovation and product deployment risk. However, if the participant was not sure what impact experimentation had on product innovation, he/she was also unsure about its impact on product improvement and product deployment. As such it is possible to conclude that the answers to the qualitative and quantitative portions of the survey related to the 65
  • impact of various product aspects were strongly correlated to knowledge of the interactive marketing product itself. Research Objective Six The remaining research question was designed to ascertain the impact of experimentation on the competitive standing of the company. It was represented by the following qualitative and quantitative survey questions: (a) Please Describe in Your Own Words How Experimentation Has Reflected upon Competitive Standing of the Company; (b) Please Describe in Your Own Words How Experimentation Has Reflected upon Competitive Standing of the Company; (c) Please Rate The Impact of Experimentation on Revenue of the Company; (d) Please Describe in Your Own Words How Experimentation Has Reflected upon Revenue of the Company; (e) Please Rate The Impact of Experimentation on Profit of the Company; (f) Please Describe in Your Own Words How Experimentation Has Reflected upon Profit of the Company; (g) Please Rate The Impact of Experimentation on Market Size of the Company; (h) Please Describe in Your Own Words How Experimentation Has Reflected upon Market Size of the Company; (j) Please Rate The Impact of Experimentation on Market Penetration of the Company; (k) Please Describe in Your Own Words How Experimentation Has Reflected upon Market Penetration of the Company; (l) Please Rate The Impact of Experimentation on Market Segmentation of the Company; and (m) Please Describe in Your Own Words How Experimentation Has Reflected upon Market Segmentation of the Company. Horizontalization According to the statistical analysis of the quantitative question 29 of the survey (see Figure 30), five participants or 45.5% that had completed this question found that experimentation had a positive impact on the competitive standing of the company. 36.6% of the 66
  • survey participants indicated that they were not sure how to answer this question. Lastly, two participants or 18.2% felt that experimentation had neutral impact on the competitive standing of the company. Other questions related to competitive standing of the company that touched upon market penetration, market segmentation, market targeting and market size in the context of experimentation, exhibited a similar percentage breakdown, with the notable exceptions of two questions that inquired about the impact of experimentation on the revenue and profit of the company. 100% of the participants of the survey felt that experimentation positively impacted upon the revenue and profit of the company (see Figure 44). In order to gain further insights into qualitative and quantitative answers given by the participants, several cross-tabulations were produced (see Figure 31), (see Figure 32). The first cross-tabulation correlated the participants’ views on the impact of experimentation on the competitive standing of the company with their job function. Opinions of the participants of the survey were uniformly distributed across all job functions represented in the survey. The second cross-tabulation cross-referenced participants’ answers with their self- assessed experimentation experience (see Figure 32). Similarly, the opinions of the self- identified experts, intermediate and novice users were uniformly distributed and ranged from positive, to neutral and not sure. The complete extraction of the questions, related to research question five, was grouped and analyzed for significant statements (see Figure 44). Clusters of Meaning The following answers expressed the prevailing opinion of the survey participants in each category, “Experimentation made the company more competitive by remaining relevant to our customers“, “I am not sure how to answer this question” and “Competitive impact is difficult to ascertain.” Based on the divergent statements above, it was clear that study participants were 67
  • divided on the impact on the experimentation on the competitive standing on the company. A similar divergence of opinion was observed during the composite analysis of the question related to the competitive standing of the company that touched upon market penetration, market segmentation, market targeting and market size in the context of experimentation, even though a slightly higher percentage of participants had a positive opinion about the impact of the experimentation on these components of the competitive standing of the company. A drastically different picture emerged during clustered analysis of the impact of experimentation on the revenue and profits of the company. The following statements express the prevailing opinions, “Experimentation had a direct impact on the bottom line of the company” and “Experimentation allowed the company to expand our customer base, resulting in higher revenues.” It was clear that study participants did not consider revenue and company profits as part of the competitive standing of the company. Textual Description Based on the analysis of the clustered data it was clear that study participants did not provide a uniform opinion related to impact of the experimentation on competitive standing of the company. While examining the impact of the various elements that make up competitive standing of the company their opinion remained mixed. At the same time study participants were unhesitant in expressing their strong opinion in support of experimentation as one of the key contributors to the profits and revenue of the company. It is also clear that some participants had a difficulty interpreting the exact nature of the questions related to competitive standing of the company. Analysis of the qualitative data revealed the highest percentage of participants were either unsure how to answer this question or gave the shortest answer possible. In contrast, the revenue and profit questions yielded expressive answers and indicated a strong degree and confidence of conviction in the given answers. 68
  • Composite Description One of the biggest anomalies and the point of focus of this research question is the fact that study participants felt drastically different about the impact of experimentation on the overall competitive standing of the company and the impact of experimentation on the profit and revenue of the company. It is highly likely that study participants did not consider revenue and profit to be part of the competitive standing of the company. Validity and Bias In order to insure validity and eliminate bias study participants were invited to complete the survey under false pretenses by the COO of the company where the research is conducted and the researcher employed. The survey was presented to the participants as the means to ascertain the effectiveness of Experimentation in Interactive Marketing in the context of their work. Since all of the study participants work in the same organization as the researcher of the study, deception was absolutely vital to avoid credibility and bias issues. Even if any possible undue influence of the researcher could be somehow mitigated, knowledge of the true nature of the study could affect the results by any attempts of the participants to be helpful to the researcher and thus contributing erroneous data that they would feel would advance the aims of the research. Since the researcher of the study holds an executive position at the company even visibility of impropriety was too great for the true nature of the study to be revealed. If participants of the study became aware of the true intentions of the research they would potentially try to skew their answers in the way that they would perceive to be helpful to the researcher, thus affecting the credibility of the study. Since the study employed a phenomenological approach, it was essential for the researcher to mitigate his personal bias on the subject while interpreting the results of the study. In order to do that the researcher utilized a Bracketing approach. Namely prior to the analysis of 69
  • the survey results the researcher expressed his own thoughts regarding each of the research questions. While analyzing the results, the researcher compared his opinions with the opinions of the participants to make sure that they did not become misinterpreted. The full set of results is presented in the next chapter. The issues of credibility, validity and bias are further explored in the next chapter as well. 70
  • CHAPTER 5: RESULTS, CONCLUSIONS, AND RECOMMENDATIONS The last chapter presented a detailed analysis of the collected data. The issues of credibility, validity and bias were specifically outlined and discussed. This chapter centers on presenting the study results, outlining potential gaps, as well as providing recommendations for future research. The results of the study were summarized as a cohesive narrative that allowed the final conclusions of the study to be formulated. In addition, the study results and conclusions were examined in the light of existing research. Results One of the key aims of the study was to examine interactive marketing through the prism of experimentation as a way of propelling interactive marketing forward and enabling it to keep pace with technological advances. The following researched questions were examined and analyzed: (a) What is the impact of experimentation on interactive marketing goals? (b) What are the key experimentation levers pertaining to interactive marketing? (c) What is the impact of experimentation on product innovation in interactive marketing? (d) What is the impact of experimentation on interactive marketing product improvement? (e) What is the impact of experimentation on interactive marketing product development and deployment risk? (f) What is the impact of experimentation on the competitive standing of the companies involved in interactive marketing? In addition to these research questions the study proposed the following research null hypotheses: (a) experimentation has a positive impact on interactive marketing goals; (b) experimentation has a positive impact on product innovation in interactive marketing; (c) experimentation has a positive impact on product improvement in interactive marketing; (d) 71
  • experimentation has a mitigating impact on product development and deployment risk; and (e) experimentation has a positive impact on the competitive standing of the companies involved in interactive marketing. In order to confirm or reject these null hypotheses, the study comprised a mixed-method survey where participants of the study were asked to reflect on various aspects of experimentation in interactive marketing in the context of their job. The study was conducted in the organization where the researcher is employed. The participants of the study were invited to participate in the study under false pretenses by the Chief Operating Officer (COO) of the company. Only the researcher of the study and the COO were aware of the true intentions of the survey. Keeping the true intentions of the study a secret was a key to increasing validity, credibility and reducing bias. The study was conducted in a double blind manner in order to preserve confidentially and insure impartiality. In other words, the study participants were unaware of the true reasons for the survey and the researcher was unaware of the identities of the survey participants. The survey was sent out in two installments to 23 participants. Based on the application of the phenomenological methodology consisting of horizontalization, clustering, textual description and composite description, the analysis of the of the received data yielded the following results; (a) the null hypothesis asserting that experimentation has a positive impact on interactive marketing goals was strongly confirmed. The 93.3% f the participants indicated that experimentation had a positive impact on interactive marketing goals. This result was consistent with the conclusions reached by Raman (1996), where user-tailored interactive advertising had a high conversion outcome; (b) the null hypothesis asserting that experimentation has a positive impact on product innovation in interactive marketing was confirmed as well. The 81.8% of the 72
  • participants felt that experimentation strongly contributed to product innovation. The results of this particular research question were consistent with the conclusions outlined by Von Hippel (1998), which concluded that product innovation was successfully driven by product users; (c) the null hypothesis asserting that experimentation had a positive impact on interactive marketing product improvement was also strongly confirmed. 81.8% of the participants indicated that experimentation had a positive and lasting impact on product improvement. Similar conclusions were reached by Thomke (1995) who found that experimentation significantly contributed to product improvement; (d) the hypothesis asserting that experimentation had a positive impact on product development and product deployment risk was similarly confirmed with 72.7% of study participants indicating that experimentation had mitigated product development and deployment risk. Both Von Hippel (1998) and Thomke (1995) reached conclusions in support of these findings. In particular, their studies asserted that the economics of product development were significantly affected by experimentation practice. Experimentation resulted in reduced product cost and consequently reduced product development and deployment risk; (e) the hypothesis asserting that experimentation has a positive impact on the competitive standing of the company was rejected by the participants of the study. Only 45.5% of the survey participants felt that the experimentation impact on the competitive standing of the company had been positive. There are currently no empirical studies relating the competitive standing of the company with experimentation efforts. As such, it is impossible to make inferences related to consistency of the research findings. The only research question, which examined the impact of experimentation on interactive marketing levers, which did not have an associated research hypothesis yielded following results: 80% of the participants felt that visual experimentation levers had the strongest impact on 73
  • interactive marketing goals. This was closely followed by functional levers with 53.3% of the vote. These findings are supported in part by the conclusions from Jebakumari (2002), Mark (2003) and Milley (2002). In particular Milley (2002) concluded that interactive marketing results were dependent on capturing and applying behavioral data. Conclusions The results of the study indicated that experimentation had a profound impact on almost all aspects of the company where the research was conducted. The majority of the research results was supported by the conclusions of previous empirical research studies and was consistent with the reference material. It was clear that the study participants felt that the experimentation efforts of the company were overwhelmingly positive even though non-uniform results were achieved at the different phases of adoption of experimentation. More specifically, some participants felt that the initial experimentation efforts lacked focus and were disorganized. As such, poor experimentation results were not deemed to be a function of the experimentation efforts but rather due to organizational maturity. When the experimentation practice was fully established the experimentation results remained consistently positive. In an overwhelming majority of the cases, the qualitative data was fully confirmed by quantitative responses. The results of the research question relating experimentation efforts with the competitive standing of the study were inconclusive. It appears that participants of the study did not consider the profit and revenue of the company as significant contributors to the competitive standing of the company. More specifically, if the research question that deals with the competitive standing of the company was dropped from the study and replaced by the question that related experimentation to the profit and revenue of the company it would have been overwhelmingly 74
  • confirmed. In fact 100% of the participants felt that experimentation positively impacted upon the profit and revenue of the company. This aspect of experimentation warrants further study and exploration. Another possibly confounding aspect is that it is also possible that study participants experienced “survey fatigue”, and the questions related to the competitive standing, placed at the end of the very long survey did not obtain an accurate or honest response. It can be argued however that the results have a general applicability to the eCommerce and the Interactive Marketing industry at large. If these results are confirmed in a broader study, then experimentation in the context of interactive marketing may serve as a disruptive technology that would drive almost universal adoption and eventual commoditization, which in turn may spur the emergence of an entire experimentation industry with a diverse offering of experimentation services and tools. Recommendations One of the key recommendations is to confirm the results of this qualitative study by conducting a broad quantitative study on the impact of experimentation on interactive marketing goals in a diverse set of interactive marketing companies. The results of the present study indicate that this impact is fairly significant. If these results are confirmed in a broad study involving a significantly diverse number of interactive marketing companies, it is possible that experimentation in interactive marketing might lead to a paradigm shift in interactive marketing practices. It is also recommended that technology departments be included in future studies. There are number of factors that could potentially impact on the technological efforts of the interactive marketing companies. Future studies may include such topics as: exploring the experimentation process in interactive marketing companies; exploring tools used to facilitate 75
  • experimentation in interactive marketing companies; and exploring metrics related to the experimentation efforts of interactive marketing companies. Based on the fact that this study received an inconclusive result regarding the impact of experimentation on the competitive standing of the company it is recommended that this subject is explored in isolation. This researcher believes that such an impact is significant and needs to be explored in more detail. Additional research may be centered on comparing and contrasting interactive marketing companies that utilize experimentation versus those that do not. A specific focus of the study should concentrate on the use of experimentation to achieve competitive advantage. Another topic that requires strong attention is related to closed-loop experimentation. There is strong anecdotal evidence to suggest that close-loop experimentation, that ties together all facets of interactive marketing such as Search Marketing, Display Marketing, Affiliate Marketing and Email Marketing, is more effective than isolated interactive marketing experimentation which has very few interactive marketing components. However, to date no empirical research has been conducted. Lastly, the results of this study point to a strong correlation between a well established experimentation process and the success of the experimentation. Additional study could focus on researching the impact of a well established experimentation process and its contribution to the overall experimentation success. Limitations of the Study One of the key limitations of the study is the fact that it was conducted in a single organization and as such, the quantitative results cannot be considered statistically significant. 76
  • The quantitative questions of the study verified the consistency of the paired qualitative questions. In addition participants of Data Analysis and Business Development groups were sparsely represented. The study used terminology and concepts that were significant and specific to the organization where the study was conducted. Even though such terminology is prevalent in the industry at large it is possible that some aspects are either interpreted differently or are not considered significant in similar companies. Another significant limitation of the study was the fact that Technology groups were excluded from participating in the survey. Since the study researcher was responsible for all of the technology, quality assurance and project management efforts of the company, the risk of bias and potential loss of validity was too great to allow Technology groups to participate. As such, a number of research questions that dealt with the technological aspects of experimentation in interactive marketing had to be excluded. Lastly, the results of the study are based on the interpretations of the researcher. As such, credibility and bias are potential limiting factors. Attempts were made to mitigate validity and bias by using a number of techniques and approaches however; it is impossible to totally exclude the presence of the personal bias. 77
  • REFERENCES Alvesson M. & Sköldberg K. (2000). Reflexive methodology: New vistas for Qualitative Research, London: Sage Publications. Anderson, A. D. (1972). Designs with partial factorial balance. The Annals of Mathematics and Statistics, 43(4), 1333-1341. Becker, C. (1992). Living and relating: An introduction to phenomenology. Thousand Oaks, CA: Sage. Beerenwinkel, N., Pachter, L. and Sturmfels, B. (2007). Epistasis and shapes of fitness landscapes, Statistica Sinica, 17, 1317-1342. Bezjian-Avery, A. M. (1997). Cognitive processing of interactive marketing (Doctoral Dissertation, Northwestern University, 1997) Biskin, B. H. (1980). Multivariate analysis in experimental counseling research. The Counseling Psychologist, 8, 69-72. Chulho, P. (1988). Interactive marketing decision system for highly differentiated product in a growing market, (Doctoral Dissertation, Stanford University, 1988) Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage. Daugherty, T. M. (2001). Consumer leaning and 3-D ecommerce: The effects of sequential exposure of a virtual experience relative to indirect and direct product experience on product knowledge, brand attitude and purchase intention (Doctoral Dissertation, Michigan State University, 2001) Deighton, J. (1996). The future of interactive marketing, Harvard Business Review, 74(6), 151- 162. Donne, D. G. (1996). Enhancements of experiment design, sensitivity, and information content in multidimensional NMR (Doctoral Dissertation, Purdue University, 1996) Dou, W. (1999). Modeling the online sales system: A catastrophe theory approach (Doctoral Dissertation, University of Wisconsin-Milwaukee, 1999) eMarketer Research. (2007). Analyst reports. eMarketer. Retrieved June 27, 2008, from http://www.emarketer.com/ 78
  • Fisher R. A. (1926). The arrangement of field experiments. Journal of the Ministry of Agriculture of Great Britain, 33, 503-513. Fisher, R. A. (1935). The design of experiments. Oliver and Boyd. Giorgi, A. (1985). The phenomenological psychology of learning and the verbal learning tradition. In A. Giorgi, (Ed.), Phenomenology and Psychological Research. Pittsburgh: Duquesne University Press. Giorgi, A. (2006). Difficulties encountered in the application of the Phenomenological Method in the social sciences. Analise Psicologica, 3 (24), 353-361. Hansen, F. R. (1989). A fractional representation approach to closed-loop system identification and experiment design (Doctoral Dissertation, Stanford University, 1989) Huberty, C. J (1986). Questions addressed by multivariate analysis of variance and discriminant analysis. Georgia Educational Researcher, 5, 47-60. Huberty, C.J. & Morris, J.D. (1989). Multivariate analysis versus multiple univariate analysis, Psychological Bulletin, 105(2), 302-308. Husserl, E. (1931). Ideas: General introduction to pure Phenomenology. (Boyce Gibson, W.R. Trans.). London: George Allen and Unwin. (Originally published 1913). Husserl, E. (1983). Ideas pertaining to a pure phenomenology and to a phenomenological philosophy. First book (F. Kersten, trans.). The Hague: M. Nijhoff (Original work published 1913). Jebakumari, G. (2002). The dimensionality of interactivity and its effect on key consumer variables (Doctoral Dissertation, Southern Illinois University, 1987) Jiang, Z. (2004). An investigation of virtual product experience and its effect mechanism (Doctoral Dissertation, University of British Columbia, 2004) Kohavi, R., Henne, M. R. & Sommerfield, D. (2007). Practical guide to controlled experiments on the web: Listen to your customers not to the HiPPO. Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 959– 967 Kwan, T. (1996). Taguchi methods of design of experiment (Masters Dissertation, California State University Dominguez Hills, 1996) Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, 62(3), 279-300. Morgan, G., & Smircich, L. (1980). The case for qualitative research. Academy of Management 79
  • Review, 5, 491-500. Lee, F., Edmondson, A.C., Thomke, S. and Worline, M. (2004). The mixed effects of inconsistency on experimentation in organizations, Organization Science, 15(3), 310. Li, S. (2003). Essays on interactive marketing, (Doctoral Dissertation, Carnegie Mellon University, 2003) Liu, Y. (2002). Interactivity and its implications for consumer behavior, (Doctoral Dissertation, State University of New Jersey, 2002) Macias, W.A. (2000). The effect of interactivity on comprehension and persuasion of interactive marketing (Doctoral Dissertation, University of Texas, 2000) Mark, J. (2002). Examination of internet marketing relative to traditional promotion in the development of web site traffic, (Doctoral Dissertation, Capella University, 2003) Miley, R. (2000). Web-enabled consumer marketing: Research constructs, propositions and managerial implications (Doctoral Dissertation, University of Calgary, 2000) Moustakas, C. (1994). Phenomenological research methods. Thousand Oaks, CA: Sage. Newman, E. (2001). Integrated marketing on the internet: Congruency effect of banner advertisements and websites, (Doctoral Dissertation, Washington State University, 2001) Ozdemiz, M. (2002). Evolutionary computing for feature selection and predictive data mining, (Doctoral Dissertation, Rensselaer Polytechnic Institute, 2002) Raman, N. V. (1996). Determinants of desired exposure to interactive advertizing (Doctoral, Dissertation, University of Texas, 1996) Schwartz, J. (Speaker). (2005). The participation age [Audio Recording]. IT Conversations Legacy Programs. Shane, S.A. (2004). Technological innovation, product development, and entrepreneurship in management science, Management Science, 50(2), 133-144. Sheffrin, S. M. (1983). Rational expectations. Cambridge: Cambridge University Press Thomke, S. (1995). The economics of experimentation in the design of new products and processes (Doctoral Dissertation, Massachusetts Institute of Technology, 1995) Thomke, S. (1998). Managing experimentation in the design of new products, Management Science, 44(6),743. Thomke, S., von Hippel, E. & Franke, R. (1998). Modes of experimentation: An innovation 80
  • process-and competitive-variable, Research Policy, 27(3), 315-332. Thomke, S. (2001). Enlightened experimentation: The new imperative for innovation, Harvard Business Review, 79(2), 66-75. Torre, A. & Rallet, A. (2005). Proximity and localization, Regional Studies, 39(1), pp. 47–59. Van Manen, M. (1990). Researching lived experience: Human science for an action sensitive pedagogy. State University of New York Press, New York. Von Hippel, E. (1998). Economics of product development by users: The impact of “sticky” local information, Management Science, 44(5), 629. Von Hippel, E. & Katz, R. (2002). Shifting innovation to users via toolkits, Management Science, 48(7), 821. Wang, W. (1999). Predictive modeling based on classification and pattern matching methods (Masters Dissertation, Simon Fraser University, 1999) Weng, W. C. (2007). Taguchi’s method for optimizing electromagnetic applications (Doctoral Dissertation, University of Mississippi, 2007) Xu, H. & Wu C.F.J. (2000). Generalized minimum aberration for asymmetrical fractional factorial designs, The Annals of Statistics, 29(2), 549-560. 81
  • APPENDIX A. SURVEY TEMPLATE Survey Page 1 82
  • 83
  • Survey Page 2 84
  • Survey Page 3 85
  • 86
  • Survey Page 4 87
  • APPENDIX B. CONCEPTUAL FRAMEWORK 88
  • APPENDIX C. INTERACTIVE MARKETING 89
  • APPENDIX D. MARKETING 90
  • APPENDIX E. EXPERIMENTATION CYCLE 91
  • APPENDIX F. DATA ANALYSIS Figure F1. Response summary Figure F2. Job function breakdown 92
  • Figure F3. Job function and experimentation experience cross-tabulation Figure F4. Experimentation impact breakdown 93
  • Figure F5. Experimentation impact and department cross-tabulation Figure F6. Experimentation impact and experimentation experience cross-tabulation 94
  • Figure F7. Experimentation levers impact breakdown Figure F8. Experimentation levers impact and experimentation experience cross-tabulation 95
  • Figure F9. Experimentation levers impact and job function cross-tabulation Figure F10. Experimentation conditions impact breakdown 96
  • Figure F11. Experimentation conditions impact and experimentation experience cross-tabulation Figure F12. Experimentation conditions impact and job function cross-tabulation 97
  • Figure F13. Experimentation CPA impact and experimentation overall impact cross-tabulation Figure F14. Experimentation CPI impact and experimentation overall impact cross-tabulation 98
  • Figure F15. Experimentation CPA impact and experimentation overall impact cross-tabulation Figure F16. Experimentation upsell impact and experimentation overall impact cross-tabulation 99
  • Figure F17. Experimentation CPI impact and experimentation overall impact cross-tabulation Figure F18. Experimentation product lifecycle impact breakdown 100
  • Figure F19. Experimentation impact on product lifecycle and job function cross-tabulation Figure F20. Experimentation impact on product lifecycle and experimentation experience cross- tabulation 101
  • Figure F21. Experimentation product innovation impact breakdown Figure F22. Experimentation product innovation impact and experimentation experience cross- tabulation 102
  • Figure F23. Experimentation product innovation impact and job function cross-tabulation Figure F24. Experimentation product improvement impact breakdown 103
  • Figure F25. Experimentation product improvement impact and job function cross-tabulation Figure F26. Experimentation product improvement impact and experimentation experience cross-tabulation 104
  • Figure F27. Experimentation product deployment risk impact breakdown Figure F28. Experimentation product deployment risk impact and experimentation experience cross-tabulation 105
  • Figure F29. Experimentation product deployment risk and job function breakdown Figure F30. Experimentation competitive standing of the company impact breakdown 106
  • Figure F31. Experimentation competitive standing of the company impact and job function cross-tabulation Figure F32. Experimentation competitive standing of the company impact and experimentation experience cross-tabulation 107
  • Figure F33. Experimentation revenue impact Figure F34. Experimentation profit impact Figure F35. Experimentation market size impact 108
  • Figure F36. Experimentation market penetration impact 109
  • Figure F37. Experimentation marketing goals impact horizontalization 110
  • Figure F38. Experimentation marketing levers impact horizontalization 111
  • Figure F39. Experimentation conditions impact horizontalization 112
  • Figure F40. Experimentation product lifecycle impact horizontalization 113
  • Figure F41. Experimentation product innovation impact horizontalization 114
  • Figure F42. Experimentation product improvement impact horizontalization 115
  • Figure F43. Experimentation product deployment risk impact horizontalization 116
  • Figure F44. Experimentation competitive standing impact horizontalization 117