Best Fit Journal Ratings Paper (Social Change Sales Pitch Mix)


Published on

The powerpoint presentation to accompany my 2006 paper on the ranking of journals

The paper proposes a new framework for assessing research output based on the research goals of the selected university, internal research strengths of a department, and the ongoing support for development and enhancement of disciplinary thought within the marketing academy. Presented here is a system of ‘best fit’ between institutional research output and ideal publication mediums, rather than a universal rank of “best” journal

The paper can be found at as part of my work on journal ratings

Published in: Business, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Best Fit Journal Ratings Paper (Social Change Sales Pitch Mix)

  1. 2. Dr Stephen Dann The Australian National University Best Journals versus Best Fit Journals A Strategic Orientation to Research Quality
  2. 3. Knight to RQF’s Bishop 3 <ul><li>&quot;What we're looking to do is distribute funding according to research that's of high quality and high impact.&quot; </li></ul>
  3. 4. So what’s quality? <ul><li>Quality is a nebulous entity </li></ul><ul><ul><li>Eye of Beholder Model </li></ul></ul><ul><ul><ul><li>I know what quality looks like when I see it </li></ul></ul></ul><ul><ul><li>Groucho Marx Model </li></ul></ul><ul><ul><ul><li>I wouldn’t want to be published in a journal that would publish me </li></ul></ul></ul><ul><ul><li>Quality is Greener on the Other Side </li></ul></ul><ul><ul><ul><li>Whatever they did is better than what we did. </li></ul></ul></ul>
  4. 5. RQF on Metrics <ul><li>Recommendation: </li></ul><ul><ul><li>Metrics should be used to inform the peer review process rather than replace it. </li></ul></ul><ul><ul><li>This recommendation was elaborated to make the process for the interaction of metrics and the Discipline Panel assessment process more explicit: </li></ul></ul><ul><ul><ul><li>Metrics should not be used in isolation, such as in the ‘shadow exercise’ proposed for the 2008 UK RAE. </li></ul></ul></ul><ul><ul><ul><li>The data should be made available to Assessment Panels at the beginning of the assessment process. </li></ul></ul></ul><ul><ul><ul><li>No attempt should be made to aggregate indicators to produce a single ‘quality score’. </li></ul></ul></ul>
  5. 6. RQF on Ranked Outputs <ul><li>Ranked outputs : Distribution of selected category(ies) of research output listed in the Research Group’s ‘body of work’ into ranked tiers, based on predetermined discipline specific rankings for the selected forms of output. </li></ul><ul><li>The basis of the rankings will vary by discipline (they could be derived for </li></ul><ul><ul><li>refereed journals, </li></ul></ul><ul><ul><li>professional journals, </li></ul></ul><ul><ul><li>book publishers, </li></ul></ul><ul><ul><li>conferences, or </li></ul></ul><ul><ul><li>performance venues </li></ul></ul><ul><li>No weightings should be applied to the different tiers </li></ul>
  6. 7. RQF on research output <ul><li>2.3 Ranked outputs </li></ul><ul><ul><li>the main thrust of this indicator is to encourage researchers to publish in the most prestigious outlets for their discipline </li></ul></ul><ul><li>We set the rules on what’s prestigious. </li></ul><ul><ul><li>So it’s time to set the rules in our favour </li></ul></ul>
  7. 8. RQF on impact <ul><li>The RQFDAG recommends that verifiable impact indicators </li></ul><ul><li>The Impact Statement, of no more than ten pages, must be verifiable and auditable, and include a statement of claims against the impact criteria. The broad criteria will require a demonstration of: </li></ul><ul><ul><li>the engagement with end-users recognising the importance of the research to address a defined social, economic, environmental and/or cultural issue; </li></ul></ul><ul><ul><li>the adoption of the research by the relevant end-user community to generate new policies, products, , processes, attitudes, behaviours and/ or outlooks; </li></ul></ul><ul><ul><li>how the research has produced social, economic, environmental and/or cultural benefits for end-users regionally, nationally and/or internationally; and </li></ul></ul><ul><ul><li>the extent of the benefit arising from the research. </li></ul></ul>
  8. 9. Background to the Model <ul><li>Over 20 years, over 20 different ranking scales </li></ul><ul><li>Once the reader looks outside of the top three or five journals, the inconsistency of these lists stand as a tribute to &quot;one size fits nobody&quot; production orientations </li></ul><ul><ul><li>Polonsky and Whitelaw (2004, 2005, 2006) </li></ul></ul><ul><li>If only the only consistency was change </li></ul><ul><ul><li>Top 3 unshakeable </li></ul></ul>
  9. 10. Ageing of the (sample) Population 1961 Journal of Advertising Research (JAR) (10) 1972 Journal of Advertising (JA) (9) 1978 Journal of Business Research (JBR) (8) 1959 International Journal of Research in Marketing (IJRM) (7) 1925 Journal of Retailing (JR) (6) 1973 Journal of the Academy of Marketing Science (JAMS) (5) 1982 Marketing Science (MS) (4) 1964 Journal of Marketing Research (JMR) (3) 1974 Journal of Consumer Research (JCR) (2) 1937 Journal of Marketing (JM) (1) D.O.B. Name Rank
  10. 11. Quadrant Model Break the old rules to make the new rules
  11. 12. Shifting from the League Table <ul><li>the use of a one-dimensional scale of &quot;best&quot; is inconsistent with the fundamental principles of the discipline and the practice of the industry </li></ul><ul><ul><li>(Corfman, 1991) </li></ul></ul>
  12. 13. Assumption of the Model <ul><li>Research output is assumed to have two core components </li></ul><ul><ul><li>the goal of developing knowledge within a specific discipline (academic freedom) </li></ul></ul><ul><ul><li>the business strategy component which represents the organisational research goals of the University (academic pragmatism) </li></ul></ul>
  13. 14. Discipline Research Goals <ul><li>Development of the marketing discipline </li></ul><ul><ul><li>Pure marketing </li></ul></ul><ul><ul><li>Applied marketing (impact) </li></ul></ul><ul><ul><li>Industry focused research. (impact) </li></ul></ul><ul><li>Two forms </li></ul><ul><ul><li>Academy driven goals </li></ul></ul><ul><ul><ul><li>Eg Vargo and Lusch, services dominant logic </li></ul></ul></ul><ul><ul><li>Academic research freedom </li></ul></ul><ul><ul><ul><li>Personal pursuit of research interests </li></ul></ul></ul>
  14. 15. University Research Goals <ul><li>Organisational research goals / objectives </li></ul><ul><ul><li>University / Funding body </li></ul></ul><ul><ul><li>Faculty / School / Marketing Unit </li></ul></ul><ul><li>AACSB (2006) </li></ul><ul><ul><li>intellectual contributions should be consistent with the school’s mission and strategic management processes </li></ul></ul>
  15. 16. Four categories <ul><li>Developing the discipline of marketing </li></ul><ul><li>Addressing Institutional Research Goals </li></ul><ul><li>Research Agenda Based Objectives </li></ul><ul><li>Independent Research Output </li></ul><ul><li>First draft mistakenly presented these as tiers of objectives, and naturally, it was seen as a best to worst priority list </li></ul>
  16. 17. Discipline and University Research Goals
  17. 18. Discipline and University Research Goals engagement with end-users adoption of the research by the relevant end-user community adoption and benefits for relevant end-users adoption of the research by the relevant end-user community
  18. 19. Quadrant 1 Developing the discipline of marketing <ul><li>Research Agenda: </li></ul><ul><ul><li>pursuit of the enhancement and development of the discipline </li></ul></ul><ul><ul><ul><li>Theoretical developments </li></ul></ul></ul><ul><ul><ul><li>Methodological developments </li></ul></ul></ul><ul><ul><ul><li>Pure science approaches </li></ul></ul></ul><ul><ul><ul><li>RQF End user is the marketing academy. </li></ul></ul></ul><ul><li>Which journals are listed? </li></ul><ul><ul><li>Constitutes a list of the journals published by the respective academies of marketing across the major geographic areas </li></ul></ul><ul><ul><ul><li>Australasia, Europe and America in current list </li></ul></ul></ul><ul><ul><li>Same nature, same cluster </li></ul></ul><ul><li>Only list that should remain constant between universities </li></ul>
  19. 20. Quadrant 1 List <ul><li>Academy of Marketing Science Review </li></ul><ul><li>Australasian Marketing Journal </li></ul><ul><li>International Journal of Research in Marketing </li></ul><ul><li>Journal of International Marketing </li></ul><ul><li>Journal of Marketing </li></ul><ul><li>Journal of Marketing Management </li></ul><ul><li>Journal of Marketing Research </li></ul><ul><li>Journal of Public Policy & Marketing </li></ul><ul><li>Journal of the Academy of Marketing Science </li></ul>
  20. 21. Quadrant 2 Addressing Institutional Research Goals <ul><li>Research Agenda: </li></ul><ul><ul><li>framework to encourage academics to pursue the research interest of the university / funding agency </li></ul></ul><ul><ul><li>End user: University stakeholders, university </li></ul></ul><ul><li>Which journals are listed? </li></ul><ul><ul><li>Journal inclusion is based on providing a matching the research strength of the organisation against the market demand for the research output product </li></ul></ul><ul><ul><ul><li>Requires research and hack work </li></ul></ul></ul><ul><li>Should be relatively similar for organisations with shared/common goals </li></ul><ul><ul><li>Opportunity to pursue diversity as sought by Bishop & DEST </li></ul></ul>
  21. 22. Sample Quadrant 2 <ul><li>Sample Institutional Goal </li></ul><ul><ul><li>Advance knowledge of Australian research </li></ul></ul><ul><li>Journals selected based on the extent to which they publish Australian research </li></ul><ul><ul><li>Method 1: Journal keyword search </li></ul></ul><ul><ul><ul><li>search by university title </li></ul></ul></ul><ul><ul><ul><li>Search by “australian/australia” </li></ul></ul></ul><ul><ul><li>Method 2: Michael Polonsky </li></ul></ul><ul><ul><ul><ul><li>Polonsky, Garma and Mittelstaedt (2006) “An examination of the globalisation of authorship in publishing in 20 leading marketing journals” European Business Review 18 (6) </li></ul></ul></ul></ul><ul><ul><li>Method 3: Cross institutional dataset </li></ul></ul>
  22. 23. Sample Quadrant 2 <ul><li>Key journals </li></ul><ul><ul><li>Asia-Pacific Journal of Marketing and Logistics </li></ul></ul><ul><ul><li>European Journal of Marketing* </li></ul></ul><ul><ul><li>Industrial Marketing Management </li></ul></ul><ul><ul><li>Journal of Brand Management (The) </li></ul></ul><ul><ul><li>Journal of Business Research </li></ul></ul><ul><ul><li>Journal of Consumer Behaviour </li></ul></ul><ul><ul><li>Journal of Consumer Marketing (The) </li></ul></ul><ul><ul><li>Journal of Product and Brand Management </li></ul></ul><ul><ul><li>Marketing Intelligence and Planning </li></ul></ul><ul><li>*Side note: I was once reprimanded by the editorial board of EJM for describing them on ELMAR as “The European Journal of Australians Publishing in Marketing” </li></ul>
  23. 24. Quadrant 3: R esearch Agenda <ul><li>Research Agenda </li></ul><ul><ul><li>Diversification based on collective group research agenda </li></ul></ul><ul><ul><ul><li>Collective agenda of is based on the aggregated areas of research specialisations of the staff </li></ul></ul></ul><ul><ul><ul><li>Self reported identification of research interest </li></ul></ul></ul><ul><ul><li>RQF End user: Business, students, academics </li></ul></ul><ul><li>Which journals are listed? </li></ul><ul><ul><li>Discipline specific journals </li></ul></ul><ul><ul><li>Sub-disciplinary specific journals </li></ul></ul><ul><ul><li>Journals with a specific research concentration in an sub discipline / sub set of marketing </li></ul></ul><ul><li>Should have a wide range of variation between universities </li></ul>
  24. 25. Create the Research Agenda <ul><li>Method: Aggregated Research Agenda </li></ul><ul><ul><li>Step 1: Have the research staff of the unit self identify their research areas against the ANZMAC track stream lists </li></ul></ul><ul><ul><ul><li>(eg primary research, secondary research, PhD topic, grants research) </li></ul></ul></ul><ul><ul><li>Step 2: Research streams are tallied to form an aggregate list of research areas, which is used to form a rank order list. </li></ul></ul><ul><ul><li>Step 3: Cluster track streams against either Polonsky and Whitelaw’s (2006) “areas of research interest” or Lehmann (2006) journal categories </li></ul></ul>
  25. 26. Convert the Research Agenda <ul><li>Method: Convert Research Agenda into Target Journals </li></ul><ul><ul><li>Step 1: Identify Research Agenda Output Journals </li></ul></ul><ul><ul><ul><li>Option 1: Using Lehmann (2006) “Journal Evolution and the Development of Marketing” Journal of Public Policy & Marketing 24(1) pp138-140 </li></ul></ul></ul><ul><ul><ul><li>Option 2: Independent research to select journals based on the number of articles containing the relevant research keyword, and the stated research goal of the journal. </li></ul></ul></ul><ul><ul><ul><li>Option 3: Cross institutional dataset of categorised journals </li></ul></ul></ul><ul><ul><li>Step 2: Prioritise research agenda journals according to the match between the research project and the needs of the target market journal </li></ul></ul><ul><ul><ul><li>Is the paper the best match for the objectives of the journal? </li></ul></ul></ul>
  26. 27. Sample Research Agenda <ul><li>Advertising / Marketing Communication / IMC, Branding </li></ul><ul><li>Marketing Communications </li></ul>10 <ul><li>Entrepreneurship, Firms in Networks, Marketing Strategy, New Product Development, Supply Chain Management, Financial & Value-Based Marketing </li></ul><ul><li>Marketing Strategy </li></ul>11 <ul><li>Corporate Responsibility, Social Issues and Social Responsibility Marketing and Society Non profit marketing Political Marketing Social Marketing </li></ul><ul><li>Marketing & Society </li></ul>12 ANZMAC Track Areas P&W(2006) Sum
  27. 28. <ul><li>Sample Quadrant based on Lehmann 2006 </li></ul><ul><ul><li>Journal of Advertising Research </li></ul></ul><ul><ul><li>Journal of Advertising </li></ul></ul><ul><ul><li>Journal of Current Issues and Research in Advertising </li></ul></ul><ul><ul><li>International Journal of Advertising </li></ul></ul><ul><ul><li>Journal of Marketing Communications </li></ul></ul><ul><ul><li>Journal of Promotion Management </li></ul></ul><ul><li>Advertising / Marketing Communication / IMC, Branding </li></ul><ul><li>Marketing Communications </li></ul>10
  28. 29. Quadrant 4 Independent Research Output <ul><li>Research Agenda </li></ul><ul><ul><li>Academic freedom. </li></ul></ul><ul><ul><li>Personal research agenda that is not constrained by the discipline, organisation or unit goals </li></ul></ul><ul><ul><li>RQF End user: Other academics/practitioners within the research area </li></ul></ul><ul><li>Which journals are listed? </li></ul><ul><ul><li>The rest of the journals </li></ul></ul><ul><li>Should vary between individuals within a group </li></ul>
  29. 30. Quadrant 4 <ul><li>recognition that it is possible for a quality research outlet to not be relevant to the research agenda of an institution. </li></ul><ul><ul><li>Inclusion in Quadrant 4 is not a criticism of the quality of the journal or the research. </li></ul></ul><ul><li>used to defend the ability of the individual researcher to pursue research beyond the goal of the university, academy or marketing unit agenda. </li></ul><ul><li>Can easily be destroyed if it is used for sniping between marketing academics </li></ul><ul><li>This is not the Fourth Tier (Group of Death) of the UQBS /AoU lists </li></ul>
  30. 31. Supporting materials
  31. 32. Cross Institution Journal Verification <ul><li>Cross verification of journals </li></ul><ul><ul><li>Potential ANZMAC Collaboration </li></ul></ul><ul><ul><ul><li>Database shared between marketing groups to identify the match between </li></ul></ul></ul><ul><ul><ul><ul><li>common objectives and journals </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Research interest clusters and journals </li></ul></ul></ul></ul><ul><ul><ul><li>Creative Commons Licensed (Attribution Share-Alike) model </li></ul></ul></ul><ul><ul><ul><ul><li>allows for institutions to contribute to the dataset without seeking prior permission, but prevents any single entity from capturing the dataset. </li></ul></ul></ul></ul>
  32. 33. Supporting Behaviours <ul><li>Publicise the research clusters </li></ul><ul><ul><li>Know what is being published where </li></ul></ul><ul><li>Lay down covering fire </li></ul><ul><ul><li>Cite each other, Cite early, cite often </li></ul></ul><ul><ul><li>Cite ANZMAC papers </li></ul></ul><ul><li>Create an ANZMAC marketing research output index </li></ul><ul><ul><li>shows who’s publishing what and where to accelerate the process of cross-citing </li></ul></ul><ul><li>Specify RQF target end user community. </li></ul><ul><ul><li>Target academics. Target teaching. Cross cite, and use in the classrooms. </li></ul></ul><ul><li>Cheat to Win : RQF is one of the largest prisoner’s dilemma games invented – so let’s cheat and communicate with each other. </li></ul>
  33. 34. Questions?