Presentation at the #AoIR2019 conference, Brisbane, 3 October 2019.

1. Aljosha Karim Schapals | @aljoshakarim
Axel Bruns | @snurb_dot_info
AoIR 2019 | Brisbane, Australia
Trust in Journalism Metrics
Are journalism metrics trustworthy enough to be used in
editorial decision making?

2. • Project title: three-year ARC Discovery project “Journalism beyond the
Crisis”, investigating what we refer to as the ‘post-journalism era’
• Context: “The people formerly known as journalists” (cf. Rosen)
 Is what they do journalism?
 Where do we draw the boundaries?
• Journalism in the digital age characterised by a “participatory information
environment in which long-established processes of production and
consumption of journalism coexist and increasingly compete with new
modes and styles of fact-based discourse” (Hanusch; Bruns)
 Journalism changing both its forms as well as its functions
Research background (1)
@qutdmrc

3. • Context: journalism going through a period of transition,
allegedly “a crisis”, but: “crisis of journalism” actually a crisis
of adaptation to the digital environment
 Journalism in transition rather than in decline
 Positive developments in journalism that normative critiques
fail to acknowledge
• How are fact-based media evolving in the digital age?
 examining the development of journalism in the Western
world, comparative perspective: Australia, Germany, UK
Research background (2)
@qutdmrc

4. Research background (3): Four research steps
@qutdmrc
Journalism beyond the Crisis
Forms & Styles Professional
Practices
Audience Uses
Democratic
Implications

5. - Who are the journalists in the digital age?
- Where do we draw the boundaries between professional
and amateur content?
- Methodology: 90 in-depth interviews with journalists (30 in
each country)
 Questionnaire includes a range of questions on journalists‘
professional practices in the digital age
 Specifically, it includes evaluations of journalism metrics
(positive/negative) and their impact on editorial decision-
making
 Goal: understanding the definitions and re-definitions of
journalism in an age of digital transformation
Research background (4): Professional practices
@qutdmrc
Journalism beyond the
Crisis
Professional
Practices

6. • Context: journalism undergoing an intense process of
quantification ever since shift towards online platforms as
primary medium for news consumption (Newman et al. 2018)
• Journalism’s ‘quantitative turn’ includes metrics such as
page views, reading time, arrival and departure pathways, as
well as audience engagement across various social media
platforms
 Such disruptions lead to (valid) questions on the impact of
such insights on editorial decision-making: stories
deemed to be ‘successful’ receiving further coverage – vs. –
the (extreme) view of stories receiving poor levels of
engagement not sufficiently covered / not covered at all
Literature Review: Journalism’s ‘quantitative turn’
@qutdmrc

7. • Metrics in the newsroom presented to journalists in such granular detail that it even
enables tracking of story performance on live dashboards, e.g. to track headlines,
synopses, or images  enabling real-time changes to ‘boost’ performance (Cherubini &
Nielsen 2016)
• Previous studies highlight both journalists’ increasingly strategic editorial decision-
making as a result of insights into these analytics – but it also details their concerns:
(1) Strategically ‘exploiting’ metrics: “The natural inclination, if one metric is seen as
the ‘important’, ‘true’ metric … is to game it” (Jonah Peretti, Buzzfeed founder)
(2) (Economic) imperative to ‘chase better metrics’: “If the company’s not making
enough money, then I might get laid off … That’s just the way it is” (Web editor qtd. in
Tandoc Jr. 2014)
 At this stage in their evolution: approaches to the incorporation of metrics into
editorial decision-making still highly divergent across the industry 
comprehensive – and crucially, comparative – picture thus far missing
Literature Review: Journalism’s ‘quantitative turn’
@qutdmrc

8. “We have to be aware of the responsibility we have as journalists! It’s not
to simply drop any stories that don’t have any readers. Because, in that
sense, that would mean stopping journalism if we just drop what people
don’t want to read.”
“If our job as journalists is to make sense of the world; trying to explain
things, then I think it is completely horrible when everything is narrowed
down to a single score. … The goal should be maximum diversity covering
a maximum diversity of topics, and this binary thinking – it’s just too top-
down.”
“It’s really important that we have all these metrics – but it is just as
important that they don’t start impacting on your decision-making too
heavily. We look very carefully at what people are interested in – but the
final decision really needs to rest with your own judgment.”
Findings: Perceptions of journalism metrics (1)
@qutdmrc

9. “I think it’s important to know what stories get interest, and maybe push them further or place them more
prominently on our website, but I think we should still inform people about things they may not think they
are interested in. I think that’s our job as journalists and we shouldn’t get caught up in all those
analytics, and we should still focus on doing our job to the best of our abilities – even if it doesn’t get any
clicks.”
“It does influence the kind of stories we then end up doing. But having said that, it influences the kind of
stories we’re still doing – we do – but it doesn’t dictate that. Because if we know, for example, when we
do immigration stories – they don’t do that well. You don’t get that many people watching. But that
doesn’t mean that we shouldn’t do them.”
“It does affect editorial decisions and it does affect decisions about what is seen on air on the day as
well.”
“What you end up with is that you’re feeding an audience a restricted menu: it’s like being in a restaurant
where you’re literally served five dishes, because it has been proved that those are the popular dishes.
But that’s not allowing for some amazing, new dishes, and also dishes that might not be popular – but
they’re really valuable. So, we’re fighting that fight, but obviously we have to do it while keep getting
traffic, while keeping advertisers happy. This is the type of journalist that works these days.”
Findings: Perceptions of journalism metrics (2)
@qutdmrc

10. “Almost without exception the stories that do best are long form pieces, thoughtfully written,
that took a long time to research – that are slow journalism, essentially. So there’s that
longevity of that I think a lot of people miss when they’re looking at what works online.
Because a lot of people aren’t writing with a view for it to be read in a year’s time; there’s so
much of a need to hit those marks and get those KPI’s and show them to their bosses and
stay in the job. So once you’re unshackled from that I think there’s a real opportunity
to use that data in a really interesting way.”
“There's always a temptation to just keep producing, 100%, 24 hours a day – cat videos.
The same stuff over and over again. Not to try anything new, not to innovate. But then you
kind of end up dying by doing that anyway. You've got keep changing, stay interesting.
So that's the disadvantage. The advantage is that it allows you to see that when the
audience does respond to quality. … That kind of gives us the concrete data to take it to
management, take it to people with money, and say, “Hey, we want to make more of this,
it works, check out the numbers.” That's a reason I like it.”
“Yeah, I mean, absolutely! I mean, it's tragic, we get that with climate change, for
example. We're getting told that the world is heading towards a certain cataclysm and from
what the numbers I'm seeing on Google Analytics, no one gives a shit. It just puts people to
sleep.”
Findings: Perceptions of journalism metrics (3)
@qutdmrc

11. • Aim of this study: new, comparative perspective on the trustworthiness of journalistic metrics – in the view of
practicing journalists themselves…
• Preliminary findings overall indicate a prevailing sense of ambiguity on the role of journalism metrics in
contemporary editorial decision-making:
 The contemporary conundrum: delivering on the journalistic mission (providing recipients with valuable
information pertinent to their daily lives) vs. delivering on its economic imperative (keeping advertisers ‘happy’)
 So, do journalists ‘trust the system’? To an extent: metrics provide added insights and are welcome so long
as the pendulum doesn’t swing towards exclusively catering to the economic imperative pertinent to much of
journalism these days
Conclusion
@qutdmrc
THREATS OPPORTUNITIES
• ‘Chasing metrics’ may lead to situations in which journalism no longer
fulfils its central role – merely giving people what they want rather
than providing information in the public interest does not cater to that
• ‘Chasing metrics’ could, in the medium to long-term, lead to issues of
story diversity: following patterns in the data – and exclusively
catering to that pattern – does not allow for content diversification
• Metrics are important – so long as they are merely used as a useful
indicator, but not as a guiding principle
• Metrics are important – but what weighs heavier is a journalist’s own
news judgment, developed over years, if not decades

12. Thank You!