From Publisher To Platform: How The Guardian Used Content, Search, and Open Source To Build a Powerful New Business Model

1 From publisher to platform How the guardian used content, search, and open source to build a powerful new business model Stephen Dunn, Guardian News and Media Apache Lucene EuroCon 21 May 2010

The publishing era Apache Lucene EuroCon 21 May 2010 2

We started a long time ago: Apache Lucene EuroCon 21 May 2010

“To secure the ﬁnancial and editorial independence of To secure the financial and editorial The Guardian in perpetuity.” independence of the Guardian in perpetuity. Topromote freedom in the press press and liberal “To promote freedom in the and liberal journalism journalism globally. globally.” Apache Lucene EuroCon 21 May 2010

2010 Apache Lucene EuroCon 21 May 2010

2010 Keyword page Live blogs iPhone app Mobile site Twitter updates Swine flu Comment Content partnerships Newspapers Audio Video Data API Apache Lucene EuroCon 21 May 2010

1996 Apache Lucene EuroCon 21 May 2010 6

1999 Apache Lucene EuroCon 21 May 2010 7

01-> 06 Apache Lucene EuroCon 21 May 2010 8

2009 1.5M pages and counting 250M+ pages/ month 30M visitors/ month 4x Webby award winner (best newspaper site) Apache Lucene EuroCon 21 May 2010 9

Part of the Web Apache Lucene EuroCon 21 May 2010 10

1. Permanent http://www.ﬂickr.com/photos/fstorr/ • “A cool URI is one that does not change” Tim Berners-Lee 1998 • 1.5 million resources redirected to new scheme Apache Lucene EuroCon 21 May 2010 11

2. Addressable ★ Resources are “about” something - ready for the social web. ★ We live in “the age of point-at-things” (Coates 2005) Apache Lucene EuroCon 21 May 2010 12

3. Discoverable ★ Multiple routes to content ★ Tagging drives discovery Apache Lucene EuroCon 21 May 2010 13

Apache Lucene EuroCon 21 May 2010 14

The hackable guardian.co.uk http://www.guardian.co.uk/.... Apache Lucene EuroCon 21 May 2010

The hackable guardian.co.uk http://www.guardian.co.uk/.... /technology/internet /technology/all /environment/climatechange Apache Lucene EuroCon 21 May 2010

The hackable guardian.co.uk http://www.guardian.co.uk/.... /technology/internet /technology/all /environment/climatechange +business/globaleconomy Apache Lucene EuroCon 21 May 2010

The hackable guardian.co.uk http://www.guardian.co.uk/.... /technology/internet/rss /technology/all/rss /environment/climatechange +business/globaleconomy/rss Apache Lucene EuroCon 21 May 2010

Results... Apache Lucene EuroCon 21 May 2010 16

Site trafﬁc growth Final Release Unique Users First release Apache Lucene EuroCon 21 May 2010 17

Site trafﬁc growth Final Release Unique Users 30,000,000 26,250,000 First release 22,500,000 Unique Users 18,750,000 15,000,000 11,250,000 7,500,000 3,750,000 Sep 2005 Feb 2006 Jul 2006 Dec 2006 May 2007 Oct 2007 Mar 2008 Aug 2008 Jan 2009 Apache Lucene EuroCon 21 May 2010 17

Site trafﬁc growth Final Release Unique Users 30,000,000 26,250,000 First release 22,500,000 Unique Users Pre - project 18,750,000 15,000,000 11,250,000 7,500,000 3,750,000 Sep 2005 Feb 2006 Jul 2006 Dec 2006 May 2007 Oct 2007 Mar 2008 Aug 2008 Jan 2009 Apache Lucene EuroCon 21 May 2010 17

Site trafﬁc growth Final Release Unique Users 30,000,000 26,250,000 First release 22,500,000 Unique Users Pre - project 18,750,000 15,000,000 11,250,000 36M 7,500,000 3,750,000 Sep 2005 Feb 2006 Jul 2006 Dec 2006 May 2007 Oct 2007 Mar 2008 Aug 2008 Jan 2009 Apache Lucene EuroCon 21 May 2010 17

However... Apache Lucene EuroCon 21 May 2010 18

1 Billion+ Internet Users! Apache Lucene EuroCon 21 May 2010 19

....”How I stopped worrying about my website and learned to love the whole Internet.” Matt McAlister Apache Lucene EuroCon 21 May 2010 23

The Open Strategy OPEN IN OPEN OUT Bring in data and Enable partners to apps from the build applications Internet using Guardian content and services for other digital platforms Apache Lucene EuroCon 21 May 2010 24

"Our most interesting experiments lie in combining what we know with the experience, opinions and expertise of the people who want to participate rather than passively receive.” Apache Lucene EuroCon 21 May 2010 28

TA BE The Open Platform Apache Lucene EuroCon 21 May 2010 29

TA BE OPEN IN OPEN OUT Bring in data and apps Allow partners to build from the Internet applications using Guardian content and services for other digital platforms Apache Lucene EuroCon 21 May 2010 30

TA BE The suite of services enabling partners to build applications with the Guardian Apache Lucene EuroCon 21 May 2010 31

TA BE Apache Lucene EuroCon 21 May 2010

TA BE CONTENT API DATA STORE POLITICS API A service for A directory of Open database of selecting and useful data candidates, voting collecting content curated by records, from the Guardian Guardian constituencies, editors election results, for re-use live data on election day Apache Lucene EuroCon 21 May 2010

TA BE Your App Here! CONTENT API A service for selecting REST API and collecting content from the Guardian for re-use Search engine CMS Guardian database Apache Lucene EuroCon 21 May 2010

TA BE Apache Lucene EuroCon 21 May 2010 34

• Stamen Design - APIMaps.org Apache Lucene EuroCon 21 May 2010 35

Text Apache Lucene EuroCon 21 May 2010 36

TA BE DATA STORE A directory of useful data curated by Guardian editors Apache Lucene EuroCon 21 May 2010

TA BE POLITICS API Open database of candidates, voting records, constituencies, election results, live data on election day Apache Lucene EuroCon 21 May 2010

TA BE POLITICS API Open database of candidates, voting records, constituencies, election results, live data on election day Apache Lucene EuroCon 21 May 2010 39

TA BE Open for Business Apache Lucene EuroCon 21 May 2010 40

Open for Business Apache Lucene EuroCon 21 May 2010 40

1 3 Tiers of access, 3 Revenue models BESPOKE: Take, reformat, augment our content. Same access as Guardian. Revenue model to be negotiated. Combination of Media, Fees, Downloads. APPROVED: Take our full article content, with an advert. Guardian keeps ad revenue, you keep rest-of-page revenue KEYLESS: Take our headlines. You keep associated revenues Apache Lucene EuroCon 21 May 2010 41

What this means OPEN OUT: Developers can now access our full content APIs on demand with keys post-approved. We are now positioning the platform as a place to do business with us. So, rapid scalability, reliability, performance, are now core requirements Apache Lucene EuroCon 21 May 2010 43

2 Open In CONTENT API DATA STORE POLITICS API A service for selecting A directory of Open database of and collecting content useful data curated candidates, voting from the Guardian for by Guardian records, re-use editors constituencies, election results, live data on election day Apache Lucene EuroCon 21 May 2010

2 Open In CONTENT API DATA STORE POLITICS API MICROAPPS A service for selecting A directory of Open database of A framework for and collecting content useful data curated candidates, voting integrating 3rd party from the Guardian for by Guardian records, applications into re-use editors constituencies, guardian.co.uk. election results, live data on election day Apache Lucene EuroCon 21 May 2010

OPEN OUT OPEN IN Allow partners to build Bring in data and apps applications using from the Internet Guardian content and services for other digital platforms Apache Lucene EuroCon 21 May 2010 45

App showcase Apache Lucene EuroCon 21 May 2010 48

What this means Open In: Partners can now more easily integrate into our core The Open Platform will become key to our commercial future. Apache Lucene EuroCon 21 May 2010 49

Evolving the architecture Apache Lucene EuroCon 21 May 2010 50

From Publisher to Platform ★Seeking massive growth, but no longer only broadcasting content ★User/partner engagement & contribution on ★journalism ★data ★software ★applications ★revenue and ads ★ Support developers and partners with data and APIs, need scalability, reliability, speed Apache Lucene EuroCon 21 May 2010 51

Web server Web server Web server App server App server App server Memcached Oracle CMS Apache Lucene EuroCon 21 May 2010

Web server Web server Web server Why RDBMS? App server App server App server 5 years ago, fewer alternatives Understand operations procedures Memcached Can easily recruit DBAs / devs Developer/ops tools Oracle Business critical system: a safe choice CMS Data feeds Apache Lucene EuroCon 21 May 2010

Scaling Apache Lucene EuroCon 21 May 2010 54

Unique Users Apache Lucene EuroCon 21 May 2010 55

30,000,000 Unique Users 26,250,000 22,500,000 Unique Users 18,750,000 15,000,000 11,250,000 7,500,000 3,750,000 Sep 2005 Feb 2006 Jul 2006 Dec 2006 May 2007 Oct 2007 Mar 2008 Aug 2008 Jan 2009 Apache Lucene EuroCon 21 May 2010 55

Unique Users Apache Lucene EuroCon 21 May 2010 56

28,000,000 25,750,000 Unique Users 23,500,000 21,250,000 19,000,000 16,750,000 14,500,000 12,250,000 May 2008 Jul 2008 Sep 2008 Nov 2008 Jan 2009 Apache Lucene EuroCon 21 May 2010 56

Whatʼs going on? ★We tag our content (multifaceted) ★Guardian.co.uk is a faceted browse through our tag- space, with editorial teams “spotlighting” key resources on selected nodes. ★Can apply multiple facets in queries faster in a search-like architecture, than an RDBMS Apache Lucene EuroCon 21 May 2010 57

“Related content” from search engine Apache Lucene EuroCon 21 May 2010 58

5 Apache Lucene EuroCon 21 May 2010

Your App Here! CONTENT API A service for selecting REST API and collecting content from the Guardian for re-use Search engine CMS Guardian database Apache Lucene EuroCon 21 May 2010

We used Solr/Lucene Can perform complex queries, including full text search We can change the schema with no downtime. On our dataset most queries are of a similar cost Scales very well horizontally Replication makes it easy to work in the cloud Apache Lucene EuroCon 21 May 2010 62

Core Web servers App server Memcached rdbms CMS Apache Lucene EuroCon 21 May 2010 63

Core Content API Web servers Solr App server Solr Memcached Solr rdbms Solr Solr Solr CMS Cloud, EC2 Apache Lucene EuroCon 21 May 2010 63

Open in? Simple REST/ HTTP framework MICROAPPS allows lightweight development A framework for Applications proxied for integrating 3rd party performance applications into guardian.co.uk. Apps generally hosted in the cloud, hot deployment into production Apache Lucene EuroCon 21 May 2010

Core Apps Web servers Proxy App App server App Memcached App App rdbms App App CMS external hosting app engine etc Apache Lucene EuroCon 21 May 2010 65

OPEN IN OPEN OUT Web servers Solr Proxy App App servers App Memcached Solr App Solr App CMS Solr Solr App Solr App rdbms Cloud, EC2 external hosting app engine etc Apache Lucene EuroCon 21 May 2010

C I O CONTENT r external Clo C I O ??????? r external Clo Apache Lucene EuroCon 21 May 2010

Thank you http://www.guardian.co.uk/open-platform Twitter: @openplatform @cuica (Stephen Dunn) Apache Lucene EuroCon 21 May 2010 68

http://www.guardian.co.uk	759
http://www.slideshare.net	271
http://rafaelsidi.blogspot.com	15
http://diarium.usal.es	8
http://www.pcanete.com.ar	3
http://www.guprod.gnl	1
http://localhost	1

From Publisher To Platform: How The Guardian Used Content, Search, and Open Source To Build a Powerful New Business Model

by The Guardian Open Platform on May 20, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 1,058

Statistics

From Publisher To Platform: How The Guardian Used Content, Search, and Open Source To Build a Powerful New Business Model — Presentation Transcript