Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1kNlmuF. Graham Tackley dives into the details of ophan - obstacles faced by the newsroom that prompted them to build the system, how it works for alerting and how the tool has made the Guardian's readers - and staffers - lives better. Shay Banon covers the technical underpinnings of ophan with a deep dive into the Elasticsearch features and functionality that power the ophan system. Filmed at qconlondon.com. Graham Tackley is Director of Architecture at Guardian News and Media. He spent a few years leading the Web Platform Team engineering guardian.co.uk to scale to ever increasing demand. Shay Banon is the creator of Elasticsearch, the open source, distributed, RESTful, search and analytics engine.

1. How Elasticsearch powers
the Guardian’s newsroom
graham tackley ■ @tackers
director of architecture
guardian news and media
shay banon ■ @kimchy
creator, co-founder and cto
elasticsearch

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/devops-banking-ci-cd

3. Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

6. “created in 1936 ... to secure the financial
and editorial independence of
the Guardian in perpetuity”

8. our in-house real-time traffic tool

10. my desktop
workstation
production
apaches
something
htmly ?

11. ssh $SERVER "nice tail -f /apache2/logs/guardian-access_log"

12. my desktop
workstation
2 x production
apaches
publisher
ssh “tail”
zeromq
x
SEO
dashboard

16. my desktop
workstationx

17. Javascript in browser
SNS
SQS
hidden pixel
Dashboard
Tracker

20. Javascript in browser
Tracker
SNS
SQS
hidden pixel
SQS
Dashboard
Serf
elasticsearch
Dashboard

22. 12 * m3.xlarge
in an autoscaling group
(with manual scaling)
instance store (SSD)
https://github.com/guardian/status-app

24. {
"dt": "2014-03-03T02:01:48.026Z",
"url": "http://www.theguardian.com/film/2014/mar/03/oscars-2014-winners-list",
"queryString": "",
"host": "www.theguardian.com",
"path": "/film/2014/mar/03/oscars-2014-winners-list",
"section": "film",
"platform": "r2",
"userAgent": {
"type": "Browser",
"family": "Safari 5.1.9",
"os": "OS X 10.6.8",
"device": "Personal computer"
},
"documentReferrer": "http://www.theguardian.com/world",
"browser": {
"id": "gA6RUFLhWNQvWdt0rW4r78Fg",
"isNew": false
},
"referringHost": "theguardian.com",
"referringPath": "/world",
"isContent": true,
"contentPublicationDate": "2014-03-03",
"countryCode": "US",
"countryName": "United States",
"location": {
"lonlat": [-73.4409, 41.2094]
}
}
⇠filter
⇠filter
⇠count per minute

25. {
"query" : {
"filtered" : {
"query" : { "match_all" : { } },
"filter" : {
"term" : { "path" : "/film/2014/mar/03/oscars-2014-winners-list" }
}
}
},
…

26. …
"facets": {
"Reddit": {
"date_histogram": { "field": "dt", "interval": "1m" },
"facet_filter": { "term": { "referringHost": "reddit.com" } }
},
"Facebook": {
"date_histogram": { "field": "dt", "interval": "1m" },
"facet_filter": { "term": { "referringHost": "facebook.com" } }
},
"Google": {
"date_histogram": { "field": "dt", "interval": "1m" },
"facet_filter": {
"or": {
"filters": [
{ "prefix": { "referringHost": "www.google." } },
{ "prefix": { "referringHost": "news.google." } }
]
}
}
}
}
}

28. /graph/breakdown?section=commentisfree

29. ?section=commentisfree
ophan.StandardFilters
ophan.StandardFiltersToElasticsearch
org.elasticsearch.index.
query.FilterBuilder

30. {
"query" : {
"filtered" : {
"query" : { "match_all" : { } },
"filter" : {
"term" : { "path" : "/film/2014/mar/03/oscars-2014-winners-list" }
}
}
},
…

31. "filter": {
"and": {
"filters": [
{
"range": {
"dt": {
"from": "2014-03-03T00:00:00.000Z",
"to": "2014-03-03T22:30:59.999Z",
"include_lower": true,
"include_upper": false
}
}
},
{
"not": {
"filter": { "term": { "countryCode": "GNM" } }
}
},
{
"not": {
"filter": { "term": { "userAgent.type": "Robot" } }
}
},
{
"filter": { "terms": { "section": [ "commentisfree" ] }}
}
]
}
}

34. thank you
graham tackley ■ @tackers
director of architecture
guardian news and media
shay banon ■ @kimchy
creator, co-founder and cto
elasticsearch

35. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/elasticsearch
-guardian