SlideShare a Scribd company logo
1 of 56
Mashing up

        Michael Brunton-Spall
michael.brunton-spall@guardian.co.uk
            @mibgames
About Us

 Online since 1995
 250M+ pages per month
 30M+ visitors per month
1995 - Guardian Online
1999 - Guardian Unlimited
1999 - Removed the registration wall
2007 - Rebuild and Redesign
2007 - Rebuild and Redesign
2007 - Rebuild and Redesign
Developing The Guardian
The Hackable Guardian

 Url Hacking
 Keyword Combiners
 RSS Feeds
Url Hacking

http://www.guardian.co.uk/
    [section]/[keyword]
        technology/internet
    [section]/all
        environment/all
     [publication]/[date]/[newspapersection]
        theguardian/2009/jun/11/technologyguardian
    profile/[name]
        profile/bobbiejohnson
RSS Feeds

RSS Everywhere!
  [section]/[keyword]/rss
      technology/internet/rss
  [section]/all /rss
      environment/all/rss
   [publication]/[date]/[newspapersection]/rss
      theguardian/2009/jun/11/technologyguardian/rss
  profile/[name]/rss
      profile/bobbiejohnson/rss
Full Fat!


http://www.flickr.com/photos/snapperwolf/
Url Combiners

  A Logical AND
  Almost any combination from before
     technology/internet+profile/bobbiejohnson
     theguardian/technologyguardian+technology/internet
  Except things that aren't actual tags:
     Dated newspaper sections
        theguardian/2009/jun/11... + anything = 404
     the all page
        .../all + anything = 404
Problems with this approach

  Fragile
  No Discovery
  Not well documented
  Copyright Issues
Build applications with the Guardian
Open Platform

  Opening up how we work with people both internally
  and externally
  A suite of services enabling partners to build
  applications with the Guardian
     Content API
     Data Store
Data Store
  A directory of useful data curated by Guardian editors
Data Store
  A directory of useful data curated by Guardian editors
Data Store
Data Store
Content API

  A service for selecting and collecting content from the
  Guardian for re-use
Content API

  A service for selecting and collecting content from the
  Guardian for re-use
Content API




              Search content
Content API




              Xml and Json
Content API




              Tag Metadata
Content API




              Full Article Body
Content API




              Filters
So why?


 guardian.co.uk has an amazing amount of
 quality content
 Incredible amounts of meta-data, curated by
 editors
 Aim to allow the guardian to become a rich
 source of facts and journalism for the web
How?
 Backed of our search platform
 Provides access to 10 years of article content
 and metadata
 Supports multiple output formats: XML, JSON,
 ATOM
 Supported free text search across content
 Search for keywords
 Guardian supported api projects in Java, PHP,
 Python and Ruby
 Community supported api projects in Perl,
 ActionScript and Coldfusion (and probably
 more?)
Pricing
Pricing




          FREE!
How free is free?

  You can publish full articles from the guardian on your
  website
  5k queries per day limit
  24 hour maximum cache lifetime
  Online support
  Partner with us on advertising
Beta trial


  Limited number of keys
  Collecting feedback
  Will open more widely at end of beta program
How do I use it?

  Home page
     http://www.guardian.co.uk/open-platform
  Sign up for an API key at
     http://guardian.mashery.com
  Use the API Explorer at
     http://api.guardianapis.com/docs/
  Use the python library at
     http://code.google.com/p/openplatform-python/
What can I do with it?


  Search our tag hierarchy
  Find content by tag
  Find content by search terms




         Display on your website!
MP's Expenses
MP's Expenses

 Written in Django
 Hosted on EC2
 Easy to modify
MP's Expenses




                An MP's Page
View

def mp(request, id):
  mp = get_object_or_404(MP, pk = id)
  ...

  return render(request, 'mp.html', {
     'mp': mp,
     'documents': mp.documents.all(),
     'top_users': top_users[:5],
  })
View

def mp(request, id):
  mp = get_object_or_404(MP, pk = id)
  ...

  return render(request, 'mp.html', {
     'mp': mp,
     'documents': mp.documents.all(),
     'top_users': top_users[:5],
  })
View

def mp(request, id):
  mp = get_object_or_404(MP, pk = id)
  ...

  return render(request, 'mp.html', {
     'mp': mp,
     'documents': mp.documents.all(),
     'top_users': top_users[:5],
  })
Using the Guardian API

  Getting all articles about an MP
     results = client.search(q='"%S"' % (name))
     returns a paginating iterator
     for x in results:
         print x['headline']
     Shows only first 10 by default
Get the client

def mp(request, id):
  mp = get_object_or_404(MP, pk = id)
  ...
  client = Client(settings.GUARDIAN_APIKEY)

  return render(request, 'mp.html', {
     'mp': mp,
     'documents': mp.documents.all(),
     'top_users': top_users[:5],
  })
Make the request

def mp(request, id):
  mp = get_object_or_404(MP, pk = id)
  ...
  client = Client(settings.GUARDIAN_APIKEY)

  return render(request, 'mp.html', {
     'mp': mp,
     'documents': mp.documents.all(),
     'top_users': top_users[:5],
     'articles': client.search(q='"%s"'%(mp.name)),
  })
Update the template

<div id="about-mp">
  <ul>
     {% for article in articles %}
     <li>
        <a href="{{article.webUrl}}">{{article.headline}}</a>
     </li>
     {% endfor %}
  </ul>
</div>
And the result!
Additions?

  Use Memcached to cache the response from guardian API
  Don't create a new client each time
  Article bodies
Who else is using it?
Who else is using it?
Who else is using it?
Who else is using it?
Public API Key?

 jbynv3fwdp8ju5625mt2axw3
 Only 5k queries a day
 Will be closed on Monday 6th July
 Will be closed if abused
 Strongly encourage you to sign up for your own key
Questions?

More Related Content

Similar to Mashing Up The Guardian

SymfonyCon Berlin 2016 - Symfony Plugin for PhpStorm - 3 years later
SymfonyCon Berlin 2016 - Symfony Plugin for PhpStorm - 3 years laterSymfonyCon Berlin 2016 - Symfony Plugin for PhpStorm - 3 years later
SymfonyCon Berlin 2016 - Symfony Plugin for PhpStorm - 3 years laterHaehnchen
 
Generating Unified APIs with Protocol Buffers and gRPC
Generating Unified APIs with Protocol Buffers and gRPCGenerating Unified APIs with Protocol Buffers and gRPC
Generating Unified APIs with Protocol Buffers and gRPCC4Media
 
Benefits of the CodeIgniter Framework
Benefits of the CodeIgniter FrameworkBenefits of the CodeIgniter Framework
Benefits of the CodeIgniter FrameworkToby Beresford
 
Adopting F# at SBTech
Adopting F# at SBTechAdopting F# at SBTech
Adopting F# at SBTechAntya Dev
 
API Workshop: Deep dive into REST APIs
API Workshop: Deep dive into REST APIsAPI Workshop: Deep dive into REST APIs
API Workshop: Deep dive into REST APIsTom Johnson
 
IBM Omnifind Enterprise Portal Seach To Improve Productivity
IBM Omnifind Enterprise   Portal Seach To Improve ProductivityIBM Omnifind Enterprise   Portal Seach To Improve Productivity
IBM Omnifind Enterprise Portal Seach To Improve ProductivityFrancis Ricalde
 
ITCamp 2013 - Adam Granicz - Developing for W8 with F# and WebSharper
ITCamp 2013 - Adam Granicz - Developing for W8 with F# and WebSharperITCamp 2013 - Adam Granicz - Developing for W8 with F# and WebSharper
ITCamp 2013 - Adam Granicz - Developing for W8 with F# and WebSharperITCamp
 
Need for Speed: Removing speed bumps in API Projects
Need for Speed: Removing speed bumps in API ProjectsNeed for Speed: Removing speed bumps in API Projects
Need for Speed: Removing speed bumps in API ProjectsŁukasz Chruściel
 
Programming With Amazon, Google, And E Bay
Programming With Amazon, Google, And E BayProgramming With Amazon, Google, And E Bay
Programming With Amazon, Google, And E BayPhi Jack
 
A Basic Django Introduction
A Basic Django IntroductionA Basic Django Introduction
A Basic Django IntroductionGanga Ram
 
High quality ap is with api platform
High quality ap is with api platformHigh quality ap is with api platform
High quality ap is with api platformNelson Kopliku
 
Web develop in flask
Web develop in flaskWeb develop in flask
Web develop in flaskJim Yeh
 
Drupal 8 preview_slideshow
Drupal 8 preview_slideshowDrupal 8 preview_slideshow
Drupal 8 preview_slideshowTee Malapela
 
Googleappengineintro 110410190620-phpapp01
Googleappengineintro 110410190620-phpapp01Googleappengineintro 110410190620-phpapp01
Googleappengineintro 110410190620-phpapp01Tony Frame
 
The Big Picture and How to Get Started
The Big Picture and How to Get StartedThe Big Picture and How to Get Started
The Big Picture and How to Get Startedguest1af57e
 
Making the most out of kubernetes audit logs
Making the most out of kubernetes audit logsMaking the most out of kubernetes audit logs
Making the most out of kubernetes audit logsLaurent Bernaille
 
Making Things Work Together
Making Things Work TogetherMaking Things Work Together
Making Things Work TogetherSubbu Allamaraju
 
The goodies of zope, pyramid, and plone (2)
The goodies of zope, pyramid, and plone (2)The goodies of zope, pyramid, and plone (2)
The goodies of zope, pyramid, and plone (2)Dylan Jay
 

Similar to Mashing Up The Guardian (20)

SymfonyCon Berlin 2016 - Symfony Plugin for PhpStorm - 3 years later
SymfonyCon Berlin 2016 - Symfony Plugin for PhpStorm - 3 years laterSymfonyCon Berlin 2016 - Symfony Plugin for PhpStorm - 3 years later
SymfonyCon Berlin 2016 - Symfony Plugin for PhpStorm - 3 years later
 
Generating Unified APIs with Protocol Buffers and gRPC
Generating Unified APIs with Protocol Buffers and gRPCGenerating Unified APIs with Protocol Buffers and gRPC
Generating Unified APIs with Protocol Buffers and gRPC
 
Benefits of the CodeIgniter Framework
Benefits of the CodeIgniter FrameworkBenefits of the CodeIgniter Framework
Benefits of the CodeIgniter Framework
 
Introduction to python scrapping
Introduction to python scrappingIntroduction to python scrapping
Introduction to python scrapping
 
Adopting F# at SBTech
Adopting F# at SBTechAdopting F# at SBTech
Adopting F# at SBTech
 
API Workshop: Deep dive into REST APIs
API Workshop: Deep dive into REST APIsAPI Workshop: Deep dive into REST APIs
API Workshop: Deep dive into REST APIs
 
IBM Omnifind Enterprise Portal Seach To Improve Productivity
IBM Omnifind Enterprise   Portal Seach To Improve ProductivityIBM Omnifind Enterprise   Portal Seach To Improve Productivity
IBM Omnifind Enterprise Portal Seach To Improve Productivity
 
ITCamp 2013 - Adam Granicz - Developing for W8 with F# and WebSharper
ITCamp 2013 - Adam Granicz - Developing for W8 with F# and WebSharperITCamp 2013 - Adam Granicz - Developing for W8 with F# and WebSharper
ITCamp 2013 - Adam Granicz - Developing for W8 with F# and WebSharper
 
Need for Speed: Removing speed bumps in API Projects
Need for Speed: Removing speed bumps in API ProjectsNeed for Speed: Removing speed bumps in API Projects
Need for Speed: Removing speed bumps in API Projects
 
Programming With Amazon, Google, And E Bay
Programming With Amazon, Google, And E BayProgramming With Amazon, Google, And E Bay
Programming With Amazon, Google, And E Bay
 
A Basic Django Introduction
A Basic Django IntroductionA Basic Django Introduction
A Basic Django Introduction
 
High quality ap is with api platform
High quality ap is with api platformHigh quality ap is with api platform
High quality ap is with api platform
 
Web develop in flask
Web develop in flaskWeb develop in flask
Web develop in flask
 
Drupal 8 preview_slideshow
Drupal 8 preview_slideshowDrupal 8 preview_slideshow
Drupal 8 preview_slideshow
 
Googleappengineintro 110410190620-phpapp01
Googleappengineintro 110410190620-phpapp01Googleappengineintro 110410190620-phpapp01
Googleappengineintro 110410190620-phpapp01
 
The Big Picture and How to Get Started
The Big Picture and How to Get StartedThe Big Picture and How to Get Started
The Big Picture and How to Get Started
 
Making the most out of kubernetes audit logs
Making the most out of kubernetes audit logsMaking the most out of kubernetes audit logs
Making the most out of kubernetes audit logs
 
Making Things Work Together
Making Things Work TogetherMaking Things Work Together
Making Things Work Together
 
Django introduction
Django introductionDjango introduction
Django introduction
 
The goodies of zope, pyramid, and plone (2)
The goodies of zope, pyramid, and plone (2)The goodies of zope, pyramid, and plone (2)
The goodies of zope, pyramid, and plone (2)
 

Recently uploaded

Foreign Relation of Pakistan with Neighboring Countries.pptx
Foreign Relation of Pakistan with Neighboring Countries.pptxForeign Relation of Pakistan with Neighboring Countries.pptx
Foreign Relation of Pakistan with Neighboring Countries.pptxunark75
 
Political-Ideologies-and-The-Movements.pptx
Political-Ideologies-and-The-Movements.pptxPolitical-Ideologies-and-The-Movements.pptx
Political-Ideologies-and-The-Movements.pptxSasikiranMarri
 
Transforming Andhra Pradesh: TDP's Legacy in Road Connectivity
Transforming Andhra Pradesh: TDP's Legacy in Road ConnectivityTransforming Andhra Pradesh: TDP's Legacy in Road Connectivity
Transforming Andhra Pradesh: TDP's Legacy in Road Connectivitynarsireddynannuri1
 
Geostrategic significance of South Asian countries.ppt
Geostrategic significance of South Asian countries.pptGeostrategic significance of South Asian countries.ppt
Geostrategic significance of South Asian countries.pptUsmanKaran
 
Mitochondrial Fusion Vital for Adult Brain Function and Disease Understanding...
Mitochondrial Fusion Vital for Adult Brain Function and Disease Understanding...Mitochondrial Fusion Vital for Adult Brain Function and Disease Understanding...
Mitochondrial Fusion Vital for Adult Brain Function and Disease Understanding...The Lifesciences Magazine
 
13042024_First India Newspaper Jaipur.pdf
13042024_First India Newspaper Jaipur.pdf13042024_First India Newspaper Jaipur.pdf
13042024_First India Newspaper Jaipur.pdfFIRST INDIA
 
16042024_First India Newspaper Jaipur.pdf
16042024_First India Newspaper Jaipur.pdf16042024_First India Newspaper Jaipur.pdf
16042024_First India Newspaper Jaipur.pdfFIRST INDIA
 
Emerging issues in migration policies.ppt
Emerging issues in migration policies.pptEmerging issues in migration policies.ppt
Emerging issues in migration policies.pptNandinituteja1
 
Power in International Relations (Pol 5)
Power in International Relations (Pol 5)Power in International Relations (Pol 5)
Power in International Relations (Pol 5)ssuser583c35
 
11042024_First India Newspaper Jaipur.pdf
11042024_First India Newspaper Jaipur.pdf11042024_First India Newspaper Jaipur.pdf
11042024_First India Newspaper Jaipur.pdfFIRST INDIA
 
15042024_First India Newspaper Jaipur.pdf
15042024_First India Newspaper Jaipur.pdf15042024_First India Newspaper Jaipur.pdf
15042024_First India Newspaper Jaipur.pdfFIRST INDIA
 
14042024_First India Newspaper Jaipur.pdf
14042024_First India Newspaper Jaipur.pdf14042024_First India Newspaper Jaipur.pdf
14042024_First India Newspaper Jaipur.pdfFIRST INDIA
 
lok sabha Elections in india- 2024 .pptx
lok sabha Elections in india- 2024 .pptxlok sabha Elections in india- 2024 .pptx
lok sabha Elections in india- 2024 .pptxdigiyvbmrkt
 
12042024_First India Newspaper Jaipur.pdf
12042024_First India Newspaper Jaipur.pdf12042024_First India Newspaper Jaipur.pdf
12042024_First India Newspaper Jaipur.pdfFIRST INDIA
 

Recently uploaded (14)

Foreign Relation of Pakistan with Neighboring Countries.pptx
Foreign Relation of Pakistan with Neighboring Countries.pptxForeign Relation of Pakistan with Neighboring Countries.pptx
Foreign Relation of Pakistan with Neighboring Countries.pptx
 
Political-Ideologies-and-The-Movements.pptx
Political-Ideologies-and-The-Movements.pptxPolitical-Ideologies-and-The-Movements.pptx
Political-Ideologies-and-The-Movements.pptx
 
Transforming Andhra Pradesh: TDP's Legacy in Road Connectivity
Transforming Andhra Pradesh: TDP's Legacy in Road ConnectivityTransforming Andhra Pradesh: TDP's Legacy in Road Connectivity
Transforming Andhra Pradesh: TDP's Legacy in Road Connectivity
 
Geostrategic significance of South Asian countries.ppt
Geostrategic significance of South Asian countries.pptGeostrategic significance of South Asian countries.ppt
Geostrategic significance of South Asian countries.ppt
 
Mitochondrial Fusion Vital for Adult Brain Function and Disease Understanding...
Mitochondrial Fusion Vital for Adult Brain Function and Disease Understanding...Mitochondrial Fusion Vital for Adult Brain Function and Disease Understanding...
Mitochondrial Fusion Vital for Adult Brain Function and Disease Understanding...
 
13042024_First India Newspaper Jaipur.pdf
13042024_First India Newspaper Jaipur.pdf13042024_First India Newspaper Jaipur.pdf
13042024_First India Newspaper Jaipur.pdf
 
16042024_First India Newspaper Jaipur.pdf
16042024_First India Newspaper Jaipur.pdf16042024_First India Newspaper Jaipur.pdf
16042024_First India Newspaper Jaipur.pdf
 
Emerging issues in migration policies.ppt
Emerging issues in migration policies.pptEmerging issues in migration policies.ppt
Emerging issues in migration policies.ppt
 
Power in International Relations (Pol 5)
Power in International Relations (Pol 5)Power in International Relations (Pol 5)
Power in International Relations (Pol 5)
 
11042024_First India Newspaper Jaipur.pdf
11042024_First India Newspaper Jaipur.pdf11042024_First India Newspaper Jaipur.pdf
11042024_First India Newspaper Jaipur.pdf
 
15042024_First India Newspaper Jaipur.pdf
15042024_First India Newspaper Jaipur.pdf15042024_First India Newspaper Jaipur.pdf
15042024_First India Newspaper Jaipur.pdf
 
14042024_First India Newspaper Jaipur.pdf
14042024_First India Newspaper Jaipur.pdf14042024_First India Newspaper Jaipur.pdf
14042024_First India Newspaper Jaipur.pdf
 
lok sabha Elections in india- 2024 .pptx
lok sabha Elections in india- 2024 .pptxlok sabha Elections in india- 2024 .pptx
lok sabha Elections in india- 2024 .pptx
 
12042024_First India Newspaper Jaipur.pdf
12042024_First India Newspaper Jaipur.pdf12042024_First India Newspaper Jaipur.pdf
12042024_First India Newspaper Jaipur.pdf
 

Mashing Up The Guardian

  • 1. Mashing up Michael Brunton-Spall michael.brunton-spall@guardian.co.uk @mibgames
  • 2. About Us Online since 1995 250M+ pages per month 30M+ visitors per month
  • 4. 1999 - Guardian Unlimited
  • 5. 1999 - Removed the registration wall
  • 6. 2007 - Rebuild and Redesign
  • 7. 2007 - Rebuild and Redesign
  • 8. 2007 - Rebuild and Redesign
  • 10. The Hackable Guardian Url Hacking Keyword Combiners RSS Feeds
  • 11. Url Hacking http://www.guardian.co.uk/ [section]/[keyword] technology/internet [section]/all environment/all [publication]/[date]/[newspapersection] theguardian/2009/jun/11/technologyguardian profile/[name] profile/bobbiejohnson
  • 12. RSS Feeds RSS Everywhere! [section]/[keyword]/rss technology/internet/rss [section]/all /rss environment/all/rss [publication]/[date]/[newspapersection]/rss theguardian/2009/jun/11/technologyguardian/rss profile/[name]/rss profile/bobbiejohnson/rss
  • 14.
  • 15.
  • 16. Url Combiners A Logical AND Almost any combination from before technology/internet+profile/bobbiejohnson theguardian/technologyguardian+technology/internet Except things that aren't actual tags: Dated newspaper sections theguardian/2009/jun/11... + anything = 404 the all page .../all + anything = 404
  • 17. Problems with this approach Fragile No Discovery Not well documented Copyright Issues
  • 18. Build applications with the Guardian
  • 19. Open Platform Opening up how we work with people both internally and externally A suite of services enabling partners to build applications with the Guardian Content API Data Store
  • 20. Data Store A directory of useful data curated by Guardian editors
  • 21. Data Store A directory of useful data curated by Guardian editors
  • 24. Content API A service for selecting and collecting content from the Guardian for re-use
  • 25. Content API A service for selecting and collecting content from the Guardian for re-use
  • 26. Content API Search content
  • 27. Content API Xml and Json
  • 28. Content API Tag Metadata
  • 29. Content API Full Article Body
  • 30. Content API Filters
  • 31. So why? guardian.co.uk has an amazing amount of quality content Incredible amounts of meta-data, curated by editors Aim to allow the guardian to become a rich source of facts and journalism for the web
  • 32. How? Backed of our search platform Provides access to 10 years of article content and metadata Supports multiple output formats: XML, JSON, ATOM Supported free text search across content Search for keywords Guardian supported api projects in Java, PHP, Python and Ruby Community supported api projects in Perl, ActionScript and Coldfusion (and probably more?)
  • 34. Pricing FREE!
  • 35. How free is free? You can publish full articles from the guardian on your website 5k queries per day limit 24 hour maximum cache lifetime Online support Partner with us on advertising
  • 36. Beta trial Limited number of keys Collecting feedback Will open more widely at end of beta program
  • 37. How do I use it? Home page http://www.guardian.co.uk/open-platform Sign up for an API key at http://guardian.mashery.com Use the API Explorer at http://api.guardianapis.com/docs/ Use the python library at http://code.google.com/p/openplatform-python/
  • 38. What can I do with it? Search our tag hierarchy Find content by tag Find content by search terms Display on your website!
  • 40. MP's Expenses Written in Django Hosted on EC2 Easy to modify
  • 41. MP's Expenses An MP's Page
  • 42. View def mp(request, id): mp = get_object_or_404(MP, pk = id) ... return render(request, 'mp.html', { 'mp': mp, 'documents': mp.documents.all(), 'top_users': top_users[:5], })
  • 43. View def mp(request, id): mp = get_object_or_404(MP, pk = id) ... return render(request, 'mp.html', { 'mp': mp, 'documents': mp.documents.all(), 'top_users': top_users[:5], })
  • 44. View def mp(request, id): mp = get_object_or_404(MP, pk = id) ... return render(request, 'mp.html', { 'mp': mp, 'documents': mp.documents.all(), 'top_users': top_users[:5], })
  • 45. Using the Guardian API Getting all articles about an MP results = client.search(q='"%S"' % (name)) returns a paginating iterator for x in results: print x['headline'] Shows only first 10 by default
  • 46. Get the client def mp(request, id): mp = get_object_or_404(MP, pk = id) ... client = Client(settings.GUARDIAN_APIKEY) return render(request, 'mp.html', { 'mp': mp, 'documents': mp.documents.all(), 'top_users': top_users[:5], })
  • 47. Make the request def mp(request, id): mp = get_object_or_404(MP, pk = id) ... client = Client(settings.GUARDIAN_APIKEY) return render(request, 'mp.html', { 'mp': mp, 'documents': mp.documents.all(), 'top_users': top_users[:5], 'articles': client.search(q='"%s"'%(mp.name)), })
  • 48. Update the template <div id="about-mp"> <ul> {% for article in articles %} <li> <a href="{{article.webUrl}}">{{article.headline}}</a> </li> {% endfor %} </ul> </div>
  • 50. Additions? Use Memcached to cache the response from guardian API Don't create a new client each time Article bodies
  • 51. Who else is using it?
  • 52. Who else is using it?
  • 53. Who else is using it?
  • 54. Who else is using it?
  • 55. Public API Key? jbynv3fwdp8ju5625mt2axw3 Only 5k queries a day Will be closed on Monday 6th July Will be closed if abused Strongly encourage you to sign up for your own key