Open Data,
Visualization &
Usability for
Online News Delivery
a presentation by Mohan & Sze
1

Background
2

Who are we?
• Lack of structured access to information on
local news sites
• POV of two technologists whose interests in
technology and current issues/activism
overlap
• Propagation of more aware and savvy
communities
3

Questions
4

How many of you…
• know HTML and/or CSS?
• know Javascript?
• know server side programming?
• maintain a news site / website?
• monetize your content?
5

Open data
Visualisation
6

Your data wants to be free
7

Why?
8

Why?
You cannot define how people will consume your
information, leverage them
• Don’t turn your user base into pirates

Why?
Monetization cannot solely be a result of exclusivity
but a result of mass access and authority
• The more people use your data, the more
valuable you become (advertising)
• How are you making your data (not just website)
more accessible

The NY Times said the project had met expectations, drawing 227,000
paying subscribers — out of 787,000 over all — and generating about
$10 million a year in revenue.
“But our projections for growth on
that paid subscriber base were low,
compared to the growth of online
advertising...”
“...what wasn’t anticipated was the explosion in
how much of our traffic
would be
generated by Google, by Yahoo and
some others.” Vivian L. Schiller, senior VP and GM of NYTimes.com

Why?
If you ain’t the top dog, use openness as a way to
partner, grow market share and relevance
• Even Google does it - OpenSocial
• Leverage exiting platforms - don’t roll another
SNS

Why ?
No matter how smart you are, there are smarter
people out there
• Do you want them to be your competitor or
partner
• Who will be making your killer Facebook
application ?

How?
17

How
• Provide a way to access data without
presentation clutter. Turn your sites into
information services
• Make it structured & machine readable - Web APIs,
Microformats

How
• Provide simple ways to export and import data
into your system
• Syndication, public wikis

How
• Provide ways to simplify the querying of your
indexed data
• Opensearch, Better back end search

Microformats
21

Microformats
• Small bits of HTML that represent things like people,
events, tags, etc. in web pages. (annotations)
• These small bit of standardized markup adds
semantic meaning to data in a page in a non-intrusive
and machine readable manner, data becomes more
structured
• A bottom up approach - community defines what and
how the markup will be

Let’s jump right in!
23

<table class=\"contentpaneopen\">
<tbody>
<tr>
<td colspan=\"2\" class=\"createdate\" valign=\"top\">Wednesday, 28 May 2008
</tr>
<tr>
<td colspan=\"2\" valign=\"top\">
<p><img src=\"forum_location_files/pjmap.jpg\" alt=\"Image\" title=\"Image\" border=\"0\"
height=\"367\" hspace=\"6\" width=\"497\"></p>
<p>Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E.
Malaysia </p>
<p>Hotel Information: <a href=\"http://www.hspj.com.my/HSGS/index.php\">Singgahsana Hotel
</a>,&nbsp;<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's Village Hotel</
a></p>
</td>
</tr>
<tr>
<td colspan=\"2\" class=\"modifydate\" align=\"left\">Last Updated ( Friday, 06 June 2008 )</td>
</tr>
</tbody>
</table>

<table class=\"contentpaneopen\">
<tbody>
<tr>
<td colspan=\"2\" class=\"createdate\" valign=\"top\">Wednesday, 28 May 2008
</tr>
<tr>
<td colspan=\"2\" valign=\"top\">
<p><img src=\"forum_location_files/pjmap.jpg\" alt=\"Image\" title=\"Image\" border=\"0\"
height=\"367\" hspace=\"6\" width=\"497\"></p>
<p>Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E.
Malaysia </p>
<p>Hotel Information: <a href=\"http://www.hspj.com.my/HSGS/index.php\">Singgahsana Hotel
</a>,&nbsp;<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's Village Hotel</
a></p>
</td>
</tr>
<tr>
<td colspan=\"2\" class=\"modifydate\" align=\"left\">Last Updated ( Friday, 06 June 2008 )</td>
</tr>
</tbody>
</table>

<div class=\"vcard\">
<div class=“vcard”>
Shah Village Hotel
Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya,
Selangor D.E. Malaysia
No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia
</div>
</div>
<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's
<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's Village
Village Hotel</a>
Hotel</a>

<div class=\"vcard\">
<div class=“vcard”>
<div class=\"fn org\"> Shah Village Hotel</div>
Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya,
Selangor D.E. Malaysia
No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor D.E. Malaysia
</div>
</div>
<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's
<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's Village
Village Hotel</a>
Hotel</a>

<div class=\"vcard\">
<div class=“vcard”>
<div class=\"fn org\">Shah Village Hotel</div>
Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya,
Selangor D.E. Malaysia
<div class=\"adr\"> No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya, Selangor
D.E. Malaysia
</div>
</div>
<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's
Village Hotel</a>
</div>
<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's Village
Hotel</a>

<div class=“vcard”>
<div class=\"vcard\">
<div class=\"fn org\">Shah Village Hotel</div>
Shah Village Hotel<br>No. 3 &amp; 5 Lorong Sultan , 46200 Petaling Jaya,
<div class=\"adr\">
Selangor D.E. Malaysia
<div class=\"street-address\"> No. 3 &amp; 5 Lorong Sultan </div>,
<span class=\"postal-code\"> 46200 </span>
</div> class=\"locality\"> Petaling Jaya </span>,
<span
<span class=\"region\"> Selangor D.E. </span>
<span class=\"country-name\"> Malaysia </span>
<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's
</div>
</div> Hotel</a>
Village
<a href=\"http://www.shahsresorts.com/petalingjaya/default.htm\">Shah's Village
Hotel</a>

<div class=“vcard”>
<a href= \"http://www.shahsresorts.com/petalingjaya/default.htm\" class=“url fn
org”>Shah Village Hotel</a>
<div class=\"adr\">
<div class=\"street-address\"> No. 3 &amp; 5 Lorong Sultan </div>,
<span class=\"postal-code\"> 46200 </span>
<span class=\"locality\"> Petaling Jaya </span> ,
<span class=\"region\"> Selangor D.E. </span>
<span class=\"country-name\"> Malaysia </span>
</div>
</div>

<div class=“vcard”>
<abbr title=\"3.102233488513198;101.64600670337677\"class=\"geo\">
<a href= \"http://www.shahsresorts.com/petalingjaya/default.htm\" class=“url fn
org”>Shah Village Hotel</a>
</abbr>
<div class=\"adr\">
<div class=\"street-address\"> No. 3 &amp; 5 Lorong Sultan </div>,
<span class=\"postal-code\"> 46200 </span>
<span class=\"locality\"> Petaling Jaya </span> ,
<span class=\"region\"> Selangor D.E. </span>
<span class=\"country-name\"> Malaysia </span>
</div>
</div>

Second example
32

Why?
• Search engine support
• http://kitchen.technorati.com/search/
• google creative commons search
• yahoo search support

Why?
• Machine readable
• Cheap API - Low barrier for community to build
add ons to your site tools in (Javascript - FF3 ,
PHP etc.)
34

• End user application support (Firefox3 with
extension,IE8 OpenService - WebSlices , soon more
apps)
35

How to incorporate
Integrate into news sites:
•Listings of classifieds - hListings
•News stories - markup with hAtom, markup
authors,personalities and organizations with hCard,
markup locations with geo and adr
•Reviews of products,restaurants, movies - hReview
e.g. http://www.australianit.news.com.au/contactus

How to incorporate
Build tools internally that leverage the
markup
•Internal search engine that is markup aware for your
site
•Build visualisation tools/widgets that leverage the
markup

How to incorporate
Build communities that leverage the markup
•Have a developer section on your site

Produce
• DreamweaverExtension - http://www.webstandards.org/action/dwtf/microformats
• Textpattern Microformats Plugin (http://placenamehere.com/TXP/pnh_mf/)
• Wordpress Microformat plugins - http://microformats.org/wiki/wordpress
• TYPO3 - tt_address , TIMTAB Weblog plugins
• Joomla - No support
• Drupal - Drupal Upcoming.org syndication module-
http://hybernaut.com/upcoming-hcal - Weak support
• PHP Lib - http://enarion.net/phpmicroformats/ (Note: Search for other
language libs to generate microfrmats)

Consume:
• Client end
• Microformat bookmarklet - http://leftlogic.com/lounge/articles/
microformats_bookmarklet
• Firefox add on - Operator-http://www.kaply.com/weblog/operator/
• Server end
• Python based microformat parser - http://code.google.com/p/aump/
• PHP based microformat parser - http://code.google.com/p/hkit/
• TransFormr -A Simple set of XSLT and PHP tools for Transforming
• Microformats - http://code.google.com/p/transformr/
• http://suda.co.uk/projects/x2v - webservices

Challenges:
• Weak tool-ing for production in open source
components such as Joomla
and Drupal

Syndication
Access of content + metadata through feeds. A data API for your site.

Why?
Users will consume your data in the way they find most
comfortable. Syndication allows developers/users to
build the right method for themselves. (Mobile readers,
Screen readers). Bottom line - increase in circulation.
Most Malaysian news sites do it badly

Why?
Additional metadata contained within a feed
allows for remixing and discovery. Categories and tags
(metadata) should be taken seriously. (Technorati,
wordpress,yahoo pipes )

Why?
Allows for 3rd party applets/widgets to be built on
top of your content

How:
• Integrate into news sites:
• Syndicate everything on your site
• All sections , all columns ,search results, listings
(classifieds etc.)

Produce
• Drupal Atom module - http://drupal.org/project/atom - Overall good support
• Joomla - DS-Syndicate-http://extensions.joomla.org/component/option,com_mtree/
task,viewlink/link_id,2497/Itemid,35/
(Only Joomla 1.0 though) - Bad support
• Wordpress - Good support
• Universal Feed Generator -
http://www.phpclasses.org/browse/package/4427.html - Roll your own

Consume
•Simplepie PHP library - http://
simplepie.org/development/
•Drupal - FeedAPI - http://
drupal.org/project/feedapi

Best practice:
•Use Atom 1.0
•Cache, cache, cache (Support conditional GETs, Slugs etc.)
•Autodiscover links, on all pages - leverage support in browsers
•Use service to redistribute - such as Feedburner

Challenges:
Monetization:
If people aren't returning to my
site for content, how do I
monetize ?

3.0 OpenSearch
A collection of simple formats that help search
engines share their search results.

Why?
Browser support is already available
• Firefox, IE7 onwards.

Why?
Leverage search aggregation, deep search
• Growing field - A9, Technorati, MS Search Server
2008 OpenSearch

Why?
Weak search access results in searching through Google
or Yahoo?
• Your customers are looking for your data on
other sites - hence you are loosing usage metrics
as well as monetization opportunities.

How
• Integrate into news sites:
• Make search a 1st class interface to your website.
• Why are you not using this browser UI element ?

62

How
• Ensure everything is indexed and can be searched
• Elevate search from an afterthought into a first
class access tool of your site
63

Produce
• Textmate - wet_opensearch plugin
• Drupal - OpenSearch Plugin - - http://drupal.org/
project/opensearchplugin
• Online creator tool - http://ready.to/search/en/
• Wordpress - plugin - http://wordpress.org/extend/
plugins/open-search-document/

Consume
Drupal - OpenSearch client - http://drupal.org/project/
opensearchclient

Best practice
1. Use autodiscovery links
2. Write informative OpenSearch description documents
3. Provide atom + microformat + extensions (geo) based
search results
where applicable

Open data
Visualisation
67

Bad design obscures.
Good design reveals.
68

So, what’s the deal with
design?
• Design is about constraints – understanding,
respecting and breaking.
• An effective design allows users to get what
they’re looking for in an easy, trusted and fast
manner.
69

Traditional vs social media
How are they different?
70

71

72

1. Dynamic content (static vs changeable)
2. Interactivity (comments)
3. Relevance / Popularity (unique visits, comments, linkbacks)
4. Accessibility (archives)
5. Mashup (mix media, mix services – ads, widgets)
6. Creative freedom
7. Continuous presence
8. Reusable (syndicatable, linkable, tagable)
73

Social media = interactions
between people using
applications / interfaces
74

Why?
• improve the interactions between audience
• prioritize information – hierarchical structure –
popularity, relevance, recency et. al.
• allow users to reorganise and remix content and
represent content in a manner they desire.
75

SIMILE Timeline
a Javascript-based widget for visualizing time-based information
76

Why?
• linear narrative – visual representation of
chronological events
• allows users to be able to appreciate and interact
with news in a more fluid, continuous form – better
context to events & stories
• Mashup – information can be easily extracted from
different sources (e.g. http://www.bbc.co.uk/history/
british/launch_tl_british.shtml )
77

How
• http://simile.mit.edu/timeline/
• Timeline is similar to Google Maps: zero
application installation, server-side/client-side
• Data is easily populated into Timeline using: RDF,
RSS, JSON, Atom, XML
78

<html>
<head>
...
<script src=\"http://simile.mit.edu/
timeline/api/timeline-api.js“ type=\"text/
javascript\"> </script>
...
</head>
<body>
...
</body>
</html>
79

<html>
<head>
...
<script src=\"http://simile.mit.edu/
timeline/api/timeline-api.js“ type=\"text/
javascript\"> </script>
...
</head>
<body>
...
</body>
</html>
80

81

Timestamping
a Javascript-based widget for visualizing time-based information
82

Why?
• Video markers create easy transitions for
readers to move between different
paragraphs of a news article and its
corresponding video clip
• Limited time - short attention span – users
has the freedom to view specific portions of a
clip
83

How:
• Interactive markers – viddler.com allows
users to
1. attach comment markers at any point of
the video clip;
2. include tags in the video timeline to
describe elements in the video clip
84

85

86

87

88

Google Maps
Building humane interfaces by respecting standards and
habituation
89

Why?
• Topographical view of how news is scattered allows
for readers to focus on hyper local content
• Maps also serve as infographics to augment readers’
experience in relating or understanding news,
especially if there is a location specific context e.g.
Cyclone Nargis (http://www.nytimes.com/interactive/
2008/05/08/world/20080508_MYANMAR.html)
90

How:
• zero application installation, server-side/
client-side
• data is easily populated with XML
91

92

Summary
• Human readable also machine readable
• Better manipulate the data - move towards machines
autonomously aggregating, remixing and manipulating
data
• Make your data accessible - make yourself important
• Monetize by selling attention -but don’t lose relevance
• Data + Visualization = More attention

Contact details
• Mohan
• mohangk@gmail.com
• Sze
• szeying@gmail.com