Web 2.0 & Mashups: How People can Tap into the "Grid" for Fun & Profit

Web 2.0 Mashups : How People Can Tap into the "Grid" for Fun & Profit Pamela Fox USC/Google http://www.slideshare.net/wuzziwug/web-20-mashups-how-people-can-tap-into-the-grid-for-fun-profit/

What’s Web2.0?

“ Web 2.0, a phrase coined by O'Reilly Media in 2004, refers to a supposed second-generation of Internet-based services such as social networking sites, wikis, communication tools, and folksonomies that let people collaborate and share information online in previously unavailable ways.” - Wikipedia

It’s all about the user…

The users define the content, rate (“digg”) the content, review the content, create the content, share the content

What the user wants, the user gets

The Web isn’t ruled by the relatively static “What’s cool!” directories of the early ’90s

Now, a website can go from 0 to thousands of hits in 1 day, if other users discover & share it

Case in point: The Profane Game. Gadget got blogged about Sunday morning, had 6,000 players by day’s end. (compared to 20 the day before)

So what’s a mashup anyway?

“ A mashup is a website or application that combines content from more than one source into an integrated experience. Content used in mashups is typically sourced from a third party via a public interface or API . Other methods of sourcing content for mashups include Web feeds (e.g. RSS or Atom ) and JavaScript .” – Wikipedia

A mashup is the ultimate user-generated content: user likes data source A, data source B, & puts them together how they like.

* There are also music & video mashups.

Some stats about me…

In past 3 months, I’ve entered 3 mashup contests and placed in all of them (1 st /2 nd /3 rd ) - $2000 in winnings, and a Google bean bag… not bad for 3am hacks!

On programmableweb.com, only 2 out of 217 developers have registered more mashups than me

I now work for Google on the Web API Support team (and I’ve used all of their APIs except AdSense)

Before July 2006 (8 months ago), I’d never used a Web API

How did I get started in Web 2.0?

Or, more appropriately…

How did I not get started in Web 2.0?

By watching Geoffrey Fox’s SOAP lectures (Sorry, dad!)

Why not?

It could be that I’m dumb, and I’ve entertained that idea many times, but…

It could be that SOAP/ WS-* isn’t a very attractive entry point for the casual or novice user

What scared me about SOAP?

Standards, protocols, standards, formats

I’ve had a sequence of slides ingrained in my head since a talk I sat on by Geoffrey for USC AIST in 2004

I dug them up for you.. Let’s flip through them quickly

A List of Web Services I

a) Core Service Architecture

XSD XML Schema (W3C Recommendation) V1.0 February 1998, V1.1 February 2004

WSDL 1.1 Web Services Description Language Version 1.1, (W3C note) March 2001

WSDL 2.0 Web Services Description Language Version 2.0, (W3C under development) March 2004

SOAP 1.1 (W3C Note) V1.1 Note May 2000

SOAP 1.2 (W3C Recommendation) June 24 2003

b) Service Discovery

UDDI (Broadly Supported OASIS Standard) V3 August 2003

WS-Discovery Web services Dynamic Discovery (Microsoft, BEA, Intel …) February 2004

WS-IL Web Services Inspection Language, (IBM, Microsoft) November 2001

A List of Web Services II

c) Security

SAML Security Assertion Markup Language (OASIS) V1.1 May 2004

XACML eXtensible Access Control Markup Language (OASIS) V1.0 February 2003

WS-Security 2004 Web Services Security: SOAP Message Security (OASIS) Standard March 2004

WS-SecurityPolicy Web Services Security Policy (IBM, Microsoft, RSA, Verisign) Draft December 2002

WS-Trust Web Services Trust Language (BEA, IBM, Microsoft, RSA, Verisign …) May 2004

WS-SecureConversation Web Services Secure Conversation Language (BEA, IBM, Microsoft, RSA, Verisign …) May 2004

WS-Federation Web Services Federation Language (BEA, IBM, Microsoft, RSA, Verisign) July 2003

A List of Web Services III

d) Messaging

WS-Addressing Web Services Addressing (BEA, IBM, Microsoft) March 2004

WS-MessageDelivery Web Services Message Delivery (W3C Submission by Oracle, Sun ..) April 2004

WS-Routing Web Services Routing Protocol (Microsoft) October 2001

WS-RM Web Services Reliable Messaging (BEA, IBM, Microsoft, Tibco) v0.992 March 2004

WS-Reliability Web Services Reliable Messaging (OASIS Web Services Reliable Messaging TC) March 2004

SOAP MOTM SOAP Message Transmission Optimization Mechanism (W3C) June 2004

e) Notification

WS-Eventing Web Services Eventing (BEA, Microsoft, TIBCO) January 2004

WS-Notification Framework for Web Services Notification with WS-Topics , WS-BaseNotification , and WS-BrokeredNotification (OASIS) OASIS Web Services Notification TC Set up March 2004

JMS Java Message Service V1.1 March 2002

A List of Web Services IV

f) Coordination and Workflow, Transactions and Contextualization

WS-CAF Web Services Composite Application Framework including WS-CTX , WS-CF and WS-TXM below (OASIS Web Services Composite Application Framework TC) July 2003

WS-CTX Web Services Context (OASIS Web Services Composite Application Framework TC) V1.0 July 2003

WS-CF Web Services Coordination Framework (OASIS Web Services Composite Application Framework TC) V1.0 July 2003

WS-TXM Web Services Transaction Management (OASIS Web Services Composite Application Framework TC) V1.0 July 2003

WS-Coordination Web Services Coordination (BEA, IBM, Microsoft) September 2003

WS-AtomicTransaction Web Services Atomic Transaction (BEA, IBM, Microsoft) September 2003

WS-BusinessActivity Web Services Business Activity Framework (BEA, IBM, Microsoft) January 2004

BTP Business Transaction Protocol (OASIS) May 2002 with V1.0.9.1 May 2004

BPEL Business Process Execution Language for Web Services (OASIS) V1.1 May 2003

WS-Choreography (W3C) V1.0 Working Draft April 2004

WSCI (W3C) Web Service Choreography Interface V1.0 (W3C Note from BEA, Intalio, SAP, Sun, Yahoo)

WSCL Web Services Conversation Language (W3C Note) HP March 2002

A List of Web Services V

h) Metadata and State

RDF Resource Description Framework (W3C) Set of recommendations expanded from original February 1999 standard

DAML+OIL combining DAML (Darpa Agent Markup Language) and OIL (Ontology Inference Layer) (W3C) Note December 2001

OWL Web Ontology Language (W3C) Recommendation February 2004

WS-DistributedManagement Web Services Distributed Management Framework with MUWS and MOWS below (OASIS)

WSDM-MUWS Web Services Distributed Management: Management Using Web Services (OASIS) V0.5 Committee Draft April 2004

WSDM-MOWS Web Services Distributed Management: Management of Web Services (OASIS) V0.5 Committee Draft April 2004

WS-MetadataExchange Web Services Metadata Exchange (BEA,IBM, Microsoft, SAP) March 2004

WS-RF Web Services Resource Framework including WS-ResourceProperties , WS-ResourceLifetime , WS-RenewableReferences , WS-ServiceGroup , and WS-BaseFaults (OASIS) Oasis TC set up April 2004 and V1.1 Framework March 2004

ASAP Asynchronous Service Access Protocol (OASIS) with V1.0 working draft G June 2004

WS-GAF Web Service Grid Application Framework (Arjuna, Newcastle University) August 2003

A List of Web Services VI

g) General Service Characteristics

WS-Policy Web Services Policy Framework (BEA, IBM, Microsoft, SAP) May 2003

WS-PolicyAssertions Web Services Policy Assertions Language (BEA, IBM, Microsoft, SAP) May 2003

WS-Agreement Web Services Agreement Specification (GGF under development) May 2004

i) User Interfaces

WSRP Web Services for Remote Portlets (OASIS) OASIS Standard August 2003

JSR168 : JSR-000168 Portlet Specification for Java binding (Java Community Process) October 2003

And thus, I repressed SOAP

It was depressing to think how little I understood what the hell he was talking about, so I resolved to put it out of my head until I had a good few weeks/months/years to sort through it all

And then my web2.0 awakening began…

I was working on a survey about the similarity of video games, and wondered if I could compare our human results to Amazon data

It was time to get over my fear of web services...

I tried enlisting the help of my boyfriend, an employee of AWS, but he was busy coding

So I sucked it up and googled how to do it-

And all it took was ONE URL!

Putting ONE URL in the browser would report back the similarity for a product

And thus I discovered the simplicity of REST*, and the addiction began

*pseudo-REST. Google it.

My first Mashup, by Pamela Fox, Age 22

The Similarity Web (completed a few days after discovery of the one URL)

Wow, my little mashup sure increased my site traffic

With 1 URL, Amazon had given me a simple tool that let me share a new way of exploring products with thousands of consumers

Thank you, Amazon! (for the referral fees, too)

Let the mashup madness begin! Note: Most of these were coded at 3am or while waiting for homework to compile.

Why is mashing up so damn fun?

I’ll admit - find myself daydreaming about new mashups I can make every day

Mashing is fast, especially once you’ve done a few – I’ve got enough PHP/flash/python scripts now that it doesn’t take long to adapt them to a slightly different purpose

Mashing is fun – you can share your mashup with the online world (not just the geeks!), and get feedback instantly

The research community isn’t usually like that  - must wait for papers to be published, research to be polished, etc., etc.

Because of the various mashups I’ve posted on my blog, I’ve gotten comments from writers, linguists, game designers, businessmen

So what makes mashups so cool?

"We know we don't have a corner on creativity. There are creative people all around the world, hundreds of millions of them, and they are going to think of things to do with our basic platform that we didn't think of. So the mashup stuff is a wonderful way of allowing people to find new ways of applying the basic infrastructures we're propagating.” – Vint Cerf (via Wikipedia)

Diagram from http:// blogs.zdnet.com/Hinchcliffe/?p =63

Basically…

You provide the API to access the structured information, and we find new and exciting ways to use the information (free labor!)

Ref: “Why Google Loves the Little Guys”

http://www.forbes.com/technology/2006/09/17/mashup-search-google-tech-cz_qh_0918google.html

Mashup technologies: AJAX

AJAX: Asynchronous javascript + XML

Before AJAX: Every time the user wanted to interact with the server, they had to press a button and reload the page

AJAX is what allows the user to stay on one page but continually see new information upon interaction with the page

The user presses a button/changes a parameter, the javascript sends a request to the server and assigns it a callback function

Once server has returned response, callback function takes care of it

AJAX uses the XMLHttpRequest/Microsoft.XMLHTTP object

Unfortunately IE/FF couldn’t decide on one

Many Javascript libraries provide wrappers that generalize the object to be cross-browser compatible

A basic AJAX example using GMaps API:

GDownloadUrl("data.xml", function(data, responseCode) { var xml = GXml.parse(data);

var markers = xml.documentElement.getElementsByTagName("marker"); for (var i = 0; i < markers.length; i++) {

var point = new GLatLng(parseFloat(markers[i].getAttribute("lat")), parseFloat(markers[i].getAttribute("lng")));

map.addOverlay(new GMarker(point)); } });

Users like AJAX-y pages because it feels fast to not reload the page, but it can be problematic as it doesn’t have built in history/bookmark-ability

Perfect for dynamic customization of search results, etc.

As long as there’s some portion of the page the user’s interested in, it’s OK for other parts to be loading

The term AJAX is also used incorrectly to describe nifty visual effects commonly used on web2.0 sites (actually just DHTML, as there’s no XML involved)

Ubiquity of the ajax loading gif: http://www.ajaxload.info/

Mashup technologies: JSON

JSON: JavaScript Object Notation

Solves the cross-site scripting restriction inherent in AJAX

JavaScript can’t make requests that aren’t on its own server

Script includes *can* reference javascripts on other servers, and those javascripts can inject variables/functions into the website that it can then use

So mashups can be completely client-side ! (but usually always involve server-side processing)

Mashup technologies: JSON

JSON is similar to XML in it’s hierarchical structure, but doesn’t require any non-native Javascript functionality as it actually is an injected JavaScript object

A basic JSON example:

<script>

function listEntries(root) {

var feed = root.feed;

var entries = feed.entry || [];

for (var i = 0; i < feed.entry.length; ++i) {

var entry = feed.entry[i];

var title = entry.title.$t;

var content = entry.content.$t;

var link = entry.link[0].href;

html.push('<li>','<a target="_blank" href="', link, '" >', title, '</a>','</li>');

}

document.getElementById("agenda").innerHTML = html.join("");

}

</script>

<script src="http://uscgamepipeinterviews.blogspot.com/feeds/posts/default?alt=json-in-script&callback=listEntries"> </script>

Mashup Technologies: PHP, Ruby, Python, Perl

Many web-friendly scripting languages are used for interacting with web services

Even better are the languages that natively interact with mySQL databases (e.g. PHP, ruby)

Often-times developers build wrappers over web APIs in the scripting language to make them even easier to use

E.g. I just created a Python wrapper for the Google Spreadsheets API

The flow of a mashup: Google Gadget

Gadgets are mini-apps that are easy to share &embed

Gadgets are often used to provide quick views of information from other sites

WordHunter Xtreme, diagrammed below, won “Most Addictive” in Google Gadget University Awards

The flow of a mashup: Google Map

MashPlanet is a mashup of map mashups (say that 10x fast)

The Web2.0 is seeing an explosion of geo-tagged data and maps (we can visualize data as more than just a list now!)

Google Maps API is the most used of all web APIs currently

Google Maps API Google group has 17822 members & gets average 150 messages/day – majority of these are answered within a few hours (and not usually by Google employees)

Why? JavaScript may not be the cleanest language, but its client-side and that makes it easy for others to debug

The Mashup/Web2.0 Community

Generally, everyone’s happy & quick to link up with everyone else (networking never does anyone any harm)

We’re all in it together philosophy. API providers in constant interaction with their users. It can only make it better.

Conferences:

Mashup Camp: The unconference for the uncomputer ( http:// www.mashupcamp.com / )

BarCamp, Where2.0, etc.

Directories:

http://www.programmableweb.com (THE directory)

http:// www.emilychang.com/go/ehub /

Blogs:

http://web2.wsj2.com/ (Web2.0, great diagrams/analyses)

http://www.mashable.com/ (Social networking)

http://www.readwriteweb.com (Products/debates)

http://www.techcrunch.com (Startups)

Mashup Types

There ~1500 mashups registered on programmableweb.com, but most have the same purpose: helping web-surfers discover and sift through information in new and useful ways

The web’s always been about sharing information, and it still is.

Some basic types: aggregators, search aggregators, visualizers, mobile, maps, games

Mashup type: Aggregators

Combine feeds/info from various related websites into one site.

More info + less clicks = happier websurfer

Examples: http://www.originalsignal.com , http://doggdot.us , http://reader.google.com

Mashup Types: Search/search aggregators

Search: Let you search an API’s data

Search aggregators: Let you search once, find info from many search engines/APIs at once.

Examples: http://pulpsite.net/zontube/ , http://imagine-it.org/google/musicmash.htm

Mashup Types: Visualizers

Visualizers: Take related data and visualize in a new and meaningful way (e.g. clouds, maps)

Examples: http://imagine-it.org/amazong/vissimweb.htm , http://www.coverpop.com , http://mathias.cianci.free.fr/

Mashup Types: Maps/Earth

Take data with geo info from other sources (including users!) and plot on map

Geographical info could be latitude/longitude, could just be an address, city or place name as geocoding APIs are quite common

Examples: http://www.81nassau.com/apnews/ , http://www.bikely.com , http://imagine-it.org/mashplanet/

Mashup Types: Mobile

Take online data from APIs/feeds and put in mobile-friendly format

Increasingly needed as many fancy web2.0 websites just won’t work on cell phones (AJAX) but people still want their information, and quick!

Examples: http://www.411sync.com/cgi-bin/search_api?query=mydigg+txttester , http://www.411sync.com/cgi-bin/search_api?query=daily , http://www.frucall.com

Mashup Types: Games

Make players guess more data about a web object (photo, friend, map clue)

My personal favorite type 

Examples here: http://imagine-it.org/flickr/PhotoMunchrs.html , http://imagine- it.org/google/wordhunter.htm , http://imagine-it.org/flickr/flicktionary.htm , http:// www.facebook.com

Mashup Types: Other!

Obligatory slide to say that there are other types of mashups

Also easy way to talk about my work with ClearForest Semantic Web Services

Newsmakers of the Day:

Sends news articles from CNN to be analyzed by ClearForest

Retrieves entity names, finds the most relatively popular entities of the day

Displays them as a daily calendar item

Motivates me to read the news!

Won 3 rd place in their contest

Most contest entries did similar thing: navigate news by entity, not title

The Future of Mashing

Currently mashing up is for the programmers and the people-not-afraid-to-try-programming, but that’s changing

Visual development environments let non-techies connect APIs together, style the output, and embed in various formats

It’s not fair for techies to have all the mashing fun!

E.g. OpenKapow.com

The APIs Making it Happen

Let’s give a hand to all the Web APIs letting mashups happen

Programmableweb.com is a site tracking mashups and APIs

363 registered Web APIs

APIs per Protocol Distribution REST SOAP XML-RPC REST, XML-RPC REST, XML-RPC, SOAP REST, SOAP JS Other google maps netvibes live.com virtual earth google search amazon S3 amazon ECS flickr ebay youtube 411sync del.icio.us yahoo! search yahoo! geocoding technorati yahoo! images trynt yahoo! local - Analysis of the API protocols offered vs. mashup count on programmableweb.com

What, no API available?

Then make one!

Feed dapper.com example pages from the site you’d like to screen-scrape, select the page elements you’re interested in, and presto – You’ve got a custom API!

Note: may be illegal to screen-scrape data from some sites. Oops. 

Suggestions for (current/would-be) API Providers

Provide a REST API or Javascript API, depending on what’s more appropriate for your API, to attract casual users

Continue providing SOAP API for other users

= Best of both worlds!

Quote from user trying to use ClearForest SOAP API when first released:

“ How about a REST interface or at least a simpler web interface with a GET or POST form (minus the frames). This would be a preferable option for many mashup environments, compared to SOAP.”