YQL: Select * from internet - Campus Party 2010

Yahoo Query Language select * from internet Pedro Valente @pedrovalente

http://meme.yahoo.com

Hoje vou fazer o seguinte:

Hoje vou fazer o seguinte: • Explicar o que é YQL

Hoje vou fazer o seguinte: • Explicar o que é YQL • Dar exemplos de como funciona

Hoje vou fazer o seguinte: • Explicar o que é YQL • Dar exemplos de como funciona • Mostrar onde aprender mais

Hoje vou fazer o seguinte: • Explicar o que é YQL • Dar exemplos de como funciona • Mostrar onde aprender mais • Dar dois recados que interessam a vocês! Mas só no ﬁnal :)

De um lado, temos desenvolvedores

http://farm4.static.ﬂickr.com/ 3643/3516125392_8583644c56_b.jpg

Do outro lado, um monte de sites e serviços

Juntando desenvolvedores com sites e serviços

Juntando desenvolvedores com sites e serviços surgem Mashups (aplicações)

Aplicação =

manipulação de dados Aplicação =

manipulação de dados Aplicação = + visualização de dados

var map = new YMap (document.getElementById('map')); … dev var currentGeoPoint = new YGeoPoint ( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint); …

var map = new YMap (document.getElementById('map')); … dev var currentGeoPoint = new YGeoPoint ( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint); … … if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … } …

var map = new YMap (document.getElementById('map')); … dev var currentGeoPoint = new YGeoPoint ( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint); … … if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … } … http://search.yahooapis.com/ ImageSearchService/V1/imageSearch? appid=YahooDemo&query=Corvette&results=2

var map = new YMap (document.getElementById('map')); … dev var currentGeoPoint = new YGeoPoint ( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint); … … if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … } … http://search.yahooapis.com/ ImageSearchService/V1/imageSearch? appid=YahooDemo&query=Corvette&results=2 http://weather.yahooapis.com/ forecastrss?p=FRXX0076&u=c

var map = new YMap (document.getElementById('map')); … dev var currentGeoPoint = new YGeoPoint ( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint); ? … … if (flickcurl_prepare(fc, ? "flickr.photos.geo.correctLocation", parameters, count)) { … } … ? http://search.yahooapis.com/ ImageSearchService/V1/imageSearch? appid=YahooDemo&query=Corvette&results=2 ? http://weather.yahooapis.com/ forecastrss?p=FRXX0076&u=c

Yahoo! My Yahoo! Mail Welcome, guest Sign In Search MY PROJECTS APIs & WEB SERVICES var map = new YMap RESOURCES SUPPORT YDN Maps Suite AJAX Maps (document.getElementById('map')); Yahoo! Maps Web Services - AJAX API Getting … READY TO GET STARTED? You aren't signed in Sign In Help Started Guide Get an App ID var currentGeoPoint = new YGeoPoint dev Home The Tour Sign Up Explore Search Learn the fundamentals of the Yahoo! Maps AJAX API and start creating your own maps. This page contains: Introduction and Prerequisites ( _c.Lat, _c.Lon ); Yahoo! Maps Cheatsheets Examples Flickr Services Yahoo! My Yahoo! Mail RECENT BLOG ARTICLES Welcome, guest Sign In API Documentation Feeds Your API Keys map.addMarker(currentGeoPoint); Display a Yahoo! Map ? Apply for a new API Key Add controls to your Yahoo! Map Leonardo da Vinci: Hacker Search Add a logger to a Yahoo! Map Place a marker on a Yahoo! Map … MY PROJECTS January 24, 2009 APIs & WEB SERVICES RESOURCES SUPPORT Draw Polylines on a Yahoo! Map Whaddyathink? & the Yahoo! Application flickr.places.getPlaceTypes Place Smart Markers on a Yahoo! Map Add multiple custom markers on Yahoo! Map YDN Weather RSS Feed Platform January 22, 2009 Traffic information overlay Yahoo! Pipes RSS output information overlay Yahoo! Weather RSS Feed Making the new Yahoo! Currency Converter Yahoo! My Yahoo! Mail Welcome, guest Sign In Local search results overlay Fetches a list of available place types for Flickr. Yahoo! Pipes JSON output information overlay YMap accessible The Weather RSS feed enables you Januaryto get up-to-date weather information for your location. You can save this 21, 2009 Overlay GeoRSS XML data Overlay custom objects at YGeoPoint location … feed in My Yahoo! or your favorite feed aggregator, or incorporate the RSS data into your own web site or client Search application. The Weather RSSRetrievingdynamically-generatedfrom based on zip code or Location ID. feed is a and displaying data feed Authentication Overlay custom objects at YCoordPoint location Overlay polyline on Yahoo! Map if (flickcurl_prepare(fc, Wikipedia with YQL PROJECTS MY APIs & WEB SERVICES RESOURCES This page describes the format of the 18, 2009 URL and the RSS response for developers. It contains these January request SUPPORT ? Export Yahoo! Map in GeoRSS format YDN Image Search Documentation for Yahoo! Search Web Services This method does not require authentication. GeoRSS overlay with polyline "flickr.photos.geo.correctLocation", sections: Get zoom level given radius from center Custom position of controls Arguments RSS Request Image Search Documentation for Yahoo! Search READY TO GET STARTED? Change Marker Images Dynamically Use YMapTypeControl to position map types parameters, count)) { … } Summary Request Parameters By applying for an Application ID for this service, you hereby agree to the Terms of Use Submitting Image Queries Rate Limit api_key (Required) Other Features of the Yahoo! Maps AJAX API … RSS Response YEvent, YGeoRSS, YMarker Your API application key. See here for moreTop-Level Elements details. The Image Search Web Service allows you to search the Internet for images. Get an App ID Terms of Service Channel Elements Support & Community Image Elements Example Response ? Item Elements Download all of the cheatsheets and example code in the Yahoo Request URL Maps API Reference Bundle. Condition Codes YAHOO! GROUPS DISCUSSIONS view all <place_types> Examples http://search.yahooapis.com/ImageSearchService/V1/imageSearch For detailed descriptions of the classes and methods in the API, place_type_id="22">neighbourhood</place_type> <place_type yahoo local search code in php Terms of Use Tue, 27 Jan 2009 check out v3.8 Reference Manual. <place_type place_type_id="7">locality</place_type> <place_type <place_type http://search.yahooapis.com/ place_type_id="9">county</place_type> RSS Request place_type_id="8">region</place_type> Remaining, Supporting Classes Request parameters My Web Search API shut down See information on constructing REST queries announcement Introduction and <place_type Prerequisites <place_type ImageSearchService/V1/imageSearch? place_type_id="12">country</place_type> The Weather RSS feed request follows simple HTTP GET syntax: start with a base URL and then add place_type_id="29">continent</place_type> (?). Multiple parameters are separated by an ampersand (&). parameters and values after a question mark Wed, 21 Jan 2009 Parameter Value Description </place_types> appid=YahooDemo&query=Corvette&results=2 The Yahoo! AJAX Maps API lets developers add maps to their web sites using DHTML and JavaScript®. the Weather RSS feed is The base URL for Maps appid string (required) The application ID. See Application IDs for more information. Re: Site Explorer API returns 999 Rate Limit Exceeded are fully embeddable and scriptable using the JavaScript programming language. Yahoo! Maps AS-Flash API's query string (required) The query to search for. Use + to include terms, - to exclude terms, and Wed, 21 Jan 2009 ? Error Codes built-in geocoder means that you can specify a physical address or latitude/longitude coordinates for your map's http://weather.yahooapis.com/forecastrss put quotes around "exact phrase". location, as you like. type all (default), any, or The kind of search to submit: Reminder: Old obsolete versions of Local In this section, Summary shows how to make an RSS request with different parameters; Request Parameters phrase Search (V1, V2) shutting do In order to create and test applications 100: Yahoo! Maps AJAX API, you'll need to use a supported summarizing those parameters. using Invalid API Key shows a table web all returns results with all query terms. Mon, 19 Jan 2009 The API key passed was not valid or has expired. browser: Firefox 2, Internet Explorer 6 or 7, Opera 9 or Safari 3. Newer versions of these browsers are also any returns results with one or more of the query terms. supported. You can code your JavaScript applications usingcurrently unavailable 105: Service your favorite text editor or IDE. phrase returns results containing the query terms as a phrase. SUMMARY results integer: default 10, The number of results to return. The requested service is temporarily unavailable. An application ID is required to use the Yahoo! Maps AJAX API. You can get an application ID here. max 50 The Weather RSS feed request follows simple HTTP GET syntax: start with a base URL and then add 111: Format "xxx" not found start integer: default 1 parameters and values after a question mark (?). Multiple parameters are separated by an starting result position to return (1-based). The finishing position (start The ampersand (&). For For best results, you should launch your web pagesrequested response format was not found. applications The from a web server. However, it's possible to run + results - 1) cannot exceed 1000. the Weather RSS feed there are two parameters: directly from your hard drive by double-clickingMethod This can be donefound 112: on them. "xxx" not by lowering the security settings in format any (default), bmp, Specifies the kind of image file to search for. the Internet Explorer browser or by setting the The requested method was not found. p for location. UniversalBrowserRead property in Firefox. gif, jpeg, png u for degrees units (Fahrenheit or Celsius). adult_ok 114: Invalid SOAP envelope http://weather.yahooapis.com/ no value or 1 The service filters out adult content by default. Enter a 1 to allow adult content. The SOAP envelope send in the requestlocationnot be parsed. required. Use this parameter to indicate the location for the weather forecast as a The could parameter p is Displaying a Yahoo! Map zip code or Location ID. coloration any (default), color, The service returns only the images of the coloration specified (color or 115: Invalid XML-RPC Method Call forecastrss?p=FRXX0076&u=c All the API classes and methods necessary to interact with Yahoo! Mapsdocument couldyour application when The XML-RPC request are availablehttp://weather.yahooapis.com/forecastrss?p=location to not be parsed. bw black-and-white). site string: default no A domain to restrict your searches to (e.g. www.yahoo.com). You may you include Yahoo! Maps AJAX API library in your web page.This example uses the demonstration application value submit up to 30 values (site=www.yahoo.com&site=www.cnn.com). ID "YD-eQRpTl0_JX2E95l_xAFs5UwZUlNQhhn7lj1H"; you cannot use this Application Forand must requestweather for Yahoo! Headquarters in Sunnyvale, CA, use the zip ID example, to get one code for Sunnyvale here. API Explorer (94089): output string: xml The format for the output. If json is requested, the results will be returned (default), json, php in JSON format. If php is requested, the results will be returned in Serialized PHP format. . <html> API Explorer : flickr.places.getPlaceTypes http://weather.yahooapis.com/forecastrss?p=94089 . <head> callback string The name of the callback function to wrap around the JSON data. The . <b><script type="text/javascript" Or use the Location ID for Sunnyvale (USCA1116): following characters are allowed: A-Z a-z 0-9 . [] and _. If output=json has . src="http://api.maps.yahoo.com/ajaxymap?v=3.8&appid=YD- not been requested, this parameter has no effect. More information on the

var map = new YMap (document.getElementById('map')); … dev var currentGeoPoint = new YGeoPoint ( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint); … … if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … } … http://search.yahooapis.com/ ImageSearchService/V1/imageSearch? appid=YahooDemo&query=Corvette&results=2 http://weather.yahooapis.com/ forecastrss?p=FRXX0076&u=c

dev YQL

dev Bindings YQL

dev Bindings YQL stmt YQL

dev Bindings YQL stmt YQL select

dev Bindings YQL stmt YQL select desc

dev Bindings YQL stmt YQL select desc show

select {what} from {service} where {condition}

select {what} from {service} where {condition} show tables

desc table select {what} from {service} where {condition} show tables

desc table select {what} from {service} where {condition} show tables filtros remotos e locais

A ferramenta http://developer.yahoo.com/yql/console

http://developer.yahoo.com/yql/console

• Tudo ﬁca num arquivo XML

• Tudo ﬁca num arquivo XML • Qualquer um pode fazer

• Tudo ﬁca num arquivo XML • Qualquer um pode fazer • Manda lá pro Github http://www.datatables.org/ http://github.com/yql/yql-tables

FIGHT!

Começamos com um usuário do Meme http://meme.yahoo.com/tommarques

SELECT * FROM meme.info WHERE name='tommarques'

SELECT * FROM meme.info WHERE name='tommarques' "results":{ "meme":{ "guid":"66VIWI4DGQ44QE2VHU5EAT545E", "name":"tommarques", "title":"Tom Marques", "description":null, "url":"http://meme.yahoo.com/tommarques/", "avatar_url":"http://d.yimg.com/gg/tommarques/ avatars/ 9bc435ebdc410b85f2971aba0800f806610d49ca.jpeg", "language":"pt", "followers":"131" } }

Não esqueça! Você pode testar todas as queries mostradas aqui no console do YQL (se tiver internet, é claro) http://developer.yahoo.com/yql/console

SELECT * FROM meme.info WHERE name='tommarques' "results":{ "meme":{ "guid":"66VIWI4DGQ44QE2VHU5EAT545E", "name":"tommarques", "title":"Tom Marques", "description":null, "url":"http://meme.yahoo.com/tommarques/", "avatar_url":"http://d.yimg.com/gg/tommarques/ avatars/ 9bc435ebdc410b85f2971aba0800f806610d49ca.jpeg", "language":"pt", "followers":"131" } }

SELECT guid FROM meme.info WHERE name='tommarques'

SELECT * FROM meme.posts WHERE owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' );

SELECT * FROM meme.posts WHERE owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' ); "post":[{ "pubid":"4OSYwPo", "guid":"66VIWI4DGQ44QE2VHU5EAT545E", "url":"http://meme.yahoo.com/tommarques/p/4OSYwPo/", "timestamp":"1252393121879", "type":"photo", "content":"http://d.yimg.com/gg/patypegorin/ afa34f168e337617e625d978b6e5b32dcd6c45a9.jpeg", "repost_count":"29", "comment":"", "caption":"", ...

Um post dele http://meme.yahoo.com/tommarques/p/oOaPL9d/

SELECT * FROM meme.post.info WHERE pubid='oOaPL9d' AND owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' );

SELECT * FROM meme.post.info(300) WHERE pubid='oOaPL9d' AND owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' ) AND comment != '';

SELECT * FROM meme.post.info(300) WHERE pubid='oOaPL9d' AND owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' ) AND comment != ''; { "type":"comment", "guid":"OKF7PJQWUFNCNWKVOB3BGHM37I", "pubid":"Md_v-Hw", "origin":"66VIWI4DGQ44QE2VHU5EAT545E", "origin_pubid":"oOaPL9d", "via":"OKF7PJQWUFNCNWKVOB3BGHM37I", "timestamp":"1256438370", "comment":"ops, moro em Goias, mas sou do RJ" },

Olha o que aconteceu nos bastidores: <diagnostics> <publiclyCallable>true</publiclyCallable> <url execution-time="29"><![CDATA[http://meme/api/v1/public/info/? name=tommarques]]></url> <url execution-time="381"> <![CDATA[http://meme/api/v1/public/permalink/info/? pubid=oOaPL9d&owner_guid=66VIWI4DGQ44QE2VHU5EAT545E&page=1&count=100]]> </url> <url execution-time="204"> <![CDATA[http://meme/api/v1/public/permalink/info/? pubid=oOaPL9d&owner_guid=66VIWI4DGQ44QE2VHU5EAT545E&page=2&count=100]]> </url> <user-time>651</user-time> <service-time>614</service-time> <build-version>4265</build-version> </diagnostics>

Conseguimos uma lista de lugares...

Conseguimos uma lista de lugares... "comment":"ops, moro em Goias, mas sou do RJ"

Conseguimos uma lista de lugares... "comment":"ops, moro em Goias, mas sou do RJ" "comment":"Pindamonhangaba e sim, a cidade existe."

Conseguimos uma lista de lugares... "comment":"ops, moro em Goias, mas sou do RJ" "comment":"Pindamonhangaba e sim, a cidade existe." "comment":"putiz sou de cotia sp"

Conseguimos uma lista de lugares... "comment":"ops, moro em Goias, mas sou do RJ" "comment":"Pindamonhangaba e sim, a cidade existe." "comment":"putiz sou de cotia sp" "comment":"Salvador...Baeeaaaaaaaaa"

Conseguimos uma lista de lugares... "comment":"ops, moro em Goias, mas sou do RJ" "comment":"Pindamonhangaba e sim, a cidade existe." "comment":"putiz sou de cotia sp" "comment":"Salvador...Baeeaaaaaaaaa" "comment":"Boa noite memers, ah sou de Belém do Pará"

Conseguimos uma lista de lugares... "comment":"ops, moro em Goias, mas sou do RJ" "comment":"Pindamonhangaba e sim, a cidade existe." "comment":"putiz sou de cotia sp" "comment":"Salvador...Baeeaaaaaaaaa" "comment":"Boa noite memers, ah sou de Belém do Pará" ... mas em linguagem beeem natural :(

SELECT * FROM geo.places WHERE text = "o Rio de Janeiro continua lindo"

SELECT * FROM geo.places WHERE text = "o Rio de Janeiro continua lindo" "place":{ "lang":"en-US", "uri":"http://where.yahooapis.com/v1/place/ 455825", "woeid":"455825", "placeTypeName":{ "code":"7", "content":"Town" }, "name":"Rio de Janeiro", "country":{ "code":"BR", "type":"Country", "content":"Brazil" }, ...

SELECT * FROM geo.places WHERE text = "o Rio de Janeiro continua lindo" "place":{ "lang":"en-US", "uri":"http://where.yahooapis.com/v1/place/ 455825", "woeid":"455825", "placeTypeName":{ "code":"7", "content":"Town" Hadouken! }, "name":"Rio de Janeiro", "country":{ "code":"BR", "type":"Country", "content":"Brazil" }, ...

SELECT * FROM geo.places WHERE text = "o Rio de Janeiro continua lindo" Hadouken!

SELECT woeid FROM geo.places WHERE text in ( SELECT comment FROM meme.post.info(300) WHERE pubid='oOaPL9d' AND owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' ) AND comment != ''; ) | unique(field='woeid')

Conseguimos uma lista de woeids...

Conseguimos uma lista de woeids... ...mas como descobrir onde usar? :(

DESC flickr.photos.search

DESC flickr.photos.search <key name="machine_tags" type="xs:string"/> <key name="safe_search" type="xs:string"/> <key name="privacy_filter" type="xs:string"/> <key name="contacts" type="xs:string"/> <key name="tag_mode" type="xs:string"/> <key name="user_id" type="xs:string"/> <key name="license" type="xs:string"/> <key name="geo_context" type="xs:string"/> <key name="lat" type="xs:string"/> <key name="tags" type="xs:string"/> <key name="lon" type="xs:string"/> <key name="text" type="xs:string"/> <key name="max_taken_date" type="xs:string"/> <key name="is_commons" type="xs:string"/> <key name="radius" type="xs:string"/> <key name="woe_id" type="xs:string"/> <key name="has_geo" type="xs:string"/>

SELECT * FROM flickr.photos.search WHERE text='campus party'

SELECT * FROM flickr.photos.search WHERE text='campus party' SELECT * FROM flickr.photos.search WHERE is_commons='true'

SELECT * FROM flickr.photos.search WHERE text='campus party' SELECT * FROM flickr.photos.search WHERE is_commons='true' SELECT * FROM flickr.photos.search WHERE user_id='foobar'

SELECT woeid FROM geo.places WHERE text in ( SELECT comment FROM meme.post.info(300) WHERE pubid='oOaPL9d' AND owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' ) AND comment != ''; ) | unique(field='woeid')

SELECT * FROM flickr.photos.search(100) WHERE has_geo="true" and woe_id in ( SELECT woeid FROM geo.places WHERE text in ( SELECT comment FROM meme.post.info(300) WHERE pubid='oOaPL9d' AND owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' ) AND comment != ''; ) | unique(field='woeid') )

SELECT * FROM flickr.photos.search(100) WHERE has_geo="true" and woe_id in ( SELECT woeid FROM geo.places WHERE text in ( SELECT comment FROM meme.post.info(300) WHERE pubid='oOaPL9d' AND owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' ) AND comment != ''; ) | unique(field='woeid') ) <results> <photo farm="3" id="4125642160" isfamily="0" isfriend="0" ispublic="1" owner="21381694@N07" secret="af0b6091b2" server="2629" title="Numa noite dessas, por ai..."/> <photo farm="3" id="4122278478" isfamily="0" isfriend="0" ispublic="1" owner="11264537@N05" secret="34f6ce4517" server="2660" title="AST Facilities 02"/> <photo farm="3" id="4055857251" isfamily="0" isfriend="0" ispublic="1" owner="44116004@N03" secret="5115fe3d9d" server="2538" title="Negao e Vinny"/> ...

E o resultado...

SELECT * FROM flickr.photos.search(100) WHERE has_geo="true" and woe_id in ( SELECT woeid FROM geo.places WHERE text in ( SELECT comment FROM meme.post.info(300) WHERE pubid='oOaPL9d' AND owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' ) AND comment != ''; ) | unique(field='woeid') )

SELECT * FROM flickr.photos.search(100) WHERE has_geo="true" and woe_id in ( SELECT woeid FROM geo.places WHERE text in ( SELECT comment FROM meme.post.info(300) WHERE pubid='oOaPL9d' AND owner_guid in ( SELECT guid FROM meme.info WHERE name='tommarques' ) AND comment != ''; ) | unique(field='woeid') ) Em uma query, 3 sites/serviços diferentes, 3 APIs distintas

É mágico e revolucionário!

INSERT, UPDATE, DELETE

insert into twitter.status (status,username,password) values ("new tweet from YQL",”xxx”,”yyy”);

delete from twitter.status where id="2108869549" and username="xxx" and password="yyy"

use ‘http://url-to-def.xml’ as tableName; select * from tableName where ...

Fork me! http://github.com/yql/yql-tables

http://datatables.org

EXECUTE

• javascript rodando no servidor • acessa APIs com autenticação • juntar pesquisas distintas

• melhorar os dados • criar API a partir de html (scraping) • transformar dados • mover lógica da aplicação para o cloud

<execute><![CDATA[ // Your javascript goes here. response.object = <item> <url>{request.url}</url> <a>{a}</a> <b>{b}</b> </item>; ]]></execute>

Mão na massa: http://developer.yahoo.com/yql/guide/yql-code-examples.html

entendeu pra que serve o YQL?

pra construir aplicações de forma mais fácil

• menos linhas de código • sintaxe consistente para acesso a dados

pra construir aplicações mais rápidas mais rápido

• serviço no cloud faz o trabalho • processamento, ﬁltros, conversões • menos chamadas de serviços • infraestrutura conﬁável

quer saber mais?

• http://developer.yahoo.com/yql/ • http://developer.yahoo.com/meme/

Yahoo Query Language select * from internet Pedro Valente @pedrovalente

Obrigado!

YQL: Select * from internet - Campus Party 2010

by Pedro Valente on Feb 01, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 6

Statistics

YQL: Select * from internet - Campus Party 2010 — Presentation Transcript

http://www.slideshare.net	4
http://www.cpdev.com.br	2