This document introduces Yahoo Query Language (YQL) and provides an overview of its features and capabilities. YQL allows users to query, filter, and join data across web services using an SQL-like language. It provides a single API specification and allows accessing and modifying internet data through a uniform query language. Examples demonstrate how to retrieve twitter trending topics for a location and get related news articles through a single YQL query. The document also discusses open data tables, the YQL console, and how to contribute new tables to the system.
Slides from the talk at Skillsmatter on my F# erasing SQL provider
http://skillsmatter.com/podcast/scala/sql-type-provider-deep-dive-with-ross-mckinlay
ELUNA2014: Developing and Testing an open source web applicationMichael Cummings
ELUNA2014: Developing and Testing an open source web application provides information about a custom library catalog developed at the George Washington University, as well as a description of how developers use CasperJS to test the system.
Presented at Devoxx US (http://cfp.devoxx.us/2017/talk/PEV-2089)
Collections are a staple in any programming language: the need to collect, sort or iterate over values is needed by nearly all developers.
The Java language introduced the Collections framework long ago and that's what many (incredibly not all!) developers learn when they start programming. It has plenty to offer, but many find it lacking: the amount of collection libraries as active open source projects demonstrate the need for something else.
If we were to search to find a holistic comparison of famous collection libraries, there is not much literature available. The talk is poised to fill this gap. In this session, we will explore the most common collections (pun intended!) frameworks, what they have to offer and what you should consider for your next project.
We will also show common programmer use cases, how each library handles them and the impact on memory, processing power and ease of use/coding.
After this session, you will be able to choose the right bag for you tricks!
Multi-Model Data Query Languages and Processing ParadigmsJiaheng Lu
Specifying users' interests with a formal query language is a typically challenging task, which becomes even harder in the context of multi-model data management because we have to deal with data variety. It usually lacks a unified schema to help the users issuing their queries, or has an incomplete schema as data come from disparate sources. Multi-Model DataBases (MMDBs) have emerged as a promising approach for dealing with this task as they are capable of accommodating and querying the multi-model data in a single system. This tutorial aims to offer a comprehensive presentation of a wide range of query languages for MMDBs and to make comparisons of their properties from multiple perspectives. We will discuss the essence of cross-model query processing and provide insights on the research challenges and directions for future work. The tutorial will also offer the participants hands-on experience in applying MMDBs to issue multi-model data queries.
Apache Calcite (a tutorial given at BOSS '21)Julian Hyde
Apache Calcite is a dynamic data management framework. Think of it as a toolkit for building databases: it has an industry-standard SQL parser, validator, highly customizable optimizer (with pluggable transformation rules and cost functions, relational algebra, and an extensive library of rules), but it has no preferred storage primitives.
In this tutorial (given at BOSS '21 in Copenhagen as part of VLDB '21) the attendees will use Apache Calcite to build a fully fledged query processor from scratch with very few lines of code. This processor is a full implementation of SQL over an Apache Lucene storage engine. (Lucene does not support SQL queries and lacks a declarative language for performing complex operations such as joins or aggregations.) Attendees will also learn how to use Calcite as an effective tool for research.
Presenters: Julian Hyde and Stamatis Zampetakis
Slick (part of the Typesafe stack) is a modern database query and access library for Scala, based on functional principles. It allows you to write queries as if you are working with regular Scala collections.
In this presentation we’ll have a deep dive into how you can use this library in real projects. How to map your tables and queries to structured objects, how to create more advanced queries with multiple joins, how to setup integration tests against an in-memory database and how you can integrate Slick with the Play Framework are all questions which will have been answered at the end of this presentation.
Originally presented on the BeScala user group.
Slides from the talk at Skillsmatter on my F# erasing SQL provider
http://skillsmatter.com/podcast/scala/sql-type-provider-deep-dive-with-ross-mckinlay
ELUNA2014: Developing and Testing an open source web applicationMichael Cummings
ELUNA2014: Developing and Testing an open source web application provides information about a custom library catalog developed at the George Washington University, as well as a description of how developers use CasperJS to test the system.
Presented at Devoxx US (http://cfp.devoxx.us/2017/talk/PEV-2089)
Collections are a staple in any programming language: the need to collect, sort or iterate over values is needed by nearly all developers.
The Java language introduced the Collections framework long ago and that's what many (incredibly not all!) developers learn when they start programming. It has plenty to offer, but many find it lacking: the amount of collection libraries as active open source projects demonstrate the need for something else.
If we were to search to find a holistic comparison of famous collection libraries, there is not much literature available. The talk is poised to fill this gap. In this session, we will explore the most common collections (pun intended!) frameworks, what they have to offer and what you should consider for your next project.
We will also show common programmer use cases, how each library handles them and the impact on memory, processing power and ease of use/coding.
After this session, you will be able to choose the right bag for you tricks!
Multi-Model Data Query Languages and Processing ParadigmsJiaheng Lu
Specifying users' interests with a formal query language is a typically challenging task, which becomes even harder in the context of multi-model data management because we have to deal with data variety. It usually lacks a unified schema to help the users issuing their queries, or has an incomplete schema as data come from disparate sources. Multi-Model DataBases (MMDBs) have emerged as a promising approach for dealing with this task as they are capable of accommodating and querying the multi-model data in a single system. This tutorial aims to offer a comprehensive presentation of a wide range of query languages for MMDBs and to make comparisons of their properties from multiple perspectives. We will discuss the essence of cross-model query processing and provide insights on the research challenges and directions for future work. The tutorial will also offer the participants hands-on experience in applying MMDBs to issue multi-model data queries.
Apache Calcite (a tutorial given at BOSS '21)Julian Hyde
Apache Calcite is a dynamic data management framework. Think of it as a toolkit for building databases: it has an industry-standard SQL parser, validator, highly customizable optimizer (with pluggable transformation rules and cost functions, relational algebra, and an extensive library of rules), but it has no preferred storage primitives.
In this tutorial (given at BOSS '21 in Copenhagen as part of VLDB '21) the attendees will use Apache Calcite to build a fully fledged query processor from scratch with very few lines of code. This processor is a full implementation of SQL over an Apache Lucene storage engine. (Lucene does not support SQL queries and lacks a declarative language for performing complex operations such as joins or aggregations.) Attendees will also learn how to use Calcite as an effective tool for research.
Presenters: Julian Hyde and Stamatis Zampetakis
Slick (part of the Typesafe stack) is a modern database query and access library for Scala, based on functional principles. It allows you to write queries as if you are working with regular Scala collections.
In this presentation we’ll have a deep dive into how you can use this library in real projects. How to map your tables and queries to structured objects, how to create more advanced queries with multiple joins, how to setup integration tests against an in-memory database and how you can integrate Slick with the Play Framework are all questions which will have been answered at the end of this presentation.
Originally presented on the BeScala user group.
Speed Up Your APEX Apps with JSON and HandlebarsMarko Gorički
With new APEX JSON API and Oracle 12c features, it's really easy to work with JSON. How can it be used in APEX? There are numerous ways to enhance your applications with JSON, and one of them is by using JavaScript templating. Handlebars is one of most popular template engines – small and simple, yet very powerful.
This presentation will cover Handlebars' integration with APEX and all the benefits you can get from this.
Graph databases are used to represent graph structures with nodes, edges and properties. Neo4j, an open-source graph database is reliable and fast for managing and querying highly connected data. Will explore how to install and configure, create nodes and relationships, query with the Cypher Query Language, importing data and using Neo4j in concert with SQL Server... Providing answers and insight with visual diagrams about connected data that you have in your SQL Server Databases!
How Solr Search Works - A tech Talk at Atlogys Delhi Office by our Senior Technologist Rajat Jain. The lecture takes a deep dive into Solr - what it is, how it works, what it does and its inbuilt architecture. A wonderful technical session with many live examples, a sneak peak into solr code and config files and a live demo. Part of Atlogys Academy Series.
Introduction to SQL Alchemy - SyPy June 2013Roger Barnes
A very brief introduction to SQLAlchemy, covering the core, ORM, database concepts and a high-level comparison to the Django ORM.
IPython notebook demo content is at http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/s/90s3abt64vxd4r4/SQLAlchemy.ipynb?token_hash=AAEWmGa8Kng0qijeH29NnPtjblOCTe387vRUxLDOpbyCKg&dl=1
Think *inside* the box. Inside the *search* box, that is.
The "best"* search results incorporate many more factors than (just) textual matching and relevancy. Search experience owners manage query context rules, signals automatically feed back machine learned factors, users implicit and explicit behaviors filter and weight future interactions. Synergy emerges with several cooperating (just) searches.
This talk will showcase and detail several (just) search examples including rules, typeahead/suggest, signals, and location awareness, bringing them all together into a cohesive search experience.
The next major release of Solr is right around the corner! Join Solr Committer Cassandra Targett and Lucidworks SVP of Engineering Trey Grainger for a first look into what’s included in the upcoming release.
Speed Up Your APEX Apps with JSON and HandlebarsMarko Gorički
With new APEX JSON API and Oracle 12c features, it's really easy to work with JSON. How can it be used in APEX? There are numerous ways to enhance your applications with JSON, and one of them is by using JavaScript templating. Handlebars is one of most popular template engines – small and simple, yet very powerful.
This presentation will cover Handlebars' integration with APEX and all the benefits you can get from this.
Graph databases are used to represent graph structures with nodes, edges and properties. Neo4j, an open-source graph database is reliable and fast for managing and querying highly connected data. Will explore how to install and configure, create nodes and relationships, query with the Cypher Query Language, importing data and using Neo4j in concert with SQL Server... Providing answers and insight with visual diagrams about connected data that you have in your SQL Server Databases!
How Solr Search Works - A tech Talk at Atlogys Delhi Office by our Senior Technologist Rajat Jain. The lecture takes a deep dive into Solr - what it is, how it works, what it does and its inbuilt architecture. A wonderful technical session with many live examples, a sneak peak into solr code and config files and a live demo. Part of Atlogys Academy Series.
Introduction to SQL Alchemy - SyPy June 2013Roger Barnes
A very brief introduction to SQLAlchemy, covering the core, ORM, database concepts and a high-level comparison to the Django ORM.
IPython notebook demo content is at http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/s/90s3abt64vxd4r4/SQLAlchemy.ipynb?token_hash=AAEWmGa8Kng0qijeH29NnPtjblOCTe387vRUxLDOpbyCKg&dl=1
Think *inside* the box. Inside the *search* box, that is.
The "best"* search results incorporate many more factors than (just) textual matching and relevancy. Search experience owners manage query context rules, signals automatically feed back machine learned factors, users implicit and explicit behaviors filter and weight future interactions. Synergy emerges with several cooperating (just) searches.
This talk will showcase and detail several (just) search examples including rules, typeahead/suggest, signals, and location awareness, bringing them all together into a cohesive search experience.
The next major release of Solr is right around the corner! Join Solr Committer Cassandra Targett and Lucidworks SVP of Engineering Trey Grainger for a first look into what’s included in the upcoming release.
YUIConf 2010. A look at the how and why of YUI 3's module infrastructure and DOM abstraction layer (Node) and how to add special attributes, new methods, and synthetic DOM events for Node.
Efficient working with Databases in LabVIEW - Sam Sharp (MediaMongrels Ltd) -...MediaMongrels Ltd
Sam Sharp's presentation from GDevCon#2 on Efficient Working with Databases in LabVIEW.
This presentation discusses some best practice hints & tips for working with databases in LabVIEW and uses Yii's ActiveRecord implementation as an example of how we can work more efficiently with databases in LabVIEW.
Effective data governance is imperative to the success of Data Lake initiatives. Without governance policies and processes, information discovery and analysis is severely impaired. In this session we will provide an in-depth look into the Data Governance Initiative launched collaboratively between Hortonworks and partners from across industries. We will cover the objectives of Data Governance Initiatives and demonstrate key governance capabilities of the Hortonworks Data Platform.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
Yui conf nov8-2010-introtoyql
1. Yahoo Query Language:
select * from internet
an Introduction
Mirek Grymuza – mirek@yahoo-inc.com
Josh Gordineer – joshgord@yahoo-inc.com
2. What are we going to cover?
• What, why and brief history of YQL
• Overview of YQL features, YQL Console
• Get into more detail with: YQL in practice
4. My application
my awesome application
•multiple data sources
•different specs and formats
•multiple connections
•api changes to deal with
•no arbitrary sources without work
5. Enter YQL
•single API spec
•SQL-like
•select/insert/update/delete
•let YQL optimize queries
•powerful
my awesome application
6. So what can YQL do?
SELECT * FROM flickr.photos.info WHERE photo_id IN (SELECT id FROM flickr.photos.search(1) WHERE text IN (SELECT content FROM
search.termextract WHERE context IN (SELECT body FROM nyt.article.search WHERE apikey='key' AND query='obama' LIMIT 1)))
show: lists the supported tables
desc: describes the structure of a table
select: fetches data
insert/update/delete: modify data
use: use an Open Data Table
set: define key-values across Open Data Tables
The statement
7. Filtering, paging, projection
• Table data can be filtered in the WHERE clause either:
–Remotely by the table data source provider or
–Locally by the YQL engine
• YQL tries to present “rows” of data
–Abstracts away “paging” views of data sources
–Presents a “subset” of paging tables by default
• In YQL fields are analogous to the columns of a table,
multiple fields are delimited by commas
select Title,Address from local.search(0,10) where query="sushi" and
location="san francisco, ca" and Rating.AverageRating="4.5" LIMIT 2
8. Joining across sources
• Sub-select works the same as normal select except it can
only return a “leaf” element value or attribute
• Parallelizes execution
• Example: How to get an international weather forecast?
Join two services in different companies:
select * from weather.forecast where location in (select id from xml where
url=http://xoap.weather.com/search/search?where=prague and
itemPath="search.loc")
9. Post-query manipulation
• YQL includes built-in functions such as sort, unique,
truncate, tail, reverse...
• Simple post-SELECT processing can be performed by
appending the “pipe” symbol to the end of the
statement SELECT … | sort(field=item.date) SELECT
… | unique(field=item.title) | …
• Functions only operate on the data being returned by the
query, nothing to do with the tables or data sources
themselves
select * from social.profile where guid in (select guid from
social.connections where owner_guid=me) | sort(field="nickname")
10. How do you benefit?
SELECT * FROM INTERNET
(INSERT/UPDATE/DELETE)
Uniform method for accessing and modifying
internet data and services
Simplify and enrich data and service
access via uniform query language and
execute tables
11. Now let’s review - what is YQL?
• Cloud web service with SQL-Like Language
–Familiar to developers
• Synonymous with Data access
–Expressive enough to get the right data.
• Self describing - show, desc table
• Allows you to query, filter, join and update data across any
structured data on the web / web services
–And Yahoo’s Sherpa cloud storage
• All in Real time
• Inject business logic with execute element
12. YQL Since Launch...
• open data tables, environment files
• execute element - April
• new paging model
• insert/update/delete, jsonp-x - July
• set verb, yql.storage, debug mode, multi env
• y.rest, y.query with timeouts
• custom cache, query alias
• meta element
• extend execute to add libraries, functions
• console cache, shortener and query builder
• lots of various data tables since then and more being added
Launched October 28 2008
2010
an enhancement
or new feature
added every
month since
2009
13. ...where is YQL today?
Most popular tables this month?
~6B table requests in October
on track to 7B in November
Popular since launch?
14. YQL Console
• http://developer.yahoo.com/yql/console/
• Hosted site which executes YQL queries
• Swiss Army Knife for YQL Developers
• Design and debug quickly
How many tables?
• default tables – 175
• community tables – 772
• total - 947
19. What is YQL?
• “The Yahoo! Query Language is an expressive SQL-like
language that lets you query, filter, and join data across
Web services.”
• So what does that mean?
• Be “lazy” – Let YQL take care of the data
–Allows you to focus on innovation not on API’s
20. The Problem
• Fetch the Yahoo! News articles for Twitter trending topics
in San Francisco
• And be “lazy” i.e. use YQL
21. YQL Tables
• Built-in Tables
–Maintained by the YQL Team (or Yahoo!)
–fantasy sports, weather, answers, flickr, geo, music,
search, upcoming, mail …
• Data Tables
–Specialized tables to fetch raw data from the web
–atom, csv, html, json, xml …
23. Open Data Tables
• Brings the power of YQL to any API
• Open Data Table Schema defines mapping between YQL
and Endpoint
–http://query.yahooapis.com/v1/schema/table.xsd
• Supply the open table with the “use” statement
• Supply multiple open tables with an “env” query parameter
–ENV file contains multiple USE statements
–Loads environment prior to executing YQL query
24. Open Data Table Example
<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
<bindings>
<select itemPath="matching_trends.trends.trend"
produces="XML">
<urls>
<url>http://api.twitter.com/1/trends/{woeid}.xml</url>
</urls>
<inputs>
<key id="woeid” paramType="path" required="true" />
</inputs>
</select>
</bindings>
</table>
25. url and key Elements
<url>http://api.twitter.com/1/trends/{woeid}.xml</url>
• Provides the resource location for your API
<key id="woeid" paramType="path" required="true" />
• Defines the parameters for the API and provides a binding
for the YQL where clause
• paramType can be query or path
• required is optional
26. Running YQL Queries
• Console
–http://developer.yahoo.com/yql/console
–Quickly discover tables and iterate on queries
• Public Endpoint
–http://query.yahooapis.com/v1/public/yql
–No Auth
–Rate limit 1K/hour per IP
• Authenticated Endpoint
–http://query.yahooapis.com/v1/yql
–OAuth
–10x higher rate limits
27. YQL Webservice Basics cont’d
• Query passed in as the “q” query parameter
–http://query.yahooapis.com/v1/public/yql?q=show%20ta
bles
• Execute as a simple HTTP GET
–curl
http://query.yahooapis.com/v1/public/yql?q=show%20ta
bles
• Also available for PUT, POST and DELETE
–curl -d "q=show%20tables"
http://query.yahooapis.com/v1/public/yql
32. Community Tables
• Someone may have done the work for you already
–http://datatables.org
• Tables are hosted on GitHub
–https://github.com/yql/yql-tables
• Use the env query parameter to include all community
tables in a request
–env=store://datatables.org/alltableswithkeys
34. Contributing
Process for adding/updating tables on Git
1. Fork the YQL Tables project
2. Clone your Fork
3. Make your changes
4. Push Changes / Commit
5. Make Pull Request
6. YQL Table Admin will moderate and merge changes
and generate new push to datatables.org
• Steps 1-5 are standard Git procedures, step 6 is unique
• Git Tutorials
–http://help.github.com/forking
–http://thinkvitamin.com/code/starting-with-git-cheat-
sheet
35. Twitter Trending News Query
select abstract, url from search.news where query in (
select trend from twitter.trends.location where
woeid=2487956
)
Retrieves news results for the latest twitter trending topics in
San Francisco
• Combines numerous API calls into a single YQL query
• Filters search.news response from 5 fields into just 2