This document discusses Pipes and Yahoo! Query Language (YQL) as tools for combining and transforming data from various sources on the web. Pipes allows users to visually combine and transform data feeds without programming. YQL provides a similar capability but with an SQL-like language instead of a visual editor. Both tools allow users to easily access, filter, and combine data from different sources on the web to build custom "data mashups". The document provides examples of common data mashups and discusses lessons learned from the widespread adoption of Pipes, as well as the advantages and future directions of YQL.
API's, Freebase, and the Collaborative Semantic webDan Delany
A presentation about the state of the collaborative semantic web, including:
- What?
- Why?
- Where do we stand?
- A case study on Metaweb's Freebase project
API's, Freebase, and the Collaborative Semantic webDan Delany
A presentation about the state of the collaborative semantic web, including:
- What?
- Why?
- Where do we stand?
- A case study on Metaweb's Freebase project
An overview of Firebug's features I presented at the July, 2009 meeting of the West Suburban Chicago PHP Meetup. This presentation was followed by at least a 1/2 hour demo, and then another member of the group went into the details of FirePHP for almost an hour.
It was a really intense night of great presentations and fun.
Visit us on the web at http://php.meetup.com/381/ and http://suburbanchicagophp.org
This talk explains best practices and techniques use to build high performance mobile sites. The talk covers a wide range of different topics centred on how to improve performance over the network and how to improve performance in the software (including both server side & client side).
Lessons from Highly Scalable Architectures at Social Networking SitesPatrick Senti
What are the techniques and technolgies used by popular social networking sites such as Facebook, Twitter, Tumblr, Pinterest or Instagram? How do they architect their systems to scale to multiples of 100 million of visits per day?
The key to a successful mobile site is high performance and reliability across a wide range of device capabilities and network latencies. However, the mobile web is a hostile environment with support for HTML5, JavaScript and CSS varying widely across browsers and devices. This talk will explain best practices to build high performance mobile sites that work across a wide range of devices and capabilities. The focus will be on lessons learnt at Betfair while rewriting the entire mobile web stack and how we used techniques to maximise performance and reliability. After discussing the problems faced in mobile the talk will explain how adaptive techniques can be used to provide progressive enhancement. This will be followed by an explanation of why and where performance bottlenecks occur and how these can be solved.
Video: http://www.youtube.com/watch?v=BT8WvQMMaV0
Hadoop is the technology of choice for processing large data sets. At salesforce.com, we service internal and product big data use cases using a combination of Hadoop, Java MapReduce, Pig, Force.com, and machine learning algorithms. In this webinar, we will discuss an internal use case and a product use case:
Product Metrics: Internally, we measure feature usage using a combination of Hadoop, Pig, and the Force.com platform (Custom Objects and Analytics).
Community-Based Recommendations: In Chatter, our most successful people and file recommendations are built on a collaborative filtering algorithm that is implemented on Hadoop using Java MapReduce.
An overview of Firebug's features I presented at the July, 2009 meeting of the West Suburban Chicago PHP Meetup. This presentation was followed by at least a 1/2 hour demo, and then another member of the group went into the details of FirePHP for almost an hour.
It was a really intense night of great presentations and fun.
Visit us on the web at http://php.meetup.com/381/ and http://suburbanchicagophp.org
This talk explains best practices and techniques use to build high performance mobile sites. The talk covers a wide range of different topics centred on how to improve performance over the network and how to improve performance in the software (including both server side & client side).
Lessons from Highly Scalable Architectures at Social Networking SitesPatrick Senti
What are the techniques and technolgies used by popular social networking sites such as Facebook, Twitter, Tumblr, Pinterest or Instagram? How do they architect their systems to scale to multiples of 100 million of visits per day?
The key to a successful mobile site is high performance and reliability across a wide range of device capabilities and network latencies. However, the mobile web is a hostile environment with support for HTML5, JavaScript and CSS varying widely across browsers and devices. This talk will explain best practices to build high performance mobile sites that work across a wide range of devices and capabilities. The focus will be on lessons learnt at Betfair while rewriting the entire mobile web stack and how we used techniques to maximise performance and reliability. After discussing the problems faced in mobile the talk will explain how adaptive techniques can be used to provide progressive enhancement. This will be followed by an explanation of why and where performance bottlenecks occur and how these can be solved.
Video: http://www.youtube.com/watch?v=BT8WvQMMaV0
Hadoop is the technology of choice for processing large data sets. At salesforce.com, we service internal and product big data use cases using a combination of Hadoop, Java MapReduce, Pig, Force.com, and machine learning algorithms. In this webinar, we will discuss an internal use case and a product use case:
Product Metrics: Internally, we measure feature usage using a combination of Hadoop, Pig, and the Force.com platform (Custom Objects and Analytics).
Community-Based Recommendations: In Chatter, our most successful people and file recommendations are built on a collaborative filtering algorithm that is implemented on Hadoop using Java MapReduce.
Everyone is mashing up in Flex these days, why aren't you? Drop in and learn how to mash up various cloud or other APIs from MTV, Amazon, Twitter, eBay, or various others listed on programmableweb.com. Emphasis will be placed on consuming XML by way of e4x utilizing REST or Soap.
GDD Japan 2009 - Designing OpenSocial Apps For Speed and ScalePatrick Chanezon
Google Developer Days Japan 2009 - Designing OpenSocial Apps For Speed and Scale
Original slides from Arne Roomann-Kurrik & Chris Chabot with a few Zen quotes and references added by me:-)
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
3. Apt near Park
How do you find an apartment near a park?
--
4. Apt near Park
• Apartment listings
– For each apartment:
• Click on map link or enter an address into a browser
• Check distance to a park on the map
• Tedious
--
5. Apt near Park
• Data is available
– Apartment RSS feeds, craigslist, backpage.com
– Yahoo! Local API to find “things” like parks
• Can do it in about 50 lines of Perl code
#!/usr/bin/perl -w
use strict;
use LWP::Simple;
use XML::Simple;
...
--
6. Apt near Park
• Basically combine feeds + web services
• Yet another custom mashup
– HousingMaps, ChicagoCrime, ...
• Would be nice if there was an easier way...
--
7. Pipes
grep -iv yahoo.com squid.log | sort |
uniq -c | sort -n > top_sources.txt
• Unix Pipes for the Web
• Build useful applications from simple primitives
--
8. Pipes
• A free service that lets you remix and create data mashups
using a visual editor
• No need to host, we do it for you
Craigslist
Yahoo!
--
9. Pipes
• A free service that lets you remix and create data mashups
using a visual editor
• No need to host, we do it for you
Craigslist
Yahoo!
--
24. Any Process
Yahoo! Local
Fetch
Your Web Service
Here!
Sort
--
25. Openness
Craigslist
RSS
Yahoo!
Yahoo! Fetch Local
Badges
Google
Your Web
HTML
Service Here!
Ebay
Your app
Sort
Your data here!
here!
--
26. Hot Deals Search
• Searches across many different deal hunting sites on the
internet looking for the best prices. You can search for
particular items or just let the pipe find the best of what's
available
--
27. Geoannotated Reuters News
• Takes an RSS feed from the Reuters news service, and
quot;geocodesquot; each item - making it possible to show where that
news item is happening on a map of the world.
--
29. Kiva Loans by Location
• Gets a list of the micro-loans people have been making
through the Kiva site, and shows the amazing variety of
people and places that these loan are helping out.
--
33. Yahoo Unanswered Questions
• Finds those questions in the Y! answers site that don't
currently have an answer - so you can show how smart you
are and answer those tricky questions.
--
34. Babbler by Max Case
• Translates IM messages in Second Life
--
35. Advantages to developers
• Why use an online service to do this?
– Leveraging large infrastructure
• Faster access to network resources
• Faster access to network services
– System-wide knowledge
– Leverage inter-organizational agreements
– Easy to “string” together with other services
– Easy to use (REST-style URLs)
--
36. Run / Get the data
• Each Pipe gets its own “hosted” page
• Use the REST-style URLs to get the data
--
37. Run / Get the data
• Each Pipe gets its own “hosted” page
• Use the REST-style URLs to get the data
--
38. Edit REST-style queries
http://pipes.yahoo.com/pipes/pipe.run?
_id=1mrlkB232xGjJDdwXqIxGw
&_render=json
&location=palo+alto%2C+ca
&mindist=2
&what=parks
&_callback=foofunction
The ID of the Pipe
--
39. Edit REST-style queries
http://pipes.yahoo.com/pipes/pipe.run?
_id=1mrlkB232xGjJDdwXqIxGw
&_render=json
&location=palo+alto%2C+ca
&mindist=2
&what=parks
&_callback=foofunction
The format of the output (rss, json, kml,
ical, csv)
--
40. Edit REST-style queries
http://pipes.yahoo.com/pipes/pipe.run?
_id=1mrlkB232xGjJDdwXqIxGw
&_render=json
&location=palo+alto%2C+ca
&mindist=2
&what=parks
&_callback=foofunction
The per Pipe user customizable parameters
--
42. A year and a half in the wild:
a few observations and lessons
• 20+ releases, 600k+ Pipes later
• Unexpected breadth
– Experts who want to exploit the service
– Non-programers with much simpler needs
--
43. Web addressable data...
• is very malformed
• can be slow
• needs considerate access
• can be untrustworthy
• can be inaccessible from “here” (behind firewall etc)
--
44. Data in the Engine...
• is “cleaned” (and repaired) into UTF-8
• is cached for
– performance
– playing well with others
– several HTTP proxy layers
• serve stale and force caching
• is “sanitized”
--
45. Making it easy to consume
• Its easy to make useful data in the cloud
– Its not easy enough (for many) to use it after
– Visualization beyond lists in RSS readers
• Badges are frequently requested
• Three variants for common types of data in Pipes
--
46. Typical Pipes/mashups
• Four types of mashup
– Feed aggregation with filtering
– Two-source mashups
– Data transformation and geocoding
– Complex mashups using REST APIs
• Geocoding remains a “mashup” favorite
--
47. Reasons for adoption
• Lower barrier to use
– Graphical editor made it quick to write Pipes, attracted non-
developers
– “View Source” and “Clone” for learning/tweaking
• Wide array of data input formats and data output formats
enabled Pipes to become a useful “component” in a larger
ecology
• Web 2.0 responsiveness to community
--
48. Inaccessible data
• Lots of requests for more rich and personal data
– Text documents, word documents, mail, Excel spreadsheets
– Also organizational data
• Workarounds (to some) emerged
– Online spreadsheets, calendars (gcal) with private RSS feeds
and so on
--
49. Power...
• We started by focusing on RSS
– high-level building blocks and operations
– good for common tasks and novice users
• We listened to our user’s desires
--
50. ...vs Complexity
• Added sources for parsing JSON, XML, CSV, ICAL ...
• Added modules that could do more and be combined in many
ways
• At the cost of simplicity
– Harder to explain, use, compose
– Stretching the capabilities of many users and a visual
development environment
--
52. YQL
• Part of the recent Y!OS release
– Social APIs, Universal profile, Application platform...
• Mediator service that enables developers to query, filter and
combine Y! data and beyond
– Yahoo! web services and any URL-addressable structured data
sources
• Exposes a SQL-like SELECT syntax that is both familiar to
developers and expressive enough for getting the right data
– YQL operates on hierarchical documents, not relational tuples
• Like Pipes but with a simple textual language
--
53. The language and service
• Provides three SQL-like statements:
SELECT, SHOW, DESC
• Single URL endpoint for executing everything
– Mix and match external data and Yahoo! APIs
http://query.yahooapis.com/v1/yql?q=show%20tables
• Uses Oauth for authentication
– Open standard that enables users to grant applications access
to (selected) private data
--
68. (Very) High Level Architecture
Query Web Service JSON
3rd party Web CSV XML
Service / data ATOM
XML
Execution
engine
Factory
YQL
statement
Cache
Source Source
Parser
Filter
Optimizer
and Union
XML
builder
Table mapping
Existing Web
Filter
Service
Sort
Project
Partially/not optimized
Un-optimized/whole doc
--
69. Mapping tables to data sources
• YQL wants to push as much of the query as possible to the
remote data provider/service
• Typically REST query/path parameters do not map closely to
result structure
– We call these “keys” and are named differently than dot-path
– Simple REST definition language describes how YQL executes
queries on “table” providers
--
70. Remote and Local filtering, paging
• Table data can be filtered in the WHERE clause either:
– Remotely by the table data source provider or
– Locally by the YQL engine
• YQL tries to present “rows” of data
– Abstracts away “paging” views of data sources
– Presents a “subset” of paging tables by default
select * from local.search(500,1000) where
zip='94085' and query='pizza’
--
71. IN (SELECT…): Joining across data sources
• No left joins, sub-select only
• Get an international weather forecast? Join two services in
different companies:
select * from weather.forecast where location in
(select id from xml where
url=quot;http://xoap.weather.com/search/search?where=praguequot;
and itemPath=quot;search.locquot;)
• Sub-select works the same as normal select except it can
only return a “leaf” element value or attribute
• Parallelizes execution
--
72. Post-query manipulation
• Simple post-SELECT processing can be performed by appending
the “pipe” symbol to the end of the statement:
SELECT … |sort(field=item.date)
SELECT … |unique(field=item.title)| …
• Functions only operate on the data being returned by the query,
nothing to do with the tables or data sources themselves
--
73. Use it!
• Public+private YQL tables can be accessed at:
http://query.yahooapis.com/v1/yql?q=…
• Oauth protected, URLs must be signed
– 2-legged for public tables
– 3-legged for social tables
• Public YQL tables (soon)
– No signing required
--
74. Next steps, challenges
• “Open” tables
• Multiple authentication authority support
• Better YQL query optimization for endpoints
• Foreign key consistency
• Scripting/language bindings
--
75. Finally: Pipes without the GUI editor
• Very popular Pipe pattern is easy to represent in YQL:
SELECT * FROM rss WHERE url in
(SELECT title FROM atom WHERE url=quot;http://
spreadsheets.google.com/feeds/list/pg_T0M/
od6/public/basicquot;)
AND description LIKE quot;%wall street%quot; LIMIT 10
| unique (field=title)
--
76. Conclusion: Pipes and YQL
• Provides powerful data functions to any client
• Consumes data from many services
• Common data formats means any part of the cloud can
become the input
– Dapper, AWS, Google spreadsheets
• ...or take the output
– 1/3 Google mashups are powered by Pipes
--
77. Conclusion: Pipes and YQL
• Enable developers to easily access, combine, and filter data
to fit their application requirements
– Self-documenting model
• YQL provides developers with consistent and unified
semantics for accessing data, not just Yahoo! services
• Low overhead
• Reduce roundtrip traffic by reducing the number of requests
--
78. Thank you
• Pipes
– http://pipes.yahoo.com
• YQL
– http://query.yahooapis.com/v1/yql
– http://developer.yahoo.com/yql
– http://developer.yahoo.com/yql/console
• Get in touch
– jtrevor@yahoo-inc.com
– yql-questions@yahoo-inc.com
--