Monetize your APIs and datasets or make them available as open data
•Download as PPTX, PDF•
2 likes•1,042 views
Presentation for FIWARE Developers Weeks, Bootcamps and FI-PPP Laisons
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Monetize your APIs and datasets or make them available as open data
Similar to Monetize your APIs and datasets or make them available as open data (20)
Recently uploaded
Recently uploaded (16)
Monetize your APIs and datasets or make them available as open data
- 1. Monetize your APIs and Datasets or Make Them Available as Open Data Aitor Magán (UPM) amagan@conwet.com @AitorMagan
- 2. Agenda 1. Open Data • Open Data in FIWARE Lab 2. Data Market: CKAN + WStore + Extensions • Providing Data Creating a dataset Publish the Dataset in the WStore Increase Data Value Publishing Context Information • Consuming Data Search Acquisition APIs Data Requests 3. Monetizing APIs 4. LAB 2
- 3. “A piece of data or content is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the requirement to attribute and/or share-alike.” [http://opendefinition.org/#sthash.6ieidzit.dpuf] Sergio García Gómez 3
- 4. • Open Data implies often hidden costs – Lack of quality – Insufficient documentation – Unstructured information – Limited amount of information • All data cannot be open – Conflicting interests – Security 4
- 5. Smat Cities: OASC initiative – 31 cities 6 Helsinki Copenhagen Brussels Ghent Lisbon Porto Milan Sevilla Valencia Porto Alegre Oulu Málaga Palermo Antwerp Santander Aarhus Tocantis Ventaa Vitória Fundão Lecce Espoo Olinda Colinas de Tocantins Taquaritinga Penela Anapólis
- 6. Open Data/Content approaches Datasets Existing Datasets (census, geographical, tourism,...) Historic Data (from sensors, events...) Real Time Vertical Systems (mobility, events...) Internet of Things (sensors, Smart meters...) Media Video streams (traffic, surveillance..) Audio (microphones), speaches... Applications NGSICKAN WEBRTC KURENTO 7Sergio García Gómez
- 7. DATA MARKET + + Extensions
- 8. Open Data Platform • De facto standard platform for Open Data in Europe and beyond • Plenty of extensions: harvesting, geographical information, data visualization… • Search & Discover Data: – Search by keywords – Browse by facets – Explore data with previews & visualization – REST/JSON APIs to access data and metadata • Data Management for publishers – Easy store & update of metadata Sergio García Gómez 9
- 9. WStore • Generic Online Store – Publish offerings with services – Acquire these offerings • Extension to support specific resource types – WireCloud components: widgtets, operators and MashUps – Datasets – … • Key features – Support accounting callbacks – Support for rich price models – Support for charging – Support for billing – Integration with PayPal – Support for FIWARE MarketPlace, Repository and RSSS 10 Offering 1 Free use Resource 1 Widget Resource 2 Dataset Offering 2 8 €
- 10. FIWARE Data Market: + + Extns. • Integrated with FIWARE Lab IdM (OAuth2) – Users do not need to have different accounts – Same user than in the rest of FIWARE Portals (Store, WireCloud,…) – Users can access the portal without log in to read open data • Ability to create private datasets – Accessible only by certain users • Ability to publish datasets in the WStore – Manage the users that can access private datasets – Charge users for accessing your data – Ensure that users only access your data under some legal terms – … • https://data.lab.fiware.org 11
- 11. FIWARE Data Market: + + Extns. 12 CKAN Publish Data Offer Notify Acquisition Data Access Payment Gateway Charge Users Data Access Dataset WStore Store Publisher Private Datasets Accounting (Not implemented yet)
- 13. Publishing and Managing Data Sergio García Gómez 14
- 14. Publishing and Managing Data 15
- 15. Publishing a Dataset in the • WStore is the showcase of FIWARE – Advertise your data for free – Make your data visible to prospective users • Manage users that can access your data – Update the list of allowed users automatically • Force users to accept some terms to use your data • Charge users for accessing your data – Single Payment – Subscription – Pay per use (soon) 16
- 16. Publishing a Dataset in the 17 Open Data can only be sold for free
- 17. Increasing your datasets value • MashUp Platform widgets 18
- 18. Increasing your datasets value • Sell your datasets together with the Mashup to visualize the data – Add extra value – Ease users the utilization of the information • Enable buyers to embed the Mashup in other web pages and increase the benefits – Online Newspapers – Government – Social Networks – … 19
- 19. Increasing your datasets value 20
- 20. Publishing Context Information • Context information from Context Broker is very important but it can be difficult to find • You can use CKAN data portal as a showcase of your context information – Create a resource with format NGSI10 – Specify the entities that you want to retrieve • Users can visualize your data in a very simple way – A map will be shown if your context information refers to geographical data 21
- 21. Publishing Context Information 22 Your Context Broker Instance URL
- 23. Publishing Historical Context Information • IoT-STH is a component to store context information and access it in the future – Data is retrieved from Context Broker and stored in a MongoDB – https://github.com/telefonicaid/IoT-STH • IoT-STH is already integrated with CKAN – You can create resources that include historical context information – You can download the information and display it in graphs 24
- 24. Publishing Historial Context Information 25
- 26. Search and discovery Sergio García Gómez 27
- 28. APIs • Full query/search through a JSON API – Get full datasets meta-information – Create, Update (metadata), Delete datasets and resources • CKAN (DataStore plugin) provides APIs for – Inserting, updating and deleting data – Querying data (JSON Filters, SQL, HTSQL) • API Key – X-Auth-Token: FIWARE Lab IdM Token (OAuth2) – Authorization: CKAN default token 29
- 29. APIs: Package Show 30 GET /api/action/package_show?id=european-elections-results-spain HTTP/1.1 Host: data.lab.fi-ware.org { "help": "[...]", "success": true, "result": { "license_title": "...", "maintainer": "", "relationships_as_object": [], "maintainer_email": "", "revision_timestamp": "...", "id": "...", "metadata_created": "...", "metadata_modified": "...", "author": "", "author_email": "", "state": "active", "version": "", "license_id": "notspecified", "type": "dataset", "resources": [ { [...] "state": "active", "description": "...", "format": "CSV", [...], "url_type": "upload", "name": "...", "created": "...", "url": "https://.../dataset/.../resource/.../dow nload/....csv", "webstore_url": null, "mimetype_inner": null, "position": 0, "revision_id": "...", "resource_type": null } ], + tags, group, organization and much more…
- 30. APIs: DataStore SQL GET /api/action/datastore_search_sql?sql=SELECT * from "3fd9115d-4ccb-4e37- 9418-4a7fe54605b9" HTTP/1.1 Host: data.lab.fi-ware.org 31 { "help": "[...]", "success": true, "result": { "records": [ { "Partido": "P.P.", "Votos": "4074363", "_id": 1, "_full_text": "'p.p':1", "Escanyos": "16" }, { "Partido": "PSOE", "Votos": "3596324", "_id": 2, "_full_text": "'pso':1", "Escanyos": "14" }, […] ] "fields": [ { "type": "int4", "id": "_id" }, { "type": "tsvector", "id": "_full_text" }, { "type": "text", "id": "Partido" }, { "type": "numeric", "id": "Votos" } ], "sql": "..." } }
- 31. APIs: DataStore JSON GET /api/action/datastore_search?resource_id=3fd9115d-4ccb-4e37-9418- 4a7fe54605b9&fields=Partido,Votos HTTP/1.1 Host: data.lab.fi-ware.org 32 { "help": "[...]", "success": true, "result": { "fields": [ { "type": "text", "id": "Partido" }, { "type": "numeric", "id": "Votos" } ], "records": [ { "Partido": "P.P.", "Votos": "4074363" }, { "Partido": "PSOE", "Votos": "3596324" }, { "Partido": "LA IZQUIERDA PLURAL", "Votos": "1562567" }, { "Partido": "PODEMOS", "Votos": "1245948" }, [...] ], [...] } } + the number of results, links to the next results,…
- 32. Visualizing the Dataset in the MashUp Platform 33
- 33. 34
- 34. Data Requests • What happen if the dataset a user is looking for is not published? – Users can request for data that is not already published – Every data request can be attached to an organization – Users can close the data request when it is fulfilled – Data requests can be commented to add more information • Benefits for publishers – Publishers can know the real data demands – Publishers can create specific datasets based on these demands – Publishers can increase their benefits • Currently working on: http://data.beta.nyc 35
- 35. Data Requests 36
- 36. Data Requests 37
- 37. MONETIZING APIS
- 38. Monetizing APIs • Not all is about CKAN and Data – Monetizing CKAN datasets it is a very easy process – You can take advantage of it now – You can deploy your own Data Market • Developers also want to monetize their own APIs. Is this possible? Yes!!!!!! – A proxy is required – Your APIs should be behind this proxy – The proxy will handle the requests and will account them – Proxy sends account information to the WStore periodically – WStore charge the users based on their consumption 39
- 39. 3) API Token Monetizing your APIs 40 Your Service 1)AcquireOffering 2) Request Access Token 3)APIToken Acquisition Process 5)Request 6)Data WStore API Access Accouting Proxy Accounting Information Send Accounting Info periodically
- 40. Monetizing APIs • API Key – Identifies the context • Used offering • Organization – An API can be included in more than one offering: • Offering 1: 0,05 €/call • Offering 2: 0,50 €/MB – If the user acquires the two offerings, two API keys are generated. You can save money!! • Big Request: API key for Offering 1 • Some small requests: API key for Offering 2 41
- 42. LABS Publishing and Consuming Data
- 43. Thanks!Thanks!
Editor's Notes
- Hello! I want to thank you for coming this workshop where I’ll try to teach you to monetize your API and datasets or make them available as Open Data. First of all I want to introduce myself. I’m Aitor Magán, a PhD student at Universidad Politécnica de Madrid. Specifically, I’m researcher at CoNWeT, an investigation group that takes part in the Apps/Service and Data Delivery work package of FICORE. You can contact me using the methods specified in this slide.
- Here you have the agenda. In first place, I’ll give you a brief introduction to Open Data and how Open Data is used in FIWARE. Then, I’ll elaborate myself on CKAN, an Open Data platform that is now integrated in FIWARE. On the one hand we’ll review all the options available for publishers and on the other one we’ll evaluate how clients can access data. Finally, I’ll present you a life demo where you’ll learn to publish your datasets and to gain access to datasets published by others. I hope you enjoy it!
- We are talking about Open Data but, what Open Data is? As can be seen in this slide, Open Data is a piece of information that can be used, reused, and redistributed for free. This is a very big advantage, since we don’t have to pay for this information. There are a lot of sources of Open Data: government, institutions, universities, laboratories… And we can do a lot of things with this information. For example, we can build applications based on that data or analyse that data to generate a new one mixing it with yours.
- However, it’s not all advantages. In first place, you should consider that Open Data can lack of quality. In addition, sometimes the documentation for this data is missing or insufficient. Finally, you should also take into account that not all the data is free for a lot of reasons: security, conflicting interests… So, if you want some specific information, you are forced to pay.
- FIWARE people know that Open Data is gaining a lot of importance due to the apparition of Smart Cities and they want to take advantage of it. In this way, FIWARE is the element that provides the technology to build entrepreneurs and developers ecosystem while the Lab is the place where stakeholders meet together around innovation.
- This important agreement encourages cities to take concrete actions. Cities adopt an initial open-licensed standard API (Application Programming Interface), FIWARE NGSI, which provides lightweight and simple means to gather, publish, query and subscribe context-based, real-time information. The cities will also use and improve standard data models based on experimentation and actual usage. The initial data models were chosen by mature European smart cities in the CitySDK initiative, forming the basis for a joint City Service Development Kit. Cities in the OASC Task Force will further harmonise the data models, extending the work to other domains in constant dialog with the developer community. [from: http://connectedsmartcities.eu/open-agile-smart-cities/]
- In FIWARE Lab you can find three types of data accessible by Open APIs: Datasets: geographical, census, wheather, tourism or even historical information. This data is generally provided by CKAN, the main Open Data platform in FIWARE and the one I will explain in a few minutes. Real Time information: information coming from IoT or vertical system. In this case we can find the Context Broker, an element that offers real time information about SmartCities (for example, the battery level of lampposts…) Media: Video, audio, images…
- Ok. Let’s introduce CKAN, one of the most important Open Data Platforms all over the world. CKAN is an open source project that you can install easily in your system. It’s written in Python and use a PostgresSQL to store the meta-data and the data. Regarding to the documentation you can relax since it counts with a very large documentation that ease the process of using it. Additionally, it offers extensions, a very valuable feature that allows users to modify and extend its behaviour in very different ways.
- Clients can search and discover data in a very simple way. In this fashion, users can define facets and/or keywords to get only interesting datasets based on their needs. This information can be accessed via the web interface or the REST/JSON APIs. In addition, you should also consider that CKAN offers simple tools for visualize data in tables, graphs and maps. On the other hands we find the providers, which can easily store and update their datasets to make them available. It’s important to remark that the original CKAN only allow providers to create open datasets. But do not worry! This behaviour can be modified as we will the next slide using extensions as explained before.
- Now I’ll introduce you the CKAN instance running in the FIWARE Lab. This instance is fully integrated with the IdM so users don’t have to create different accounts. Nevertheless, you must note that users are allowed to explore the datasets without logging in. Moreover, this instance allows providers to create private datasets. This has been achieved by creating a CKAN extension. Last but not least, the instance allows providers to publish datasets directly in the FIWARE Store. With this integration, providers can manage in a simple way the users that can access their private datasets. Moreover, providers can charge users and force them to conform some legal terms for using their data.
- Here we can see a brief diagram of the provided solution. CKAN delegates the authentication of users in filab identity manager. The CKAN publishes private datasets as offerings in the store which manages acquisition, including the registering of the users, accepting of terms and conditions etc. Note that this approach allows the creation of offerings that can include more digital assets than only a dataset When a concrete user acquires an offering that contains a dataset the store notifies CKAN in order to give the user access to the dataset.
- Once that we have reviewed CKAN, let’s explore how providers can uploader their own datasets. Data can be uploaded in two different ways: Harvesting data from external repositories (geospatial servers, other CKAN servers, HTML, Socrata…) Entering data via the web interface and/or the provided APIs.
- As we’ll see in the next slide, CKAN allows providers to manage the visibility, workflow and other aspects: Visibility allows users choose if the dataset is readable by anyone or only by a certain list of users. Searchable is a field used to define if your dataset is shown in the searches made by users. Only private datasets can be marked as non-searchable. Allowed users is the list of users that can access the dataset. You can fill this form or let the Store make it for you.
- Once that you have published your dataset, you must publish it in the WStore since it’s the main showcase of FIWARE. By publishing the dataset in the Store, you will be able to advertise your dataset for free and make it available for prospective users. Additionally, WStore allows you not to worry about the authorization. This process is fully done by the Store that will update automatically the list of allowed users every time a user acquires a dataset. Finally, by publishing your dataset in the WStore, you are able to charge users for using your data and/or force them to accept some legal terms.
- In this slide you can see how easy is to publish an offering in the WStore from CKAN. You have to fill the fields and your offering will be automatically published in the FIWARE Store.
- However, you must note that raw data cannot be interesting to some of your clients. For this reason, you must look for ways to increase the value of your datasets. One of this ways can be offering methods to visualize this data in tables, graphs or even maps. To do that, and taking advantage of the MashUp technology explained in the previous workshop, FIWARE offers you a set of widgets and operators that ease the representation of data. In this slide you can see some of the widgets that are being used currently.
- With these widgets and operators you can add extra value to your data. In addition, using the MashUp platform embedding capabilities, clients will be able to embed the data in external web sites and increase the benefits that you receive for your data. For example, in an election day, you can offer newspapers a MashUp with the elections result that can embedded in their webs easily, reducing drastically the development that they have to perform in this situations.
- The screenshot in this slide shows the process of creating a new offering at the point of adding resources. As we can see, we should check both the dataset and the MashUp to visualize the data. This process involves more steps that will be covered in the “Hands On” that we are performing in a few minutes.
- But that’s not all since the FIWARE CKAN instance allows users to publish context information in a very easy way. As you may know, Context Information can be difficult to find, so you can use CKAN as a showcase of the Context Information that you are publishing. Doing it is a very simple procccess: just create a resource with and set “NGSI10” as format. You also have to set the payload needed to retrieve your information from the Context Broker. Once that these parameters are set, users will be able to visualize the Context Information in a very simple way, and if your context information contains geographical data, a map will appear showing the points.
- As you know, Context Broker only provides you with the last value, but in some occasions it’s very interesting to get historical information. To do so, Telefonica has developed a new component called IoT-STH that retrieves the current data from Context Broker and stores it in a MongoDB so you can get it at any time. This tool is already integrated with CKAN (extension) so can create a dataset that contains historical context information. In addition, a graph will try to display your data.
- Once that we have reviewed how provider can publish their own data, let’s learn how clients can consume it. Clients can use three methods to discover and search data. First of all, they can access CKAN directly and use the search methods offered that allow to filter by text or facets (for example: tags, formats…). Clients can also use the API offered by CKAN to discover data in a programmatic way. And last but not least, clients can also use the FIWARE Store, where they don’t find only datasets but also MashUps and widgets that allow them to visualize the data in a proper way.
- When clients search datasets in the CKAN portal, they will observe that some dataset are marked as “private”. This dataset cannot be accessed until they acquire it. To do so, users are offered a link Store where they can see all the offerings that contain this dataset. This link is provided via the “Acquire URL” that I mentioned before. Once that the user is in the Store, they must complete the acquiring process. Then, they will be redirected to CKAN again and they’ll be able to access the dataset. 1.- The users tries to access a dataset that is private and he/she has not acquired. 2.- A link to the Store is provided where the user can acquire the dataset. 3.- The user acquire the dataset in the Store and is redirected again to the dataset since he/she can access it now.
- Users are able to access the data via the web interface provided by CKAN but one of the most important ways to access data is through the APIs offered. CKAN offers REST/JSON APIs to get datasets meta-data (for example: links to the files, tags, authors, date…). Additionally, users can create or delete datasets or update the meta-data. However, this is not enough sometimes as users want to access the data itself and update or delete it. To do that, CKAN offers an extension (called DataStore) that parse all the CSV files uploaded and creates a SQL table in order to enable users access the data in a SQL way. Moreover, users are able to use other languages such as JSON or HTSQL to access the data. Relating to the authorization you should note that you are able to set two authentication headers. The first one (“Authorization”) is the default one provided by CKAN that can be obtained in the user profile page. The second one (“X-Auth-Toke”n) is the token provided by the FIWARE IdM that allows communication among the different enablers (for example: to get a private dataset for a user in the MashUp Platform). datastore_search: Easy to use. Low flexibility. JSON. Resources can NOT be joined. datastore_search_sql: Difficult to use (SQL statements). High flexibility. Resources can be joined. HTSQL: Medium complexity and flexibility. Language: HTSQL. Resources can NOT be joined.
- This API only allows us to get meta-data from the dataset: resources (link to download), tags, organization, group, license,…
- This API is very flexible and allows us to join differnet resources. However, it’s more difficult since we have to set up the complete SQL statement.
- It’s less flexible but it’s use is simpler that the previous one.
- Finally, when users acquire an offering that includes both the dataset and the MashUp to visualize data, they must go to the MashUp portal to visualize the data. This is something that is going to be explained in a few minutes in the “Hands On” section but as can be seen is an easy process that can be performed in just three steps.