1. The document discusses ways to monetize APIs and datasets through an open data platform or data market. It describes publishing datasets on CKAN and making them available or chargeable on the WStore storefront.
2. Publishers can increase the value of their datasets by creating mashups on the platform or selling datasets bundled with visualization tools. Context and historical context information from IoT systems can also be published.
3. Consumers can search for open and private datasets, acquire offerings on the WStore to access private data, and use APIs and data requests to access additional data. Publishers are notified of acquisitions and can charge users.
We present various approaches to providing an API as part of an IoT product and discuss certain business models for monetizing it.
Centaur Technologies develops and markets end-to-end solutions for the Internet of Things, focused on the Industrial and Enterprise sectors.
- WireCloud is a web-based platform for developing and sharing mashups and applications. It allows users to visually combine ("mashup") widgets, data, and services from various sources.
- WStore is a generic online store for publishing and acquiring digital offerings such as APIs, datasets, and mashups. It supports pricing models, billing, and integration with payment systems.
- WMarket is a marketplace that aggregates offerings from multiple WStores. It allows buyers to compare and review offerings from different providers in order to select those with the best value.
Session 8 - Creating Data Processing Services | Train the Trainers ProgramFIWARE
This technical session for Local Experts in Data Sharing (LEBDs), this session will explain how to create data processing services that are key to i4Trust.
We present various approaches to providing an API as part of an IoT product and discuss certain business models for monetizing it.
Centaur Technologies develops and markets end-to-end solutions for the Internet of Things, focused on the Industrial and Enterprise sectors.
- WireCloud is a web-based platform for developing and sharing mashups and applications. It allows users to visually combine ("mashup") widgets, data, and services from various sources.
- WStore is a generic online store for publishing and acquiring digital offerings such as APIs, datasets, and mashups. It supports pricing models, billing, and integration with payment systems.
- WMarket is a marketplace that aggregates offerings from multiple WStores. It allows buyers to compare and review offerings from different providers in order to select those with the best value.
Session 8 - Creating Data Processing Services | Train the Trainers ProgramFIWARE
This technical session for Local Experts in Data Sharing (LEBDs), this session will explain how to create data processing services that are key to i4Trust.
Orion context broker webminar 2013 06-19Fermin Galan
The document discusses Orion Context Broker, an implementation of a context information broker within the FI-WARE platform. It implements the OMA NGSI9/10 specification for managing context information and availability. Orion acts as both a Pub/Sub Context Broker generic enabler and Configuration Management generic enabler, allowing context producers to publish context data and consumers to retrieve it via operations like updateContext, queryContext, and subscribeContext. Context brokers can also federate by subscribing and registering with each other.
Building a modern in-house analytics pipelineSergey Burkov
Data-driven organizations need a modern data pipeline to ensure that right data is available at all times to make decisions, and as a foundation for building smart innovative services.
A carefully managed data pipeline can provide access to reliable and well-structured datasets for analytics, machine learning and research stakeholders.
Automating the movement and transformation of data allows the consolidation of data from multiple sources so that it can be used strategically.
WSO2 Data Services Server augments service-oriented architecture development efforts by providing an easy-to-use platform for integrating data stores, creating composite data views, and hosting data services. It supports secure and managed data access across federated data stores, data service transactions, and data transformation and validation using a lightweight, developer friendly, agile development approach. It provides federation support, combining data from multiple sources in single response or resource and also supports nested queries across data sources.
Scalable Data Management: Automation and the Modern Research Data PortalGlobus
Globus is an established service from the University of Chicago that is widely used for managing research data in national laboratories, campus computing centers, and HPC facilities. While its interactive web browser interface addresses simple file transfer and sharing scenarios, large scale automation typically requires integration of the research data management platform it provides into bespoke applications.
We will describe one such example, the Petrel data portal (https://petreldata.net), used by researchers to manage data in diverse fields including materials science, cosmology, machine learning, and serial crystallography. The portal facilitates automated ingest of data, extraction and addition of metadata for creating search indexes, assignment of persistent identifiers faceted search for rapid data discovery, and point-and-click downloading of datasets by authorized users. As security and privacy are often critical requirements, the portal employs fine-grained permissions that control both visibility of metadata and access to the datasets themselves. It is based on the Modern Research Data Portal design pattern, jointly developed by the ESnet and Globus teams, and leverages capabilities such as the Science DMZ for enhanced performance and to streamline the user experience.
Session 3 - i4Trust components for Identity Management and Access Control i4T...FIWARE
This session consists of two parts. The first part of the session will introduce you to i4Trust IAM components in detail while the second will introduce i4Trust Marketplace Services. Technical session for Local Experts in Data Sharing (LEBDs)
FIWARE Wednesday Webinars - FIWARE Building the FutureFIWARE
FIWARE Building the Future - 3 June 2020
Corresponding webinar recording: https://youtu.be/REoJA7yxJ_0
An in-depth look at where FIWARE is going next and integrates with blockchain and distributed ledger technologies, Artificial Intelligence or Robotics.
Chapter: Cross-Domain
Difficulty: 2
Audience: Any Technical
Presenter: Juanjo Hierro (CTO, FIWARE Foundation
Gimel is a data abstraction framework built on Apache Spark - providing unified Data Access via API & SQL to different technologies such as kafka, elastic, HBASE, Rest API, File, Object stores, Relational , etc.
We spoke about this recently in the "cloud track" in the "Scale By The Bay" Conference.
https://www.scale.bythebay.io/schedule
https://sched.co/e55D
Youtube - https://www.youtube.com/watch?v=cy8g2WZbEBI&ab_channel=FunctionalTV
https://youtu.be/m6_0iI4XDpU
1) The document discusses Orion Context Broker, which is a component of the FI-WARE platform that intermediates between context producers and consumers to manage context data or context information.
2) Context information always relates to "entities" and has a name, type, and value. Orion Context Broker uses the NGSI information model and stores context information along with metadata in a database.
3) Orion Context Broker provides REST APIs for context availability management and context management to update, query, and subscribe to context information from distributed context sources.
Day 13 - Creating Data Processing Services | Train the Trainers ProgramFIWARE
This technical session for Local Experts in Data Sharing (LEBDs), this session will explain how to create data processing services that are key to i4Trust.
Architecting An Enterprise Storage Platform Using Object StoresNiraj Tolia
This document discusses architecting an enterprise storage platform using object stores. It summarizes MagFS, a file system designed for the cloud that is layered on top of object storage. Key points include:
- MagFS provides a consistent, elastic, secure and mobile-enabled file system experience while leveraging low-cost object storage.
- The client architecture pushes intelligence to edges for heavy lifting like encryption, deduplication, and caching while coordinating with metadata servers.
- Metadata servers enforce strong consistency, authentication, and garbage collection while optimizing performance through virtualization and lease-based caching.
- Security is ensured through server-driven request signing and scrubbing writes to object storage after client acknowledgment
Interoperability in the Internet of Things is critical for emerging services and applications. In this presentation we advocate the use of IoT ‘hubs’ to aggregate things using web protocols, and suggest a staged approach to interoperability. In the context of a UK government funded project involving 8 IoT projects to address cross-domain IoT interoperability, we introduce the HyperCat IoT catalogue specification. We then describe the tools and techniques we developed to adapt an existing data portal and IoT platform to this specification, and provide an IoT hub focused on the highways industry called ‘Smart Streets’. Based on our experience developing this large scale IoT hub, we outline lessons learned which we hope will contribute to ongoing efforts to create an interoperable global IoT ecosystem.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Unified Data Access with Gimel
Deepak Chandramouli, Engineering Lead
Anisha Nainani, Sr. Software Engineer
Dr. Vladimir Bacvanski, Principal Architect (Paypal)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Myth Busters IV: I Access My Data Through APIs–Data Virtualization Can't Do ThisDenodo
Watch full webinar here: https://bit.ly/3frkTmj
When you hear data virtualization, do you think BI and analytics? If so, you’re misinformed and missing out on a whole set of possibilities and capabilities of this technology. It is probably also why you think data virtualization is of no use to you if you need to access your data through APIs.
This is why we’re back with another episode of Myth Busters!
We enter the world, or ecosystem, of APIs and API-based architectures to investigate whether data virtualization plays a role in it. Maybe we’ll even learn that it possibly can enhance API capabilities and increase the benefits?
Here’s what we’ll be exploring:
- Is there a place for data virtualization in an API strategy?
- Can data virtualization enhance the deployment and exposure of APIs?
- Can data virtualization work as a service container or as an API gateway?
- Data virtualization and GraphQL...are they really like oil and water?
ODPi Egeria provides a framework for open metadata management, supporting many use cases in the governance of data in Data Lakes – as described in the Egeria webinar on 2nd June: “Data Lake Design with Egeria”. As described in that webinar, Egeria operates across the Data Lake, without needing centralization of metadata from the different tools into a central tool or repository.
Metadata frequently describe relationships between things like Assets, Schemas, Glossaries, and Terms – and these relationships form graphs. Egeria is distributed in nature, enabling you to see a federated view of the metadata contained in multiple tools and metadata repositories. As a result, the discrete graphs naturally federate to form a distributed graph. In this session, we’ll cover the Open Metadata Repository Services (OMRS) layer that enables Egeria to operate across this distributed graph.
The document discusses internet architecture patterns for connecting embedded devices in the Internet of Things. It describes common design patterns including using embedded intelligence, connected intelligence through virtualization and abstraction, and combining local and cloud-based services through feedback loops. It also reviews standards like CoAP, LWM2M, and IPSO smart objects that provide interoperability through modular protocol stacks and common data models.
The document summarizes a presentation on developing hybrid applications with Informix. It discusses how the Informix wire listener allows unified access to JSON, relational, and time series data through MongoDB and REST APIs. It enables applications to execute SQL statements and perform joins across different data sources. Sample applications demonstrate basic CRUD operations using MongoDB and REST interfaces with Informix.
Orion context broker webminar 2013 06-19Fermin Galan
The document discusses Orion Context Broker, an implementation of a context information broker within the FI-WARE platform. It implements the OMA NGSI9/10 specification for managing context information and availability. Orion acts as both a Pub/Sub Context Broker generic enabler and Configuration Management generic enabler, allowing context producers to publish context data and consumers to retrieve it via operations like updateContext, queryContext, and subscribeContext. Context brokers can also federate by subscribing and registering with each other.
Building a modern in-house analytics pipelineSergey Burkov
Data-driven organizations need a modern data pipeline to ensure that right data is available at all times to make decisions, and as a foundation for building smart innovative services.
A carefully managed data pipeline can provide access to reliable and well-structured datasets for analytics, machine learning and research stakeholders.
Automating the movement and transformation of data allows the consolidation of data from multiple sources so that it can be used strategically.
WSO2 Data Services Server augments service-oriented architecture development efforts by providing an easy-to-use platform for integrating data stores, creating composite data views, and hosting data services. It supports secure and managed data access across federated data stores, data service transactions, and data transformation and validation using a lightweight, developer friendly, agile development approach. It provides federation support, combining data from multiple sources in single response or resource and also supports nested queries across data sources.
Scalable Data Management: Automation and the Modern Research Data PortalGlobus
Globus is an established service from the University of Chicago that is widely used for managing research data in national laboratories, campus computing centers, and HPC facilities. While its interactive web browser interface addresses simple file transfer and sharing scenarios, large scale automation typically requires integration of the research data management platform it provides into bespoke applications.
We will describe one such example, the Petrel data portal (https://petreldata.net), used by researchers to manage data in diverse fields including materials science, cosmology, machine learning, and serial crystallography. The portal facilitates automated ingest of data, extraction and addition of metadata for creating search indexes, assignment of persistent identifiers faceted search for rapid data discovery, and point-and-click downloading of datasets by authorized users. As security and privacy are often critical requirements, the portal employs fine-grained permissions that control both visibility of metadata and access to the datasets themselves. It is based on the Modern Research Data Portal design pattern, jointly developed by the ESnet and Globus teams, and leverages capabilities such as the Science DMZ for enhanced performance and to streamline the user experience.
Session 3 - i4Trust components for Identity Management and Access Control i4T...FIWARE
This session consists of two parts. The first part of the session will introduce you to i4Trust IAM components in detail while the second will introduce i4Trust Marketplace Services. Technical session for Local Experts in Data Sharing (LEBDs)
FIWARE Wednesday Webinars - FIWARE Building the FutureFIWARE
FIWARE Building the Future - 3 June 2020
Corresponding webinar recording: https://youtu.be/REoJA7yxJ_0
An in-depth look at where FIWARE is going next and integrates with blockchain and distributed ledger technologies, Artificial Intelligence or Robotics.
Chapter: Cross-Domain
Difficulty: 2
Audience: Any Technical
Presenter: Juanjo Hierro (CTO, FIWARE Foundation
Gimel is a data abstraction framework built on Apache Spark - providing unified Data Access via API & SQL to different technologies such as kafka, elastic, HBASE, Rest API, File, Object stores, Relational , etc.
We spoke about this recently in the "cloud track" in the "Scale By The Bay" Conference.
https://www.scale.bythebay.io/schedule
https://sched.co/e55D
Youtube - https://www.youtube.com/watch?v=cy8g2WZbEBI&ab_channel=FunctionalTV
https://youtu.be/m6_0iI4XDpU
1) The document discusses Orion Context Broker, which is a component of the FI-WARE platform that intermediates between context producers and consumers to manage context data or context information.
2) Context information always relates to "entities" and has a name, type, and value. Orion Context Broker uses the NGSI information model and stores context information along with metadata in a database.
3) Orion Context Broker provides REST APIs for context availability management and context management to update, query, and subscribe to context information from distributed context sources.
Day 13 - Creating Data Processing Services | Train the Trainers ProgramFIWARE
This technical session for Local Experts in Data Sharing (LEBDs), this session will explain how to create data processing services that are key to i4Trust.
Architecting An Enterprise Storage Platform Using Object StoresNiraj Tolia
This document discusses architecting an enterprise storage platform using object stores. It summarizes MagFS, a file system designed for the cloud that is layered on top of object storage. Key points include:
- MagFS provides a consistent, elastic, secure and mobile-enabled file system experience while leveraging low-cost object storage.
- The client architecture pushes intelligence to edges for heavy lifting like encryption, deduplication, and caching while coordinating with metadata servers.
- Metadata servers enforce strong consistency, authentication, and garbage collection while optimizing performance through virtualization and lease-based caching.
- Security is ensured through server-driven request signing and scrubbing writes to object storage after client acknowledgment
Interoperability in the Internet of Things is critical for emerging services and applications. In this presentation we advocate the use of IoT ‘hubs’ to aggregate things using web protocols, and suggest a staged approach to interoperability. In the context of a UK government funded project involving 8 IoT projects to address cross-domain IoT interoperability, we introduce the HyperCat IoT catalogue specification. We then describe the tools and techniques we developed to adapt an existing data portal and IoT platform to this specification, and provide an IoT hub focused on the highways industry called ‘Smart Streets’. Based on our experience developing this large scale IoT hub, we outline lessons learned which we hope will contribute to ongoing efforts to create an interoperable global IoT ecosystem.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Unified Data Access with Gimel
Deepak Chandramouli, Engineering Lead
Anisha Nainani, Sr. Software Engineer
Dr. Vladimir Bacvanski, Principal Architect (Paypal)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Myth Busters IV: I Access My Data Through APIs–Data Virtualization Can't Do ThisDenodo
Watch full webinar here: https://bit.ly/3frkTmj
When you hear data virtualization, do you think BI and analytics? If so, you’re misinformed and missing out on a whole set of possibilities and capabilities of this technology. It is probably also why you think data virtualization is of no use to you if you need to access your data through APIs.
This is why we’re back with another episode of Myth Busters!
We enter the world, or ecosystem, of APIs and API-based architectures to investigate whether data virtualization plays a role in it. Maybe we’ll even learn that it possibly can enhance API capabilities and increase the benefits?
Here’s what we’ll be exploring:
- Is there a place for data virtualization in an API strategy?
- Can data virtualization enhance the deployment and exposure of APIs?
- Can data virtualization work as a service container or as an API gateway?
- Data virtualization and GraphQL...are they really like oil and water?
ODPi Egeria provides a framework for open metadata management, supporting many use cases in the governance of data in Data Lakes – as described in the Egeria webinar on 2nd June: “Data Lake Design with Egeria”. As described in that webinar, Egeria operates across the Data Lake, without needing centralization of metadata from the different tools into a central tool or repository.
Metadata frequently describe relationships between things like Assets, Schemas, Glossaries, and Terms – and these relationships form graphs. Egeria is distributed in nature, enabling you to see a federated view of the metadata contained in multiple tools and metadata repositories. As a result, the discrete graphs naturally federate to form a distributed graph. In this session, we’ll cover the Open Metadata Repository Services (OMRS) layer that enables Egeria to operate across this distributed graph.
The document discusses internet architecture patterns for connecting embedded devices in the Internet of Things. It describes common design patterns including using embedded intelligence, connected intelligence through virtualization and abstraction, and combining local and cloud-based services through feedback loops. It also reviews standards like CoAP, LWM2M, and IPSO smart objects that provide interoperability through modular protocol stacks and common data models.
The document summarizes a presentation on developing hybrid applications with Informix. It discusses how the Informix wire listener allows unified access to JSON, relational, and time series data through MongoDB and REST APIs. It enables applications to execute SQL statements and perform joins across different data sources. Sample applications demonstrate basic CRUD operations using MongoDB and REST interfaces with Informix.
WireCloud is a platform for developing web mashups and visualizing data and applications. It allows users to integrate different data sources, applications, and UI components to create customized applications. Developers can create reusable widgets and operators using standard web technologies. Users can assemble these components visually into mashups without programming. The platform also supports consuming external APIs and FIWARE services. WireCloud fosters sharing of mashups and components through public offerings in its app store.
WireCloud is the FIWARE Application Mashup Generic Enabler Reference Implementation. This talk gives an overview of WireCloud, describes its integration with other Generic Enablers, and gives some technical information on how to write composite applications using WireCloud
Presentation of FIWARE Application Mashup GE and its reference implementation, WireCloud. It describes how to build web-based front-ends through widgets, and the features that WireCloud offers for that. Finally it goes into a hands-on workshop to build an example application
The document discusses key performance indicators (KPIs) for a real estate director position. It provides examples of KPIs, steps for creating KPIs, common mistakes to avoid, and how to design effective KPIs. The document recommends visiting kpi123.com for additional KPI samples, performance appraisal forms, methods and review phrases to help evaluate a real estate director's job performance.
Este documento describe tres tipos de riesgos ocupacionales: activos, mitigados y públicos. Explica que los riesgos son la probabilidad de lesiones o daños dependiendo del ambiente de trabajo, e incluyen enfermedades ocupacionales causadas por agentes físicos, condiciones anti-ergonómicas u otros factores. También señala que es importante identificar los peligros y evaluar los riesgos en el lugar de trabajo a través de un análisis de riesgos para establecer medidas preventivas y reducir los riesgos ocup
The document discusses the benefits of exercise for mental health. Regular physical activity can help reduce anxiety and depression and improve mood and cognitive functioning. Exercise causes chemical changes in the brain that may help boost feelings of calmness, happiness and focus.
Our mission as the voice of the students at MIT is to spread news, information, events, experiences, artistic vent outs and beautiful literature within our design habitat to each and everyone via a (sort of) bi-monthly newsletter available in print as well as the web.
Amado Nervo was a renowned Mexican poet, journalist, and diplomat from the 19th century. He successfully served as the Mexican ambassador to Argentina and Uruguay. Considered one of Mexico's most prominent poets, Nervo wrote poetry and worked as a journalist. He held occupations as a diplomat, poet, and educator in his home country of Mexico.
The document discusses key performance indicators (KPIs) for real estate administrators. It provides steps to create KPIs for real estate administrators, including defining objectives, identifying key result areas and tasks, and determining how to measure results. The document also discusses types of KPIs and cautions against creating too many KPIs or ones that do not change to suit objectives. Employers can visit kpi123.com for additional KPI samples and materials.
The Power and Value of Professional Mentoring - GWBC by Michelle DealKey Services
Key Services, Inc. President Michelle Deal presented at the GWBC 2014 Power of Partnering Marketplace on September 29, Deal shared of her experience in the Mentor Protege Program. View her speech here. Deal will serve as the Regional Voice Chair for North Carolina.
Amado Nervo was a renowned Mexican poet, ambassador, journalist and educator from the 19th century. He successfully served as the Mexican ambassador to Argentina and Uruguay. Nervo is considered one of the most prominent Mexican poets of his time. The document also includes recipes for traditional Mexican dishes like quesadillas, chapulines (grasshoppers), and chicken enchiladas.
The document discusses different ways to show respect, including respecting elders by doing as told, cleaning up, and doing chores. It also mentions following instructions, not interrupting others, obeying laws, and being a good role model by thinking positively and having a positive impact. Overall, the document provides guidance on demonstrating respect through responsible actions like cleaning up, doing chores, obeying rules and authorities, and being a good role model.
This document discusses key performance indicators (KPIs) for real estate negotiators. It provides examples of KPIs, steps for creating a KPI system, common mistakes to avoid, and how to design effective KPIs. The document recommends visiting an external website for additional KPI materials, performance appraisal forms, methods, and review phrases related to real estate negotiators.
The document discusses API and big data solutions using WSO2 products. It begins by introducing WSO2 and its open source middleware platform. It then defines APIs and API management, describing how APIs can be used for both public and internal consumption. Next, it covers big data concepts like collecting, storing, and analyzing large datasets. It proposes several patterns for integrating APIs and big data, such as using API analytics for monitoring and control, billing and metering, targeted recommendations, and exposing datasets and analytics via APIs. Finally, it provides an example use case of using API and big data products to trigger alerts when new API versions become slower.
This document discusses Linked Open Data and how to publish open government data. It explains that publishing data in open, machine-readable formats and linking it to other external data sources increases its value. It provides examples of published open government data and outlines best practices for making data open through licensing, standard formats like CSV and XML, using URIs as identifiers, and linking to related external data. The key benefits outlined are empowering others to build upon the data and improving transparency, competition and innovation.
The document provides an overview of data mesh principles and hands-on examples for implementing a data mesh. It discusses key concepts of a data mesh including data ownership by domain, treating data as a product, making data available everywhere through self-service, and federated governance of data wherever it resides. Hands-on examples are provided for creating a data mesh topology with Apache Kafka as the underlying infrastructure, developing data products within domains, and exploring consumption of real-time and historical data from the mesh.
Systems on the edge - your stepping stones into Oracle Public PaaS Cloud - AM...Lucas Jellema
Adoption of the cloud will not start with the core enterprise applications. There are several ways to start the adoption. One is to move in from training environments through development and test to production. Another takes the importance of applications into consideration, starting with secondary, supporting systems. The approach discussed in this session is to start with edge systems that are already in the DMZ, on the fringes of an enterprises, where they engage in interaction with the outside world.
Systems on the edge of an enterprise have special challenges regarding availability, scalability, security and external interactions with systems or people. This applies for example to external portals, B2B interactions, workflows that involve external actors, mobile APIs and integrations with SaaS instances. These systems are obvious candidates to move to a public cloud - and handle these special requirements on the PaaS platform. This session discusses and demonstrates a number of Oracle PaaS Cloud Services, their mutual interaction and how they can be leveraged to move these systems over the edge and into the cloud: Java Cloud Service, Integration Cloud Service, Process Cloud Service, IoT CS, Mobile Cloud Service, SOA Suite Cloud Service and Message Cloud Service. We will go over a number of scenarios for moving edge systems from on premises to the public cloud. Essential in this discussion is of course the integration from the edge system in the Oracle Public Cloud to the on premises backend systems.
The document describes the EGI Marketplace, which allows users to easily discover, access, order, and use EGI-related computing and data services. The marketplace brings several benefits, including discoverability of services, efficient sharing of resources, and facilitating interdisciplinary research. Users can gain access to EGI services through a three-step process: 1) authenticating and registering, 2) discovering and customizing available services, and 3) ordering and requesting services. The marketplace aims to enhance the user experience and support various types of users and ordering processes.
EUBra-BIGSEA is a European-Brazilian consortium that developed a cloud-centric big data scientific research platform comprising a quality of service data analytics platform with six layers and 15 new components integrated with 5 existing components. The platform supports defining and running highly scalable and privacy-aware data analytic applications on cloud infrastructures. It was demonstrated by implementing applications for analyzing public transportation data.
This document discusses analytics and IoT. It covers key topics like data collection from IoT sensors, data storage and processing using big data tools, and performing descriptive, predictive, and prescriptive analytics. Cloud platforms and visualization tools that can be used to build end-to-end IoT and analytics solutions are also presented. The document provides an overview of building IoT solutions for collecting, analyzing, and gaining insights from sensor data.
This document discusses data marketplaces and the potential benefits of linked data for data marketplaces. It provides an overview of several existing data marketplaces including Factual, InfoChimps, Azure DataMarket, Freebase, Socrata, and Kasabi. These marketplaces vary in their data domains, models, sizes, monetization approaches, and tools for data access. The document also outlines benefits of the semantic web and linked data for data marketplaces, such as unified data representation, global identifiers, interlinked datasets, and easy integration of existing linked open data. However, challenges include ensuring data quality and performing large-scale data integration across different schemas.
What exactly is big data? The definition of big data is data that contains greater variety, arriving in increasing volumes and with more velocity. This is also known as the three Vs. Put simply, big data is larger, more complex data sets, especially from new data sources.
How to scale your PaaS with OVH infrastructure?OVHcloud
ForePaaS provides a platform for data infrastructure automation that allows customers to collect, store, transform and analyze data across multiple cloud providers or on-premise in a unified manner. Key features of the ForePaaS platform include being end-to-end, multi-cloud, providing a marketplace for sharing elements of work, and offering automated infrastructure that scales based on customer needs. ForePaaS has partnered with OVH to leverage their public cloud, private cloud, and bare metal server offerings to power ForePaaS infrastructure globally.
Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...Seeling Cheung
Citizens Bank was implementing a BigInsights Hadoop Data Lake with PureData System for Analytics to support all internal data initiatives and improve the customer experience. Testing BigInsights on the ViON Hadoop Appliance yielded the productivity, maintenance, and performance Citizens was looking for. Citizens Bank moved some analytics processing from Teradata to Netezza for better cost and performance, implemented BigInsights Hadoop for a data lake, and avoided large capital expenditures for additional Teradata capacity.
This document provides an introduction to a course on big data. It outlines the instructor and TA contact information. The topics that will be covered include data analytics, Hadoop/MapReduce programming, graph databases and analytics. Big data is defined as data sets that are too large and complex for traditional database tools to handle. The challenges of big data include capturing, storing, analyzing and visualizing large, complex data from many sources. Key aspects of big data are the volume, variety and velocity of data. Cloud computing, virtualization, and service-oriented architectures are important enabling technologies for big data. The course will use Hadoop and related tools for distributed data processing and analytics. Assessment will include homework, a group project, and class
This document provides an introduction to a course on big data and analytics. It outlines the following key points:
- The instructor and TA contact information and course homepage.
- The course will cover foundational data analytics, Hadoop/MapReduce programming, graph databases, and other big data topics.
- Big data is defined as data that is too large or complex for traditional database tools to process. It is characterized by high volume, velocity, and variety.
- Examples of big data sources and the exponential growth of data volumes are provided. Real-time analytics and fast data processing are also discussed.
Talk given at Open Knowledge Foundation 'Opening Up Metadata: Challenges, Standards and Tools' Workshop, Queen Mary University of London, 13th June 2012.
Info on the event at http://openglam.org/2012/05/31/last-places-left-for-opening-up-metadata-challenges-standards-and-tools/
FIWARE Wednesday Webinars - NGSI-LD and Smart Data Models: Standard Access to...FIWARE
NGSI-LD and Smart Data Models: Standard Access to Digital Twin Data - 15 July 2020
Corresponding webinar recording: https://youtu.be/MBx23ypORLk
Understanding the basis of context information management, NGSI-LD and smart Data Models
Chapter: Core
Difficulty: 2
Audience: Any Technical
Speaker: Juanjo Hierro (CTO, FIWARE Foundation), Alberto Abella (Data Modeling Expert and Technical Evangelist, FIWARE Foundation)
Big data is generated from a variety of sources like web data, purchases, social networks, sensors, and IoT devices. Telecom companies process exabytes and zettabytes of data daily, including call detail records, network configuration data, and customer information. This big data is analyzed to enhance customer experience through personalization, predict churn, and optimize networks. Analytics also helps with operations, data monetization through services, and identifying new revenue streams from IoT and M2M data. Frameworks like Hadoop and MapReduce are used to analyze this distributed big data across clusters in a distributed manner for faster insights.
This document provides an overview of big data and business analytics. It discusses the key characteristics of big data, including volume, variety, and velocity. Volume refers to the enormous and growing amount of data being generated. Variety means data comes in all types from structured to unstructured. Velocity indicates that data is being created in real-time and needs to be analyzed rapidly. The document also outlines some of the challenges of big data and how cloud computing and technologies like Hadoop are helping to manage and analyze large, complex data sets.
Similar to Monetize your APIs and datasets or make them available as open data (20)
Explore the key differences between silicone sponge rubber and foam rubber in this comprehensive presentation. Learn about their unique properties, manufacturing processes, and applications across various industries. Discover how each material performs in terms of temperature resistance, chemical resistance, and cost-effectiveness. Gain insights from real-world case studies and make informed decisions for your projects.
Monetize your APIs and datasets or make them available as open data
1. Monetize your APIs and Datasets or Make Them
Available as Open Data
Aitor Magán (UPM)
amagan@conwet.com
@AitorMagan
2. Agenda
1. Open Data
• Open Data in FIWARE Lab
2. Data Market: CKAN + WStore + Extensions
• Providing Data
Creating a dataset
Publish the Dataset in the WStore
Increase Data Value
Publishing Context Information
• Consuming Data
Search
Acquisition
APIs
Data Requests
3. Monetizing APIs
4. LAB
2
3. “A piece of data or content is open if
anyone is free to use, reuse, and
redistribute it — subject only, at most,
to the requirement to attribute and/or
share-alike.”
[http://opendefinition.org/#sthash.6ieidzit.dpuf]
Sergio García Gómez 3
4. • Open Data implies often hidden costs
– Lack of quality
– Insufficient documentation
– Unstructured information
– Limited amount of information
• All data cannot be open
– Conflicting interests
– Security
4
5. Smat Cities: OASC initiative – 31 cities
6
Helsinki
Copenhagen
Brussels
Ghent
Lisbon
Porto
Milan
Sevilla
Valencia
Porto Alegre
Oulu
Málaga
Palermo
Antwerp
Santander
Aarhus
Tocantis
Ventaa
Vitória
Fundão
Lecce
Espoo
Olinda
Colinas de
Tocantins
Taquaritinga
Penela
Anapólis
6. Open Data/Content approaches
Datasets
Existing Datasets
(census,
geographical,
tourism,...)
Historic Data (from
sensors, events...)
Real Time
Vertical Systems
(mobility, events...)
Internet of Things
(sensors, Smart
meters...)
Media
Video streams
(traffic,
surveillance..)
Audio
(microphones),
speaches...
Applications
NGSICKAN WEBRTC
KURENTO
7Sergio García Gómez
8. Open Data Platform
• De facto standard platform for Open Data in Europe and
beyond
• Plenty of extensions: harvesting, geographical information,
data visualization…
• Search & Discover Data:
– Search by keywords
– Browse by facets
– Explore data with previews & visualization
– REST/JSON APIs to access data and metadata
• Data Management for publishers
– Easy store & update of metadata
Sergio García Gómez 9
9. WStore
• Generic Online Store
– Publish offerings with services
– Acquire these offerings
• Extension to support specific resource types
– WireCloud components: widgtets, operators and MashUps
– Datasets
– …
• Key features
– Support accounting callbacks
– Support for rich price models
– Support for charging
– Support for billing
– Integration with PayPal
– Support for FIWARE MarketPlace, Repository and RSSS
10
Offering 1
Free use
Resource 1
Widget
Resource 2
Dataset
Offering 2
8 €
10. FIWARE Data Market: + + Extns.
• Integrated with FIWARE Lab IdM (OAuth2)
– Users do not need to have different accounts
– Same user than in the rest of FIWARE Portals (Store,
WireCloud,…)
– Users can access the portal without log in to read open data
• Ability to create private datasets
– Accessible only by certain users
• Ability to publish datasets in the WStore
– Manage the users that can access private datasets
– Charge users for accessing your data
– Ensure that users only access your data under some legal terms
– …
• https://data.lab.fiware.org
11
11. FIWARE Data Market: + + Extns.
12
CKAN
Publish Data Offer
Notify Acquisition
Data Access
Payment
Gateway
Charge
Users
Data
Access
Dataset
WStore
Store
Publisher
Private
Datasets
Accounting
(Not implemented yet)
15. Publishing a Dataset in the
• WStore is the showcase of FIWARE
– Advertise your data for free
– Make your data visible to prospective users
• Manage users that can access your data
– Update the list of allowed users automatically
• Force users to accept some terms to use your data
• Charge users for accessing your data
– Single Payment
– Subscription
– Pay per use (soon)
16
18. Increasing your datasets value
• Sell your datasets together with the Mashup to
visualize the data
– Add extra value
– Ease users the utilization of the information
• Enable buyers to embed the Mashup in other web
pages and increase the benefits
– Online Newspapers
– Government
– Social Networks
– …
19
20. Publishing Context Information
• Context information from Context Broker is very
important but it can be difficult to find
• You can use CKAN data portal as a showcase of
your context information
– Create a resource with format NGSI10
– Specify the entities that you want to retrieve
• Users can visualize your data in a very simple
way
– A map will be shown if your context information refers
to geographical data
21
23. Publishing Historical Context Information
• IoT-STH is a component to store context
information and access it in the future
– Data is retrieved from Context Broker and stored in a
MongoDB
– https://github.com/telefonicaid/IoT-STH
• IoT-STH is already integrated with CKAN
– You can create resources that include historical context
information
– You can download the information and display it in
graphs
24
28. APIs
• Full query/search through a JSON API
– Get full datasets meta-information
– Create, Update (metadata), Delete datasets and
resources
• CKAN (DataStore plugin) provides APIs for
– Inserting, updating and deleting data
– Querying data (JSON Filters, SQL, HTSQL)
• API Key
– X-Auth-Token: FIWARE Lab IdM Token (OAuth2)
– Authorization: CKAN default token
29
34. Data Requests
• What happen if the dataset a user is looking for is
not published?
– Users can request for data that is not already published
– Every data request can be attached to an organization
– Users can close the data request when it is fulfilled
– Data requests can be commented to add more information
• Benefits for publishers
– Publishers can know the real data demands
– Publishers can create specific datasets based on these demands
– Publishers can increase their benefits
• Currently working on: http://data.beta.nyc
35
38. Monetizing APIs
• Not all is about CKAN and Data
– Monetizing CKAN datasets it is a very easy process
– You can take advantage of it now
– You can deploy your own Data Market
• Developers also want to monetize their own APIs. Is
this possible? Yes!!!!!!
– A proxy is required
– Your APIs should be behind this proxy
– The proxy will handle the requests and will account them
– Proxy sends account information to the WStore periodically
– WStore charge the users based on their consumption
39
39. 3) API Token
Monetizing your APIs
40
Your Service
1)AcquireOffering
2) Request Access Token
3)APIToken
Acquisition Process
5)Request
6)Data
WStore
API Access
Accouting
Proxy
Accounting Information
Send Accounting Info
periodically
40. Monetizing APIs
• API Key
– Identifies the context
• Used offering
• Organization
– An API can be included in more than one offering:
• Offering 1: 0,05 €/call
• Offering 2: 0,50 €/MB
– If the user acquires the two offerings, two API keys
are generated. You can save money!!
• Big Request: API key for Offering 1
• Some small requests: API key for Offering 2
41
Hello! I want to thank you for coming this workshop where I’ll try to teach you to monetize your API and datasets or make them available as Open Data.
First of all I want to introduce myself. I’m Aitor Magán, a PhD student at Universidad Politécnica de Madrid. Specifically, I’m researcher at CoNWeT, an investigation group that takes part in the Apps/Service and Data Delivery work package of FICORE. You can contact me using the methods specified in this slide.
Here you have the agenda. In first place, I’ll give you a brief introduction to Open Data and how Open Data is used in FIWARE. Then, I’ll elaborate myself on CKAN, an Open Data platform that is now integrated in FIWARE. On the one hand we’ll review all the options available for publishers and on the other one we’ll evaluate how clients can access data. Finally, I’ll present you a life demo where you’ll learn to publish your datasets and to gain access to datasets published by others. I hope you enjoy it!
We are talking about Open Data but, what Open Data is? As can be seen in this slide, Open Data is a piece of information that can be used, reused, and redistributed for free. This is a very big advantage, since we don’t have to pay for this information. There are a lot of sources of Open Data: government, institutions, universities, laboratories… And we can do a lot of things with this information. For example, we can build applications based on that data or analyse that data to generate a new one mixing it with yours.
However, it’s not all advantages. In first place, you should consider that Open Data can lack of quality. In addition, sometimes the documentation for this data is missing or insufficient. Finally, you should also take into account that not all the data is free for a lot of reasons: security, conflicting interests… So, if you want some specific information, you are forced to pay.
FIWARE people know that Open Data is gaining a lot of importance due to the apparition of Smart Cities and they want to take advantage of it. In this way, FIWARE is the element that provides the technology to build entrepreneurs and developers ecosystem while the Lab is the place where stakeholders meet together around innovation.
This important agreement encourages cities to take concrete actions. Cities adopt an initial open-licensed standard API (Application Programming Interface), FIWARE NGSI, which provides lightweight and simple means to gather, publish, query and subscribe context-based, real-time information. The cities will also use and improve standard data models based on experimentation and actual usage. The initial data models were chosen by mature European smart cities in the CitySDK initiative, forming the basis for a joint City Service Development Kit. Cities in the OASC Task Force will further harmonise the data models, extending the work to other domains in constant dialog with the developer community.
[from: http://connectedsmartcities.eu/open-agile-smart-cities/]
In FIWARE Lab you can find three types of data accessible by Open APIs:
Datasets: geographical, census, wheather, tourism or even historical information. This data is generally provided by CKAN, the main Open Data platform in FIWARE and the one I will explain in a few minutes.
Real Time information: information coming from IoT or vertical system. In this case we can find the Context Broker, an element that offers real time information about SmartCities (for example, the battery level of lampposts…)
Media: Video, audio, images…
Ok. Let’s introduce CKAN, one of the most important Open Data Platforms all over the world.
CKAN is an open source project that you can install easily in your system. It’s written in Python and use a PostgresSQL to store the meta-data and the data. Regarding to the documentation you can relax since it counts with a very large documentation that ease the process of using it. Additionally, it offers extensions, a very valuable feature that allows users to modify and extend its behaviour in very different ways.
Clients can search and discover data in a very simple way. In this fashion, users can define facets and/or keywords to get only interesting datasets based on their needs. This information can be accessed via the web interface or the REST/JSON APIs. In addition, you should also consider that CKAN offers simple tools for visualize data in tables, graphs and maps.
On the other hands we find the providers, which can easily store and update their datasets to make them available. It’s important to remark that the original CKAN only allow providers to create open datasets. But do not worry! This behaviour can be modified as we will the next slide using extensions as explained before.
Now I’ll introduce you the CKAN instance running in the FIWARE Lab. This instance is fully integrated with the IdM so users don’t have to create different accounts. Nevertheless, you must note that users are allowed to explore the datasets without logging in. Moreover, this instance allows providers to create private datasets. This has been achieved by creating a CKAN extension.
Last but not least, the instance allows providers to publish datasets directly in the FIWARE Store. With this integration, providers can manage in a simple way the users that can access their private datasets. Moreover, providers can charge users and force them to conform some legal terms for using their data.
Here we can see a brief diagram of the provided solution.
CKAN delegates the authentication of users in filab identity manager.
The CKAN publishes private datasets as offerings in the store which manages acquisition, including the registering of the users, accepting of terms and conditions etc.
Note that this approach allows the creation of offerings that can include more digital assets than only a dataset
When a concrete user acquires an offering that contains a dataset the store notifies CKAN in order to give the user access to the dataset.
Once that we have reviewed CKAN, let’s explore how providers can uploader their own datasets. Data can be uploaded in two different ways:
Harvesting data from external repositories (geospatial servers, other CKAN servers, HTML, Socrata…)
Entering data via the web interface and/or the provided APIs.
As we’ll see in the next slide, CKAN allows providers to manage the visibility, workflow and other aspects:
Visibility allows users choose if the dataset is readable by anyone or only by a certain list of users.
Searchable is a field used to define if your dataset is shown in the searches made by users. Only private datasets can be marked as non-searchable.
Allowed users is the list of users that can access the dataset. You can fill this form or let the Store make it for you.
Once that you have published your dataset, you must publish it in the WStore since it’s the main showcase of FIWARE. By publishing the dataset in the Store, you will be able to advertise your dataset for free and make it available for prospective users. Additionally, WStore allows you not to worry about the authorization. This process is fully done by the Store that will update automatically the list of allowed users every time a user acquires a dataset. Finally, by publishing your dataset in the WStore, you are able to charge users for using your data and/or force them to accept some legal terms.
In this slide you can see how easy is to publish an offering in the WStore from CKAN. You have to fill the fields and your offering will be automatically published in the FIWARE Store.
However, you must note that raw data cannot be interesting to some of your clients. For this reason, you must look for ways to increase the value of your datasets. One of this ways can be offering methods to visualize this data in tables, graphs or even maps. To do that, and taking advantage of the MashUp technology explained in the previous workshop, FIWARE offers you a set of widgets and operators that ease the representation of data. In this slide you can see some of the widgets that are being used currently.
With these widgets and operators you can add extra value to your data. In addition, using the MashUp platform embedding capabilities, clients will be able to embed the data in external web sites and increase the benefits that you receive for your data. For example, in an election day, you can offer newspapers a MashUp with the elections result that can embedded in their webs easily, reducing drastically the development that they have to perform in this situations.
The screenshot in this slide shows the process of creating a new offering at the point of adding resources. As we can see, we should check both the dataset and the MashUp to visualize the data. This process involves more steps that will be covered in the “Hands On” that we are performing in a few minutes.
But that’s not all since the FIWARE CKAN instance allows users to publish context information in a very easy way. As you may know, Context Information can be difficult to find, so you can use CKAN as a showcase of the Context Information that you are publishing. Doing it is a very simple procccess: just create a resource with and set “NGSI10” as format. You also have to set the payload needed to retrieve your information from the Context Broker. Once that these parameters are set, users will be able to visualize the Context Information in a very simple way, and if your context information contains geographical data, a map will appear showing the points.
As you know, Context Broker only provides you with the last value, but in some occasions it’s very interesting to get historical information. To do so, Telefonica has developed a new component called IoT-STH that retrieves the current data from Context Broker and stores it in a MongoDB so you can get it at any time. This tool is already integrated with CKAN (extension) so can create a dataset that contains historical context information. In addition, a graph will try to display your data.
Once that we have reviewed how provider can publish their own data, let’s learn how clients can consume it. Clients can use three methods to discover and search data. First of all, they can access CKAN directly and use the search methods offered that allow to filter by text or facets (for example: tags, formats…). Clients can also use the API offered by CKAN to discover data in a programmatic way. And last but not least, clients can also use the FIWARE Store, where they don’t find only datasets but also MashUps and widgets that allow them to visualize the data in a proper way.
When clients search datasets in the CKAN portal, they will observe that some dataset are marked as “private”. This dataset cannot be accessed until they acquire it. To do so, users are offered a link Store where they can see all the offerings that contain this dataset. This link is provided via the “Acquire URL” that I mentioned before. Once that the user is in the Store, they must complete the acquiring process. Then, they will be redirected to CKAN again and they’ll be able to access the dataset.
1.- The users tries to access a dataset that is private and he/she has not acquired.
2.- A link to the Store is provided where the user can acquire the dataset.
3.- The user acquire the dataset in the Store and is redirected again to the dataset since he/she can access it now.
Users are able to access the data via the web interface provided by CKAN but one of the most important ways to access data is through the APIs offered. CKAN offers REST/JSON APIs to get datasets meta-data (for example: links to the files, tags, authors, date…). Additionally, users can create or delete datasets or update the meta-data. However, this is not enough sometimes as users want to access the data itself and update or delete it. To do that, CKAN offers an extension (called DataStore) that parse all the CSV files uploaded and creates a SQL table in order to enable users access the data in a SQL way. Moreover, users are able to use other languages such as JSON or HTSQL to access the data.
Relating to the authorization you should note that you are able to set two authentication headers. The first one (“Authorization”) is the default one provided by CKAN that can be obtained in the user profile page. The second one (“X-Auth-Toke”n) is the token provided by the FIWARE IdM that allows communication among the different enablers (for example: to get a private dataset for a user in the MashUp Platform).
datastore_search: Easy to use. Low flexibility. JSON. Resources can NOT be joined.
datastore_search_sql: Difficult to use (SQL statements). High flexibility. Resources can be joined.
HTSQL: Medium complexity and flexibility. Language: HTSQL. Resources can NOT be joined.
This API only allows us to get meta-data from the dataset: resources (link to download), tags, organization, group, license,…
This API is very flexible and allows us to join differnet resources. However, it’s more difficult since we have to set up the complete SQL statement.
It’s less flexible but it’s use is simpler that the previous one.
Finally, when users acquire an offering that includes both the dataset and the MashUp to visualize data, they must go to the MashUp portal to visualize the data. This is something that is going to be explained in a few minutes in the “Hands On” section but as can be seen is an easy process that can be performed in just three steps.