IoT and Business don't depend on data, but on processes.
So choosing a relational Db is not always the correct choice. In an IoT scenario, is better finding a data solution to store data with more performance: NoSQL databases. We'll see DocumentDb, the NoSql Db from Microsoft in Azure. But there are also other alternatives!
MongoDB IoT City Tour EINDHOVEN: Industry 4.0 and the Internet of Things: Inm...MongoDB
Presented by, Timo Klingenmeier, General Manager, inmation GmbH & Co. KG
Industry 4.0 and the Internet of Things translate to new opportunities and new challenges for manufacturing corporations. The informed workforce requires access to vast amounts of data. Handling floods of data from the production floor across the organisation and in exchange with selected business partners is not a simple task, but when achieved it turns into a major competitive advantage. MongoDB’s strategic partner inmation has developed a middleware solution which combines intelligent industrial real-time connectivity with unlimited scalability and one uniform storage layer for all kinds of data structures – MongoDB.
DOAN DuyHai – Cassandra: real world best use-cases and worst anti-patterns - ...NoSQLmatters
DOAN DuyHai – Cassandra: real world best use-cases and worst anti-patterns
In this session, you'll see how to leverage the best features of Cassandra to solve real world issues (Rate limiting/anti fraud system, account validation, security token …). We'll also highlight some common anti-patterns (queue,partition key miss,CQL3 null) and see how to solve them in the Cassandra way.
MongoDB IoT City Tour LONDON: Why your Dad's database won't work for IoT. Joe...MongoDB
Presented by, Joe Drumgoole, Director of Solutions Architecture EMEA, MongoDB.
IoT is the next big paradigm shift in computing. The move to super-dense sensor networks creates a completely new set of opportunities and challenges for developers, designers and end-users. In this context we need a new kind of storage medium.
MongoDB IoT City Tour EINDHOVEN: Managing the Database ComplexityMongoDB
The value of the fast growing class of NoSQL databases is the ability to handle high velocity and volumes of data while enabling greater agility with dynamic schemas. MongoDB gives you those benefits while also providing a rich querying capability and a document model for developer productivity. Arthur Viegers will outline the reasons for MongoDB's popularity in IoT applications and how you can leverage the core concepts of NoSQL to build robust and highly scalable IoT applications.
Webinar: The Performance Challenge: Providing an Amazing Customer Experience ...DataStax
Building and managing cloud applications is not easy. Delivering one with an amazing customer experience is even harder. Join us for “The Performance Challenge: Providing an Amazing Customer Experience No Matter What” webinar where we will deep dive into the challenges customers face with providing a consistent experience no matter where customers are, providing real-time access to data and how DataStax Enterprise can help.
Link to recording: https://youtu.be/qBGsyNulCOs
View past DataStax webinars: http://www.datastax.com/resources/webinars
MongoDB IoT City Tour EINDHOVEN: Industry 4.0 and the Internet of Things: Inm...MongoDB
Presented by, Timo Klingenmeier, General Manager, inmation GmbH & Co. KG
Industry 4.0 and the Internet of Things translate to new opportunities and new challenges for manufacturing corporations. The informed workforce requires access to vast amounts of data. Handling floods of data from the production floor across the organisation and in exchange with selected business partners is not a simple task, but when achieved it turns into a major competitive advantage. MongoDB’s strategic partner inmation has developed a middleware solution which combines intelligent industrial real-time connectivity with unlimited scalability and one uniform storage layer for all kinds of data structures – MongoDB.
DOAN DuyHai – Cassandra: real world best use-cases and worst anti-patterns - ...NoSQLmatters
DOAN DuyHai – Cassandra: real world best use-cases and worst anti-patterns
In this session, you'll see how to leverage the best features of Cassandra to solve real world issues (Rate limiting/anti fraud system, account validation, security token …). We'll also highlight some common anti-patterns (queue,partition key miss,CQL3 null) and see how to solve them in the Cassandra way.
MongoDB IoT City Tour LONDON: Why your Dad's database won't work for IoT. Joe...MongoDB
Presented by, Joe Drumgoole, Director of Solutions Architecture EMEA, MongoDB.
IoT is the next big paradigm shift in computing. The move to super-dense sensor networks creates a completely new set of opportunities and challenges for developers, designers and end-users. In this context we need a new kind of storage medium.
MongoDB IoT City Tour EINDHOVEN: Managing the Database ComplexityMongoDB
The value of the fast growing class of NoSQL databases is the ability to handle high velocity and volumes of data while enabling greater agility with dynamic schemas. MongoDB gives you those benefits while also providing a rich querying capability and a document model for developer productivity. Arthur Viegers will outline the reasons for MongoDB's popularity in IoT applications and how you can leverage the core concepts of NoSQL to build robust and highly scalable IoT applications.
Webinar: The Performance Challenge: Providing an Amazing Customer Experience ...DataStax
Building and managing cloud applications is not easy. Delivering one with an amazing customer experience is even harder. Join us for “The Performance Challenge: Providing an Amazing Customer Experience No Matter What” webinar where we will deep dive into the challenges customers face with providing a consistent experience no matter where customers are, providing real-time access to data and how DataStax Enterprise can help.
Link to recording: https://youtu.be/qBGsyNulCOs
View past DataStax webinars: http://www.datastax.com/resources/webinars
Real World Use Case with Cassandra (Eddie Satterly, DataNexus) | C* Summit 2016DataStax
Discuss how IAG, Australia's leading Insurance company, uses Cassandra to change the way they integrate and leverage their data. First use case is in moving from Silos of data to enabling near real time analytics on critical digital application data leveraging kafka for data flows to Cassandra. Then leveraging Solr and Spark nodes to enable real time search and analytics. Second use case is in collecting new telemetry and telematics data to enable real time feedback to customers that initially shows 7-20% fuel savings for gas vehicles. This will also help customers to avoid heavy accident risk areas of cities during peak times and eventually suggest alternate routes and commute timing to reduce risk.
About the Speaker
Eddie Satterly Co-Founder, DataNexus
Eddie has served in a variety of roles including developer, engineer, architect and CTO over his 27+ year career. At DataNexus he is building a new big data application. Previously he was CTO of the Emerging Technologies at CSC running product portfolio and R&D teams in: Cyber, Analytics, Cloud, Mobile & IoT. Prior in CTO Office at Splunk where he presented at events and set data strategy. Prior he revolutionized how Expedia delivers service to improve user experience using Cassandra. MVP
VoltDB and HPE Vertica Present: Building an IoT Architecture for Fast + Big DataVoltDB
This webinar with Chris Selland of HPE Vertica and Dennis Duckworth of VoltDB addresses the growing challenges with managing a complex IoT solution and how to enable real-time operational interaction with comprehensive data analytics.
Hype, buzzword, threat; however you want to characterize it, the Internet of Things (IoT) is here.
IoT scenarios that were hypothetical only a few years ago are real today. Still thinking along the line of fleet management and temperature measurements? You’re out. Endless possibilities of IoT applications are surfacing every day, from the connected cow (huh?) to things that monitor and analyze your daily life (really?).
In this webinar, we will discuss architecture of IoT data management solutions and the challenges that arise. We will explore how MongoDB features provide solutions to those problems. Time permitting, we will demonstrate an IoT Cloud service built on top of MongoDB.
(SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:I...Amazon Web Services
Amazon DynamoDB is a fully managed, highly scalable distributed database service. In this technical talk, we show you how to use DynamoDB to build high-scale applications like social gaming, chat, and voting. We show you how to use building blocks such as secondary indexes, conditional writes, consistent reads, and batch operations to build the higher-level functionality such as multi-item atomic writes and join queries. We also discuss best practices such as index projections, item sharding, and parallel scan for maximum scalability.
Data Modeling is the one of the first things to sink your teeth into when trying out a new database. That's why we are going to cover this foundational topic in enough detail for you to get dangerous. Data Modeling for relational databases is more than a touch different than the way it's approached with Cassandra. We will address the quintessential query-driven methodology through a couple of different use cases, including working with time series data for IoT. We will also demo a new tool to get you bootstrapped quickly with MovieLens sample data. This talk should give you the basics you need to get serious with Apache Cassandra.
Real-time Data Processing with Amazon DynamoDB Streams and AWS LambdaAmazon Web Services
DynamoDB Streams is a feature of DynamoDB that allows you to access a stream of all changes made to your DynamoDB tables in the last rolling 24 hours. You can use AWS Lambda to process event data generated from a DynamoDB Stream.
In this webinar, we will cover key Amazon DynamoDB Streams and AWS Lambda features, walk through sample use cases for real-time data processing, and discuss best practices on using the services together. We'll then demonstrate setting up Amazon DynamoDB Streams and an associated Lambda function to capture and perform custom computations on database table updates, all without setting up any infrastructure
Learning Objectives:
· Understand key Amazon DynamoDB Streams and AWS Lambda features
· Learn how to set up a real-time data modification framework using Amazon DynamoDB Streams AWS Lambda
· Learn sample use cases, best practices and tips on using AWS Lambda with Amazon DynamoDB Streams
A management introduction to IoT - Myths - Pitfalls - ChallengesSven Beauprez
An introduction of what The Internet of Things is based on an overview of our society, how an implementation of The Internet of Things looks like from a bird eye view and some pitfalls and challenges that come with IoT.
This presentation was given on several occasions to C-Level management, lawyers, students, techies,...
Getting started with azure event hubs and stream analytics servicesEastBanc Tachnologies
Author: Vladimir Bychkov, www.eastbanctech.com
The total amount of data in the world almost doubles every 2 years. Storing data for offline processing is no longer a viable business model. In the past few years, new technologies for real-time data processing emerged. Microsoft Azure offers a comprehensive set of tools to ingest and process data in motion. In this presentation we will go over and learn how to collect data from devices, how to process data in real time using Azure Stream Analytic jobs, and how to produce and handle actionable insights.
Amazon DynamoDB is a fully managed, highly scalable NoSQL database service. We will deep dive into how DynamoDB scaling and partitioning works, how to do data modeling based on access patterns using primitives such as hash/range keys, secondary indexes, conditional writes and query filters. We will also discuss how to use DynamoDB Streams to build cross-region replication and integrate with other services (such as Amazon S3, Amazon CloudSearch, Amazon ElastiCache, Amazon Redshift) to enable logging, search, analytics and caching. You will learn design patterns and best practices on how to use DynamoDB to build highly scalable applications, with the right performance characteristics at the right cost.
Data Culture Series - Keynote & Panel - Reading - 12th May 2015Jonathan Woodward
Big data. Small data. All data. You have access to an ever-expanding volume of data inside the walls of your business and out across the web. The potential in data is endless – from predicting election results to preventing the spread of epidemics. But how can you use it to your advantage to help move your business forward?
Data is growing exponentially and it’s now possible to mine and unlock insights from data in new and unexpected ways. Empower your business to take advantage of this data by harnessing the rich capabilities of Microsoft SQL Server and the familiarity of Microsoft Office to help organize, analyze, and make sense of your data—no matter the size.
A Connected Data Landscape: Virtualization and the Internet of ThingsInside Analysis
The Briefing Room with Dr. Robin Bloor and Cisco
Live Webcast March 3, 2015
Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=a75f0f379405de155800a37b2bf104db
Data at rest, data in motion - regardless of its trajectory, data remains the lifeblood of today's information economy. But finding a way to bridge old systems with new opportunities requires an innovative data strategy, one that takes advantage of multiple processing technologies. With the optimal architecture in place, companies can harness years of work in traditional information systems, while opening the door to the flood of new data sources available.
Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Robin Bloor, as he explains how data virtualization and other data technologies fundamentally change what's possible with data access, movement and analysis. He'll be briefed by David Besemer of Cisco, who will discuss how this new kind of data strategy can enable the integration of legacy systems, Cloud computing and the Internet of Things. He'll also answer questions about how Big Data and the IoT are helping to redefine the practice of data management.
Visis InsideAnalysis.com for more information.
Hoe het Azure ecosysteem een cruciale rol speelt in uw IoT-oplossing (Glenn C...Codit
“Internet of Things” verandert onze wereld. Alles kan nu via de cloud met elkaar worden verbonden. Van consumentenapparatuur, innovatieve producten voor thuis tot industriële machines... In deze sessie zal Glenn u leiden door het Azure IoT ecosysteem en uitleggen welke belangrijke onderdelen u helpen bij het integreren van uw oplossingen op het Azure IoT-platform.
Real World Use Case with Cassandra (Eddie Satterly, DataNexus) | C* Summit 2016DataStax
Discuss how IAG, Australia's leading Insurance company, uses Cassandra to change the way they integrate and leverage their data. First use case is in moving from Silos of data to enabling near real time analytics on critical digital application data leveraging kafka for data flows to Cassandra. Then leveraging Solr and Spark nodes to enable real time search and analytics. Second use case is in collecting new telemetry and telematics data to enable real time feedback to customers that initially shows 7-20% fuel savings for gas vehicles. This will also help customers to avoid heavy accident risk areas of cities during peak times and eventually suggest alternate routes and commute timing to reduce risk.
About the Speaker
Eddie Satterly Co-Founder, DataNexus
Eddie has served in a variety of roles including developer, engineer, architect and CTO over his 27+ year career. At DataNexus he is building a new big data application. Previously he was CTO of the Emerging Technologies at CSC running product portfolio and R&D teams in: Cyber, Analytics, Cloud, Mobile & IoT. Prior in CTO Office at Splunk where he presented at events and set data strategy. Prior he revolutionized how Expedia delivers service to improve user experience using Cassandra. MVP
VoltDB and HPE Vertica Present: Building an IoT Architecture for Fast + Big DataVoltDB
This webinar with Chris Selland of HPE Vertica and Dennis Duckworth of VoltDB addresses the growing challenges with managing a complex IoT solution and how to enable real-time operational interaction with comprehensive data analytics.
Hype, buzzword, threat; however you want to characterize it, the Internet of Things (IoT) is here.
IoT scenarios that were hypothetical only a few years ago are real today. Still thinking along the line of fleet management and temperature measurements? You’re out. Endless possibilities of IoT applications are surfacing every day, from the connected cow (huh?) to things that monitor and analyze your daily life (really?).
In this webinar, we will discuss architecture of IoT data management solutions and the challenges that arise. We will explore how MongoDB features provide solutions to those problems. Time permitting, we will demonstrate an IoT Cloud service built on top of MongoDB.
(SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:I...Amazon Web Services
Amazon DynamoDB is a fully managed, highly scalable distributed database service. In this technical talk, we show you how to use DynamoDB to build high-scale applications like social gaming, chat, and voting. We show you how to use building blocks such as secondary indexes, conditional writes, consistent reads, and batch operations to build the higher-level functionality such as multi-item atomic writes and join queries. We also discuss best practices such as index projections, item sharding, and parallel scan for maximum scalability.
Data Modeling is the one of the first things to sink your teeth into when trying out a new database. That's why we are going to cover this foundational topic in enough detail for you to get dangerous. Data Modeling for relational databases is more than a touch different than the way it's approached with Cassandra. We will address the quintessential query-driven methodology through a couple of different use cases, including working with time series data for IoT. We will also demo a new tool to get you bootstrapped quickly with MovieLens sample data. This talk should give you the basics you need to get serious with Apache Cassandra.
Real-time Data Processing with Amazon DynamoDB Streams and AWS LambdaAmazon Web Services
DynamoDB Streams is a feature of DynamoDB that allows you to access a stream of all changes made to your DynamoDB tables in the last rolling 24 hours. You can use AWS Lambda to process event data generated from a DynamoDB Stream.
In this webinar, we will cover key Amazon DynamoDB Streams and AWS Lambda features, walk through sample use cases for real-time data processing, and discuss best practices on using the services together. We'll then demonstrate setting up Amazon DynamoDB Streams and an associated Lambda function to capture and perform custom computations on database table updates, all without setting up any infrastructure
Learning Objectives:
· Understand key Amazon DynamoDB Streams and AWS Lambda features
· Learn how to set up a real-time data modification framework using Amazon DynamoDB Streams AWS Lambda
· Learn sample use cases, best practices and tips on using AWS Lambda with Amazon DynamoDB Streams
A management introduction to IoT - Myths - Pitfalls - ChallengesSven Beauprez
An introduction of what The Internet of Things is based on an overview of our society, how an implementation of The Internet of Things looks like from a bird eye view and some pitfalls and challenges that come with IoT.
This presentation was given on several occasions to C-Level management, lawyers, students, techies,...
Getting started with azure event hubs and stream analytics servicesEastBanc Tachnologies
Author: Vladimir Bychkov, www.eastbanctech.com
The total amount of data in the world almost doubles every 2 years. Storing data for offline processing is no longer a viable business model. In the past few years, new technologies for real-time data processing emerged. Microsoft Azure offers a comprehensive set of tools to ingest and process data in motion. In this presentation we will go over and learn how to collect data from devices, how to process data in real time using Azure Stream Analytic jobs, and how to produce and handle actionable insights.
Amazon DynamoDB is a fully managed, highly scalable NoSQL database service. We will deep dive into how DynamoDB scaling and partitioning works, how to do data modeling based on access patterns using primitives such as hash/range keys, secondary indexes, conditional writes and query filters. We will also discuss how to use DynamoDB Streams to build cross-region replication and integrate with other services (such as Amazon S3, Amazon CloudSearch, Amazon ElastiCache, Amazon Redshift) to enable logging, search, analytics and caching. You will learn design patterns and best practices on how to use DynamoDB to build highly scalable applications, with the right performance characteristics at the right cost.
Data Culture Series - Keynote & Panel - Reading - 12th May 2015Jonathan Woodward
Big data. Small data. All data. You have access to an ever-expanding volume of data inside the walls of your business and out across the web. The potential in data is endless – from predicting election results to preventing the spread of epidemics. But how can you use it to your advantage to help move your business forward?
Data is growing exponentially and it’s now possible to mine and unlock insights from data in new and unexpected ways. Empower your business to take advantage of this data by harnessing the rich capabilities of Microsoft SQL Server and the familiarity of Microsoft Office to help organize, analyze, and make sense of your data—no matter the size.
A Connected Data Landscape: Virtualization and the Internet of ThingsInside Analysis
The Briefing Room with Dr. Robin Bloor and Cisco
Live Webcast March 3, 2015
Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=a75f0f379405de155800a37b2bf104db
Data at rest, data in motion - regardless of its trajectory, data remains the lifeblood of today's information economy. But finding a way to bridge old systems with new opportunities requires an innovative data strategy, one that takes advantage of multiple processing technologies. With the optimal architecture in place, companies can harness years of work in traditional information systems, while opening the door to the flood of new data sources available.
Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Robin Bloor, as he explains how data virtualization and other data technologies fundamentally change what's possible with data access, movement and analysis. He'll be briefed by David Besemer of Cisco, who will discuss how this new kind of data strategy can enable the integration of legacy systems, Cloud computing and the Internet of Things. He'll also answer questions about how Big Data and the IoT are helping to redefine the practice of data management.
Visis InsideAnalysis.com for more information.
Hoe het Azure ecosysteem een cruciale rol speelt in uw IoT-oplossing (Glenn C...Codit
“Internet of Things” verandert onze wereld. Alles kan nu via de cloud met elkaar worden verbonden. Van consumentenapparatuur, innovatieve producten voor thuis tot industriële machines... In deze sessie zal Glenn u leiden door het Azure IoT ecosysteem en uitleggen welke belangrijke onderdelen u helpen bij het integreren van uw oplossingen op het Azure IoT-platform.
apidays LIVE London 2021 - API Horror Stories from an Unnamed Coworking Compa...apidays
apidays LIVE London 2021 - Reaching Maximum Potential in Banking & Insurance with API Mindset
October 27 & 28, 2021
Future of API Design
API Horror Stories from an Unnamed Coworking Company
Phil Sturgeon, DevRel at Stoplight
The explosive growth of the “Internet of Things” is changing our world and today the Internet of Things knows almost as many applications as there are types of devices connected.
From consumer equipment, to innovate new designs and products at home, to industrial machinery… Everything is connected to the cloud.
In this session Glenn will guide you through the Azure IoT Ecosystem and show you some of the key components of the Azure IoT Platform.
Internet of Things (IoT) - in the cloud or rather on-premises?Guido Schmutz
You want to implement a Big Data or Internet of Things (IoT) solution and like to know if it should be implemented in the cloud or on-premises. You are interested in the cloud offerings of vendors and what benefits they provide and if a similar solution would not be possible on-premises.
This presentation deals with this and other questions. Starting from a vendor-independent reference architecture and corresponding design patterns, different cloud solutions from various vendors are compared and rated. Additionally, it will be shown how such solution could be implemented on-premises and how a hybrid IoT solution could look like.
Session about "Microsoft and Internet of Things" at #NuvolaRosa - Naples (Italy) 12 May 2016
http://www.nuvolarosa.eu/corsi-napoli/
Main Themes:
Internet of Things
Windows 10 IoT Core
Windows Azure Services
Windows IoT Hub
Stream Analytics
Azure Blob Storage
Power Bi
The recently launched Microsoft IoT Central is a fully managed IoT SaaS solution that makes it easy to connect, monitor and manage your IoT assets at scale. It dramatically lowers the barriers of entry for companies looking to revolutionize their business with IoT.
We know there’s more than one approach when building an IoT Solution with the Microsoft Azure platform. With the recent arrival of Microsoft IoT Central, it’s important to determine whether you need a PaaS or SaaS offering.
In this presentation, Glenn Colpaert, Codit Azure/IoT Domain Lead and Microsoft Azure MVP, will guide you through the different offerings of the Azure platform and show you the capabilities of this new solution.
Real-time big data analytics based on product recommendations case studydeep.bi
We started as an ad network. The challenge was to recommend the best product (out of millions) to the right person in a given moment (thousands of users within a second). We have delivered 5 billion ad views since 24 months. To put it in the scale context: If we would serve 1 ad per second it will take 160 years to serve 5 billion ads.
So we needed a solution. SQL databases did not work. Popular NoSQL databases did not work. Standard data warehouse approaches (pre-aggregations, creating schemas) - did not work too.
Re-thinking all the problems with huge data streams flowing to us every second we have built a complete solution based on open-source technologies and fresh, smart ideas from our engineering team. It is called deep.bi and now we make it available to other companies.
deep.bi lets high-growth companies solve fast data problems by providing scalable, flexible and real-time data collection, enrichment and analytics.
It was built using:
- Node.js - API
- Kafka - collecting and distributing data
- Spark Streaming - ETL, data enrichments
- Druid - real-time analytics
- Cassandra - user events store
- Hadoop + Parquet + Spark - raw data store + ad-hoc queries
Meetup Toulouse Microsoft Azure : Bâtir une solution IoTAlex Danvy
Un tour d'horizon des solutions disponibles chez Microsoft pour bâtir une solution IoT. Il est question de Microsoft Azure bien-sûr, mais pas seulement. Windows, Machine Learning, Bots, OCF/AllJoyn, Hololens
Slides from my talk at NDC IoT day in Oslo 2014-11-06.
Code is available on GitHub at https://github.com/codeplanner/NDC-InternetOfThingsDay-2014-11-06
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape?
Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.
Similar to NoSQL Database in Azure for IoT and Business (20)
Normalmente parliamo e presentiamo Azure IoT (Central) con un taglio un po' da "maker". In questa sessione, invece, vediamo di parlare allo SCADA engineer. Come si configura Azure IoT Central per il mondo industriale? Dov'è OPC/UA? Cosa c'entra IoT Plug & Play in tutto questo? E Azure IoT Central...quali vantaggi ci da? Cerchiamo di rispondere a queste e ad altre domande in questa sessione...
Allo sviluppatore Azure piacciono i servizi PaaS perchè sono "pronti all'uso". Ma quando proponiamo le nostre soluzioni alle aziende, ci scontriamo con l'IT che apprezza gli elementi infrastrutturali, IaaS. Perchè non (ri)scoprirli aggiungendo anche un pizzico di Hybrid che con il recente Azure Kubernetes Services Edge Essentials si può anche usare in un hardware che si può tenere anche in casa? Quindi scopriremo in questa sessione, tra gli altri, le VNET, le VPN S2S, Azure Arc, i Private Endpoints, e AKS EE.
Static abstract members nelle interfacce di C# 11 e dintorni di .NET 7.pptxMarco Parenzan
Did interfaces in C# need evolution? Maybe yes. Are they violating some fundamental principles? We see. Are we asking for some hoops? Let's see all this by telling a story (of code, of course)
Azure Synapse Analytics for your IoT SolutionsMarco Parenzan
Let's find out in this session how Azure Synapse Analytics, with its SQL Serverless Pool, ADX, Data Factory, Notebooks, Spark can be useful for managing data analysis in an IoT solution.
Power BI Streaming Data Flow e Azure IoT Central Marco Parenzan
Dal 2015 gli utilizzatori di Power BI hanno potuto analizzare dati in real-time grazie all'integrazione con altri prodotti e servizi Microsoft. Con streaming dataflow, si porterà l'analisi in tempo reale completamente all'interno di Power BI, rimuovendo la maggior parte delle restrizioni che avevamo, integrando al contempo funzionalità di analisi chiave come la preparazione dei dati in streaming e nessuna creazione di codice. Per vederlo in funzione, studieremo un caso specifico di streaming come l'IoT con Azure IoT Central.
Power BI Streaming Data Flow e Azure IoT CentralMarco Parenzan
Dal 2015 gli utilizzatori di Power BI hanno potuto analizzare dati in real-time grazie all'integrazione con altri prodotti e servizi Microsoft. Con streaming dataflow, si porterà l'analisi in tempo reale completamente all'interno di Power BI, rimuovendo la maggior parte delle restrizioni che avevamo, integrando al contempo funzionalità di analisi chiave come la preparazione dei dati in streaming e nessuna creazione di codice. Per vederlo in funzione, studieremo un caso specifico di streaming come l'IoT con Azure IoT Central.
Power BI Streaming Data Flow e Azure IoT CentralMarco Parenzan
Since 2015, Power BI users have been able to analyze data in real-time thanks to the integration with other Microsoft products and services. With streaming dataflow, you'll bring real-time analytics completely within Power BI, removing most of the restrictions we had, while integrating key analytics features like streaming data preparation and no coding. To see it in action, we will study a specific case of streaming such as IoT with Azure IoT Central.
What are the actors? What are they used for? And how can we develop them? And how are they published and used on Azure? Let's see how it's done in this session
Generic Math, funzionalità ora schedulata per .NET 7, e Azure IoT PnP mi hanno risvegliato un argomento che nel mio passato mi hanno portato a fare due/tre viaggi, grazie all'Università di Trieste, a Cambridge (2006/2007 circa) e a Seattle (2010, quando ho parlato pubblicamente per la prima volta di Azure :) e che mi ha fatto conoscere il mito Don Box!), a parlare di codice in .NET che aveva a che fare con la matematica e con la fisica: le unità di misura e le matrici. L'avvento dei Notebook nel mondo .NET e un vecchio sogno legato alla libreria ANTLR (e tutti i miei esercizi di Code Generation) mi portano a mettere in ordine 'sto minestrone di idee...o almeno ci provo (non so se sta tutto in piedi).
322 / 5,000
Translation results
.NET is better every year for a developer who still dreams of developing a video game. Without pretensions and without talking about Unity or any other framework, just "barebones" .NET code, we will try to write a game (or parts of it) in the 80's style (because I was a kid in those years). In Christmas style.
Building IoT infrastructure on edge with .net, Raspberry PI and ESP32 to conn...Marco Parenzan
IoT scenarios necessarily pass through the Edge component and the Raspberry PI is a great way to explore this world. If we need to receive IoT events from sensors, how do I implement an MQTT endpoint? Kafka is a clever way to do this. And how do I process the data? Kafka? Spark? Rabbit ?. How do we write custom code for these environments? .NET, now in version 6 is another clever way to do it! And maybe, we can also communicate with Azure. We'll see in this session if we can make it all work!
How can you handle defects? If you are in a factory, production can produce objects with defects. Or values from sensors can tell you over time that some values are not "normal". What can you do as a developer (not a Data Scientist) with .NET o Azure to detect these anomalies? Let's see how in this session.
Quali vantaggi ci da Azure? Dal punto di vista dello sviluppo software, uno di questi è certamente la varietà dei servizi di gestione dei dati. Questo ci permette di cominciare a non essere SQL centrici ma utilizzare il servizio giusto per il problema giusto fino ad applicare una strategia di Polyglot Persistence (e vedremo cosa significa) nel rispetto di una corretta gestione delle risorse IT e delle pratiche di DevOps.
C'è ancora diffidenza nei confronti dell'Internet of Things e il costo delle soluzioni custom non aiuta. Azure IoT Central è un servizio SaaS personalizzabile che rende accessibile a costi sostenibili. Vediamo quali sonole peculiarità di questo servizio.
Come puoi gestire i difetti? Se sei in una fabbrica, la produzione può produrre oggetti con difetti. Oppure i valori dei sensori possono dirti nel tempo che alcuni valori non sono "normali". Cosa puoi fare come sviluppatore (non come Data Scientist) con .NET o Azure per rilevare queste anomalie? Vediamo come in questa sessione.
It happens that we have to develop several services and deploy them in Azure. They are small, repetitive but different, often not very different. Why not use code generation techniques to simplify the development and implementation of these services? Let's see with .NET comes to meet us and helps us to deploy in Azure.
Running Kafka and Spark on Raspberry PI with Azure and some .net magicMarco Parenzan
IoT scenarios necessarily pass through the Edge component and the Raspberry PI is a great way to explore this world. If we need to receive IoT events from sensors, how do I implement an MQTT endpoint? Kafka is a clever way to do this. And how do I process the data in Kafka? Spark is another clever way of doing this. How do we write custom code for these environments? .NET, now in version 6 is another clever way to do it! And maybe, we also communicate with Azure. We'll see in this session if we can make it all work!
Time Series Anomaly Detection with Azure and .NETTMarco Parenzan
f you have any device or source that generates values over time (also a log from a service), you want to determine if in a time frame, the time serie is correct or you can detect some anomalies. What can you do as a developer (not a Data Scientist) with .NET o Azure? Let's see how in this session.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
GraphSummit Paris - The art of the possible with Graph TechnologyNeo4j
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
In the ever-evolving landscape of technology, enterprise software development is undergoing a significant transformation. Traditional coding methods are being challenged by innovative no-code solutions, which promise to streamline and democratize the software development process.
This shift is particularly impactful for enterprises, which require robust, scalable, and efficient software to manage their operations. In this article, we will explore the various facets of enterprise software development with no-code solutions, examining their benefits, challenges, and the future potential they hold.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
3. IoT day 2015
Speaker info/Marco Parenzan
www.slideshare.net/marco.parenzan
www.github.com/marcoparenzan
marco [dot] parenzan [at] 1nn0va [dot] it
www.1nnova.it
@marco_parenzan
Formazione ,Divulgazione e Consulenza con 1nn0va
Microsoft MVP 2014 for Microsoft Azure
Cloud Architect, NET developer
Loves Functional Programming, Html5 Game Programming and Internet of Things
Microservices
Saturday 2015:
un viaggio con
NServiceBus LI
VE
AZURE
COMMUNITY
BOOTCAMP 2015
5. IoT day 2015
Data Ecosystem
Where do I put data
received in EventHub?
6. From private to public Cloud
A Continuous offering
Microsoft Relational Storage Options
7. IoT day 2015
SQL Server database technology “as a Service”
Fully managed database-as-a-service built on SQL with near zero administration
Enterprise-ready with automatic support for HA, DR, Backups, replication and more
Highly available and elastically scalable for unpredictable SaaS workloads
Uptime SLA of 99.99%
Predictable performance & Pricing
Built-in regional database geo-replication for additional protection
All core search capabilities - faceting, suggestions, geospatial
Secure and compliant for your sensitive data
Fully compatible with SQL Server 2014 databases
SQL Azure features
10. IoT day 2015
Business, no longer data, is the foundation of software design
DDD!=OOP
Don’t start from Data
Data are not unique
No more ACID…ACID transactions are not useful with a
distributed model over different storages
Paradigm Shift
11. IoT day 2015
How many queries can be determined at level analysis?
“A repository should offer an explicit and well defined contract
and avoid arbitrary query”
In business … don’t‘ delete anything (Repository doesn’t
delete anything)
From theory to practice
14. CQRS for IoT (Service Bus Powered)
Event Handler
UI
Event
Command Handler
Event
Device
Queue
Topics/Subscription
Event Hub
Write
Model
Read
/Search
Model
15. IoT day 2015
No longer build on data…but on “what happens”
No more one single data store
Data store typess
Logs
Persistence
Saga (long transactions)
Search
Event-based systems
23. What is a document database?
Definitely NOT this
kind of document !
24. What is a document database?
Not ideal, but it can work -
{
"id": "13244_post",
"text": "Lorizzle ghetto dolor tellivizzle boofron, stuff pimpin' elizzle. Nullam sapizzle
velizzle, my shizz tellivizzle, suscipizzle funky fresh, shizzle my nizzle crocodizzle
vizzle, arcu. Pellentesque eget tortizzle. Sizzle erizzle. Mammasay mammasa mamma oo sa
break it down dolor own yo' things fo shizzle mah nizzle fo rizzle, mah home g-dizzle
sure. Maurizzle pellentesque dawg ghetto turpizzle. Shiz izzle my shizz. Pellentesque
eleifend rhoncizzle nisi. In its fo rizzle owned ma nizzle dictumst. Sizzle gangsta.
Curabitur tellizzle urna, pretizzle go to hizzle, mattizzle izzle, eleifend vitae,
tellivizzle. Dawg shizzlin dizzle. Integer semper velit sizzle stuff.
Boofron mofo auctizzle ma nizzle. Pot a elizzle ut nibh pretium tincidunt. Maecenizzle
things erat. Own yo' in lacizzle sed maurizzle elementizzle tristique. I'm in the
shizzle yippiyo sizzle daahng dawg eros ultricizzle . In velit tortor, ultricizzle
ghetto, hendrerizzle fo shizzle mah nizzle fo rizzle, mah home g-dizzle, adipiscing
crunk, boom shackalack. Etizzle velit doggy, hizzle consequizzle, pharetra get down
get down, dictizzle sed, shut the shizzle up. Fo shizzle neque. Fo lorizzle. Bling
bling vitae pizzle ut libero commodo gizzle. Fusce izzle augue eu yo mamma dang.
Phasellizzle break it down fo nizzle erat. Suspendisse shizzlin dizzle owned,
sollicitudin sizzle, mah nizzle izzle, commodo nec, justo. Donizzle fizzle
porttitizzle ligula. Nunc feugizzle, tellus tellivizzle ornare tempor, sapizzle break
it down tincidunt gangster, eget dapibus daahng dawg enizzle izzle that's the shizzle.
Stuff quizzle leo, imperdizzle izzle, fo shizzle my nizzle izzle, semper izzle,
sapien. Ut boofron magna vizzle ghetto. I'm in the shizzle ante bling bling,
suscipizzle vitae, yo mamma stuff, rutrizzle pizzle, velizzle.
Mauris da bomb go to zzle. Sizzle mammasay mammasa mamma oo sa magna own yo' amet risus
congue. Boofron mofo auctizzle ma nizzle. Pot a elizzle ut nibh pretium tincidunt.
things erat. Own yo' in lacizzle sed maurizzle elementizzle tristique. I'm in the
shizzle yippiyo sizzle daahng dawg eros ultricizzle . In velit tortor, ultricizzle
ghetto, hendrerizzle fo shizzle mah nizzle fo rizzle, mah home g-dizzle, adipiscing
crunk, boom shackalack. Etizzle velit doggy, hizzle consequizzle, pharetra get down
get down, dictizzle sed, shut the shizzle up. Fo shizzle neque. Fo lorizzle. Bling "
}
26. IoT day 2015
JSON can represent complex containment relationships that are
difficult to represent in RDBMS
Schema-less – great for growing requirements during dev unlike
RDBMS where you must know the structure up front and its
painful to modify it
Native notation for JavaScript
Why JSON?
27. IoT day 2015
try to treat your entities as self-contained documents represented in JSON
When working with relational databases, we've been taught for years to normalize, normalize,
normalize.
There are contains relationships between entities.
There are one-to-few relationships between entities.
There is embedded data that changes infrequently.
There is embedded data won't grow without bound.
There is embedded data that is integral to data in a document.
Embedding
better read performance
28. IoT day 2015
Representing one-to-many relationships.
Representing many-to-many relationships.
Related data changes frequently.
Referenced data could be unbounded
Provides more flexibility than embedding
More round trips to read data
Referencing
Normalizing typically provides better write performance
29. •
No magic bullet
Think about how your data
is going to be written, read
and model accordingly
Hybrid models ~ denormalize + reference + aggregate
{
"id": "1",
"firstName": "Thomas",
"lastName": "Andersen",
"countOfBooks": 3,
"books": [1, 2, 3],
"images": [
{"thumbnail": "http://....png"}
{"profile": "http://....png"}
]
}
{
"id": 1,
"name": "DocumentDB 101",
"authors": [
{"id": 1, "name": "Thomas Andersen", "thumbnail": "http://....png"},
{"id": 2, "name": "William Wakefield", "thumbnail": "http://....png"}
]
}
30. IoT day 2015
Promote code first development (mapping objects to json)
Resilient to iterative schema changes
Richer query and indexing (compared to KV stores)
Low impedance as object / JSON store; no ORM required
It just works
It’s fast
Developer Appeal
32. IoT day 2015
Store schema-less JSON documents
Excels at search w/ SQL syntax
JavaScript for Stored Procs, Triggers and UDFs
Elastic capacity (not in specific Azure sense, up to now)
Multi-document transaction (Batch)
Tweak everything (read/write performance vs. consistency, index
performance, security)
Designed for massive scale
What is DocumentDb?
33. IoT day 2015
Applications that need managed elastic scale
Customer does not want to add additional IT resources for
support and maintenance
Avoiding CAPEX and OPEX
Built-for-the-cloud database technology
Access via RESTful HTTP API or client library
DocumentDB: DbaaS
34. IoT day 2015
Catalog data
Preferences and state
Event store
User generated content
Data exchange
Typical usage
42. IoT day 2015
a container of JSON documents and the associated JavaScript
application logic
JSON docs inside of a collection can vary dramatically
A unit of scale for transaction and query throughput (capacity
units allocated uniformly across all collections)
A unit of scale for capacity
A unit of replication
What is a collection?
43. IoT day 2015
Collections in DocumentDB are not just logical containers, but
also physical containers
They are the transaction boundary for stored procedures and
triggers
entry point to queries and CRUD operations
Each collection is assigned a reserved amount of throughput
which is not shared with other collections in the same account
Collections do not enforce schema
Collections
45. Design: Partitioning
Why Partition?
• Data Size
A single collection (currently*) holds 10GB
• Throughput
3 Performance tiers with a max of 2,500 RU/sec
46. IoT day 2015
In hash partitioning, partitions are assigned based on the value
of a hash function, allowing you to evenly distribute requests
and data across a number of partitions. This is commonly used
to partition data produced or consumed from a large number of
distinct clients, and is useful for storing user profiles, catalog
items, and IoT ("Internet of Things") telemetry data.
Hash Partitioning
47. IoT day 2015
In range partitioning, partitions are assigned based on whether
the partition key is within a certain range
This is commonly used for partitioning with time
stamp properties
Keep current data hot, Warm historical data, Scale-down older
data, Purge / Archive
Range partitioning
48. IoT day 2015
In lookup partitioning, partitions are assigned based on a
lookup map that assigns discrete partition values to specific
partitions a.k.a. a partition or shard map
This is commonly used for partitioning by region
Lookup partitioning
Tenant Partition Id
Customer 1
Big Customer 2
Another 3
51. IoT day 2015
Query / transaction throughput (and reliability – i.e., hardware failure) depend on
replication!
All writes to the primary are replicated across two secondary replicas
All reads are distributed across three copies
“Scalability of throughput” – allowing different clients to read from different replicas helps prevent
bottlenecks
BUT replication takes time!
Potential scenario: some clients are
reading while another is writing
Now, the data is out-of-date, inconsistent!
Why worry about consistency?
52. IoT day 2015
Trade-off: speed (performance & availability) or consistency
(data correctness)?
“Does every read need the MOST current data?”
“Or do I need every request to be handled and handled quickly?”
No “one size fits all” answer … so it’s up to you!
4 options …
For the entire Db…
…In a future release, we intend to support overriding the default consistency level on
a per collection basis.
Tweakable Consistency
53. IoT day 2015
client always sees completely consistent data
Slowest reads / writes
Mission critical: e.x. stock market, banking, airline reservation
Strong
54. IoT day 2015
Default – even trade-off between performance & availability vs.
data correctness
client reads its own writes, but other clients reading this same
data might see older values
Session
55. IoT day 2015
client might see old data, but it can specify a limit for how old
that data can be (ex. 2 seconds)
Updates happen in order received
similar to Session consistency, but speeds up reads while still
preserving the order of updates
Bounded Staleness
56. IoT day 2015
client might see old data for as long as it takes a write to
propagate to all replicas
High performance & availability, but a client might sometimes
read out-of-date information or see updates out of order
Eventual
57. IoT day 2015
At the database level (see preview portal)
On a per-read or per-query basis (optional parameter on
CreateDocumentQuery method)
Setting Consistency
58. IoT day 2015
Use Weaker Consistency Levels for better Read latencies
• IoT
• Data Analysis
http://azure.microsoft.com/blog/2015/01/27/performance-tips-
for-azure-documentdb-part-2/
Consistency Tips
60. IoT day 2015
Efficient, rich hierarchical and relational queries without any schema or
index definitions.
Consistent query results while handling a sustained volume of writes. For
high write throughput workloads with consistent queries, the index is
updated incrementally, efficiently, and online while handling a sustained
volume of writes.
Storage efficiency. For cost effectiveness, the on-disk storage overhead of
the index is bounded and predictable.
Indexing
61. var collection = new DocumentCollection
{
Id = "lazyCollection"
};
collection.IndexingPolicy.IndexingMode = IndexingMode.Lazy;
client.CreateDocumentCollectionAsync(databaseLink, collection);
Indexing modes
Consistent
Default mode
Index updated synchronously on writes
Lazy
Useful for bulk ingestion scenarios
Indexing policies
Automatic
Default
Manual
Can choose to index documents via
RequestOptions
Can read non-indexed documents
via selflink
Indexing – Modes and policies
Set indexing mode
Set indexing policy
var collection = new DocumentCollection
{
Id = "manualCollection"
};
collection.IndexingPolicy.Automatic = false;
client.CreateDocumentCollectionAsync(databaseLink, collection);
62. Setting paths, types, and precision
var collection = new DocumentCollection
{
Id = "Orders"
};
collection.IndexingPolicy.ExcludedPaths.Add("/"metaData"/*");
collection.IndexingPolicy.IncludedPaths.Add(new IndexingPath
{
IndexType = IndexType.Hash,
Path = "/",
});
collection.IndexingPolicy.IncludedPaths.Add(new IndexingPath
{
IndexType = IndexType.Range,
Path = @"/""shippedTimestamp""/?",
NumericPrecision = 7
});
client.CreateDocumentCollectionAsync(databaseLink, collection);
Index paths
Include and/or Exclude paths
Index types
Hash
Supported for strings and numbers
Optimized for equality matches
Range
Supported for numbers
Optimized for comparison queries
Index precision
String precision
Default is 3
Numeric precision
Default is 3
Increase for larger number fields
Indexing – Paths and types
63. IoT day 2015
Use lazy indexing for faster peak time ingestion rates
Exclude unused paths from indexing for faster writes
Specify range index path type for all paths used in range queries
Vary index precision for write vs query performance vs storage
tradeoffs
http://azure.microsoft.com/blog/2015/01/27/performance-tips-
for-azure-documentdb-part-2/
Indexing tips
65. IoT day 2015
Optimize for queries with small result sets for scalability
Limit use of scans (no range index, NOT, UDFs in WHERE)
Use page size (MaxItemCount) and continuation tokens
For large result sets, use a larger page size (1000)
Querying
66. Query over heterogeneous documents without defining
schema or managing indexes
Query arbitrary paths, properties and values without
specifying secondary indexes or indexing hints
Execute queries with consistent results
Supported SQL features; predicates, iterations (arrays),
sub-queries, logical operators, UDFs, intra-document
JOINs, JSON transforms
In general, more predicates result in a larger request
charge.
Additional predicates can help if they result in narrowing
the overall result set.
from book in client.CreateDocumentQuery<Book>(collectionSelfLink)
where book.Title == "War and Peace"
select book;
from book in client.CreateDocumentQuery<Book>(collectionSelfLink)
where book.Author.Name == "Leo Tolstoy"
select book.Author;
-- Nested lookup against index
SELECT B.Author
FROM Books B
WHERE B.Author.Name = "Leo Tolstoy"
-- Transformation, Filters, Array access
SELECT { Name: B.Title, Author: B.Author.Name }
FROM Books B
WHERE B.Price > 10 AND B.Language[0] = "English"
-- Joins, User Defined Functions (UDF)
SELECT udf.CalculateRegionalTax(B.Price, "USA", "WA")
FROM Books B
JOIN L IN B.Languages
WHERE L.Language = "Russian"
LINQ Query
SQL Query Grammar
Query
68. function region(doc)
{
switch (doc.Location.Region)
{
case 0:
return "North";
case 1:
return "Middle";
case 2:
return "South";
}
}
The complexity of a query impacts the
request units consumed for an operation:
Use of user-defined functions (UDFs)
SELECT or WHERE clauses
To take advantage of indexing, try and have at least
one filter against an indexed property when
leveraging a UDF in the WHERE clause
.
Query with user-defined function
69. function count(filterQuery, continuationToken) {
var collection = getContext().getCollection();
var maxResult = 25; // MAX number of docs to process in one
batch, when reached, return to client/request continuation.
// intentionally set low to demonstrate the concept. This can
be much higher. Try experimenting.
// We've had it in to the high thousands before seeing the
stored proceudre timing out.
// The number of documents counted.
var result = 0;
tryQuery(continuationToken);
}
Execute “explicit” Javascript
code on collection
Executing Stored Procedures
70. function normalize() {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();
var doc = getContext().getRequest().getBody();
var newDoc = {
"Sensor": {
"Id": doc.sensorId,
"Class": 0
},
"Degree": {
"Value": doc.degreeValue,
"Type": 0
},
"Location": {
"Name": doc.locationName,
"Region": doc.locationRegion,
"Longitude": doc.locationLong,
"Latitude": doc.locationLat
},
"id": doc.id
};
// Update the request -- this is what is going to be inserted.
getContext().getRequest().setBody(newDoc);
}
Execute “implicit” Javascript
code on CRUD operations
(Insert, Update, Delete) on
collections
Triggers!
72. IoT day 2015
Data is saved on SSD
All writes to the primary are replicated across two secondary
replicas
(Replicas are spread on different hardware in same region to protect
against failures)
All reads are distributed across the
three copies (when and how depend
on consistency level for db account
and query)
DocumentDb Performance
73. IoT day 2015
Measure and Tune for lower request units/second usage
DocumentDB offers a rich set of database operations including relational and hierarchical queries with UDFs, stored procedures and triggers – all operating on the
documents within a database collection. The cost associated with each of these operations will vary based on the CPU, IO and memory required to complete the operation.
Instead of thinking about and managing hardware resources, you can think of a request unit (RU) as a single measure for the resources required to perform various database
operations and service an application request.
Handle Server throttles/request rate too large
When a client attempts to exceed the reserved throughput for an account, there will be no performance degradation at the server and no use of throughput capacity beyond the reserved
level. The server will preemptively end the request with RequestRateTooLarge (HTTP status code 429) and return the x-ms-retry-after-ms header indicating the amount of time, in
milliseconds, that the user must wait before reattempting the request.
Delete empty collections to utilize all provisioned throughput
Every document collection created in a DocumentDB account is allocated reserved throughput capacity based on the number of Capacity Units (CUs) provisioned, and the number of
collections created. A single CU makes available 2,000 request units (RUs) and supports up to 3 collections
Design for smaller documents for higher throughput
The Request Charge (i.e. request processing cost) of a given operation is directly correlated to the size of the document
http://azure.microsoft.com/blog/2015/01/27/performance-tips-for-azure-documentdb-part-2/
Performance Tips
75. IoT day 2015
User generated content
Many specific data (varbinary(MAX) in SQL)
Catalog data
Log data
User preferences data
Device sensor data
IoT use cases commonly share some patterns in how they ingest, process and store data. First, these
systems allow for data intake that can ingest bursts of data from device sensors of various locales. Next,
these systems process and analyze streaming data to derive real time insights. And last but not least,
most if not all data will eventually land in a data store for adhoc querying and offline analytics.
Usage: what is DocumentDb for?
76. IoT day 2015
Maturity: Balancing embedding (ok) and relating (limits)
Searching and Denormalizing
Opportunity
Storing transient Data
Better Opportunities
Storing Files
Append Only
(Table) Storage
Limits from DocumentDb
78. IoT day 2015
Targeted at streaming workloads (E.g. files read from beginning
to end like media files)
Each blob consists of a sequence of blocks
Each block is identified by a Block ID
Each block can be a maximum of 64 MB in size
Size limit 200GB per blob
Azure Storage Blob: Block Blob
Block Blob:
79. IoT day 2015
Targeted at random read/write workloads (E.g. backing storage
for the VHDs used in Azure VMs)
Each blob consists of an array of pages
Each page is identified by its offset from the start of the blob
Size limit 1TB per blob
Azure Storage Blob: Page Blob
80. IoT day 2015
Not an RDBMS Table!
The mental picture is ‘Entities’
Entity can have up to 255 properties
Up to 1MB per entity
Partitioning
PartitionKey & RowKey are mandatory properties
Composite key which uniquely identifies an entity
They are the only indexed properties
Defines the sort order
Purpose of the PartitionKey:
Entity Locality
Entities in the same partition will be stored together
Efficient querying and cache locality
Entity Group Transactions
Target throughput – 500 tps/partition, several thousand tps/account
Microsoft Azure monitors the usage patterns of partitions
Automatically load balance partitions
Each partition can be served by a different storage node
Scale to meet the traffic needs of your table
Supports full manipulation (CRUD)
Table Scalability
Azure Table Storage Details
81. IoT day 2015
Embed a sophisticated search experience into web and mobile
applications without having to worry about the complexities of
full-text search and without having to deploy, maintain or
manage any infrastructure.
Perfect for enterprise cloud developers, cloud software vendors,
cloud architects who need a fully-managed search solution.
Search is a natural backend for Cortana
Take a bunch of words apply linguistics return relevant results
Azure Search
82. IoT day 2015
“Search service”
Scope for capacity
Bound to a region
Has keys, indexes, indexers, data sources
Provisioning
Azure Portal
Azure resource management API
Elastic scale
Capacity can be changed dynamically
Replicas ~ more QPS, HA
Partitions ~ more documents, write throughput
Azure Search Service
83. IoT day 2015
Simple HTTP/JSON API for creating indexes, pushing documents, searching
Keyword search with user-friendly operators (+, -, *, “”, etc.)
Hit highlighting
Faceting (histograms over ranges, typically used in catalog browsing)
Based on ElasticSearch
Search Functionality
84. IoT day 2015
Linguistics are key in search
Support for 50 languages
Word breaking, stop words, inflections
Lucene analyzers
Well-known analyzer stack
Stemming
Microsoft analyzers
Same NLP stack used by parts of Office, Bing
Lematization in many languages
Linguistics
85. IoT day 2015
Suggestions (auto-complete)
Rich structured queries (filter, select, sort) that combines with search
Scoring profiles to model search result relevance
Geo-spatial support integrated in filtering, sorting and ranking (such as finding all
restaurants within 5 KM of your current location)
Search Functionality
86. IoT day 2015
Redis is an open source, BSD licensed, networked, single-
threaded, in-memory key-value cache and store.
Key-value cache and store (value can be a couple of things)
In-memory (no persistence, but you can)
Single-threaded (atomic operations & transactions)
Networked (it’s a server and it does master/slave)
Some other stuff (scripting, pub/sub, Sentinel, snapshot
Caching: Redis
88. IoT day 2015
Pro:
partitioning, replica and scaling at it’s core
self contained documents
programmability in Javascript
SQL like “intradocument” queries
Cons:
No SQL generic queries
Can work alone just in few scenarios
So DocumentDb…
89. IoT day 2015
Great storage opportunities in Azure
• Log
• Search
• Transient
• Files/Attachments
• SQL!
• And all new Data Analysis/Machine Learning opportunities
Other Not Only SQL alternatives
90. IoT day 2015
http://bit.do/documentdb-pricing
Capacity Units (CU)
Capacity
Throughput (in terms of rate of transactions / second)
• Request Units (RU) = 2000 request per second
• “Request” depends on the size of the document – ex. Uploading 1000 large JSON documents
might count as more than one request
Pricing
91. Standard pricing tier with hourly
billing
1 hr from just $0.034!
Performance levels can be
adjusted
Each collection = 10GB of SSD
Collection* perf is set by S1, S2,
S3
Limit of 100 collections (1 TB)
Soft limit, can be lifted as
needed per account
What does DocumentDB cost?
* collection != table of homogenous entities
collection ~ a data partition
92. IoT day 2015
NoSQL in Azure per l’IoT
(e il Business)
Marco Parenzan
Microsoft Azure MVP
@marco_parenzan
marco [dot] parenzan [at] 1nn0va [dot] it
Editor's Notes
Slide Objectives:
Show Microsoft’ continuous Private to Public Cloud Offering, but this presentation will focus on Microsoft’s relational database PaaS offering.
Transition:
Microsoft provides a continuous solution from private cloud to the public cloud. No matter where you are on your technology roadmap we have a solution to fit your needs.
We are a trusted advisor and platform in the traditional enterprise and ISV space with new IaaS offerings that making it easier to bring this same level of trust and ease of use to the public cloud. However, Microsoft Azure SQL Database extends SQL Server capabilities to the cloud by offering SQL Server as a relational database service.
Speaking Points:
SQL Database provides SQL Server as a relational service.
Slide Objectives:
Understand the overall concepts and benefits of SQL Database
Transition:
Let’s clear up any confusion and look at the basics of what SQL Database really is and some of its benefits.
Speaking Points:
The same great SQL Server database technology that you know, love, and use on-premises provided as a service
Enterprise-ready
Automatic support for High-Availability
DR = Disaster Recovery
Designed to scale on-demand to provide the same great elasticity
Notes:
High-availability – 3 copies of the database free for the cost of the one database. Always in sync. The cost to do this on-premises isn’t cheap. This is FREE in SQL Database.
Notes
A data lake is a massive, easily accessible, centralized repository of large volumes of structured and unstructured data
Slide Objective
Speaker Notes
Notes
Slide Objective
Speaker Notes
Notes
Slide Objective
Understand block blob
Speaker Notes
Block blobs are comprised of blocks, each of which is identified by a block ID.
You create or modify a block blob by uploading a set of blocks and committing them by their block IDs.
Each block can be a maximum of 64 MB in size. The maximum size for a block blob in version 2009-09-19 is 200 GB, or up to 50,000 blocks.
Notes
http://msdn.microsoft.com/en-us/library/dd135734.aspx
Slide Objective
Understand page blob
Speaker Notes
Page blobs are a collection of pages.
A page is a range of data that is identified by its offset from the start of the blob.
To create a page blob, you initialize the page blob by calling Put Blob and specifying its maximum size.
To add content to or update a page blob, you call the Put Page operation to modify a page or range of pages by specifying an offset and range. All pages must align 512-byte page boundaries.
Unlike writes to block blobs, writes to page blobs happen in-place and are immediately committed to the blob.
The maximum size for a page blob is 1 TB.
A page written to a page blob may be up to 1 TB in size but will typically be much smaller
Notes
http://msdn.microsoft.com/en-us/library/dd135734.aspx
Slide Objectives
Understand Tables
Speaker Notes
Within a storage account, a developer may create named tables.
Tables store data as entities.
An entity is a collection of named properties and their values, similar to a row.
Tables are partitioned to support load balancing across storage nodes.
Each table has as its first property a partition key that specifies the partition an entity belongs to.
The second property is a row key that identifies an entity within a given partition.
The combination of the partition key and the row key forms a primary key that identifies each entity uniquely within the table.
The Table service does not enforce any schema.
A developer may choose to implement and enforce a schema on the client side
Notes
http://msdn.microsoft.com/en-us/library/dd573356.aspx
Azure Search is a fully managed search solution that allows developers to enable search experiences in applications.
What Azure Search does is that it sits right next to your data store (relational or NOSQL) which can be on-prem or on the Cloud (which may be Azure or any other public cloud) and provides the necessary index that can be used to search the operational data.
This service is used only by the application developer and saves him the overhead of developing a search function specifically for his app.
Faceted navigation is a filtering mechanism that provides self-directed drilldown navigation in search applications.