Virtual Global Azure 2020 - Azure MonitorPedro Sousa
This presentation was given at Global Azure 2020 Lisbon, about Azure Monitor.
This session focused on:
- steps of the Monitoring Lifecycle;
- Conceptual Architecture of Azure Monitoring;
- Data Collection & Onboarding;
- Metrics & Logs;
- Demos.
Recordings for the event sessions will be available soon.
Hello All,
It is time for the second Tokyo Azure Meetup!
As a natural continuation of our first topic, we will proceed with Big Data.
Until recently you needed to learn new language or master new concepts in order get started with Big Data.
Moreover, you needed to spend a lot of time setting up infrastructure that will meet the business demands for Big Data processing.
Not any more!
If you know C# and T-SQL you are ready to become Big Data master!
Public cloud and especially Microsoft Azure are very well suited for working with Big Data.
Join us for our next event and and I can assure you that after the session you will be ready to start working with Big Data.
And maybe you are asking why this is important.
I believe that we don't have choice but build smart applications and get as much possible insights from the data we collect from various sources in order to take the best business decisions and please our customers.
Today we have so much data available publicly or coming from our customers and it is very challenging to process it and turn it into valuable business asset.
Not any more!
Join for our next meetup and you will see how Microsoft create amazing opportunity for each .Net developer to become Big Data expert and every company to start using Big Data to accelerate its growth.
I have been working closely with the product team developing U-SQL language that empower Azure Data Lake Analytics, which is one of the processing engines for Azure Data Lake and I will be very happy to share my experience with you!
See you very soon!
Kanio
Monitoring real-life Azure applications: When to use what and whyKarl Ots
Slides from my presentation at Intelligent Cloud Conf on 29.5.2018 in Copenhagen
Modern applications leverage a variety of services, and often span across on premises, IaaS, PaaS and SaaS. Monitoring these environments is different from traditional systems. We have more and more data available from the platform with the likes of ARM Activity Logs, Azure Monitor, Log Analytics and Application Insights.
With a massive amount of signal and noise being generated in all these systems, how do we get our arms around what is happening? Is my application impacted in an ongoing Azure outage? Are my integrations intact? Which services from Azure should I use to monitor my application end-to-end? Come and hear how to answer these questions. After the session, you’ll have deeper understanding of end-to-end monitoring techniques in Azure solutions and know which services to choose for which scenario.
.
Virtual Global Azure 2020 - Azure MonitorPedro Sousa
This presentation was given at Global Azure 2020 Lisbon, about Azure Monitor.
This session focused on:
- steps of the Monitoring Lifecycle;
- Conceptual Architecture of Azure Monitoring;
- Data Collection & Onboarding;
- Metrics & Logs;
- Demos.
Recordings for the event sessions will be available soon.
Hello All,
It is time for the second Tokyo Azure Meetup!
As a natural continuation of our first topic, we will proceed with Big Data.
Until recently you needed to learn new language or master new concepts in order get started with Big Data.
Moreover, you needed to spend a lot of time setting up infrastructure that will meet the business demands for Big Data processing.
Not any more!
If you know C# and T-SQL you are ready to become Big Data master!
Public cloud and especially Microsoft Azure are very well suited for working with Big Data.
Join us for our next event and and I can assure you that after the session you will be ready to start working with Big Data.
And maybe you are asking why this is important.
I believe that we don't have choice but build smart applications and get as much possible insights from the data we collect from various sources in order to take the best business decisions and please our customers.
Today we have so much data available publicly or coming from our customers and it is very challenging to process it and turn it into valuable business asset.
Not any more!
Join for our next meetup and you will see how Microsoft create amazing opportunity for each .Net developer to become Big Data expert and every company to start using Big Data to accelerate its growth.
I have been working closely with the product team developing U-SQL language that empower Azure Data Lake Analytics, which is one of the processing engines for Azure Data Lake and I will be very happy to share my experience with you!
See you very soon!
Kanio
Monitoring real-life Azure applications: When to use what and whyKarl Ots
Slides from my presentation at Intelligent Cloud Conf on 29.5.2018 in Copenhagen
Modern applications leverage a variety of services, and often span across on premises, IaaS, PaaS and SaaS. Monitoring these environments is different from traditional systems. We have more and more data available from the platform with the likes of ARM Activity Logs, Azure Monitor, Log Analytics and Application Insights.
With a massive amount of signal and noise being generated in all these systems, how do we get our arms around what is happening? Is my application impacted in an ongoing Azure outage? Are my integrations intact? Which services from Azure should I use to monitor my application end-to-end? Come and hear how to answer these questions. After the session, you’ll have deeper understanding of end-to-end monitoring techniques in Azure solutions and know which services to choose for which scenario.
.
Building and deploying an analytic service on Cloud is a challenge. A bigger challenge is to maintain the service. In a world where users are gravitating towards a model where cluster instances are to provisioned on the fly, in order for these to be used for analytics or other purposes, and then to have these cluster instances shut down when the jobs get done, the relevance of containers and container orchestration is more important than ever. In short Customers are looking for Serverless Spark Clusters. The Intent of this presentation is to share what is Serverless Spark and what are the benefits of running Spark in serverless manner.
Many organizations are embracing the latest practices for DevOps agility and cloud innovations to manage their heterogeneous environments (Hybrid DC). Yet they are also concerned about their ability to make appropriate and responsible decisions about how to monitor those workloads. How to monitor application and infrastructure in a centralized location?
Let's meet and talk about Microsoft Azure PaaS offerings. The PaaS layer provides many scalable and globally deployed services completely manged by Microsoft that allow developer to focus on specific business requirements and to leave the infrastructure bits to the cloud provider. We will underline the differences between Virtual Machines, Cloud Services and Azure Web Apps on the compute layer. Later we will compare SQL Server and Azure SQL.
Then we will focus on Data Storage and Data Analytics services that gives incredible power to developers and data professionals.
Most of the examples we cover are platform agnostic so people from any programming background are welcome to join and share their unique experience. Microsoft Azure is getting more open and open source friendly with every new day!
Come and join us to learn more about Microsoft Azure and enjoy your journey with the public cloud!
Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...Tom Kerkhove
Azure Data Factory is a hybrid data integration service in Azure that allows you to create, manage & operate data pipelines in Azure. It is a serverless orchestrator that allows you to create data pipelines to either move, transform, load data; a fully managed Extract, Transform, Load (ETL) & Extract, Load, Transform (ELT) service if you will.
In this talk I'll cover the basics of Azure Data Factory and show you how you can create, manage & operate data pipelines.
Mining public datasets using opensource tools: Zeppelin, Spark and Jujuseoul_engineer
There are plenty of public datasets out there available and the number is growing. Few recent and most useful of BigData ecosystem tools are showcased: Apache Zeppelin (incubating), Apache Spark and Juju.
Must Know Azure Kubernetes Best Practices And Features For Better Resiliency ...CodeOps Technologies LLP
Running day-1 Ops on your Kubernetes is somewhat easy, but it is quite daunting to manage day two challenges. Learn about AKS best practices for your cloud-native applications so that you can avoid blow up your workloads.
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...HostedbyConfluent
The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Event driven workloads on Kubernetes with KEDANilesh Gule
Slide deck of the presentation done at the Pune User Group on 27th February 2021. Demonstrate how Kubernetes based event driven autoscaling (KEDA) can be used with RabbitMQ as the event source.
Cloud solutions could not be best solution if it is not chosen. One factor businesses deviates from cloud solutions is unawareness of getting best out of cloud solutions with increasing efficiency.
This presentation addresses gaps between discussion had at the global azure bootcamp New Jersey.
Azure Automation delivers cloud-based automation, operating system updates, and configuration service that supports consistent management across your Azure and non-Azure environments. It includes process automation, configuration management, update management, shared capabilities, and heterogeneous features.
Building and deploying an analytic service on Cloud is a challenge. A bigger challenge is to maintain the service. In a world where users are gravitating towards a model where cluster instances are to provisioned on the fly, in order for these to be used for analytics or other purposes, and then to have these cluster instances shut down when the jobs get done, the relevance of containers and container orchestration is more important than ever. In short Customers are looking for Serverless Spark Clusters. The Intent of this presentation is to share what is Serverless Spark and what are the benefits of running Spark in serverless manner.
Many organizations are embracing the latest practices for DevOps agility and cloud innovations to manage their heterogeneous environments (Hybrid DC). Yet they are also concerned about their ability to make appropriate and responsible decisions about how to monitor those workloads. How to monitor application and infrastructure in a centralized location?
Let's meet and talk about Microsoft Azure PaaS offerings. The PaaS layer provides many scalable and globally deployed services completely manged by Microsoft that allow developer to focus on specific business requirements and to leave the infrastructure bits to the cloud provider. We will underline the differences between Virtual Machines, Cloud Services and Azure Web Apps on the compute layer. Later we will compare SQL Server and Azure SQL.
Then we will focus on Data Storage and Data Analytics services that gives incredible power to developers and data professionals.
Most of the examples we cover are platform agnostic so people from any programming background are welcome to join and share their unique experience. Microsoft Azure is getting more open and open source friendly with every new day!
Come and join us to learn more about Microsoft Azure and enjoy your journey with the public cloud!
Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...Tom Kerkhove
Azure Data Factory is a hybrid data integration service in Azure that allows you to create, manage & operate data pipelines in Azure. It is a serverless orchestrator that allows you to create data pipelines to either move, transform, load data; a fully managed Extract, Transform, Load (ETL) & Extract, Load, Transform (ELT) service if you will.
In this talk I'll cover the basics of Azure Data Factory and show you how you can create, manage & operate data pipelines.
Mining public datasets using opensource tools: Zeppelin, Spark and Jujuseoul_engineer
There are plenty of public datasets out there available and the number is growing. Few recent and most useful of BigData ecosystem tools are showcased: Apache Zeppelin (incubating), Apache Spark and Juju.
Must Know Azure Kubernetes Best Practices And Features For Better Resiliency ...CodeOps Technologies LLP
Running day-1 Ops on your Kubernetes is somewhat easy, but it is quite daunting to manage day two challenges. Learn about AKS best practices for your cloud-native applications so that you can avoid blow up your workloads.
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...HostedbyConfluent
The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Event driven workloads on Kubernetes with KEDANilesh Gule
Slide deck of the presentation done at the Pune User Group on 27th February 2021. Demonstrate how Kubernetes based event driven autoscaling (KEDA) can be used with RabbitMQ as the event source.
Cloud solutions could not be best solution if it is not chosen. One factor businesses deviates from cloud solutions is unawareness of getting best out of cloud solutions with increasing efficiency.
This presentation addresses gaps between discussion had at the global azure bootcamp New Jersey.
Azure Automation delivers cloud-based automation, operating system updates, and configuration service that supports consistent management across your Azure and non-Azure environments. It includes process automation, configuration management, update management, shared capabilities, and heterogeneous features.
Slides from my talk at Big Data Conference 2018 in Vilnius
Doing data science today is far more difficult than it will be in the next 5-10 years. Sharing, collaborating on data science workflows in painful, pushing models into production is challenging.
Let’s explore what Azure provides to ease Data Scientists’ pains. What tools and services can we choose based on a problem definition, skillset or infrastructure requirements?
In this talk, you will learn about Azure Machine Learning Studio, Azure Databricks, Data Science Virtual Machines and Cognitive Services, with all the perks and limitations.
Dans cette session nous vous présenterons les différentes manières d'utiliser SQL Server dans une infrastructure Cloud (Microsoft Azure). Seront présentés des scénarios hybrides, de migration, de backup, et d'hébergement de bases de données SQL Server en mode IaaS ou PaaS.
Migrating on premises workload to azure sql databasePARIKSHIT SAVJANI
Azure SQL Database is a fully managed cloud database service with built-in intelligence, elastic scale, performance, reliability, and data protection that enables enterprises and ISVs to reduce their total cost of ownership and operational cost and overheads. In this session, I will share real-world experience of successfully migrated existing SaaS application and on-premises workload for some our tier 1 customers and ISV partners to Azure SQL Database service. The session walks through planning, assessment, migration tools and best practices from the proven experiences and practices of migrating real world applications to Azure SQL Database service.
How to Architect a Serverless Cloud Data Lake for Enhanced Data AnalyticsInformatica
This presentation is geared toward enterprise architects and senior IT leaders looking to drive more value from their data by learning about cloud data lake management.
As businesses focus on leveraging big data to drive digital transformation, technology leaders are struggling to keep pace with the high volume of data coming in at high speed and rapidly evolving technologies. What's needed is an approach that helps you turn petabytes into profit.
Cloud data lakes and cloud data warehouses have emerged as a popular architectural pattern to support next-generation analytics. Informatica's comprehensive AI-driven cloud data lake management solution natively ingests, streams, integrates, cleanses, governs, protects and processes big data workloads in multi-cloud environments.
Please leave any questions or comments below.
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
2. #azuresatpn
Nice to meet you
Riccardo Perico | rperico@solidq.com | @R1k91
SolidQ
Data Platform & BI Specialist
10 years working, training and speaking in Microsoft «Data Realm»
MCP: MTA, MCSA
https://www.linkedin.com/in/riccardo-perico-8b942384/
5. #azuresatpn
What ADF really is?
Cloud based
Data
integration
service
Orchestrates &
Automates
Data
movement and
transformation
Allows
Monitoring
and Debugging
Programmable
7. #azuresatpn
Sample Workflow
On-premises
data mart
Customer
web logs
Product table
Azure DB
Product
recommendations
Visualize
Azure Blob storage
Customer web
Logs
Product table
Data set
(Collection of files,
DB table, etc.)
Pipeline: A sequence of
activities (logical group)
Activity: A processing step
(Hadoop job, custom code, ML model, etc.)
…
Data sources Ingest Transform and analyze Publish
Combined
input table
Mapping
Transform,
combine, etc. Analyze Move
8. #azuresatpn
Devices Device Connectivity Storage Analytics Presentation & Action
Event Hubs SQL Database
Machine
Learning
App Service
IoT Hubs
Table/Blob
Storage
Stream Analytics Power BI
Service Bus Cosmos DB HDInsight
Notification
Hubs
External Data
Sources
External Data
Sources
Data Factory Mobile Services
BizTalk Services
Data Lake
Analytics
11. #azuresatpn
Activities & Pipelines
An Activity is a single task in workflow:
• Copy from input to output
• Transform
• C#
• Stored Procedure
• Hadoop (Map/Reduce, Hive, Pig)
• ML, Data Lake Analytics
• Databricks
• Control
• IF, ForEach, Until, Wait, Execute Pipeline
• Web
Pipeline groups activities
SQL
Serve
r
SQL
DB
SQL
Server
VMs
12. #azuresatpn
Integration Runtime
• Bridge between Activity and Linked Service
• Compute environment where activity runs or it’s dispatched from
3 types of IR:
• IR Azure
• IR Self-hosted
• IR Azure-SSIS
14. #azuresatpn
ADF Location vs IR Location
• ADF location metadata store and triggering pipeline start
• IR location backend compute engine location (data movement,
activity dispatch and SSIS execution)
ADF Location and IR location could be different
IR can use “Auto Resolve”
15. #azuresatpn
Mapping Data Flows
• Based on Spark
• Use Databricks behind the scene
• A lot of transformations already available
• Few sources available for now
• This week GA announced!
17. #azuresatpn
Developer Tools
• Azure Portal: Create, Edit. Visual and Textual
• Visual Studio: Integrated in VS project
• Powershell: cmdlets https://docs.microsoft.com/en-
us/powershell/module/azurerm.datafactories/?view=azurermps-
6.13.0
• Azure RM Template
18. #azuresatpn
Pricing
Multiple factors affect pricing
• Number of Activities run
• Volume of data moved
• SQL Server Integration Services Compute Hours
• Whether you re-running an activity
https://azure.microsoft.com/en-us/pricing/details/data-factory/v2/
Enterprises have data of various types that are located in disparate sources on-premises, in the cloud, structured, unstructured, and semi-structured, all arriving at different intervals and speeds.
The first step in building an information production system is to connect to all the required sources.
Without Data Factory, enterprises must build custom data movement components, they often lack the enterprise-grade monitoring, alerting, and the controls that a fully managed service can offer.
After data is present in a centralized data store in the cloud, process or transform the collected data by using compute services such as HDInsight Hadoop, Spark, Data Lake Analytics, and Machine Learning.
After the raw data has been refined into a business-ready consumable form, load the data into Azure Data Warehouse, Azure SQL Database, Azure CosmosDB, or whichever analytics engine your business users can point to from their business intelligence tools.
The Data Set is a view of input/output data
Data sets identify the data from different data stores.
Azure: public accessible endpoints, serverless, fully managed, pay for use only, scaled up automagically according to copy activity properties
Self-hosted: everything works in a private network behind corporate firewall, only HTTP outbound. A Windows server is needed and IR must be installed. Supports active-active load balancing.
Azure SSIS: Set of VMs natively executes SSIS. Supports BYO SSISDB on Azure SQL DB or Managed Instance. To On-prem use Azure Virtual Network with VPN site-to-site.
Mapping Data Flows are visually designed data transformations in Azure Data Factory
Copy activity from S3 to SQL
Rest to SQL
Datasets transform with SP vs MDF
Trigger & Monitor