Bi and AI updates in the Microsoft Data Platform stack

•

0 likes•647 views

This document provides an overview and agenda for a presentation on new features in the Microsoft data platform universe including SQL Server, Azure, and Power BI, with a focus on business intelligence (BI) and artificial intelligence (AI). The presentation covers new features in SQL Server 2019 for BI like calculation groups and dynamic formatting in Analysis Services tabular mode. It also highlights important updates in Azure including new serverless and hyperscale options for Azure SQL Database, updates to data ingestion and storage with Azure Data Factory v2 and Azure Data Lake Gen 2, and changes to data modeling, preparation, and serving capabilities in Azure SQL Data Warehouse. The document concludes by discussing options for incorporating AI and machine learning into BI solutions using either

Technology

Powered by
BI and AI in the
Microsoft data
platform
universe
(with a dash of cloud)
Ivan Donev

Agenda
• What’s new in SQL 2019 for BI
• Important updates on Azure for Data platform
• AI and ML for the masses

What is new in 2019 for BI
• SQL Server AS
• MD nothing
• Tabular – Calc groups, M:M relationships, dynamic formatting
• SQL Server IS
• Nothing
• SQL Server RS
• Nothing
• SQL Server MDM and DQS
• Almost nothing

DEMO with AS 2019
Calculation groups
Dynamic formatting
M:M in tabular

Updates in Azure SQL DB
• Azure SQL DB – Hyperscale• Azure SQL DB – Serverless

Azure SQL DB – Serverless
• Single DB Serverless compute tier
• Billed on compute used per SECOND
• Used only in the vCore model
• Parametrize the min/max vCores
• Scenarios
• Intermittent usage
• Frequently rescaled DBs
• New deployments prior historical usage data

Azure SQL DB – Hyperscale
• Up to 100TB
• Fast backups (filesystem snapshots)
• Up to a minute restores
• Faster throughput
• Fast scale out and scale up
• Distributed architecture

Modern DWH – important updates in Ingest
• Azure Data Factory v2
• Integration runtimes to run SSIS
as-is
• Storing SSIS catalogue in SQL DB
• Mapping workflow
• Wrangling workflow

Modern DWH – important updates in Store
• Azure Data Lake Gen 2
• Hierarchical file system
• Security
• Performance
• Much easier to integrate with
other services

Modern DWH – important updates in Prep and
Train
• Azure Databricks Delta
• Spark engine with RDBMS features

Databricks Delta
• ACID transactions
• Versioned PARQUET files
• Streaming writes to a table (i.e.
Kafka)
• Batch upserts
• High performance reads
• Schema enforcement

Modern DWH – important updates in Model and
Serve
• Changes in Azure DWH
• Concurrency increased to 128
• Adaptive caching (NVMe !!!)
• Unlimited Columnstore storage
capacity
• Workload classification and
importance improvements
• Changes in PowerBI

PowerBI Updates worth noting
• PowerBI Dataflows
• Self-service data transformation
• Shared and certified datasets
(preview)
• Paginated Reports (SSRS)
• Premium
• XMLA Endpoints
• Premium
• Auto ML
• Premium
• CDS integration

The AI in BI
• Options to use in DWH and BI

The ML in BI
Not scalableSelf-service AI
•Prototyping
•Do not need additional configuration or tuning
•Options are
•Microsoft Cognitive services with PowerBI (demo)
•AutoML in Dataflows in PowerBI Premium
•R/Python visuals
Scalable, configurable, needs specialized staffEnterprise AI
•Mandatory to run ML and store it in the Store/Serve model
•Options are
•Databricks
•Azure ML
•R/Python in Dataflow as data sources

How to choose?
• The aim?
• Prototype/Test/Verify
• Production/O16N
• The knowledge
• R/Python/Scala/Java/…
• The task
• Image processing/Text analytics/Prediction/Classification
• The post-production support
• Can you support the solution afterwards?

Demo
• PowerBI Dataflows
• Using External Cognitive Services APIs

THANK YOU
All my demos will be described and uploaded on our blog:
http://sqlmasteracademy.com/techblog/

This document summarizes new features in SQL Server 2019 including: - Support for big data analytics using SQL Server, Spark, and data lakes for high volume data storage and access. - Enhanced data virtualization capabilities to combine data from multiple sources. - An integrated AI platform to ingest, prepare, train, store, and operationalize machine learning models. - Improved security, performance, and high availability features like Always Encrypted with secure enclaves and accelerated database recovery. - Continued improvements based on community feedback.

Discovery Day 2019 Sofia - Big data clusters

Ivan Donev

This document provides an overview of the architecture and components of SQL Server 2019 Big Data Clusters. It describes the key Kubernetes concepts used in Big Data Clusters like pods, services, and nodes. It then explains the different planes (control, compute, data) and nodes that make up a Big Data Cluster and their roles. Components in each plane like the SQL master instance, compute pools, storage pools, and data pools are also outlined.

Odessa .net-user-group-sql-server-2019-hidden-gems by Denis Reznik

Alex Tumanoff

This document discusses hidden gems in SQL Server 2019, including lightweight query profiling, DBCC PAGE, dm_os_host_info, dm_os_schedulers, and QUERY OPTIMIZER COMPATIBILITY LEVEL. It provides demos of these lesser known features and notes that the string truncation error message is one of the most well-known hidden features. The author concludes that more hidden features will continue to be added to SQL Server.

The Roadmap for SQL Server 2019

Amit Banerjee

Azure Cosmos DB: Features, Practical Use and Optimization "

GlobalLogic Ukraine

This presentation is dedicated to Azure Cosmos DB, it's history, characteristics, tasks and solutions. The presentation deals with performance optimization, practical experience of usage and an overview of the news about Cosmos DB from Microsoft Build 2017 conference (https://build.microsoft.com). This presentation by Andriy Gorda (Engineering Manager & Lead Software Engineer, Consultant, GlobalLogic Kharkiv) was delivered at GlobalLogic Kharkiv MS TechTalk on June 13, 2017.

Overview SQL Server 2019

Juan Fabian

The document outlines the roadmap for SQL Server, including enhancements to performance, security, availability, development tools, and big data capabilities. Key updates include improved intelligent query processing, confidential computing with secure enclaves, high availability options on Kubernetes, machine learning services, and tools in Azure Data Studio. The roadmap aims to make SQL Server the most secure, high performing, and intelligent data platform across on-premises, private cloud and public cloud environments.

How to Set Up ApsaraDB for RDS on Alibaba Cloud

Alibaba Cloud

RDS is Alibaba Cloud's relational database service that provides a managed database service. It offers high availability, high performance, and scalability. Key benefits include usability through easy deployment and management, security through features like IP whitelisting and SQL attack protection, and availability through an architecture with primary and standby instances in different zones for failover. RDS instances can be easily scaled up or down and offer backups, read replicas, and temporary instances for recovery.

Upgrade your SQL Server like a Ninja

Amit Banerjee

Amit Banerjee is a senior program manager at Microsoft focusing on performance and high availability disaster recovery (HADR) for SQL Server. He has nearly a decade of experience with SQL Server and was previously part of Microsoft's SQL escalation services and premier field engineering teams. Banerjee is also an author of books on SQL Server internals and troubleshooting as well as migration to SQL Server on Azure. In this presentation, he discusses SQL Server 2017's focus on choice, intelligence, and easy migration. He also outlines the upgrade journey and provides an overview of tools and services for database assessment, migration, and modernization.

This document discusses modern Extract, Transform, Load (ETL) tools in Azure, including Azure Data Factory, Azure Data Lake, and Azure SQL Database. It provides an overview of each tool and how they can be used together in a data warehouse architecture with Azure Data Lake acting as the data hub and Azure SQL Database being used for analytics and reporting through the creation of data marts. It also includes two demonstrations, one on Azure Data Factory and another showing Azure Data Lake Store and Analytics.

Customer migration to Azure SQL database, December 2019

George Walters

AliCloud Object Storage Service (OSS) Core Features

Alibaba Cloud

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

HostedbyConfluent

The challenge with today’s “data explosion” is finding the most appropriate answer to the question, “So where do I put my data?” while avoiding the longer-term problem: data warehouses, data lakes, cloud storage, NoSQL databases, … are often the places where “big” data goes to die. Enter Physics 101, and my corollary to Newton’s First Law of Motion: Data in motion tends to stay in motion until it comes rest on disk. Similarly, if data is at rest, it will remain at rest until an external “force” puts it in motion again. Data inevitably comes to rest at some point. Without “external forces”, data often gets lost or becomes stale where it lands. “Modern” architectures tend to involve data pipelines where downstream consumers of data make use of data generated upstream, often with built-for-purpose repositories at each stage. This session will explore how data that has come to rest can be put in motion again; how Kafka can keep it in motion longer; and how pipelined architectures might be created to make use of that data.

Netflix's Big Leap from Oracle to Cassandra

Roopa Tangirala

This document summarizes Netflix's migration from Oracle to Cassandra. It discusses how Netflix moved its backend database from Oracle to Cassandra to gain scalability and reduce costs. The migration strategy involved dual writes to both databases, fork lifting the existing Oracle dataset, and a consistency checker. Challenges included security, denormalization, and engineering effort. Real use cases like APIs and viewing history are discussed along with lessons learned around data modeling, performance testing, and thinking of Cassandra as just storage.

Running Scylla on Kubernetes with Scylla Operator

ScyllaDB

- The document discusses running Scylla, a NoSQL database, on Kubernetes using the Scylla Operator. The Operator allows Kubernetes to leverage for workload management and provides a management layer for Scylla. - A demo shows deploying a Scylla cluster on Kubernetes with the Operator, stress testing the deployment, and performing common procedures like scaling up and upgrading Scylla versions. - The Operator uses custom resources and controllers to map Scylla concepts like members, clusters, and datacenters to Kubernetes concepts like statefulsets and pods. This provides capabilities like topology changes and rolling upgrades.

Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...

Lucas Jellema

Big Data Quickstart Series 3: Perform Data Integration

Alibaba Cloud

This document summarizes Derek Meng's presentation on data integration using Alibaba Cloud's MaxCompute big data platform. It discusses the general process of data integration including data acquisition, transformation, and governance. It provides an overview of MaxCompute basics, including its architecture, basic concepts such as projects and tables, and how to use MaxCompute's data channel and SQL. The document concludes with a brief introduction to DataWorks for data integration and a demo.

Logging infrastructure for Microservices using StreamSets Data Collector

Cask Data

This document discusses using StreamSets Data Collector (SDC) to build a logging infrastructure for microservices. SDC can ingest logs from microservices running in containers and handle issues like schema changes and new log formats. It processes and transforms the logs, sending them to destinations like Kafka. SDC pipelines can run on Spark clusters on Yarn and Mesos to handle large volumes of log data and load it into systems like HDFS, HBase and Elasticsearch for analysis.

Beyond Relational

Lynn Langit

The document discusses building data pipelines in the cloud. It covers serverless data pipeline patterns using services like BigQuery, Cloud Storage, Cloud Dataflow, and Cloud Pub/Sub. It also compares Cloud Dataflow and Cloud Dataproc for ETL workflows. Key questions around ingestion and ETL are discussed, focusing on volume, variety, velocity and veracity of data. Cloud vendor offerings for streaming and ETL are also compared.

Cloud-based Linked Data Management for Self-service Application Development

Peter Haase

Peter Haase and Michael Schmidt of fluid Operations AG presented on developing applications using linked open data. They discussed the increasing amount of linked open data available and challenges in building applications that integrate data from different sources and domains. Their Information Workbench platform aims to address these challenges by allowing users to discover, integrate, and customize applications using linked data in a no-code environment. Key components of the platform include virtualized integration of data sources and the vision of accessing linked data as a cloud-based data service.

Organizational compliance and security SQL 2012-2019 by George Walters

George Walters

Microsoft ignite 2018 SQL Server 2019 big data clusters - intro session

Travis Wright

Building a data warehouse with AWS Redshift, Matillion and Yellowfin

Lynn Langit

Benchmarking Aerospike on the Google Cloud - NoSQL Speed with Ease

Lynn Langit

Sql connections germany - migration considerations when migrating your on pre...

Charley Hanania

This document discusses considerations for migrating an on-premise SQL Server database system to SQL Azure. It provides an overview of the existing on-premise system and SQL Azure capabilities. Key challenges of the migration include determining which data to store where based on cost and performance, implementing connectivity retry logic to handle throttling and faults, and designing a cost-driven architecture in the cloud. The document recommends migrating sensor data and databases to SQL Azure while using Azure services like Data Sync and Analysis Services.

Personalization Journey: From Single Node to Cloud Streaming

Databricks

In the online gaming industry we receive a vast amount of transactions that need to be handled in real time. Our customers get to choose from hundreds or even thousand options, and providing a seamless experience is crucial in our industry. Recommendation systems can be the answer in such cases but require handling loads of data and need to utilize large amounts of processing power. Towards this goal, in the last two years we have taken down the road of machine learning and AI in order to transform our customer’s daily experience and upgrade our internal services.

Automated Metadata Management in Data Lake – A CI/CD Driven Approach

Databricks

We as data engineers are aware of trade off’s between development speed, metadata governance and schema evolution (or restriction) in rapidly evolving organization. Our day to day activities involve adding/removing/updating tables, protecting PII Information, curating and exposing data to our consumers. While our data lake keeps growing exponentially, there is equal increase in our downstream consumers. Struggle is to maintain balance between quickly promoting metadata changes with robust validation for downstream systems stability. In relational world DDL, DML changes can be managed through numerous options available for every kind of database from the vendor or 3rd party. As engineers we developed a tool which uses centralized git managed repository of data schemas in yml structure with ci/cd capabilities which maintains stability of our data lake and downstream systems. In this presentation Northwestern Mutual Engineers, will discuss how they designed and developed new end-to-end ci/cd driven metadata management tool to make introduction of new tables/views, managing access requests etc in a more robust, maintainable and scalable way, all with only checking in yml files. This tool can be used by people who have no or minimal knowledge of spark. Key focus will be: Need for metadata management tool in a data lake Architecture and Design of the tool Maintaining information on databases/tables/views like schema, owner, PII, description etc in simple to understand yml structure Live demo of creating a new table with CI/CD promotion to production

10 Things Learned Releasing Databricks Enterprise Wide

Databricks

Implementing tools, let alone an entire Unified Data Platform, like Databricks, can be quite the undertaking. Implementing a tool which you have not yet learned all the ins and outs of can be even more frustrating. Have you ever wished that you could take some of that uncertainty away? Four years ago, Western Governors University (WGU) took on the task of rewriting all of our ETL pipelines in Scala/Python, as well as migrating our Enterprise Data Warehouse into Delta, all on the Databricks platform. Starting with 4 users and rapidly growing to over 120 users across 8 business units, our Databricks environment turned into an entire unified platform, being used by individuals of all skill levels, data requirements, and internal security requirements. Through this process, our team has had the chance and opportunity to learn while making a lot of mistakes. Taking a look back at those mistakes, there are a lot of things we wish we had known before opening the platform to our enterprise. We would like to share with you 10 things we wish we had known before WGU started operating in our Databricks environment. Covering topics surrounding user management from both an AWS and Databricks perspective, understanding and managing costs, creating custom pipelines for efficient code management, learning about new Apache Spark snippets that helped save us a fortune, and more. We would like to provide our recommendations on how one can overcome these pitfalls to help new, current and prospective users to make their environments easier, safer, and more reliable to work in.

Vitalii Bondarenko "Machine Learning on Fast Data"

DataConf

This document discusses machine learning on fast data. It presents an agenda covering ML on production systems, TensorFlow, Kafka, Docker and Kubernetes. It then describes the machine learning process and shows how an enterprise analytics platform can integrate data sources, a machine learning cluster using Kafka, and data destinations. Details are provided on using TensorFlow for linear regression and neural networks. Apache Kafka is explained as a distributed streaming platform using topics, brokers, and consumer groups. The Confluent platform, KStream and KTable APIs are also summarized. Docker and Kubernetes are mentioned for containerization.

Migrating to Amazon RDS with Database Migration Service

Amazon Web Services

The document provides an overview of database migration options using AWS Database Migration Service (DMS) and AWS Schema Conversion Tool (SCT). It discusses how DMS can be used to migrate databases across different database platforms with minimal downtime. It also outlines how SCT can be used to convert schemas from commercial databases to open-source databases like PostgreSQL. The document shares customer examples and benefits of using DMS and SCT for heterogeneous, scale-up, and split migrations. It also lists available resources for customers on DMS and SCT.

Microsoft Cloud BI Update 2012 for SQL Saturday Philly

Mark Kromer

This document provides an overview and update of Microsoft's Cloud Business Intelligence (BI) solutions in version 3.0 from June 2012. It discusses the objectives of Cloud BI including providing data access and answers to business questions anytime from mobile devices. An overview of the session covers Windows Azure, SQL Azure, SQL Azure Reporting Services, mobile BI delivery, cloud data integration, data mining in the cloud, and hybrid scenarios. Key features of SQL Azure like import/export, data-tier applications, data sync, and federations for database scale-out are also summarized.

What's hot

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Eric Bragas

Customer migration to Azure SQL database, December 2019

George Walters

AliCloud Object Storage Service (OSS) Core Features

Alibaba Cloud

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

HostedbyConfluent

Netflix's Big Leap from Oracle to Cassandra

Roopa Tangirala

Running Scylla on Kubernetes with Scylla Operator

ScyllaDB

Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...

Lucas Jellema

Big Data Quickstart Series 3: Perform Data Integration

Alibaba Cloud

Logging infrastructure for Microservices using StreamSets Data Collector

Cask Data

Beyond Relational

Lynn Langit

Cloud-based Linked Data Management for Self-service Application Development

Peter Haase

Organizational compliance and security SQL 2012-2019 by George Walters

George Walters

Microsoft ignite 2018 SQL Server 2019 big data clusters - intro session

Travis Wright

Building a data warehouse with AWS Redshift, Matillion and Yellowfin

Lynn Langit

Benchmarking Aerospike on the Google Cloud - NoSQL Speed with Ease

Lynn Langit

Sql connections germany - migration considerations when migrating your on pre...

Charley Hanania

Personalization Journey: From Single Node to Cloud Streaming

Databricks

Automated Metadata Management in Data Lake – A CI/CD Driven Approach

Databricks

10 Things Learned Releasing Databricks Enterprise Wide

Databricks

Vitalii Bondarenko "Machine Learning on Fast Data"

DataConf

What's hot (20)