Here are a few questions I have after the material covered so far: 1. What is the difference between the compute pool and app pool in a SQL Server 2019 big data cluster? 2. How does Polybase work differently in a SQL Server 2019 big data cluster compared to a traditional Polybase setup? 3. What components make up the storage pool in a SQL Server 2019 big data cluster?

1. Data & AI Revolution
18/9/2019

2. 300K+MEMBERS
WORLDWIDE
5 GLOBAL
NEWSLETTERS
22VIRTUAL
GROUPS
279 LOCAL
GROUPS
100+LOCAL EVENTS
4000 ATTENDEES
AT PASS SUMMIT

3. Registrarse para una membresía gratuita y asociarse al grupo Local SQL
Argentina:
• pass.org
• Sqlargentina.pass.org
https://www.facebook.com/groups/SQLArgentina/
@sqlargentina
https://www.linkedin.com/groups/3664741
https://www.youtube.com/channel/UCGadNDwP8nAVfQwqoeYHw
#SQLHelp

4. Introducción a SQL 2019 Big data Cluster
Maximiliano D. Accotto
Owner Triggerdb Consulting SRL.
Microsoft MVP Data Platform
www.triggerdb.com
@maxiaccotto
Gonzalo Bissio
SQL Engineering Team at Ernst & Young
MCSA 2012 - 2014- 2016 Database
administration /
MCSE: Data Management and Analytics.

5. Acerca de Maxi Accotto
Microsoft MVP Data Platform y speaker desde el año 2005.
Fundador de TriggerDB Consulting SRL.
Technical Solution Specialist Data Platform & BI con mas de 15 años de experiencia.
https://www.triggerdb.com
@maxiaccotto
https://www.linkedin.com/in/maxiaccotto
maxi@triggerdb.com

6. Acerca de Gonzalo Bissio
Flamante y orgulloso papá de Pedrito. Amo el mundo DBA tanto como al Club Independiente y
a Rocky Balboa ☺ ☺
https://sqlserverrules.wordpress.com
@Gbissio
https://www.linkedin.com/in/gonzalo-juan-jose-bissio-19b3bb32/
gonzalobissio@Hotmail.com

7. Polybase
7

8. SQL Server
T-SQLAnalytics Apps
ODBC NoSQL Relational databases Big data
PolyBase external tables
What is SQL Server Polybase?
✓ Distributed compute engine integrated with SQL Server
✓ Query data where it lives using T-SQL
✓ Distributed, scalable query performance
✓ Manual/deploy with SQL Server
✓ Auto deploy/optimize with Big Data Clusters
Intelligence over all data
“It’s all about
Data Virtualization”

9. Using Polybase in SQL Server: T-SQL EXTERNAL TABLE
Setup and
configure
Polybase
Setup
authentication
Create
EXTERNAL
DATA SOURCE
Create
EXTERNAL FILE
FORMAT
Create
EXTERNAL
TABLE
Create
statistics on
key columns
Query like any
other table
SQL

10. The SQL Server Polybase Architecture
SQL Engine
Polybase
Engine
Polybase Data
Movement
Service
SQL Engine
Polybase
Engine
Polybase Data
Movement
Service
SQL Engine
Polybase
Engine
Polybase Data
Movement
Service
mpdwsvc.exe

11. SQL Server 2019: Data Virtualization
SQL

12. SQL Server Polybase and Hadoop
Polybase
“Classic”
Connect to Hadoop: Cloudera or HortonWorks on Windows and Linux
Azure Blob Storage
Direct Steaming or Java MapReduce
SQL Server or Azure SQL Data Warehouse
Polybase
and HDFS
in Big Data
Clusters
Direct Access to HDFS via SQL Server Engine
Hadoop cluster pre-installed with HDFS and Spark
HDFS metadata handled within the cluster

13. Polybase as a Semantic Layer
Use EXTERNAL table to map common names familiar to your SQL Server
Exclude fields or columns from data source that are not needed (Planned)
Use SQL Server views to abstract joins and data source access
Use UNION to join similar data from data sources and local SQL Server

14. External Tables vs Linked Servers
External Tables
Database Scoped
ODBC Drivers
Read-only*
Scale out queries with push-down
Failover with AG
Basic authentication
Distributed Transactions not supported
Linked Servers
Instance Scoped
OLEDB Providers
Read/write
Single threaded queries with push-down
Requires separate config from AG
Basic and integrated authentication
Distributed Transactions supported

15. SQL Server 2019
Big Data Cluster
1

16. SQL Server 2019 Big Data Clusters and Polybase
The Problem
• Customers want to integrate with Big Data and other data sources easily
• Polybase is not simple to setup, configure, maintain, and scale elastically
• Customers may not have a Hadoop cluster or want to build one
• Polybase “classic” (MapReduce) could be better

17. Kubernetes concepts

18. Kubernetes concepts

19. BDC Components

20. Control Plane
• These components are used to manage and monitor the solution
Control Plane: Controller Service
• The Controller in the BDC is a service that is deployed with the azdata utility. It bridges the interactions
with SQL Server, Kubernetes, Spark and HDFS.
BDC Components - 1

21. BDC: SQL Server Master Instance
• The SQL Server Master Instance is an installation of SQL Server 2019 in a Pod on a Node in the
Kubernetes cluster. You access it the same way as any SQL Server Instance, and use it for high-value,
OLTP, OLAP or other types of workloads. It has Machine Learning Services already configured, so you
have the full range of R, Python, and Java to work with on the data in the Cluster environment.
• The master instance is a SQL Server instance running in a big data cluster to manage connectivity,
scale-out queries, metadata and user databases, and machine learning services.
BDC: Compute Pool
• The Compute Pool holds one or more SQL Server Pods used for distributed processing under the
direction of the SQL Server Master Instance. It makes the calls out to the PolyBase connectors for a
distributed Compute layer of the BDC.
BDC Components - 2

22. BDC: App Pool
• The App Pool is a set of Pods within a Node that hold multiple types of end-points into the system. SQL Server Integration
Services lives in the App Pool, and other Job systems are possible. You could instatiate a long-running job (such as IoT
streaming) or Machine Learning (ML) endpoints used for scoring a prediction or returning a classification.
• These components are used in the Compute Pool of the BDC:
BDC: Data Pool
• The Data Pool in a BDC consists of one or more SQL Server data pool instances. SQL data pool instances provide persistent
SQL Server storage for the cluster. A data pool is used to ingest data from SQL queries or Spark jobs, or other locations. To
provide better performance across large data sets, data in a data pool is distributed into shards across the member SQL
data pool instances.
BDC: Storage Pool
• The storage pool consists of storage nodes comprised of SQL Server on Linux, Spark, and HDFS. All the storage nodes in a
SQL big data cluster are members of an HDFS cluster. You can use these as a "Data Lake" construct to work with large sets of
data stored on disparate data sources.
BDC Components - 3

23. BDC data Pool
23

24. BDC Storage Pool
24

25. Polybase in SQL Server 2019 Big Data Clusters
Compute pool
SQL Compute
Node
SQL Compute
Node
SQL Compute
Node
…
Compute pool
SQL Compute
Node
IoT data
Directly
read from
HDFS
Persistent storage
…
Storage pool
SQL
Server
Spark
HDFS Data Node
SQL
Server
Spark
HDFS Data Node
SQL
Server
Spark
HDFS Data Node
Kubernetes pod
Analytics
Custom
apps BI
SQL Server
master instance
Node Node Node Node Node Node Node
SQL
Data mart
SQL Data
Node
SQL Data
Node
Compute pool
SQL Compute
Node
Storage Storage
Polybase head node
In Linux containerPolybase compute nodes
In Linux containers
“Built-in”
Data Sources
MapReduce
Not used
mpdwsvc.exe
uses SQLPALControl Plane
Controller Svc
Configuration Store (SQL Server)
Grafana
Elastic Search
Azure FSM Engine
InfluxDB
Kibana

26. SQL Server 2019 BDC
Deploy
2

27. BDC – Deploy
27
AKS Python Script
Bash Script
Azure Data Studio
https://docs.microsoft.com/en-us/sql/big-data-cluster/deploy-notebooks?view=sqlallproducts-allversions

28. SQL Server 2019 BDC
Monitoring
2

29. BDC – Monitoring - Using commands
29
Using kubectl commands
• Since the BDC lives within a Kubernetes cluster, you'll work with the kubectl command to deal
with those specific components. The following list is a short version of some of the commands
you can use to manage and monitor the BDC implementation of a Kubernetes cluster:
Using azdata commands
• The azdata utility enables cluster administrators to bootstrap and manage big data clusters via
the REST APIs exposed by the Controller service. The controller is deployed and hosted in the
same Kubernetes namespace where the customer wants to build out a big data cluster. The
Controller is responsible for core logic for deploying and managing a big data cluster.
• The Controller service is installed by a Kubernetes administrator during cluster bootstrap, using
the azdata command-line utility.

30. BDC – Monitoring - Using Grafana and Kibana
30
The Azure Data Studio big data clusters management panel shows the TCP/IP addresses for each of these systems.

31. Break!
Questions up to now?
3

32. 3