Databricks resources deployed to a pre-provisioned VNET Databricks traffic isolated from regular network traffic Prevent data exfiltration Internal traffic between cluster nodes internal and encrypted Access to Databricks control plane limited and controlled

1. Databricks Oslo
User Group MeetUp #5
March 16th 2022

2. ©2019 Avanade Inc. All Rights Reserved
• This is a group for anyone interested in Databricks. All skill levels are welcome.
• Established in June 2019
• We are about 220 members
• The user group aims to arrange 3-5 physical meetings per year in the Oslo area.
• Avanade Norway started this group to meet other people excited about the possibilities in
Databricks and to exchange knowledge and experiences.
• This MeetUp group is not sponsored by (or affiliated with) Databricks. It is an unofficial
community group.
• The MeetUp group is part of Azure Tech Communities and sponsored by Microsoft
Databricks Oslo User Group - DOUG
2

3. ©2019 Avanade Inc. All Rights Reserved
17:00 - Doors open / mingling / coffee
17:30 – Welcome
17:40 – Databricks Enterprise Security
Databricks secure deployments and security baselines
Azure Data Factory and Databricks interaction - with security focus
Vacuum of delta tables for GDPR compliance
Antonio Abalos Castillo and Marino Grønseth (Avanade)
18:25 – Break – 15 minutes
18:40 - Experiences building an enterprise Data Lakehouse using Azure Databricks
Halvar Trøyel Nerbø & Sindre Grindheim (Glitni)
19:25 - Wrap up / mingling
Agenda
3

4. Databricks secure deployments and security baselines
DOUG, March 2022
Antonio Abalos Castillo, @antonioabalos

5. ©2019 Avanade Inc. All Rights Reserved
The perfectly secure deployment is like travelling at light-speed, the
more secure we want to be, the more energy it requires.
Travelling at light-speed
5

6. ©2019 Avanade Inc. All Rights Reserved
Where is data stored?
What kind of questions can we expect?
6
Is data protected while in-transit?
Is data encrypted at-rest?
What region is data stored on?
Can data be transferred to another region?

7. ©2019 Avanade Inc. All Rights Reserved
1. Databricks resources deployed to a pre-provisioned VNET
2. Databricks traffic isolated from regular network traffic
• Prevent data exfiltration
3. Internal traffic between cluster nodes internal and encrypted
4. Access to Databricks control plane limited and controlled
Goals for secure deployments
7

8. ©2019 Avanade Inc. All Rights Reserved
Databricks
8

9. ©2019 Avanade Inc. All Rights Reserved
Databricks
9
Databricks + Azure
User subscription
Isolated VNET
At-rest encryption
Access control
Deploy Azure Databricks in your Azure virtual network (VNet injection)
Service endpoint / private link

10. ©2019 Avanade Inc. All Rights Reserved
Most secure features require Premium (*)
• Single sign-on
• Role-based access control
• Credentials passthrough
• VNET injection
• Secure cluster connectivity
• IP access list
• Customer-managed keys for control plane data encryption
• Customer-managed keys for DBFS (data plane) data encryption
Databricks pricing plan
10
(*) All but SSO require the Premium plan.
Enterprise security for Azure Databricks - Azure Databricks
1 VM x 1 DBU x 1 hour
Standard kr6,97
Premium kr8,31 (+19%)

11. ©2019 Avanade Inc. All Rights Reserved
- Service Endpoints
- No extra cost
- Private Link
- Inbound traffic: kr0,0909/GB
- Outbound traffic: kr0,0909/GB
Other price considerations
11
Pricing - Azure Private Link | Microsoft Azure
Azure virtual network service endpoints | Microsoft Docs

12. ©2019 Avanade Inc. All Rights Reserved
Default Azure Databricks deployment
- VNET created by Databricks
- Resources created in locked resource group
- Subnets created and managed by Databricks
VNET injection
12
Deploy Azure Databricks in your Azure virtual network (VNet injection)
VNET injection
- VNET managed by user
- Subnets created and managed by the
user
- Custom DNS settings
- Custom route tables
- On-premises data connection available
(*) You cannot replace the VNet for an existing workspace

13. ©2019 Avanade Inc. All Rights Reserved
- Workspace and VNET must reside in the same region
- Workspace and VNET must reside in the same subscription
- Address space for VNET: between CIDR /16 and /24 (65k and 256 respectively)
- Several workspaces can share the same VNET
- 2 dedicated subnets exclusive for each workspace:
- Public/host subnet
- Private/container subnet
- 5 IP addresses reserved for Azure in each subnet
VNET injection requirements
13

14. ©2019 Avanade Inc. All Rights Reserved
VNET injection example
14
VNET Databricks
Container/private
subnet
Host/public subnet
Cluster VMs
VNET Hub
Azure Firewall
VNET peering
Service connections
Credentials passthrough
Service Principal
TLS inter-node
communications
Storage Blob
ADLS
Route tables
Key Vault
Enterprise App VNET
VMs
How to Deploy Databricks Clusters in Your Own Custom VNET
Deploy Azure Databricks in your Azure virtual network (VNet injection)

15. ©2019 Avanade Inc. All Rights Reserved
Service connections
15
Azure virtual network service endpoints
Securely Accessing Azure Data Sources from Azure Databricks
No additional price
Endpoints allow you to secure your critical Azure service resources to only
your virtual networks. Traffic from your VNet to the Azure service always remains on
the Microsoft Azure network backbone.

16. ©2019 Avanade Inc. All Rights Reserved
Data exfiltration protection
Private-endpoints
16
Securely Accessing Azure Data Sources from Azure Databricks
Pricing - Azure Private Link | Microsoft Azure
Extends your private network address space to Azure Data services, i.e. the Azure
data service effectively gets a private IP in one of your VNETs and could be treated
as part of your larger private network.

17. ©2019 Avanade Inc. All Rights Reserved
Secure Cluster
Connectivity
(No Public-IP)
17
Secure cluster connectivity (No Public IP / NPIP)
* You cannot add secure cluster connectivity to an existing workspace
* Using Secure Cluster Connectivity with default (managed) VNET
creates a NAT gateway, incurring in extra costs

18. ©2019 Avanade Inc. All Rights Reserved
Data exfiltration full overview
18
How to protect Data Exfiltration with Azure Databricks to help ensure Cloud Security

19. ©2019 Avanade Inc. All Rights Reserved
Other secure characteristics
19

20. ©2019 Avanade Inc. All Rights Reserved
Restrict access to Azure Databricks Control Plane by using Conditional Access.
Authenticated connections allowed only from pre-defined IP addresses.
- Requires Azure AD Premium P1
Azure AD Conditional access
20
What is Conditional Access in Azure Active Directory?

21. ©2019 Avanade Inc. All Rights Reserved
Access control
• Workspace object access control: folder and notebook access
• Cluster access control: access to changing clusters (attach, restart, manage)
• Pool access control: changing access to pools (attach, manage)
• Jobs access control: job results (view, manage run, owner, manage)
• Table access control (premium): table access (deny)
• Secret access control: create, view, delete
Databricks Access Control
21
Security guide - Azure Databricks | Microsoft Docs

22. ©2019 Avanade Inc. All Rights Reserved
DBFS encryption
22
Configure customer-managed keys for DBFS root
Key vault in the same region and same Azure Active
Directory (Azure AD) tenant as your Azure Databricks
workspace.
They can be in different subscriptions.
Key identifier
Encryption
Keys can be rotated
without re-encrypting
storage account content.
Databricks workspace needs
key permissions in Key Vault
to perform wrap and unwrap
key operations.

23. ©2019 Avanade Inc. All Rights Reserved
1. Network Security
2. Logging and Monitoring
3. Identity and Access Control
4. Data Protection
5. Vulnerability Management
6. Inventory and Asset Management
7. Secure Configuration
8. Malware Defense
9. Data Recovery
10. Incident Response
11. Penetration Tests and Red Team Exercises
Azure Databricks security baseline (34 pages)
https://docs.microsoft.com/en-us/security/benchmark/azure/baselines/databricks-security-baseline

24. Vacuum of delta tables for GDPR compliance - Demo
DOUG, March 2022
Marino Bråthen Grønseth, LinkedIn

25. ©2019 Avanade Inc. All Rights Reserved
We hope to see you next time!
The DOUG crew
Henrik Brattlie, LinkedIn