This document discusses AI at scale using Apache Spark on Azure. It provides an overview of Apache Spark, how it can be used for machine learning with tools like MLlib and Databricks, and how cognitive services can be combined with Spark. It also discusses using Azure services like Databricks, HDInsight and AKS for running Spark workloads at scale, and the roles of data engineers and data scientists.

1. AI at Scale
Adi Polak | Microsoft
@adipolak

2. Start from the beginning
• Massive datasets
• Intelligent product
@adipolak

3. • Apache Spark
• Databricks for Production
• Build machine learning models with
Pipelines
• Cognitive Services and Apache Spark
• Data Engineer vs Data Scientist
• MMLSpark
Overview
@adipolak

4. What is
Apache Spark

5. What is Apache Spark?
@adipolak

6. Motivation
@adipolak

7. Apache Spark architecture
@adipolak

8. Apache Spark submit job
storage storage storage
@adipolak

9. Framework libraries- Software Stracture
@adipolak

10. Python, R & .NET for Apache Spark
@adipolak

11. Embedded Service
PySpark
Spark Scala Process
Pyspark
Protocol
Spark Worker
@adipolak

12. Structured
Streaming
MLlib GraphFrame TensorFrames
SQL SparkSession / DataFrame / Dataset APIs
Data Source
Connectors
Catalyst Optimization & Tungsten Execution
Spark Core (RDD APIs)
SQL
API
More API
@adipolak

13. Why Apache Spark is fast and efficient?
CATALYST OPTIMIZER IN MEMORY
COMPUTING
TUNGSTEN
@adipolak

14. CATALYST
@adipolak

15. Fundamentals of Catalyst Optimizer
SUB
Attribute(x) SUB
some_func(1) some_func(2)
Tree Rules
SUB
Attribute(x) some_func(-1)
@adipolak

16. Spark SQL Execution Plan
Logical optimization –> Optimization rules
• Constant folding
• Predicate pushdown
• Projection pruning
• …
Catalyst
Frontend Backend
@adipolak

17. Apache Spark
Connectors

18. Spark Connector for SQL Server
@adipolak

19. Spark connector for Cosmos DB
@adipolak

20. Many more connectors…
@adipolak

21. How to run
Apache
Spark on
Azure?
HDInsight
Azure Databricks
AKS

22. Azure Databricks

23. What is Azure Databricks
@adipolak

24. Workspace and Notebooks

25. Build for Production
• - Databricks limits and Azure limits
• - Workspace isolation
• - vnet
• - Security and key vault
• - Enable log analytics for monitoring

26. Databricks limits and Azure limits
• Key Databricks limits ( per workspace):
• - 150 concurrent jobs
• - 150 max notebooks
• - 1000 max hourly job submission
Key Azure limits :
- Storage accounts per region per
subscription: 250
- VMs per subscription per
region: 25,000
- Resource groups per subscription: 980
@adipolak

27. Workspace isolation
@adipolak

28. Cluster modes and their characteristics
@adipolak

29. Batch vs. Interactive workloads
@adipolak

30. Data Scientists and Analysts –
Hight concurrency mode
@adipolak

31. Data Engineers
@adipolak

32. Troubleshooting with cluster logs
• Azure Databricks provides three kinds of logging of cluster-related
activity:
• Cluster event logs, which capture cluster lifecycle events, like creation,
termination, configuration edits, and so on.
• Apache Spark driver and worker logs, which you can use for
debugging.
• Cluster init-script logs, valuable for debugging init scripts.
@adipolak

33. Cluster logs

34. Cluster
Metric
Report
@adipolak

35. Log Analytics Workspace
• Leverage Azure log analytics and connect it to the workspace
• with init scripts and visualize it using Kibana or other tools.
@adipolak

36. What
About
Machine
Learning?

37. Gather Data
ML Process / Basic Life Cycle
Feature Extract, Clean and Normalize
Select algorithm
Evaluate model
Data visualization
4
5
1
2
3
Repeat!
@adipolak

38. H O W ?
M L P i p e l i n e s !
@adipolak

39. What are pipelines?

40. Tools
Spark
Streaming
Spark ML/Other
Spark SQL MLflow
@adipolak

41. @adipolak

42. Demo
@adipolak

43. But in real life:
Accuracy < 0.5
AUC L
Results are far
from being good

44. Building machine learning models
from scratch is hard!
@adipolak

45. We need a Data Science
@adipolak

46. Our Data scientist needs tools
@adipolak

47. Stand on the
shoulder of giants
@adipolak

48. Azure Cognitive Services
@adipolak

49. Cognitive Services capabilities
Infuse your apps, websites, and bots with human-like intelligence

50. A variety of real-world applications
Vision Speech
Intent: PlayCall
Language Knowledge Search

51. Let’s Combine them
+
@adipolak

52. Apache Spark + Cognitive Services
@adipolak

53. @adipolak

54. Data Engineer vs Data Scientist
Advanced programming
Distributed systems
Data pipeline
Basic analytics
Data Engineer
Advance math/statistics
ML/AI
Advanced analytics
Basic programming
Data Science
Big Data constrains
Both should understand:
@adipolak

55. MMLSpark:
Microsoft Machine
Learning for
Apache Spark
@adipolak

56. aka.ms/mlflow_and_azure_ml
aka.ms/azure_databricks
aka.ms/twitter_sentiment_analysis
Want to learn more?
@adipolak
Thank you! Dank u! Merci!