The document discusses the integration of .NET with Apache Spark, emphasizing the demand from .NET developers for big data capabilities. It outlines Spark's features such as speed and scalability for processing large data sets, and introduces .NET Spark bindings for seamless usage within the .NET ecosystem. Additionally, it highlights the evolution of notebooks for data science and presents upcoming events related to SQL and data science.

1. 1
TOPIC
.net developer for Jupyter
Notebooks and Apache
Spark and viceversa
Marco Parenzan
(c)2020FrancescaParenzan

2. Who I am
https://twitter.com/marco_parenzan
https://github.com/marcoparenzan
https://www.linkedin.com/in/marcoparenzan/
Marco Parenzan
Prossimo evento 1nn0va: SQL Saturday Pordenone, 30 maggio 2020 https://www.sqlsaturday.com/921/

3. Road to Data Science

4. Big Data
Volume Velocity and Variety of Data
Thousands of IoT sensors in a fatory, producing petabytes of
data

5. What is Apache Spark?
• General-purpose distributed
processing engine for analytics over
large datasets
• For
• SparkSQL
• Streaming
• Machine Learning
• Visualization
• OSS anlutics engine
• Access data from variety of sources
and types
• ADLS, HDFS, S3, Kafka
• Text files, CSVs, JSONs, Parquet
• In-memory computation improves
efficiency and speed
• 100x faster that disk-based Hadoop
MapReduce
• 10x faster event when still on-disk
• Faster speeds  more interactive
analysis
• Developers can harness the power
of big data in a language of their
chiuce with the various high-
performance Spark APIs

6. DataFrame: the core of Spark programming
CSV Data
JSON Data
RDBMS Data
Parquet Data
Binary Data
MySQL Data
DataFrame
User programs
against the
DataFrame
abstraction

7. A Spark Recipe
Create Session
Create Dataframe
Define a user defined function
Manipulate and view Data

8. Elements of Spark
Spark
Session
User’s
Program
Task1 Task 2
Task 3 Task 4
Driver Executor
Cluster Manager

9. Batch vs. Notebooks
Work on slow data stored
into a Datalake
Submit a complete app in
one single deploy
Receive the entire output
Use «spart-submit»
Run cell by cell (but also all
at once)

10. What about .NET?
In a recent survey, more than 70% of .NET devs expressed
interest in Apache Spark
Millions of lines of big data-usable business logic are written
in .NET
But .NET devs are locked out from big data processing –
lack of .NET support in OSS big data solutions
We want a first-class .net experience in Spark

11. .NET for Apache Spark
• .NET bindings (C# e F#) to
Spark
• Written on the Spark interop
layer, designed to provide
high performance bindings to
multiple languages
• Re-use knowledge, skills,
code you have as a .NET
developer
• Compliant with .NET Standard
• You can use .NET for Apache
Spark anywhere you write
.NET code
• Original project Moebius
• https://github.com/microsoft/
Mobius

12. .NET Spark support
Spark DataFramews
with SparkSQL
• Spark 2.3.x, 2.4.x
• ~300 SparkSQL
function
• DeltaLake
.NET Standard 2.0
• C#/F#
• .NET Framework
4.6.1+
• .NET Core 2.1+
Batch&Streaming
• Structured
Streaming
Data Science
• ML.NET
• Notebooks

13. DEMO

14. The .NET Notebook experience

15. Evolution of REPL
• At the beginning there
was mono
• Then Dynamic/DLR (C#
4)
• C#/F# interactive
• .NET Try
In a world of:
• Python
• Mathematica

16. Jupyter
• Evolution and generalization of the seminal role of
Mathematica (notebook)
• +Python adoption (ipynb)
• +Web (HTTP+Markdown)
• +Kernel

17. DEMO

18. Conclusions
• .NET for Spark
• 1.0 GA in May/June
• https://github.com/dotnet/spark/
• .NET Interactive
• https://github.com/dotnet/interactive/

19. Thanks
Questions?
https://twitter.com/marco_parenzan
https://github.com/marcoparenzan
https://www.linkedin.com/in/marcoparenzan/