NoSQL databases have grown in popularity in recent years due to the flexibility of data modeling and scaling up capabilities. NoSQL databases also have been used in the big data landscape. The demo rich session will elaborate the difference between SQL and NoSQL. And data moving capabilities from NoSQL database MongoDB to Azure Data Lake by using Azure data factory.

1. SQL vs NoSQL and moving data from
MongoDB to Azure data lake by
using Azure Data Factory
Diponkar Paul
@Baton Rouge User Groups

2. Father and
Husband
Blogger &
Speaker
Profession:
Data Engineer
Working with
BI, data
warehouse 12
years
Diverse
background:
South Asia,
Nordic
region, North
America
Community:
Lead Toronto
Data
Professionals
Community
Twitter: @Paulswengrr
Blog: www.allaboutdata.ca

3. What we
cover
Refresh our memory with traditional SQL
Know about NoSQL (MongoDB)
Demo: No SQL
Comparison
Azure data factory: Copy data from MongoDB
Demo: MongoDB with ADF

4. SQL Syntax
SELECT Id, Product, Price
From Product
Where ProductCategory=’Bikes’
Join, Insert, Update, Delete

5. Well defined
Schema
CREATE TABLE [Production].[Product](
[ProductID] [int] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](100) NOT NULL,
[ProductNumber] [nvarchar](25) NOT NULL,
[MakeFlag] [dbo].[Flag] NOT NULL,
[FinishedGoodsFlag] [dbo].[Flag] NOT NULL,
[Color] [nvarchar](15) NULL,
[SafetyStockLevel] [smallint] NOT NULL,
[StandardCost] [money] NOT NULL,
[ListPrice] [money] NOT NULL,
[Size] [nvarchar](5) NULL)

6. Relationship/Normalization
Customer Bridge table (Order)
Product
Id Name Price Description
1 “Mountain Bike “ 2500 “Bike for mountain trek”
2 “City Bike” 1000 “Best fit to roam around city”
Id Customer_ID Product_ID
1 2 1
2 2 2
3 1 1
Id Name Email
1 Morten
Sorenson
m.s@outlook.com
2 Andersen Lu al@yahoo.com
3 Derek Paul dp@outlook.com

7. Type of
relationships

8. NoSQL
• MongoDB
• Azure Cosmos DB
• Amazon Document DB
• Oracle NoSQL
• Google BigTable
Not Only SQL!!

9. NoSQL- MongoDB
“MongoDB” derives from the word “humongous”

10. How we call them?
Database E-Commerce
Collections Table –Customer, Product…
Documents {“Name”: ”Anders”, age:36}
{“Name”: “Carsten”, age:42}

11. No defined Schema
Id:1 Age:36
Name: ‘Anders’ …..
Id:2
Age:36
Name: ‘Carsten’ …..
Id:3 …..

12. NoSQL –No relation
Profession
{id:1,profession:’Developer’}
{id:2, profession: ’Data Engineer’}
{id:3, profession: ’Actor’}
Users
{id:1,name:’Tom Hanks’, age:20}
{id:2,name:’Casper Ruther’, age:42}
{id:3,name:’Paul Anders’, age:63}
db.Users.insert(
{
id:"01",
name:"Tom Hanks",
age:20
email:"th@hollywood.com",
Profession:["Developer","Data
Engineer","Actor"]
}
)
Usersprofession
{id:1,userId:1,professionId:1}
{id:2,userId:1, professionId: 2}
{id:3,userId: 1, professionId: 3}
{id:4,userId: 2, professionId: 2}

13. Tools: MongoDB
https://www.mongodb.com/products/compass
Robo 3T: https://robomongo.org/
https://docs.mongodb.com/manual/core/data-model-design/
https://docs.mongodb.com/manual/reference/method/db.collection.update/

14. Languages
• MONGO SHELL
• Python
• java
• C#
• Scala
• GO and many more.

15. Demo

16. SQL vs NoSQL
SQL NoSQL
Data uses Schema Schema-less (Schema Agnostic)
Maintain Relationship No relations– though you can design relationship
Data distributed in multiple tables Data in one table (embedded)

17. Move your NoSQL data from OnPrem to Data
Lake Gen2

18. Azure Data Lake
Azure Data Lake is a scalable data storage
and analytics service
-Fully HDFS compliance file system
-Azure AD integrated
-Microsoft’s PAAS service big data solution

19. Azure Data Factory
-ETL/ELT Tool
-Code free
-Azure Cloud
-a lot more…

20. Pre-requisite
• azure account
• Azure data factory resource
• Linked services (Source and
target connection)
• Integration run time
Integration
Runtime
Linked Service

21. Demo

22. Be cautious!
• MongoDB version supported for ADF copy activity (V 3.4)
*https://docs.microsoft.com/en-us/azure/data-factory/connector-mongodb

23. Questions
@paulswengrr
Diponkarpaul