Successfully reported this slideshow.
Your SlideShare is downloading. ×

Sql vs no sql and azure data factory glasgow data UG

Sql vs no sql and azure data factory glasgow data UG

Download to read offline

NoSQL databases have grown in popularity in recent years due to the flexibility of data modeling and scaling up capabilities. NoSQL databases also have been used in the big data landscape. The demo rich session will elaborate the difference between SQL and NoSQL. And data moving capabilities from NoSQL database MongoDB to Azure Data Lake by using Azure data factory.

NoSQL databases have grown in popularity in recent years due to the flexibility of data modeling and scaling up capabilities. NoSQL databases also have been used in the big data landscape. The demo rich session will elaborate the difference between SQL and NoSQL. And data moving capabilities from NoSQL database MongoDB to Azure Data Lake by using Azure data factory.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Sql vs no sql and azure data factory glasgow data UG

  1. 1. SQL vs NoSQL and moving data from MongoDB to Azure data lake by using Azure Data Factory Diponkar Paul
  2. 2. Father and Husband Blogger & Speaker Profession: Data Engineer Working with BI, data warehouse 12 years Diverse background: South Asia, Nordic region, North America Community: Lead Toronto Data Professionals Community Twitter: @Paulswengrr Blog: www.allaboutdata.ca
  3. 3. What we cover Refresh our memory with traditional SQL Know about NoSQL (MongoDB) Demo: No SQL Comparison Azure data factory: Copy data from MongoDB Demo: MongoDB with ADF
  4. 4. SQL Syntax SELECT Id, Product, Price From Product Where ProductCategory=’Bikes’ Join, Insert, Update, Delete
  5. 5. Well defined Schema CREATE TABLE [Production].[Product]( [ProductID] [int] IDENTITY(1,1) NOT NULL, [Name] [nvarchar](100) NOT NULL, [ProductNumber] [nvarchar](25) NOT NULL, [MakeFlag] [dbo].[Flag] NOT NULL, [FinishedGoodsFlag] [dbo].[Flag] NOT NULL, [Color] [nvarchar](15) NULL, [SafetyStockLevel] [smallint] NOT NULL, [StandardCost] [money] NOT NULL, [ListPrice] [money] NOT NULL, [Size] [nvarchar](5) NULL)
  6. 6. Relationship/Normalization Customer Bridge table (Order) Product Id Name Price Description 1 “Mountain Bike “ 2500 “Bike for mountain trek” 2 “City Bike” 1000 “Best fit to roam around city” Id Customer_ID Product_ID 1 2 1 2 2 2 3 1 1 Id Name Email 1 Morten Sorenson m.s@outlook.com 2 Andersen Lu al@yahoo.com 3 Derek Paul dp@outlook.com
  7. 7. Type of relationships
  8. 8. NoSQL • MongoDB • Azure Cosmos DB • Amazon Document DB • Oracle NoSQL • Google BigTable Not Only SQL!!
  9. 9. NoSQL- MongoDB “MongoDB” derives from the word “humongous”
  10. 10. How we call them? Database E-Commerce Collections Table –Customer, Product… Documents {“Name”: ”Anders”, age:36} {“Name”: “Carsten”, age:42}
  11. 11. No defined Schema Id:1 Age:36Name: ‘Anders’ ….. Id:2 Age:36 Name: ‘Carsten’ ….. Id:3 …..
  12. 12. NoSQL –No relation Profession {id:1,profession:’Developer’} {id:2, profession: ’Data Engineer’} {id:3, profession: ’Actor’} Users {id:1,name:’Tom Hanks’, age:20} {id:2,name:’Casper Ruther’, age:42} {id:3,name:’Paul Anders’, age:63} db.Users.insert( { id:"01", name:"Tom Hanks", age:20 email:"th@hollywood.com", Profession:["Developer","Data Engineer","Actor"] } ) Usersprofession {id:1,userId:1,professionId:1} {id:2,userId:1, professionId: 2} {id:3,userId: 1, professionId: 3} {id:4,userId: 2, professionId: 2}
  13. 13. Tools: MongoDB https://www.mongodb.com/products/compass Robo 3T: https://robomongo.org/ https://docs.mongodb.com/manual/core/data-model-design/ https://docs.mongodb.com/manual/reference/method/db.collection.update/
  14. 14. Languages • MONGO SHELL • Python • java • C# • Scala • GO and many more.
  15. 15. Demo
  16. 16. SQL vs NoSQL SQL NoSQL Data uses Schema Schema-less (Schema Agnostic) Maintain Relationship No relations– though you can design relationship Data distributed in multiple tables Data in one table (embedded)
  17. 17. Move your NoSQL data from OnPrem to Data Lake Gen2
  18. 18. Azure Data Lake Azure Data Lake is a scalable data storage and analytics service -Fully HDFS compliance file system -Azure AD integrated -Microsoft’s PAAS service big data solution
  19. 19. Azure Data Factory -ETL/ELT Tool -Code free -Azure Cloud -a lot more…
  20. 20. Pre-requisite • azure account • Azure data factory resource • Linked services (Source and target connection) • Integration run time Integration Runtime Linked Service
  21. 21. Demo
  22. 22. Be cautious! • MongoDB version supported for ADF copy activity (V 3.4) *https://docs.microsoft.com/en-us/azure/data-factory/connector-mongodb
  23. 23. Questions @paulswengrr Diponkarpaul

×