Montreal Sql saturday: moving data from no sql db to azure data lake

•

0 likes•75 views

NoSQL database have grown popularity in recent years due to the flexibility of data modeling and scaling up capabilities. NoSQL database also have been using in big data landscape. The demo rich session will elaborate difference between SQL and NoSQL. And end to end solution for data moving capabilities from NoSQL database MongoDB by using Azure data factory.

Technology

SQL vs NoSQL and moving data from
MongoDB to Azure data lake by using
Azure Data Factory
Diponkar Paul

Thanks to our sponsors
And
Global
Microsoft RedHat Google Cloud

Title
Content
Father and
Husband
Blogger &
Speaker
Data
Engineer
Diverse
background
Community
Twitter: @Paulswengrr
Blog: www.allaboutdata.ca

What we cover
Refresh our memory with traditional SQL
Know about NoSQL (MongoDB)
Demo: No SQL
Comparison
Azure data factory: Copy data
from MongoDB
Demo: MongoDB with
ADF

SQL Syntax
SELECT Id, Product, Price
From Product
Where ProductCategory=’Bikes’
Join, Insert, Update, Delete

Well defined Schema
CREATE TABLE [Production].[Product](
[ProductID] [int] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](100) NOT NULL,
[ProductNumber] [nvarchar](25) NOT NULL,
[MakeFlag] [dbo].[Flag] NOT NULL,
[FinishedGoodsFlag] [dbo].[Flag] NOT NULL,
[Color] [nvarchar](15) NULL,
[SafetyStockLevel] [smallint] NOT NULL,
[StandardCost] [money] NOT NULL,
[ListPrice] [money] NOT NULL,
[Size] [nvarchar](5) NULL)

Relationship/Normalization
Customer Bridge table (Order)
Product
Id Name Price Description
1 “Mountain
Bike “
2500 “Bike for mountain trek”
2 “City Bike” 1000 “Best fit to roam around
city”
Id Customer_ID Product_ID
1 2 1
2 2 2
3 1 1
Id Name Email
1 Morten
Sorenson
m.s@outlook.com
2 Andersen Lu al@yahoo.com
3 Derek Paul dp@outlook.com

Vendors in the market
• MongoDB
• Azure Cosmos DB
• Amazon Document DB
• Oracle NoSQL
• Google BigTable

How we call them?
Database E-Commerce
Collections Table –Customer, Product…
Documents {“Name”: ”Anders”, age:36}
{“Name”: “Carsten”, age:42}

No defined Schema
Id:1 Age:36Name: ‘Anders’ …..
Id:2
Age:36
Name: ‘Carsten’ …..
Id:3 …..
Age:36 …..
Age:36
Name: ‘Carsten’ …..
…..
Id:1 Name: ‘Anders’
Id:2
Age:36 …..
Name: ‘Carsten’ …..

NoSQL –No relation
Profession
{id:1,profession:’Developer’}
{id:2, profession: ’Data
Engineer’}
{id:3, profession: ’Actor’}
Users
{id:1,name:’Tom Hanks’, age:20}
{id:2,name:’Casper Ruther’, age:42}
{id:3,name:’Paul Anders’, age:63}
db.Users.insert(
{
id:"01",
name:"Tom Hanks",
age:20
email:"th@hollywood.com",
Profession:["Developer","Data Engineer","Actor"]
}
)
Usersprofession
{id:1,userId:1,professionId:1}
{id:2,userId:1, professionId: 2}
{id:3,userId: 1, professionId: 3}
{id:4,userId: 2, professionId: 2}

Tools: MongoDB
https://www.mongodb.com/products/compass
Robo 3T: https://robomongo.org/
https://docs.mongodb.com/manual/core/data-model-design/
https://docs.mongodb.com/manual/reference/method/db.collection.update/

Languages
• MONGO SHELL
• Python
• java
• C#
• Scala
• GO and many more.

SQL vs NoSQL
SQL NoSQL
Data uses Schema Schema-less (Schema Agnostic)
Maintain Relationship No relations– though you can design
relationship
Data distributed in multiple tables Data in one table (embedded)
Monolithic, you can easily Scale-Up.
Scale out is also possible but difficult
(e.g. Azure Elastic Database tools)
Scale up and scale out- Globally
distributed

Move your NoSQL data from OnPrem to
Data Lake Gen2

Azure Data Lake
Azure Data Lake is a scalable data storage and analytics
service
-Fully HDFS compliance file system
-Azure AD integrated
-Microsoft’s PAAS service big data solution

Azure Data Factory
• ETL/ELT Tool
• Code free
• Azure Cloud
• a lot more…

Pre-requisite
• azure account
• Azure data factory resource
• Linked services (Source and
target connection)
• Integration run time
Integration
Runtime
Linked Service

Be cautious!
• MongoDB version supported for ADF copy activity (V 3.4)
*https://docs.microsoft.com/en-us/azure/data-factory/connector-mongodb

Key Takeaways:
• SQL vs. NoSQL
• Choose SQL/ NoSQL
• Integration from OnPrem MongoDB
• Starting new NoSQL?- choose cloud solution Azure CosmosDB or
Atlas

This document discusses using MongoDB to store and query location data. Key points include: - MongoDB supports geospatial indexing and querying using Geo2D indexes on location fields. - Queries like $near, $box, $center, and $polygon allow searching by proximity or within regions. - Data should be structured with location as an array [long, lat] for spherical queries. - Scaling involves sharding on a geo field, but has query limitations in MongoDB 1.8.

NoSQL with MongoDB

Ikram Manseri

In this presentation we start by making a comparison between SQL (relational) and NoSQL DBMS. Next, we go through NoSQL DBMS types : column based, key-value based, document oriented and graph oriented dbms. MongoDB is one of the popular document oriented NoSQL DBMS. In this presentation we discovered how data is structured in MongoDB between collections and databases. Then, we jumped to MongoDB queries, we identified all the possible queries : searching, inserting, updating and removing data queries. MongoDB has a set of relevant indexes : text indexes, geographical indexes, unique indexes ...etc. Indexes are a data structures that help the DBMS administrator in accelerating his search queries. At the end, we had an overview about Sharding and Replication in MongoDB.

Building a Location-based platform with MongoDB from Zero.

Ravi Teja

Raviteja Dodda presented on building a location-based backend platform using MongoDB and MongoEngine. He discussed using MongoDB's geospatial indexing and query capabilities to build a check-in rewards app that allows users to check-in to places and earn rewards. He outlined the data models for places, users, check-ins and rewards, and provided examples of geospatial queries to find places and rewards near a given location. Raviteja concluded by encouraging attendees to build their own location-based applications using MongoDB's simple and powerful geospatial features.

MongoDB - A Document NoSQL Database

Ruben Inoto Soto

MongoDB is a document-oriented NoSQL database that uses a document-data model. It provides horizontal scaling with auto-sharding and replication. MongoDB can store documents in collections without a defined schema and support dynamic queries and indexing. RealNetworks uses MongoDB with Scala and other technologies for an Android app to send notifications to devices with installed RealNetworks applications at scale.

Mongo db – document oriented database

Wojciech Sznapka

This document discusses MongoDB, a document-oriented NoSQL database. It begins with an introduction to NoSQL databases and their advantages over relational databases. MongoDB is then described in more detail, including its features like storing dynamic JSON documents, indexing, replication, querying and MapReduce capabilities. Examples of CRUD operations on MongoDB documents are provided. The document concludes by discussing when MongoDB may be applicable and who uses it, as well as comparing MongoDB to SQL databases.

MongoDB @ Frankfurt NoSql User Group

Chris Harris

When to Use MongoDB

MongoDB

Engineers often ask "how do I know if I should build my application on MongoDB?" IT executives ask a similar question, "which applications in my application portfolio should I migrate to MongoDB?" This presentation will present a framework for answering these questions. We will cover two sets of criteria: (1) how to determine when to migrate a legacy application to MongoDB and (2) when should MongoDB be used for new applications? The presentation will also include a brief introduction to MongoDB to provide enough MongoDB technical background for analyzing when to use MongoDB? Learning Objectives: The basics of MongoDB document model, query capabilities, and architecture required for analyzing when to use MongoDB? Criteria for determining when to use MongoDB to re-platform legacy applications Criteria for determining when to use MongoDB for new applications

Back to Basics Webinar 4: Advanced Indexing, Text and Geospatial Indexes

MongoDB

This document discusses optimizing slow queries in MongoDB through the use of indexes and schema design. It begins with examples of common performance issues like inefficient reads from table scans and writes causing locks. Each example shows the slow query, explains the problem, and proposes a solution like adding an index. It emphasizes that new indexes only scale up to a point and sometimes the best approach is to redesign the schema or add hardware. The talk provides guidance on measuring performance, finding slow queries, determining if they can be indexed, and rephrasing problems in terms of desired queries.

MongoDB .local Munich 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pip...

MongoDB

MongoDB .local Toronto 2019: Tips and Tricks for Effective Indexing

MongoDB

MongoDB .local Munich 2019: Tips and Tricks++ for Querying and Indexing MongoDB

MongoDB

MongoDB Indexing Constraints and Creative Schemas

MongoDB

This document summarizes Chris Winslett's talk on MongoDB indexing and improving performance. It discusses: 1) Common performance issues in MongoDB like inefficient queries causing high CPU from table scans, and how to address them through indexing and schema design. 2) Examples of performance problems like social networking queries and messages, and how indexing on viewable data rather than relationships improved queries. 3) Tools for debugging like mongostat, logs, and currentOp to find slow queries and operations. 4) The importance of measuring performance, finding slow queries, and refactoring schemas and queries to optimize for desired queries through indexing rather than always adding more indexes.

Mapping Flatland: Using MongoDB for an MMO Crossword Game (GDC Online 2011)

Grant Goodale

WordSquared is a massively multiplayer online scrabble crossword that uses MongoDB to store location data. Players place tiles in realtime, and compete to build the longest chain of words and earn points. WordSquared leverages MongoDB's built-in geospatial indexing, storing an x and y coordinate with each tile and using bounding box queries to display a view of the board. This presentation covers the architecture of the game, with a specific focus on the use of MongoDB, storing and querying location data, and learning how to structure and even shard geo data through the unlikely use case of an infinitely large board game.

MongoDB .local Munich 2019: Best Practices for Working with IoT and Time-seri...

MongoDB

Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe. This talk covers: • Common components of an IoT solution • The challenges involved with managing time-series data in IoT applications • Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance. • How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

MongoDB .local Munich 2019: Still Haven't Found What You Are Looking For? Use...

MongoDB

Come and hear more about our new full-text search operator for MongoDB Atlas. This is a significant enhancement to MongoDB search features and is the easiest and most powerful full-text search solution for databases on MongoDB Atlas. This talk is important for anyone who has implemented search or is considering a search feature in their MongoDB application. You will see a demo of $searchBeta, learn about how it works, discover specific features to help you deliver relevant search results, and learn how you can start using full-text search in your application today.

Webinar: Building Your First App with MongoDB and Java

MongoDB

The document discusses building Java applications that use MongoDB as the database. It covers connecting to MongoDB from Java using the driver, designing schemas for embedded documents and arrays, building Java objects to represent and insert data, and performing basic operations like inserts. The document also mentions using an object-document mapper like Morphia to simplify interactions between Java objects and MongoDB documents.

Creating, Updating and Deleting Document in MongoDB

Wildan Maulana

This document discusses various methods for creating, updating, and deleting documents in MongoDB collections, including inserting documents, batch inserts, removing documents, updating documents using modifiers like $set and $inc, and choosing appropriate write speeds and safety levels. Key points covered include using batch inserts to speed up multiple document inserts, removing entire collections being faster than removing individual documents, and partial document updates being done efficiently using atomic update modifiers.

Running Production MongoDB Lightning Talk

chrisckchang

Back to Basics: My First MongoDB Application

MongoDB

- The document is a slide deck for a webinar on building a basic blogging application using MongoDB. - It covers MongoDB concepts like documents, collections and indexes. It then demonstrates how to install MongoDB, connect to it using the mongo shell, and insert documents. - The slide deck proceeds to model a basic blogging application using MongoDB, creating collections for users, articles and comments. It shows how to query, update, and import large amounts of seeded data.

Storing tree structures with MongoDB

Vyacheslav

In a real life almost any project deals with the tree structures. Different kinds of taxonomies, site structures etc require modeling of hierarchy relations. Typical approaches used ● Model Tree Structures with Child References ● Model Tree Structures with Parent References ● Model Tree Structures with an Array of Ancestors ● Model Tree Structures with Materialized Paths ● Model Tree Structures with Nested Sets

Going Schema-Free

spraints

Swift & JSON

John Sundell

MongoDB crud

Darshan Jayarama

MongoDB .local Paris 2020: La puissance du Pipeline d'Agrégation de MongoDB

MongoDB

Le pipeline d'agrégation a été en mesure d'alimenter votre analyse de données depuis la version 2.2. Dans la version 4.2, nous avons ajouté plus de puissance et vous pouvez maintenant l'utiliser pour des requêtes plus puissantes, des mises à jour et la sortie de vos données dans des collections existantes. Venez découvrir comment vous pouvez tout faire avec le pipeline, y compris les vues uniques, ETL, les cumuls de données et les vues matérialisées.

The rise of json in rdbms land jab17

alikonweb

This document discusses the rise of JSON usage in relational database management systems (RDBMS). It provides an overview of JSON and its benefits as a lightweight data format. It then examines different approaches to storing and querying JSON data in an RDBMS, including using JSON columns, entity-attribute-value modeling, and document-oriented modeling that treats documents as records. The document discusses pros and cons of each approach and emphasizes the importance of good data modeling and structure. It also shows examples of using JSON functions and indexing in MySQL.

An introduction to MongoDB

Universidade de São Paulo

Back to Basics Webinar 3: Schema Design Thinking in Documents

MongoDB

Sql vs no sql and azure data factory glasgow data UG

Diponkar Paul

Baton rouge - sql vs no sql and azure data factory

Diponkar Paul

What's hot

Optimizing Slow Queries with Indexes and Creativity

MongoDB

MongoDB .local Munich 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pip...

MongoDB

MongoDB .local Toronto 2019: Tips and Tricks for Effective Indexing

MongoDB

MongoDB .local Munich 2019: Tips and Tricks++ for Querying and Indexing MongoDB

MongoDB

MongoDB Indexing Constraints and Creative Schemas

MongoDB

Mapping Flatland: Using MongoDB for an MMO Crossword Game (GDC Online 2011)

Grant Goodale

MongoDB .local Munich 2019: Best Practices for Working with IoT and Time-seri...

MongoDB

MongoDB .local Munich 2019: Still Haven't Found What You Are Looking For? Use...

MongoDB

Webinar: Building Your First App with MongoDB and Java

MongoDB

Creating, Updating and Deleting Document in MongoDB

Wildan Maulana

Running Production MongoDB Lightning Talk

chrisckchang

Back to Basics: My First MongoDB Application

MongoDB

Storing tree structures with MongoDB

Going Schema-Free

Swift & JSON

MongoDB crud

MongoDB .local Paris 2020: La puissance du Pipeline d'Agrégation de MongoDB

MongoDB

The rise of json in rdbms land jab17

alikonweb

An introduction to MongoDB

Universidade de São Paulo

Back to Basics Webinar 3: Schema Design Thinking in Documents

MongoDB

What's hot (20)

Optimizing Slow Queries with Indexes and Creativity

MongoDB .local Munich 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pip...

MongoDB .local Toronto 2019: Tips and Tricks for Effective Indexing

MongoDB .local Munich 2019: Tips and Tricks++ for Querying and Indexing MongoDB

MongoDB Indexing Constraints and Creative Schemas

Mapping Flatland: Using MongoDB for an MMO Crossword Game (GDC Online 2011)

MongoDB .local Munich 2019: Best Practices for Working with IoT and Time-seri...

MongoDB .local Munich 2019: Still Haven't Found What You Are Looking For? Use...

Webinar: Building Your First App with MongoDB and Java

Creating, Updating and Deleting Document in MongoDB

Running Production MongoDB Lightning Talk

Back to Basics: My First MongoDB Application

Storing tree structures with MongoDB

Going Schema-Free

Swift & JSON

MongoDB crud

MongoDB .local Paris 2020: La puissance du Pipeline d'Agrégation de MongoDB

The rise of json in rdbms land jab17

An introduction to MongoDB

Back to Basics Webinar 3: Schema Design Thinking in Documents

Similar to Montreal Sql saturday: moving data from no sql db to azure data lake

Sql vs no sql and azure data factory glasgow data UG

Diponkar Paul

Baton rouge - sql vs no sql and azure data factory

Diponkar Paul

SQL vs. NoSQL and Moving data by Azure Data Factory

Diponkar Paul

Sql vs no sql and azure data factory

Diponkar Paul

GraphQL - when REST API is not enough - lessons learned

MarcinStachniuk

This document discusses lessons learned from implementing GraphQL APIs. It begins by describing some limitations of REST APIs, such as requiring multiple roundtrips to fetch nested data. GraphQL is introduced as an alternative that allows clients to specify exactly what data they need in a single request. The document then covers various GraphQL concepts like queries, mutations, and type systems. It also discusses best practices like using a schema-first approach, pagination support, and the DataLoader library to solve the N+1 problem. Testing GraphQL APIs and integrating with Relay are also briefly outlined. The overall message is that GraphQL is a good alternative to REST when clients have complex data needs.

The Rise of NoSQL

Arnd Kleinbeck

After a brief introduction into the history of Database Management Systems different types of NoSQL data stores are characterized. Theoretical background information about sharding mechanisms, horizontal scaling and the CAP theorem are getting explained. After a comparison of different NoSQL stores you will get to know the pros and cons of the different approaches and you will learn how to take the decision for the best fitting database in your project.

Why no sql

Anand Srinivasan

Eagle6 mongo dc revised

MongoDB

Eagle6 Enterprise Situational Awareness

MongoDB

Eagle6 is a product that use system artifacts to create a replica model that represents a near real-time view of system architecture. Eagle6 was built to collect system data (log files, application source code, etc.) and to link system behaviors in such a way that the user is able to quickly identify risks associated with unknown or unwanted behavioral events that may result in unknown impacts to seemingly unrelated down-stream systems. This session is designed to present the capabilities of the Eagle6 modeling product and how we are using MongoDB to support near-real-time analysis of large disparate datasets.

PostgreSQL Open SV 2018

artgillespie

This document describes a scalable, versioned document store built within PostgreSQL. It discusses the motivation for moving from multiple data stores and repositories to a single PostgreSQL database. It then covers the design of storing immutable content as Merkle DAG nodes linked by cryptographic hashes, with references and tags allowing different versions. It also explains how the system was implemented using PostgreSQL functions to generate hashes, insert nodes, and handle migrations from the original data model.

Modeling JSON data for NoSQL document databases

Ryan CrawCour

ER/Studio and DB PowerStudio Launch Webinar: Big Data, Big Models, Big News!

Embarcadero Technologies

MongoDB

techwhizbang

MongoDB is a non-relational database that uses a document-based data model. It is an alternative to traditional relational databases and is optimized for storing large amounts of unstructured and semi-structured data. MongoDB does not require a predefined schema and allows flexible, dynamic queries against documents using JavaScript. While relational databases are better suited for transactions, MongoDB is designed for horizontal scalability, faster queries, and flexible data modeling.

Sql vs no sql diponkar paul-april 2020-Toronto PASS

Diponkar Paul

Building your first app with mongo db

MongoDB

Dev Jumpstart: Build Your First App with MongoDB

MongoDB

This document provides an overview of MongoDB and how to build a simple blogging application with it. It describes MongoDB as a document database where documents are analogous to JSON objects and collections are analogous to tables in a relational database. It demonstrates how to install and run MongoDB, insert sample data for users and blog posts, query the data, and update documents. The document also lists various MongoDB drivers and provides pointers for next steps in learning about data modeling, replication, performance, and scaling in MongoDB.

How to leverage what's new in MongoDB 3.6

Maxime Beugnet

Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015

NoSQLmatters

During this live-coding session, Tugdual will move an old fashion full SQL application (JavaEE) to the new NoSQL world.Using MongoDB, and REST, he will show the benefits of this new architecture: * Easyness * Flexibility * High availability * Scalability; During this presentation, you will learn more about: * Document Oriented Model * JSON * REST * Iterative development; This demonstration is also a good opportunity to see how you can migrate data from a relational database, and the various schema options.

[OSC 2020 Online/Nagoya] MySQLドキュメントストア

Ryusuke Kajiyama

This document discusses MySQL's support for both SQL and NoSQL through its MySQL Document Store feature. It provides an overview of how MySQL 8.0 allows users to store JSON documents and access them through either SQL queries or NoSQL CRUD APIs. This hybrid model provides the flexibility of NoSQL with the reliability of SQL in a single database, meeting the needs of modern applications that require both relational and non-relational data storage and access.

SlamData - How MongoDB Is Powering a Revolution in Visual Analytics

John De Goes

Similar to Montreal Sql saturday: moving data from no sql db to azure data lake (20)

Sql vs no sql and azure data factory glasgow data UG

Baton rouge - sql vs no sql and azure data factory

SQL vs. NoSQL and Moving data by Azure Data Factory

Sql vs no sql and azure data factory

GraphQL - when REST API is not enough - lessons learned

The Rise of NoSQL

Why no sql

Eagle6 mongo dc revised

Eagle6 Enterprise Situational Awareness

PostgreSQL Open SV 2018

Modeling JSON data for NoSQL document databases

ER/Studio and DB PowerStudio Launch Webinar: Big Data, Big Models, Big News!

MongoDB

Sql vs no sql diponkar paul-april 2020-Toronto PASS

Building your first app with mongo db

Dev Jumpstart: Build Your First App with MongoDB

How to leverage what's new in MongoDB 3.6

Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015

[OSC 2020 Online/Nagoya] MySQLドキュメントストア

SlamData - How MongoDB Is Powering a Revolution in Visual Analytics

Recently uploaded

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/ The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this! We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model. Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward. These topics will be covered - Reducing license cost by finding and fixing misconfigurations and superfluous accounts - How do CCB and CCX licenses really work? - Understanding the DLAU tool and how to best utilize it - Tips for common problem areas, like team mailboxes, functional/test users, etc - Practical examples and best practices to implement right away

Mariano G Tinti - Decoding SpaceX

Mariano Tinti

Pushing the limits of ePRTC: 100ns holdover for 100 days

Adtran

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

GenAI Pilot Implementation in the organizations

kumardaparthi1024

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Paige Cruz

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack. I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:

Infrastructure Challenges in Scaling RAG with Custom AI models

Zilliz

Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.

Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!

SOFTTECHHUB

As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.

Serial Arm Control in Real Time Presentation

tolgahangng

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Safe Software

Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency. During the hour, we’ll take you through: Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board. Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes. Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI. We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI. This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

SOFTTECHHUB

The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing. One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Neo4j

Dr. Sean Tan, Head of Data Science, Changi Airport Group Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

“I’m still / I’m still / Chaining from the Block”

Claudio Di Ciccio

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

UiPath Test Automation using UiPath Test Suite series, part 5

DianaGray10

Building Production Ready Search Pipelines with Spark and Milvus

Zilliz

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

Neo4j

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

Neo4j

UiPath Test Automation using UiPath Test Suite series, part 6

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI. UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities. Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes. What will you get from this session? 1. Insights into integrating generative AI. 2. Understanding how this integration enhances test automation within the UiPath platform 3. Practical demonstrations 4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath Topics covered: What is generative AI Test Automation with generative AI and Open AI. UiPath integration with generative AI Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

Recently uploaded (20)

HCL Notes and Domino License Cost Reduction in the World of DLAU

Mariano G Tinti - Decoding SpaceX

Pushing the limits of ePRTC: 100ns holdover for 100 days

Climate Impact of Software Testing at Nordic Testing Days

GenAI Pilot Implementation in the organizations

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Infrastructure Challenges in Scaling RAG with Custom AI models

Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!

Serial Arm Control in Real Time Presentation

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Essentials of Automations: The Art of Triggers and Actions in FME

“I’m still / I’m still / Chaining from the Block”

Removing Uninteresting Bytes in Software Fuzzing

UiPath Test Automation using UiPath Test Suite series, part 5

Building Production Ready Search Pipelines with Spark and Milvus

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

UiPath Test Automation using UiPath Test Suite series, part 6

Montreal Sql saturday: moving data from no sql db to azure data lake

1. SQL vs NoSQL and moving data from MongoDB to Azure data lake by using Azure Data Factory Diponkar Paul

2. Thanks to our sponsors And Global Microsoft RedHat Google Cloud

3. Title Content Father and Husband Blogger & Speaker Data Engineer Diverse background Community Twitter: @Paulswengrr Blog: www.allaboutdata.ca

4. What we cover Refresh our memory with traditional SQL Know about NoSQL (MongoDB) Demo: No SQL Comparison Azure data factory: Copy data from MongoDB Demo: MongoDB with ADF

5. SQL

6. SQL Syntax SELECT Id, Product, Price From Product Where ProductCategory=’Bikes’ Join, Insert, Update, Delete

7. Well defined Schema CREATE TABLE [Production].[Product]( [ProductID] [int] IDENTITY(1,1) NOT NULL, [Name] [nvarchar](100) NOT NULL, [ProductNumber] [nvarchar](25) NOT NULL, [MakeFlag] [dbo].[Flag] NOT NULL, [FinishedGoodsFlag] [dbo].[Flag] NOT NULL, [Color] [nvarchar](15) NULL, [SafetyStockLevel] [smallint] NOT NULL, [StandardCost] [money] NOT NULL, [ListPrice] [money] NOT NULL, [Size] [nvarchar](5) NULL)

8. Relationship/Normalization Customer Bridge table (Order) Product Id Name Price Description 1 “Mountain Bike “ 2500 “Bike for mountain trek” 2 “City Bike” 1000 “Best fit to roam around city” Id Customer_ID Product_ID 1 2 1 2 2 2 3 1 1 Id Name Email 1 Morten Sorenson m.s@outlook.com 2 Andersen Lu al@yahoo.com 3 Derek Paul dp@outlook.com

9. Type of relationship

10. NoSQL

11. Vendors in the market • MongoDB • Azure Cosmos DB • Amazon Document DB • Oracle NoSQL • Google BigTable

12. NoSQL-MongoDB

13. How we call them? Database E-Commerce Collections Table –Customer, Product… Documents {“Name”: ”Anders”, age:36} {“Name”: “Carsten”, age:42}

14. No defined Schema Id:1 Age:36Name: ‘Anders’ ….. Id:2 Age:36 Name: ‘Carsten’ ….. Id:3 ….. Age:36 ….. Age:36 Name: ‘Carsten’ ….. ….. Id:1 Name: ‘Anders’ Id:2 Age:36 ….. Name: ‘Carsten’ …..

15. NoSQL –No relation Profession {id:1,profession:’Developer’} {id:2, profession: ’Data Engineer’} {id:3, profession: ’Actor’} Users {id:1,name:’Tom Hanks’, age:20} {id:2,name:’Casper Ruther’, age:42} {id:3,name:’Paul Anders’, age:63} db.Users.insert( { id:"01", name:"Tom Hanks", age:20 email:"th@hollywood.com", Profession:["Developer","Data Engineer","Actor"] } ) Usersprofession {id:1,userId:1,professionId:1} {id:2,userId:1, professionId: 2} {id:3,userId: 1, professionId: 3} {id:4,userId: 2, professionId: 2}

16. Tools: MongoDB https://www.mongodb.com/products/compass Robo 3T: https://robomongo.org/ https://docs.mongodb.com/manual/core/data-model-design/ https://docs.mongodb.com/manual/reference/method/db.collection.update/

17. Languages • MONGO SHELL • Python • java • C# • Scala • GO and many more.

18. Demo

19. SQL vs NoSQL SQL NoSQL Data uses Schema Schema-less (Schema Agnostic) Maintain Relationship No relations– though you can design relationship Data distributed in multiple tables Data in one table (embedded) Monolithic, you can easily Scale-Up. Scale out is also possible but difficult (e.g. Azure Elastic Database tools) Scale up and scale out- Globally distributed

20. Move your NoSQL data from OnPrem to Data Lake Gen2

21. Azure Data Lake Azure Data Lake is a scalable data storage and analytics service -Fully HDFS compliance file system -Azure AD integrated -Microsoft’s PAAS service big data solution

22. Azure Data Factory • ETL/ELT Tool • Code free • Azure Cloud • a lot more…

23. Pre-requisite • azure account • Azure data factory resource • Linked services (Source and target connection) • Integration run time Integration Runtime Linked Service

24. Demo-ADF

25. Be cautious! • MongoDB version supported for ADF copy activity (V 3.4) *https://docs.microsoft.com/en-us/azure/data-factory/connector-mongodb

26. Key Takeaways: • SQL vs. NoSQL • Choose SQL/ NoSQL • Integration from OnPrem MongoDB • Starting new NoSQL?- choose cloud solution Azure CosmosDB or Atlas

27. Questions @paulswengrr Diponkarpaul

Montreal Sql saturday: moving data from no sql db to azure data lake

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Montreal Sql saturday: moving data from no sql db to azure data lake

Similar to Montreal Sql saturday: moving data from no sql db to azure data lake (20)

Recently uploaded

Recently uploaded (20)

Montreal Sql saturday: moving data from no sql db to azure data lake