See the latest features of SSIS in ADF. We will show you how to join your Azure-SSIS Integration Runtime (IR) to an ARM VNet, so you can use Azure SQL Managed Instance to host your SSISDB and access data on premises. You will learn how to select Enterprise Edition for your IR, enabling you to use advanced/premium features, e.g. Oracle/Teradata/SAP BW connectors, CDC components, Fuzzy Grouping/Lookup transformations, etc. You will also learn how to customize your IR via a custom setup interface to modify system configurations/install additional components, e.g. (un)licensed 3rd party/Open Source extensions, assemblies, drivers, tools, APIs, etc. Finally, we will show you how to trigger/schedule/orchestrate SSIS package executions as first-class activities in ADF pipelines.
Embrace and Extend - First-Class Activity and 3rd Party Ecosystem for SSIS in...Sandy Winarko
This session focuses on the deeper integration of SQL Server Integration Services (SSIS) in Azure Data Factory (ADF) and the broad extensibility of Azure-SSIS Integration Runtime (IR). We will first show you how to provision Azure-SSIS IR – dedicated ADF servers for lifting & shifting SSIS packages – and extend it with custom/3rd party components. Preserving your skillsets, you can then use the familiar SQL Server Data Tools (SSDT)/SQL Server Management Studio (SSMS) to design/deploy/configure/execute/monitor your SSIS packages in the cloud just like you do on premises. Next, we will guide you to trigger/schedule SSIS package executions as first-class activities in ADF pipelines and combine/chain them with other activities, allowing you to inject/splice built-in/custom/3rd party tasks/data transformations in your ETL/ELT workflows, automatically provision Azure-SSIS IR on demand/just in time, etc. And finally, you will learn about the licensing model for ISVs to develop paid components/extensions and join the growing 3rd party ecosystem for SSIS in ADF with a few examples from our partners.
PaaSport to Paradise: Back to the Future with SSIS in Azure Data FactorySandy Winarko
This session focuses on the all Platform as a Service (PaaS) solution of Azure SQL Database (DB)/Managed Instance (MI) + SQL Server Integration Services (SSIS) in Azure Data Factory (ADF) to lift & shift, modernize, and extend Extract, Transform & Load (ETL) workflows. We will first show you how to provision Azure-SSIS Integration Runtime (IR) – dedicated ADF servers for running SSIS – with SSIS catalog (SSISDB) hosted by Azure SQL DB/MI, configure it to access data on premises using Windows authentication and Virtual Network (VNet) injection/Self-Hosted IR as a proxy, and extend it with custom/Open Source/3rd party components. We will next show you how to use the familiar SQL Server Data Tools (SSDT)/SQL Server Management Studio (SSMS) to design/debug/deploy/configure/execute your SSIS packages in the cloud just like you do on premises. We will then show you how to modernize your ETL workflows by invoking/scheduling SSIS package executions as first-class activities in ADF pipelines and combining/chaining them with other activities, allowing you to trigger your pipeline runs by events, automatically (de)provision SSIS IR just in time, etc. We will also show you how to enable the following scenarios: assessing/testing SSIS packages in ADF from SSDT, running SSIS packages stored in file system/Azure Files/SQL Server database (MSDB) hosted by Azure SQL MI, accessing Azure SQL DB/MI with OLEDB connectors that are configured for Azure Active Directory (AAD) authentication with ADF managed identity, scheduling SSIS jobs in ADF from SQL Server Agent/3rd party orchestrators that invoke Azure-enabled dtexec.exe, etc. We will finally show you the complete SSIS Migration Playbook to perform package assessments and package/job migrations in batches using Data(base) Migration Assistant/Service, dtutil.exe, scripts, etc.
PaaSport to Paradise - Azure SQL and SSIS in Azure Data Factory - Better Toge...Sandy Winarko
Learn about enablers/features that can unblock and accelerate legacy SSIS migrations into ADF with no/minimal changes to existing packages and tools, e.g. Azure-enabled SSDT and SSMS, Package Deployment Model support, SSIS Integration Runtime (IR) package store, dtutil command prompt utility, Azure SQL Managed Instance (MI) Agent, SSIS scheduling feature and SSIS Job Migration Wizard on SSMS, Azure-enabled DTExec (AzureDTExec) command prompt utility, Virtual Network (VNet) injection of SSIS IR, Self-Hosted IR (SHIR) as a proxy for SSIS IR via ConnectByProxy property, Windows authentication feature, Azure Key Vault (AKV) integration, Azure Active Directory (AAD) authentication with ADF managed identity via ConnectUsingManagedIdentity property and OLEDB driver for SQL Server (MSOLEDBSQL), Azure Monitor integration, etc.
See how to use the latest SSIS 2017 to modernize your traditional on-premises ETL workflows and transform them into scalable hybrid ETL/ELT workflows.
First, we will take a deep dive into SSIS Scale-Out feature, guiding you end-to-end from cluster installations to parallel executions on premises/IaaS, to help shorten the overall duration of your workflows. Next, we will guide you to lift & shift your ETL workloads into SSIS PaaS in ADFv2. Finally, we will show you how to trigger/schedule/orchestrate SSIS package executions in ADF pipelines.
Make Web, Not War - Installfest: Extend Your Web Server, Rodney BuikeMake Web Not War
Install Windows Server 2008 R2 RC (valid up to one year) and find out what's new in IIS and the Microsoft Web Platform. Then walk through a variety of Demos including ASP.NET & PHP on Server Core, Remote Management of Server Core, Windows PowerShell Web Administration on Server Core & Web Deployment Tool.
Best Practices for Running Microsoft SQL Server on AWSGianluca Hotz
The document discusses best practices for running Microsoft SQL Server on AWS. It provides an overview of options for deploying SQL Server on AWS, including using Amazon RDS or Amazon EC2. When using RDS, AWS manages the SQL Server instance and provides features like automated backups and read replicas. When using EC2, the user has more control but must manage SQL Server, backups, and high availability. The document discusses considerations and techniques for optimizing SQL Server performance on EC2, including storage options and configuration.
AWS has a number of services that help enterprise customers deploy solutions that meet high performance, security, and reliability requirements. SQL Server is no exception. In this session, we will explore the different options that exist today to help enterprises meet those types of requirements. Another key capability in AWS is flexibility. Multiple options exist for how enterprises can deploy SQL Server in AWS. We will talk in detail about how to choose between a managed database model like Relational Database Service (RDS) or core compute model like Elastic Compute Cloud (EC2). Finally, we’ll wrap up with an exploration of different operational aspects of SQL Server in AWS.
AWS Webcast - Implementing Windows and SQL Server for High Availability on AWSAmazon Web Services
• Deploy the virtual network infrastructure on multiple subnets
• Launch Amazon Machine Images (AMIs) of Windows Server 2008 R2
• Set up Active Directory and DNS • Launch and configure the WSFC nodes
• Create a SQL Server AlwaysOn Availability Group Who should attend:
• IT Executives, Developers, DBAs, Solutions Architects, Systems Engineers
• Network and Database Professionals and Leaders
Embrace and Extend - First-Class Activity and 3rd Party Ecosystem for SSIS in...Sandy Winarko
This session focuses on the deeper integration of SQL Server Integration Services (SSIS) in Azure Data Factory (ADF) and the broad extensibility of Azure-SSIS Integration Runtime (IR). We will first show you how to provision Azure-SSIS IR – dedicated ADF servers for lifting & shifting SSIS packages – and extend it with custom/3rd party components. Preserving your skillsets, you can then use the familiar SQL Server Data Tools (SSDT)/SQL Server Management Studio (SSMS) to design/deploy/configure/execute/monitor your SSIS packages in the cloud just like you do on premises. Next, we will guide you to trigger/schedule SSIS package executions as first-class activities in ADF pipelines and combine/chain them with other activities, allowing you to inject/splice built-in/custom/3rd party tasks/data transformations in your ETL/ELT workflows, automatically provision Azure-SSIS IR on demand/just in time, etc. And finally, you will learn about the licensing model for ISVs to develop paid components/extensions and join the growing 3rd party ecosystem for SSIS in ADF with a few examples from our partners.
PaaSport to Paradise: Back to the Future with SSIS in Azure Data FactorySandy Winarko
This session focuses on the all Platform as a Service (PaaS) solution of Azure SQL Database (DB)/Managed Instance (MI) + SQL Server Integration Services (SSIS) in Azure Data Factory (ADF) to lift & shift, modernize, and extend Extract, Transform & Load (ETL) workflows. We will first show you how to provision Azure-SSIS Integration Runtime (IR) – dedicated ADF servers for running SSIS – with SSIS catalog (SSISDB) hosted by Azure SQL DB/MI, configure it to access data on premises using Windows authentication and Virtual Network (VNet) injection/Self-Hosted IR as a proxy, and extend it with custom/Open Source/3rd party components. We will next show you how to use the familiar SQL Server Data Tools (SSDT)/SQL Server Management Studio (SSMS) to design/debug/deploy/configure/execute your SSIS packages in the cloud just like you do on premises. We will then show you how to modernize your ETL workflows by invoking/scheduling SSIS package executions as first-class activities in ADF pipelines and combining/chaining them with other activities, allowing you to trigger your pipeline runs by events, automatically (de)provision SSIS IR just in time, etc. We will also show you how to enable the following scenarios: assessing/testing SSIS packages in ADF from SSDT, running SSIS packages stored in file system/Azure Files/SQL Server database (MSDB) hosted by Azure SQL MI, accessing Azure SQL DB/MI with OLEDB connectors that are configured for Azure Active Directory (AAD) authentication with ADF managed identity, scheduling SSIS jobs in ADF from SQL Server Agent/3rd party orchestrators that invoke Azure-enabled dtexec.exe, etc. We will finally show you the complete SSIS Migration Playbook to perform package assessments and package/job migrations in batches using Data(base) Migration Assistant/Service, dtutil.exe, scripts, etc.
PaaSport to Paradise - Azure SQL and SSIS in Azure Data Factory - Better Toge...Sandy Winarko
Learn about enablers/features that can unblock and accelerate legacy SSIS migrations into ADF with no/minimal changes to existing packages and tools, e.g. Azure-enabled SSDT and SSMS, Package Deployment Model support, SSIS Integration Runtime (IR) package store, dtutil command prompt utility, Azure SQL Managed Instance (MI) Agent, SSIS scheduling feature and SSIS Job Migration Wizard on SSMS, Azure-enabled DTExec (AzureDTExec) command prompt utility, Virtual Network (VNet) injection of SSIS IR, Self-Hosted IR (SHIR) as a proxy for SSIS IR via ConnectByProxy property, Windows authentication feature, Azure Key Vault (AKV) integration, Azure Active Directory (AAD) authentication with ADF managed identity via ConnectUsingManagedIdentity property and OLEDB driver for SQL Server (MSOLEDBSQL), Azure Monitor integration, etc.
See how to use the latest SSIS 2017 to modernize your traditional on-premises ETL workflows and transform them into scalable hybrid ETL/ELT workflows.
First, we will take a deep dive into SSIS Scale-Out feature, guiding you end-to-end from cluster installations to parallel executions on premises/IaaS, to help shorten the overall duration of your workflows. Next, we will guide you to lift & shift your ETL workloads into SSIS PaaS in ADFv2. Finally, we will show you how to trigger/schedule/orchestrate SSIS package executions in ADF pipelines.
Make Web, Not War - Installfest: Extend Your Web Server, Rodney BuikeMake Web Not War
Install Windows Server 2008 R2 RC (valid up to one year) and find out what's new in IIS and the Microsoft Web Platform. Then walk through a variety of Demos including ASP.NET & PHP on Server Core, Remote Management of Server Core, Windows PowerShell Web Administration on Server Core & Web Deployment Tool.
Best Practices for Running Microsoft SQL Server on AWSGianluca Hotz
The document discusses best practices for running Microsoft SQL Server on AWS. It provides an overview of options for deploying SQL Server on AWS, including using Amazon RDS or Amazon EC2. When using RDS, AWS manages the SQL Server instance and provides features like automated backups and read replicas. When using EC2, the user has more control but must manage SQL Server, backups, and high availability. The document discusses considerations and techniques for optimizing SQL Server performance on EC2, including storage options and configuration.
AWS has a number of services that help enterprise customers deploy solutions that meet high performance, security, and reliability requirements. SQL Server is no exception. In this session, we will explore the different options that exist today to help enterprises meet those types of requirements. Another key capability in AWS is flexibility. Multiple options exist for how enterprises can deploy SQL Server in AWS. We will talk in detail about how to choose between a managed database model like Relational Database Service (RDS) or core compute model like Elastic Compute Cloud (EC2). Finally, we’ll wrap up with an exploration of different operational aspects of SQL Server in AWS.
AWS Webcast - Implementing Windows and SQL Server for High Availability on AWSAmazon Web Services
• Deploy the virtual network infrastructure on multiple subnets
• Launch Amazon Machine Images (AMIs) of Windows Server 2008 R2
• Set up Active Directory and DNS • Launch and configure the WSFC nodes
• Create a SQL Server AlwaysOn Availability Group Who should attend:
• IT Executives, Developers, DBAs, Solutions Architects, Systems Engineers
• Network and Database Professionals and Leaders
Help, I need to migrate my On Premise Database to Azure, which Database Tier ...Erwin de Kreuk
Azure SQL Database provides several deployment options including single databases and elastic pools. The single database option provides resource guarantees at the database level while elastic pools allow for sharing of resources across multiple databases for better cost efficiency. Azure SQL Database offers different service tiers including Basic, Standard, and Premium that provide different performance levels and features. Customers can choose between DTU-based and vCore-based purchasing models, with vCores offering more flexibility and control over compute and storage. The Data Migration Assistant and Data Migration Service can help customers assess, plan, and execute migrations of databases to Azure SQL Database.
Amazon Aurora is a MySQL-compatible database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. This session introduces you to Amazon Aurora, explains common use cases for the service, and helps you get started with building your first Amazon Aurora–powered application.
The document provides an overview of AWS Database Migration Service (AWS DMS). It explains that AWS DMS allows users to easily and securely migrate or replicate databases to AWS. It describes how to use AWS DMS by creating a replication instance, specifying source and target endpoints, and then creating a migration task to transfer data from the source to target. Key aspects of the replication instance, endpoints, and tasks are also defined.
Implement a disaster recovery solution for your on-prem SQL with Azure? Easy!Marco Obinu
Slides presented at SQL Saturday 980 Plovdiv, talking about the different architectures you can implement to protect your on-premises SQL Server workloads on Azure for DR purposes.
An Amazon EC2 Dedicated Host is a physical server with EC2 instance capacity fully dedicated to your use. Dedicated Hosts can help you address compliance requirements and reduce costs by allowing you to use your existing server-bound software licenses.
This webinar will provide you an introduction to Dedicated Hosts, including a demonstration of how to bring your own Windows software and reviewing best practices. The webinar will be focused on Windows BYOL (e.g., Windows Server, SQL Server, etc)
Learning Objectives:
Understand key features of Dedicated Hosts, including billing and instance placement controls
Learn how to create a Dedicated Host and provision instances on it
Learn how to bring your own software into EC2 and provision instances of your software on Dedicated Hosts
Learn best practices for managing BYOL instances on Dedicated Hosts
Who Should Attend:
Customers who want to use their own server-bound licenses (e.g., Windows Server, SQL Server, SUSE Enterprise Linux Server) in EC2
Customers who want to learn more about Dedicated Hosts
Customers with a compliance or regulatory use case that requires the use of physical servers fully dedicated to their use. (e.g., customers with a life sciences use case (HIPAA, GxP)
AWS offers customers a range of different database options. These include Amazon DynamoDB, a fully-managed NoSQL database service that makes it simple and cost-effective to store and retrieve any amount of data as well as Amazon Relational Database Service (RDS), a service that makes it easy to set up, operate, and scale a relational database in the cloud with support for MySQL, Microsoft SQL Server, PostgreSQL, and Oracle Database. In this session you’ll get an overview of AWS database options and how they might help support your application and see how to get started.
This document summarizes Amazon Web Services database migration and replication services. It discusses how the AWS Database Migration Service can migrate databases between on-premises and cloud environments within 10 minutes with no application downtime. It also describes how the AWS Schema Conversion Tool can help migrate databases off Oracle and SQL Server to other database engines like MySQL. Finally, it provides an overview of Amazon RDS managed database services and high availability features.
Amazon Web Services (AWS) can make hosting scalable, highly-available websites and web applications easier and less expensive for the Enterprise Education customers. Join us for an informative webinar on tools AWS provides to elastically scale your architecture to avoid underutilized resources while reducing complexity with templates, partners, and tools to do much of the heavy lifting of creating and running a website for you.
This document contains the resume of Suman Chandra Jha, who has 12 years of experience as a Senior IT Infrastructure Solution Architect and Freelancer. He has various certifications in cloud computing from Amazon, Microsoft, and VMware, as well as certifications in data center design. He currently works as a Senior IT Infrastructure Solution Architect for the UAE Government and also works as a freelancer providing corporate training on Amazon and Microsoft certification courses.
ENT313 Deploying a Disaster Recovery Site on AWS: Minimal Cost with Maximum E...Amazon Web Services
In the event of a disaster, you need to be able to recover lost data quickly to ensure business continuity. For critical applications, keeping your time to recover and data loss to a minimum as well as optimizing your overall capital expense can be challenging. This session presents AWS features and services along with Disaster Recovery architectures that you can leverage when building highly available and disaster resilient applications. We will provide recommendations on how to improve your Disaster Recovery plan and discuss example scenarios showing how to recover from a disaster.
AWS re:Invent 2016: Bring Microsoft Applications to AWS to Save Money and Sta...Amazon Web Services
Running Microsoft workloads on AWS is easy and can save you money. This session will cover how to bring your own Microsoft licenses to AWS, and then demonstrate using PowerShell to import your Windows Server image from Vmware or Hyper-V, configure Windows KMS with your license key, and launch an EC2 Dedicated Host. We will discuss ways you can use AWS Config rules to manage license compliance.
This document provides an overview of Azure SQL database and related services including:
- Azure SQL Database which provides single database and elastic pool models for predictable or shared workloads.
- Azure SQL Managed Instance which provides high compatibility with SQL Server in a PaaS model.
- Related Azure data and analytics services for ingestion, storage, preparation, modeling and serving of data.
- Key capabilities of Azure SQL Database around data migration, programmability, security and operations.
Database performance is a key factor for keeping your app running fast. As the amount of data and request throughput grow, it becomes harder to keep it nimble and performant. In this webinar, we will discuss using Redis, a popular NoSQL key-value store, to run your data fast at scale. Redis can be used as a cache in front of a database or as a fast in-memory data store. We will also cover how you can use ElastiCache for Redis to easily set up a large multi-terabyte Redis environment and operate it with zero management.
Learning Objectives:
• Understand fast data
• Understand the capabilities of Amazon Elasticache for Redis to cache Relational and NoSQL databases
• Use Amazon Elasticache for Redis as a primary data store
Who Should Attend:
• Developers, Data Architects
Cloud for Developers: Azure vs. Google App Engine vs. Amazon vs. AppHarborSvetlin Nakov
Software Development for the Public Cloud Platforms: Windows Azure vs. Google App Engine vs. Amazon Web Services (AWS) vs AppHarbor.
In this talk the speaker will compare the most widely used public PaaS clouds (Azure, GAE and AWS) from the software developer’s perspective.
A parallel between Azure, GAE, AWS and few other clouds (like AppHarbor, Heroku, Cloudfoundry and AppForce) will be made based on several criteria: architecture, pricing, storage services (non-relational databases, relational databases in the cloud and blob/file storage), business-tier services (like queues, notifications, email, CDN, etc.), supported languages, platforms and frameworks and front-end technologies.
A live demo will be made to compare the way we build and deploy a multi-tiered application in Azure, Amazon and GAE and how to implement its back-end (using a cloud database), business tier (based on REST services) and front-end (based on HTML5).
The speaker Svetlin Nakov (http://www.nakov.com) is well-known software development expert and trainer, a head of the Telerik Software Academy and a main organizer of the Cloud Development course (http://clouddevcourse.telerik.com).
Spark 101 – First Steps To Distributed Computing - Demi Ben-Ari @ Ofek AlumniDemi Ben-Ari
The world has changed and having one huge server won’t do the job anymore, when you’re talking about vast amounts of data, growing all the time the ability to Scale Out would be your saviour.
This lecture will be about the basics of Apache Spark and distributed computing and the development tools needed to have a functional environment.
Bio:
Demi Ben-Ari, Sr. Data Engineer @Windward, Ofek Alumni
Has over 9 years of experience in building various systems both from the field of near real time applications and Big Data distributed systems.
Co-Founder of the “Big Things” Big Data community: http://somebigthings.com/big-things-i...
One of the most fundamental challenges of CI/CD is the ability to balance between Quality, Time, and Cost. Amazon EC2 Container Service (ECS), along with Docker and Amazon EC2 Container Registry (ECR), has changed the game for many by making resource management very simple. For Okta, it has enabled the Continuous Integration team to maximize throughput while minimizing cost. In this session we will show you how Okta has created a flexible CI system with ECS, Docker, ECR, AWS Lambda, AWS CloudFormation, Amazon RDS, and Amazon SQS. Okta runs 30,000 tests with each developer commit, and releases 10,000 new lines of code each week to production. The CI system, built 100% on AWS, must be able to handle load while keeping cost under control. This talk is oriented toward developers looking to achieve efficient resource and cost management without compromising speed or quality.
AWS re:Invent 2016: How to Launch a 100K-User Corporate Back Office with Micr...Amazon Web Services
Learn how to build a scalable, compliance-ready, and automated deployment of the Microsoft “backoffice” servers for 100K users running on AWS. In this session, we show a reference architecture deployment of Exchange, SharePoint, Skype for Business, SQL Server and Active Directory in a single VPC. We discuss the following: (1) how the solution is automated for 100K users, (2) how the solution is enabled for compliance (e.g., FedRAMP, HIPAA, PCI), and (3) how the solution is built from modular 10K user blocks. Attendees should have knowledge of AWS CloudFormation, PowerShell, instance bootstrapping, VPCs, and Amazon Route 53, as well as the relevant Microsoft technologies.
In our first Windows webinar, find out about the benefits of migrating your Windows workloads to AWS. During the session, we will explain why AWS makes your Windows applications faster, more reliable and more secure. He will also talk about how to bring your own license (BYOL), how to architect, deploy, and manage your Windows platforms on AWS.
New capabilities for modern data integration in the cloudGaurav Malhotra
The document describes how Azure Data Factory can be used to orchestrate data pipelines and execute SSIS packages in the cloud. It provides an overview of key capabilities like lifting and shifting existing SSIS packages to Azure, using various Azure services like SQL Data Warehouse and Databricks in data pipelines, and orchestrating workflows. It also discusses how Azure Data Factory allows scheduling SSIS package execution in a managed cloud environment using familiar SSIS authoring and monitoring tools.
This document provides an overview of how to configure an Azure-SSIS Integration Runtime (IR) within Azure Data Factory. It describes the steps to name and describe the IR, select its location, node size, cluster size, edition/license, and connect it to an existing Azure SQL Database or Managed Instance that will host the SSIS catalog database. It also covers options for using Azure Active Directory authentication or bringing your own Azure SQL credentials. The document guides users through the entire configuration process for an Azure-SSIS IR within Azure Data Factory.
Help, I need to migrate my On Premise Database to Azure, which Database Tier ...Erwin de Kreuk
Azure SQL Database provides several deployment options including single databases and elastic pools. The single database option provides resource guarantees at the database level while elastic pools allow for sharing of resources across multiple databases for better cost efficiency. Azure SQL Database offers different service tiers including Basic, Standard, and Premium that provide different performance levels and features. Customers can choose between DTU-based and vCore-based purchasing models, with vCores offering more flexibility and control over compute and storage. The Data Migration Assistant and Data Migration Service can help customers assess, plan, and execute migrations of databases to Azure SQL Database.
Amazon Aurora is a MySQL-compatible database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. This session introduces you to Amazon Aurora, explains common use cases for the service, and helps you get started with building your first Amazon Aurora–powered application.
The document provides an overview of AWS Database Migration Service (AWS DMS). It explains that AWS DMS allows users to easily and securely migrate or replicate databases to AWS. It describes how to use AWS DMS by creating a replication instance, specifying source and target endpoints, and then creating a migration task to transfer data from the source to target. Key aspects of the replication instance, endpoints, and tasks are also defined.
Implement a disaster recovery solution for your on-prem SQL with Azure? Easy!Marco Obinu
Slides presented at SQL Saturday 980 Plovdiv, talking about the different architectures you can implement to protect your on-premises SQL Server workloads on Azure for DR purposes.
An Amazon EC2 Dedicated Host is a physical server with EC2 instance capacity fully dedicated to your use. Dedicated Hosts can help you address compliance requirements and reduce costs by allowing you to use your existing server-bound software licenses.
This webinar will provide you an introduction to Dedicated Hosts, including a demonstration of how to bring your own Windows software and reviewing best practices. The webinar will be focused on Windows BYOL (e.g., Windows Server, SQL Server, etc)
Learning Objectives:
Understand key features of Dedicated Hosts, including billing and instance placement controls
Learn how to create a Dedicated Host and provision instances on it
Learn how to bring your own software into EC2 and provision instances of your software on Dedicated Hosts
Learn best practices for managing BYOL instances on Dedicated Hosts
Who Should Attend:
Customers who want to use their own server-bound licenses (e.g., Windows Server, SQL Server, SUSE Enterprise Linux Server) in EC2
Customers who want to learn more about Dedicated Hosts
Customers with a compliance or regulatory use case that requires the use of physical servers fully dedicated to their use. (e.g., customers with a life sciences use case (HIPAA, GxP)
AWS offers customers a range of different database options. These include Amazon DynamoDB, a fully-managed NoSQL database service that makes it simple and cost-effective to store and retrieve any amount of data as well as Amazon Relational Database Service (RDS), a service that makes it easy to set up, operate, and scale a relational database in the cloud with support for MySQL, Microsoft SQL Server, PostgreSQL, and Oracle Database. In this session you’ll get an overview of AWS database options and how they might help support your application and see how to get started.
This document summarizes Amazon Web Services database migration and replication services. It discusses how the AWS Database Migration Service can migrate databases between on-premises and cloud environments within 10 minutes with no application downtime. It also describes how the AWS Schema Conversion Tool can help migrate databases off Oracle and SQL Server to other database engines like MySQL. Finally, it provides an overview of Amazon RDS managed database services and high availability features.
Amazon Web Services (AWS) can make hosting scalable, highly-available websites and web applications easier and less expensive for the Enterprise Education customers. Join us for an informative webinar on tools AWS provides to elastically scale your architecture to avoid underutilized resources while reducing complexity with templates, partners, and tools to do much of the heavy lifting of creating and running a website for you.
This document contains the resume of Suman Chandra Jha, who has 12 years of experience as a Senior IT Infrastructure Solution Architect and Freelancer. He has various certifications in cloud computing from Amazon, Microsoft, and VMware, as well as certifications in data center design. He currently works as a Senior IT Infrastructure Solution Architect for the UAE Government and also works as a freelancer providing corporate training on Amazon and Microsoft certification courses.
ENT313 Deploying a Disaster Recovery Site on AWS: Minimal Cost with Maximum E...Amazon Web Services
In the event of a disaster, you need to be able to recover lost data quickly to ensure business continuity. For critical applications, keeping your time to recover and data loss to a minimum as well as optimizing your overall capital expense can be challenging. This session presents AWS features and services along with Disaster Recovery architectures that you can leverage when building highly available and disaster resilient applications. We will provide recommendations on how to improve your Disaster Recovery plan and discuss example scenarios showing how to recover from a disaster.
AWS re:Invent 2016: Bring Microsoft Applications to AWS to Save Money and Sta...Amazon Web Services
Running Microsoft workloads on AWS is easy and can save you money. This session will cover how to bring your own Microsoft licenses to AWS, and then demonstrate using PowerShell to import your Windows Server image from Vmware or Hyper-V, configure Windows KMS with your license key, and launch an EC2 Dedicated Host. We will discuss ways you can use AWS Config rules to manage license compliance.
This document provides an overview of Azure SQL database and related services including:
- Azure SQL Database which provides single database and elastic pool models for predictable or shared workloads.
- Azure SQL Managed Instance which provides high compatibility with SQL Server in a PaaS model.
- Related Azure data and analytics services for ingestion, storage, preparation, modeling and serving of data.
- Key capabilities of Azure SQL Database around data migration, programmability, security and operations.
Database performance is a key factor for keeping your app running fast. As the amount of data and request throughput grow, it becomes harder to keep it nimble and performant. In this webinar, we will discuss using Redis, a popular NoSQL key-value store, to run your data fast at scale. Redis can be used as a cache in front of a database or as a fast in-memory data store. We will also cover how you can use ElastiCache for Redis to easily set up a large multi-terabyte Redis environment and operate it with zero management.
Learning Objectives:
• Understand fast data
• Understand the capabilities of Amazon Elasticache for Redis to cache Relational and NoSQL databases
• Use Amazon Elasticache for Redis as a primary data store
Who Should Attend:
• Developers, Data Architects
Cloud for Developers: Azure vs. Google App Engine vs. Amazon vs. AppHarborSvetlin Nakov
Software Development for the Public Cloud Platforms: Windows Azure vs. Google App Engine vs. Amazon Web Services (AWS) vs AppHarbor.
In this talk the speaker will compare the most widely used public PaaS clouds (Azure, GAE and AWS) from the software developer’s perspective.
A parallel between Azure, GAE, AWS and few other clouds (like AppHarbor, Heroku, Cloudfoundry and AppForce) will be made based on several criteria: architecture, pricing, storage services (non-relational databases, relational databases in the cloud and blob/file storage), business-tier services (like queues, notifications, email, CDN, etc.), supported languages, platforms and frameworks and front-end technologies.
A live demo will be made to compare the way we build and deploy a multi-tiered application in Azure, Amazon and GAE and how to implement its back-end (using a cloud database), business tier (based on REST services) and front-end (based on HTML5).
The speaker Svetlin Nakov (http://www.nakov.com) is well-known software development expert and trainer, a head of the Telerik Software Academy and a main organizer of the Cloud Development course (http://clouddevcourse.telerik.com).
Spark 101 – First Steps To Distributed Computing - Demi Ben-Ari @ Ofek AlumniDemi Ben-Ari
The world has changed and having one huge server won’t do the job anymore, when you’re talking about vast amounts of data, growing all the time the ability to Scale Out would be your saviour.
This lecture will be about the basics of Apache Spark and distributed computing and the development tools needed to have a functional environment.
Bio:
Demi Ben-Ari, Sr. Data Engineer @Windward, Ofek Alumni
Has over 9 years of experience in building various systems both from the field of near real time applications and Big Data distributed systems.
Co-Founder of the “Big Things” Big Data community: http://somebigthings.com/big-things-i...
One of the most fundamental challenges of CI/CD is the ability to balance between Quality, Time, and Cost. Amazon EC2 Container Service (ECS), along with Docker and Amazon EC2 Container Registry (ECR), has changed the game for many by making resource management very simple. For Okta, it has enabled the Continuous Integration team to maximize throughput while minimizing cost. In this session we will show you how Okta has created a flexible CI system with ECS, Docker, ECR, AWS Lambda, AWS CloudFormation, Amazon RDS, and Amazon SQS. Okta runs 30,000 tests with each developer commit, and releases 10,000 new lines of code each week to production. The CI system, built 100% on AWS, must be able to handle load while keeping cost under control. This talk is oriented toward developers looking to achieve efficient resource and cost management without compromising speed or quality.
AWS re:Invent 2016: How to Launch a 100K-User Corporate Back Office with Micr...Amazon Web Services
Learn how to build a scalable, compliance-ready, and automated deployment of the Microsoft “backoffice” servers for 100K users running on AWS. In this session, we show a reference architecture deployment of Exchange, SharePoint, Skype for Business, SQL Server and Active Directory in a single VPC. We discuss the following: (1) how the solution is automated for 100K users, (2) how the solution is enabled for compliance (e.g., FedRAMP, HIPAA, PCI), and (3) how the solution is built from modular 10K user blocks. Attendees should have knowledge of AWS CloudFormation, PowerShell, instance bootstrapping, VPCs, and Amazon Route 53, as well as the relevant Microsoft technologies.
In our first Windows webinar, find out about the benefits of migrating your Windows workloads to AWS. During the session, we will explain why AWS makes your Windows applications faster, more reliable and more secure. He will also talk about how to bring your own license (BYOL), how to architect, deploy, and manage your Windows platforms on AWS.
New capabilities for modern data integration in the cloudGaurav Malhotra
The document describes how Azure Data Factory can be used to orchestrate data pipelines and execute SSIS packages in the cloud. It provides an overview of key capabilities like lifting and shifting existing SSIS packages to Azure, using various Azure services like SQL Data Warehouse and Databricks in data pipelines, and orchestrating workflows. It also discusses how Azure Data Factory allows scheduling SSIS package execution in a managed cloud environment using familiar SSIS authoring and monitoring tools.
This document provides an overview of how to configure an Azure-SSIS Integration Runtime (IR) within Azure Data Factory. It describes the steps to name and describe the IR, select its location, node size, cluster size, edition/license, and connect it to an existing Azure SQL Database or Managed Instance that will host the SSIS catalog database. It also covers options for using Azure Active Directory authentication or bringing your own Azure SQL credentials. The document guides users through the entire configuration process for an Azure-SSIS IR within Azure Data Factory.
Embrace and extend first-class activity and 3rd party ecosystem for ssis in adfTillmann Eitelberg
This session focuses on the deeper integration of SQL Server Integration Services (SSIS) in Azure Data Factory (ADF) and the broad extensibility of Azure-SSIS Integration Runtime (IR). We will first show you how to provision Azure-SSIS IR – dedicated ADF servers for lifting & shifting SSIS packages – and extend it with custom/3rd party components. Preserving your skillsets, you can then use the familiar SQL Server Data Tools (SSDT)/SQL Server Management Studio (SSMS) to design/deploy/configure/execute/monitor your SSIS packages in the cloud just like you do on premises. Next, we will guide you to trigger/schedule SSIS package executions as first-class activities in ADF pipelines and combine/chain them with other activities, allowing you to inject/splice built-in tasks/data transformations in your ETL/ELT workflows, automatically provision Azure-SSIS IR on demand/just in time, etc. And finally, you will learn about the licensing model for ISVs to develop paid components/extensions and join the growing 3rd party ecosystem for SSIS in ADF with a few examples from our partners.
The document discusses Azure Data Factory and its capabilities for cloud-first data integration and transformation. ADF allows orchestrating data movement and transforming data at scale across hybrid and multi-cloud environments using a visual, code-free interface. It provides serverless scalability without infrastructure to manage along with capabilities for lifting and running SQL Server Integration Services packages in Azure.
This session focuses on the needs of the data integrator and data engineer whether that be for data warehousing & BI, advanced analytics of data for SaaS applications. We walk through a comprehensive set of new additions to Azure Data Factory and SSIS for moving and integrating data across on-premises and cloud. Topics and examples will include simple, scalable and reliable data pipelines in ADF using a serverless, parallel data movement service to/from the cloud, provisioning of Azure-SSIS Integration Runtime (IR) – dedicated servers for lifting & shifting SSIS packages to cloud– and its customization with your own/3rd party extensions, the execution of SSIS packages as first-class activities in ADF pipelines and their combination with other ADF activities to create modern ETL/ELT workflows all through the new code-free experience.
Lift SSIS package to Azure Data Factory V2Manjeet Singh
Manjeet Singh gives a presentation on lifting SSIS packages to Azure using Data Factory v2. He discusses how the Integration Runtime in ADF v2 allows existing on-premises SSIS packages to be lifted to the cloud. He demonstrates deploying a SSIS package to an Azure SQL database, running it using SQL Server Management Studio and Azure Data Factory pipelines, and provides tips on using the SSIS Integration Runtime.
Microsoft SQL Azure - Building Applications Using SQL Azure PresentationMicrosoft Private Cloud
Building Applications Using SQL Azure provides an overview of using Microsoft's SQL Azure platform as a service. It covers setting up a SQL Azure account, connecting applications, managing security, creating database objects, migrating schemas and data, performance considerations, and building a simple application connected to SQL Azure. The presentation aims to help database developers and architects understand how to build applications using the SQL Azure relational database service.
Azure Data Factory ETL Patterns in the CloudMark Kromer
This document discusses ETL patterns in the cloud using Azure Data Factory. It covers topics like ETL vs ELT, the importance of scale and flexible schemas in cloud ETL, and how Azure Data Factory supports workflows, templates, and integration with on-premises and cloud data. It also provides examples of nightly ETL data flows, handling schema drift, loading dimensional models, and data science scenarios using Azure data services.
PaaSport to Paradise: Lifting & Shifting with Azure SQL Database/Managed Inst...Sandy Winarko
This session focuses on the all PaaS solution of Azure SQL DB/Managed Instance (MI) + SSIS in Azure Data Factory (ADF) to lift & shift, modernize, and extend ETL workflows. We will first show you how to provision Azure-SSIS Integration Runtime (IR) – dedicated ADF servers for running SSIS – with SSIS catalog (SSISDB) hosted by Azure SQL DB/MI, configure it to access data on premises using Windows authentication and Virtual Network injection/Self-Hosted IR as a proxy, and extend it with custom/Open Source/3rd party components. We will next show you how to use the familiar SSDT/SSMS tools to design/test/deploy/execute your SSIS packages in the cloud just like you do on premises. We will finally show you how to modernize your ETL workflows by invoking/scheduling SSIS package executions as first-class activities in ADF pipelines and combining/chaining them with other activities, allowing you to trigger your pipeline runs by events, automatically (de)provision SSIS IR just in time, etc.
SQL Saturday Redmond 2019 ETL Patterns in the CloudMark Kromer
This document discusses ETL patterns in the cloud using Azure Data Factory. It covers topics like ETL vs ELT, scaling ETL in the cloud, handling flexible schemas, and using ADF for orchestration. Key points include staging data in low-cost storage before processing, using ADF's integration runtime to process data both on-premises and in the cloud, and building resilient data flows that can handle schema drift.
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)Cathrine Wilhelmsen
This document discusses Azure Data Factory (ADF) and how it can be used to build and orchestrate data pipelines without code. It describes how ADF is a hybrid data integration service that improves on its previous version. It also explains how existing SSIS packages can be "lifted and shifted" to ADF to modernize solutions while retaining investments. The document demonstrates creating pipelines and data flows in ADF, handling schema drift, and best practices for development.
This document provides an agenda and summary for a Data Analytics Meetup (DAM) on March 27, 2018. The agenda covers topics such as disruption opportunities in a changing data landscape, transitioning from traditional to modern BI architectures using Azure, Azure SQL Database vs Data Warehouse, data integration with Azure Data Factory and SSIS, Analysis Services, Power BI reporting, and a wrap-up. The document discusses challenges around data growth, digital transformation, and the shrinking time for companies to adapt to disruption. It provides overviews and comparisons of Azure SQL Database, Data Warehouse, and related Azure services to help modernize analytics architectures.
Dell Acceleration Appliance for Databases 2.0 and Microsoft SQL Server 2014: ...Principled Technologies
As this guide has shown, installing and configuring a Microsoft Windows Server 2012 R2 with SQL Server 2014 powered by the Dell Acceleration Appliance for Databases is a straightforward procedure. A key benefit from implementing DAAD 2.0 into your infrastructure is the ability to accelerate workloads without a complete storage area network redesign. This can be ideal for businesses that have snapshot and deduplication features within their software stack or are looking to improve database performance without investing in large storage solutions that may contain features they do not need. Consider DAAD 2.0 for your business—a storage acceleration solution that requires only 4U of rack space and can potentially give your database workloads a boost.
Azure Data Factory for Redmond SQL PASS UG Sept 2018Mark Kromer
Azure Data Factory is a fully managed data integration service in the cloud. It provides a graphical user interface for building data pipelines without coding. Pipelines can orchestrate data movement and transformations across hybrid and multi-cloud environments. Azure Data Factory supports incremental loading, on-demand Spark, and lifting SQL Server Integration Services packages to the cloud.
ODTUG The Necessary Knowledge and Tools You Need to Have for SOA Suite 11gEdwin Biemond
This document discusses the necessary knowledge and tools for SOA Suite 11g, including MDS, avoiding invalid composites, deployment, testing with SOA TestSuite, AIA CAVS, and continuous integration with Hudson. MDS provides centralized storage for artifacts accessed at design and run-time. Invalid composites can occur when composite references are unavailable; the solution is to import references into MDS. Deployment options include JDeveloper scripts, AIA, and custom WLST/resource adapters. Testing tools include SOA TestSuite, AIA CAVS for validation across composites, and Hudson for continuous integration.
The document discusses transforming an independent software vendor's (ISV's) applications to run on Microsoft Azure. There is growing demand for software-as-a-service (SaaS) offerings globally. Azure provides a broad set of services that can help ISVs offer their existing solutions as SaaS. Technical challenges include migrating databases and file storage to Azure services, and adapting applications for a multi-tenant architecture. Non-technical challenges involve scaling operations and support to handle more customers. The presentation recommends consulting with ISVs on migration and offering managed services to help address scaling and support issues.
SSIS provides capabilities for ETL operations using a control flow and data flow engine. It allows importing and exporting data, integrating heterogeneous data sources, and supporting BI solutions. Key concepts include packages, control flow, data flow, variables, and event handlers. SSIS can be optimized for scalability through techniques like parallelism, avoiding blocking transformations, and leveraging SQL for aggregations. Performance can be monitored using tools like SQL Server logs, WMI, and MOM. SSIS is interoperable with data sources like Oracle, Excel, and flat files.
This document provides an overview of the Microsoft Windows Azure platform, including its core components and capabilities. It begins with definitions of service-oriented architecture (SOA) and cloud computing. It then discusses the various cloud service models of infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). The remainder of the document focuses on the key Microsoft Azure services, including Windows Azure, SQL Azure, AppFabric, and connectivity options. It describes how applications can be deployed on Azure and scaled across roles and instances for availability and performance. It also covers core Azure services like storage, tables, queues, and monitoring.
Global Situational Awareness of A.I. and where its headedvikram sood
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeWalaa Eldin Moustafa
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
Natural Language Processing (NLP), RAG and its applications .pptxfkyes25
1. In the realm of Natural Language Processing (NLP), knowledge-intensive tasks such as question answering, fact verification, and open-domain dialogue generation require the integration of vast and up-to-date information. Traditional neural models, though powerful, struggle with encoding all necessary knowledge within their parameters, leading to limitations in generalization and scalability. The paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" introduces RAG (Retrieval-Augmented Generation), a novel framework that synergizes retrieval mechanisms with generative models, enhancing performance by dynamically incorporating external knowledge during inference.
End-to-end pipeline agility - Berlin Buzzwords 2024Lars Albertsson
We describe how we achieve high change agility in data engineering by eliminating the fear of breaking downstream data pipelines through end-to-end pipeline testing, and by using schema metaprogramming to safely eliminate boilerplate involved in changes that affect whole pipelines.
A quick poll on agility in changing pipelines from end to end indicated a huge span in capabilities. For the question "How long time does it take for all downstream pipelines to be adapted to an upstream change," the median response was 6 months, but some respondents could do it in less than a day. When quantitative data engineering differences between the best and worst are measured, the span is often 100x-1000x, sometimes even more.
A long time ago, we suffered at Spotify from fear of changing pipelines due to not knowing what the impact might be downstream. We made plans for a technical solution to test pipelines end-to-end to mitigate that fear, but the effort failed for cultural reasons. We eventually solved this challenge, but in a different context. In this presentation we will describe how we test full pipelines effectively by manipulating workflow orchestration, which enables us to make changes in pipelines without fear of breaking downstream.
Making schema changes that affect many jobs also involves a lot of toil and boilerplate. Using schema-on-read mitigates some of it, but has drawbacks since it makes it more difficult to detect errors early. We will describe how we have rejected this tradeoff by applying schema metaprogramming, eliminating boilerplate but keeping the protection of static typing, thereby further improving agility to quickly modify data pipelines without fear.
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
State of Artificial intelligence Report 2023kuntobimo2016
Artificial intelligence (AI) is a multidisciplinary field of science and engineering whose goal is to create intelligent machines.
We believe that AI will be a force multiplier on technological progress in our increasingly digital, data-driven world. This is because everything around us today, ranging from culture to consumer products, is a product of intelligence.
The State of AI Report is now in its sixth year. Consider this report as a compilation of the most interesting things we’ve seen with a goal of triggering an informed conversation about the state of AI and its implication for the future.
We consider the following key dimensions in our report:
Research: Technology breakthroughs and their capabilities.
Industry: Areas of commercial application for AI and its business impact.
Politics: Regulation of AI, its economic implications and the evolving geopolitics of AI.
Safety: Identifying and mitigating catastrophic risks that highly-capable future AI systems could pose to us.
Predictions: What we believe will happen in the next 12 months and a 2022 performance review to keep us honest.
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Aggregage
This webinar will explore cutting-edge, less familiar but powerful experimentation methodologies which address well-known limitations of standard A/B Testing. Designed for data and product leaders, this session aims to inspire the embrace of innovative approaches and provide insights into the frontiers of experimentation!
8. Phase 1: Use ADFv2 to provision SSIS
PaaS – Launched at MS Ignite (Sep’17)
Create ADFv2, if you have not done so already
Use ADFv2 App/SDK/API/PSH to provision SSIS
PaaS w/ ADF compute resource called Azure-SSIS
Integration Runtime (IR)
Still use SQL Server Data Tools (SSDT) to
design/deploy SSIS packages
Still use SQL Server Management Studio (SSMS) to
execute/manage SSIS packages
Serve SSIS customers who want to move all/part of
their on-premises workloads and just “lift & shift”
many existing packages to Azure
Independent Software Vendors (ISVs) can build
extensions/Software as a Service (SaaS) on SSIS
PaaS
Execute/Manage
Provision
SSDT SSMS
Cloud
On-Premises
Design/Deploy
SSIS Server
Design/Deploy Execute/Manage
ISVs
SSIS PaaS w/
ADF compute
resource
ADF (PROVISIONING)
HDI
ML
9. Introducing Azure-SSIS IR: Managed
cluster of Azure VMs (nodes) dedicated
to run your SSIS packages and no other
activities
You can scale it up/out by specifying the node size
/number of nodes in the cluster
You can bring your own Azure SQL Database
(DB)/Managed Instance (MI) server to host the
catalog of SSIS projects/packages (SSISDB) that will
be attached to it
You can join it to a Virtual Network (VNet) that is
connected to your on-prem network to enable on-
prem data access
Once provisioned, you can enter your Azure SQL
DB/MI server endpoint on SSDT/SSMS to deploy
SSIS projects/packages and configure/execute them
just like using SSIS on premises
Execute/Manage
Provision
SSDT SSMS
Cloud
On-Premises
Design/Deploy
SSIS Server
Design/Deploy Execute/Manage
ISVs
SSIS PaaS w/
ADF compute
resource
ADF (PROVISIONING)
HDI
ML
10. Customer cohorts for Phase 1:
1. "SQL Migrators"
These are SSIS customers who want to retire
their on-prem SQL Servers and migrate all apps
+ data (“complete/full lift & shift”) into Azure
SQL MI – For them, SSISDB can be hosted by
Azure SQL MI inside their VNet
2. "ETL Cost Cutters“
These are SSIS customers who want to lower
their operational costs and gain High Availability
(HA)/scalability for just their ETL workloads w/o
managing their own infra (“partial lift & shift") –
For them, SSISDB can be hosted by Azure SQL
DB in the public network
Execute/Manage
Provision
SSDT SSMS
Cloud
On-Premises
Design/Deploy
SSIS Server
Design/Deploy Execute/Manage
ISVs
SSIS PaaS w/
ADF compute
resource
ADF (PROVISIONING)
HDI
ML
11. LIVE now in Public Preview
Six locations: East US/East US 2/Central US/North
Europe/West Europe/Australia East
Six node sizes: Av2/Dv2 series VMs
Classic VNet support
Standard edition/license
24/7 live-site support
What’s new
Azure Resource Manager (ARM) VNet support
Enterprise edition/license (Private Preview)
Custom setup interface (Private Preview)
ADFv2 App/GUI web tool
Coming soon
More locations, e.g. West US, UK South
More node sizes, e.g. Dv3/Ev3 series VMs
Azure Hybrid Use Benefit (AHUB)/Bring Your Own
License (BYOL) support
Execute/Manage
Provision
SSDT SSMS
Cloud
On-Premises
Design/Deploy
SSIS Server
Design/Deploy Execute/Manage
ISVs
SSIS PaaS w/
ADF compute
resource
ADF (PROVISIONING)
HDI
ML
12. Phase 2: Use ADFv2 to
execute/manage SSIS packages
deployed to SSIS PaaS as first-class
activities – ETA Q2CY18 (TBD)
Chain/group them with Azure HDInsight
(HDI)/Machine Learning (ML)/other activities
inside data pipelines
Serve SSIS/ADF customers who want to
combine ETL/ELT workflows on a single
platform
Execute/Manage
Provision
Execute/
Manage
SSDT SSMS
Cloud
On-Premises
Design/Deploy
SSIS Server
Design/Deploy Execute/Manage
ISVs
SSIS PaaS w/
ADF compute
resource
ADF (PROVISIONING,
SCHEDULING)
HDI
ML
13. Customer cohorts for Phase 2:
3. “ETL Modernizers"
These are SSIS customers who want to
modernize their workflows and explore Big Data
Analytics in the cloud
4. “ELT Gap Fillers"
These are ADF customers who want to fill some
gaps in their workflows w/ code-free authoring
of transformations/built-in tasks from SSIS
Execute/Manage
Provision
Execute/
Manage
SSDT SSMS
Cloud
On-Premises
Design/Deploy
SSIS Server
Design/Deploy Execute/Manage
ISVs
SSIS PaaS w/
ADF compute
resource
ADF (PROVISIONING,
SCHEDULING)
HDI
ML
14.
15. ADF is Microsoft’s unified platform for ETL/ELT services in the cloud
ADF allows you to build data pipelines and trigger/schedule their runs
Data pipeline is a chain/group of activities to be performed on your data, e.g. data movements/transformations
Activities take data sets, which are named references/pointers to data, as inputs/outputs
Some activities target data store/compute resources allocated outside ADF, e.g. ADLS/HDI/ML/etc.
Linked services represent those resources and provide the connection info for ADF to orchestrate activities targeting
them
16. Integration Runtime (IR) is ADF compute resource
that can perform data movements/transformations,
including SSIS package executions
Customers can deploy one/many instances of IR
per ADF as required to run pipelines/process data
IR can run in the public network, inside VNet, or
behind corporate firewalls
SSIS PaaS runs on Azure-SSIS IR, internal/native to
ADF that provisions it, while SSISDB is hosted by
your own Azure SQL DB/MI server, external to ADF
Consequently, Azure-SSIS IR is billed under your
ADF subscription, separately from SSISDB that is
billed under your Azure SQL DB/MI subscription
ADF
Data Movements
SSIS Package Executions
Hadoop/Spark (Hive, Pig, etc.)
Data Lake Analytics (DLA)
Machine Learning (ML)
SQL Stored Procedures (Sprocs)
Custom Activities
Integration Runtime (IR)
DATA PIPELINES
Processing
activity
Orchestration
and monitoring
Processing
activity
Azure HDI, DLA, ML, SQL
DB/MI/DW, Batch, etc.
Orchestration
and monitoring
17.
18.
19. Enterprise Features Descriptions
CDC components • CDC Source/Splitter/Control Task are preinstalled on your Azure-SSIS IR Enterprise
Edition
• To connect to Oracle, you will also need to install CDC Designer/Service on another
machine
Oracle connectors • Oracle Connection Manager/Source/Destination are preinstalled on your Azure-SSIS
IR Enterprise Edition
• You will also need to install Oracle Call Interface (OCI) driver, and if necessary
configure Oracle Transport Network Substrate (TNS), on your Azure-SSIS IR (via
Custom Setup Interface)
Teradata connectors • You will need to install Teradata Connection Manager/Source/Destination and
Teradata Parallel Transporter (TPT) API + Teradata ODBC driver on your Azure-SSIS IR
Enterprise Edition (via Custom Setup Interface)
20. Enterprise Features Descriptions
SAP BW connectors • SAP BW Connection Manager/Source/Destination are preinstalled on your Azure-SSIS
IR Enterprise Edition
• You will also need to install SAP BW driver on your Azure-SSIS IR (via Custom Setup
Interface)
• These connectors support SAP BW 7.0 or earlier versions
• To connect to later versions, you can install SAP connectors from our partners (e.g.
Theobald Software) on your Azure-SSIS IR (via Custom Setup Interface)
21. Enterprise Features Descriptions
SSAS/AAS components • Data Mining Model Training/Dimension Processing/Partition Processing Destinations
and Data Mining Query Transform are preinstalled on your Azure-SSIS IR Enterprise
Edition
• All support SSAS, but only Partition Processing Destination supports AAS
• To connect to SSAS, you will also need to configure Windows Authentication
credentials in your SSISDB
• On top of these components, Analysis Services Execute DDL/Processing and Data
Mining Query Tasks are also preinstalled on your Azure-SSIS IR Standard/Enterprise
Edition
Fuzzy Grouping/Lookup
Transforms
• They are preinstalled on your Azure-SSIS IR Enterprise Edition and support SQL
Server/Azure SQL DB for storing reference data
Term Extraction/Lookup
Transforms
• They are preinstalled on your Azure-SSIS IR Enterprise Edition and support SQL
Server/Azure SQL DB for storing reference data
42. PSH scripts to manipulate your Azure Storage
account
Azure Blob Download Task
Execute Process Task that executes it
Teradata’s site
43.
44. Azure-SSIS IR node
Container
ISV Setup1. Specify Product Key in setup script ISV Activation Server
2. Get Activation Key by submitting Cluster ID + Product Key
Local Store
(e.g. Registry)
3. Write Activation Key
SSIS Executor
ISV Extension
4. Read Activation Key and
validate it with Cluster ID
Setup
Runtime
4. Get Cluster ID
4. Report on Node Count (Optional)
SSIS Runtime
2. Activation Key
54. Select its location
Select the right location for your Azure-SSIS IR to achieve high
performance in ETL workflows – East US/East US 2/Central US/North
Europe/West Europe/Australia East are available in Public Preview,
more will be provided in the future (e.g. West US, UK South, etc.)
It needs not be the same as the location of your ADFv2, but it should
be the same as the location of your own Azure SQL DB/MI server
where SSISDB will be hosted and or the location of VNet connected
to your on-prem network
Avoid Azure-SSIS IR accessing SSISDB/data movements across
different locations
55. Select its node size
A selection of node sizes specifying the number of cores (CPU) and
size of memory (RAM) per node is provided –
Standard_A4_v2/Standard_A8_v2/
Standard_D1_v2/Standard_D2_v2/Standard_D3_v2/Standard_D4_v2
are available in Public Preview, more will be provided in the future
(e.g. Dv3/Ev3 series VMs, etc.)
Select a large node size (scale up), if you want to run many compute/
memory –intensive packages
56. Select its cluster size
A range of 1-10 for the number of nodes per cluster is provided
Select a large cluster with many nodes (scale out), if you want to run
many packages in parallel
Like Azure VMs, you can manage the cost of running Azure-SSIS IR
by stopping/starting it, effectively releasing/reallocating its nodes, as
you see fit
57. Select its edition/license
Standard edition/license is available in Public Preview, Enterprise
edition/license will be provided in the future
58. Bring your own Azure SQL DB/MI
server to host SSISDB
A selection of your existing Azure SQL DB/MI servers in the same
location as your Azure-SSIS IR is provided – If there is none, create a
new one to ensure that your Azure-SSIS IR can easily access SSISDB
without incurring excessive traffics between different locations
If you select your existing Azure SQL MI server to host SSISDB inside
a VNet, you must also join your Azure-SSIS IR to the same VNet, so
we can prepare and manage SSISDB on your behalf
The endpoint of your selected Azure SQL DB/MI server becomes the
connection info to enter on SSDT/SSMS
59. Enter your Azure SQL DB/MI server
admin credentials
Your Azure SQL DB/MI server admin username and password are
required for us to prepare and manage SSISDB on your behalf
60. Select the service tier and provide
access consent for your Azure SQL
DB server hosting SSISDB
A selection of service tiers (e.g. S1/S2/S3/etc.) for your Azure SQL DB
server hosting SSISDB is provided – Not applicable to Azure SQL MI
Select a large service tier, if you want to achieve high database performance
and store many SSIS projects/packages/execution logs
By default, you must allow Azure services to access your Azure SQL
DB server hosting SSISDB – Not applicable to Azure SQL MI
This is required for us to prepare and manage SSISDB on your behalf
If there is an access issue, you will be guided to resolve it
61. Select the concurrency level of your
Azure-SSIS IR nodes
A range of 1-8 for the number of parallel executions per node is
provided
Select a low concurrency level, if you want to use more than one
cores to run a single large/compute-intensive package
Select a high concurrency level, if you want to run one or more
small/light-weight packages in a single core
62. Optionally join your Azure-SSIS IR
to a VNet and provide consent for
VNet configuration
If you use Azure SQL MI to host SSISDB inside a VNet, you must also
join your Azure-SSIS IR to the same VNet, so we can prepare and
manage SSISDB on your behalf
If you have on-prem data sources/destinations in your SSIS
packages, you must join your Azure-SSIS IR to a VNet that is
connected to your on-prem network to enable on-prem data access
By default, you must allow Azure services to configure the
permissions/settings of selected VNet for your Azure-SSIS IR to join
If there is a configuration issue, you will be guided to resolve it
63. Select VNet and subnet for your
Azure-SSIS IR to join
A selection of existing VNets and subnets in the location of your
Azure-SSIS IR is provided – ARM VNet is recommended, since Classic
VNet will be deprecated soon
Ideally, your Azure-SSIS IR, Azure SQL DB/MI server to host SSISDB,
and VNet connected to your on-prem network should all be in the
same location
If that is not the case, you can first create your Azure-SSIS IR in the location
of Azure SQL DB/MI server and join it to a VNet in the same location, then
configure a VNet-to-VNet connection with the VNet connected to your on-
prem network
64. When you submit a
provisioning request, it
takes 10 – 20 minutes to
allocate/prepare the
nodes of your Azure-SSIS
IR and start billing
65. Once your Azure-SSIS IR
is started, you can deploy
SSIS packages to execute
on it and you can stop it
as you see fit
We start with SSIS and ADF as two separate tools for traditional on-prem ETL and modern cloud ELT workflows, respectively. We are converging these tools to create a unified platform for Microsoft ETL/ELT services in the cloud.
This opens the 1st door on ADF for provisioning SSIS PaaS and still preserves the skillsets of old-school SSIS customers who are familiar with SSDT/SSMS.
SSIS PaaS requires VNet to enable on-prem data access for now, but we may also reuse ADF Data Management Gateway (DMG) to do the same in the future.
"SQL Migrators": They want Azure SQL MI inside their VNet to maintain network isolation. To support them, we need to have both SSISDB hosted by Azure SQL MI and Azure-SSIS IR inside the same VNet.
"ETL Cost Cutters": They don't necessarily want to migrate their SQL Servers to the cloud, in fact some of their sensitive data must stay on premises. Unlike cohort 1), they don't have strict VNet requirements and accept creating a new VNet to enable on-prem data access. Some of them have just cloud<->cloud ETL scenarios that don't require VNet at all. To support them, we can have SSISDB hosted by Azure SQL DB in the public network and Azure-SSIS IR optionally inside their VNet.
This opens the 2nd door on ADF for triggering/scheduling executions of SSIS packages deployed to SSIS PaaS.
“ETL Modernizers": They can also be considered as “ETL Gap Fillers”, since they want to use Big Data processing beyond the basic tasks included in SSIS Azure Feature Pack (e.g. HDI Hive/Pig Tasks). Some of them already use SSIS ADLS Destination in their packages to ingest data into ADLS and want to use ADF ADLA U-SQL Activity to transform data there, chaining ingestion and transformation activities in a single pipeline.
“ELT Gap Fillers": Most of them already use ADF to move data to staging stores and want to use SSIS to transform data before ingesting it into destination stores. Some of them already use ADFv1 Sproc Activity to schedule SSIS transformations in their pipelines and want a code-free ADFv2 Scheduler to do the same.
Unlike cohorts 1 and 2, who only need ADFv2 for provisioning Azure-SSIS IR, cohorts 3 and 4 need ADFv2 for scheduling SSIS package executions as first-class activities/jobs.
SSIS PaaS always supports the latest SSIS version, so new features will be frequently released cloud-first to SSIS PaaS and then backported to on-prem box products later.
Running isdeploymentwizard.exe from the command prompt might only be needed for automatic deployments from batch files.
SSIS PaaS connection info and SQL authentication credentials need to be entered again, since Integration Services Deployment Wizard is a separate executable from SSMS.
Running dtexec.exe from the command prompt might only be needed for custom scheduling solutions.
No need to create a new sproc, just use the system sproc sp_executesql that executes T-SQL script.
By default, SSISDB sproc [catalog].[start_execution] is asynchronous with the package execution on Azure-SSIS IR. You can synchronize them by setting SYNCHRONIZED execution parameter, so that ADF Sproc Activity exits only when the package execution exits and you can raise an error for unsuccessful execution to be investigated on SSMS.
The availability property of output dataset drives scheduling.