Continuous integration is not normally associate with data warehouse projects due to the perceived complexity of implementation. John shows how modern tools make it simple to apply CI to the data warehouse. The session covers:
* The benefits of the SQL Server Data Tools declarative model
* Using PowerShell and psake to automate your build and deployments
* Implementing the TeamCity build server
* Integration and regression testing
* Auto-code generation within SSDT using T4 templates and DacFx
SSDT Workshop @ SQL Bits X (2012-03-29)Gert Drapers
Mastering SSDT with the DataDude
This is your chance to hear the real story behind SSDT, directly from the man who built it. SQL Server Data Tools is effectively the 3rd version of the DataDude project, started in 2005 by Gert Drapers. SQL Server Data Tools (SSDT) lets you develop, test, and maintain SQL Server and SQL Azure databases offline on your desktop. SSDT's modern T-SQL development environment supports declarative model-driven development whether working connected or offline, and integrates with Visual Studio's project and application lifecycle management tools to enable team development and source code control support for SQL Server and SQL Azure databases.
This master class will get you started using SSDT; provides you with the architectural ins and outs of schema management using SSDT; team oriented database development and leveraging the command line and programmatic interfaces that accompany SSDT for importing, comparing and deploying database schemas. Learn it from the DataDude himself.
Writing simple web services in java using eclipse editorSantosh Kumar Kar
This is a simple steps showing how you can write a simple web service, host into a server, write a client class to access the service on web server. Just for a beginners...
Microsoft SQL Azure - Developing And Deploying With SQL Azure WhitepaperMicrosoft Private Cloud
SQL Azure is built on the SQL Server’s core engine, so developing against SQL Azure is very similar to developing against on-premise SQL Server. While there are certain features that are not compatible with SQL Azure, most T-SQL syntax is compatible. The MSDN link http://msdn.microsoft.com/en-us/library/ee336281.aspx provides a comprehensive description of T-SQL features that are supported, not supported and partially supported in SQL Azure.
SSDT Workshop @ SQL Bits X (2012-03-29)Gert Drapers
Mastering SSDT with the DataDude
This is your chance to hear the real story behind SSDT, directly from the man who built it. SQL Server Data Tools is effectively the 3rd version of the DataDude project, started in 2005 by Gert Drapers. SQL Server Data Tools (SSDT) lets you develop, test, and maintain SQL Server and SQL Azure databases offline on your desktop. SSDT's modern T-SQL development environment supports declarative model-driven development whether working connected or offline, and integrates with Visual Studio's project and application lifecycle management tools to enable team development and source code control support for SQL Server and SQL Azure databases.
This master class will get you started using SSDT; provides you with the architectural ins and outs of schema management using SSDT; team oriented database development and leveraging the command line and programmatic interfaces that accompany SSDT for importing, comparing and deploying database schemas. Learn it from the DataDude himself.
Writing simple web services in java using eclipse editorSantosh Kumar Kar
This is a simple steps showing how you can write a simple web service, host into a server, write a client class to access the service on web server. Just for a beginners...
Microsoft SQL Azure - Developing And Deploying With SQL Azure WhitepaperMicrosoft Private Cloud
SQL Azure is built on the SQL Server’s core engine, so developing against SQL Azure is very similar to developing against on-premise SQL Server. While there are certain features that are not compatible with SQL Azure, most T-SQL syntax is compatible. The MSDN link http://msdn.microsoft.com/en-us/library/ee336281.aspx provides a comprehensive description of T-SQL features that are supported, not supported and partially supported in SQL Azure.
SQL Server 2016: Just a Few of Our DBA's Favorite ThingsHostway|HOSTING
Join Rodney Landrum, Senior DBA Consultant for Ntirety, a division of HOSTING, as he demonstrates his favorite new features of the latest Microsoft SQL Server 2016 Service Pack 1.
During the accompanying webinar and slides, Rodney will touch on the following:
• A demo of his favorite new features in SQL Server 2016 and SP1 including:
o Query Store
o Database Cloning
o Dynamic Data Masking
o Create or Alter
• A review of Enterprise features that are now available in standard edition
• New information in Dynamic Management Views and SQL Error Log that will make your DBAs job easier.
Cause 2013: A Flexible Approach to Creating an Enterprise Directoryrwgorrel
Leveraging Microsoft Active Directory LDS to create a flexible enterprise directory.
As UNCG sought to replace Novell Directory Services with the next generation enterprise authentication and directory services (LDAP), we examined OpenLDAP, Active Directory, and Active Directory Lightweight Domain Services. Hear why we picked a somewhat uncommon approach in the less known AD LDS product and the flexibility it afforded us a middle ground between OpenLDAP and the urge to use existing Active Directory domain. We will also discuss the ADAMSync tool used to populate this environment as well as the MSUserProxy object to centralize authentication.
In Microsoft CSS, Setting up and Configuring Kerberos for MSBI is one of the top call volume generators which makes us realize there is definitely some gap in Understanding on how to setup and configure Kerberos for MSBI stack in a multi-server farm environment. In the session, we intend to explain and more importantly simplify the steps to setup Kerberos for SQL Server, SSAS, SSRS & Sharepoint along with the Demo of the issues which can occur based on real live experiences with troubleshooting and configuring for Customers.
Presentation by Shree Prasad Khanal, Leader, Himalayan SQL Server User Group, on "Where should I be encrypting my data? " at "Braindigit 9th National ICT Conference 2013" organized by Information Technology Society, Nepal at Alpha House, Kathmandu, Nepal on 26th January, 2013
With Microsoft Cloud Solutions Azure, Enterprises now has four options to host SQL Server in environment
SQL Server on Physical Machines
SQL Server on VMs in DataCenters
SQL Server on Azure VMs (IaaS)
SQL Server on Azure SQL Database (PaaS)
The Slide can be used as a decision matrix for where to host my SQL Server instance
PASS VC: SQL Server Performance Monitoring and BaseliningPARIKSHIT SAVJANI
When managing large scale deployment of SQL Server instances, it is important for DBAs to setup proactive monitoring & establishing performance baselines which helps in performance tuning, capacity planning & identifying workload patterns. Attend this session to learn what data should a DBAs collect & how, to monitor & establish performance baseline in SQL Server.
SQL Server 2016: Just a Few of Our DBA's Favorite ThingsHostway|HOSTING
Join Rodney Landrum, Senior DBA Consultant for Ntirety, a division of HOSTING, as he demonstrates his favorite new features of the latest Microsoft SQL Server 2016 Service Pack 1.
During the accompanying webinar and slides, Rodney will touch on the following:
• A demo of his favorite new features in SQL Server 2016 and SP1 including:
o Query Store
o Database Cloning
o Dynamic Data Masking
o Create or Alter
• A review of Enterprise features that are now available in standard edition
• New information in Dynamic Management Views and SQL Error Log that will make your DBAs job easier.
Cause 2013: A Flexible Approach to Creating an Enterprise Directoryrwgorrel
Leveraging Microsoft Active Directory LDS to create a flexible enterprise directory.
As UNCG sought to replace Novell Directory Services with the next generation enterprise authentication and directory services (LDAP), we examined OpenLDAP, Active Directory, and Active Directory Lightweight Domain Services. Hear why we picked a somewhat uncommon approach in the less known AD LDS product and the flexibility it afforded us a middle ground between OpenLDAP and the urge to use existing Active Directory domain. We will also discuss the ADAMSync tool used to populate this environment as well as the MSUserProxy object to centralize authentication.
In Microsoft CSS, Setting up and Configuring Kerberos for MSBI is one of the top call volume generators which makes us realize there is definitely some gap in Understanding on how to setup and configure Kerberos for MSBI stack in a multi-server farm environment. In the session, we intend to explain and more importantly simplify the steps to setup Kerberos for SQL Server, SSAS, SSRS & Sharepoint along with the Demo of the issues which can occur based on real live experiences with troubleshooting and configuring for Customers.
Presentation by Shree Prasad Khanal, Leader, Himalayan SQL Server User Group, on "Where should I be encrypting my data? " at "Braindigit 9th National ICT Conference 2013" organized by Information Technology Society, Nepal at Alpha House, Kathmandu, Nepal on 26th January, 2013
With Microsoft Cloud Solutions Azure, Enterprises now has four options to host SQL Server in environment
SQL Server on Physical Machines
SQL Server on VMs in DataCenters
SQL Server on Azure VMs (IaaS)
SQL Server on Azure SQL Database (PaaS)
The Slide can be used as a decision matrix for where to host my SQL Server instance
PASS VC: SQL Server Performance Monitoring and BaseliningPARIKSHIT SAVJANI
When managing large scale deployment of SQL Server instances, it is important for DBAs to setup proactive monitoring & establishing performance baselines which helps in performance tuning, capacity planning & identifying workload patterns. Attend this session to learn what data should a DBAs collect & how, to monitor & establish performance baseline in SQL Server.
Schmutz im Code behindert mittel- und langfristig effiziente Software-Entwicklung. Seine Ursachen sind vielfältig. Clean Code zeigt, wie wir effizienter arbeiten und gleichzeitig höhere Komplexität handhaben können. Ein spezieller Clean-Code-Prozess hilft, Clean-Code-Maßnahmen im Projektalltag lebendig werden zu lassen. Muss wartbarer Code höher bewertet werden als funktionierender Code?
Weitere Vorträge, die wir auch gern in Ihrem Unternehmen halten, finden Sie unter: https://www.iks-gmbh.com/impulsvortraege
Agile Methods and Data Warehousing (2016 update)Kent Graziano
This presentation takes a look at the Agile Manifesto and the 12 Principles of Agile Development and discusses how these apply to Data Warehousing and Business Intelligence projects. Several examples and details from my past experience are included. Includes more details on using Data Vault as well. (I gave this presentation at OUGF14 in Helsinki, Finland and again in 2016 for TDWI Nashville.)
Dev ops for big data cluster management toolsRan Silberman
What are the tools that we can find to day to manage Hadoop cluster and its ecosystem?
There are two tools ready today:
Cloudera Manager and Ambari from Hortonworks.
In this presentation I explain what they do and why to use them, as well as Pros. and Cons.
Unter http://www.youtube.com/watch?v=8fATk1esGj4 gibt es den Vortrag auch als Film.
Während sich agile Verfahren in der Entwicklung immer mehr durchsetzen, fehlt es im IT-Betrieb häufig noch an der Akzeptanz und Implementierung agiler Prozesse. Die Folge: eine Kluft zwischen Entwicklern und Administratoren.
DevOps ist eine IT-Bewegung, die es sich zur Aufgabe macht, ein Umdenken in den Lagern der Entwicklung (Development) und des IT-Betriebs (Operations) zu bewirken. Ziel ist es:
bei Entwicklern und Administratoren ein Bewusstsein für die übergeordneten, gemeinsamen Ziele zu wecken (auch wenn sich individuelle Teilziele und Anforderungen der zwei Bereiche zumindest auf den ersten Blick zu unterscheiden scheinen).
Lösungen zu entwickeln, mit denen die Zusammenarbeit zwischen Entwicklern und Administratoren optimiert werden kann und die zentralen Grundsätze agiler Methoden auch auf den Betrieb ausgedehnt werden können.
Schlomo Schapiro, Systemarchitekt und Open-Source-Evangelist bei ImmobilienScout24, gibt eine Einführung in das Thema und berichtet über die DevOps-Erfolge bei ImmobilienScout24. Nach 2 Jahren agilem Umdenken in der Entwicklung hat sich gezeigt, dass die klassische Gewaltenteilung Plan - Build - Run nicht mehr praktikabel ist, weil sich die Anforderungen und Erwartungen sowohl in der Entwicklung als auch im Betrieb kontinuierlich ändern und neue Formen der Zusammenarbeit erfordern.
Zielgruppe des TechTalks sind Mitarbeiter und Führungskräfte aus IT- und Web-Unternehmen, die sich für agile Arbeitsweisen und crossfunktionale Teams interessieren. Der Vortrag dreht sich weniger um technische Errungenschaften (die hier eher als konkretes Beispiel dienen), sondern vielmehr um die Art und Weise, wie die DevOps-Idee ein Unternehmen weiterentwickeln kann und dabei alle Mitarbeiter aus der IT und darüber hinaus mitnehmen kann.
DevOps and Continuous Delivery Reference Architectures - Volume 2Sonatype
CONTINUOUS DELIVERY REFERENCE ARCHITECTURES Including Sonatype Nexus and other popular DevOps tools Derek E. Weeks (@weekstweets) VP and DevOps Advocate Sonatype.
Continuous Delivery and DevOps Reference Architectures include many common tool choices. The most common tool choices we find in these reference architectures are: Eclipse, git, Cloudbees Jenkins / Atlassian Bamboo, Sonatype Nexus, Atlassian JIRA, SonarQube, Puppet, Chef, Rundeck, Maven / Ant / Gradle, Subversion (svn), Junit, LiveRebel, ServiceNow
As software teams transition to cloud-based architectures and adopt more agile processes, the tools they need to support their development cycles will change. In this session, we'll take you through the transition that Amazon made to a service-oriented architecture over a decade ago. We will share the lessons we learned, the processes we adopted, and the tools we built to increase both our agility and reliability. We will also introduce you to AWS CodeCommit, AWS CodePipeline, and AWS CodeDeploy, three new services born out of Amazon's internal DevOps experience.
Im letzten Jahr entstand für einen großen Kunden ein Mobile Game für iOS, das serverseitig auf Windows Azure basiert. Zum Einsatz kommen dabei Computer-Instanzen, Backup, Table Storage, SQL Database. Die Spiel-Logik wurde serverseitig mit CQRS umgesetzt, das für Azure ideal einsetzbar ist. Der Vortrag gibt einen Einblick in die Architektur und zeigt den Einsatz der gewählten Azure-Komponenten.
This article describe entity framework code first migration steps in a simple way .Code first migrations commands and how to deployed to the Azure cloud .
Presentación sobre Integration Services en SQL Server 2008.
Ing. Eduardo Castro Martinez, PhD
Microsoft SQL Server MVP
http://ecastrom.blogspot.com
http://comunidadwindows.org
Apoyo en la administración de bases de datos mediante microsoft data toolsSpanishPASSVC
Aprovechar las características de Microsoft Data Tools para la Ingeniería inversa de Bases de Datos, manejo organizado de Scripts, Comparación de esquemas entre Servidores, Aplicación organizada de cambios y Refactoring.
DWX 2023 - Datenbank-Schema Deployment im Kubernetes ReleaseMarc Müller
Kubernetes bietet viel Funktionalität, um Zero-Downtime Deployments durchzuführen. Etwas herausfordernder wird es dann, wenn der Service-Update auch mit einem Datenbank-Schema Update verbunden ist. Nebst den verschiedenen Strategien, um ein Datenbankschema in einem Zero-Downtime-Release auszurollen, lernen Sie in diesem Vortrag, wie das Datenbank-Schema sowie die Deployment-Tools in einem Container Verpackt mit der Applikation ausgerollt werden können. Somit erhalten wir ein einziges, in sich konsistentes, Helm Paket, welches den Service samt Datenbank-Schema ausrollen kann.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
20 Comprehensive Checklist of Designing and Developing a WebsitePixlogix Infotech
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
UiPath Test Automation using UiPath Test Suite series, part 6
Continuous Integration and the Data Warehouse - PASS SQL Saturday Slovenia
1. Continuous Integration
and the
Data Warehouse
Dr. John Tunnicliffe
Independent BI Architect &
Hands-On Developer
Mobile: +44 7771 818770
john@bovi.co.uk
Blog: http://sqlblogcasts.com/blogs/drjohn/
2. Agenda
What is CI?
SQL Server Data Tools (SSDT)
PowerShell and psake build tools
TeamCity Build Server
Nbi Test framework
SSDT and T4 Templates
3. Different approaches to
SQL database development
Migration Scripts
Manually prepare
upgrade scripts
Must be run in correct order
Scripts are combination
of DDL and data fixes
Manual deployment
Declarative Model
Define what we want
Tool creates migration
scripts
Still need data fixes
Automated deployments
4. Hands up
Who has entire BI solution under source code control?
Including SQL databases, all SSIS packages, all SSAS cubes
SSDT and the declarative model approach?
Who already uses CI in a data warehouse context?
Does CI also deploy & test your code?
5. What is Continuous Integration?
Continuous Integration is a development practice that
requires developers to integrate code into a shared
repository several times a day.
Each check-in is verified by an automated build,
deploy and test cycle which allows errors to be
detected and thereby corrected quickly.
6. Benefits of Continuous Integration
Eradication of manual integrations and deployments
Catches issues fast
Reduces production & staging errors
Automated testing provides quality assurance
CI allows developers to proceed with confidence
CI provides reporting on quality of the code base
Facilitates Continuous Delivery
7. Applying CI to the Data Warehouse
Why is it so difficult?
BI developers are not familiar with CI
Never seen it in action in a DWH context
Not familiar with the tools and techniques
Cannot comprehend how it can be applied
Data is a BIG barrier
PROD data is different to DEV
Problems often only manifest themselves in PROD data
Time taken to load data
8. Data Warehouse
Project started 2010
SQL 2008 R2 – tools & techniques have remained static
Various “initiatives” have resulted in a confused data architecture
Three competing data streams and lots of data marts
Overly complicated and complex!
100,000 files in the code-base!
14 database, 200 SSIS packages, 4 multidimensional cubes
7 developers, 4 testers, 3 BAs, 1 S2T, 1 DBA, 1 PM
Each database had 2 VSDT projects – split “storage” and “code”
Configuration scattered across code base
Manual build and very manual deploy – 1 guy, 4 days per month!
Each production release was getting longer, complex and error prone!
9. Data Warehouse
Imported all databases into SQL Server Data Tools
New SCC repository with simplified structure: only 8,900 files
PowerShell script written to do local build & deployment
Development tools
Visual Studio 2013 for SQL Server Data Tools (SSDT)
Visual Studio 2008 for SSIS and SSAS (servers still SQL 2008R2)
TeamCity build server
Creates single Nuget package containing all 14 databases, 200+
SSIS package and SSAS cubes plus deployment scripts
Automatically deploys and tests everything
Automatic deployment to test, UAT and other environments!
10. Key Considerations for CI
Deployable artifacts
What is generated from the code that I can deploy?
Deployment
How do I deploy the artifact?
Configuration
How can I set up the environment so that everything works?
11. Tools and techniques for CI
Source Code Control (SCC)
ALL code MUST be under Source Code Control
SQL Server, SSIS, BIML, SSAS, SSRS, Test Suites etc. etc.
Recommend Using Three Branches
1. Development branch
2. Release branch
3. Prod branch
SCC Locking strategy:
Use optimistic locking for SQL, C#
Use pessimistic locking for SSIS, SSAS and SSRS
12. SQL Server Data Tools
Declarative model design environment for SQL Server
Requires Visual Studio 2012+
Supports SQL Server 2005+
Very easy to import an existing database
Must replace hard-coded database names with variables
Add references to other databases and setup SQLCmd variables
Don’t upgrade VSDT projects
Instead start a new SSDT project and import a deployed
database instance
Then transfer over pre- and post-deploy scripts
17. SQL Server Data Tools
Build generates a DACPAC => “deployable artifact”
Contains entire database model
Conforms to Open Packaging Convention = ZIP file!
Use MsBuild to create DACPAC from solutions
Use SQLPackage.exe to deploy DACPAC
18. The many actions of
SQLPackage.exe
Publish
Deploys a DACPAC
Script
Generates a T-SQL
deployment script from
a DACPAC
Extract
Creates a DACPAC
from a database
Export
Creates a BACPAC
Import
Imports a BACPAC
DeployReport
List of changes new
deploy will make
DriftReport
List of changes since
last deploy
19. PowerShell
PowerShell is the “glue” for your CI project
Part of Windows Management Framework
4.0 built into Windows Server 2012 R2 and Windows 8.1
Download and install WMF 4.0 for prior OS versions
Advanced scripting
Proper error handling Try-Catch
Easy to read/write XML
Ideal for dynamically generation of config files
CmdLets
Invoke-SQLCmd
Invoke-ASCmd & SQLAS
20. psake
Task-orientated build tool written in PowerShell
Uses a dependency pattern similar to MSBuild or Make
Download from https://github.com/psake/psake
Build script consist of Tasks
Each task can depend on other tasks
31. Tools and techniques for CI
SSIS Artifacts
File Mode Deployment (pre-2012)
Build does nothing useful!
SSIS package => “deployable artifact”
Deployment simply copies packages into target location
File Mode Configuration
Update Package Configurations using PowerShell to:
Update XML config
Set SQLCmd variables and run a SQLCmd script which updates the
SSIS configuration table
32. Tools and techniques for CI
SSIS Artifacts
Project Mode Deployment (SQL Server 2012+)
Build generates an ISPAC => “deployable artifact”
Use IsDeploymentWizard.exe to deploy the packages
Project Mode Configuration
Use SSISDB catalog stored procedures to:
Create Projects & Folders
Create Environments & Variables
etc.
Run as SQLCmd script from PowerShell
33. SSAS Artifacts
Build
Cannot be done with MsBuild!
Must invoke Visual Studio (devenv.exe) to perform build
Windows EXE => so pipe to Out-Null to force PowerShell to wait
Generates a .AsDatabase file => “deployable artifact”
Deploy
Microsoft.AnalysisServices.Deployment.exe /s
PowerShell to generate XML config files defining target server etc.
Windows EXE => so pipe to Out-Null to force PowerShell to wait
SSAS Partition Manager
Deploys using AMO and also dynamically creates partitions
SsasPartitionManager.codeplex.com
34. NBi Test Suite
Open source framework to test your BI solutions
Tests run using NUnit GUI or command-line runner
Tests written in XML
Can be dynamically manipulated using PowerShell
Automatic generation of tests via:
Tools provided with NBi suite – GenBI and GenBIL
SELECT … FOR XML
PowerShell etc.
Run SQL, MDX and DAX queries
Extract queries from SSRS reports
Run SSIS packages and check “side effects”
Check model structure
35. NBi Test Suite
Extensive support for testing the model structure
36. NBi Test Suite
Compare results of a query
Static data set
CSV
Results of another query
Check datatypes, nulls, empty strings, format etc.
Apply tolerance and rounding
Validate SQL query syntax
Performance tests
Clear the cache in test setup
Check cube member count, ordering, existence etc.
39. Unit Testing with SSDT
Add a Unit Test C# project to your SSDT solution
40. Code Analysis with SSDT
T-SQL Design Issues
Checks for code that might not behave the way you expect
Deprecated syntax
Issues that could cause problems in the design
T-SQL Naming Issues
Naming issues arise when a database object violates generally
accepted conventions
T-SQL Performance Issues
Code that might noticeably reduce speed of database operations
Extensible model
Write your own Code Analysis
41. Top tips to using SSDT & CI
No User Permissions
Database roles only
Assign Windows Groups to Roles in a post-deploy script
Don’t deploy SQL Agent Jobs to CI server
Use PowerShell to call the commands instead
Hold SSIS configuration scripts outside SSDT
Hold pre- and post-deploy data fix scripts external to
SSDT in a “release” folder
42. SSDT + T4 Templates
T4 = Text Template Transformation Toolkit
Built into Visual Studio 2005+
Create code from code
Example: History tables and triggers
Old data should be written to a history table every time an update
done to the main table
Problem: 200 main tables = 200 history tables + 200 triggers
43. SSDT + T4 Templates
Three ways you could implement T4 Templates
1. Use ADO.NET to query catalog views on a materialized
(deployed) copy of the database
2. Use the Data-tier Application Framework (DACFx) to query the
content of the DACPAC generated by SSDT on your last build
3. Use the Data-tier Application Framework (DACFx) to query the
in-memory content of your SSDT project
Very few examples
Download my toolkit from t4dacfx2tsql.codeplex.com
Read my blog on http://sqlblogcasts.com/blogs/drjohn/
45. Continuous Integration
and the
Data Warehouse
Dr. John Tunnicliffe
Independent BI Architect &
Hands-On Developer
Mobile: +44 7771 818770
john@bovi.co.uk
Blog: http://sqlblogcasts.com/blogs/drjohn/