SlideShare a Scribd company logo
1 of 45
Agile Data Warehousing 
From Start to Finish 
Presenter: Davide Mauri, Architect & Mentor, SolidQ 
Moderator: Alex Whittles
Technical Assistance 
2 
If you require assistance 
during the session, type 
your inquiry into the 
question pane on the 
right side. 
Maximize your screen 
with the zoom button 
on the top of the 
presentation window 
Type your questions in 
the question pane on 
the right side
Thank You Sponsors 
Welcome to the Azure family! 
Try DocumentDB today! 
http://documentdb.com 
Solutions from Dell help you 
monitor, manage, protect and 
improve your SQL Server 
environment. 
http://software.dell.com/sql-pass-vc- 
dell-sql-server-solutions
www.PASSSummit.com 
Planning on attending PASS Summit 2014? Start saving 
today! 
• The world’s largest gathering of SQL Server & BI professionals 
• Take your SQL Server skills to the next level by learning from the world’s 
SQL Server experts, in 190+ technical sessions 
• Over 5000 attendees, representing 2000 companies, from 52 countries, 
ready to network & learn 
Use discount code 24HOP14 
to save $200! 
$1,895 
UNTIL SEPTEMBER 26, 
2014
Davide Mauri 
 SolidQ Mentor 
 Board of Directors, SolidQ Italy 
 Microsoft SQL Server MVP 
 Works with managers to build effective, 
tailor-made BI solutions for customers 
@mauridb
Agile Data Warehousing 
From Start to Finish 
Davide Mauri, Architect & Mentor, SolidQ
Agenda 
What is a DWH, really? 
Agile: the only way to succeed 
Engineering the DWH 
ETL Design Patterns 
ETL Automation 
Testing
What is a DWH, really?
The Data-Driven Age
Isn’t the DWH and “old” thing? 
Big Data, In Memory and all the new stuff, can’t just replace 
the Data Warehouse? 
The answer would be “yes”, if a DWH would be a simple 
“container” of data. 
But it’s much more than this.
What is a DWH, really? 
In this new era, data is like water. 
Who will ever drink from 
untested, untrusted, 
uncertified data?
What is a DWH, really? 
Would a manager or a decision maker, take a decision 
based on data of which he doesn’t know the source, the 
integrity and the correctness?
What is a DWH, really? 
The Data Warehouse is the place where managers and 
decision makers will look for 
• Correct 
• Trusted 
• Updated 
Data in order to make a 
conscious decision
What is a DWH, really? 
The answer is now easy:
What is DWH, really? 
A place to store consolidated data coming from the whole 
company 
A place where cleanse, verify and certify data 
A place where historic data is stored 
A place that holds the single version of truth (if there is one!) 
Forms the core of a BI solution 
User friendly Data models, designed to make data analysis 
easier
Modern Data Environment 
Master 
Data 
EDW 
Data Mart 
Big Data 
Unstructured 
Data 
BI Environment 
Analytics Environment 
Structured 
Data Data Scientist 
Decision Maker
Agility: the only way to succeed
EDW: Reality Check 
EDW is the trusted container of all company data 
It cannot be created in “one day” 
It has to grow and evolve with business needs. 
It will never be 100% complete
The story so far
Adapt to Survive 
“50% of requirements change in the first year of a BI 
project” 
Andreas Bitterer, Research VP, Gartner
Agile Principles 
Small design upfront. Prototype. 
Delivery quickly, Deliver frequently. 
Users are part of the development team! 
Feedback is a key part of the success 
They’ll grow with the solution and the solution will grow with them 
Embrace Changes! 
http://agilemanifesto.org/principles.html
Agile Challenges 
Delivery Quickly and Fast 
 Challenge: keep high quality, no matter who’s doing the work 
Embrace Changes 
 Challenge: don’t introduce bugs. Change the smallest part 
possible. Use automatic Testing to preserve and assure data 
quality.
Engineering the DWH
Engineering the solution 
To be Agile, some engineering practices needs to be included in 
our work model 
Agility != Anarchy 
Engineering: 
 Apply well-known models 
 Define, Apply & Enforce rules 
 Automate and/or Check rules application 
 Measure 
 Test 
2
Engineering the solution 
Favor Kimball Approach (for user-facing models) 
 Dimensional Modeling 
 Fact & Measures 
 Dimensions 
Use views to introduce abstraction layers 
 Reduce the “friction” between layers (source / stage / dwh / dm) 
 Apply the “Information Hiding Principle”
Engineering the solution 
Define & Force the application of well-known ETL patterns 
 SCD1 / SCD2 
 Incremental / Partition Load 
Divide Et Impera 
 At least two SSIS solutions 
 many small SSIS Packages 
 5 Databases (STG, CFG, LOG, MD, DWH)
Design Pattern 
“A general reusable solution 
to a commonly occurring 
problem within a given 
context”
Design Pattern 
Generic ETL Pattern 
 Partition Load 
 Incremental/Differential Load 
Generic DWH/BI Design Pattern 
 Slowly Changing Dimension 
 SCD1, SCD2, ecc. 
 Fact Table 
 Transactional, Snapshot, Temporal Snapshot
Design Pattern 
Specific SQL Server Patterns 
 Change Data Capture 
 Change Tracking 
 Partition Load 
 SSIS Parallelism
ETL Automation
No Monkey Work! 
Let the people think and let 
the machines do the 
«monkey» work.
Invest on Automation? 
Faster development 
 Reduce Costs 
 Embrace Changes 
Less bugs 
Increase solution quality and make it consistent throughout 
the whole product
Hi-Level Vision 
Technical Process 
Technical Process 
ETL ETL 
STG 
OLTP DWH 
ETL 
Business Process
ETL Phases 
«E» and «L» must be 
 Simple, Easy and Straightforward 
 Completely Automated 
 Completely Reusable 
«E» and «L» have ZERO value in a DWH Solution 
 Should be done in the most economic way
Automation Tools 
PowerShell / .NET 
 Supported by SMO & SSIS API 
 Microsoft creates platforms not only products! 
BIML – BI Markup Language 
 From Varigence 
 Free with BIDS Helper 
 Full support with MIST
Metadata 
Metadata is needed in order to make automation a 
repeatable process 
 Source to Staging Info 
 Staging to DWH info 
 Dimension Keys 
 Dimension & Fact Table relationship 
Extended Properties + SQL Server DMVs help to maintain 
metadata coherent
Unit Testing
Unit Testing 
Data MUST be tested. 
It’s like water, remember? 
If trust is lost, DWH is an 
#epicfail
Unit Testing 
Before releasing anything data in the DW must be tested. 
User has to validate a sample of data 
 (e.g.:total invoice amount of January 2012) 
That validated value will become the reference value 
Before release, the same query will be executed again. 
 If the data is the expected reference data then test is green 
 otherwise the test fails
Unit Testing 
Of course test MUST be automated when possible 
 Visual Studio 
 NUnit extensions 
 NBI 
 BI.Quality 
What to test? 
 Aggregated results 
 Specific values of some «special» rule 
 Fixed bugs/tickets 
4
The perfect BI process & architecture 
Iterative!
Questions?
Like What You Heard? 
Davide will be presenting at PASS Summit 2014! 
 PreConference: 
 Agile Data Warehousing: Start to Finish 
 General Session: 
 Agile BI: Unit Testing and Continuos Integration 
Use discount code 24HOP14 
to save $200! 
@mauridb
Coming up next … 
DAX Formulas in Action 
Alberto Ferrari
Thank You for Attending

More Related Content

What's hot

From Personal BI to Managed BI with Power BI
From Personal BI to Managed BI with Power BIFrom Personal BI to Managed BI with Power BI
From Personal BI to Managed BI with Power BIJean-Pierre Riehl
 
Spca2014 sp ci with tfs online and azure matthias einig
Spca2014 sp ci with tfs online and azure matthias einigSpca2014 sp ci with tfs online and azure matthias einig
Spca2014 sp ci with tfs online and azure matthias einigNCCOMMS
 
Tableau product overview 10.3
Tableau product overview 10.3Tableau product overview 10.3
Tableau product overview 10.3Irwan Syah
 
Top Five Cool Features in Oracle SQL Developer Data Modeler
Top Five Cool Features in Oracle SQL Developer Data ModelerTop Five Cool Features in Oracle SQL Developer Data Modeler
Top Five Cool Features in Oracle SQL Developer Data ModelerKent Graziano
 
PowerApps & Flow @ Power BI World Tour Copenhagen
PowerApps & Flow @ Power BI World Tour CopenhagenPowerApps & Flow @ Power BI World Tour Copenhagen
PowerApps & Flow @ Power BI World Tour CopenhagenDaniel Laskewitz
 
Tableau Drive, A new methodology for scaling your analytic culture
Tableau Drive, A new methodology for scaling your analytic cultureTableau Drive, A new methodology for scaling your analytic culture
Tableau Drive, A new methodology for scaling your analytic cultureTableau Software
 
Snowflake: The Good, the Bad and the Ugly
Snowflake: The Good, the Bad and the UglySnowflake: The Good, the Bad and the Ugly
Snowflake: The Good, the Bad and the UglySamanthaBerlant
 
ACCELERATE THE VALUE OF DATA ANALYTICS - TABLEAU SERVICES
ACCELERATE THE VALUE OF DATA ANALYTICS - TABLEAU SERVICESACCELERATE THE VALUE OF DATA ANALYTICS - TABLEAU SERVICES
ACCELERATE THE VALUE OF DATA ANALYTICS - TABLEAU SERVICESTaction Software LLC
 
Conceptional Data Vault
Conceptional Data VaultConceptional Data Vault
Conceptional Data VaultTorsten Glunde
 
Prototyping like it is 2022
Prototyping like it is 2022 Prototyping like it is 2022
Prototyping like it is 2022 Michael Yagudaev
 
O365Con19 - Office 365 Groups Surviving the Real World - Jasper Oosterveld
O365Con19 - Office 365 Groups Surviving the Real World - Jasper OosterveldO365Con19 - Office 365 Groups Surviving the Real World - Jasper Oosterveld
O365Con19 - Office 365 Groups Surviving the Real World - Jasper OosterveldNCCOMMS
 
5 Reasons not to use Dita from a CCMS Perspective
5 Reasons not to use Dita from a CCMS Perspective5 Reasons not to use Dita from a CCMS Perspective
5 Reasons not to use Dita from a CCMS PerspectiveMarcus Kesseler
 
20220205 Getting started with power bi
20220205 Getting started with power bi20220205 Getting started with power bi
20220205 Getting started with power biAroh Shukla
 
Don't Drop ACID - Data Love - April 2021
Don't Drop ACID - Data Love - April 2021Don't Drop ACID - Data Love - April 2021
Don't Drop ACID - Data Love - April 2021Matthew Groves
 
5 Reasons to Move Your BI to the Cloud
5 Reasons to Move Your BI to the Cloud5 Reasons to Move Your BI to the Cloud
5 Reasons to Move Your BI to the CloudTableau Software
 
Power BI Advanced Data Modeling Virtual Workshop
Power BI Advanced Data Modeling Virtual WorkshopPower BI Advanced Data Modeling Virtual Workshop
Power BI Advanced Data Modeling Virtual WorkshopCCG
 
Optimizing the DITA Authoring Experience
Optimizing the DITA Authoring ExperienceOptimizing the DITA Authoring Experience
Optimizing the DITA Authoring Experiencedclsocialmedia
 
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile ApproachUsing OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile ApproachKent Graziano
 
Why Migrate from MySQL to Cassandra
Why Migrate from MySQL to CassandraWhy Migrate from MySQL to Cassandra
Why Migrate from MySQL to CassandraDATAVERSITY
 

What's hot (20)

From Personal BI to Managed BI with Power BI
From Personal BI to Managed BI with Power BIFrom Personal BI to Managed BI with Power BI
From Personal BI to Managed BI with Power BI
 
Spca2014 sp ci with tfs online and azure matthias einig
Spca2014 sp ci with tfs online and azure matthias einigSpca2014 sp ci with tfs online and azure matthias einig
Spca2014 sp ci with tfs online and azure matthias einig
 
Tableau product overview 10.3
Tableau product overview 10.3Tableau product overview 10.3
Tableau product overview 10.3
 
Top Five Cool Features in Oracle SQL Developer Data Modeler
Top Five Cool Features in Oracle SQL Developer Data ModelerTop Five Cool Features in Oracle SQL Developer Data Modeler
Top Five Cool Features in Oracle SQL Developer Data Modeler
 
PowerApps & Flow @ Power BI World Tour Copenhagen
PowerApps & Flow @ Power BI World Tour CopenhagenPowerApps & Flow @ Power BI World Tour Copenhagen
PowerApps & Flow @ Power BI World Tour Copenhagen
 
Tableau Drive, A new methodology for scaling your analytic culture
Tableau Drive, A new methodology for scaling your analytic cultureTableau Drive, A new methodology for scaling your analytic culture
Tableau Drive, A new methodology for scaling your analytic culture
 
Snowflake: The Good, the Bad and the Ugly
Snowflake: The Good, the Bad and the UglySnowflake: The Good, the Bad and the Ugly
Snowflake: The Good, the Bad and the Ugly
 
ACCELERATE THE VALUE OF DATA ANALYTICS - TABLEAU SERVICES
ACCELERATE THE VALUE OF DATA ANALYTICS - TABLEAU SERVICESACCELERATE THE VALUE OF DATA ANALYTICS - TABLEAU SERVICES
ACCELERATE THE VALUE OF DATA ANALYTICS - TABLEAU SERVICES
 
Conceptional Data Vault
Conceptional Data VaultConceptional Data Vault
Conceptional Data Vault
 
Prototyping like it is 2022
Prototyping like it is 2022 Prototyping like it is 2022
Prototyping like it is 2022
 
Tableau vs PowerBI
Tableau vs PowerBITableau vs PowerBI
Tableau vs PowerBI
 
O365Con19 - Office 365 Groups Surviving the Real World - Jasper Oosterveld
O365Con19 - Office 365 Groups Surviving the Real World - Jasper OosterveldO365Con19 - Office 365 Groups Surviving the Real World - Jasper Oosterveld
O365Con19 - Office 365 Groups Surviving the Real World - Jasper Oosterveld
 
5 Reasons not to use Dita from a CCMS Perspective
5 Reasons not to use Dita from a CCMS Perspective5 Reasons not to use Dita from a CCMS Perspective
5 Reasons not to use Dita from a CCMS Perspective
 
20220205 Getting started with power bi
20220205 Getting started with power bi20220205 Getting started with power bi
20220205 Getting started with power bi
 
Don't Drop ACID - Data Love - April 2021
Don't Drop ACID - Data Love - April 2021Don't Drop ACID - Data Love - April 2021
Don't Drop ACID - Data Love - April 2021
 
5 Reasons to Move Your BI to the Cloud
5 Reasons to Move Your BI to the Cloud5 Reasons to Move Your BI to the Cloud
5 Reasons to Move Your BI to the Cloud
 
Power BI Advanced Data Modeling Virtual Workshop
Power BI Advanced Data Modeling Virtual WorkshopPower BI Advanced Data Modeling Virtual Workshop
Power BI Advanced Data Modeling Virtual Workshop
 
Optimizing the DITA Authoring Experience
Optimizing the DITA Authoring ExperienceOptimizing the DITA Authoring Experience
Optimizing the DITA Authoring Experience
 
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile ApproachUsing OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
 
Why Migrate from MySQL to Cassandra
Why Migrate from MySQL to CassandraWhy Migrate from MySQL to Cassandra
Why Migrate from MySQL to Cassandra
 

Viewers also liked

SSIS Monitoring Deep Dive
SSIS Monitoring Deep DiveSSIS Monitoring Deep Dive
SSIS Monitoring Deep DiveDavide Mauri
 
Datarace: IoT e Big Data (Italian)
Datarace: IoT e Big Data (Italian)Datarace: IoT e Big Data (Italian)
Datarace: IoT e Big Data (Italian)Davide Mauri
 
Iris Multi-Class Classifier with Azure ML
Iris Multi-Class Classifier with Azure MLIris Multi-Class Classifier with Azure ML
Iris Multi-Class Classifier with Azure MLDavide Mauri
 
Azure Machine Learning
Azure Machine LearningAzure Machine Learning
Azure Machine LearningDavide Mauri
 
SQL Server 2016 Temporal Tables
SQL Server 2016 Temporal TablesSQL Server 2016 Temporal Tables
SQL Server 2016 Temporal TablesDavide Mauri
 
AzureML - Creating and Using Machine Learning Solutions (Italian)
AzureML - Creating and Using Machine Learning Solutions (Italian)AzureML - Creating and Using Machine Learning Solutions (Italian)
AzureML - Creating and Using Machine Learning Solutions (Italian)Davide Mauri
 
Real Time Power BI
Real Time Power BIReal Time Power BI
Real Time Power BIDavide Mauri
 
Getting started with Scrum
Getting started with ScrumGetting started with Scrum
Getting started with ScrumTecsisa
 
SQL Server 2016 What's New For Developers
SQL Server 2016  What's New For DevelopersSQL Server 2016  What's New For Developers
SQL Server 2016 What's New For DevelopersDavide Mauri
 
AzureDay - Introduction Big Data Analytics.
AzureDay  - Introduction Big Data Analytics.AzureDay  - Introduction Big Data Analytics.
AzureDay - Introduction Big Data Analytics.Łukasz Grala
 
Dashboarding with Microsoft: Datazen & Power BI
Dashboarding with Microsoft: Datazen & Power BIDashboarding with Microsoft: Datazen & Power BI
Dashboarding with Microsoft: Datazen & Power BIDavide Mauri
 
Introduction to Big Data & Analytics
Introduction to Big Data & AnalyticsIntroduction to Big Data & Analytics
Introduction to Big Data & AnalyticsPrasad Chitta
 
Azure Machine Learning (Italian)
Azure Machine Learning (Italian)Azure Machine Learning (Italian)
Azure Machine Learning (Italian)Davide Mauri
 
Temporal Snapshot Fact Tables
Temporal Snapshot Fact TablesTemporal Snapshot Fact Tables
Temporal Snapshot Fact TablesDavide Mauri
 
On Big Data Analytics - opportunities and challenges
On Big Data Analytics - opportunities and challengesOn Big Data Analytics - opportunities and challenges
On Big Data Analytics - opportunities and challengesPetteri Alahuhta
 
Agile data warehouse
Agile data warehouseAgile data warehouse
Agile data warehouseDao Vo
 
Azure ML: from basic to integration with custom applications
Azure ML: from basic to integration with custom applicationsAzure ML: from basic to integration with custom applications
Azure ML: from basic to integration with custom applicationsDavide Mauri
 
Bancos colombia
Bancos colombiaBancos colombia
Bancos colombiaivanhhh
 
Agile Data Warehouse Design for Big Data Presentation
Agile Data Warehouse Design for Big Data PresentationAgile Data Warehouse Design for Big Data Presentation
Agile Data Warehouse Design for Big Data PresentationVishal Kumar
 

Viewers also liked (20)

SSIS Monitoring Deep Dive
SSIS Monitoring Deep DiveSSIS Monitoring Deep Dive
SSIS Monitoring Deep Dive
 
Datarace: IoT e Big Data (Italian)
Datarace: IoT e Big Data (Italian)Datarace: IoT e Big Data (Italian)
Datarace: IoT e Big Data (Italian)
 
Iris Multi-Class Classifier with Azure ML
Iris Multi-Class Classifier with Azure MLIris Multi-Class Classifier with Azure ML
Iris Multi-Class Classifier with Azure ML
 
Azure Machine Learning
Azure Machine LearningAzure Machine Learning
Azure Machine Learning
 
SQL Server 2016 Temporal Tables
SQL Server 2016 Temporal TablesSQL Server 2016 Temporal Tables
SQL Server 2016 Temporal Tables
 
Data modeling facts
Data modeling factsData modeling facts
Data modeling facts
 
AzureML - Creating and Using Machine Learning Solutions (Italian)
AzureML - Creating and Using Machine Learning Solutions (Italian)AzureML - Creating and Using Machine Learning Solutions (Italian)
AzureML - Creating and Using Machine Learning Solutions (Italian)
 
Real Time Power BI
Real Time Power BIReal Time Power BI
Real Time Power BI
 
Getting started with Scrum
Getting started with ScrumGetting started with Scrum
Getting started with Scrum
 
SQL Server 2016 What's New For Developers
SQL Server 2016  What's New For DevelopersSQL Server 2016  What's New For Developers
SQL Server 2016 What's New For Developers
 
AzureDay - Introduction Big Data Analytics.
AzureDay  - Introduction Big Data Analytics.AzureDay  - Introduction Big Data Analytics.
AzureDay - Introduction Big Data Analytics.
 
Dashboarding with Microsoft: Datazen & Power BI
Dashboarding with Microsoft: Datazen & Power BIDashboarding with Microsoft: Datazen & Power BI
Dashboarding with Microsoft: Datazen & Power BI
 
Introduction to Big Data & Analytics
Introduction to Big Data & AnalyticsIntroduction to Big Data & Analytics
Introduction to Big Data & Analytics
 
Azure Machine Learning (Italian)
Azure Machine Learning (Italian)Azure Machine Learning (Italian)
Azure Machine Learning (Italian)
 
Temporal Snapshot Fact Tables
Temporal Snapshot Fact TablesTemporal Snapshot Fact Tables
Temporal Snapshot Fact Tables
 
On Big Data Analytics - opportunities and challenges
On Big Data Analytics - opportunities and challengesOn Big Data Analytics - opportunities and challenges
On Big Data Analytics - opportunities and challenges
 
Agile data warehouse
Agile data warehouseAgile data warehouse
Agile data warehouse
 
Azure ML: from basic to integration with custom applications
Azure ML: from basic to integration with custom applicationsAzure ML: from basic to integration with custom applications
Azure ML: from basic to integration with custom applications
 
Bancos colombia
Bancos colombiaBancos colombia
Bancos colombia
 
Agile Data Warehouse Design for Big Data Presentation
Agile Data Warehouse Design for Big Data PresentationAgile Data Warehouse Design for Big Data Presentation
Agile Data Warehouse Design for Big Data Presentation
 

Similar to Agile Data Warehousing

Agile Data Warehousing
Agile Data WarehousingAgile Data Warehousing
Agile Data WarehousingDavide Mauri
 
Big Data Made Easy: A Simple, Scalable Solution for Getting Started with Hadoop
Big Data Made Easy:  A Simple, Scalable Solution for Getting Started with HadoopBig Data Made Easy:  A Simple, Scalable Solution for Getting Started with Hadoop
Big Data Made Easy: A Simple, Scalable Solution for Getting Started with HadoopPrecisely
 
DataOps , cbuswaw April '23
DataOps , cbuswaw April '23DataOps , cbuswaw April '23
DataOps , cbuswaw April '23Jason Packer
 
The Right Data Warehouse: Automation Now, Business Value Thereafter
The Right Data Warehouse: Automation Now, Business Value ThereafterThe Right Data Warehouse: Automation Now, Business Value Thereafter
The Right Data Warehouse: Automation Now, Business Value ThereafterInside Analysis
 
How to implement hadoop successfuly
How to implement hadoop successfulyHow to implement hadoop successfuly
How to implement hadoop successfulyAdir Sharabi
 
How to implement Hadoop successfully
How to implement Hadoop successfullyHow to implement Hadoop successfully
How to implement Hadoop successfullyAdir Sharabi
 
Self-Service Analytics with Guard Rails
Self-Service Analytics with Guard RailsSelf-Service Analytics with Guard Rails
Self-Service Analytics with Guard RailsDenodo
 
Tdwi march 2015 presentation
Tdwi march 2015 presentationTdwi march 2015 presentation
Tdwi march 2015 presentationAlison Macfie
 
2015 02 12 talend hortonworks webinar challenges to hadoop adoption
2015 02 12 talend hortonworks webinar challenges to hadoop adoption2015 02 12 talend hortonworks webinar challenges to hadoop adoption
2015 02 12 talend hortonworks webinar challenges to hadoop adoptionHortonworks
 
Innovative Data Strategies for Advanced Analytics Solutions and the Role of D...
Innovative Data Strategies for Advanced Analytics Solutions and the Role of D...Innovative Data Strategies for Advanced Analytics Solutions and the Role of D...
Innovative Data Strategies for Advanced Analytics Solutions and the Role of D...Denodo
 
Enable Better Decision Making with Power BI Visualizations & Modern Data Estate
Enable Better Decision Making with Power BI Visualizations & Modern Data EstateEnable Better Decision Making with Power BI Visualizations & Modern Data Estate
Enable Better Decision Making with Power BI Visualizations & Modern Data EstateCCG
 
Business in the Driver’s Seat – An Improved Model for Integration
Business in the Driver’s Seat – An Improved Model for IntegrationBusiness in the Driver’s Seat – An Improved Model for Integration
Business in the Driver’s Seat – An Improved Model for IntegrationInside Analysis
 
Building The Agile Database
Building The Agile DatabaseBuilding The Agile Database
Building The Agile Databaseelliando dias
 
Big Data's Impact on the Enterprise
Big Data's Impact on the EnterpriseBig Data's Impact on the Enterprise
Big Data's Impact on the EnterpriseCaserta
 
Power to the People: A Stack to Empower Every User to Make Data-Driven Decisions
Power to the People: A Stack to Empower Every User to Make Data-Driven DecisionsPower to the People: A Stack to Empower Every User to Make Data-Driven Decisions
Power to the People: A Stack to Empower Every User to Make Data-Driven DecisionsLooker
 
DevOps and the DBA- 24 Hours of Pass
DevOps and the DBA-  24 Hours of PassDevOps and the DBA-  24 Hours of Pass
DevOps and the DBA- 24 Hours of PassKellyn Pot'Vin-Gorman
 
OCSL - VMware, vSphere Webinar May 2013
OCSL - VMware, vSphere Webinar May 2013OCSL - VMware, vSphere Webinar May 2013
OCSL - VMware, vSphere Webinar May 2013OCSL
 
Introduction To SQL Server 2014
Introduction To SQL Server 2014Introduction To SQL Server 2014
Introduction To SQL Server 2014Vishal Pawar
 
Slalom DevOps Automacon 2016
Slalom DevOps Automacon 2016Slalom DevOps Automacon 2016
Slalom DevOps Automacon 2016George Peden
 

Similar to Agile Data Warehousing (20)

Agile Data Warehousing
Agile Data WarehousingAgile Data Warehousing
Agile Data Warehousing
 
Operational Data Vault
Operational Data VaultOperational Data Vault
Operational Data Vault
 
Big Data Made Easy: A Simple, Scalable Solution for Getting Started with Hadoop
Big Data Made Easy:  A Simple, Scalable Solution for Getting Started with HadoopBig Data Made Easy:  A Simple, Scalable Solution for Getting Started with Hadoop
Big Data Made Easy: A Simple, Scalable Solution for Getting Started with Hadoop
 
DataOps , cbuswaw April '23
DataOps , cbuswaw April '23DataOps , cbuswaw April '23
DataOps , cbuswaw April '23
 
The Right Data Warehouse: Automation Now, Business Value Thereafter
The Right Data Warehouse: Automation Now, Business Value ThereafterThe Right Data Warehouse: Automation Now, Business Value Thereafter
The Right Data Warehouse: Automation Now, Business Value Thereafter
 
How to implement hadoop successfuly
How to implement hadoop successfulyHow to implement hadoop successfuly
How to implement hadoop successfuly
 
How to implement Hadoop successfully
How to implement Hadoop successfullyHow to implement Hadoop successfully
How to implement Hadoop successfully
 
Self-Service Analytics with Guard Rails
Self-Service Analytics with Guard RailsSelf-Service Analytics with Guard Rails
Self-Service Analytics with Guard Rails
 
Tdwi march 2015 presentation
Tdwi march 2015 presentationTdwi march 2015 presentation
Tdwi march 2015 presentation
 
2015 02 12 talend hortonworks webinar challenges to hadoop adoption
2015 02 12 talend hortonworks webinar challenges to hadoop adoption2015 02 12 talend hortonworks webinar challenges to hadoop adoption
2015 02 12 talend hortonworks webinar challenges to hadoop adoption
 
Innovative Data Strategies for Advanced Analytics Solutions and the Role of D...
Innovative Data Strategies for Advanced Analytics Solutions and the Role of D...Innovative Data Strategies for Advanced Analytics Solutions and the Role of D...
Innovative Data Strategies for Advanced Analytics Solutions and the Role of D...
 
Enable Better Decision Making with Power BI Visualizations & Modern Data Estate
Enable Better Decision Making with Power BI Visualizations & Modern Data EstateEnable Better Decision Making with Power BI Visualizations & Modern Data Estate
Enable Better Decision Making with Power BI Visualizations & Modern Data Estate
 
Business in the Driver’s Seat – An Improved Model for Integration
Business in the Driver’s Seat – An Improved Model for IntegrationBusiness in the Driver’s Seat – An Improved Model for Integration
Business in the Driver’s Seat – An Improved Model for Integration
 
Building The Agile Database
Building The Agile DatabaseBuilding The Agile Database
Building The Agile Database
 
Big Data's Impact on the Enterprise
Big Data's Impact on the EnterpriseBig Data's Impact on the Enterprise
Big Data's Impact on the Enterprise
 
Power to the People: A Stack to Empower Every User to Make Data-Driven Decisions
Power to the People: A Stack to Empower Every User to Make Data-Driven DecisionsPower to the People: A Stack to Empower Every User to Make Data-Driven Decisions
Power to the People: A Stack to Empower Every User to Make Data-Driven Decisions
 
DevOps and the DBA- 24 Hours of Pass
DevOps and the DBA-  24 Hours of PassDevOps and the DBA-  24 Hours of Pass
DevOps and the DBA- 24 Hours of Pass
 
OCSL - VMware, vSphere Webinar May 2013
OCSL - VMware, vSphere Webinar May 2013OCSL - VMware, vSphere Webinar May 2013
OCSL - VMware, vSphere Webinar May 2013
 
Introduction To SQL Server 2014
Introduction To SQL Server 2014Introduction To SQL Server 2014
Introduction To SQL Server 2014
 
Slalom DevOps Automacon 2016
Slalom DevOps Automacon 2016Slalom DevOps Automacon 2016
Slalom DevOps Automacon 2016
 

More from Davide Mauri

Azure serverless Full-Stack kickstart
Azure serverless Full-Stack kickstartAzure serverless Full-Stack kickstart
Azure serverless Full-Stack kickstartDavide Mauri
 
Dapper: the microORM that will change your life
Dapper: the microORM that will change your lifeDapper: the microORM that will change your life
Dapper: the microORM that will change your lifeDavide Mauri
 
When indexes are not enough
When indexes are not enoughWhen indexes are not enough
When indexes are not enoughDavide Mauri
 
Building a Real-Time IoT monitoring application with Azure
Building a Real-Time IoT monitoring application with AzureBuilding a Real-Time IoT monitoring application with Azure
Building a Real-Time IoT monitoring application with AzureDavide Mauri
 
SSIS Monitoring Deep Dive
SSIS Monitoring Deep DiveSSIS Monitoring Deep Dive
SSIS Monitoring Deep DiveDavide Mauri
 
Azure SQL & SQL Server 2016 JSON
Azure SQL & SQL Server 2016 JSONAzure SQL & SQL Server 2016 JSON
Azure SQL & SQL Server 2016 JSONDavide Mauri
 
SQL Server & SQL Azure Temporal Tables - V2
SQL Server & SQL Azure Temporal Tables - V2SQL Server & SQL Azure Temporal Tables - V2
SQL Server & SQL Azure Temporal Tables - V2Davide Mauri
 
Azure Stream Analytics
Azure Stream AnalyticsAzure Stream Analytics
Azure Stream AnalyticsDavide Mauri
 
Event Hub & Azure Stream Analytics
Event Hub & Azure Stream AnalyticsEvent Hub & Azure Stream Analytics
Event Hub & Azure Stream AnalyticsDavide Mauri
 
SQL Server 2016 JSON
SQL Server 2016 JSONSQL Server 2016 JSON
SQL Server 2016 JSONDavide Mauri
 
Back to the roots - SQL Server Indexing
Back to the roots - SQL Server IndexingBack to the roots - SQL Server Indexing
Back to the roots - SQL Server IndexingDavide Mauri
 
Schema less table & dynamic schema
Schema less table & dynamic schemaSchema less table & dynamic schema
Schema less table & dynamic schemaDavide Mauri
 
Data Science Overview
Data Science OverviewData Science Overview
Data Science OverviewDavide Mauri
 
Delayed durability
Delayed durabilityDelayed durability
Delayed durabilityDavide Mauri
 
Hekaton: In-memory tables
Hekaton: In-memory tablesHekaton: In-memory tables
Hekaton: In-memory tablesDavide Mauri
 
Hardware planning & sizing for sql server
Hardware planning & sizing for sql serverHardware planning & sizing for sql server
Hardware planning & sizing for sql serverDavide Mauri
 

More from Davide Mauri (17)

Azure serverless Full-Stack kickstart
Azure serverless Full-Stack kickstartAzure serverless Full-Stack kickstart
Azure serverless Full-Stack kickstart
 
Dapper: the microORM that will change your life
Dapper: the microORM that will change your lifeDapper: the microORM that will change your life
Dapper: the microORM that will change your life
 
When indexes are not enough
When indexes are not enoughWhen indexes are not enough
When indexes are not enough
 
Building a Real-Time IoT monitoring application with Azure
Building a Real-Time IoT monitoring application with AzureBuilding a Real-Time IoT monitoring application with Azure
Building a Real-Time IoT monitoring application with Azure
 
SSIS Monitoring Deep Dive
SSIS Monitoring Deep DiveSSIS Monitoring Deep Dive
SSIS Monitoring Deep Dive
 
Azure SQL & SQL Server 2016 JSON
Azure SQL & SQL Server 2016 JSONAzure SQL & SQL Server 2016 JSON
Azure SQL & SQL Server 2016 JSON
 
SQL Server & SQL Azure Temporal Tables - V2
SQL Server & SQL Azure Temporal Tables - V2SQL Server & SQL Azure Temporal Tables - V2
SQL Server & SQL Azure Temporal Tables - V2
 
Azure Stream Analytics
Azure Stream AnalyticsAzure Stream Analytics
Azure Stream Analytics
 
Event Hub & Azure Stream Analytics
Event Hub & Azure Stream AnalyticsEvent Hub & Azure Stream Analytics
Event Hub & Azure Stream Analytics
 
SQL Server 2016 JSON
SQL Server 2016 JSONSQL Server 2016 JSON
SQL Server 2016 JSON
 
Back to the roots - SQL Server Indexing
Back to the roots - SQL Server IndexingBack to the roots - SQL Server Indexing
Back to the roots - SQL Server Indexing
 
Schema less table & dynamic schema
Schema less table & dynamic schemaSchema less table & dynamic schema
Schema less table & dynamic schema
 
Data juice
Data juiceData juice
Data juice
 
Data Science Overview
Data Science OverviewData Science Overview
Data Science Overview
 
Delayed durability
Delayed durabilityDelayed durability
Delayed durability
 
Hekaton: In-memory tables
Hekaton: In-memory tablesHekaton: In-memory tables
Hekaton: In-memory tables
 
Hardware planning & sizing for sql server
Hardware planning & sizing for sql serverHardware planning & sizing for sql server
Hardware planning & sizing for sql server
 

Recently uploaded

Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingScyllaDB
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data PlatformLess Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data PlatformWSO2
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...caitlingebhard1
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxFIDO Alliance
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Victor Rentea
 
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....rightmanforbloodline
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMKumar Satyam
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard37
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuidePixlogix Infotech
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingWSO2
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxFIDO Alliance
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptxFIDO Alliance
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDropbox
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
API Governance and Monetization - The evolution of API governance
API Governance and Monetization -  The evolution of API governanceAPI Governance and Monetization -  The evolution of API governance
API Governance and Monetization - The evolution of API governanceWSO2
 

Recently uploaded (20)

Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data PlatformLess Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation Computing
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
API Governance and Monetization - The evolution of API governance
API Governance and Monetization -  The evolution of API governanceAPI Governance and Monetization -  The evolution of API governance
API Governance and Monetization - The evolution of API governance
 

Agile Data Warehousing

  • 1. Agile Data Warehousing From Start to Finish Presenter: Davide Mauri, Architect & Mentor, SolidQ Moderator: Alex Whittles
  • 2. Technical Assistance 2 If you require assistance during the session, type your inquiry into the question pane on the right side. Maximize your screen with the zoom button on the top of the presentation window Type your questions in the question pane on the right side
  • 3. Thank You Sponsors Welcome to the Azure family! Try DocumentDB today! http://documentdb.com Solutions from Dell help you monitor, manage, protect and improve your SQL Server environment. http://software.dell.com/sql-pass-vc- dell-sql-server-solutions
  • 4. www.PASSSummit.com Planning on attending PASS Summit 2014? Start saving today! • The world’s largest gathering of SQL Server & BI professionals • Take your SQL Server skills to the next level by learning from the world’s SQL Server experts, in 190+ technical sessions • Over 5000 attendees, representing 2000 companies, from 52 countries, ready to network & learn Use discount code 24HOP14 to save $200! $1,895 UNTIL SEPTEMBER 26, 2014
  • 5. Davide Mauri  SolidQ Mentor  Board of Directors, SolidQ Italy  Microsoft SQL Server MVP  Works with managers to build effective, tailor-made BI solutions for customers @mauridb
  • 6. Agile Data Warehousing From Start to Finish Davide Mauri, Architect & Mentor, SolidQ
  • 7. Agenda What is a DWH, really? Agile: the only way to succeed Engineering the DWH ETL Design Patterns ETL Automation Testing
  • 8. What is a DWH, really?
  • 10. Isn’t the DWH and “old” thing? Big Data, In Memory and all the new stuff, can’t just replace the Data Warehouse? The answer would be “yes”, if a DWH would be a simple “container” of data. But it’s much more than this.
  • 11. What is a DWH, really? In this new era, data is like water. Who will ever drink from untested, untrusted, uncertified data?
  • 12. What is a DWH, really? Would a manager or a decision maker, take a decision based on data of which he doesn’t know the source, the integrity and the correctness?
  • 13. What is a DWH, really? The Data Warehouse is the place where managers and decision makers will look for • Correct • Trusted • Updated Data in order to make a conscious decision
  • 14. What is a DWH, really? The answer is now easy:
  • 15. What is DWH, really? A place to store consolidated data coming from the whole company A place where cleanse, verify and certify data A place where historic data is stored A place that holds the single version of truth (if there is one!) Forms the core of a BI solution User friendly Data models, designed to make data analysis easier
  • 16. Modern Data Environment Master Data EDW Data Mart Big Data Unstructured Data BI Environment Analytics Environment Structured Data Data Scientist Decision Maker
  • 17. Agility: the only way to succeed
  • 18. EDW: Reality Check EDW is the trusted container of all company data It cannot be created in “one day” It has to grow and evolve with business needs. It will never be 100% complete
  • 20. Adapt to Survive “50% of requirements change in the first year of a BI project” Andreas Bitterer, Research VP, Gartner
  • 21. Agile Principles Small design upfront. Prototype. Delivery quickly, Deliver frequently. Users are part of the development team! Feedback is a key part of the success They’ll grow with the solution and the solution will grow with them Embrace Changes! http://agilemanifesto.org/principles.html
  • 22. Agile Challenges Delivery Quickly and Fast  Challenge: keep high quality, no matter who’s doing the work Embrace Changes  Challenge: don’t introduce bugs. Change the smallest part possible. Use automatic Testing to preserve and assure data quality.
  • 24. Engineering the solution To be Agile, some engineering practices needs to be included in our work model Agility != Anarchy Engineering:  Apply well-known models  Define, Apply & Enforce rules  Automate and/or Check rules application  Measure  Test 2
  • 25. Engineering the solution Favor Kimball Approach (for user-facing models)  Dimensional Modeling  Fact & Measures  Dimensions Use views to introduce abstraction layers  Reduce the “friction” between layers (source / stage / dwh / dm)  Apply the “Information Hiding Principle”
  • 26. Engineering the solution Define & Force the application of well-known ETL patterns  SCD1 / SCD2  Incremental / Partition Load Divide Et Impera  At least two SSIS solutions  many small SSIS Packages  5 Databases (STG, CFG, LOG, MD, DWH)
  • 27. Design Pattern “A general reusable solution to a commonly occurring problem within a given context”
  • 28. Design Pattern Generic ETL Pattern  Partition Load  Incremental/Differential Load Generic DWH/BI Design Pattern  Slowly Changing Dimension  SCD1, SCD2, ecc.  Fact Table  Transactional, Snapshot, Temporal Snapshot
  • 29. Design Pattern Specific SQL Server Patterns  Change Data Capture  Change Tracking  Partition Load  SSIS Parallelism
  • 31. No Monkey Work! Let the people think and let the machines do the «monkey» work.
  • 32. Invest on Automation? Faster development  Reduce Costs  Embrace Changes Less bugs Increase solution quality and make it consistent throughout the whole product
  • 33. Hi-Level Vision Technical Process Technical Process ETL ETL STG OLTP DWH ETL Business Process
  • 34. ETL Phases «E» and «L» must be  Simple, Easy and Straightforward  Completely Automated  Completely Reusable «E» and «L» have ZERO value in a DWH Solution  Should be done in the most economic way
  • 35. Automation Tools PowerShell / .NET  Supported by SMO & SSIS API  Microsoft creates platforms not only products! BIML – BI Markup Language  From Varigence  Free with BIDS Helper  Full support with MIST
  • 36. Metadata Metadata is needed in order to make automation a repeatable process  Source to Staging Info  Staging to DWH info  Dimension Keys  Dimension & Fact Table relationship Extended Properties + SQL Server DMVs help to maintain metadata coherent
  • 38. Unit Testing Data MUST be tested. It’s like water, remember? If trust is lost, DWH is an #epicfail
  • 39. Unit Testing Before releasing anything data in the DW must be tested. User has to validate a sample of data  (e.g.:total invoice amount of January 2012) That validated value will become the reference value Before release, the same query will be executed again.  If the data is the expected reference data then test is green  otherwise the test fails
  • 40. Unit Testing Of course test MUST be automated when possible  Visual Studio  NUnit extensions  NBI  BI.Quality What to test?  Aggregated results  Specific values of some «special» rule  Fixed bugs/tickets 4
  • 41. The perfect BI process & architecture Iterative!
  • 43. Like What You Heard? Davide will be presenting at PASS Summit 2014!  PreConference:  Agile Data Warehousing: Start to Finish  General Session:  Agile BI: Unit Testing and Continuos Integration Use discount code 24HOP14 to save $200! @mauridb
  • 44. Coming up next … DAX Formulas in Action Alberto Ferrari
  • 45. Thank You for Attending

Editor's Notes

  1. Welcome to 24 hours of PASS: Summit Preview! We’re excited you could join us today for Davide Mauri’s session, Agile Data Warehousing: Start to Finish. This 24 Hours of PASS event consists of 24 consecutive live webcasts. Sessions will be recorded and posted online soon after the event. My name is Alex Whittles [add brief intro about yourself] and I have a few quick introduction slides before I hand over the reigns to Davide. He will speak for 40-45 minutes and then we’ll move on to the Q&A where you can ask any questions you may have. [move to next slide]
  2. If you’re having any issues, type your issue into the question pane and someone will assist you. To maximize your screen, use the zoom button located on the top of the presentation window. Feel free to enter your questions in the Q&A field at any time. The questions pane is located on the right side of your screen. Once we get to the Q&A portion of the session, I’ll read off your questions to the speaker. Note that there will be a short evaluation at the end of the session, your feedback is important to us so please take a moment to complete it. It will show up on your screen. [Note to moderators: You need to determine which questions are the most relevant and ask them out loud to the presenter].
  3. I’d like to take a moment to thank our event partners. The staging of 24 Hours of PASS would not be possible without their support and dedication, they are the reason this event is available free of charge. Thank you to our Presenting Sponsors: Microsoft and Dell Software. Move to next slide
  4. Next, as you all may now, this 24 Hours of PASS is a preview of PASS Summit 2014, the largest conference for SQL Server and BI professionals. With over 5000 attendees representing 2000 companies, from 52 countries, Summit is a time to share, connect and learn with your peers and industry partners. PASS Summit is not only a week of intensive learning and knowledge sharing that’ll offer strategic insights, it’s a time to network and rub shoulders with industry experts. Taking place in Seattle, WA from November 4-7, PASS Summit will feature over 190 world-class sessions across 5 topic tracks. These 24 Hours of PASS sessions provide a mere glimpse of what you can expect from PASS Summit. Find out more at www.passSummit.com and if you register by September 26 using discount code 24HOP14, you’ll get $200 off the registration fee. [move to next slide]
  5. And now, please allow me to present the speakers of the hour: Davide Mauri [move to next slide, speaker’s presentation]
  6. http://en.wikipedia.org/wiki/Software_design_pattern
  7. http://en.wikipedia.org/wiki/Software_design_pattern
  8. http://en.wikipedia.org/wiki/Software_design_pattern
  9. http://chartporn.org/2012/05/10/repetitive-tasks/
  10. Like what you heard here? Davide will be presenting at PASS Summit 2014: catch Davide in his general session, Agile BI: Unit Testing and Continuos Integration and the full presentation of this PreConference, Agile Data Warehousing: Start to Finish at PASS Summit 2014. And don’t forget to use the discount code 24HOP14 to save $200 on PASS Summit registration.
  11. Stay tuned for our next session, DAX Formulas in Action with Alberto Ferrari, happening in a couple of minutes.