Distributing Data Techniques with 
Distributing Data Techniques with 
Distributing Data Techniques with
       SQL Server ...
Distributing Data Techniques 
Distributing Data Techniques
  with SQL Server 2008 R2




       Copyright 2010 All Rights ...
Agenda
                        g
• Determining  where data is needed

• What SQL Server options do you have

• How to get ...
Reasons to Distribute Data
Reasons to Distribute Data
SQL Server 2008                               SQL Server 2008
                            Reporting/ODS
      Primary     ...
Distributing periodic updates to
 Data Marts from a “master” Data Warehouse



Data
Mart
                                 ...
Enabling a Data Warehouse
    from Legacy Systems
                              ORACLE



    Master
Data Warehouse


    ...
Initialization of tables                      Heterogeneous
for a heterogeneous Subscriber                        Subscrib...
Pulling in Sales Order information
Sales Force Automation         from disparate sources
      Application         for imp...
Data Mart




  OLTP
 Database                              SQL Server
                                         2008




S...
Connection



          Results                  Results

                      SQL Server             SQL Server
        ...
Decision‐Tree Path Traversal
                                                      Database
                              ...
Data         Log      Database Mirroring   Linked                 Backup/   Detach/      Sync
                            ...
Data Replication
Data Replication
Central Publisher
SQL Server                               SQL Server
  2008                (default option)     2005

   ...
Central Publisher
SQL Server
  19FIG11           Remote Distributor    SQL Server
  2008                                  ...
Publishing Subscriber
SQL Server                                             SQL Server
  2008                            ...
Publication SQL Server                                        SQL Server




                                             ...
Multiple Publishers
                                                    of a Single Table
19FIG14Server
           on




...
SQL Server
                                                           Updating
                                           ...
SQL Server                                                                     SQL Server
   2008                         ...
Publisher




                                                                                  Publication
              ...
Publisher



                                                                                       Publication
          ...
Publisher



                                                                                       Publication

         ...
AW_H_and_J (Publication)

                                                                                                ...
Publisher
                                    AW (Publication)
                                       (P bli ti )         ...
Data Access   Latency
                        Distributing Data
                        Distributing Data
                ...
Log Shipping
 Yes, it is still here..
    ,
Log Shipping
                                       “Source”
                                      SQL Server 2008
       ...
Database Mirroring
   with Snapshots
   with Snapshots
Client         Client     Client      Client


                                   Network

                         A

   ...
SQL Server 2008                                   SQL Server 2008
                                                        ...
SQL Server 2008

               01AdventureWork
          Source                                 CREATE DB ‘xyz’
         ...
Source Data
                         Pages

  Source

    02
SQL Server 2008


     SQL                             Sparse...
Source
                             Data
                            Pages

       Source
         03
     SQL Server 2008...
Snapshot
                            Source Data                   Users
                              Pages
             ...
SQL Server 2008                                   SQL Server 2008
                                                        ...
PH Topology With Snapshots
                                                                                               ...
Publisher                                        DB Mirror and 
SQL Server 2008       SQL Server 2008                     ...
Integration Services
Excel
                                         OLAP
                                    Data Warehouse

   Access




    ...
Package
P k
                Task          Task


                Task



        ainer
                          Source


...
SSIS
                                                           Data Mart




                                 Join and Ag...
Linked Servers 
OLE DB Provider
                                                         for MS SQL Server
                               ...
Might be to easy?
                 Might be to easy?
-------------------------------------------------------------
-- Sele...
Microsoft Sync Framework
 Microsoft Sync Framework
A comprehensive synchronization platform that 
enables collaboration an...
Others
• BCP
• Detach/Attach
• Backup/Restore
     k /
Copy on Write Technology
DB Mirroring
            Client         Client     Client      Client


                               Network

          ...
Source
                             Data
                            Pages

       Source
         03
     SQL Server 2008...
Database Architechs




Paul Bertucci
   Founder Database Architechs
   Former Chief Data Architect – Veritas & Symantec...
To learn more or inquire about speaking opportunities, please contact:

                Mark Ginnebaugh, User Group Leader...
Upcoming SlideShare
Loading in …5
×

Microsoft SQL Server Distributing Data with R2 Bertucci

3,637 views

Published on

This presentation by Paul Bertucci describes an ordered method of determining what users need and which SQL Server data distribution solution is best to use.

There are many needs of data throughout an organization. Getting data to those who need it can be accomplished many different ways with SQL Server 2008 technologies.

This presentation covers data replication, database mirroring and snapshots, older methods such as log shipping and linked servers, and new methods such as using the sync framework.

You'll Learn

* Each of SQL Server’s main data distribution solutions
* How to determine which solution to use to solve different purposes

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,637
On SlideShare
0
From Embeds
0
Number of Embeds
905
Actions
Shares
0
Downloads
41
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Microsoft SQL Server Distributing Data with R2 Bertucci

  1. 1. Distributing Data Techniques with  Distributing Data Techniques with  Distributing Data Techniques with SQL Server 2008 R2 Q Speaker: Paul Bertucci Founder, Database Architechs d b hi h Silicon Valley SQL Server User Group October 19, 2010 Mark Ginnebaugh, User Group Leader,  mark@designmind.com
  2. 2. Distributing Data Techniques  Distributing Data Techniques with SQL Server 2008 R2 Copyright 2010 All Rights Reserved
  3. 3. Agenda g • Determining  where data is needed • What SQL Server options do you have • How to get it there • Work within your constraints  Security  Regulatory (like PII…)  Administratively Ad i i t ti l  Data volatility/sizes  SLA’s you must deliver  Disaster Recovery/Business Continuity  Skills/Knowledge of your team  Resources (like $$’s, so on)
  4. 4. Reasons to Distribute Data Reasons to Distribute Data
  5. 5. SQL Server 2008 SQL Server 2008 Reporting/ODS Primary Reporting OLTP Server 19FIG02 OLTP DB Rpt DB pt SQL Server 2008 Enabling/Partitioning Europe Server SQL Server 2008 USA xyz DB SQL Server 2008 (Headquarters) Asia xyz DB Server xyz DB SQL Server 2008 SQL Server 2008 North America Regionalization Europe Region Region xyz DB y (multiple owners) xyz DB y SQL Server 2008 SQL Server 2008 Primary Failover Hot Spare (Fail-over) xyz DB xyz DB
  6. 6. Distributing periodic updates to Data Marts from a “master” Data Warehouse Data Mart Master SQL Server Data Warehouse 2008 Data Mart Data Mart SQL Server 2000 SQL Server 2005
  7. 7. Enabling a Data Warehouse from Legacy Systems ORACLE Master Data Warehouse SQL Server 2005 SQL Server 2008
  8. 8. Initialization of tables Heterogeneous for a heterogeneous Subscriber Subscriber in Replication ORACLE Publisher Subscriber SQL Server 2008 SQL Server 2008 Subscriber SQL Server 2005
  9. 9. Pulling in Sales Order information Sales Force Automation from disparate sources Application for import into SQL Server 2008 Access SQL Server Excel 2008
  10. 10. Data Mart OLTP Database SQL Server 2008 SQL Server 2008 Excel Export from the OLTP Database to a business intelligence Data Mart & Excel
  11. 11. Connection Results Results SQL Server SQL Server 2008 2008 Client Local Server Remote Server
  12. 12. Decision‐Tree Path Traversal Database Mirroring Integration I t ti With Snapshots S h t Services 1a2c3 1a2d3 Detach/ 1a2e3 Attach BCP 1a2b3 b c d a 1a2 e 1a2a3 1 2 3 a b 1b2 1e2 Log Sync e 1 c d Shipping Framework 1c2 1d2 Backup/ Data Restore Replication Linked Servers
  13. 13. Data Log Database Mirroring Linked Backup/ Detach/ Sync SSIS BCP Replication Shipping with Snapshots Servers Restore Attach Framework Large Data High Volatility On Line Many Targets Low Latency Transformation/ Filtering
  14. 14. Data Replication Data Replication
  15. 15. Central Publisher SQL Server SQL Server 2008 (default option) 2005 Adventure Adventure Works2008 Works2008 translog SQL Server translog Subscription 2008 Subscription Server 19FIG10 Server Adventure Works2008 translog Publication Server distribution SQL Server 2000 Oracle Distribution Server Adventure Adventure Works2008 Works2008 translog Subscription Subscription Server Server 15
  16. 16. Central Publisher SQL Server 19FIG11 Remote Distributor SQL Server 2008 2005 Adventure Adventure Works2008 Works2008 SQL Server translog 2008 translog Subscription Subscription Server Adventure Server Works2008 translog Publication Server SQL Server 2008 SQL Server 2000 Oracle distribution Remote Distribution Adventure Adventure Works2008 Server Works2008 translog Subscription Subscription Server Server
  17. 17. Publishing Subscriber SQL Server SQL Server 2008 2005 SQL Server Publication 2008 Adventure Adventure Server Works2008 Works2008 Adventure 19FIG12 Works2008 translog S translog Subscription translog Subscription Distribution Server Server Server distribution Subscription SQL Server Serveer 2008 Adventure SQL Server Works2008 2000 tion Oracle er translog Publicat Serve Adventure Adventure Works2008 Works2008 translog Subscription ion Subscription Distributi Server r Server Server distribution 17
  18. 18. Publication SQL Server SQL Server tion 2008 2008 19FIG13 er er Central Serve Publicat Serve Adventure Adventure Works2008 Works2008 translog Subscriber translog ion Distribution Distributi Server r Server distribution distribution SQL Server 2008 Subscription Server Adventure Works2008 SQL Server SQL Server Publication ation 2008 2008 ver ver Publica Serv Serv Adventure Adventure Works2008 Works2008 translog translog ution ution er er Serve Serve Distribu Distribu distribution distribution
  19. 19. Multiple Publishers of a Single Table 19FIG14Server on on Subscriptio Subscriptio SQL S SQL Server S Server Server 2008 2008 Adventure Adventure on on Works2008 Works2008 Publicatio Publicatio Serverr Server translog translog Distribution Distribution Server Server distribution distribution Territory 1 Territory 2 Customers Table CustomerID TerritoryID AccountNumber CustomerType rowguid ModifiedDate 1345 1 AW1345 I X69G9.. 120203 1356 2 AW1356 I W211G.. 051605 2354 1 AW2354 S 7SQ78K.. 106705 3346 2 AW3346 I W12DV.. 022305 7643 2 AW7643 S WZ8R4.. 122205 7901 2 AW7901 I S2345X.. 041506 8921 1 AW8921 I RT66Y.. 031206
  20. 20. SQL Server Updating U d ti blication 2008 Subscriber Server 19FIG15 Adventure Works2008 Pub S translog SQL Server Queue stribution (Queue Reader Agent) Server S distribution Dis Update Customers set AccountNumber = null where CustomerID = 12345 SQL Server SQL Server 2008 2008 Adventure Adventure Works2008 Works2008 W k 2008 translog translog Subscription Subscription Server Server
  21. 21. SQL Server SQL Server 2008 2008 19FIG16 Adventure Works2008 Peer-to-Peer Adventure Works2008 translog translog Publication Publication Server Server North American o e ca Asia sa SQL Server 2008 Server Server distribution Peer Originator id: 1 Distribution Server Peer Originator id: 2 Customers Table CustomerID C t ID TerritoryID T it ID AccountNumber C A tN b CustomerType T rowguid id ModifiedDate M difi dD t 1345 1 AW1345 I X69G9.. 120203 1356 2 AW1356 I W211G.. 051605 2354 1 AW2354 S 7SQ78K.. 106705 3346 2 AW3346 I W12DV.. W12DV 022305 7643 3 AW7643 S WZ8R4.. 122205 7901 5 AW7901 I S2345X.. 041506 8921 4 AW8921 I RT66Y.. 031206
  22. 22. Publisher Publication P bli ti AW_Vertical Adventure Works2008 translog CustomerV (Article)19FIG05 SQL Server 2008 CustomerID TerritoryID AccountNumber CustomerType rowguid ModifiedDate CustomerID TerritoryID CustomerType 1345 1 AW1345 I X69G9.. 120203 1345 1 I 1356 2 AW1356 I W211G.. 051605 1356 2 I 2354 1 AW2354 S 7SQ78K.. 106705 2354 1 S 3346 2 AW3346 I W12DV.. 022305 3346 2 I 7643 3 AW7643 S WZ8R4.. 122205 7643 3 S 7901 5 AW7901 I S2345X.. 041506 7901 5 I 8921 4 AW8921 I RT66Y.. 031206 8921 4 I
  23. 23. Publisher Publication AW_Horizontal Adventure Works2008 19FIG06 CustomerH (Article) translog SQL Server 2008 CustomerID TerritoryID AccountNumber CustomerType rowguid ModifiedDate 1345 1 AW1345 I X69G9.. 120203 1356 2 AW1356 I W211G.. 051605 2354 1 AW2354 S 7SQ78K.. 106705 3346 2 AW3346 I W12DV.. 022305 Only these Rows! 7643 3 AW7643 S WZ8R4.. 122205 7901 5 AW7901 I S2345X.. 041506 8921 4 AW8921 I RT66Y.. 031206 CustomerID TerritoryID AccountNumber CustomerType yp rowguid g ModifiedDate 2354 1 AW2354 S 7SQ78K.. 106705 7643 3 AW7643 S WZ8R4.. 122205
  24. 24. Publisher Publication Adventure Ad t 19FIG07 AW_H_and_V Works2008 07FIG08 translog CustomerHV (Article) SQL Server 2008 CustomerID TerritoryID AccountNumber CustomerType rowguid ModifiedDate 1345 1 AW1345 I X69G9.. 120203 1356 2 AW1356 I W211G.. 051605 Only these Columns 2354 1 AW2354 S 7SQ78K.. 106705 and 3346 2 AW3346 I W12DV.. 022305 these Rows! 7643 3 AW7643 S WZ8R4.. 122205 7901 5 AW7901 I S2345X.. 041506 8921 4 AW8921 I RT66Y.. 031206 CustomerID TerritoryID y CustomerType 2354 1 S 7643 3 S
  25. 25. AW_H_and_J (Publication) CustomerH (Article) 19FIG08 07FIG09 OrdersHJ (Article) CustomerID TerritoryID AccountNumber CustomerType rowguid ModifiedDate Only the orders 1345 1 AW1345 I X69G9.. X69G9 120203 for 1356 2 AW1356 I W211G.. 051605 customer 2354 1 AW2354 S 7SQ78K.. 106705 CustomerID 2354 type “S” 3346 2 AW3346 I W12DV.. 022305 7643 3 AW7643 S WZ8R4.. WZ8R4 122205 7643 7901 5 AW7901 I S2345X.. 041506 8921 4 AW8921 I RT66Y.. 031206 join SalesOrderID RevisionNumber OrderDate DueDate Status CustomerID 43659 1 7/1/2001 7/13/2001 5 1345 43660 1 7/1/2001 7/13/2001 5 1356 43661 1 7/1/2001 7/13/2001 5 2354 SalesOrderID OrderDate CustomerID 43662 2 7/1/2001 7/13/2001 5 3346 43661 7/1/2001 2354 43663 1 7/1/2001 7/13/2001 5 7643 43664 1 7/1/2001 7/13/2001 5 7901 43663 7/1/2001 7643 43665 1 7/1/2001 7/13/2001 5 8921 25
  26. 26. Publisher AW (Publication) (P bli ti ) Subscriber SQL Server cation 19FIG17 2008 Customers (Article) rver SQL Server Public Adventure 2008 Ser Works2008 Adventure translog Works2008 bution PUSH rver translog Distrib Ser distribution di t ib ti Subscription Server PUSH: initiated by Publisher PULL (agent will be on the Publisher if local distribution, or on the Distributor if remote distribution) PULL: initiated by Subscriber y (agent will be on the subscriber)
  27. 27. Data Access Latency Distributing Data Distributing Data Autonomy Sites (locations) Frequency Network Machines Owner Other REPLICATION Read Only Each site only Central Publisher fast/ n Transactional repl Reporting short high many high 1 OLTP needs regional stable server/site site data filter by region Read Only Each site only Central Publisher fast/ n Snapshot repl Reporting long high many low 1 OLTP needs regional stable server/site site data filter by region Read Mostly ead ost y Regional updates Central Publisher g p fast/ n Transactional repl A few updates short high < 10 medium 1 OLTP on one table stable server/site site Updating Subs Read Mostly Regional update Central Publisher slow/ n All Merge repl A few updates medium high < 10 medium all tables unreliab server/site update Read equal Regional update Peer-to-Peer fast/ 1 All Transactional Equal updates short high < 10 medium all tables stable server/site update repl Inserts 1 Each site only Central Subscriber fast/ 1 Transactional repl (new orders) short high many high needs regional stable server/site report data site Central Publisher Hot/Warm / Very high <2 high fast/ f / 1 Fail-over Remote Distributor 1 OLTP Spare short stable server/site site Transactional repl
  28. 28. Log Shipping Yes, it is still here.. ,
  29. 29. Log Shipping “Source” SQL Server 2008 “Destination” SQL Server 2008 Q TxnLog T L Primary CallOne DB backups Secondary Server Server translog CallOne DB BackupCallOne_tlog_200405141120.TRN pped wer LogShareCallOne_tlog_200405141120.TRN LogShareCallOne tlog 200405141120 TRN TxnLog Last log ship Delay Answ TxnLog T L Copies Restores “Monitor” SQL Server 2008 S Delay between logs loaded Delay Answer Monitor Server MSDB DB Don’t build on this, it is being deprecated !
  30. 30. Database Mirroring with Snapshots with Snapshots
  31. 31. Client Client Client Client Network A SQL Server 2008 SQL Server 2008 Mirror Server Mi S DB Mirroring Adventure D Principal Adventure D Works DB Server B Works DB translog t l translog C SQL Server xyz Witness t ess Server MSDB DB 31
  32. 32. SQL Server 2008 SQL Server 2008 Mirror Server Principal 10 Server AdventureWorks AdventureWorks DB DB translog translog Database Database Snapshot Snapshot comes the mirror. If this server node SQL Server 2008 m n Witness MSDB DB Server bec t 32
  33. 33. SQL Server 2008 01AdventureWork Source CREATE DB ‘xyz’ s DB AS SNAPSHOT OF Server AdventureWorks translog Database D t b Snapshot --- Reverting --- if needed --- RESTORE DB AdventureWorks FROM DATABASE_SNAPSHOT = ‘xyz’
  34. 34. Source Data Pages Source 02 SQL Server 2008 SQL Sparse Server File Pages Snapshot AdventureWorks AdventureWorks DB DB Empty Sparse file of a Snapshot just created (no p ( updates to original g data pages have occurred yet) 34
  35. 35. Source Data Pages Source 03 SQL Server 2008 SQL Sparse Server File Pages Snapshot AdventureWorks AdventureWorks DB DB System Catalog Copy of original of changed pages pages for snapshot only when a page is changed (Copy-on-write) 35
  36. 36. Snapshot Source Data Users Pages SELECT …..data……. FROM AdventureWorks SNAPSHOT 04 SQL Server 2008 SQL Server Snapshot AdventureWorks AdventureWorks DB DB System Catalog Sparse of changed pages File Pages 36
  37. 37. SQL Server 2008 SQL Server 2008 Mirror Server Principal 10 Server AdventureWorks AdventureWorks DB DB translog translog Database Database Snapshot Snapshot comes the mirror. If this server node SQL Server 2008 m n Witness MSDB DB Server bec t 37
  38. 38. PH Topology With Snapshots Real time Reporting R ti Users Network SQL Server 2008 N Principal Server SQL Server 2008 OLTP Application n Active Mirror Server stered Replication Adventure Adventure Works DB Works DB Clus translog translog Database Passive PH Snapshot Topology Principal Network Server SQL Server 2008 Point in time 38 Reporting Users
  39. 39. Publisher DB Mirror and  SQL Server 2008 SQL Server 2008 Replication Principal Mirror SQL Server 2008 Server Server Subscriber Distributor s bu o SQL Server 2008 SQL Server 2008 Principal Mirror Server Server SQL Server 2008 SQL Server 2008 Witness Server Subscriber 39
  40. 40. Integration Services
  41. 41. Excel OLAP Data Warehouse Access Multidimensional OLTP Cube Databases Other Data Sources OLTP Databases SQL Server Flat XML Web Services files
  42. 42. Package P k Task Task Task ainer Source Cache e Conta Data Flow Transformation Task Destination D ti ti Task Task Task
  43. 43. SSIS Data Mart Join and Aggregate e OLTP Database Tas to SQL Server sk AdventureWorks 2008 d UnleashedDataMart SQL Server 2008 Excel .csv flat file Export from the OLTP Database to a business intelligence Data Mart & Excel/.csv flat file
  44. 44. Linked Servers 
  45. 45. OLE DB Provider for MS SQL Server SQL Server 6.5 – 2008 OLE DB Provider DLL’s OLE DB Provider Sybase for ODBC Informix DB2 ODBC Others OLE DB Provider Excel spreadsheet for Jet Jet FoxPro Database database Dbase file MS OLE DB OLE DB Provider file Access Database E Provider P id for Oracle f O l SQL Server ORACLE 2008 OLE DB Provider P id for Indexing Service Active Directory Services Index Server Others Client Tier Server Tier Remote Data Source Tier (can be “Linked”)
  46. 46. Might be to easy? Might be to easy? ------------------------------------------------------------- -- Selecting data from Excel spreadsheet via Linked Server -- ------------------------------------------------------------- SELECT substring(CustomerID,1,5) as 'CustID', substring(ContactName,1,18) as ContactName, substring(City,1,10) as 'Cit ' substring(Country,1,15) as 'C b t i (Cit 1 10) 'City', b t i (C t 1 15) 'Country'' t FROM [Linked ExcelSW]...[SWCustomers$] ORDER BY ContactName g go CustID ContactName City Country ------ ------------------ ---------- --------------- PICCO Adam Greifer Los Angele USA BLAUS John Monroe Concord USA CHOPS Martin Sommer Berkeley USA
  47. 47. Microsoft Sync Framework Microsoft Sync Framework A comprehensive synchronization platform that  enables collaboration and offline access for  enables collaboration and offline access for applications, services, and devices. It features  technologies and tools that enable roaming,  sharing, and taking data offline.  By using Sync Framework, developers can build  B i S F k d l b ild sync ecosystems that integrate any application  with any data from any store that uses any  y y y protocol over any network.  Mentioned here  because of its replication‐like behavior for  “occasionally connected” applications. occasionally connected applications
  48. 48. Others • BCP • Detach/Attach • Backup/Restore k /
  49. 49. Copy on Write Technology
  50. 50. DB Mirroring Client Client Client Client Network A SQL Server 2008 SQL Server 2008 Mirror Server Mi S Principal Adventure D Adventure Works DB D Server B Works DB translog t l translog C SQL Server xyz Witness t ess Server MSDB DB 50
  51. 51. Source Data Pages Source 03 SQL Server 2008 SQL Sparse Server File Pages Snapshot AdventureWorks AdventureWorks DB DB System Catalog Copy of original of changed pages pages for snapshot only when a page is changed (Copy-on-write) 51
  52. 52. Database Architechs Paul Bertucci  Founder Database Architechs  Former Chief Data Architect – Veritas & Symantec  Currently C rrentl Chief Architect - A todesk Autodesk  Author of SQL Server 2000, 2005 & 2008/R2 Unleashed  Co- Co-Author of ADO.NET in 24 hours  Author MS SQL Server High Availability  Author Sybase Performance & Tuning  Author Sybase Physical DB Design  Author Chen Entity/Relationship Modeling  Veritas SQL Server Performance Series pbertucci@dbarchitechs.com
  53. 53. To learn more or inquire about speaking opportunities, please contact: Mark Ginnebaugh, User Group Leader Mark Ginnebaugh User Group Leader mark@designmind.com

×