More Related Content
What's hot
What's hot (15)
Viewers also liked
Viewers also liked (16)
Similar to Implementing Change Systems in SQL Server 2016
Similar to Implementing Change Systems in SQL Server 2016 (20)
Recently uploaded
Recently uploaded (20)
Implementing Change Systems in SQL Server 2016
- 1. Change Systems Critical Component Series
- 3. The goal of this series is to give you the tools you need to push analytics forward at your company. • The nature and importance of change systems in an overall data platform • Compare and contrast traditional and modern data warehouse architectures • Discuss a key technology that is core to change systems in the enterprise • Compare the SQL Server features that enable robust change data capture Change Systems Agenda
- 4. Database Engine MDF LDF Overview The Source of Change • A database engine manages files. • Data structures • Transaction logs • Change systems accurately track modifications inside data structures. • The source of record for change is the transaction log. Using this log directly is a characteristic of passive change systems. • Active change systems watch the data structure and record observable change.
- 5. Overview Modeling Change AccountID CustomerID AccountBalance ModifyDate 4568456 2342 1234758.23 2017-03-11 04:11:05 4624572 9875 5768.01 2017-03-11 04:13:15 4745733 8735 478893.33 2017-03-11 04:13:01 AccountID CustomerID Type Amount EventDate 4568456 2342 Deposit 1198575.32 2017-03-08 09:09:04 4624572 9875 Deposit 4438.70 2017-03-08 09:10:01 4745733 8735 Deposit 460436.02 2017-03-07 10:13:20 4568456 2342 Deposit 528.11 2017-03-08 06:13:45 4624572 9875 Deposit 1345.23 2017-03-09 10:22:25 4745733 8735 Deposit 635.20 2017-03-08 11:13:01 4568456 2342 Withdrawal 23.21 2017-03-09 12:12:02 4624572 9875 Fee 21.34 2017-03-09 06:13:45 4745733 8735 Withdrawal 42.66 2017-03-10 13:13:12 4568456 2342 Transfer 35678.01 2017-03-11 04:11:05 4624572 9875 Deposit 5.42 2017-03-11 04:13:15 4745733 8735 Deposit 17864.77 2017-03-11 04:13:01 Table Log =* *Record the CRUD operations to the table and you get a changelog. The duality is that a table supports data at rest and logs capture change. If you have a log you can not only create the original table but a myriad of other derived tables. Logs therefore seem to be a more fundamental data structure.
- 6. Overview Modeling Change Valid Time John Doe who lived in Flat Rock, NC made his first visit to us on April 1st, 1985 and changed his permanent address during a sale on November 12th 2005. Name Address ValidFrom ValidTo John Doe 81 Carl Sandberg Ln, Flat Rock, NC 28731 1985-04-01 10:00:00 2005-11-12 09:05:00 John Doe 9433 Collingdale Way, Raleigh, NC 27617 2005-11-12 09:06:01 9999-12-31 23:59:99 Transaction Time Our data warehouse went live on November 1st 2005. The ETL runs daily at 4 AM. Name Address CreateDate ExpireDate John Doe 81 Carl Sandberg Ln, Flat Rock, NC 28731 2005-11-01 09:25:11 2005-11-13 04:54:11 John Doe 9433 Collingdale Way, Raleigh, NC 27617 2005-11-13 04:54:12 9999-12-31 23:59:99
- 7. Overview Modeling Change ID Name Address ModifyDate 12345 John Doe 81 Carl Sandberg Ln, Flat Rock, NC 28731 1985-04-01 10:00:00 12345 John Doe 9433 Collingdale Way, Raleigh, NC 27617 2005-11-12 09:06:01 Key ID Name Address ValidFrom ValidTo CreateDate ExpireDate 1 12345 John Doe 81 Carl Sandberg Ln, Flat Rock, NC 28731 1985-04-01 10:00:00 2005-11-12 09:06:00 2005-11-01 09:25:11 2005-11-13 04:54:11 2 12345 John Doe 9433 Collingdale Way, Raleigh, NC 27617 2005-11-12 09:06:01 9999-12-31 23:59:99 2005-11-13 04:54:12 9999-12-31 23:59:99 ETL Source Target SCD 2 Dimension This column creates risk Latency of 1 Day at best
- 8. Application Database SQL DB2 SQL … Enterprise Data Warehouse Mart Mart Batch ETL Jobs Storage and Query Traditional Architecture The Pull Method
- 9. Application Database SQL DB2 SQL … Enterprise Data Warehouse Mart Mart Batch ETL Jobs Storage and Query Traditional Architecture Focus on the Source Focus Area One Friction & Frustration Data Quality • Timeliness • Latency of change • Latency of build • Consistency • Redundant ETL • Accuracy • Filters • Logic • Source Lead Time • Custom ETL • Manual ETL • Business case and ceremony • Domain knowledge Dependencies • Business logic • Redundancy • Downstream effects • Team
- 10. Collect and Route Events Query | Model | Automate Modern Architecture The Push Method (Lambda) Speed Layer Batch Layer Serving Layer Real-time Views Batch Views
- 11. Events Query | Model | Automate Stream Modern Architecture The Push Method (Kappa) Unified Log Storage Archive Collect
- 12. Modern Architecture The Fungibility of Data LOG
- 13. • Ingest (don’t extract) disparate silos of data • Store data in its atomic form (no transform) • Collect changes as if they were events (immutable) • Run downstream ETL more often (process less data each cycle) Modern Architecture Lessons Learned
- 14. Mart ETL Application Database Enriched Source Mirror Layer Mart Storage and Query Micro-Batch ETL Jobs Modern Architecture Phase 1 Homogenize, Protect, and Standardize = database transaction log
- 15. Mirror Layer Analytical Model Temporary Staging Source Why Have a Mirror Layer? 1. Improve the data structure of a source system (add primary keys, indexes) 2. Hide complexity related to the type of source system (SQL, API, Mainframe) 3. Improve the quality and performance of change tracking 4. Enable data governance programs by homogenizing sources 5. Enable prototyping of new automation solutions without developer support Risks/Assumptions This layer must be real-time and simple, close to the metal. The more it looks like another ETL layer, the more the risks will outweigh the benefits. Transform Near Real-time Intensive Transform Mirror layer Overview
- 16. But all I read is hate for replication on the internets!
- 17. Mirror layer Replication in Production Sale Transaction Customer Profile Source Database Server T-LOGT-LOG Pub Sub ArticleArticle Push Dist cmd • Set up everything in a lower environment and replay production activity to get an idea of load. • The source database is placed into an Always On availability group so that the database and replication can failover. • Distributor and subscriber are moved to their own failover cluster. • Subscribers connect to an availability group listener so they can find the right server after a failover. • Database and log backups are still taken regularly to support disaster recovery, but additional preparations are made to enable a smooth restore of replication.
- 18. Mirror Layer Demo Features of SQL Server AccountID CustomerID AccountBalance ModifyDate 4568456 2342 1234758.23 2017-03-11 04:11:05 4624572 9875 5768.01 2017-03-11 04:13:15 AccountID Operation Columns 4568456 INSERT 4624572 UPDATE AccountBalance 4745733 DELETE Base Table Change Table (Internal) AccountID CustomerID AccountBalance ModifyDate CreateDate ExpireDate 4624572 9875 5001.01 2017-03-10 06:19:01 2017-03-10 06:20:35 2017-03-11 04:14:22 4745733 8735 478893.33 2017-03-11 04:13:01 2017-03-11 04:14:59 2017-03-12 09:01:12 History Table Change Tracking • Net changes only • No data • Internal tables • Internal functions • Retention period only Temporal Tables • Net changes not automatic • Data • Normal tables • T-SQL language integration • Full support for archiving
Editor's Notes
- #1 challenge: active = observation is difficult, passive = logs are esoteric and sheltered Which is more valuable? In general logs are more valuable; however, it can be expensive to implement. It all depends on controls, trust, and latency. If your source database has the right audit fields, they’re trustworthy, and bath ETL is OK, then active change systems are the easy and cheap choice.
- These two things are equivalent and they had better be because this is the foundation of reliability, availability, and resilience of SQL Server and all other RDBMS systems. It’s how why we mirror databases, log ship, and how create high availability in data center infrastructure. A log is essentially a backup of all possible states of the table and any other possible derived table.
- Which is more valuable? As you approach real-time integration these two concepts are effectively interchangeable at the margin.
- John spent $30,000 over 600 transactions with us while he lived in Flat Rock ($50/tr). He has spent $250 over 2 transactions since he “moved” to Raleigh ($125/tr). Raleigh is where our brick and mortar store resides. What if you have to side-load new data from say an acquisition, merger, or archive situation? What if you have to trunc and reload this table? What about failure? You want valid time more than transaction time. Most data warehouses keep only transaction time. If ModifyDate does not exist or you don’t trust it then you’ve got a rough road ahead.
- LOTS OF CEREMONY. This is a synchronous world. Application have their own databases. We reach in and extract large amounts of data, bring it down to disk, and search for changes. We transform the data and load complex schemas with information. Who do you scale this system? You can’t do it horizontally. You can only scale up: bigger SQL servers, SSD SANs. Schemas must be designed and built before the Business can discover and analyze. Arbitrary questions are difficult to ask of the system and typically involve data points not yet modeled. In almost every experience, I have seen the Business’ need for information out pace IT’s capacity to build. Of course, it’s never this simple….
- This general architecture is called a lambda architecture. The traditional Extract portion of ETL is no longer relevant. We now “ingest” data in this architecture. Applications (and even devices) are “emitting” their events. VALUE: robustness, fault-tolerance, low latency reads/writes, scalability, generalization, extensibility, minimal maintenance, ad hoc queries, debuggability. I could fill this slide with the companies that implement this architecture including Microsoft, Walmart, Yahoo, LinkedIn, and Netflix. Nathan Marz – creator of Storm
- Jay Kreps – originally a software engineer at LinkedIn who was a key contributor to Kafka. LinkedIn is an interesting case because during Kreps’ tenure at the company they went from an VLB Oracle DW to Teradata to Hadoop and along that path they discovered the value of immutable streams of data in stark contrast to batch analytics. Clients have to keep track of their place in the log. The unified log made data fungible.
- Oil is fungible because it has equivalent value regardless of source. It is commoditized. Data, like oil, derives its value from an even quality and its ability to flow freely between endpoints as a homogenous commodity. By simplifying the data to its most fundamental structure (a log), we can unify organizations around a data platform, and (click) we can unify our analytics process because we have standardized our data flow.
- Most organizations already have a data warehouse. Frankly, the architectures previously presented make a lot of assumptions about the organization like the reliance on software engineers, the way the organization approaches data integration and information management. So how do we get started? Well, luckily SQL Server has been around for a very long time, and we can bootstrap a progressive architecture by understanding and applying the good parts of what we’ve learned from others in the field.
- Cloud-born data should remain in the cloud.
- Here we move from a pull to a push architecture. We are closer to applications emitting their own events. This is not another ETL layer, we are ingesting database transactions as they appear in real-time. This satisfies the principles of a mirror layer. Indeed, if you cannot satisfy these principles, it is best to move back to the traditional architecture. With this architecture, we can support micro-batch and batch processes with a robust, fault-tolerate tool that is close to the metal and simple. Downstream development becomes simpler and more confident where the focus is more on steering the analytical model and less to do with tracking source system data changes. Data quality and governance metrics become trustworthy because the mirror layer is sentient.
- Fundamentally, a mirror layer reflects source systems exactly table-by-table. It makes every source system look like a SQL Server database: DB2, flat files, APIs, etc. We are hiding the complexity of a heterogeneous environment. It becomes the source for analytical data models. This is where you improve a system that you cannot control: you add primary keys, indexing, etc. Remember, a source system database is designed for transactional speed. We would rather it be design for query speed; you can do this in the mirror layer. This is also where you track changes. Especially when a source system tracks changes poorly, a mirror helps you iron out transactional history in a way that is robust and fault tolerant. It should be able to heal when a source system changes its schema during a release, for example. By homogenizing sources, we are supporting data governance from a lineage perspective. Moreover, we are allowing governance personnel to access system data that they would not otherwise have access to in the production operation environment. Metrics can be compared more easily and new metrics can be created in less time. Maybe most importantly, we are providing a layer for the Business to prototype the next valuable solution. The business should not report from or run ad hoc on source system databases. They can run it on this layer and help us build high quality solution faster through prototyping.
- This is all possible with SQL Server Standard edition. Always Encrypted makes this not possible.
- Change tracking is essentially just metadata. But it sure is powerful and can transform your ETL process if all you need is to know what has changed. Temporal tables is a fully supported time machine. Unfortunately, getting net changes is not automatic. This is transaction time but if replication is setup as continuous then we are very close to valid time at the margin. You will still need to seed valid time as part of an initial load.