Recommended
More Related Content
What's hot
What's hot (20)
Similar to Evolving Architecture
Similar to Evolving Architecture (20)
Recently uploaded
Recently uploaded (20)
Evolving Architecture
- 2. Build a DATA PLATFORM not a DATA WAREHOUSE Analytics as a competitive advantage is born from STRATEGY not PROJECTS We require REAL-TIME and BATCH data pipelines
- 3. Event Producer Collect and Route Storage Query | Model | Automate Emerging Architecture Azure Stream Analytics Azure Event Hubs Azure Data Lake Store Azure SQL Data Warehouse
- 4. Application Database Enterprise Data Warehouse Mart Mart Batch ETL Jobs Storage and Query Traditional Architecture
- 5. Mirror Layer Analytical Model Temporary Staging Source Why Have a Mirror Layer? 1. Improve the data structure of a source system (add primary keys, indexes) 2. Hide complexity related to the type of source system (SQL, API, Mainframe) 3. Improve the quality and performance of change tracking 4. Enable data governance programs by homogenizing sources 5. Enable prototyping of new marketing automation solutions without developer support Risks/Assumptions This layer must be real-time and simple, close to the metal. The more it looks like another ETL layer, the more the risks will outweigh the benefits. Transform Near Real-time Intensive Transform
- 6. Application Database Mirror Layer Mart Mart Storage and Query Micro-Batch ETL Jobs Modern Architecture: Phase 1 Homogenize, Protect, and Standardize = database transaction log
- 7. Customer Metrics & History Event/Log Producer (Web Application) Stream Processing (Event Hubs) 1MB Max Event Archive Storage Query Model Automate DIGITAL ENTERPRISE On Premises Off Premises Modern Architecture vNext Events Micro-Batch Processing In EDW
- 8. Sale Transaction Customer Profile Source Database MS CDC is REQUIRED on tables where PKs don’t exist T-LOGT-LOG MS CDC is NOT required on tables where PKs exist Pub Dist Sale Transaction Customer Profile Transaction Changes Profile Changes Source Database Pub Dist T-LOGT-LOG SQL Replication vs CDC on Source
- 9. Sale Transaction Customer Profile Transaction Changes Profile Changes Source Database Pub Dist T-LOGT-LOG Proc Attunity Software Attunity Server Data Warehouse Sale Transaction Customer Profile Customer Profile Changes Server Purpose • Pass DML and DDL changes • Identify and suppress archiving activity • Monitor and notify table sync errors Trigger Trigger filters out “BEFORE” state Real-time integration Complete Attunity Ecosystem Sale Transaction Changes
Editor's Notes
- What’s the difference between a data platform and a data warehouse? The former implies that analytics is upstream of our operational systems. If you understand and accept this, you are ready to implement a robust analytics program. Also, we need a new set of terminology to drive culture. If you ignore Real-time as something that the “business isn’t asking for” or “isn’t ready to use” then you forget that IT leads from the front! Analytics as a competitive advantage means capturing revenue by seizing the marketing opportunity. Who here thinks that the stabilization project will deliver a competitive advantage? What about the next project? Where are we going exactly? How exactly can we drive revenue in a data-driven way? What if we didn’t start another project or hire another consulting firm until we have a strategy?
- This general architecture is called a lambda architecture. There’s a speed, batch, and serving layer. Notice that this flows as Extract, Load, Transform rather than ETL; in fact, extract is no longer relevant: it should be “ingest”. Applications (and even devices) are “emitting” their events. VALUE: robustness, fault-tolerance, low latency reads/writes, scalability, generalization, extensibility, minimal maintenance, ad hoc queries, debuggability. What parts are relevant to us? Should we do any of this? Notice the arrows go in both directions. A machine learning result can be pushed back to an application. Everything scales linearly and is highly available – even the app itself. I could fill this slide with the companies that implement this architecture including Microsoft, Walmart, Yahoo, LinkedIn, and Netflix. It’s worth noting that some architects are pushing for the collapse of the speed and batch layer into a single layer. New technologies are supporting this concept. This is especially possible at smaller scales.
- This is a synchronous world. Application have their own databases. We reach in and extract large amounts of data, bring it down to disk, and search for changes. We transform the data and load complex schemas with information. Who do you scale this system? You can’t do it horizontally. You can only scale up: bigger SQL servers, SSD SANs. Schemas must be designed and built before the Business can discover and analyze. Arbitrary questions are difficult to ask of the system and typically involve data points not yet modeled. In almost every experience, I have seen the Business’ need for information out pace IT’s capacity to build. The ETL layers become more complex. Sometime you create layers just to track changes… Of course, it’s never this simple….
- Here we move from a pull to a push architecture. We are closer to applications emitting their own events. This is not another ETL layer, we are ingesting database transactions as they appear in real-time. This satisfies the principles of a mirror layer. Indeed, if you cannot satisfy these principles, it is best to move back to the traditional architecture. With this architecture, we can support micro-batch and batch processes with a robust, fault-tolerate tool that is close to the metal and simple. This is a key driver of high data quality which is defined as timeliness, consistency, and accuracy. Downstream development becomes simpler and more confident where the focus is more on steering the analytical model and less to do with tracking source system data changes. Data quality and governance metrics become trustworthy because the mirror layer is basically sentient.
- Cloud-born data should remain in the cloud. But we can bring our enterprise customer insights to the cloud in a secure, scalable and efficient way. Attunity can also support this on a per hour basis with CloudBeam which can send our enterprise data to Azure SQL Data Warehouse at high speed. The most expensive path is $2.24 per hour and the cheapest is $.018 per hour. Auto-scaling is possible. But we can start small…very small. The query environment is a SQL-like interface that could easily be switched out for R or even pure Scala. It’s up to the analyst. Code is translated into processes on the storage system that will bring back data. We could also could pre-make tables for analysts. The point is that a “schema” does not have to exist from the beginning. An analyst can apply schema at the time of query (also called late binding). This allows your data engineering team to focus on ingesting and storing data while the analyst has the ability to take an arbitrary question and apply structure to the data in order to answer that question. This is the modern architecture.