Hand Coding ETL Scenarios and Challenges
•
3 likes•3,320 views
Overview of some of the scenarios that lead one to hand-coding over tools, description of the challenges faces, and some practices to deal with the problems.
Report
Share
Report
Share
![Course Outline and Overview ,[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Introduction to ETL and Data Integration
This presenation explains basics of ETL (Extract-Transform-Load) concept in relation to such data solutions as data warehousing, data migration, or data integration. CloverETL is presented closely as an example of enterprise ETL tool. It also covers typical phases of data integration projects.
Etl - Extract Transform Load
Data Warehouse:
A physical repository where relational data are specially organized to provide enterprise-wide, cleansed data in a standardized format.
Reconciled data: detailed, current data intended to be the single, authoritative source for all decision support.
Extraction:
The Extract step covers the data extraction from the source system and makes it accessible for further processing. The main objective of the extract step is to retrieve all the required data from the source system with as little resources as possible.
Data Transformation:
Data transformation is the component of data reconcilation that converts data from the format of the source operational systems to the format of enterprise data warehouse.
Data Loading:
During the load step, it is necessary to ensure that the load is performed correctly and with as little resources as possible. The target of the Load process is often a database. In order to make the load process efficient, it is helpful to disable any constraints and indexes before the load and enable them back only after the load completes. The referential integrity needs to be maintained by ETL tool to ensure consistency.
To Study E T L ( Extract, Transform, Load) Tools Specially S Q L Server I...
To Study E T L ( Extract, Transform, Load) Tools Specially S Q L Server Integration Services
Why shift from ETL to ELT?
The process of data warehousing is undergoing rapidtransformation, giving rise to various new terminologies, especially due to theshift from the traditional ETL to the new ELT. Forsomeone new to the process, these additional terminologies and abbreviationsmight seem overwhelming, some may even ask, “Why does it matter if the L comesbefore the T?”
The answer lies in the infrastructure and the setup. Here iswhat the fuss is all about, the sequencing of the words and more importantly,why you should be shifting from ETL to ELT.
Recommended
Introduction to ETL and Data Integration
This presenation explains basics of ETL (Extract-Transform-Load) concept in relation to such data solutions as data warehousing, data migration, or data integration. CloverETL is presented closely as an example of enterprise ETL tool. It also covers typical phases of data integration projects.
Etl - Extract Transform Load
Data Warehouse:
A physical repository where relational data are specially organized to provide enterprise-wide, cleansed data in a standardized format.
Reconciled data: detailed, current data intended to be the single, authoritative source for all decision support.
Extraction:
The Extract step covers the data extraction from the source system and makes it accessible for further processing. The main objective of the extract step is to retrieve all the required data from the source system with as little resources as possible.
Data Transformation:
Data transformation is the component of data reconcilation that converts data from the format of the source operational systems to the format of enterprise data warehouse.
Data Loading:
During the load step, it is necessary to ensure that the load is performed correctly and with as little resources as possible. The target of the Load process is often a database. In order to make the load process efficient, it is helpful to disable any constraints and indexes before the load and enable them back only after the load completes. The referential integrity needs to be maintained by ETL tool to ensure consistency.
To Study E T L ( Extract, Transform, Load) Tools Specially S Q L Server I...
To Study E T L ( Extract, Transform, Load) Tools Specially S Q L Server Integration Services
Why shift from ETL to ELT?
The process of data warehousing is undergoing rapidtransformation, giving rise to various new terminologies, especially due to theshift from the traditional ETL to the new ELT. Forsomeone new to the process, these additional terminologies and abbreviationsmight seem overwhelming, some may even ask, “Why does it matter if the L comesbefore the T?”
The answer lies in the infrastructure and the setup. Here iswhat the fuss is all about, the sequencing of the words and more importantly,why you should be shifting from ETL to ELT.
Managing Data Integration Initiatives
This powerpoint slide deck is the presentation given at the Microsoft center in Waltham, MA titled Leading Practices and Insights for Managing Data Integration Initiatives.
Topics covered include:
Key Drivers
Approaches and Strategy
Tools and Products
Useful Case Studies
Success Factors
Data Integration: the Beginner's Guide
Stop wasting time and money manually entering data. Invest in software data integration. Break down data silos and get your data talking!
CloverETL Basic Training Excerpt
Excerpt from the CloverETL Basic Training slides.
The basic course lasts 3 days and covers basic principles, CloverETL Designer walkthrough, transaction analysis, lookups, database connections, working with structured data, XML etc.
More at www.cloveretl.com/services/training
3 tier data warehouse
This is the 3- Tier architecture of Data Warehouse. This is the topic under Data Mining subject. Data mining is extracting knowledge from large amount of data.
Data Warehouse 101
Introduction to Data Warehouse. Summarized from the first chapter of 'The Data Warehouse Lifecyle Toolkit : Expert Methods for Designing, Developing, and Deploying Data Warehouses' by Ralph Kimball
ETL Fundamentals
Presentation of ETL Fundamentals for Data Warehousing and All That Jazz Meetup Group
DATASTAGE AND QUALITY STAGE 9.1 ONLINE TRAINING
IF ANY ONE NEED DATASTAGE ONLINE TRAINING (OR) JOB SUPPORT (OR) CLASS VIDEO'S PLEASE CONTACT ---> trainingonds@gmail.com or +91-9000380723
ETL Market Webcast
Short overview of the components of the data integration market, some statistics and trends related to ETL, and how ETL products are evolving.
More Related Content
What's hot
Managing Data Integration Initiatives
This powerpoint slide deck is the presentation given at the Microsoft center in Waltham, MA titled Leading Practices and Insights for Managing Data Integration Initiatives.
Topics covered include:
Key Drivers
Approaches and Strategy
Tools and Products
Useful Case Studies
Success Factors
Data Integration: the Beginner's Guide
Stop wasting time and money manually entering data. Invest in software data integration. Break down data silos and get your data talking!
CloverETL Basic Training Excerpt
Excerpt from the CloverETL Basic Training slides.
The basic course lasts 3 days and covers basic principles, CloverETL Designer walkthrough, transaction analysis, lookups, database connections, working with structured data, XML etc.
More at www.cloveretl.com/services/training
3 tier data warehouse
This is the 3- Tier architecture of Data Warehouse. This is the topic under Data Mining subject. Data mining is extracting knowledge from large amount of data.
Data Warehouse 101
Introduction to Data Warehouse. Summarized from the first chapter of 'The Data Warehouse Lifecyle Toolkit : Expert Methods for Designing, Developing, and Deploying Data Warehouses' by Ralph Kimball
ETL Fundamentals
Presentation of ETL Fundamentals for Data Warehousing and All That Jazz Meetup Group
DATASTAGE AND QUALITY STAGE 9.1 ONLINE TRAINING
IF ANY ONE NEED DATASTAGE ONLINE TRAINING (OR) JOB SUPPORT (OR) CLASS VIDEO'S PLEASE CONTACT ---> trainingonds@gmail.com or +91-9000380723
What's hot (20)
Viewers also liked
ETL Market Webcast
Short overview of the components of the data integration market, some statistics and trends related to ETL, and how ETL products are evolving.
Le processus ETL (Extraction, Transformation, Chargement)
Présentation général des étapes du processus ETL (Extract,Transform, Load) d'un projet décisionnel.
ETL, acronyme de Extraction, Transformation, Loading, est un système de chargement de données depuis les différentes sources d'information de l'entreprise (hétérogènes) jusqu'à l'entrepôt de données (modèles multidimensionnels).
Processing Near Real-Time Global Vessel Data
exactEarth describes its close work with Safe to allow the seamless integration of exactAIS global vessel data that is managed and processed through FME 2014.
ETL DW-RealTime
Apresentação sobre os métodos aplicados no processo de ETL, aprofundando sobre os métodos CDC que são utilizados em ETL de DataWarehouse de Tempo Real.
kafka for db as postgres
Introduction to Apache Kafka And Real-Time ETL for DBAs and others who are interested in new ways of working with relational databases
Osmosit case study_asl2
Presentazione del caso di studio di Business Intelligence USL2 di Perugia durante il meeting Open Opportunity 2010
Migrazione da PostNuke a Plone: la banca dati CROP (S. Carluccio, CNR-ISMAR)
Migrazione da PostNuke a Plone: la banca dati CROP (S. Carluccio, CNR-ISMAR)Plone for Research and University
Presentazione Plone4UR Bologna, 20 maggio 2010#materaodd15
Slide utilizzate per il matera open data day del 21 febbraio 2015. Il tema analizzato riguarda la reperibilità e l'analisi dei dati delle royalties petrolifere in Basilicata in particolare e dello sviluppo del territorio in generale.
Geoweb Framework
Geoweb è il framework OpenSource per creare sistemi Tecnico-Gestionali, basati su standard e protocolli 'aperti', completamente personalizzabili.
Viewers also liked (20)
Le processus ETL (Extraction, Transformation, Chargement)
Le processus ETL (Extraction, Transformation, Chargement)
Using Continuous Etl With Real Time Queries To Eliminate My Sql Bottlenecks
Using Continuous Etl With Real Time Queries To Eliminate My Sql Bottlenecks
Porting evolutivo di un’applicazione per la gestione di note spese in ambient...
Porting evolutivo di un’applicazione per la gestione di note spese in ambient...
Migrazione da PostNuke a Plone: la banca dati CROP (S. Carluccio, CNR-ISMAR)
Migrazione da PostNuke a Plone: la banca dati CROP (S. Carluccio, CNR-ISMAR)
Similar to Hand Coding ETL Scenarios and Challenges
Getting Unstuck: Working with Legacy Code and Data
From this presentation for the IASA in 2007, Cory covers common challenges in dealing with Legacy Code and Data, and some tools and techniques for handling them.
Agile Business Intelligence
Applying Agile Delivery to Business Intelligence, focusing on ETL development
Etl data processing system which is very useful for the engineering students
Welcome to Gboard clipboard, any text that you copy will be saved here.
Best practice adoption (and lack there of)
This is a short presentation I created some time ago that details some of the developmental, procedural, and infrastructure best practices that I discovered while working with various customers.
Database Management System, Lecture-1
This is prepared from the book- Database System Concepts, written by ©Silberschatz, Korth and Sudarshan.
Building High Performance MySQL Query Systems and Analytic Applications
This presentation describes how to build fast running MySQL applications that service read-based systems. It takes a special look at column databases and Calpont's InfiniDB
Building High Performance MySql Query Systems And Analytic Applications
This presentation gives practical advice and tips on how to build high-performance read intensive databases, and discusses innovations such as column-oriented databases
Unlocking the Power of Your Data: Working with Databases in FME
Data integration can be a daunting task, with challenges such as formatting differences, siloed data, scalability issues, and ensuring clean and validated data. FME is a powerful tool that can help overcome these challenges.
In this webinar, we will show you how the no-code FME Platform works with relational databases like:
- MySQL
- PostgreSQL
- ArcSDE Enterprise Geodatabase
- Oracle
- Microsoft SQL Server
Join us to learn how you can leverage FME’s support for databases to build your ETL and other data integration workflows. We'll also discuss best practices for how FME can help you work more efficiently with your database.
By the end of the webinar, you'll have a clear understanding of the power and flexibility FME provides in streamlining your ETL and other data integration workflows with databases.
What are the benefits of learning ETL Development and where to start learning...
Learning ETL (Extract, Transform, Load) development can unlock numerous career opportunities in data management and analytics. By mastering ETL skills, individuals can integrate data from various sources, ensuring its quality and efficiency for decision-making processes. To embark on this journey, understanding ETL concepts, acquiring SQL proficiency, familiarizing oneself with ETL tools like Informatica or Talend, practicing with real-world datasets, and staying updated with industry trends are essential steps. Institutes offering specialized ETL Development training in Pune provide hands-on experience and expert guidance, preparing individuals for success in the dynamic field of data engineering. Benefits of learning ETL development include lucrative career prospects, data integration skills, data quality assurance, efficiency through automation, and the ability to derive actionable insights from raw data. To start learning ETL development, one should understand the core concepts, learn SQL, explore ETL tools, practice with real-world data, stay updated with industry trends, and build a portfolio of projects to showcase expertise. Optimizing ETL workflows for performance and scalability involves strategies such as data profiling, incremental loading, parallel processing, partitioning, indexing, data compression, memory management, resource monitoring, and scalable architecture design. Handling data consistency issues in ETL development requires strategies like data profiling and cleansing, validation rules, transformation and mapping, master data management, change data capture, error handling and logging, transactional processing, data quality monitoring, documentation, and metadata management. Data lineage tracing is crucial in ETL development for understanding data flow, identifying dependencies, ensuring data quality, compliance, and auditing. It is implemented through metadata management, data lineage tools, annotation and documentation, logging and monitoring, and integration with ETL tools. By mastering ETL development and implementing data lineage tracing effectively, individuals and organizations can enhance data management, analytics, and decision-making processes.
Domain Driven Design
I presented Domain Driven Design (and Command Query Separation) internally to my company.
FOWA Scaling The Lamp Stack Workshop
Slides from the workshop "Scaling the LAMP Stack" at the Future of Web Apps on October 5, 2007
Native eDiscovery for Lotus Notes
Description: Learn how to preserve and manage native Lotus Notes documents through the entire eDiscovery process. This webinar will provide you with methodology, tips, processes and important legal considerations such as:
-What are the best practices for conducting eDiscovery involving Lotus Notes data?
-How the legal and IT teams collaborate to help streamline eDiscovery.
-How to effectively capture and preserve native Lotus Notes email and application data for eDiscovery.
-How data fidelity is impacted when converting native Lotus Notes to PST or EML and minimizing risks related to same.
-How native processing of Lotus Notes allows improved TIFF rendering and maintains data fidelity.
-How to identify potentially relevant data, stored in non-email Lotus Notes files – unlocking and accessing content stored in non-email data.
-Discussion of how email handling differs from non-email handling.
Python + MPP Database = Large Scale AI/ML Projects in Production Faster
ODSC East virtual presentation - The best machine learning, and advanced analytics projects are often stopped when it comes time to move into large scale production, preventing them from ever impacting the business in a meaningful way. Hundreds of hours of work may never get put to use.
Python is rapidly becoming the language of choice for scientists and researchers of many types to build, test, train and score models. But when data science models need to go into production, challenges of performance and scale can be a huge roadblock.
By combining a Python application with an underlying massively parallel (MPP) database, Python users can achieve a simplified path to production. An MPP database also allows you to do data preparation and data analysis at far greater speeds, accelerating development and testing as well as production performance. It also allows greater numbers of concurrent jobs to run, while also continuously loading data for IoT or other streaming use cases.
Analyze data in the database where it sits, rather than first moving it to another framework, then analyzing it, then moving the results, taking multiple performance hits from both CPU and IO for every move and transformation.
In this talk, you will learn about combination architectures that can get your work into production, shorten development time, and provide the performance and scale advantages of an MPP database with the convenience and power of Python. Use case examples use the open source Vertica-Python project created by Uber with contributions from Twitter, Palantir, Etsy, Vertica, Kayak and Gooddata.
Introduction to Data Warehousing
A Brief History of Information Technology
Databases for Decision Support
OLTP vs. OLAP
Why OLAP & OLTP don’t mix (1)
Organizational Data Flow and Data Storage Components
Loading the Data Warehouse
Characteristics of a Data Warehouse
A Data Warehouse is Subject Oriented
For more visit : http://jsbi.blogspot.com
Similar to Hand Coding ETL Scenarios and Challenges (20)
Getting Unstuck: Working with Legacy Code and Data
Getting Unstuck: Working with Legacy Code and Data
Etl data processing system which is very useful for the engineering students
Etl data processing system which is very useful for the engineering students
Building High Performance MySQL Query Systems and Analytic Applications
Building High Performance MySQL Query Systems and Analytic Applications
Building High Performance MySql Query Systems And Analytic Applications
Building High Performance MySql Query Systems And Analytic Applications
Unlocking the Power of Your Data: Working with Databases in FME
Unlocking the Power of Your Data: Working with Databases in FME
What are the benefits of learning ETL Development and where to start learning...
What are the benefits of learning ETL Development and where to start learning...
Python + MPP Database = Large Scale AI/ML Projects in Production Faster
Python + MPP Database = Large Scale AI/ML Projects in Production Faster
More from mark madsen
Data Architecture: OMG It’s Made of People
Do you have data? Do you have users? Do they use that data to solve problems? Then you have a data architecture. Maybe your architecture is organic and accidental, or maybe it’s an accumulation of the latest practices and technologies you heard about on Stack Overflow.
Spoiler: data architecture is about people and how they use data, not the latest pipeline framework or AI model. Data architecture is about enabling users to be productive, not adding the next “shiny object” and then blaming the users for using it wrong. What you design needs to focus on a different subject than either technology or data.
Join Kevin Bogusch, Ecosystem Architect, as he talks with Mark Madsen, Fellow at the Technology Innovation Office, on the crucial elements you’re missing in a successful data architecture: people and process. Find out why Mark says, “don’t buy one problem to solve another problem.”
Solve User Problems: Data Architecture for Humans
We are bombarded with stories of the latest products to hit the market – products that will change everything we do. This causes us to focus on the latest technology, building IT for the sake of building IT. Meanwhile, the world still seems to run on Excel.
The “big innovators” who have and use unimaginably large amounts of data are not the norm. Aspiring to use the same complex technologies and patterns they do leads to poor investments and tradeoffs. This is an age-old problem rooted in the over-emphasis of technology as the agent of change. Technology isn’t the answer – it’s the platform on which people build answers.
To emphasize technology is to ignore the way tools change people and practices. The design focus in our market was on storing and making data accessible. If we want to make progress then we need to step back from the details and look at data from the perspective of the organization. Our design focus shifts to people learning and applying new insights, asking questions about how an organization can be more resilient, more efficient, or faster to sense and respond to changing conditions.
In this talk you will learn how to put your data architecture into a human frame of reference. Drawing inspiration from the history of technology and urban planning, we will see that the services provided by the things we build are what drive success, not the latest shiny distraction.
The Black Box: Interpretability, Reproducibility, and Data Management
The growing complexity of data science leads to black box solutions that few people in an organization understand. You often hear about the difficulty of interpretability—explaining how an analytic model works—and that you need it to deploy models. But people use many black boxes without understanding them…if they’re reliable. It’s when the black box becomes unreliable that people lose trust.
Mistrust is more likely to be created by the lack of reliability, and the lack of reliability is often the result of misunderstanding essential elements of analytics infrastructure and practice. The concept of reproducibility—the ability to get the same results given the same information—extends your view to include the environment and the data used to build and execute models.
Mark Madsen examines reproducibility and the areas that underlie production analytics and explores the most frequently ignored and yet most essential capability, data management. The industry needs to consider its practices so that systems are more transparent and reliable, improving trust and increasing the likelihood that your analytic solutions will succeed.
This talk will treat the black boxed of ML the way management perceives them, as black boxes.
There is much work on explainable models, interpretability, etc. that are important to the task of reproducibility. Much of that is relevant to the practitioner, but the practitioner can become too focused on the part they are most familiar with and focused on. Reproducing the results needs more.
Operationalizing Machine Learning in the Enterprise
TDWI Munich 2019
What does it take to operationalize machine learning and AI in an enterprise setting?
Machine learning in an enterprise setting is difficult, but it seems easy. All you need is some smart people, some tools, and some data. It’s a long way from the environment needed to build ML applications to the environment to run them in an enterprise.
Most of what we know about production ML and AI come from the world of web and digital startups and consumer services, where ML is a core part of the services they provide. These companies have fewer constraints than most enterprises do.
This session describes the nature of ML and AI applications and the overall environment they operate in, explains some important concepts about production operations, and offers some observations and advice for anyone trying to build and deploy such systems.
Building a Data Platform Strata SF 2019
Building a data lake involves more than installing Hadoop or putting data into AWS. The goal in most organizations is to build multi-use data infrastructure that is not subject to past constraints. This tutorial covers design assumptions, design principles, and how to approach the architecture and planning for multi-use data infrastructure in IT.
[This is a new, changed version of the presentations of the same title from last year's Strata]
Architecting a Data Platform For Enterprise Use (Strata NY 2018)
Building a data lake involves more than installing Hadoop or putting data into AWS. The goal in most organizations is to build multi-use data infrastructure that is not subject to past constraints. This tutorial covers design assumptions, design principles, and how to approach the architecture and planning for multi-use data infrastructure in IT.
Long:
The goal in most organizations is to build multi-use data infrastructure that is not subject to past constraints. This session will discuss hidden design assumptions, review design principles to apply when building multi-use data infrastructure, and provide a reference architecture to use as you work to unify your analytics infrastructure.
The focus in our market has been on acquiring technology, and that ignores the more important part: the larger IT landscape within which this technology lives and the data architecture that lies at its core. If one expects longevity from a platform then it should be a designed rather than accidental architecture.
Architecture is more than just software. It starts from use and includes the data, technology, methods of building and maintaining, and organization of people. What are the design principles that lead to good design and a functional data architecture? What are the assumptions that limit older approaches? How can one integrate with, migrate from or modernize an existing data environment? How will this affect an organization's data management practices? This tutorial will help you answer these questions.
Topics covered:
* A brief history of data infrastructure and past design assumptions
* Categories of data and data use in organizations
* Data architecture
* Functional architecture
* Technology planning assumptions and guidance
Architecting a Platform for Enterprise Use - Strata London 2018
The goal in most organizations is to build multi-use data infrastructure that is not subject to past constraints. This session will discuss hidden design assumptions, review design principles to apply when building multi-use data infrastructure, and provide a reference architecture to use as you work to unify your analytics infrastructure.
The focus in our market has been on acquiring technology, and that ignores the more important part: the larger IT landscape within which this technology lives and the data architecture that lies at its core. If one expects longevity from a platform then it should be a designed rather than accidental architecture.
Architecture is more than just software. It starts from use and includes the data, technology, methods of building and maintaining, and organization of people. What are the design principles that lead to good design and a functional data architecture? What are the assumptions that limit older approaches? How can one integrate with, migrate from or modernize an existing data environment? How will this affect an organization's data management practices? This tutorial will help you answer these questions.
Topics covered:
* A brief history of data infrastructure and past design assumptions
* Categories of data and data use in organizations
* Analytic workload characteristics and constraints
* Data architecture
* Functional architecture
* Tradeoffs between different classes of technology
* Technology planning assumptions and guidance
#strataconf
A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
A hotspot of diversity for rare plants, butterflies and birds, the Klamath-Siskiyou region of southern Oregon is a scientist's (and naturalist's) paradise. This is transverse range running from the Cascades range to the Pacific Ocean, creating an east-west corridor between the coast and the volcanic Cascades range. Mark Madsen’s love of biology while living in the area for 15 years sparked an interest in botanical taxonomy in the world of serpentine soils and the plant communities thriving in the region, including remnant species from the last ice age.
How to understand trends in the data & software market
The big challenge most analytics and IT professionals face today is dealing with complexity. Trends are still not clear. It helps to look at the past and current state to understand what’s really happening in the data technology market – a whole lot of reinvention and some innovation, but not where you expect it.
We have the (well-understood) problems that we have, with their (well-understood) limitations and intractabilities.
We deal with them in the world in which they were first codified and framed. Paradigms (world views) change as a function of political, economic, technological, cultural, use and growth, however, and when the world changes we’ll have a criteria for framing not just the problems/shortcomings/intractabilities of the prior paradigm, but that paradigm itself.
At that point, however, it will have ceased to matter because we’ll be dealing with fundamentally new problems/shortcomings/intractabilities.
Pay no attention to the man behind the curtain - the unseen work behind data ...
Goal: explain the nature of the work of an analytics team to a manager, and enable people on those teams to explain what a data science team needs to a manager.
It seems as if every organization wants to enable analytical-decision making and embed analytics into operational processes. What can you do with analytics? It looks like anything is possible. What can you really do? Probably a lot less than you expect. Why is this? Vendors promise easy-to-use analytics tools and services but they rarely deliver. The products may be easy but the work is still hard.
Using analytics to solve problems depends on many factors beyond the math: people, processes, the skills of the analyst, the technology used, the data. Technology is the easy part. Figuring out what to do and how to do it is a lot harder. Despite this, fancy new tools get all the attention and budget.
People and data are the truly hard parts. People, because many believe that data is absolute rather than relative, and that analytic models produce an answer rather than a range of answers with varying degrees of truth, accuracy and applicability. Data, because managing data for analytics is a nuanced, detail-oriented and seemingly dull task left to back-office IT.
If your goal is to build a repeatable analytics capability rather than a one-off analytics project then you will need to address the parts that are rarely mentioned. This talk will explain some of the unseen and little-discussed aspects involved when building and deploying analytics.
Assumptions about Data and Analysis: Briefing room webcast slides
In many ways, moving data is like moving furniture: it's an unpleasant process dubbed an occasional necessary evil. But as the data pipelines of old decay, a new reality is taking shape: the data-native architecture. Unlike traditional data processing for BI and Analytics, this approach works on data right where it lives, thus eliminating the pain of forklifting, narrowing the margin of error, and expediting the time to business benefit. The new architecture embodies new assumptions, some of which we will talk about here.
Register for this episode of The Briefing Room to hear veteran Analyst Mark Madsen of Third Nature explain why this shift is truly tectonic. He'll be briefed by Steve Wooledge of Arcadia Data who will showcase his company's technology, which leverages a data-native architecture to fuel rapid-fire visualization and analysis of both big data and small.
Everything Has Changed Except Us: Modernizing the Data Warehouse
Keynote, Munich, June 2016
The way we make decisions has changed. The data we use has changed. The techniques we can apply to data and decisions have changed. Yet what we build and how we build it has barely changed in 20 years.
The definition of madness is doing more of what you already do and expecting different results. The threat to the data warehouse is not from new technology that will replace the data warehouse. It is from destabilization caused by new technology as it changes the architecture, and from failure to adapt to those changes.
The technology that we use is problematic because it constrains and sometimes prevents necessary activities. We don’t need more technology and bigger machines. We need different technology that does different things. More product features from the same vendors won’t solve the problem.
The data we want to use is challenging. We can’t model and clean and maintain it fast enough. We don’t need more data modeling to solve this problem. We need less modeling and more metadata.
And lastly, a change in scale has occurred. It isn’t a simple problem of “big”. The problem with current workloads has been solved, despite the performance problems that many people still have today. Scale has many dimensions – important among them are the number of discrete sources and structures, the rate of change of individual structures, the rate of change in data use, the variety of uses and the concurrency of those uses.
In short, we need new architecture that is not focused on creating stability in data, but one that is adaptable to continuous and rapidly changing uses of data.
A Pragmatic Approach to Analyzing Customers
The business market is different today than it was 20 years ago when BI got started. We're just beginning to grasp how to work within the new economic and communication models. Companies can't rely solely on financial and operational metrics any more, and need to analyze customer behaviors in more detail.
The big change in analysis is a move from mass market metrics to individualized data, no longer analyzing or managing by averages. The stream of events and observations available from applications today combined with new platforms for collecting and processing data enables (relatively) easy analysis.
Despite this, many companies struggle to analyze customer data. This talk will describe a handful of customer metrics and models that are (relatively) easy to do, yet are often not done. It's often easier to succeed by stringing together a handful of simple techniques rather than applying advanced techniques.
Expect to come away from this session with:
- a little history of customer data use by marketing and how that has changed in the last 10 years.
- the most common behavioral data sources you have available.
- some of the basic questions that often go unanswered, and data that is not assessed in the proper context.
- some basic analyses you can perform.
Disruptive Innovation: how do you use these theories to manage your IT?
The term disruptive innovation was popularized by Harvard professor Clayton Christensen in his 1997 book “The Innovator’s Dilemma.” Nearly 20 years later “Disrupt!” is a popular leadership mantra that is more frequently uttered than experienced. You can't productize it. You can't always control it – at least what effects it has in practice. You aren't necessarily going to like every product of innovation. So are you sure you want it? If so, how do you promote a culture in which innovation can flower – and, potentially, thrive? Because that's probably the best that you can do.
Perhaps there's a better framing for innovation than just "disruption.“ This session is an overview of commmoditization and innovation theories followed by basic things you can do to apply that theory to your daily job architecting, choosing and managing a data environment in your company.
Briefing room: An alternative for streaming data collection
Knowing what’s happening in your enterprise right now can mark the difference between success and failure. The key is to have a rich view of activity, such that analysts and others can explore in a fully multidimensional fashion. Benefiting from such a detailed perspective can help professionals identify the exact nature of problems or opportunities, thus enabling precise actions that make a difference quickly.
Register for this episode of The Briefing Room to hear veteran Analyst Mark Madsen of Third Nature explain how a nexus of innovations for analyzing network traffic can help companies stay on top of their game. He’ll be briefed by Erik Giesa of ExtraHop, who will showcase his company’s stream analytics technology for wire data, which provides real-time, multidimensional views of network traffic. He’ll share success stories of how ExtraHop has solved otherwise intractable problems and enabled a new level of root-cause analysis.
Building the Enterprise Data Lake: A look at architecture
The topic is building an Enterprise Data Lake, discussing high level data and technology architecture. We will describe the architecture of a data warehouse, how a data lake needs to differ, and show a high level functional and data architecture for a data lake. This webinar will cover:
Why dumping data into Hadoop and letting users get it out doesn't work
The difference between a Hadoop application and a Data Lake
Why new ideas about data architecture are a key element
An Enterprise Data Lake reference architecture to frame what must be built
Briefing Room analyst comments - streaming analytics
Slides for Briefing Room webcast ( https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=869f964b1380f728cedde802779a1e12 )
Organizations worldwide are learning hard lessons these days about the constraints of dated information systems. The time-tested process of Extract-Transform-Load (ETL) is fast losing its ability to cope with the volume, velocity and variety of Big Data coming down the pike. Forward-thinking companies are therefore prepping the battle field by designing on-ramps to the future of streaming analytics. Register for this episode of The Briefing Room to hear Analyst Mark Madsen explain how a new era of data solutions is rising to the challenge of streaming data. He'll be briefed by Steve Wilkes, founder and CTO of the Striim platform. Steve will share how enterprises are turning to streaming data integration, in-memory transformations and continuous processing to achieve the goals of ETL in milliseconds – at a fraction of the cost and complexity of legacy systems. Several case studies will be shared.
Everything has changed except us
The way we make decisions has changed. The data we use has changed. The techniques we can apply to data and decisions have changed. Yet what we build and how we build it has barely changed in 20 years.
The definition of madness is doing more of what you already do and expecting different results. The threat to the data warehouse is not from new technology that will replace the data warehouse. It is from destabilization caused by new technology as it changes the architecture, and from failure to adapt to those changes.
The technology that we use is problematic because it constrains and sometimes prevents necessary activities. We don’t need more technology and bigger machines. We need different technology that does different things. More product features from the same vendors won’t solve the problem.
The data we want to use is challenging. We can’t model and clean and maintain it fast enough. We don’t need more data modeling to solve this problem. We need less modeling and more metadata.
And lastly, a change in scale has occurred. It isn’t a simple problem of “big”. The problem with current workloads has been solved, despite the performance problems that many people still have today. Scale has many dimensions – important among them are the number of discrete sources and structures, the rate of change of individual structures, the rate of change in data use, the variety of uses and the concurrency of those uses.
In short, we need new architecture that is not focused on creating stability in data, but one that is adaptable to continuous and rapidly changing uses of data.
Bi isn't big data and big data isn't BI (updated)
Big data is hyped, but isn't hype. There are definite technical, process and business differences in the big data market when compared to BI and data warehousing, but they are often poorly understood or explained. BI isn't big data, and big data isn't BI. By distilling the technical and process realities of big data systems and projects we can separate fact from fiction. This session examines the underlying assumptions and abstractions we use in the BI and DW world, the abstractions that evolved in the big data world, and how they are different. Armed with this knowledge, you will be better able to make design and architecture decisions. The session is sometimes conceptual, sometimes detailed technical explorations of data, processing and technology, but promises to be entertaining regardless of the level.
Yes, it’s about the data normally called “big”, but it’s not Hadoop for the database crowd, despite the prominent role Hadoop plays. The session will be technical, but in a technology preview/overview fashion. I won’t be teaching you to write MapReduce jobs or anything of the sort.
The first part will be an overview of the types, formats and structures of data that aren’t normally in the data warehouse realm. The second part will cover some of the basic technology components, vendors and architecture.
The goal is to provide an overview of the extent of data available and some of the nuances or challenges in processing it, coupled with some examples of tools or vendors that may be a starting point if you are building in a particular area.
On the edge: analytics for the modern enterprise (analyst comments)
On the Edge: Analytics for the Modern Enterprise
[these are the analyst comments on enterprise data architecture and streaming]
Webcast description: The speed of business today requires new approaches to generating and leveraging analytics. Latencies of a day, an hour or even minutes no longer suffice in many situations. For these use cases, organizations must embrace analytics at the edge: a process that involves targeted number-crunching at the fringe of the enterprise. When designed properly, these systems give companies a leg up on their competitors. Register for this episode of The Briefing Room to hear veteran Analyst Mark Madsen of Third Nature explain how a new era of information architectures is now unfolding, paving the way to much more responsive and agile business models. He'll be briefed by Kim Macpherson of the Cisco Data and Analytics Business Unit, who will explain how her company's platform is uniquely suited for this new, federated analytic paradigm. She'll demonstrate how edge analytics can help companies address opportunities quickly and effectively.
More from mark madsen (20)
The Black Box: Interpretability, Reproducibility, and Data Management
The Black Box: Interpretability, Reproducibility, and Data Management
Operationalizing Machine Learning in the Enterprise
Operationalizing Machine Learning in the Enterprise
Architecting a Data Platform For Enterprise Use (Strata NY 2018)
Architecting a Data Platform For Enterprise Use (Strata NY 2018)
Architecting a Platform for Enterprise Use - Strata London 2018
Architecting a Platform for Enterprise Use - Strata London 2018
A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
How to understand trends in the data & software market
How to understand trends in the data & software market
Pay no attention to the man behind the curtain - the unseen work behind data ...
Pay no attention to the man behind the curtain - the unseen work behind data ...
Assumptions about Data and Analysis: Briefing room webcast slides
Assumptions about Data and Analysis: Briefing room webcast slides
Everything Has Changed Except Us: Modernizing the Data Warehouse
Everything Has Changed Except Us: Modernizing the Data Warehouse
Disruptive Innovation: how do you use these theories to manage your IT?
Disruptive Innovation: how do you use these theories to manage your IT?
Briefing room: An alternative for streaming data collection
Briefing room: An alternative for streaming data collection
Building the Enterprise Data Lake: A look at architecture
Building the Enterprise Data Lake: A look at architecture
Briefing Room analyst comments - streaming analytics
Briefing Room analyst comments - streaming analytics
On the edge: analytics for the modern enterprise (analyst comments)
On the edge: analytics for the modern enterprise (analyst comments)
Recently uploaded
Memorandum Of Association Constitution of Company.ppt
www.seribangash.com
A Memorandum of Association (MOA) is a legal document that outlines the fundamental principles and objectives upon which a company operates. It serves as the company's charter or constitution and defines the scope of its activities. Here's a detailed note on the MOA:
Contents of Memorandum of Association:
Name Clause: This clause states the name of the company, which should end with words like "Limited" or "Ltd." for a public limited company and "Private Limited" or "Pvt. Ltd." for a private limited company.
https://seribangash.com/article-of-association-is-legal-doc-of-company/
Registered Office Clause: It specifies the location where the company's registered office is situated. This office is where all official communications and notices are sent.
Objective Clause: This clause delineates the main objectives for which the company is formed. It's important to define these objectives clearly, as the company cannot undertake activities beyond those mentioned in this clause.
www.seribangash.com
Liability Clause: It outlines the extent of liability of the company's members. In the case of companies limited by shares, the liability of members is limited to the amount unpaid on their shares. For companies limited by guarantee, members' liability is limited to the amount they undertake to contribute if the company is wound up.
https://seribangash.com/promotors-is-person-conceived-formation-company/
Capital Clause: This clause specifies the authorized capital of the company, i.e., the maximum amount of share capital the company is authorized to issue. It also mentions the division of this capital into shares and their respective nominal value.
Association Clause: It simply states that the subscribers wish to form a company and agree to become members of it, in accordance with the terms of the MOA.
Importance of Memorandum of Association:
Legal Requirement: The MOA is a legal requirement for the formation of a company. It must be filed with the Registrar of Companies during the incorporation process.
Constitutional Document: It serves as the company's constitutional document, defining its scope, powers, and limitations.
Protection of Members: It protects the interests of the company's members by clearly defining the objectives and limiting their liability.
External Communication: It provides clarity to external parties, such as investors, creditors, and regulatory authorities, regarding the company's objectives and powers.
https://seribangash.com/difference-public-and-private-company-law/
Binding Authority: The company and its members are bound by the provisions of the MOA. Any action taken beyond its scope may be considered ultra vires (beyond the powers) of the company and therefore void.
Amendment of MOA:
While the MOA lays down the company's fundamental principles, it is not entirely immutable. It can be amended, but only under specific circumstances and in compliance with legal procedures. Amendments typically require shareholder
ModelingMarketingStrategiesMKS.CollumbiaUniversitypdf
Implicitly or explicitly all competing businesses employ a strategy to select a mix
of marketing resources. Formulating such competitive strategies fundamentally
involves recognizing relationships between elements of the marketing mix (e.g.,
price and product quality), as well as assessing competitive and market conditions
(i.e., industry structure in the language of economics).
Premium MEAN Stack Development Solutions for Modern Businesses
Stay ahead of the curve with our premium MEAN Stack Development Solutions. Our expert developers utilize MongoDB, Express.js, AngularJS, and Node.js to create modern and responsive web applications. Trust us for cutting-edge solutions that drive your business growth and success.
Know more: https://www.synapseindia.com/technology/mean-stack-development-company.html
VAT Registration Outlined In UAE: Benefits and Requirements
Vat Registration is a legal obligation for businesses meeting the threshold requirement, helping companies avoid fines and ramifications. Contact now!
https://viralsocialtrends.com/vat-registration-outlined-in-uae/
The effects of customers service quality and online reviews on customer loyal...
The effects of customers service quality and online reviews on customer loyal...balatucanapplelovely
Research 20240425_ TJ Communications Credentials_compressed.pdf
"𝑩𝑬𝑮𝑼𝑵 𝑾𝑰𝑻𝑯 𝑻𝑱 𝑰𝑺 𝑯𝑨𝑳𝑭 𝑫𝑶𝑵𝑬"
𝐓𝐉 𝐂𝐨𝐦𝐬 (𝐓𝐉 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬) is a professional event agency that includes experts in the event-organizing market in Vietnam, Korea, and ASEAN countries. We provide unlimited types of events from Music concerts, Fan meetings, and Culture festivals to Corporate events, Internal company events, Golf tournaments, MICE events, and Exhibitions.
𝐓𝐉 𝐂𝐨𝐦𝐬 provides unlimited package services including such as Event organizing, Event planning, Event production, Manpower, PR marketing, Design 2D/3D, VIP protocols, Interpreter agency, etc.
Sports events - Golf competitions/billiards competitions/company sports events: dynamic and challenging
⭐ 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐝 𝐩𝐫𝐨𝐣𝐞𝐜𝐭𝐬:
➢ 2024 BAEKHYUN [Lonsdaleite] IN HO CHI MINH
➢ SUPER JUNIOR-L.S.S. THE SHOW : Th3ee Guys in HO CHI MINH
➢FreenBecky 1st Fan Meeting in Vietnam
➢CHILDREN ART EXHIBITION 2024: BEYOND BARRIERS
➢ WOW K-Music Festival 2023
➢ Winner [CROSS] Tour in HCM
➢ Super Show 9 in HCM with Super Junior
➢ HCMC - Gyeongsangbuk-do Culture and Tourism Festival
➢ Korean Vietnam Partnership - Fair with LG
➢ Korean President visits Samsung Electronics R&D Center
➢ Vietnam Food Expo with Lotte Wellfood
"𝐄𝐯𝐞𝐫𝐲 𝐞𝐯𝐞𝐧𝐭 𝐢𝐬 𝐚 𝐬𝐭𝐨𝐫𝐲, 𝐚 𝐬𝐩𝐞𝐜𝐢𝐚𝐥 𝐣𝐨𝐮𝐫𝐧𝐞𝐲. 𝐖𝐞 𝐚𝐥𝐰𝐚𝐲𝐬 𝐛𝐞𝐥𝐢𝐞𝐯𝐞 𝐭𝐡𝐚𝐭 𝐬𝐡𝐨𝐫𝐭𝐥𝐲 𝐲𝐨𝐮 𝐰𝐢𝐥𝐥 𝐛𝐞 𝐚 𝐩𝐚𝐫𝐭 𝐨𝐟 𝐨𝐮𝐫 𝐬𝐭𝐨𝐫𝐢𝐞𝐬."
Digital Transformation and IT Strategy Toolkit and Templates
This Digital Transformation and IT Strategy Toolkit was created by ex-McKinsey, Deloitte and BCG Management Consultants, after more than 5,000 hours of work. It is considered the world's best & most comprehensive Digital Transformation and IT Strategy Toolkit. It includes all the Frameworks, Best Practices & Templates required to successfully undertake the Digital Transformation of your organization and define a robust IT Strategy.
Editable Toolkit to help you reuse our content: 700 Powerpoint slides | 35 Excel sheets | 84 minutes of Video training
This PowerPoint presentation is only a small preview of our Toolkits. For more details, visit www.domontconsulting.com
falcon-invoice-discounting-a-premier-platform-for-investors-in-india
Falcon stands out as a top-tier P2P Invoice Discounting platform in India, bridging esteemed blue-chip companies and eager investors. Our goal is to transform the investment landscape in India by establishing a comprehensive destination for borrowers and investors with diverse profiles and needs, all while minimizing risk. What sets Falcon apart is the elimination of intermediaries such as commercial banks and depository institutions, allowing investors to enjoy higher yields.
3.0 Project 2_ Developing My Brand Identity Kit.pptx
A personal brand exploration presentation summarizes an individual's unique qualities and goals, covering strengths, values, passions, and target audience. It helps individuals understand what makes them stand out, their desired image, and how they aim to achieve it.
Putting the SPARK into Virtual Training.pptx
This 60-minute webinar, sponsored by Adobe, was delivered for the Training Mag Network. It explored the five elements of SPARK: Storytelling, Purpose, Action, Relationships, and Kudos. Knowing how to tell a well-structured story is key to building long-term memory. Stating a clear purpose that doesn't take away from the discovery learning process is critical. Ensuring that people move from theory to practical application is imperative. Creating strong social learning is the key to commitment and engagement. Validating and affirming participants' comments is the way to create a positive learning environment.
LA HUG - Video Testimonials with Chynna Morgan - June 2024
Have you ever heard that user-generated content or video testimonials can take your brand to the next level? We will explore how you can effectively use video testimonials to leverage and boost your sales, content strategy, and increase your CRM data.🤯
We will dig deeper into:
1. How to capture video testimonials that convert from your audience 🎥
2. How to leverage your testimonials to boost your sales 💲
3. How you can capture more CRM data to understand your audience better through video testimonials. 📊
Kseniya Leshchenko: Shared development support service model as the way to ma...
Kseniya Leshchenko: Shared development support service model as the way to make small projects with small budgets profitable for the company (UA)
Kyiv PMDay 2024 Summer
Website – www.pmday.org
Youtube – https://www.youtube.com/startuplviv
FB – https://www.facebook.com/pmdayconference
Unveiling the Secrets How Does Generative AI Work.pdf
At its core, generative artificial intelligence relies on the concept of generative models, which serve as engines that churn out entirely new data resembling their training data. It is like a sculptor who has studied so many forms found in nature and then uses this knowledge to create sculptures from his imagination that have never been seen before anywhere else. If taken to cyberspace, gans work almost the same way.
Recently uploaded (20)
Memorandum Of Association Constitution of Company.ppt
Memorandum Of Association Constitution of Company.ppt
ModelingMarketingStrategiesMKS.CollumbiaUniversitypdf
ModelingMarketingStrategiesMKS.CollumbiaUniversitypdf
Premium MEAN Stack Development Solutions for Modern Businesses
Premium MEAN Stack Development Solutions for Modern Businesses
VAT Registration Outlined In UAE: Benefits and Requirements
VAT Registration Outlined In UAE: Benefits and Requirements
The effects of customers service quality and online reviews on customer loyal...
The effects of customers service quality and online reviews on customer loyal...
20240425_ TJ Communications Credentials_compressed.pdf
20240425_ TJ Communications Credentials_compressed.pdf
Digital Transformation and IT Strategy Toolkit and Templates
Digital Transformation and IT Strategy Toolkit and Templates
The Influence of Marketing Strategy and Market Competition on Business Perfor...
The Influence of Marketing Strategy and Market Competition on Business Perfor...
falcon-invoice-discounting-a-premier-platform-for-investors-in-india
falcon-invoice-discounting-a-premier-platform-for-investors-in-india
3.0 Project 2_ Developing My Brand Identity Kit.pptx
3.0 Project 2_ Developing My Brand Identity Kit.pptx
LA HUG - Video Testimonials with Chynna Morgan - June 2024
LA HUG - Video Testimonials with Chynna Morgan - June 2024
Kseniya Leshchenko: Shared development support service model as the way to ma...
Kseniya Leshchenko: Shared development support service model as the way to ma...
Unveiling the Secrets How Does Generative AI Work.pdf
Unveiling the Secrets How Does Generative AI Work.pdf
Hand Coding ETL Scenarios and Challenges
- 1. Challenges and Techniques When Hand-coding ETL TDWI Webcast Series June, 26 2007 Mark Madsen http://ThirdNature.net
- 3. One Fact: ETL Market Share Sources: Forrester Research, TDWI, private survey