Introducción a la arquitectura Data Lake con Azure

•Download as PPTX, PDF•

0 likes•812 views

En esta charla la idea es ilustrar cómo funciona y qué componentes forman parte de una arquitectura Data Lake clásica pero usando tecnología cloud de Azure. La charla se dividirá en:1. ¿De dónde veníamos? DW clásico y primeras aproximaciones usando Hadoop2. Introducción del concepto de Data Lakea. Cambio de arquitecturab. Desacoplamiento de datos, consumidores y potencia de cálculoc. Descripción de componentes principales: Azure Data Lake Store, HDInsight, Data Factory, Azure SQL database3. Demo end-to-end de la arquitectura funcionando4. Lambda architecture como manera de extender a escenario real time5. Importancia del Data Warehouse en escenarios “big data” (Apache Kylin)

Introducción a la
arquitectura Data Lake

@daesgar90
describano@plainconcepts.com
Daniel Escribano García Rafael Gómez García
rgomez@plainconcepts.com

¿Qué vamos a ver?
• Data Warehouse clásico
• Arquitectura Data Lake
• Demo técnica
• Arquitectura Lambda, escenario real-time

Componentes de un DataWarehouse
clásico
DB Source
Cloud
Prem
File Source
Prem
On Premises
Source
SSIS
Staging Dimensional
model

Problemas
Velocidad: Caso Netflix
~500 billion events and ~1.3 PB per day
~8 million events and ~24 GB per second during peak hours
https://medium.com/netflix-techblog/evolution-of-the-netflix-data-pipeline-da246ca36905

DB Source
Cloud
Prem
File Source
Prem
On Premises
Source
SSIS
Staging Dimensional
model
Componentes de un DataWarehouse
clásico

Soluciones
Desacoplamiento de HDFS a su computación

Arquitectura Data Lake conceptual
Consumer client
SQL DB
Consumer client
SQL DB ReportTabular
Consumer client
MLSQL DB
Ingestion
Metadata
Data
Orchestrator
Raw Storage
Blob Storage
(raw)
Preprocesed storage
Data Lake
Storage
(optimized)
Data Lake
Storage
(optimized)
Cluster
DB Source
Cloud
Prem
File Source
Prem
On Premises
Source

Thanks and …
See you soon!
Thanks also to the organization
Without whom this would not have been posible.

This document provides an overview of Azure data analytics architecture using the Lambda architecture pattern. It covers Azure data and services, including ingestion, storage, processing, analysis and interaction services. It provides a brief overview of the Lambda architecture including the batch layer for pre-computed views, speed layer for real-time views, and serving layer. It also discusses Azure data distribution, SQL Data Warehouse architecture and design best practices, and data modeling guidance.

Five Things to Consider About Data Mesh and Data Governance

DATAVERSITY

Data mesh was among the most discussed and controversial enterprise data management topics of 2021. One of the reasons people struggle with data mesh concepts is we still have a lot of open questions that we are not thinking about: Are you thinking beyond analytics? Are you thinking about all possible stakeholders? Are you thinking about how to be agile? Are you thinking about standardization and policies? Are you thinking about organizational structures and roles? Join data.world VP of Product Tim Gasper and Principal Scientist Juan Sequeda for an honest, no-bs discussion about data mesh and its role in data governance.

Apache Spark Overview

Vadim Y. Bichutskiy

This document provides an overview of a talk on Apache Spark. It introduces the speaker and their background. It acknowledges inspiration from a previous Spark training. It then outlines the structure of the talk, which will include: a brief history of big data; a tour of Spark including its advantages over MapReduce; and explanations of Spark concepts like RDDs, transformations, and actions. The document serves to introduce the topics that will be covered in the talk.

Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cl...

Certus Solutions

Snowflake is a cloud data warehouse that provides elasticity, scalability, and simplicity. It allows organizations to consolidate their diverse data sources in one place and instantly scale up or down their compute capacity as needed. Aptus Health, a digital marketing company, used Snowflake to break down data silos, integrate disparate data sources, enable broad data sharing, and provide a scalable and cost-effective solution to meet their analytics needs. Snowflake addressed both business needs for timely access to centralized data and IT needs for flexibility, extensibility, and reducing ETL work.

Databricks Fundamentals

Dalibor Wijas

This document is a training presentation on Databricks fundamentals and the data lakehouse concept by Dalibor Wijas from November 2022. It introduces Wijas and his experience. It then discusses what Databricks is, why it is needed, what a data lakehouse is, how Databricks enables the data lakehouse concept using Apache Spark and Delta Lake. It also covers how Databricks supports data engineering, data warehousing, and offers tools for data ingestion, transformation, pipelines and more.

Data Lakehouse, Data Mesh, and Data Fabric (r1)

James Serra

Big data architectures and the data lake

James Serra

The document provides an overview of big data architectures and the data lake concept. It discusses why organizations are adopting data lakes to handle increasing data volumes and varieties. The key aspects covered include: - Defining top-down and bottom-up approaches to data management - Explaining what a data lake is and how Hadoop can function as the data lake - Describing how a modern data warehouse combines features of a traditional data warehouse and data lake - Discussing how federated querying allows data to be accessed across multiple sources - Highlighting benefits of implementing big data solutions in the cloud - Comparing shared-nothing, massively parallel processing (MPP) architectures to symmetric multi-processing (

Azure Synapse Analytics Overview (r2)

James Serra

Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.

Modern data is massive, quickly evolving, unstructured, and increasingly hard to catalog and understand from multiple consumers and applications. This presentation will guide you though the best practices for designing a robust data architecture, highlightning the benefits and typical challenges of data lakes and data warehouses. We will build a scalable solution based on managed services such as Amazon Athena, AWS Glue, and AWS Lake Formation.

Event Hub & Azure Stream Analytics

Davide Mauri

Being able to analyze data in real-time will be a very hot topic for sure in near future. Not only for IoT-related tasks but as a general approach to user-to-machine or machine-to-machine interaction. From product recommendations to fraud detection alarms, a lot of stuff would be perfect if it could happen in real time. Now, with Azure Event Hubs and Stream Analytics, it’s possible. In this session, Davide will demonstrate how to use Event Hubs to quickly ingest new real-time data and Stream Analytics to query on-the-fly data, in order to do a real-time analysis of what’s happening right now.

Big Data Fabric Capability Maturity Model

Ross Collins

The document outlines the evolution of an organization's data and analytics capabilities from initial siloed and inconsistent usage to a mature and world-class "data-driven" state. Key stages included establishing data stewardship, developing a data value council, implementing a metadata catalogue and defining data quality, and building out an enhanced "Big Data Fabric" platform to support predictive and prescriptive analytics as well as artificial intelligence applications. The organization ultimately reached a world-class level where data drives business value through insights and automation, and the data team coaches others to become "Data Citizens."

Big Data - Conceptos, herramientas y patrones

Juan José Domenech

Achieving Lakehouse Models with Spark 3.0

Databricks

It’s very easy to be distracted by the latest and greatest approaches with technology, but sometimes there’s a reason old approaches stand the test of time. Star Schemas & Kimball is one of those things that isn’t going anywhere, but as we move towards the “Data Lakehouse” paradigm – how appropriate is this modelling technique, and how can we harness the Delta Engine & Spark 3.0 to maximise it’s performance?

Migrating to Cloud: Inhouse Hadoop to Databricks (3)

Knoldus Inc.

Data Governance in a big data era

Pieter De Leenheer

This document discusses data governance challenges in the era of big data and proposes solutions. It begins by outlining the rise of data-driven businesses and the challenges they face with data quality, access, and trust issues. This has led to the rise of the Chief Data Officer role. The document then discusses how data governance approaches need to shift from hierarchical systems of record to more networked systems of engagement to manage expanding data volumes and types from sources like IoT and big data analytics. Key challenges discussed include digitalizing trust in data and addressing risks from opaque big data models. The document proposes taking a hybrid governance approach and implementing a system of record for data assets to provide findability, understandability and trust for all organizational data. Example use

Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...

HostedbyConfluent

Companies are increasingly becoming software-driven, requiring new approaches to software architecture and data integration. The "data mesh" architectural pattern decentralizes data management by organizing it around domain experts and treating data as products that can be accessed on-demand. This helps address issues with centralized data warehouses by evolving data modeling with business needs, avoiding bottlenecks, and giving autonomy to domain teams. Key principles of the data mesh include domain ownership of data, treating data as self-service products, and establishing federated governance to coordinate the decentralized system.

DW Migration Webinar-March 2022.pptx

Databricks

The document discusses migrating a data warehouse to the Databricks Lakehouse Platform. It outlines why legacy data warehouses are struggling, how the Databricks Platform addresses these issues, and key considerations for modern analytics and data warehousing. The document then provides an overview of the migration methodology, approach, strategies, and key takeaways for moving to a lakehouse on Databricks.

Architecture Document Template

Pierre-Marie Delpech

This document provides a template for an architecture document that includes: 1. An introduction section describing the purpose, scope, definitions, and references. 2. A business architecture section describing the business context, processes, events, and impacts. 3. A functional architecture section describing the functional composition, most significant functions/use cases, and functional communication/interactions. 4. Additional sections for application architecture, software architecture requirements, technical architecture, deployment architecture, development strategy, data migration strategy, deployment strategy, configuration and version management strategy, security and conformance constraints, risks, and appendices.

Modern Data architecture Design

Kujambu Murugesan

The document discusses modern data architectures. It presents conceptual models for data ingestion, storage, processing, and insights/actions. It compares traditional vs modern architectures. The modern architecture uses a data lake for storage and allows for on-demand analysis. It provides an example of how this could be implemented on Microsoft Azure using services like Azure Data Lake Storage, Azure Data Bricks, and Azure Data Warehouse. It also outlines common data management functions such as data governance, architecture, development, operations, and security.

Apache Spark and the Hadoop Ecosystem on AWS

Amazon Web Services

The document provides an overview of Apache Spark and Hadoop ecosystem tools on Amazon EMR including Spark, Hive on Tez, and Presto. It discusses building data lakes with Amazon EMR and S3, running jobs and security options, and customer use cases. The demo shows Zeppelin and Hue interfaces. Examples are given of Netflix using Presto on EMR with a 25PB dataset and FINRA saving 60% costs by moving to HBase on EMR.

Modern Data Architecture

Alexey Grishchenko

Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...

Hortonworks

How do you turn data from many different sources into actionable insights and manufacture those insights into innovative information-based products and services? Industry leaders are accomplishing this by adding Hadoop as a critical component in their modern data architecture to build a data lake. A data lake collects and stores data across a wide variety of channels including social media, clickstream data, server logs, customer transactions and interactions, videos, and sensor data from equipment in the field. A data lake cost-effectively scales to collect and retain massive amounts of data over time, and convert all this data into actionable information that can transform your business. Join Hortonworks and Informatica as we discuss: - What is a data lake? - The modern data architecture for a data lake - How Hadoop fits into the modern data architecture - Innovative use-cases for a data lake

CDMP preparation workshop EDW2016

Christopher Bradley

Snowflake Best Practices for Elastic Data Warehousing

Amazon Web Services

Data driven organizations can be challenged to deliver new and growing business intelligence requirements from existing data warehouse platforms, constrained by lack of scalability and performance. The solution for customers is a data warehouse that scales for real-time demands and uses resources in a more optimized and cost-effective manner. Join Snowflake, AWS and Ask.com to learn how Ask.com enhanced BI service levels and decreased expenses while meeting demand to collect, store and analyze over a terabyte of data per day. Snowflake Computing delivers a fast and flexible elastic data warehouse solution that reduces complexity and overhead, built on top of the elasticity, flexibility, and resiliency of AWS. Join us to learn: • Learn how Ask.com eliminates data redundancy, and simplifies and accelerates data load, unload, and administration • Learn how to support new and fluid data consumption patterns with consistently high performance • Best practices for scaling high data volume on Amazon EC2 and Amazon S3 Who should attend: CIOs, CTOs, CDOs, Directors of IT, IT Administrators, IT Architects, Data Warehouse Developers, Database Administrators, Business Analysts and Data Architects

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Cathrine Wilhelmsen

Which Change Data Capture Strategy is Right for You?

Precisely

Change Data Capture or CDC is the practice of moving the changes made in an important transactional system to other systems, so that data is kept current and consistent across the enterprise. CDC keeps reporting and analytic systems working on the latest, most accurate data. Many different CDC strategies exist. Each strategy has advantages and disadvantages. Some put an undue burden on the source database. They can cause queries or applications to become slow or even fail. Some bog down network bandwidth, or have big delays between change and replication. Each business process has different requirements, as well. For some business needs, a replication delay of more than a second is too long. For others, a delay of less than 24 hours is excellent. Which CDC strategy will match your business needs? How do you choose? View this webcast on-demand to learn: • Advantages and disadvantages of different CDC methods • The replication latency your project requires • How to keep data current in Big Data technologies like Hadoop

[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...

DataScienceConferenc1

Building the Data Lake with Azure Data Factory and Data Lake Analytics

Khalid Salama

In essence, a data lake is commodity distributed file system that acts as a repository to hold raw data file extracts of all the enterprise source systems, so that it can serve the data management and analytics needs of the business. A data lake system provides means to ingest data, perform scalable big data processing, and serve information, in addition to manage, monitor and secure the it environment. In these slide, we discuss building data lakes using Azure Data Factory and Data Lake Analytics. We delve into the architecture if the data lake and explore its various components. We also describe the various data ingestion scenarios and considerations. We introduce the Azure Data Lake Store, then we discuss how to build Azure Data Factory pipeline to ingest the data lake. After that, we move into big data processing using Data Lake Analytics, and we delve into U-SQL.

Dean Bryen: Scaling The Platform For Your Startup

huguk

Scaling the Platform for Your Startup

Amazon Web Services

What's hot

Building a Modern Data Platform in the Cloud

Amazon Web Services

Event Hub & Azure Stream Analytics

Davide Mauri

Big Data Fabric Capability Maturity Model

Ross Collins

Big Data - Conceptos, herramientas y patrones

Juan José Domenech

Achieving Lakehouse Models with Spark 3.0

Databricks

Migrating to Cloud: Inhouse Hadoop to Databricks (3)

Knoldus Inc.

Data Governance in a big data era

Pieter De Leenheer

Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...

HostedbyConfluent

DW Migration Webinar-March 2022.pptx

Databricks

Architecture Document Template

Pierre-Marie Delpech

Modern Data architecture Design

Kujambu Murugesan

Apache Spark and the Hadoop Ecosystem on AWS

Amazon Web Services

Modern Data Architecture

Alexey Grishchenko

Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...

Hortonworks

CDMP preparation workshop EDW2016

Christopher Bradley

Snowflake Best Practices for Elastic Data Warehousing

Amazon Web Services

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Cathrine Wilhelmsen

Which Change Data Capture Strategy is Right for You?

Precisely

[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...

DataScienceConferenc1

Building the Data Lake with Azure Data Factory and Data Lake Analytics

Khalid Salama

What's hot (20)

Building a Modern Data Platform in the Cloud

Event Hub & Azure Stream Analytics

Big Data Fabric Capability Maturity Model

Big Data - Conceptos, herramientas y patrones

Achieving Lakehouse Models with Spark 3.0

Migrating to Cloud: Inhouse Hadoop to Databricks (3)

Data Governance in a big data era

Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...

DW Migration Webinar-March 2022.pptx

Architecture Document Template

Modern Data architecture Design

Apache Spark and the Hadoop Ecosystem on AWS

Modern Data Architecture

Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...

CDMP preparation workshop EDW2016

Snowflake Best Practices for Elastic Data Warehousing

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Which Change Data Capture Strategy is Right for You?

[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...

Building the Data Lake with Azure Data Factory and Data Lake Analytics

Similar to Introducción a la arquitectura Data Lake con Azure

Dean Bryen: Scaling The Platform For Your Startup

huguk

Scaling the Platform for Your Startup

Amazon Web Services

Scaling on AWS for the First 10 Million Users at Websummit Dublin

Amazon Web Services

Ian Massingham gave a presentation on scaling applications on AWS from initial launch to over 1 million users. He began by discussing foundational AWS services and database options. He then walked through examples of scaling an application from 1 user to over 500,000 users by leveraging services like EC2, RDS, DynamoDB, ElastiCache, S3, CloudFront, and Auto Scaling. Key strategies included separating components across instances, adding redundancy, implementing caching, and leveraging auto scaling to dynamically scale resources based on demand. Massingham concluded by discussing strategies for scaling beyond 500,000 users such as service-oriented architectures and workload distribution across availability zones.

Scaling on AWS for the First 10 Million Users at Websummit Dublin

Ian Massingham

In this talk from the Dublin Websummit 2014 AWS Technical Evangelist Ian Massingham discusses the techniques that AWS customers can use to create highly scalable infrastructure to support the operation of large scale applications on the AWS cloud. Includes a walk-through of how you can evolve your architecture as your application becomes more popular and you need to scale up your infrastructure to support increased demand.

Building a Scalable Asset Management (DAM) Platform in the AWS

Rahul Shukla

This document outlines the architecture for building a scalable digital asset management (DAM) platform in the cloud. Key components include Amazon S3 for storage, auto scaling EC2 instances for processing metadata and generating renditions, DynamoDB for the catalog, CloudSearch for search, and Elastic Transcoder for transcoding. The architecture provides ingest of assets from S3, metadata extraction using EC2 workers, generation of renditions, building the catalog in DynamoDB and CloudSearch, and delivery of assets through CloudFront.

Scaling Up to Your First 10 Million Users

Amazon Web Services

This document discusses scaling applications on AWS to support millions of users. It begins by introducing AWS global infrastructure services like compute, storage, databases, and networking services. It then discusses starting simply with EC2 instances and expanding horizontally and vertically. The document covers database options on AWS like RDS, DynamoDB, and Redshift. It discusses adding features like auto scaling, load balancing, caching, and content delivery. It proposes a service-oriented architecture with loose coupling. It provides an example reference architecture scaled to support over 1 million users.

Big Data LDN 2018: A TALE OF TWO BI STANDARDS: DATA WAREHOUSES AND DATA LAKES

Matt Stubbs

Date: 13th November 2018 Location: Self-Service Analytics Theatre Time: 14:30 - 15:00 Speaker: Zaf Khan Organisation: Arcadia Data About: The use of data lakes continue to grow, and a recent survey by Eckerson Group shows that organizations are getting real value from their deployments. However, there’s still a lot of room for improvement when it comes to giving business users access to the wealth of potential insights in the data lake. While the data management aspect has been fairly well understood over the years, the success of business intelligence (BI) and analytics on data lakes lags behind. In fact, organizations often struggle with data lakes because they are only accessible by highly-skilled data scientists and not by business users. But BI tools have been able to access data warehouses for years, so what gives? In this talk, we’ll discuss: • Why traditional BI tools are architected well for data warehouses, but not data lakes. • Why every organization should have two BI standards: one for data warehouses and one for data lakes. • Innovative capabilities provided by BI for data lakes

Serverless Microservices - Real life story of a Web App that uses AWS Lambda

Mitoc Group

Scalable applications are by nature resource intensive, expensive to build and difficult to manage. What if we can change this perception and help developers design full-stack applications that are low cost and low maintenance? This session describes the underlying architecture behind www.deep.mg, the microservices marketplace built by Mitoc Group using AngularJS, NodeJS and powered by abstracted services like AWS Lambda, Amazon CloudFront, Amazon DynamoDB, and so on. Eugene Istrati, Technology Partner at Mitoc Group, will dive deep into their approach to microservices architecture using serverless platform from AWS and demonstrate how anyone can use serverless computing to achieve high scalability, high availability, and high performance without huge efforts or expensive resources allocation.

Building a Scalable Digital Asset Management Platform in the Cloud (MED402) |...

Amazon Web Services

With the breadth of AWS services available that are relevant to digital media, organizations can readily build out complete content/asset management (DAM/MAM/CMS) solutions in the cloud. This session provides a detailed walkthrough for implementing a scalable, rich-media asset management platform capable of supporting a variety of industry use cases. The session includes code-level walkthrough, AWS architecture strategies, and integration best practices for content storage, metadata processing, discovery, and overall library management functionality—with particular focus on the use of Amazon S3, Amazon Elastic Transcoder, Amazon DynamoDB and Amazon CloudSearch. Customer case study will highlight successful usage of Amazon CloudSearch by PBS to enable rich discovery of programming content across the breadth of their network catalog.

Lunch and Learn - Store and Move your Data To & From the AWS Cloud, Markku Le...

Amazon Web Services

This document provides an overview and summary of options for storing and moving data to and from AWS cloud storage services. It discusses the problems with traditional on-premise storage solutions and how AWS storage services can help by providing scalable, cost-effective storage across different services optimized for various data and access needs such as block storage, file storage, archive storage, and backup storage. The document also covers data transfer options, choosing the right data store, disaster recovery strategies using AWS, and examples of companies using AWS storage services successfully.

Riga dev day: Lambda architecture at AWS

Antons Kranga

Create cloud service on AWS

Amazon Web Services

Building Scalable Web Applications using Microservices Architecture and Serve...

Mitoc Group

Owning Your Own (Data) Lake House

Data Con LA

Data Con LA 2020 Description In this session, I introduce the Amazon Redshift lake house architecture which enables you to query data across your data warehouse, data lake, and operational databases to gain faster and deeper insights. With a lake house architecture, you can store data in open file formats in your Amazon S3 data lake. Speaker Antje Barth, Amazon Web Services, Sr. Developer Advocate, AI and Machine Learning

Serverless Microservices - Real life story of a Web App that uses AngularJS, ...

Mitoc Group

Microservices Architecture for Content Management Systems using AWS Lambda an...

Mitoc Group

Content Management Systems are by nature resource intensive, expensive to customize, and difficult to manage at scale. What if we can change this perception and help PHP / WordPress developers architect a content platform that is high performance and low cost, high security and low maintenance? This talk will focus on 3 key topics: 1) serverless environment, 2) microservices architecture and 3) hands-on demos. We will describe a serverless solution and propose a scalable architecture that will help WordPress community to adopt cloud-native approach without huge efforts or expensive resources allocation. Eugene Istrati, Technology Partner at Mitoc Group, will dive deep into their approach to microservices architecture using serverless platform from AWS and demonstrate how anyone can use serverless computing to achieve high scalability, high availability, and high performance without huge efforts or expensive resources allocation.

Scaling the Platform for Your Startup

Amazon Web Services

Join AWS at this session to understand how to architect an infrastructure to handle going from zero to millions of users. From leveraging highly scalable AWS services to making smart decisions on building out your application, you'll learn a number of best practices for scaling your infrastructure in the cloud. Speakers: Andreas Chatzakis, AWS Solutions Architect Pete Mounce, Senior Developer, JustEat

Beyond Relational

Lynn Langit

The document discusses building data pipelines in the cloud. It covers serverless data pipeline patterns using services like BigQuery, Cloud Storage, Cloud Dataflow, and Cloud Pub/Sub. It also compares Cloud Dataflow and Cloud Dataproc for ETL workflows. Key questions around ingestion and ETL are discussed, focusing on volume, variety, velocity and veracity of data. Cloud vendor offerings for streaming and ETL are also compared.

Data Con LA 2018 - A tale of two BI standards: Data warehouses and data lakes...

Data Con LA

A tale of two BI standards: Data warehouses and data lakes by Shant Hovsepian, Co-Founder and CTO, Arcadia Data Data lakes as part of the logical data warehouse (LDW) have entered the trough of disillusionment. Some failures are due to lack of value from businesses focusing on the big data challenges and not the big analytics opportunity. After all, data is just data until you analyze it. While the data management aspect has been fairly well understood over the years, the success of business intelligence (BI) and analytics on data lakes lags behind. In fact, data lakes often fail because they are only accessible by highly skilled data scientists and not by business users. But BI tools have been able to access data warehouses for years, so what gives? Shant Hovsepian explains why existing BI tools are architected well for data warehouses but not data lakes, the pros and cons of each architecture, and why every organization should have two BI standards: one for data warehouses and one for data lakes.

Log Analysis At Scale

Amazon Web Services

This document discusses strategies for logging at scale. It notes that logging presents challenges around temporary storage, data capture, permanent storage, and visualization. The document recommends starting with SQL databases and using NoSQL like Elasticsearch for very large datasets or fast data ingest. It presents Amazon Kinesis, Firehose, and Elasticsearch Service as tools to help with data capture, transport, and search. Visualization can be done with Kibana or by loading data into Redshift for use with existing BI tools. The key lessons are to reuse existing technologies when possible and choose the right tools for each part of the logging pipeline.

Similar to Introducción a la arquitectura Data Lake con Azure (20)

Dean Bryen: Scaling The Platform For Your Startup

Scaling the Platform for Your Startup

Scaling on AWS for the First 10 Million Users at Websummit Dublin

Building a Scalable Asset Management (DAM) Platform in the AWS

Scaling Up to Your First 10 Million Users

Big Data LDN 2018: A TALE OF TWO BI STANDARDS: DATA WAREHOUSES AND DATA LAKES

Serverless Microservices - Real life story of a Web App that uses AWS Lambda

Building a Scalable Digital Asset Management Platform in the Cloud (MED402) |...

Lunch and Learn - Store and Move your Data To & From the AWS Cloud, Markku Le...

Riga dev day: Lambda architecture at AWS

Create cloud service on AWS

Building Scalable Web Applications using Microservices Architecture and Serve...

Owning Your Own (Data) Lake House

Serverless Microservices - Real life story of a Web App that uses AngularJS, ...

Microservices Architecture for Content Management Systems using AWS Lambda an...

Scaling the Platform for Your Startup

Beyond Relational

Data Con LA 2018 - A tale of two BI standards: Data warehouses and data lakes...

Log Analysis At Scale

More from Plain Concepts

R y Python con Power BI, la ciencia y el análisis de datos, juntos

Introducción a la arquitectura Data Lake con Azure

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Introducción a la arquitectura Data Lake con Azure

Similar to Introducción a la arquitectura Data Lake con Azure (20)

More from Plain Concepts

More from Plain Concepts (20)

Recently uploaded

Recently uploaded (20)

Introducción a la arquitectura Data Lake con Azure

Editor's Notes