Etl with apache impala by athemaster
•
1 like•336 views
This document summarizes a presentation on implementing an ETL project using Apache Impala on the Hadoop platform. It discusses the components of a Hadoop ETL project, including data storage, sources, ETL processing, metadata, and targets. It then shares details about a customer case that involved replacing an SQL reporting system with Impala to access standardized data for reporting and analytics within required timeframes. The presentation emphasizes that successful Hadoop ETL projects require data engineering methodologies around organization, modularization, logging, and testing approaches.
Report
Share
Report
Share
Download to read offline
Recommended
Not Your Father's Database: How to Use Apache Spark Properly in Your Big Data...
This session will cover a series of problems that are adequately solved with Apache Spark, as well as those that are require additional technologies to implement correctly. Here’s an example outline of some of the topics that will be covered in the talk: Problems that are perfectly solved with Apache Spark: 1) Analyzing a large set of data files. 2) Doing ETL of a large amount of data. 3) Applying Machine Learning & Data Science to a large dataset. 4) Connecting BI/Visualization tools to Apache Spark to analyze large datasets internally.
By Vida Ha at Spark Summit East 2016.
Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark
Spark provides a unified platform for processing big data from multiple sources and in different formats. It allows for just-in-time processing of data without needing to wait for ETL into a data warehouse. This provides lower latency and makes it easy to combine data. Spark also unifies batch, streaming, and machine learning functionality into a single engine. This was demonstrated on a large online service company that leverages Spark for interactive queries, machine learning, and combining data from various sources for analytics and predictive services.
Transitioning from Traditional DW to Apache® Spark™ in Operating Room Predict...
The prevailing issue when working with Operating Room (OR) scheduling within a hospital setting is that it is difficult to schedule and predict available OR block times. This leads to empty and unused operating rooms leading to longer waiting times for patients for their procedures. In this three-part session, Ayad Shammout and Denny will show:
1) How we tried to solve this problem using traditional DW techniques
2) How we took advantage of the DW capabilities in Apache Spark AND easily transition to Spark MLlib so we could more easily predict available OR block times resulting in better OR utilization and shorter wait times for patients.
3) Some of the key learnings we had when migrating from DW to Spark.
Strata NYC 2015 - Supercharging R with Apache Spark
R is the favorite language of many data scientists. In addition to a language and runtime, R is a rich ecosystem of libraries for a wide range of use cases from statistical inference to data visualization. However, handling large or distributed data with R is challenging. Hence R is used along with other frameworks and languages by most data scientist. In this mode most of the friction is at the interface of R and the other systems. For example, when data is sampled by a big data platform, results need to be transferred to and imported in R as native data structures. In this talk we show an alternative, and complimentary, approach to SparkR for integrating Spark and R.
Since SparkR was released in version 1.4 of Apache Spark distributed data remains inside the JVM instead of individual R processes running on workers. This approach is more convenient when dealing with external data sources such as Cassandra, Hive, and Spark’s own distributed DataFrames. We show two specific techniques to remove the data transfer friction between R and JVM: collecting Spark DataFrames as R data frames and user space filesystems. We think this model complements and improves the day-to-day workload of many data scientists who use R. Spark’s interactive query processing, especially with in-memory datasets, closely matches the R interactive session model. When integrated together Spark and R can provide state of the art tools for the entire end-to-end data science pipeline. We will show how such a pipeline works in real world use cases in a live demo at the end of the talk.
SQL Now! How Optiq brings the best of SQL to NoSQL data.
A talk given at the August, 2013 NoSQL Now! conference in San Jose, California.
2013 is the year of that SQL meets Hadoop & NoSQL. There is a surprisingly good fit between thirty-something SQL and the upstart millennial data technologies. SQL helps you deliver faster value, improve integration with visualization and analytics tools, and bring your data to a larger audience.
Julian Hyde, author of Mondrian and Optiq, and a contributor to Apache Drill and Cascading Lingual, shows how to quickly build a SQL interface to a NoSQL system using the Optiq framework.
As a case study, he shows how Optiq has allows the Mondrian in-memory analysis engine (part of Pentaho's business intelligence suite) to leverage the memory and compute power of a cluster and pull data from hybrid SQL and NoSQL sources.
Insights into Customer Behavior from Clickstream Data by Ronald Nowling
This document discusses insights that can be gained from analyzing clickstream data collected from a company's customer knowledgebase and portal. It describes how clickstream data containing 61 million page views and 500,000 pages from 125,000 registered users can be processed and analyzed to build customer profiles, improve search and recommendations, and guide the selection of new knowledgebase topics. Potential applications include using customer profiles to aid sales teams, creating a recommendation system for the knowledgebase, and enhancing the customer portal search.
The Future of Real-Time in Spark
Reynold Xin, an Apache Spark PMC member and Chief Architect for Spark at Databricks, outlines the future of real-time in Apache Spark.
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
Vida Ha presented best practices for storing and working with data in files for optimal Spark performance. Some key tips included choosing appropriate file sizes between 64 MB to 1 GB, using splittable compression formats like gzip and Snappy, enforcing schemas for structured formats like Parquet and Avro, and reusing Hadoop libraries to read various file formats. General tips involved controlling output file size through methods like coalesce and repartition, using sc.wholeTextFiles for non-splittable formats, and processing files individually by filename.
Recommended
Not Your Father's Database: How to Use Apache Spark Properly in Your Big Data...
This session will cover a series of problems that are adequately solved with Apache Spark, as well as those that are require additional technologies to implement correctly. Here’s an example outline of some of the topics that will be covered in the talk: Problems that are perfectly solved with Apache Spark: 1) Analyzing a large set of data files. 2) Doing ETL of a large amount of data. 3) Applying Machine Learning & Data Science to a large dataset. 4) Connecting BI/Visualization tools to Apache Spark to analyze large datasets internally.
By Vida Ha at Spark Summit East 2016.
Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark
Spark provides a unified platform for processing big data from multiple sources and in different formats. It allows for just-in-time processing of data without needing to wait for ETL into a data warehouse. This provides lower latency and makes it easy to combine data. Spark also unifies batch, streaming, and machine learning functionality into a single engine. This was demonstrated on a large online service company that leverages Spark for interactive queries, machine learning, and combining data from various sources for analytics and predictive services.
Transitioning from Traditional DW to Apache® Spark™ in Operating Room Predict...
The prevailing issue when working with Operating Room (OR) scheduling within a hospital setting is that it is difficult to schedule and predict available OR block times. This leads to empty and unused operating rooms leading to longer waiting times for patients for their procedures. In this three-part session, Ayad Shammout and Denny will show:
1) How we tried to solve this problem using traditional DW techniques
2) How we took advantage of the DW capabilities in Apache Spark AND easily transition to Spark MLlib so we could more easily predict available OR block times resulting in better OR utilization and shorter wait times for patients.
3) Some of the key learnings we had when migrating from DW to Spark.
Strata NYC 2015 - Supercharging R with Apache Spark
R is the favorite language of many data scientists. In addition to a language and runtime, R is a rich ecosystem of libraries for a wide range of use cases from statistical inference to data visualization. However, handling large or distributed data with R is challenging. Hence R is used along with other frameworks and languages by most data scientist. In this mode most of the friction is at the interface of R and the other systems. For example, when data is sampled by a big data platform, results need to be transferred to and imported in R as native data structures. In this talk we show an alternative, and complimentary, approach to SparkR for integrating Spark and R.
Since SparkR was released in version 1.4 of Apache Spark distributed data remains inside the JVM instead of individual R processes running on workers. This approach is more convenient when dealing with external data sources such as Cassandra, Hive, and Spark’s own distributed DataFrames. We show two specific techniques to remove the data transfer friction between R and JVM: collecting Spark DataFrames as R data frames and user space filesystems. We think this model complements and improves the day-to-day workload of many data scientists who use R. Spark’s interactive query processing, especially with in-memory datasets, closely matches the R interactive session model. When integrated together Spark and R can provide state of the art tools for the entire end-to-end data science pipeline. We will show how such a pipeline works in real world use cases in a live demo at the end of the talk.
SQL Now! How Optiq brings the best of SQL to NoSQL data.
A talk given at the August, 2013 NoSQL Now! conference in San Jose, California.
2013 is the year of that SQL meets Hadoop & NoSQL. There is a surprisingly good fit between thirty-something SQL and the upstart millennial data technologies. SQL helps you deliver faster value, improve integration with visualization and analytics tools, and bring your data to a larger audience.
Julian Hyde, author of Mondrian and Optiq, and a contributor to Apache Drill and Cascading Lingual, shows how to quickly build a SQL interface to a NoSQL system using the Optiq framework.
As a case study, he shows how Optiq has allows the Mondrian in-memory analysis engine (part of Pentaho's business intelligence suite) to leverage the memory and compute power of a cluster and pull data from hybrid SQL and NoSQL sources.
Insights into Customer Behavior from Clickstream Data by Ronald Nowling
This document discusses insights that can be gained from analyzing clickstream data collected from a company's customer knowledgebase and portal. It describes how clickstream data containing 61 million page views and 500,000 pages from 125,000 registered users can be processed and analyzed to build customer profiles, improve search and recommendations, and guide the selection of new knowledgebase topics. Potential applications include using customer profiles to aid sales teams, creating a recommendation system for the knowledgebase, and enhancing the customer portal search.
The Future of Real-Time in Spark
Reynold Xin, an Apache Spark PMC member and Chief Architect for Spark at Databricks, outlines the future of real-time in Apache Spark.
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
Vida Ha presented best practices for storing and working with data in files for optimal Spark performance. Some key tips included choosing appropriate file sizes between 64 MB to 1 GB, using splittable compression formats like gzip and Snappy, enforcing schemas for structured formats like Parquet and Avro, and reusing Hadoop libraries to read various file formats. General tips involved controlling output file size through methods like coalesce and repartition, using sc.wholeTextFiles for non-splittable formats, and processing files individually by filename.
What's new in pandas and the SciPy stack for financial users
Wes McKinney discusses updates and planned improvements to Python packages for financial analysis, including pandas, NumPy, IPython, Cython, matplotlib, and statsmodels. Major changes include a redesign of pandas' DataFrame internals, hierarchical indexing, time series functionality in statsmodels, and performance optimizations. McKinney aims to make pandas the foundation for rich statistical computing and leverage the best of other languages in Python.
Spark Summit EU 2015: Lessons from 300+ production users
At Databricks, we have a unique view into over a hundred different companies trying out Spark for development and production use-cases, from their support tickets and forum posts. Having seen so many different workflows and applications, some discernible patterns emerge when looking at common performance and scalability issues that our users run into. This talk will discuss some of these common common issues from an engineering and operations perspective, describing solutions and clarifying misconceptions.
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
In this talk we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas.
We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark ML and GraphFrames.
Histogrammar package—a cross-platform suite of data aggregation primitives for making histograms, calculating descriptive statistics and plotting in Scala—is introduced to enable interactive data analysis in Spark REPL.
We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.
Apache Flink - Hadoop MapReduce Compatibility
Flink allows users to run Hadoop MapReduce jobs without changing any code by wrapping Hadoop's APIs. It supports Hadoop data types, file systems, and functions like mappers and reducers. Specifically, Flink can run a WordCount example written using Hadoop APIs without modifications by utilizing Hadoop input/output formats and mapper/reducer functions. Going forward, Flink aims to allow injecting entire MapReduce jobs as a unit into a Flink program while supporting custom Hadoop partitioners and sorters.
pandas: Powerful data analysis tools for Python
Wes McKinney introduced pandas, a Python data analysis library built on NumPy. Pandas provides data structures and tools for cleaning, manipulating, and working with relational and time-series data. Key features include DataFrame for 2D data, hierarchical indexing, merging and joining data, and grouping and aggregating data. Pandas is used heavily in financial applications and has over 1500 unit tests, ensuring stability and reliability. Future goals include better time series handling and integration with other Python data science packages.
Skutil - H2O meets Sklearn - Taylor Smith
Skutil brings the best of both worlds to H2O and sklearn, delivering an easy transition into the world of distributed computing that H2O offers, while providing the same, familiar interface that sklearn users have come to know and love.
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
In Spark SQL’s Catalyst optimizer, many rule based optimization techniques have been implemented, but the optimizer itself can still be improved. For example, without detailed column statistics information on data distribution, it is difficult to accurately estimate the filter factor, cardinality, and thus output size of a database operator. With the inaccurate and/or misleading statistics, it often leads the optimizer to choose suboptimal query execution plans.
We added a Cost-Based Optimizer framework to Spark SQL engine. In our framework, we use Analyze Table SQL statement to collect the detailed column statistics and save them into Spark’s catalog. For the relevant columns, we collect number of distinct values, number of NULL values, maximum/minimum value, average/maximal column length, etc. Also, we save the data distribution of columns in either equal-width or equal-height histograms in order to deal with data skew effectively. Furthermore, with the number of distinct values and number of records of a table, we can determine how unique a column is although Spark SQL does not support primary key. This helps determine, for example, the output size of join operation and multi-column group-by operation.
In our framework, we compute the cardinality and output size of each database operator. With reliable statistics and derived cardinalities, we are able to make good decisions in these areas: selecting the correct build side of a hash-join operation, choosing the right join type (broadcast hash-join versus shuffled hash-join), adjusting multi-way join order, etc. In this talk, we will show Spark SQL’s new Cost-Based Optimizer framework and its performance impact on TPC-DS benchmark queries.
Exceptions are the Norm: Dealing with Bad Actors in ETL
Stable and robust data pipelines are a critical component of the data infrastructure of enterprises. Most commonly, data pipelines ingest messy data sources with incorrect, incomplete or inconsistent records and produce curated and/or summarized data for consumption by subsequent applications.
In this talk, we go over new and upcoming features in Spark that enabled it to better serve such workloads. Such features include isolation of corrupt input records and files, useful diagnostic feedback to users and improved support for nested type handling which is common in ETL jobs.
Speaker: Sameer Agarwal
This talk was originally presented at Spark Summit East 2017.
SQL on Big Data using Optiq
SQL on Big Data is not a "one size fits all". Optiq is a framework that allows you to build a data management system on top of any back-end system, including NoSQL and Hadoop, and rules that optimize query processing for capabilities of the data source. We show how Optiq is used in the Apache Drill and Cascading Lingual projects, and how we plan to combine Optiq materialized views, Mondrian, and a data grid to create next-generation in-memory analytics.
This presentation was given at the Real-Time Big Data meetup at RichRelevance in San Francisco, 2013-04-09.
Use r tutorial part1, introduction to sparkr
This document introduces SparkR, which allows users to perform big data processing from R. It discusses how SparkR enables data exploration using DataFrames for ETL and data cleaning. SparkR also supports advanced analytics like machine learning by integrating R with Spark MLlib. The tutorial outlines exploring data, advanced analytics, and using SparkR DataFrames for tasks like aggregation, filtering, visualization with ggplot, and large-scale machine learning. It recommends signing up for an online Databricks notebook to run SparkR examples and tutorials interactively on a dedicated Spark cluster.
ISAX
ISAX is a time series data compression algorithm that can group similar patterns in billions of time series datasets. It is implemented on H2O's distributed architecture and can be used for clustering, classification, anomaly detection, and predictive analytics on compressed time series data from fields like IoT, finance, bioinformatics, and image/sound processing. Examples of ISAX code in H2O are provided.
Eventually Elasticsearch: Eventual Consistency in the Real World
Based on the experience of an ElasticSearch implementation at bol.com, we'll discuss the consequences of different modes of operation of ElasticSearch in an environment of existing SQL databases. How can you connect ElasticSearch to change queues of other databases, how can the versioning mechanism be used to implement optimistic locking, and what are the consistency consequences of using ElasticSearch as either a free text index on external data, a data cache or as the single source-of-truth system?
Open Source Big Data Ingestion - Without the Heartburn!
Big Data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Upstream data sources can 'drift' due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail, inducing heartburn in even the most resilient data scientist. This session will survey the big data ingestion landscape, focusing on how open source tools such as Sqoop, Flume, Nifi and StreamSets can keep the data pipeline flowing.
Optiq: A dynamic data management framework
Optiq is a framework that allows heterogeneous data from different locations and formats to be queried using a standard SQL interface. It helps optimize queries by caching results, pre-computing aggregates, and using column-oriented storage and materialized views to speed up analytics queries on large datasets. By treating different data sources as a single database and helping "cheat" through query optimization techniques, Optiq provides the benefits of a centralized database for managing and querying hybrid transactional and analytical workloads.
Why you should care about data layout in the file system with Cheng Lian and ...
Efficient data access is one of the key factors for having a high performance data processing pipeline. Determining the layout of data values in the filesystem often has fundamental impacts on the performance of data access. In this talk, we will show insights on how data layout affects the performance of data access. We will first explain how modern columnar file formats like Parquet and ORC work and explain how to use them efficiently to store data values. Then, we will present our best practice on how to store datasets, including guidelines on choosing partitioning columns and deciding how to bucket a table.
Overview of the Hive Stinger Initiative
Dr. Eric N. Hanson, Principal Software Development Engineer at Microsoft and Apache Hive committer presents the recent improvements in Hive
Discardable In-Memory Materialized Queries With Hadoop
What to do with all that memory in a Hadoop cluster? Should we load all of our data into memory to process it?
The goal should be to put memory into its right place in the storage hierarchy, alongside disk and solid-state drives (SSD). Data should reside in the right place for how it is being used, and should be organized appropriately for where it resides. This proposed solution requires a new kind of data set called the Discardable, In-Memory, Materialized Query (DIMMQ).
In this session we will talk through how we can build on existing Hadoop facilities to deliver three key underlying concepts that enable this approach.
Streaming SQL with Apache Calcite
With the rise of the Internet of Things (IoT) and low-latency analytics, streaming data becomes ever more important. Surprisingly, one of the most promising approaches for processing streaming data is SQL. In this presentation, Julian Hyde shows how to build streaming SQL analytics that deliver results with low latency, adapt to network changes, and play nicely with BI tools and stored data. He also describes how Apache Calcite optimizes streaming queries, and the ongoing collaborations between Calcite and the Storm, Flink and Samza projects.
This talk was given Julian Hyde at Apache Big Data conference, Vancouver, on 2016/05/09.
Scalable Data Models with Elasticsearch
At bol.com, a leading ecommerce platform in The Netherlands, we have done extensive research into what it would take to use ElasticSearch as the main search provider. We will explain the specific challenges and requirements of running an Elasticsearch cluster at bol.com-scale, and show how we have used generated data to do performance and scalability tests on different ways to model a hierarchical data model into Elasticsearch. We will describe the benefits and drawbacks of the different data model options, and their consequences for the design of the index and search applications.
Presto updates to 0.178
Presto was updated from version 0.152 to 0.178. New features in the update include lambda expressions, filtered aggregation, a VALIDATE mode for EXPLAIN, compressed exchange, and complex grouping operations. The update also added new functions and deprecated some legacy features with warnings. Future work on Presto includes disk spill optimization and a cost-based optimizer.
Introduction to HCFS
1. HCFS stands for Hadoop Compatible File System. It allows Hadoop to access cloud storage systems like AWS S3, Azure Blob Storage, and Ceph.
2. AWS S3 supports three implementations - s3:, s3n:, and s3a:. S3 cannot replace HDFS due to consistency issues but is commonly used with EMR.
3. Azure Blob Storage uses the wasbs:// scheme and hadoop-azure.jar. It supports multiple accounts and page/block blobs but lacks append and permissions.
4. CephFS can be used with Hadoop but has limited official support to Hadoop 1.1.x due to JNI issues with later versions
Cloudera security and enterprise license by Athemaster(繁中)
以開源專案為基礎的軟體產品，"付費授權功能有哪一些"經常是客戶詢問的主題。這一份簡報中介紹了Cloudera授權的種類，以及付費安全模組的差異。
More Related Content
What's hot
What's new in pandas and the SciPy stack for financial users
Wes McKinney discusses updates and planned improvements to Python packages for financial analysis, including pandas, NumPy, IPython, Cython, matplotlib, and statsmodels. Major changes include a redesign of pandas' DataFrame internals, hierarchical indexing, time series functionality in statsmodels, and performance optimizations. McKinney aims to make pandas the foundation for rich statistical computing and leverage the best of other languages in Python.
Spark Summit EU 2015: Lessons from 300+ production users
At Databricks, we have a unique view into over a hundred different companies trying out Spark for development and production use-cases, from their support tickets and forum posts. Having seen so many different workflows and applications, some discernible patterns emerge when looking at common performance and scalability issues that our users run into. This talk will discuss some of these common common issues from an engineering and operations perspective, describing solutions and clarifying misconceptions.
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
In this talk we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas.
We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark ML and GraphFrames.
Histogrammar package—a cross-platform suite of data aggregation primitives for making histograms, calculating descriptive statistics and plotting in Scala—is introduced to enable interactive data analysis in Spark REPL.
We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.
Apache Flink - Hadoop MapReduce Compatibility
Flink allows users to run Hadoop MapReduce jobs without changing any code by wrapping Hadoop's APIs. It supports Hadoop data types, file systems, and functions like mappers and reducers. Specifically, Flink can run a WordCount example written using Hadoop APIs without modifications by utilizing Hadoop input/output formats and mapper/reducer functions. Going forward, Flink aims to allow injecting entire MapReduce jobs as a unit into a Flink program while supporting custom Hadoop partitioners and sorters.
pandas: Powerful data analysis tools for Python
Wes McKinney introduced pandas, a Python data analysis library built on NumPy. Pandas provides data structures and tools for cleaning, manipulating, and working with relational and time-series data. Key features include DataFrame for 2D data, hierarchical indexing, merging and joining data, and grouping and aggregating data. Pandas is used heavily in financial applications and has over 1500 unit tests, ensuring stability and reliability. Future goals include better time series handling and integration with other Python data science packages.
Skutil - H2O meets Sklearn - Taylor Smith
Skutil brings the best of both worlds to H2O and sklearn, delivering an easy transition into the world of distributed computing that H2O offers, while providing the same, familiar interface that sklearn users have come to know and love.
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
In Spark SQL’s Catalyst optimizer, many rule based optimization techniques have been implemented, but the optimizer itself can still be improved. For example, without detailed column statistics information on data distribution, it is difficult to accurately estimate the filter factor, cardinality, and thus output size of a database operator. With the inaccurate and/or misleading statistics, it often leads the optimizer to choose suboptimal query execution plans.
We added a Cost-Based Optimizer framework to Spark SQL engine. In our framework, we use Analyze Table SQL statement to collect the detailed column statistics and save them into Spark’s catalog. For the relevant columns, we collect number of distinct values, number of NULL values, maximum/minimum value, average/maximal column length, etc. Also, we save the data distribution of columns in either equal-width or equal-height histograms in order to deal with data skew effectively. Furthermore, with the number of distinct values and number of records of a table, we can determine how unique a column is although Spark SQL does not support primary key. This helps determine, for example, the output size of join operation and multi-column group-by operation.
In our framework, we compute the cardinality and output size of each database operator. With reliable statistics and derived cardinalities, we are able to make good decisions in these areas: selecting the correct build side of a hash-join operation, choosing the right join type (broadcast hash-join versus shuffled hash-join), adjusting multi-way join order, etc. In this talk, we will show Spark SQL’s new Cost-Based Optimizer framework and its performance impact on TPC-DS benchmark queries.
Exceptions are the Norm: Dealing with Bad Actors in ETL
Stable and robust data pipelines are a critical component of the data infrastructure of enterprises. Most commonly, data pipelines ingest messy data sources with incorrect, incomplete or inconsistent records and produce curated and/or summarized data for consumption by subsequent applications.
In this talk, we go over new and upcoming features in Spark that enabled it to better serve such workloads. Such features include isolation of corrupt input records and files, useful diagnostic feedback to users and improved support for nested type handling which is common in ETL jobs.
Speaker: Sameer Agarwal
This talk was originally presented at Spark Summit East 2017.
SQL on Big Data using Optiq
SQL on Big Data is not a "one size fits all". Optiq is a framework that allows you to build a data management system on top of any back-end system, including NoSQL and Hadoop, and rules that optimize query processing for capabilities of the data source. We show how Optiq is used in the Apache Drill and Cascading Lingual projects, and how we plan to combine Optiq materialized views, Mondrian, and a data grid to create next-generation in-memory analytics.
This presentation was given at the Real-Time Big Data meetup at RichRelevance in San Francisco, 2013-04-09.
Use r tutorial part1, introduction to sparkr
This document introduces SparkR, which allows users to perform big data processing from R. It discusses how SparkR enables data exploration using DataFrames for ETL and data cleaning. SparkR also supports advanced analytics like machine learning by integrating R with Spark MLlib. The tutorial outlines exploring data, advanced analytics, and using SparkR DataFrames for tasks like aggregation, filtering, visualization with ggplot, and large-scale machine learning. It recommends signing up for an online Databricks notebook to run SparkR examples and tutorials interactively on a dedicated Spark cluster.
ISAX
ISAX is a time series data compression algorithm that can group similar patterns in billions of time series datasets. It is implemented on H2O's distributed architecture and can be used for clustering, classification, anomaly detection, and predictive analytics on compressed time series data from fields like IoT, finance, bioinformatics, and image/sound processing. Examples of ISAX code in H2O are provided.
Eventually Elasticsearch: Eventual Consistency in the Real World
Based on the experience of an ElasticSearch implementation at bol.com, we'll discuss the consequences of different modes of operation of ElasticSearch in an environment of existing SQL databases. How can you connect ElasticSearch to change queues of other databases, how can the versioning mechanism be used to implement optimistic locking, and what are the consistency consequences of using ElasticSearch as either a free text index on external data, a data cache or as the single source-of-truth system?
Open Source Big Data Ingestion - Without the Heartburn!
Big Data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Upstream data sources can 'drift' due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail, inducing heartburn in even the most resilient data scientist. This session will survey the big data ingestion landscape, focusing on how open source tools such as Sqoop, Flume, Nifi and StreamSets can keep the data pipeline flowing.
Optiq: A dynamic data management framework
Optiq is a framework that allows heterogeneous data from different locations and formats to be queried using a standard SQL interface. It helps optimize queries by caching results, pre-computing aggregates, and using column-oriented storage and materialized views to speed up analytics queries on large datasets. By treating different data sources as a single database and helping "cheat" through query optimization techniques, Optiq provides the benefits of a centralized database for managing and querying hybrid transactional and analytical workloads.
Why you should care about data layout in the file system with Cheng Lian and ...
Efficient data access is one of the key factors for having a high performance data processing pipeline. Determining the layout of data values in the filesystem often has fundamental impacts on the performance of data access. In this talk, we will show insights on how data layout affects the performance of data access. We will first explain how modern columnar file formats like Parquet and ORC work and explain how to use them efficiently to store data values. Then, we will present our best practice on how to store datasets, including guidelines on choosing partitioning columns and deciding how to bucket a table.
Overview of the Hive Stinger Initiative
Dr. Eric N. Hanson, Principal Software Development Engineer at Microsoft and Apache Hive committer presents the recent improvements in Hive
Discardable In-Memory Materialized Queries With Hadoop
What to do with all that memory in a Hadoop cluster? Should we load all of our data into memory to process it?
The goal should be to put memory into its right place in the storage hierarchy, alongside disk and solid-state drives (SSD). Data should reside in the right place for how it is being used, and should be organized appropriately for where it resides. This proposed solution requires a new kind of data set called the Discardable, In-Memory, Materialized Query (DIMMQ).
In this session we will talk through how we can build on existing Hadoop facilities to deliver three key underlying concepts that enable this approach.
Streaming SQL with Apache Calcite
With the rise of the Internet of Things (IoT) and low-latency analytics, streaming data becomes ever more important. Surprisingly, one of the most promising approaches for processing streaming data is SQL. In this presentation, Julian Hyde shows how to build streaming SQL analytics that deliver results with low latency, adapt to network changes, and play nicely with BI tools and stored data. He also describes how Apache Calcite optimizes streaming queries, and the ongoing collaborations between Calcite and the Storm, Flink and Samza projects.
This talk was given Julian Hyde at Apache Big Data conference, Vancouver, on 2016/05/09.
Scalable Data Models with Elasticsearch
At bol.com, a leading ecommerce platform in The Netherlands, we have done extensive research into what it would take to use ElasticSearch as the main search provider. We will explain the specific challenges and requirements of running an Elasticsearch cluster at bol.com-scale, and show how we have used generated data to do performance and scalability tests on different ways to model a hierarchical data model into Elasticsearch. We will describe the benefits and drawbacks of the different data model options, and their consequences for the design of the index and search applications.
Presto updates to 0.178
Presto was updated from version 0.152 to 0.178. New features in the update include lambda expressions, filtered aggregation, a VALIDATE mode for EXPLAIN, compressed exchange, and complex grouping operations. The update also added new functions and deprecated some legacy features with warnings. Future work on Presto includes disk spill optimization and a cost-based optimizer.
What's hot (20)
What's new in pandas and the SciPy stack for financial users
What's new in pandas and the SciPy stack for financial users
Spark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production users
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Exceptions are the Norm: Dealing with Bad Actors in ETL
Exceptions are the Norm: Dealing with Bad Actors in ETL
Eventually Elasticsearch: Eventual Consistency in the Real World
Eventually Elasticsearch: Eventual Consistency in the Real World
Open Source Big Data Ingestion - Without the Heartburn!
Open Source Big Data Ingestion - Without the Heartburn!
Why you should care about data layout in the file system with Cheng Lian and ...
Why you should care about data layout in the file system with Cheng Lian and ...
Discardable In-Memory Materialized Queries With Hadoop
Discardable In-Memory Materialized Queries With Hadoop
Viewers also liked
Introduction to HCFS
1. HCFS stands for Hadoop Compatible File System. It allows Hadoop to access cloud storage systems like AWS S3, Azure Blob Storage, and Ceph.
2. AWS S3 supports three implementations - s3:, s3n:, and s3a:. S3 cannot replace HDFS due to consistency issues but is commonly used with EMR.
3. Azure Blob Storage uses the wasbs:// scheme and hadoop-azure.jar. It supports multiple accounts and page/block blobs but lacks append and permissions.
4. CephFS can be used with Hadoop but has limited official support to Hadoop 1.1.x due to JNI issues with later versions
Cloudera security and enterprise license by Athemaster(繁中)
以開源專案為基礎的軟體產品，"付費授權功能有哪一些"經常是客戶詢問的主題。這一份簡報中介紹了Cloudera授權的種類，以及付費安全模組的差異。
EARTH and ME
The document summarizes key facts about Earth. It describes Earth as the third planet from the sun that is unique in harboring life. It then discusses Earth's formation over 4.5 billion years ago and its four major layers: crust, mantle, outer core, and inner core. For each layer, it provides 2-3 brief facts about its composition, thickness, and properties. It also summarizes the major atmospheric layers from the troposphere to the exosphere.
Proyecto
Este documento presenta un proyecto de innovación para mejorar la fluidez y comprensión lectora de estudiantes de segundo grado en una escuela primaria. El proyecto implementará actividades y estrategias metodológicas utilizando textos locales como adivinanzas, canciones y cuentos. Se registrarán los niveles de fluidez, comprensión y velocidad lectora de los estudiantes antes y después de las intervenciones. El objetivo es desarrollar habilidades de lectura y motivación para mejorar el rendimiento ac
Empresa LORÉAL- Meyling Cruzatty
L'Oréal es el grupo líder mundial en belleza, presente en 130 países. Con más de un siglo de experiencia fabricando productos y servicios de belleza, L'Oréal se compromete con el crecimiento responsable y ético y respeta la diversidad. La Fundación L'Oréal busca hacer la ciencia más accesible y promover una belleza generosa para los más necesitados.
Vertex profile Aiptek projector
This document profiles Vertex International Services and its subsidiary Aiptek MEA, which deliver digital products and services in the Middle East and Africa. It summarizes their history and establishes that Vertex was founded in 2013 and Aiptek MEA in 2016 as a division of Global Aiptek Corporation. The document then highlights key pico projector market statistics and describes several of Aiptek's projector products, including the iBeam Block, P800 Boom Box, and i70. Product specifications, applications, pricing, and contact information are provided.
La Filosofia y Metodos
El documento describe diferentes métodos filosóficos, incluyendo la mayéutica de Sócrates, el método deductivo de Descartes, el método inductivo de Bacon, el método fenomenológico de Husserl, el método genético, el método comparativo, el método morfológico de Spengler y el método dialéctico evolutivo de Hegel.
Proposed advert
Nike was founded in 1964 and officially became Nike Inc. in 1971. It started as a distributor for a Japanese shoemaker and was founded by Bill Bowerman and Phil Knight. By 1980, Nike had attained 50% of the US athletic shoe market. Today, Nike sells trainers, sportswear, equipment, and accessories. It also offers fitness apps. Nike's competitors include Adidas, Reebok, Puma, and New Balance. The proposed advert would showcase an athlete performing better in Nike trainers compared to ordinary trainers to demonstrate Nike's ability to improve performance.
MANEJO DE LA INFORMACION
Este documento proporciona información sobre componentes básicos de computadoras e Internet. Explica brevemente lo que son las computadoras y sus partes principales como la tarjeta madre, memoria RAM y ROM, microprocesador, disco duro, etc. También describe orígenes e historia de Internet, modelo cliente-servidor, protocolo HTTP y navegadores y buscadores web.
Kariem ali mohamed
Kariem Ali is seeking a position as a system engineer utilizing his skills and qualifications. He has a bachelor's degree in communication and electronics from Benha University. He has over 5 years of work experience as a control engineer and system engineer. He has strong technical skills in areas such as embedded systems, RF systems, networking, and programming. He is detail-oriented, able to work independently or as part of a team, and capable of problem solving.
Bull City Fit Presentation
Allison Walters interned at Bull City Fit where she developed and taught a gardening and farm to table program. The program involved planting a garden, checking on its growth, teaching the participants about seeds and vegetables, and having the participants engage in activities like taste tests and a recipe contest using vegetables from the garden. The goal was to educate the participants on where food comes from and encourage healthy eating.
Seguridad en la publicación de la información
El documento ofrece nueve indicaciones sobre la seguridad en la publicación de información en Internet. Recomienda hablar con los padres sobre la navegación, respetar las edades mínimas, informarse sobre los sitios web antes de publicar datos personales, no relacionarse con desconocidos, no publicar fotos sin permiso, mantenerse actualizado sobre tecnología y no realizar descargas sin consentimiento de un adulto.
Storage
This document discusses recent storage devices. It defines storage devices as hardware used to store, port, and extract data either permanently or temporarily. There are two main types: primary storage, like RAM, which is small and temporary; and secondary storage, like hard disks, which are larger and permanent. Recent examples highlighted include the Toshiba P300 3TB and WD Blue 1TB internal hard drives, and the Seagate IronWolf 10TB for network-attached storage. External devices discussed are the Buffalo DriveStation DDR, Samsung Portable SSD T3, and G-Technology's BlackBox Plus hard drive. Key specifications like capacity, speed, price and warranty periods are provided for each.
Viewers also liked (16)
Cloudera security and enterprise license by Athemaster(繁中)
Cloudera security and enterprise license by Athemaster(繁中)
Similar to Etl with apache impala by athemaster
Testing Big Data: Automated Testing of Hadoop with QuerySurge
Are You Ready? Stepping Up To The Big Data Challenge In 2016 - Learn why Testing is pivotal to the success of your Big Data Strategy.
According to a new report by analyst firm IDG, 70% of enterprises have either deployed or are planning to deploy big data projects and programs this year due to the increase in the amount of data they need to manage.
The growing variety of new data sources is pushing organizations to look for streamlined ways to manage complexities and get the most out of their data-related investments. The companies that do this correctly are realizing the power of big data for business expansion and growth.
Learn why testing your enterprise's data is pivotal for success with big data and Hadoop. Learn how to increase your testing speed, boost your testing coverage (up to 100%), and improve the level of quality within your data - all with one data testing tool.
Emerging technologies /frameworks in Big Data
A short overview presentation on Emerging technologies /frameworks in Big Data covering Apache Parquet, Apache Flink, Apache Drill with basic concepts of Columnar Storage and Dremel.
Data Science with the Help of Metadata
Data scientists spend too much of their time collecting, cleaning and wrangling data as well as curating and enriching it. Some of this work is inevitable due to the variety of data sources, but there are tools and frameworks that help automate many of these non-creative tasks. A unifying feature of these tools is support for rich metadata for data sets, jobs, and data policies. In this talk, I will introduce state-of-the-art tools for automating data science and I will show how you can use metadata to help automate common tasks in Data Science. I will also introduce a new architecture for extensible, distributed metadata in Hadoop, called Hops (Hadoop Open Platform-as-a-Service), and show how tinker-friendly metadata (for jobs, files, users, and projects) opens up new ways to build smarter applications.
Splice machine-bloor-webinar-data-lakes
Getting Started with
Hadoop Operational Data Lake
Splice
Bloor Groop
Webinar - Presentation Slice
Apache Spark for RDBMS Practitioners: How I Learned to Stop Worrying and Lov...
This talk is about sharing experience and lessons learned on setting up and running the Apache Spark service inside the database group at CERN. It covers the many aspects of this change with examples taken from use cases and projects at the CERN Hadoop, Spark, streaming and database services. The talks is aimed at developers, DBAs, service managers and members of the Spark community who are using and/or investigating “Big Data” solutions deployed alongside relational database processing systems. The talk highlights key aspects of Apache Spark that have fuelled its rapid adoption for CERN use cases and for the data processing community at large, including the fact that it provides easy to use APIs that unify, under one large umbrella, many different types of data processing workloads from ETL, to SQL reporting to ML.
Spark can also easily integrate a large variety of data sources, from file-based formats to relational databases and more. Notably, Spark can easily scale up data pipelines and workloads from laptops to large clusters of commodity hardware or on the cloud. The talk also addresses some key points about the adoption process and learning curve around Apache Spark and the related “Big Data” tools for a community of developers and DBAs at CERN with a background in relational database operations.
Bi on Big Data - Strata 2016 in London
There are many options for business intelligence on Hadoop and other Big Data systems. This presentation provides an overview of some of the options.
Using Databricks as an Analysis Platform
Over the past year, YipitData spearheaded a full migration of its data pipelines to Apache Spark via the Databricks platform. Databricks now empowers its 40+ data analysts to independently create data ingestion systems, manage ETL workflows, and produce meaningful financial research for our clients.
Apache hadoop
LOAD 'tweets.txt' USING PigStorage() AS (id, text, iso_language);
FILTER tweets BY iso_language == 'en';
GROUP filtered_tweets BY iso_language;
DUMP grouped_tweets;
This Pig Latin program loads tweets data from a text file, filters the data to only include tweets with an iso_language of 'en', groups the filtered tweets by iso_language, and dumps the results.
Slide 2 collecting, storing and analyzing big data
This document discusses the process of collecting, storing, processing and analyzing big data. It covers the key concepts and technologies for collecting data using tools like Apache Sqoop and Kafka, storing data using clusters, file systems, NoSQL databases and concepts like sharding and replication. It also discusses processing data using parallel and distributed processing with Hadoop, and analyzing data using Apache Phoenix which provides a SQL interface to query HBase databases.
Powering Interactive Data Analysis at Pinterest by Amazon Redshift
In the last six month, we have set up Amazon Redshift to power our interactive data analysis at Pinterest. It has tremendously improved the speed of analyzing our data.
What's New in Apache Hive 3.0 - Tokyo
The document discusses new features and enhancements in Apache Hive 3.0 including:
1. Improved transactional capabilities with ACID v2 that provide faster performance compared to previous versions while also supporting non-bucketed tables and non-ORC formats.
2. New materialized view functionality that allows queries to be rewritten to improve performance by leveraging pre-computed results stored in materialized views.
3. Enhancements to LLAP workload management that improve query scheduling and enable better sharing of resources across users.
What's New in Apache Hive 3.0?
Apache Hive is a rapidly evolving project, many people are loved by the big data ecosystem. Hive continues to expand support for analytics, reporting, and bilateral queries, and the community is striving to improve support along with many other aspects and use cases. In this lecture, we introduce the latest and greatest features and optimization that appeared in this project last year. This includes benchmarks covering LLAP, Apache Druid's materialized views and integration, workload management, ACID improvements, using Hive in the cloud, and performance improvements. I will also tell you a little about what you can expect in the future.
Demystifying data engineering
Slides for the talk at AI in Production meetup:
https://www.meetup.com/LearnDataScience/events/255723555/
Abstract: Demystifying Data Engineering
With recent progress in the fields of big data analytics and machine learning, Data Engineering is an emerging discipline which is not well-defined and often poorly understood.
In this talk, we aim to explain Data Engineering, its role in Data Science, the difference between a Data Scientist and a Data Engineer, the role of a Data Engineer and common concepts as well as commonly misunderstood ones found in Data Engineering. Toward the end of the talk, we will examine a typical Data Analytics system architecture.
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...
At StampedeCon 2014, Ronald Indeck (VelociData), "Enabling Key Business Advantage from Big Data through Advanced Ingest Processing."
All too often we see critical data dumped into a “Data Lake” causing the data waters to stagnate and become a “Data Swamp”. We have found that many data transformation, quality, and security processes can be addressed a priori on ingest to enhance goodness and improve accessibility to the data. Data can still be stored in raw form if desired but this processing on ingest can unlock operational effectiveness and competitive advantage by integrating fresh and historical data and enable the full potential of the data. We will discuss the underpinnings of stream processing engines, review several relevant business use cases, and discuss future applications.
Meta scale kognitio hadoop webinar
The webinar discusses how organizations can make big data easy to use with the right tools and talent. It presents on MetaScale's expertise in helping Sears Holdings implement Hadoop and how Kognitio's in-memory analytics platform can accelerate Hadoop for organizations. The webinar agenda includes an introduction, a case study on Sears Holdings' Hadoop implementation, an explanation of how Kognitio's platform accelerates Hadoop, and a Q&A session.
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...
http://www.bigdataspain.org/2014/conference/state-of-play-data-science-on-hadoop-in-2015-keynote
Machine Learning is not new. Big Machine Learning is qualitatively different: More data beats algorithm improvement, scale trumps noise and sample size effects, can brute-force manual tasks.
Session presented at Big Data Spain 2014 Conference
18th Nov 2014
Kinépolis Madrid
http://www.bigdataspain.org
Event promoted by: http://www.paradigmatecnologico.com
Slides: https://speakerdeck.com/bigdataspain/state-of-play-data-science-on-hadoop-in-2015-by-sean-owen-at-big-data-spain-2014
Jethro data meetup index base sql on hadoop - oct-2014
JethroData Index based SQL on Hadoop engine.
Architecture comparison of MPP / Full-Scan sql engines such as Impala and Hive to index-based access such as Jethro.
SQL and NoSQL NYC meetup Oct 20 2014
Boaz Raufman
Big data berlin
This document provides an agenda and overview of a talk on big data and data science given by Peter Wang. The key points covered include:
- An honest perspective on big data trends and challenges over time.
- Architecting systems for data exploration and analysis using tools like Continuum Analytics' Blaze and Numba libraries.
- Python's role in data science for its ecosystem of libraries and accessibility to domain experts.
Inroduction to Big Data
This document provides an introduction to big data and related technologies. It defines big data as datasets that are too large to be processed by traditional methods. The motivation for big data is the massive growth in data volume and variety. Technologies like Hadoop and Spark were developed to process this data across clusters of commodity servers. Hadoop uses HDFS for storage and MapReduce for processing. Spark improves on MapReduce with its use of resilient distributed datasets (RDDs) and lazy evaluation. The document outlines several big data use cases and projects involving areas like radio astronomy, particle physics, and engine sensor data. It also discusses when Hadoop and Spark are suitable technologies.
Seagate: Sensor Overload! Taming The Raging Manufacturing Big Data Torrent
Nicholas Berg presented on Seagate's use of big data analytics to manage the large amount of manufacturing data generated from its hard drive production. Seagate collects terabytes of data per day from testing its drives, which it analyzes using Hadoop to improve quality, predict failures, and gain other insights. It faces challenges in integrating this emerging platform due to the rapid evolution of Hadoop and lack of tools to fully leverage large datasets. Seagate is developing its data lake and data science capabilities on Hadoop to better optimize manufacturing and drive design.
Similar to Etl with apache impala by athemaster (20)
Testing Big Data: Automated Testing of Hadoop with QuerySurge
Testing Big Data: Automated Testing of Hadoop with QuerySurge
Apache Spark for RDBMS Practitioners: How I Learned to Stop Worrying and Lov...
Apache Spark for RDBMS Practitioners: How I Learned to Stop Worrying and Lov...
Slide 2 collecting, storing and analyzing big data
Slide 2 collecting, storing and analyzing big data
Powering Interactive Data Analysis at Pinterest by Amazon Redshift
Powering Interactive Data Analysis at Pinterest by Amazon Redshift
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...
Jethro data meetup index base sql on hadoop - oct-2014
Jethro data meetup index base sql on hadoop - oct-2014
Seagate: Sensor Overload! Taming The Raging Manufacturing Big Data Torrent
Seagate: Sensor Overload! Taming The Raging Manufacturing Big Data Torrent
Recently uploaded
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Communications Mining Series - Zero to Hero - Session 1
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
UiPath Test Automation using UiPath Test Suite series, part 5
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
“I’m still / I’m still / Chaining from the Block”
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
Video Streaming: Then, Now, and in the Future
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Climate Impact of Software Testing at Nordic Testing Days
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Securing your Kubernetes cluster_ a step-by-step guide to success !
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Recently uploaded (20)
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Etl with apache impala by athemaster
- 1. WE SEE, WE REFORM. 炬識科技股份有限公司 APACHE IMPALA 在 HADOOP平台實作ETL案例 Chen, Chih Chun2017/01/21
- 2. 專長： 1. 系統架構設計、 2. 系統軟體開發、 3. 系統軟體測試 講師：陳之駿 Chen Chih Chun2 www.athemaster.com
- 3. • 導言 • 實際案例分享及分析 • 方法論 • 總結 大綱3 www.athemaster.com
- 4. A Hadoop ETL Project. • What is a Hadoop ETL Project? • What are the components? • What are the characteristics? • Can those be define by “Equations”? 導言4 www.athemaster.com
- 5. Hadoop = ??? www.athemaster.com 5 Hadoop = A Ecosystem = Lots of Animals (and new one keeps coming … …) = Diversity Platform = Tools fit for a job = Chaos
- 6. ETL = ??? www.athemaster.com 6 ETL = Data X Systems = [Data(src) + Data(target)] X [System(src) + System(target)] = Many many “Pipe-lines” = Humongous Integrations… … = Troubles
- 7. People = ??? www.athemaster.com 7 People = People(persons) = SYS_ADMINS, POWER_USERS, USERS, HACKERS !!?? ... ... = Tons of requirements = Challenges
- 8. Hadoop + ETL + People =??? www.athemaster.com 8 Hadoop + ETL + People = ((Chaos)^Troubles)^Challenges = Lots of Chaos!!! = End of the world !!!! … … What is missing?
- 9. Solution!!! www.athemaster.com 9 Order = Organized logics = Rules for doing things = Methodologies Disciplines = Structures Interfaces = Engineering The Final Equation: (Hadoop + ETL + People)-Order = Success
- 10. Conclusion www.athemaster.com 10 ¨ A Successful Hadoop ETL Project needs Data Engineering. ¨ Data Engineering makes “Right people doing right things to deliver right stuffs at right time”.
- 11. • Customer Main Use Case • Final Customer Requirements • Customer Profile • Deliverable 實際案例分享及分析11 www.athemaster.com
- 12. Customer Main Use Cases www.athemaster.com 12 • Data lake: Store data from many sources for the current report system up to a specific period of time. • Data Processing: Produce standardized data by using BZ logics for analytic tools. • Data Mart: Provide standardized Data by using BZ logics for current report system . • In-time queries: Data must be accessible with in a specific time frame.
- 13. Final Customer Requirements www.athemaster.com 13 • Replace current SQL base reporting system. 1,000+ reports a day. • Minimum Code changes. 80% unchanged. • In-time Queries 3 ~ 5 GB / min 3,000 EPS
- 14. Customer Profile www.athemaster.com 14 • End User Knows • SQL • Java JDBC • New to Hadoop.
- 15. Final Deliverable – The Structure www.athemaster.com 15 The Basic Logical Component Stacks Storage Sources ETL JDBC Metadata Targets
- 16. Deliverable www.athemaster.com 16 Mark I n Data in-take n ETL process n Data Storage n File structure design HDFS Sources Data in-take
- 17. Deliverable www.athemaster.com 17 Mark II n Data Accessing n Basic Reporting function n SQL statements porting n Code migration n Data processing workflow n Meta data design n Schema n Partitioning HDFS Sources Data in-take JDBC Impala (External tables) Report System Data Proc.
- 18. HBase Deliverable www.athemaster.com 18 Mark III n Increasing in Data n Quantity n Complexity n In-time query HDFS Sources Data in-take JDBC Impala (Parquet, internal table) Report System Data Proc. In-time Query
- 19. Final Deliverable www.athemaster.com 19 • Why Impala? • Ease of use • SQL base • JDBC • Pack with Power • Many Built-in transformation functions i.e. String, date, time. • Compatible with Hadoop. • i.e. make data extraction and loading easy. • Reasonable performance • Well Supported; good documentation.
- 20. 天道酬勤, 厚德載物 Discipline and Integrity Methodology20 www.athemaster.com
- 21. Overview www.athemaster.com 21 ¤ Data Engineering Concepts n Abstraction (Interface) n Modularization n Logging
- 22. Overview www.athemaster.com 22 ¤ Data Engineering Methods n POC n Architecture Design n Implement UT n BVT n RPT n SIT