The EDW Ecosystem

•Download as PPTX, PDF•

2 likes•859 views

This document discusses leveraging Hadoop within the existing data warehouse environment of the Department of Immigration and Border Protection (DIBP) in Australia. It provides an overview of DIBP's business and why Hadoop was adopted, describes the existing EDW environment, and discusses the technical implementation of Hadoop. It also outlines next steps such as consolidating the departmental EDW and advanced analytics on Hadoop, and concludes by taking questions.

The EDW Ecosystem
Leveraging Hadoop within an existing Data Warehouse
environment
LUKE KAY
STEVE O’NEILL

Department of Immigration and Border Protection
Question: What is this?
2
Answer: A data lake…
Image Source: Linkedin , 2016

Department of Immigration and Border Protection
1. Overview of DIBP business
2. Why Hadoop in DIBP?
3. Overview of EDW environment
4. Technical implementation of Hadoop
5. Next steps
6. Questions
Agenda
3

Department of Immigration and Border Protection 4
The Department
Each week on average…
• Previously two separate organisations:
• ACBPS (Australian Customs and Border Protection Service)
• DIAC (Department of Immigration and Citizenship)
• DIBP formed on 1st July 2015 (Department Immigration and Border
Protection) and ABF (Australian Border Force)
DIBP Annual Report 2014-2015 ACBPS Annual Report 2014-2015

Department of Immigration and Border Protection 5
Hadoop drivers in DIBP?
Data Archival / ETL Offload
- Significant Legacy ‘application’ debt
- Historical data sources kept for ‘one-off’
querying or Audit purposes
- The right platform for the right workload
Business Requirements
- Load Big Data, but make it easily accessible
- Store it for long periods of time
Analytics
Advanced Analytics (Spark)
- Text mining
- Unstructured Data
New Data types &
Functionality
- ’Free text’ searching (SOLR)
- Key value store (HBASE)

Department of Immigration and Border Protection 6
EDW / Hadoop Strategy
The right data in the right place for the right outcome:
• Teradata EDW for the structured
• Hortonworks Hadoop for the Unstructured
• Teradata Aster for Discovery
All connected over high speed ‘Infiniband network’ for quick data transfer and seamless connectivity.
Our Strategy:
• Walk before you run. Start small and expect rework
• Ability to query and access the two environments
efficiently and effectively (more on this)
• Free text / unstructured searches (SOLR) compared
to full table scans in EDW

Department of Immigration and Border Protection
• Embedded Operationally in the Department
• Get everything / Integrate Data
• As close to the event as possible
• Used everywhere
• Intelligence, Operational , Analytics and Reporting
capabilities
The Enterprise Data Warehouse
7

Department of Immigration and Border Protection 8
EDW ENVIRONMENT

Department of Immigration and Border Protection
Our Requirements
Unstructured Repository
• Tens of millions of BLOBs
Years of backlog / historical data
• New projects
• Multiple source database types
• Fast and convenient retrieval
Logging Repository
• Billions of rows of infrequently accessed logging data
Timeframe
• Now

Department of Immigration and Border Protection
Our Solution
Source HBase
Hive
Avro
Sqoop
Avro
Pig
HiveQL
HDFS
Source
Archive
Landing Staging

Department of Immigration and Border Protection
The Challenges
Learning Curve
• Once the basics are covered, the curve gets steep.
Toolsets
• Vast number of tools
• Behaviour of tools
Robustness
• eg, HBase regions crashing under load
The Business
• Fear of the new

Department of Immigration and Border Protection
Advice and Lessons
Keep it Simple
• Start with your existing skillsets
• Start with a defined need
Expect Rework
• Partly from bugs
• Mostly from experience
Help is out There
• Not always easy to find, but generally good quality
• Be careful of old advice

Department of Immigration and Border Protection
Advice and Lessons
Run Your Own Race
• Don’t be afraid to not like tool X or approach Y
• Don’t feel the need to always jump to the latest and
greatest
Use Your Existing Standards
• eg, our Landing and Staging concept
Enjoy!

Department of Immigration and Border Protection 14
What’s next
- Departmental EDW consolidation (leverage lessons
learned)
- Hadoop 2.4 just complete ( Now we will take new
functionality, bug fixes, security)
- Advanced Analytics work on the Hadoop platform
- Hadoop Roadmap (next slide)

Department of Immigration and Border Protection 15

Department of Immigration and Border Protection 16
Questions

The document discusses modernizing enterprise data warehouses by using a Hadoop data lake solution with EMC Isilon storage. This provides benefits like offloading expensive ETL processing to reduce costs, archiving cold data for cheaper storage, and enabling analytics on new data sources like semi-structured data. The solution leverages Hortonworks Data Platform for open, interoperable analytics and provides enterprise-grade data management capabilities on Hadoop at lower costs than traditional EDWs.

Beyond TCO

DataWorks Summit/Hadoop Summit

This document discusses architecting Hadoop for adoption and data applications. It begins by explaining how traditional systems struggle as data volumes increase and how Hadoop can help address this issue. Potential Hadoop use cases are presented such as file archiving, data analytics, and ETL offloading. Total cost of ownership (TCO) is discussed for each use case. The document then covers important considerations for deploying Hadoop such as hardware selection, team structure, and impact across the organization. Lastly, it discusses lessons learned and the need for self-service tools going forward.

Integrating and Analyzing Data from Multiple Manufacturing Sites using Apache...

DataWorks Summit

In this talk Mark Baker (CSL) will show how CSL Behring is Integrating and Analyzing Data from Multiple Manufacturing Sites using Apache NIFI to a central Hadoop data lake at CSL Behring The challenge of merging data from disparate systems has been a leading driver behind investments in data warehousing systems, as well as, in Hadoop. While data warehousing solutions are ready-built for RDBMS integration, Hadoop adds the benefits of infinite and economical scale – not to mention the variety of structured and non-structured formats that it can handle. Whether using a data warehouse or Hadoop or both, physical data movement and consolidation is the primary method of integration. There may also be challenges with synchronizing rapidly changing data from a system of record to a consolidated Hadoop platform . This introduces the need for “data federation” , where data is integrated without copying data between systems. For historical/batch data use cases there is a replication of data across remote data hubs into a central data lake using Apache NIFI. We will demo using Apache Zeppelin for analyzing data using Apache Spark and Apache HIVE.

Scaling Data Science on Big Data

DataWorks Summit

Data science holds tremendous potential for organizations to uncover new insights and drivers of revenue and profitability. Big Data has brought the promise of doing data science at scale to enterprises, however this promise also comes with challenges for data scientists to continuously learn and collaborate. Data Scientists have many tools at their disposal such as notebooks like Juypter and Apache Zeppelin & IDEs such as RStudio with languages like R, Python, Scala and frameworks like Apache Spark. Given all the choices how do you best collaborate to build your model and then work through the development lifecycle to deploy it from test into production ? In this session learn the attributes of a modern data science platform that empowers data scientists to build models using all the data in their data lake and foster continuous learning and collaboration. We will show a demo of DSX with HDP with the focus on integration, security and model deployment and management. Speakers: Sriram Srinivasan, Senior Technical Staff Member, Analytics Platform Architect, IBM Vikram Murali, Program Director, Data Science and Machine Learning, IBM

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

NoSQLmatters

Come to this deep dive on how Pivotal's Data Lake Vision is evolving by embracing next generation in-memory data exchange and compute technologies around Spark and Tachyon. Did we say Hadoop, SQL, and what's the shortest path to get from past to future state? The next generation of data lake technology will leverage the availability of in-memory processing, with an architecture that supports multiple data analytics workloads within a single environment: SQL, R, Spark, batch and transactional.

Insights into Real World Data Management Challenges

DataWorks Summit

Data is your most valuable business asset and it's also your biggest challenge. This challenge and opportunity means we continually face significant road blocks toward becoming a data driven organisation. From the management of data, to the bubbling open source frameworks, the limited industry skills to surmounting time and cost pressures, our challenge in data is big. We all want and need a “fit for purpose” approach to management of data, especially Big Data, and overcoming the ongoing challenges around the ‘3Vs’ means we get to focus on the most important V - ‘Value’.Come along and join the discussion on how Oracle Big Data Cloud provides Value in the management of data and supports your move toward becoming a data driven organisation. Speaker Noble Raveendran, Principal Consultant, Oracle

Breakout: Hadoop and the Operational Data Store

Cloudera, Inc.

As disparate data volumes continue to be operationalized across the enterprise, data will need to be processed, cleansed, transformed, and made available to end users at greater speeds. Traditional ODS systems run into issues when trying to process large data volumes causing operations to be backed up, data to be archived, and ETL/ ELT processes to fail. Join this breakout to learn how to battle these issues.

Filling the Data Lake

DataWorks Summit/Hadoop Summit

This document discusses strategies for filling a data lake by improving the process of data onboarding. It advocates using a template-based approach to streamline data ingestion from various sources and reduce dependence on hardcoded procedures. The key aspects are managing ELT templates and metadata through automated metadata extraction. This allows generating integration jobs dynamically based on metadata passed at runtime, providing flexibility to handle different source data with one template. It emphasizes reducing the risks associated with large data onboarding projects by maintaining a standardized and organized data lake.

Spark and Hadoop are perfectly together. Spark is a key tool in Hadoop's toolbox that provides elegant developer APIs and accelerates data science and machine learning. It can process streaming data in real-time for applications like web analytics and insurance claims processing. The future of Spark and Hadoop includes innovating the core technologies, providing seamless data access across data platforms, and further accelerating data science tools and libraries.

Data-In-Motion Unleashed

DataWorks Summit

The document discusses Hortonworks DataFlow (HDF), which is a platform for data in motion. HDF allows users to collect data at the edge, route and process streaming data with Apache NiFi and Kafka, and analyze, visualize, predict and prescribe outcomes from the data using HDF platform services. The HDF platform provides scalable stream processing, security, data provenance, and management capabilities for data in motion applications across the enterprise.

Hadoop Journey at Walgreens

DataWorks Summit

This document summarizes Walgreens' journey with Hadoop and building an integrated data hub. It discusses how Walgreens has expanded its Hadoop cluster from 14 nodes in 2014 to 140 nodes currently to handle growing data volumes and variety. An integrated data hub was created to bring analytics to data instead of moving data, improve collaboration, and better manage skills. The hub provides a single platform for batch and streaming analytics across structured, unstructured, hot and cold data. Building the hub required upgrading Hadoop Distribution Platform (HDP) versions, adding new services like HBase, and involving teams early in the process.

Journey to the Data Lake: How Progressive Paved a Faster, Smoother Path to In...

DataWorks Summit

Progressive Insurance is well known for its innovative use of data to better serve its customers, and the important role that Hortonworks Data Platform has played in that transformation. However, as with most things worth doing, the path to the Data Lake was not without its challenges. In this session, I’ll share our top use cases for Hadoop – including telematics and display ads, how a skills shortage turned supporting these applications into a nightmare, and how – and why – we now use Syncsort DMX-h to accelerate enterprise adoption by making it quick and easy (or faster and easier) to populate the data lake – and keep it up to date – with data from across the enterprise. I’ll discuss the different approaches we tried, the benefits of using a tool vs. open source, and how we created our Hadoop Ingestor app using Syncsort DMX-h.

Data Governance - Atlas 7.12.2015

Hortonworks

Effective data governance is imperative to the success of Data Lake initiatives. Without governance policies and processes, information discovery and analysis is severely impaired. In this session we will provide an in-depth look into the Data Governance Initiative launched collaboratively between Hortonworks and partners from across industries. We will cover the objectives of Data Governance Initiatives and demonstrate key governance capabilities of the Hortonworks Data Platform.

Swimming Across the Data Lake, Lessons learned and keys to success

DataWorks Summit/Hadoop Summit

This document discusses strategies for successfully utilizing a data lake. It notes that creating a data lake is just the beginning and that challenges include data governance, metadata management, access, and effective use of the data. The document advocates for data democratization through discovery, accessibility, and usability. It also discusses best practices like self-service BI and automated workload migration from data warehouses to reduce costs and risks. The key is to address the "data lake dilemma" of these challenges to avoid a "data swamp" and slow adoption.

Welcome to Apache Hadoop's Teenage Years, Arun Murthy Keynote

DataWorks Summit/Hadoop Summit

The document discusses the evolution of Apache Hadoop from its origins in 2006 to the present day. It describes how Hadoop has grown from early implementations focused on HDFS, MapReduce, and batch applications to a full ecosystem that enables enterprise interoperability, deployment in data centers and clouds, and assembly of modern data applications. The document also presents several talks that will take place at the conference, focusing on topics like YARN, security, data governance, and running Hadoop in the cloud.

The DAP - Where YARN, HBase, Kafka and Spark go to Production

DataWorks Summit/Hadoop Summit

The document summarizes the Cask Data Application Platform (CDAP), which provides an integrated framework for building and running data applications on Hadoop and Spark. It consolidates the big data application lifecycle by providing dataset abstractions, self-service data, metrics and log collection, lineage, audit, and access control. CDAP has an application container architecture with reusable programming abstractions and global user and machine metadata. It aims to simplify deploying and operating big data applications in enterprises by integrating technologies like YARN, HBase, Kafka and Spark.

Intro to Spark & Zeppelin - Crash Course - HS16SJ

DataWorks Summit/Hadoop Summit

Empowering you with Democratized Data Access, Data Science and Machine Learning

DataWorks Summit

Data science with its specialized tools and knowledge has been a forte of data scientists. However, it is not easy even for data scientists to get access to data that could be in different data stores in the organization. To unleash the power of data and gain valuable insights, machine learning needs to be made easily consumable by various stake holders and access to data made simpler. As an organization's data volumes continue to grow, delivering these insights real time is a complex challenge to solve. This session will provide on overview of an approach to building a scalable solution where machine and deep learning and access to data is made much more consumable and simpler by the fastest SQL on Hadoop engine on the planet, a rich data scientist toolset and an infrastructure that can deliver the responsiveness needed for production environments. Speakers: Pandit Prasad, Program Director, IBM Ashutosh Mate, Global Senior Solutions Architect, IBM

Top Trends in Building Data Lakes for Machine Learning and AI

Holden Ackerman

Presentation by Ashish Thusoo, Co-Founder & CEO at Qubole, on exploring the big data industry trends in moving from data warehouses to cloud-based data lakes.This presentation will cover how companies today are seeing a significant rise in the success of their big data projects by moving to the cloud to iteratively build more cost-effective data pipelines and new products with ML and AI. Uncovering how services like AWS, Google, Oracle, and Microsoft Azure provide the storage and compute infrastructure to build self-service data platforms that can enable all teams and new products to scale iteratively.

High Performance Spatial-Temporal Trajectory Analysis with Spark

DataWorks Summit/Hadoop Summit

This document discusses high performance spatial-temporal trajectory analysis using Spark. It covers the background of analyzing mobile signaling data to enable smarter urban planning. The solution architecture includes data sources, distributed file system, computation engine, and visualization. Technical designs address the big data platform, data governance, algorithm models, and Spark spatial computing. Example scenarios are presented for population heatmaps, commute routes, and office-residence imbalance analysis.

Data lake – On Premise VS Cloud

Idan Tohami

The document discusses Ido Friedman and his background working with various data technologies. It then discusses the concept of a data lake and how it serves as a single store for raw and transformed data used for reporting, analytics, and machine learning. The rest of the document discusses how traditional tools like SQL have changed with the rise of Hadoop and cloud storage. It provides examples of performance and cost differences between running data workloads on Hadoop clusters versus cloud-based data processing services like BigQuery and Dataproc. The document concludes that a large data lake is now possible in the cloud and discusses various deployment options to consider.

Spark in the Enterprise - 2 Years Later by Alan Saldich

Spark Summit

Over the past 2 years, Cloudera has focused on improving and supporting Apache Spark. They have integrated Spark with Hadoop components like YARN, HBase, and Kafka. Cloudera engineers have also contributed security, monitoring, and governance features to Spark. More than 200 customers now use Spark for tasks like ETL, machine learning, and streaming analytics. Customers want Spark to have security comparable to databases, high performance, and simplicity. Cloudera is developing technologies like Sentry and Kudu to meet these needs and make Spark more powerful and useful for enterprises.

Hadoop Hadoop & Spark meetup - Altiscale

Mark Kerzner

Altiscale provides a big data-as-a-service platform based on Apache Hadoop and related technologies like Spark, Hive, and Tez. Interest in big data is growing rapidly but many independent implementations fail. Altiscale aims to help with its experienced team and fully managed platform that offers fast time to value, scalability, security, and lower total cost of ownership. The platform core is Apache open source components like Hadoop, Spark, Hive and Tez. Altiscale handles administration of the Hadoop cluster including hardware, upgrades, tuning, and addressing failures so customers can focus on their data and jobs.

Addressing Enterprise Customer Pain Points with a Data Driven Architecture

DataWorks Summit

Customers that are implementing Big Data Analytics projects in enterprise environments driven by line of business applications are faced with the three critical issues of Managing Complexity, Data Movement and Replication, and Cloud Integration. In this session you will learn about the characteristics of these pain points and how designing and implementing a data driven approach enables enterprises to implement quickly and efficiently with a future proof architecture of hybrid cloud.

Planing and optimizing data lake architecture

Milos Milovanovic

Accelerating Data Warehouse Modernization

DataWorks Summit/Hadoop Summit

Modern data warehouses need to be modernized to handle big data, integrate multiple data silos, reduce costs, and reduce time to market. A modern data warehouse blueprint includes a data lake to land and ingest structured, unstructured, external, social, machine, and streaming data alongside a traditional data warehouse. Key challenges for modernization include making data discoverable and usable for business users, rethinking ETL to allow for data blending, and enabling self-service BI over Hadoop. Common tactics for modernization include using a data lake as a landing zone, offloading infrequently accessed data to Hadoop, and exploring data in Hadoop to discover new insights.

Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)

Rittman Analytics

Oracle Data Integration Platform is a cornerstone for big data solutions that provides five core capabilities: business continuity, data movement, data transformation, data governance, and streaming data handling. It includes eight core products that can operate in the cloud or on-premise, and is considered the most innovative in areas like real-time/streaming integration and extract-load-transform capabilities with big data technologies. The platform offers a comprehensive architecture covering key areas like data ingestion, preparation, streaming integration, parallel connectivity, and governance.

Building a Data Pipeline With Tools From the Hadoop Ecosystem - StampedeCon 2016

StampedeCon

This document discusses building a data pipeline using tools from the Apache Hadoop ecosystem. It begins with an introduction to the speaker and why Hadoop is useful for data pipelines. It then provides a matrix comparing the different Hadoop distributions and their included components. It outlines the various tiers of projects in the Hadoop ecosystem and disclaims any completeness. It also presents the typical data lifecycle of capture, enrichment, analysis, presentation, reporting, archival and removal. The document concludes with a reference to demo code and soliciting questions.

Are you paying attention

Hiba Hamdan

LLAP: Sub-Second Analytical Queries in Hive

DataWorks Summit/Hadoop Summit

The document discusses LLAP (Live Long and Process), a new execution engine in Apache Hive 2.0 that enables sub-second analytical queries. LLAP keeps a small subset of frequently accessed data in memory to enable faster query processing times compared to traditional Hive architectures that rely on disk access. It works by running Hive query fragments simultaneously in both YARN containers and long-running daemon processes that cache data in memory. This allows for highly concurrent query execution without specialized YARN configurations. The document provides details on how LLAP is implemented and evaluates its performance benefits based on benchmarks and customer case studies.

What's hot

Spark and Hadoop Perfect Togeher by Arun Murthy

Spark Summit

Data-In-Motion Unleashed

DataWorks Summit

Hadoop Journey at Walgreens

DataWorks Summit

Journey to the Data Lake: How Progressive Paved a Faster, Smoother Path to In...

DataWorks Summit

Data Governance - Atlas 7.12.2015

Hortonworks

Swimming Across the Data Lake, Lessons learned and keys to success

DataWorks Summit/Hadoop Summit

Welcome to Apache Hadoop's Teenage Years, Arun Murthy Keynote

DataWorks Summit/Hadoop Summit

The DAP - Where YARN, HBase, Kafka and Spark go to Production

DataWorks Summit/Hadoop Summit

Intro to Spark & Zeppelin - Crash Course - HS16SJ

DataWorks Summit/Hadoop Summit

Empowering you with Democratized Data Access, Data Science and Machine Learning

DataWorks Summit

Top Trends in Building Data Lakes for Machine Learning and AI

Holden Ackerman

High Performance Spatial-Temporal Trajectory Analysis with Spark

DataWorks Summit/Hadoop Summit

Data lake – On Premise VS Cloud

Idan Tohami

Spark in the Enterprise - 2 Years Later by Alan Saldich

Spark Summit

Hadoop Hadoop & Spark meetup - Altiscale

Mark Kerzner

Addressing Enterprise Customer Pain Points with a Data Driven Architecture

DataWorks Summit

Planing and optimizing data lake architecture

Milos Milovanovic

Accelerating Data Warehouse Modernization

DataWorks Summit/Hadoop Summit

Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)

Rittman Analytics

Building a Data Pipeline With Tools From the Hadoop Ecosystem - StampedeCon 2016

StampedeCon

What's hot (20)

Spark and Hadoop Perfect Togeher by Arun Murthy

Data-In-Motion Unleashed

Hadoop Journey at Walgreens

Journey to the Data Lake: How Progressive Paved a Faster, Smoother Path to In...

Data Governance - Atlas 7.12.2015

Swimming Across the Data Lake, Lessons learned and keys to success

Welcome to Apache Hadoop's Teenage Years, Arun Murthy Keynote

The DAP - Where YARN, HBase, Kafka and Spark go to Production

Intro to Spark & Zeppelin - Crash Course - HS16SJ

Empowering you with Democratized Data Access, Data Science and Machine Learning

Top Trends in Building Data Lakes for Machine Learning and AI

High Performance Spatial-Temporal Trajectory Analysis with Spark

Data lake – On Premise VS Cloud

Spark in the Enterprise - 2 Years Later by Alan Saldich

Hadoop Hadoop & Spark meetup - Altiscale

Addressing Enterprise Customer Pain Points with a Data Driven Architecture

Planing and optimizing data lake architecture

Accelerating Data Warehouse Modernization

Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)

Building a Data Pipeline With Tools From the Hadoop Ecosystem - StampedeCon 2016

Viewers also liked

Are you paying attention

Hiba Hamdan

LLAP: Sub-Second Analytical Queries in Hive

DataWorks Summit/Hadoop Summit

Big Data at your Desk with KNIME

DataWorks Summit/Hadoop Summit

This document provides an agenda and overview of a presentation by Data Transformed on big data analytics using the KNIME platform. The presentation includes an introduction of Data Transformed and KNIME, a live demonstration of forecasting energy usage from customer data using KNIME and Hadoop tools, and a question and answer session. It promotes Data Transformed's services around data management, analytics, and consulting and highlights KNIME's capabilities for comprehensive data processing.

Sql Stream Intro

Chris Clabaugh

SQLstream provides a relational streaming platform that uses standard SQL to enable real-time analytics on streaming big data. It transforms sensor, system, and service data into real-time intelligence and answers through familiar SQL queries. SQLstream's technology is mature, with granted patents, and helps customers unlock value from real-time data through low-cost analysis and rapid responses to new business requirements. It complements Hadoop by focusing on real-time queries of streaming data.

Scale-Out Resource Management at Microsoft using Apache YARN

DataWorks Summit/Hadoop Summit

This document discusses Microsoft's use of Apache YARN for scale-out resource management. It describes how YARN is used to manage vast amounts of data and compute resources across many different applications and workloads. The document outlines some limitations of YARN and Microsoft's contributions to address those limitations, including Rayon for improved scheduling, Mercury and Yaq for distributed scheduling, and work on federation to scale YARN across multiple clusters. It provides details on the implementation and evaluation of these contributions through papers, JIRAs, and integration into Apache Hadoop releases.

Selective Data Replication with Geographically Distributed Hadoop

DataWorks Summit

This document discusses selective data replication with geographically distributed Hadoop. It describes running Hadoop across multiple data centers as a single cluster. A coordination engine ensures consistent metadata replication and a global sequence of updates. Data is replicated asynchronously over the WAN for fast ingestion. Selective data replication allows restricting replication of some data to specific locations for regulations, temporary data, or ingest-only use cases. Heterogeneous storage zones with different performance profiles can also be used for selective placement. This architecture aims to provide a single unified file system view, strict consistency, continuous availability, and geographic scalability across data centers.

Stream Processing made simple with Kafka

DataWorks Summit/Hadoop Summit

Kafka Streams is a lightweight stream processing library included in Apache Kafka since version 0.10. It provides a simple yet powerful API for building stream processing applications. The API uses a domain-specific language that allows developers to define stream processing topologies where data from Kafka topics acts as input streams and can be transformed before writing the results to output topics. The library handles common stream processing tasks like state management, windowing, and fault tolerance using Kafka's distributed and fault-tolerant architecture.

Enterprise Data Classification and Provenance

DataWorks Summit/Hadoop Summit

Apache Atlas provides data governance capabilities for Hadoop including data classification, metadata management, and data lineage/provenance. It models metadata using a flexible type system and stores metadata in a property graph database for relationships and lineage queries. Key features include cross-component lineage mapping, reusable tagging policies for access control, and a business catalog to organize assets by common business terms.

Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...

DataWorks Summit/Hadoop Summit

The truth about SQL and Data Warehousing on Hadoop

DataWorks Summit/Hadoop Summit

Building Stream Infrastructure across Multiple Data Centers with Apache Kafka

Guozhang Wang

To manage the ever-increasing volume and velocity of data within your company, you have successfully made the transition from single machines and one-off solutions to large distributed stream infrastructures in your data center, powered by Apache Kafka. But what if one data center is not enough? I will describe building resilient data pipelines with Apache Kafka that span multiple data centers and points of presence, and provide an overview of best practices and common patterns while covering key areas such as architecture guidelines, data replication, and mirroring as well as disaster scenarios and failure handling.

Ingest and Stream Processing - What will you choose?

DataWorks Summit/Hadoop Summit

This document discusses streaming data ingestion and processing options. It provides an overview of common streaming architectures including Kafka as an ingestion hub and various streaming engines. Spark Streaming is highlighted as a popular and full-featured option for processing streaming data due to its support for SQL, machine learning, and ease of transition from batch workflows. The document also briefly profiles StreamSets Data Collector as a higher-level tool for building streaming data pipelines.

Apache Ranger Hive Metastore Security

DataWorks Summit/Hadoop Summit

Ranger provides centralized authorization policies for Hadoop resources. A new Ranger Hive Metastore security agent was introduced to address gaps in authorizing Hive CLI and synchronizing policies with object changes. It enforces authorization for Hive CLI, synchronizes access control lists when objects are altered via DDL in HiveServer2, and provides consistent authorization between Hive and HDFS resources. The agent has been implemented through custom classes extending Hive authorization hooks. A demo showed the new capabilities working with Ranger, Hive, and Hadoop.

End-to-End Security and Auditing in a Big Data as a Service Deployment

DataWorks Summit/Hadoop Summit

1. The document discusses security considerations for deploying big data as a service (BDaaS) across multiple tenants and applications. It focuses on maintaining a single user identity to prevent data duplication and enforce access policies consistently. 2. It describes using Apache Ranger to centrally define and enforce policies across Hadoop services like HDFS, HBase, Hive. Ranger integrates with LDAP/AD for authentication. 3. The key challenge is propagating user identities from the application layer to the data layer. This can be done by connecting HDFS directly via Kerberos or using a "super-user" that impersonates other users when accessing HDFS.

Empower Data-Driven Organizations

DataWorks Summit/Hadoop Summit

HPE provides optimized server architectures for Hadoop including the Apollo 4200 server which offers high storage density. HPE also offers a reference architecture for Hadoop that separates compute and storage resources for better performance, using optimized servers like Moonshot for processing and Apollo for storage. Additionally, HPE contributes to Apache Spark through HP Labs to improve efficiency and scale of memory and performance.

Comparison of Transactional Libraries for HBase

DataWorks Summit/Hadoop Summit

File Format Benchmark - Avro, JSON, ORC & Parquet

DataWorks Summit/Hadoop Summit

This document summarizes a benchmark study of file formats for Hadoop, including Avro, JSON, ORC, and Parquet. It found that ORC with zlib compression generally performed best for full table scans. However, Avro with Snappy compression worked better for datasets with many shared strings. The document recommends experimenting with the benchmarks, as performance can vary based on data characteristics and use cases like column projections.

Deep Learning for Fraud Detection

DataWorks Summit/Hadoop Summit

This document summarizes a presentation on deep learning and fraud detection. The presentation explores the state of the art in deep learning and fraud detection, provides guidance on getting results, and includes experiments. The agenda includes discussing motivation for advanced modeling in fraud detection, explaining neural networks and deep learning, and exploring sample fraud detection features and challenges. Examples of applying clustering and autoencoders to time series anomaly detection and card velocity fraud detection are also summarized.

Cloudbreak - Technical Deep Dive

DataWorks Summit/Hadoop Summit

The document provides a technical deep dive into Cloudbreak, an open source tool for provisioning Hadoop clusters in cloud environments. It discusses Cloudbreak's goals of declarative provisioning across cloud providers, its technology stack including Ambari and Salt, lessons learned around different cloud providers' compute, storage and networking capabilities, and best practices for planning Hadoop workloads in cloud environments.

Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...

DataWorks Summit/Hadoop Summit

This document provides an overview of Apache Atlas and how it addresses big data governance issues for enterprises. It discusses how Atlas provides a centralized metadata repository that allows users to understand data across Hadoop components. It also describes how Atlas integrates with Apache Ranger to enable dynamic security policies based on metadata tags. Finally, it outlines new capabilities in upcoming Atlas releases, including cross-component data lineage tracking and a business taxonomy/catalog.

Viewers also liked (20)

Are you paying attention

LLAP: Sub-Second Analytical Queries in Hive

Big Data at your Desk with KNIME

Sql Stream Intro

Scale-Out Resource Management at Microsoft using Apache YARN

Selective Data Replication with Geographically Distributed Hadoop

Stream Processing made simple with Kafka

Enterprise Data Classification and Provenance

Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...

The truth about SQL and Data Warehousing on Hadoop

Building Stream Infrastructure across Multiple Data Centers with Apache Kafka

Ingest and Stream Processing - What will you choose?

Apache Ranger Hive Metastore Security

End-to-End Security and Auditing in a Big Data as a Service Deployment

Empower Data-Driven Organizations

Comparison of Transactional Libraries for HBase

File Format Benchmark - Avro, JSON, ORC & Parquet

Deep Learning for Fraud Detection

Cloudbreak - Technical Deep Dive

Top Three Big Data Governance Issues and How Apache ATLAS resolves it for the...

Similar to The EDW Ecosystem

Searching The Enterprise Data Lake With Solr - Watch Us Do It!: Presented by...

Lucidworks

The document discusses searching enterprise data lakes with Apache Solr. It begins with an overview of how data storage has evolved from single databases to data warehouses to modern data lakes that store vast amounts of raw and processed data. The challenge is finding needed data in this environment. The document then covers the process for indexing data lake contents with Solr, including ingesting data, configuring Solr, parsing and indexing data, searching and analyzing data. It concludes with a demonstration of performing these steps and resources for further information.

Bridging Oracle Database and Hadoop by Alex Gorbachev, Pythian from Oracle Op...

Alex Gorbachev

Modern big data solutions often incorporate Hadoop as one of the components and require the integration of Hadoop with other components including Oracle Database. This presentation explains how Hadoop integrates with Oracle products focusing specifically on the Oracle Database products. Explore various methods and tools available to move data between Oracle Database and Hadoop, how to transparently access data in Hadoop from Oracle Database, and review how other products, such as Oracle Business Intelligence Enterprise Edition and Oracle Data Integrator integrate with Hadoop.

5 Things that Make Hadoop a Game Changer

Caserta

5 Things that Make Hadoop a Game Changer Webinar by Elliott Cordo, Caserta Concepts There is much hype and mystery surrounding Hadoop's role in analytic architecture. In this webinar, Elliott presented, in detail, the services and concepts that makes Hadoop a truly unique solution - a game changer for the enterprise. He talked about the real benefits of a distributed file system, the multi workload processing capabilities enabled by YARN, and the 3 other important things you need to know about Hadoop. To access the recorded webinar, visit the event site: https://www.brighttalk.com/webcast/9061/131029 For more information the services and solutions that Caserta Concepts offers, please visit http://casertaconcepts.com/

Webinar 5-reasons-object-storage.pptx

Cloudian

Are you drowning in unstructured data? Can a private cloud, object based storage system be the solution? Join our live webinar "5 Reasons Object Storage Can Solve Unstructured Data Problems" and learn how Object Storage can: 1. Reduce Data Storage Costs 2. Increase Data Retention Capabilities 3. Improve Data Protection and Security 4. Provide "Cloud" Flexibility 5. Be easy to implement and operate

The modern analytics architecture

Joseph D'Antoni

This document summarizes the history and evolution of data warehousing and analytics architectures. It discusses how data warehouses emerged in the 1970s and were further developed in the late 1980s and 1990s. It then covers how big data and Hadoop have changed architectures, providing more scalability and lower costs. Finally, it outlines components of modern analytics architectures, including Hadoop, data warehouses, analytics engines, and visualization tools that integrate these technologies.

SCAPE - Scalable Preservation Environments

SCAPE Project

Ross King, Project Director of SCAPE, gave a short presentation of the EU funded project SCAPE, including descriptions of tools for planning and monitoring digital preservation, scalable computation and repositories, SCAPE Testbeds and where to learn more. The presentation was given at the workshop ‘Preservation at Scale’ http://bit.ly/17ppAln in connection with the iPres2013 conference in Lissabon, Portugal, in September 2013.

Innovation in the Enterprise Rent-A-Car Data Warehouse

DataWorks Summit

Big Data adoption is a journey. Depending on the business the process can take weeks, months, or even years. With any transformative technology the challenges have less to do with the technology and more to do with how a company adapts itself to a new way of thinking about data. Building a Center of Excellence is one way for IT to help drive success. This talk will explore Enterprise Holdings Inc. (which operates the Enterprise Rent-A-Car, National Car Rental and Alamo Rent A Car) and their experience with Big Data. EHI’s journey started in 2013 with Hadoop as a POC and today are working to create the next generation data warehouse in Microsoft’s Azure cloud utilizing a lambda architecture. We’ll discuss the Center of Excellence, the roles in the new world, share the things which worked well, and rant about those which didn’t. No deep Hadoop knowledge is necessary, architect or executive level.

Use and reuse: research data locally & globally #esipfed

Kevin Ashley

The document discusses the importance of research data reuse and the growing demands by funders for data management and sharing. It notes that properly managing and sharing research data can improve research quality, speed, and cost effectiveness. However, many researchers remain reluctant to share data due to various excuses. The document advocates for national research data infrastructure and services to support universities in meeting funder requirements and overcoming barriers to data sharing.

NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...

National Information Standards Organization (NISO)

Open data is a crucial prerequisite for inventing and disseminating the innovative practices needed for agricultural development. To be usable, data must not just be open in principle—i.e., covered by licenses that allow re-use. Data must also be published in a technical form that allows it to be integrated into a wide range of applications. The webinar will be of interest to any institution seeking ways to publish and curate data in the Linked Data cloud. This webinar describes the technical solutions adopted by a widely diverse global network of agricultural research institutes for publishing research results. The talk focuses on AGRIS, a central and widely-used resource linking agricultural datasets for easy consumption, and AgriDrupal, an adaptation of the popular, open-source content management system Drupal optimized for producing and consuming linked datasets. Agricultural research institutes in developing countries share many of the constraints faced by libraries and other documentation centers, and not just in developing countries: institutions are expected to expose their information on the Web in a re-usable form with shoestring budgets and with technical staff working in local languages and continually lured by higher-paying work in the private sector. Technical solutions must be easy to adopt and freely available.

Real Time Data Warehousing Mastering Business Objects June 11

jeffmonico

Logging at scale: doing more with less

André Fucs de Miranda

The document discusses using Apache NiFi to improve logging capabilities at scale for a Security Operations Center (SOC). It provides an agenda that includes an introduction to the speaker, context on the SOC's logging needs, an overview of Apache NiFi, and a demonstration. The SOC generates terabytes of logging data per day across diverse tools and cloud services. It evaluated tools to better collect, process, enrich, and analyze logs. Apache NiFi was selected for its ability to flexibly route and transform data, integrate with existing systems, query diverse sources, scale horizontally, and provide security and data provenance.

Hadoop Present - Open Enterprise Hadoop

Yifeng Jiang

UniProtKB/Swiss-Prot:Why sparql?

Jerven Bolleman

The Data Lake and Getting Buisnesses the Big Data Insights They Need

Dunn Solutions Group

Do terms like "Data Lake" confuse you? You’re not alone. With all of the technology buzzwords flying around today, it can become a task to keep up with and clearly understand each of them. However a data lake is definitely something to dedicate the time to understand. Leveraging data lake technology, companies are finally able to keep all of their disparate information and streams of data in one secure location ready for consumption at any time – this includes structured, unstructured, and semi-structured data. For more information on our Big Data Consulting Services, don’t hesitate to visit us online at: http://bit.ly/2fvV5rR

Ask bigger questions

South West Data Meetup

Nov 2014 talk to SW Data Meetup by Mike Olson, co-founder and chairman of Cloudera. In business, we often deal with hype around trends in society, politics, economy and technology. We know we need to take claims of the next big thing with a grain of salt and that we should be careful not to set expectations too high. However, with Big Data analytics, the opposite is true. The hype that accompanies it actually conceals the enormity of its impact on the way we do business. In this talk I’ll discuss how new 'Data Driven' economies are emerging through relentless innovation across the public and private sectors. Mike (co-founded Cloudera in 2008 and served as its CEO until 2013 when he took on his current role of chief strategy officer (CSO.) As CSO, Mike is responsible for Cloudera’s product strategy, open source leadership, engineering alignment and direct engagement with customers. Prior to Cloudera Mike was CEO of Sleepycat Software, makers of Berkeley DB, the open source embedded database engine. Mike spent two years at Oracle Corporation as vice president for Embedded Technologies after Oracle’s acquisition of Sleepycat in 2006. Prior to joining Sleepycat, Mike held technical and business positions at database vendors Britton Lee, Illustra Information Technologies and Informix Software. Mike has a Bachelor’s and a Master’s Degree in Computer Science from the University of California, Berkeley.

Case Study: University Alabama-Birmingham.

Red_Hat_Storage

The University of Alabama at Birmingham gives scientists and researchers a massive, on-demand, virtual storage cloud using OpenStack and Ceph for less than $0.41 per gigabyte. This session will detail how the university IT staff deployed a private storage cloud infrastructure using the Dell OpenStack cloud solution with Dell servers, storage, networking and OpenStack, and Inktank Ceph. After assessing a number of traditional storage scenarios, the University partnered with Dell and Inktank to architect a centralized cloud storage platform that was capable of scaling seamlessly and rapidly, was cost-effective, and that could leverage a single hardware infrastructure for the OpenStack compute and storage environment. Vide presentation: http://bit.ly/1o85YKV

OpenStack and Ceph case study at the University of Alabama

Kamesh Pemmaraju

The University of Alabama at Birmingham gives scientists and researchers a massive, on-demand, virtual storage cloud using OpenStack and Ceph for less than $0.41 per gigabyte. This is a session at the OpenStack summit given by Kamesh Pemmaraju at Dell and John Paul at University of Alabama. This will detail how the university IT staff deployed a private storage cloud infrastructure using the Dell OpenStack cloud solution with Dell servers, storage, networking and OpenStack, and Inktank Ceph. After assessing a number of traditional storage scenarios, the University partnered with Dell and Inktank to architect a centralized cloud storage platform that was capable of scaling seamlessly and rapidly, was cost-effective, and that could leverage a single hardware infrastructure for the OpenStack compute and storage environment.

DELWP’s Data Lake: Investing in Asset Wealth for Public/Community Benefit – B...

Amazon Web Services

In the last 10 years, Department of Environment, Land, Water and Planning (DELWP) has procured half a petabyte of aerial photography, satellite imagery, and point cloud data worth more than $45million. In addition, a range of faster products critical to the delivery of services are produced continuously. DELWP’s faster data needs is expected to grow overtime and an enterprise system to manage, publish and enable discovery and analysis of this growing catalogue is essential. Find out how DELWP is implementing pioneering data science strategies to successfully integrate disparate systems and data silos, reduce risks and increase compliance, and support a rapid pace of innovation to improve DELWP’s services to the Victorian public. Project Manager with DELWP’s Digital First program, Alena Moison will share the department’s journey through digital transformation, key successes and learning points, as well as how they worked with Bulletproof to implement solutions on AWS. Speakers: Giles Bill, Bulletproof Networks Sam Mason, Bulletproof Networks Alena Moison, Department of Environment, Land, Water and Planning

HPCC Systems Engineering Summit: Community Use Case: Because Who Has Time for...

HPCC Systems

Presenter: John Andleman, Staff Database Engineer, Citrix In this session, John will share some interesting use cases leveraging the HPCC Systems platform, including those beyond traditional big data uses. John will also share his roadmap of HPCC projects being planned for the next few months and why he feels HPCC Systems is a more suitable solution than Hadoop based on experiences and lessons learned. NOTE: The video of this presentation is the 3rd one shown in the accompanying YouTube link.

Data-intensive bioinformatics on HPC and Cloud

Ola Spjuth

The document discusses data-intensive bioinformatics and challenges with analyzing large genomic datasets on high performance computing (HPC) resources. It summarizes that storage is the biggest challenge as sequencing projects generate very large amounts of data and users do not clean up data. The strategies discussed to address this include assessing costs of storage and analysis upfront, limiting project lifetimes, moving to tiered storage, and improving efficiency. It also discusses using cloud computing resources through virtual clusters and containers to enable flexible, on-demand access and pay-per-use pricing models. Scientific workflows and microservices approaches are presented as ways to automate and orchestrate large-scale genomic analyses on distributed computing resources.

Similar to The EDW Ecosystem (20)

Searching The Enterprise Data Lake With Solr - Watch Us Do It!: Presented by...

Bridging Oracle Database and Hadoop by Alex Gorbachev, Pythian from Oracle Op...

5 Things that Make Hadoop a Game Changer

Webinar 5-reasons-object-storage.pptx

The modern analytics architecture

SCAPE - Scalable Preservation Environments

Innovation in the Enterprise Rent-A-Car Data Warehouse

Use and reuse: research data locally & globally #esipfed

NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...

Real Time Data Warehousing Mastering Business Objects June 11

Logging at scale: doing more with less

Hadoop Present - Open Enterprise Hadoop

UniProtKB/Swiss-Prot:Why sparql?

The Data Lake and Getting Buisnesses the Big Data Insights They Need

Ask bigger questions

Case Study: University Alabama-Birmingham.

OpenStack and Ceph case study at the University of Alabama

DELWP’s Data Lake: Investing in Asset Wealth for Public/Community Benefit – B...

HPCC Systems Engineering Summit: Community Use Case: Because Who Has Time for...

Data-intensive bioinformatics on HPC and Cloud

More from DataWorks Summit/Hadoop Summit

Running Apache Spark & Apache Zeppelin in Production

DataWorks Summit/Hadoop Summit

This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.

State of Security: Apache Spark & Apache Zeppelin

DataWorks Summit/Hadoop Summit

Unleashing the Power of Apache Atlas with Apache Ranger

DataWorks Summit/Hadoop Summit

The document discusses the Virtual Data Connector project which aims to leverage Apache Atlas and Apache Ranger to provide unified metadata and access governance across data sources. Key points include: - The project aims to address challenges of understanding, governing, and controlling access to distributed data through a centralized metadata catalog and policies. - Apache Atlas provides a scalable metadata repository while Apache Ranger enables centralized access governance. The project will integrate these using a virtualization layer. - Enhancements to Atlas and Ranger are proposed to better support the project's goals around a unified open metadata platform and metadata-driven governance. - An initial minimum viable product will be built this year with the goal of an open, collaborative ecosystem around shared

Enabling Digital Diagnostics with a Data Science Platform

DataWorks Summit/Hadoop Summit

This document discusses using a data science platform to enable digital diagnostics in healthcare. It provides an overview of healthcare data sources and Yale/YNHH's data science platform. It then describes the data science journey process using a clinical laboratory use case as an example. The goal is to use big data and machine learning to improve diagnostic reproducibility, throughput, turnaround time, and accuracy for laboratory testing by developing a machine learning algorithm and real-time data processing pipeline.

Revolutionize Text Mining with Spark and Zeppelin

DataWorks Summit/Hadoop Summit

This document discusses using Apache Spark and MLlib for text mining on big data. It outlines common text mining applications, describes how Spark and MLlib enable scalable machine learning on large datasets, and provides examples of text mining workflows and pipelines that can be built with Spark MLlib algorithms and components like tokenization, feature extraction, and modeling. It also discusses customizing ML pipelines and the Zeppelin notebook platform for collaborative data science work.

Double Your Hadoop Performance with Hortonworks SmartSense

DataWorks Summit/Hadoop Summit

This document compares the performance of Hive and Spark when running the BigBench benchmark. It outlines the structure and use cases of the BigBench benchmark, which aims to cover common Big Data analytical properties. It then describes sequential performance tests of Hive+Tez and Spark on queries from the benchmark using a HDInsight PaaS cluster, finding variations in performance between the systems. Concurrency tests are also run by executing multiple query streams in parallel to analyze throughput.

Hadoop Crash Course

DataWorks Summit/Hadoop Summit

The document discusses modern data applications and architectures. It introduces Apache Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. Hadoop provides massive scalability and easy data access for applications. The document outlines the key components of Hadoop, including its distributed storage, processing framework, and ecosystem of tools for data access, management, analytics and more. It argues that Hadoop enables organizations to innovate with all types and sources of data at lower costs.

Data Science Crash Course

DataWorks Summit/Hadoop Summit

This document provides an overview of data science and machine learning. It discusses what data science and machine learning are, including extracting insights from data and computers learning without being explicitly programmed. It also covers Apache Spark, which is an open source framework for large-scale data processing. Finally, it discusses common machine learning algorithms like regression, classification, clustering, and dimensionality reduction.

Apache Spark Crash Course

DataWorks Summit/Hadoop Summit

This document provides an overview of Apache Spark, including its capabilities and components. Spark is an open-source cluster computing framework that allows distributed processing of large datasets across clusters of machines. It supports various data processing workloads including streaming, SQL, machine learning and graph analytics. The document discusses Spark's APIs like DataFrames and its libraries like Spark SQL, Spark Streaming, MLlib and GraphX. It also provides examples of using Spark for tasks like linear regression modeling.

Dataflow with Apache NiFi

DataWorks Summit/Hadoop Summit

This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.

Schema Registry - Set you Data Free

DataWorks Summit/Hadoop Summit

Many Organizations are currently processing various types of data and in different formats. Most often this data will be in free form, As the consumers of this data growing it’s imperative that this free-flowing data needs to adhere to a schema. It will help data consumers to have an expectation of about the type of data they are getting and also they will be able to avoid immediate impact if the upstream source changes its format. Having a uniform schema representation also gives the Data Pipeline a really easy way to integrate and support various systems that use different data formats. SchemaRegistry is a central repository for storing, evolving schemas. It provides an API & tooling to help developers and users to register a schema and consume that schema without having any impact if the schema changed. Users can tag different schemas and versions, register for notifications of schema changes with versions etc. In this talk, we will go through the need for a schema registry and schema evolution and showcase the integration with Apache NiFi, Apache Kafka, Apache Storm.

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...

DataWorks Summit/Hadoop Summit

There is increasing need for large-scale recommendation systems. Typical solutions rely on periodically retrained batch algorithms, but for massive amounts of data, training a new model could take hours. This is a problem when the model needs to be more up-to-date. For example, when recommending TV programs while they are being transmitted the model should take into consideration users who watch a program at that time. The promise of online recommendation systems is fast adaptation to changes, but methods of online machine learning from streams is commonly believed to be more restricted and hence less accurate than batch trained models. Combining batch and online learning could lead to a quickly adapting recommendation system with increased accuracy. However, designing a scalable data system for uniting batch and online recommendation algorithms is a challenging task. In this talk we present our experiences in creating such a recommendation engine with Apache Flink and Apache Spark.

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...

DataWorks Summit/Hadoop Summit

DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.

Mool - Automated Log Analysis using Data Science and ML

DataWorks Summit/Hadoop Summit

QE automation for large systems is a great step forward in increasing system reliability. In the big-data world, multiple components have to come together to provide end-users with business outcomes. This means, that QE Automations scenarios need to be detailed around actual use cases, cross-cutting components. The system tests potentially generate large amounts of data on a recurring basis, verifying which is a tedious job. Given the multiple levels of indirection, the false positives of actual defects are higher, and are generally wasteful. At Hortonworks, we’ve designed and implemented Automated Log Analysis System - Mool, using Statistical Data Science and ML. Currently the work in progress has a batch data pipeline with a following ensemble ML pipeline which feeds into the recommendation engine. The system identifies the root cause of test failures, by correlating the failing test cases, with current and historical error records, to identify root cause of errors across multiple components. The system works in unsupervised mode with no perfect model/stable builds/source-code version to refer to. In addition the system provides limited recommendations to file/open past tickets and compares run-profiles with past runs.

How Hadoop Makes the Natixis Pack More Efficient

DataWorks Summit/Hadoop Summit

Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together. This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear: • How and why the business and IT requirements originated • How we leverage the platform to fulfill security and production requirements • How we organize a community to: o Guard all the players, no one gets left on the ground! o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead) • What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match! DETAILS This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.

HBase in Practice

DataWorks Summit/Hadoop Summit

HBase is a distributed, column-oriented database that stores data in tables divided into rows and columns. It is optimized for random, real-time read/write access to big data. The document discusses HBase's key concepts like tables, regions, and column families. It also covers performance tuning aspects like cluster configuration, compaction strategies, and intelligent key design to spread load evenly. Different use cases are suitable for HBase depending on access patterns, such as time series data, messages, or serving random lookups and short scans from large datasets. Proper data modeling and tuning are necessary to maximize HBase's performance.

The Challenge of Driving Business Value from the Analytics of Things (AOT)

DataWorks Summit/Hadoop Summit

There has been an explosion of data digitising our physical world – from cameras, environmental sensors and embedded devices, right down to the phones in our pockets. Which means that, now, companies have new ways to transform their businesses – both operationally, and through their products and services – by leveraging this data and applying fresh analytical techniques to make sense of it. But are they ready? The answer is “no” in most cases. In this session, we’ll be discussing the challenges facing companies trying to embrace the Analytics of Things, and how Teradata has helped customers work through and turn those challenges to their advantage.

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

DataWorks Summit/Hadoop Summit

In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...

DataWorks Summit/Hadoop Summit

In high-risk manufacturing industries, regulatory bodies stipulate continuous monitoring and documentation of critical product attributes and process parameters. On the other hand, sensor data coming from production processes can be used to gain deeper insights into optimization potentials. By establishing a central production data lake based on Hadoop and using Talend Data Fabric as a basis for a unified architecture, the German pharmaceutical company HERMES Arzneimittel was able to cater to compliance requirements as well as unlock new business opportunities, enabling use cases like predictive maintenance, predictive quality assurance or open world analytics. Learn how the Talend Data Fabric enabled HERMES Arzneimittel to become data-driven and transform Big Data projects from challenging, hard to maintain hand-coding jobs to repeatable, future-proof integration designs. Talend Data Fabric combines Talend products into a common set of powerful, easy-to-use tools for any integration style: real-time or batch, big data or master data management, on-premises or in the cloud.

Backup and Disaster Recovery in Hadoop

DataWorks Summit/Hadoop Summit

While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.

More from DataWorks Summit/Hadoop Summit (20)

Running Apache Spark & Apache Zeppelin in Production

State of Security: Apache Spark & Apache Zeppelin

Unleashing the Power of Apache Atlas with Apache Ranger

Enabling Digital Diagnostics with a Data Science Platform

Revolutionize Text Mining with Spark and Zeppelin

Double Your Hadoop Performance with Hortonworks SmartSense

Hadoop Crash Course

Data Science Crash Course

Apache Spark Crash Course

Dataflow with Apache NiFi

Schema Registry - Set you Data Free

Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...

Mool - Automated Log Analysis using Data Science and ML

How Hadoop Makes the Natixis Pack More Efficient

HBase in Practice

The Challenge of Driving Business Value from the Analytics of Things (AOT)

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

From Regulatory Process Verification to Predictive Maintenance and Beyond wit...

Backup and Disaster Recovery in Hadoop

Recently uploaded

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/ DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen! Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell. Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten. Diese Themen werden behandelt - Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten - Wie funktionieren CCB- und CCX-Lizenzen wirklich? - Verstehen des DLAU-Tools und wie man es am besten nutzt - Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw. - Praxisbeispiele und Best Practices zum sofortigen Umsetzen

みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...

名前です男

AI 101: An Introduction to the Basics and Impact of Artificial Intelligence

IndexBug

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Uni Systems S.M.S.A.

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

Mind map of terminologies used in context of Generative AI

Kumud Singh

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

shyamraj55

Artificial Intelligence for XMLDevelopment

Octavian Nadolu

In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject. We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup. Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved. The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring. The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise. By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

GenAI Pilot Implementation in the organizations

kumardaparthi1024

OpenID AuthZEN Interop Read Out - Authorization

David Brossard

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Columbus Data & Analytics Wednesdays - June 2024

Jason Packer

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

Malak Abu Hammad

Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers: * What is Vector Search? * Importance and benefits of vector search * Practical use cases across various industries * Step-by-step implementation guide * Live demos with code snippets * Enhancing LLM capabilities with vector search * Best practices and optimization strategies Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications. #MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology

20240609 QFM020 Irresponsible AI Reading List May 2024

Matthew Sinclair

Best 20 SEO Techniques To Improve Website Visibility In SERP

Pixlogix Infotech

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

June Patch Tuesday

Ivanti

Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

akankshawande

Recently uploaded (20)

Climate Impact of Software Testing at Nordic Testing Days

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU

AI 101: An Introduction to the Basics and Impact of Artificial Intelligence

Uni Systems Copilot event_05062024_C.Vlachos.pdf

20240605 QFM017 Machine Intelligence Reading List May 2024

Mind map of terminologies used in context of Generative AI

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

Artificial Intelligence for XMLDevelopment

Presentation of the OECD Artificial Intelligence Review of Germany

GenAI Pilot Implementation in the organizations

OpenID AuthZEN Interop Read Out - Authorization

Essentials of Automations: The Art of Triggers and Actions in FME

Columbus Data & Analytics Wednesdays - June 2024

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

20240609 QFM020 Irresponsible AI Reading List May 2024

Best 20 SEO Techniques To Improve Website Visibility In SERP

Removing Uninteresting Bytes in Software Fuzzing

June Patch Tuesday

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

The EDW Ecosystem

1. The EDW Ecosystem Leveraging Hadoop within an existing Data Warehouse environment LUKE KAY STEVE O’NEILL

2. Department of Immigration and Border Protection Question: What is this? 2 Answer: A data lake… Image Source: Linkedin , 2016

3. Department of Immigration and Border Protection 1. Overview of DIBP business 2. Why Hadoop in DIBP? 3. Overview of EDW environment 4. Technical implementation of Hadoop 5. Next steps 6. Questions Agenda 3

4. Department of Immigration and Border Protection 4 The Department Each week on average… • Previously two separate organisations: • ACBPS (Australian Customs and Border Protection Service) • DIAC (Department of Immigration and Citizenship) • DIBP formed on 1st July 2015 (Department Immigration and Border Protection) and ABF (Australian Border Force) DIBP Annual Report 2014-2015 ACBPS Annual Report 2014-2015

5. Department of Immigration and Border Protection 5 Hadoop drivers in DIBP? Data Archival / ETL Offload - Significant Legacy ‘application’ debt - Historical data sources kept for ‘one-off’ querying or Audit purposes - The right platform for the right workload Business Requirements - Load Big Data, but make it easily accessible - Store it for long periods of time Analytics Advanced Analytics (Spark) - Text mining - Unstructured Data New Data types & Functionality - ’Free text’ searching (SOLR) - Key value store (HBASE)

6. Department of Immigration and Border Protection 6 EDW / Hadoop Strategy The right data in the right place for the right outcome: • Teradata EDW for the structured • Hortonworks Hadoop for the Unstructured • Teradata Aster for Discovery All connected over high speed ‘Infiniband network’ for quick data transfer and seamless connectivity. Our Strategy: • Walk before you run. Start small and expect rework • Ability to query and access the two environments efficiently and effectively (more on this) • Free text / unstructured searches (SOLR) compared to full table scans in EDW

7. Department of Immigration and Border Protection • Embedded Operationally in the Department • Get everything / Integrate Data • As close to the event as possible • Used everywhere • Intelligence, Operational , Analytics and Reporting capabilities The Enterprise Data Warehouse 7

8. Department of Immigration and Border Protection 8 EDW ENVIRONMENT

9. Department of Immigration and Border Protection Our Requirements Unstructured Repository • Tens of millions of BLOBs Years of backlog / historical data • New projects • Multiple source database types • Fast and convenient retrieval Logging Repository • Billions of rows of infrequently accessed logging data Timeframe • Now

10. Department of Immigration and Border Protection Our Solution Source HBase Hive Avro Sqoop Avro Pig HiveQL HDFS Source Archive Landing Staging

11. Department of Immigration and Border Protection The Challenges Learning Curve • Once the basics are covered, the curve gets steep. Toolsets • Vast number of tools • Behaviour of tools Robustness • eg, HBase regions crashing under load The Business • Fear of the new

12. Department of Immigration and Border Protection Advice and Lessons Keep it Simple • Start with your existing skillsets • Start with a defined need Expect Rework • Partly from bugs • Mostly from experience Help is out There • Not always easy to find, but generally good quality • Be careful of old advice

13. Department of Immigration and Border Protection Advice and Lessons Run Your Own Race • Don’t be afraid to not like tool X or approach Y • Don’t feel the need to always jump to the latest and greatest Use Your Existing Standards • eg, our Landing and Staging concept Enjoy!

14. Department of Immigration and Border Protection 14 What’s next - Departmental EDW consolidation (leverage lessons learned) - Hadoop 2.4 just complete ( Now we will take new functionality, bug fixes, security) - Advanced Analytics work on the Hadoop platform - Hadoop Roadmap (next slide)

15. Department of Immigration and Border Protection 15

16. Department of Immigration and Border Protection 16 Questions

Editor's Notes

What will happen in the next 12 months: 37 million people crossing our border 5 million of those people will have to go through a process of some decision as to whether they are entitled to come here 150 thousand of those people will seek to join us as Australian citizens 2 million sea cargo containers will come into our ports 6.5 billion air cargo consignments will cross our border, to increase by 54% over the next couple of years 200 million mail items will come across our border Passenger volume is going to increase by 8 million passengers over the next 3 or 4 years 7.5 million Visas granted
We realised we needed to change the way we previously worked The right data in the right place for the right outcome, that is: Teradata EDW for the structured mission critical Guaranteed response times Highly secured Hortonworks Hadoop for the Unstructured Quick storage / retrieval Cheaper storage Teradata Aster for Discovery All connected over high speed ‘Infiniband network’ for quick data transfer and seamless connectivity Our Strategy: Structured / Transactional data in the traditional EDW (Teradata) Unstructured Data in the Unstructured environment (Hadoop) Walk before you run. Start small and expect rework. Ability to query and access the two environments efficiently and effectively (more on this) Free text / unstructured searches (SOLR) compared to full table scans in EDW
Embedded Operationally in the DepartmentThe Enterprise Data Warehouse is used within all business lines of he Department. Used 24/7 in operational areas (Australian Border Force) . Get everything / Integrate DataA full history or copy of source system (Staging ) is kept in the EDW allowing atomic level data analysis (our source systems DON’T keep history) so the EDW is the central / integrated point of contact). The data is then modelled / integrated – joining data together to achieve insights that aren’t possible in the source systems As close to the event as possibleData feeds and ETLs are near-real time Used everywhereApproximately 3,000 Users (internal and external) running reports / using applications 24x7 / and custom SQL queries Intelligence, Operational , Analytics and Reporting capabilitiesReal time interfaces with operational systems. Risk and Analytic models are run in the EDW environment.

The EDW Ecosystem

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to The EDW Ecosystem

Similar to The EDW Ecosystem (20)

More from DataWorks Summit/Hadoop Summit

More from DataWorks Summit/Hadoop Summit (20)

Recently uploaded

Recently uploaded (20)

The EDW Ecosystem

Editor's Notes