Big data hadoop flume spark cloudera Oracle big data appliance apache , oracle loader for hadoop, Big data copy. Exadata to Big data appliance. bilginc It academy.
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...Spark Summit
One of the key challenges in working with real-time and streaming data is that the data format for capturing data is not necessarily the optimal format for ad hoc analytic queries. For example, Avro is a convenient and popular serialization service that is great for initially bringing data into HDFS. Avro has native integration with Flume and other tools that make it a good choice for landing data in Hadoop. But columnar file formats, such as Parquet and ORC, are much better optimized for ad hoc queries that aggregate over large number of similar rows.
A Non-Standard use Case of Hadoop: High Scale Image Processing and AnalyticsDataWorks Summit
1. The Hadoop Image Processing (HIP) pipeline acquires vehicle images, identifies updates, generates URLs, crops and resizes images, copies them to asset servers, and removes duplicates.
2. It uses HBase for image storage and archiving, MapReduce for image processing, Kafka for publishing to asset servers, OpenCV for image processing, and Avro for data serialization.
3. Performance testing showed HIP scales linearly and is at least 10x faster than the previous system, and using cascading downloads provided a 20% performance gain.
Big Data Anti-Patterns: Lessons From the Front LIneDouglas Moore
This document summarizes common anti-patterns in big data projects based on lessons learned from working with over 50 clients. It identifies anti-patterns in hardware and infrastructure, tooling, and big data warehousing. Specifically, it discusses issues with referencing outdated architectures, using tools improperly for the workload, and de-normalizing schemas without understanding the implications. The document provides recommendations to instead co-locate data and computing, choose the right tools for each job, and deploy solutions matching the intended workload.
Hadoop summit 2010 frameworks panel elephant birdKevin Weil
Elephant Bird is a framework for working with structured data within Hadoop ecosystems. It allows users to specify a flexible, forward-backward compatible, self-documenting data schema and then generates code for input/output formats, Hadoop Writables, and Pig load/store functions. This reduces the amount of code needed and allows users to focus on their data. Elephant Bird underlies 20,000 Hadoop jobs per day at Twitter.
Accelerating analytics in a new era of dataArnon Shimoni
Organizations today produce exponentially more data than they did just a few years ago, but their databases weren’t built to handle these new volumes. As a result, reporting takes way too long, and some complex analytics simply cannot be done. The Era of Massive Data is upon us, and a new approach is required to overcome the limitations of traditional CPU-based data stores.
KEY TAKEAWAYS
- Flexible data exploration with minimal preparation
- Unrestricted access to your organization’s full scope of data
- Access to previously unobtainable insights, for smarter business decisions
HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...Cloudera, Inc.
Apache Drill is an interactive SQL query engine for analyzing large scale datasets. It allows for querying data stored in HBase and other data sources. Drill uses an optimistic execution model and late binding to schemas to enable fast queries without requiring metadata definitions. It leverages recent techniques like vectorized operators and late record materialization to improve performance. The project is currently in alpha stage but aims to support features like nested queries, Hive UDFs, and optimized joins with HBase.
SQream DB - Bigger Data On GPUs: Approaches, Challenges, SuccessesArnon Shimoni
This talk will present SQream’s journey to building an analytics data warehouse powered by GPUs. SQream DB is an SQL data warehouse designed for larger than main-memory datasets (up to petabytes). It’s an on-disk database that combines novel ideas and algorithms to rapidly analyze trillions of rows with the help of high-throughput GPUs. We will explore some of SQream’s ideas and approaches to developing its analytics database – from simple prototype and tech demos, to a fully functional data warehouse product containing the most important features for enterprise deployment. We will also describe the challenges of working with exotic hardware like GPUs, and what choices had to be made in order to combine the CPU and GPU capabilities to achieve industry-leading performance – complete with real world use case comparisons.
As part of this discussion, we will also share some of the real issues that were discovered, and the engineering decisions that led to the creation of SQream DB’s high-speed columnar storage engine, designed specifically to take advantage of streaming architectures like GPUs.
Teradata Partners Conference Oct 2014 Big Data Anti-PatternsDouglas Moore
Douglas Moore discusses common anti-patterns seen when implementing big data solutions based on lessons learned from working with over 50 clients. He covers anti-patterns in hardware and infrastructure like relying on outdated reference architectures, tooling like trying to do analytics directly in NoSQL databases, and big data warehousing like over-curating data during ETL. The key is to understand the strengths and weaknesses of different tools and deploy the right solution for the intended workload.
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...Spark Summit
One of the key challenges in working with real-time and streaming data is that the data format for capturing data is not necessarily the optimal format for ad hoc analytic queries. For example, Avro is a convenient and popular serialization service that is great for initially bringing data into HDFS. Avro has native integration with Flume and other tools that make it a good choice for landing data in Hadoop. But columnar file formats, such as Parquet and ORC, are much better optimized for ad hoc queries that aggregate over large number of similar rows.
A Non-Standard use Case of Hadoop: High Scale Image Processing and AnalyticsDataWorks Summit
1. The Hadoop Image Processing (HIP) pipeline acquires vehicle images, identifies updates, generates URLs, crops and resizes images, copies them to asset servers, and removes duplicates.
2. It uses HBase for image storage and archiving, MapReduce for image processing, Kafka for publishing to asset servers, OpenCV for image processing, and Avro for data serialization.
3. Performance testing showed HIP scales linearly and is at least 10x faster than the previous system, and using cascading downloads provided a 20% performance gain.
Big Data Anti-Patterns: Lessons From the Front LIneDouglas Moore
This document summarizes common anti-patterns in big data projects based on lessons learned from working with over 50 clients. It identifies anti-patterns in hardware and infrastructure, tooling, and big data warehousing. Specifically, it discusses issues with referencing outdated architectures, using tools improperly for the workload, and de-normalizing schemas without understanding the implications. The document provides recommendations to instead co-locate data and computing, choose the right tools for each job, and deploy solutions matching the intended workload.
Hadoop summit 2010 frameworks panel elephant birdKevin Weil
Elephant Bird is a framework for working with structured data within Hadoop ecosystems. It allows users to specify a flexible, forward-backward compatible, self-documenting data schema and then generates code for input/output formats, Hadoop Writables, and Pig load/store functions. This reduces the amount of code needed and allows users to focus on their data. Elephant Bird underlies 20,000 Hadoop jobs per day at Twitter.
Accelerating analytics in a new era of dataArnon Shimoni
Organizations today produce exponentially more data than they did just a few years ago, but their databases weren’t built to handle these new volumes. As a result, reporting takes way too long, and some complex analytics simply cannot be done. The Era of Massive Data is upon us, and a new approach is required to overcome the limitations of traditional CPU-based data stores.
KEY TAKEAWAYS
- Flexible data exploration with minimal preparation
- Unrestricted access to your organization’s full scope of data
- Access to previously unobtainable insights, for smarter business decisions
HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...Cloudera, Inc.
Apache Drill is an interactive SQL query engine for analyzing large scale datasets. It allows for querying data stored in HBase and other data sources. Drill uses an optimistic execution model and late binding to schemas to enable fast queries without requiring metadata definitions. It leverages recent techniques like vectorized operators and late record materialization to improve performance. The project is currently in alpha stage but aims to support features like nested queries, Hive UDFs, and optimized joins with HBase.
SQream DB - Bigger Data On GPUs: Approaches, Challenges, SuccessesArnon Shimoni
This talk will present SQream’s journey to building an analytics data warehouse powered by GPUs. SQream DB is an SQL data warehouse designed for larger than main-memory datasets (up to petabytes). It’s an on-disk database that combines novel ideas and algorithms to rapidly analyze trillions of rows with the help of high-throughput GPUs. We will explore some of SQream’s ideas and approaches to developing its analytics database – from simple prototype and tech demos, to a fully functional data warehouse product containing the most important features for enterprise deployment. We will also describe the challenges of working with exotic hardware like GPUs, and what choices had to be made in order to combine the CPU and GPU capabilities to achieve industry-leading performance – complete with real world use case comparisons.
As part of this discussion, we will also share some of the real issues that were discovered, and the engineering decisions that led to the creation of SQream DB’s high-speed columnar storage engine, designed specifically to take advantage of streaming architectures like GPUs.
Teradata Partners Conference Oct 2014 Big Data Anti-PatternsDouglas Moore
Douglas Moore discusses common anti-patterns seen when implementing big data solutions based on lessons learned from working with over 50 clients. He covers anti-patterns in hardware and infrastructure like relying on outdated reference architectures, tooling like trying to do analytics directly in NoSQL databases, and big data warehousing like over-curating data during ETL. The key is to understand the strengths and weaknesses of different tools and deploy the right solution for the intended workload.
Real-time Analytics with Trino and Apache PinotXiang Fu
Trino summit 2021:
Overview of Trino Pinot Connector, which bridges the flexibility of Trino's full SQL support to the power of Apache Pinot's realtime analytics, giving you the best of both worlds.
Building a Scalable Web Crawler with Hadoop by Ahad Rana from CommonCrawl
Ahad Rana, engineer at CommonCrawl, will go over CommonCrawl’s extensive use of Hadoop to fulfill their mission of building an open, and accessible Web-Scale crawl. He will discuss their Hadoop data processing pipeline, including their PageRank implementation, describe techniques they use to optimize Hadoop, discuss the design of their URL Metadata service, and conclude with details on how you can leverage the crawl (using Hadoop) today.
Rolling Out Apache HBase for Mobile Offerings at Visa HBaseCon
Partha Saha and CW Chung (Visa)
Visa has embarked on an ambitious multi-year redesign of its entire data platform that powers its business. As part of this plan, the Apache Hadoop ecosystem, including HBase, will now become a staple in many of its solutions. Here, we will describe our journey in rolling out a high-availability NoSQL solution based on HBase behind some of our prominent mobile offerings.
Introduction to the Hadoop Ecosystem (FrOSCon Edition)Uwe Printz
Talk held at the FrOSCon 2013 on 24.08.2013 in Sankt Augustin, Germany
Agenda:
- What is Big Data & Hadoop?
- Core Hadoop
- The Hadoop Ecosystem
- Use Cases
- What‘s next? Hadoop 2.0!
You run your SQL-centric infrastructure for 10 years and slowly starting to note you can’t do this way anymore – everything is getting too expensive but your business requires things which are simply impossible without radical changes.
This is exact situation we had 2 years before. So we’d like to show our experience:
- Why and how we came into Big Data?
- Why we choose Apache and Hadoop?
- What to do and what is already done?
- What lessons were learned?
- Hadoop and relational databases: fight or synergy?
- Reactive Big Data manifest.
HBaseConAsia2018: Track2-5: JanusGraph-Distributed graph database with HBaseMichael Stack
This document provides an introduction to JanusGraph, an open source distributed graph database that can be used with Apache HBase for storage. It begins with background on graph databases and their structures, such as vertices, edges, properties, and different storage models. It then discusses JanusGraph's architecture, support for the TinkerPop graph computing framework, and schema and data modeling capabilities. Details are given on partitioning graphs across servers and using different indexing approaches. The document concludes by explaining why HBase is a good storage backend for JanusGraph and providing examples of how the data model would be structured within HBase.
Yahoo - Moving beyond running 100% of Apache Pig jobs on Apache TezDataWorks Summit
Last year at Yahoo, we spent great effort in scaling, stabilizing and making Pig on Tez production ready and by the end of the year retired running Pig jobs on Mapreduce. This talk will detail the performance and resource utilization improvements Yahoo achieved after migrating all Pig jobs to run on Tez.
After successful migration and the improved performance we shifted our focus to addressing some of the bottlenecks we identified and new optimization ideas that we came up with to make it go even faster. We will go over the new features and work done in Tez to make that happen like custom YARN ShuffleHandler, reworking DAG scheduling order, serialization changes, etc.
We will also cover exciting new features that were added to Pig for performance such as bloom join and byte code generation. A distributed bloom join that can create multiple bloom filters in parallel was straightforward to implement with the flexibility of Tez DAGs. It vastly improved performance and reduced disk and network utilization for our large joins. Byte code generation for projection and filtering of records is another big feature that we are targeting for Pig 0.17 which will speed up processing by reducing the virtual function calls.
Presto: SQL-on-Anything. Netherlands Hadoop User Group MeetupWojciech Biela
Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes. Presto was designed and written from the ground up for interactive analytics and approaches the speed of commercial data warehouses while scaling to the size of organizations like Facebook. One key feature in Presto is the ability to query data where it lives via an uniform ANSI SQL interface. Presto’s connector architecture creates an abstraction layer for anything that can be represented in a columnar or row-like format, such as HDFS, Amazon S3, Azure Storage, NoSQL stores, relational databases, Kafka streams and even proprietary data stores. Furthermore, a single Presto query can combine data from multiple sources, allowing for analytics across an entire organization.
HBaseCon 2012 | HBase for the Worlds Libraries - OCLCCloudera, Inc.
WorldCat is the world’s largest network of library content and services. Over 25,000 libraries in 170 countries have cooperated for 40 years to build WorldCat. OCLC is currently in the process of transitioning Worldcat from Oracle to Apache HBase. This session will discuss our data design for representing the constantly changing ownership information for thousands of libraries (billions of data points, millions of daily updates) and our plans for how we’re managing HBase in an environment that is equal parts end user facing and batch.
In this talk, Ian will table about Amazon Redshift, a managed petabyte scale data warehouse, give an overview of integration with Amazon Elastic MapReduce, a managed Hadoop environment, and cover some exciting new developments in the analytics space.
A comprehensive overview on the entire Hadoop operations and tools: cluster management, coordination, injection, streaming, formats, storage, resources, processing, workflow, analysis, search and visualization
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and SparkMichael Stack
Wei Li of Alibaba
Track 2: Ecology and Solutions
https://open.mi.com/conference/hbasecon-asia-2019
THE COMMUNITY EVENT FOR APACHE HBASE™
July 20th, 2019 - Sheraton Hotel, Beijing, China
https://hbase.apache.org/hbaseconasia-2019/
This document summarizes Gareth Llewellyn's experience redesigning the network architecture at DataSift to improve performance and scalability. The initial Cisco-based design suffered from issues like buffering, head of line blocking, and oversubscription of uplinks. Gareth considered moving to an Arista leaf-spine architecture with Arista 7050 core switches and 7048 top-of-rack switches, which would provide better redundancy, scalability, and throughput while reducing complexity compared to the mesh design. Questions are welcomed about the new design.
GPU databases - How to use them and what the future holdsArnon Shimoni
GPU databases are the hottest new thing, with about 7 different companies producing their own variant. In this session, we will discuss why they were created, how they are already disrupting the database world, and what the future of computing holds for them.
This presentation demonstrates how the power of NVIDIA GPUs can be leveraged to both accelerate speed to insight and to scale the amount of hot and warm data analyzed to meet the increasing demands of data scientists and business intelligence professionals alike, as well as to find tactical and strategic insights with greater speed on exponentially growing datasets.
Organizations commonly believe that they are advancing in analytical capabilities due to the rise in the data science profession and the myriad of technologies available for analytics, business intelligence, artificial intelligence and machine learning. However, if you do the math, they are actually falling behind as the increases in the rates of data collection volume far outpace the rate of increases in hot and warm data used for analytics. This is causing organizations to rely on an ever-decreasing percentage of their information assets for decision making.
We talk about why GPU databases were created and share what sets SQream apart from other GPU databases, MPP solutions, in memory and Hadoop based analytic alternatives.
We will also outline how an organization can use GPU databases to thrive in the information revolution by using a significantly greater percentage of its data for analytical purposes, obtaining insights that are desired today, and will remain cost-effective into the next few years when data lakes are expected to balloon from petabytes to exabytes.
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...Michael Stack
This document summarizes a presentation on scaling a 30 TB data lake using Apache HBase and Scala. It introduces Apache HBase and Spark as technologies for building fast data platforms. It then describes a case study where they were used to architect a retail analytics platform capable of processing 4.6 billion events weekly from 30 TB of historical data. Key aspects included using HBase for data deduplication and as a master data management system, and connecting Spark to HBase using a Scala DSL for efficient querying and updates at scale. Performance was improved 5x by reengineering the data pipeline to be highly concurrent and asynchronous.
A New “Sparkitecture” for Modernizing your Data Warehouse: Spark Summit East ...Spark Summit
Legacy enterprise data warehouse (EDW) architecture, geared toward day-to-day workloads associated with operational querying, reporting, and analytics, are often ill-equipped to handle the volume of data, traffic, and varied data types associated with a modern, ad-hoc analytics platform. Faced with challenges of increasing pipeline speed, aggregation, and visualization in a simplified, self-service fashion, organizations are increasingly turning to some combination of Spark, Hadoop, Kafka, and proven analytical databases like Vertica as key enabling technologies to optimize their EDW architecture. Join us to learn how successful organizations have developed real-time streaming solutions with these technologies for range of use cases, including IOT predictive maintenance.
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...Sumeet Singh
This document discusses lessons learned from building a scalable, self-serve, real-time, multi-tenant monitoring service at Yahoo. It describes transitioning from a classical architecture to one based on real-time big data technologies like Storm and Kafka. Key lessons include properly handling producer-consumer problems at scale, challenges of debugging skewed data, strategically managing multi-tenancy and resources, issues optimizing asynchronous systems, and not neglecting assumptions outside the application.
Data Freeway is a system developed by Facebook to handle large volumes of data in real-time at scale. It includes components like Scribe for distributed logging, Calligraphus for persisting logs to HDFS, and Puma for real-time analytics on the data. The system is designed to handle over 10GB/second of data reliably with low latency of less than 10 seconds for 99% of data. It provides a simple interface for applications to access real-time data streams through tools like ptail. The system is open source and used at Facebook to power applications like real-time search, spam detection, and metrics analysis.
The main topic of slides is building high availability high throughput system for receiveing and saving different kind of information with horizontal scalling possibility using HBase, Flume and Grizzly hosted on Amazon EC2 low cost instances. Talk describes HBase HA cluster setup process with useful hints and EC2 pitfalls, Flume setup process with providing comparasion between standalone and embedded Flume versions and show difference and usecases of both versions. A lot of attention payed to Flume2Hbase streaming features with tweaks and different approaches for speeding up this process.
The document summarizes Oracle Cloud services including Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and Software as a Service (SaaS). It provides an overview of Oracle database, middleware, and engineered systems available on Oracle Cloud. It also discusses how to create a database on Oracle Cloud using REST APIs and cURL, and how to perform RMAN backups to Oracle Cloud Storage. Finally, it covers connecting to databases on Oracle Cloud using Enterprise Manager and SQL Developer.
Real-time Analytics with Trino and Apache PinotXiang Fu
Trino summit 2021:
Overview of Trino Pinot Connector, which bridges the flexibility of Trino's full SQL support to the power of Apache Pinot's realtime analytics, giving you the best of both worlds.
Building a Scalable Web Crawler with Hadoop by Ahad Rana from CommonCrawl
Ahad Rana, engineer at CommonCrawl, will go over CommonCrawl’s extensive use of Hadoop to fulfill their mission of building an open, and accessible Web-Scale crawl. He will discuss their Hadoop data processing pipeline, including their PageRank implementation, describe techniques they use to optimize Hadoop, discuss the design of their URL Metadata service, and conclude with details on how you can leverage the crawl (using Hadoop) today.
Rolling Out Apache HBase for Mobile Offerings at Visa HBaseCon
Partha Saha and CW Chung (Visa)
Visa has embarked on an ambitious multi-year redesign of its entire data platform that powers its business. As part of this plan, the Apache Hadoop ecosystem, including HBase, will now become a staple in many of its solutions. Here, we will describe our journey in rolling out a high-availability NoSQL solution based on HBase behind some of our prominent mobile offerings.
Introduction to the Hadoop Ecosystem (FrOSCon Edition)Uwe Printz
Talk held at the FrOSCon 2013 on 24.08.2013 in Sankt Augustin, Germany
Agenda:
- What is Big Data & Hadoop?
- Core Hadoop
- The Hadoop Ecosystem
- Use Cases
- What‘s next? Hadoop 2.0!
You run your SQL-centric infrastructure for 10 years and slowly starting to note you can’t do this way anymore – everything is getting too expensive but your business requires things which are simply impossible without radical changes.
This is exact situation we had 2 years before. So we’d like to show our experience:
- Why and how we came into Big Data?
- Why we choose Apache and Hadoop?
- What to do and what is already done?
- What lessons were learned?
- Hadoop and relational databases: fight or synergy?
- Reactive Big Data manifest.
HBaseConAsia2018: Track2-5: JanusGraph-Distributed graph database with HBaseMichael Stack
This document provides an introduction to JanusGraph, an open source distributed graph database that can be used with Apache HBase for storage. It begins with background on graph databases and their structures, such as vertices, edges, properties, and different storage models. It then discusses JanusGraph's architecture, support for the TinkerPop graph computing framework, and schema and data modeling capabilities. Details are given on partitioning graphs across servers and using different indexing approaches. The document concludes by explaining why HBase is a good storage backend for JanusGraph and providing examples of how the data model would be structured within HBase.
Yahoo - Moving beyond running 100% of Apache Pig jobs on Apache TezDataWorks Summit
Last year at Yahoo, we spent great effort in scaling, stabilizing and making Pig on Tez production ready and by the end of the year retired running Pig jobs on Mapreduce. This talk will detail the performance and resource utilization improvements Yahoo achieved after migrating all Pig jobs to run on Tez.
After successful migration and the improved performance we shifted our focus to addressing some of the bottlenecks we identified and new optimization ideas that we came up with to make it go even faster. We will go over the new features and work done in Tez to make that happen like custom YARN ShuffleHandler, reworking DAG scheduling order, serialization changes, etc.
We will also cover exciting new features that were added to Pig for performance such as bloom join and byte code generation. A distributed bloom join that can create multiple bloom filters in parallel was straightforward to implement with the flexibility of Tez DAGs. It vastly improved performance and reduced disk and network utilization for our large joins. Byte code generation for projection and filtering of records is another big feature that we are targeting for Pig 0.17 which will speed up processing by reducing the virtual function calls.
Presto: SQL-on-Anything. Netherlands Hadoop User Group MeetupWojciech Biela
Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes. Presto was designed and written from the ground up for interactive analytics and approaches the speed of commercial data warehouses while scaling to the size of organizations like Facebook. One key feature in Presto is the ability to query data where it lives via an uniform ANSI SQL interface. Presto’s connector architecture creates an abstraction layer for anything that can be represented in a columnar or row-like format, such as HDFS, Amazon S3, Azure Storage, NoSQL stores, relational databases, Kafka streams and even proprietary data stores. Furthermore, a single Presto query can combine data from multiple sources, allowing for analytics across an entire organization.
HBaseCon 2012 | HBase for the Worlds Libraries - OCLCCloudera, Inc.
WorldCat is the world’s largest network of library content and services. Over 25,000 libraries in 170 countries have cooperated for 40 years to build WorldCat. OCLC is currently in the process of transitioning Worldcat from Oracle to Apache HBase. This session will discuss our data design for representing the constantly changing ownership information for thousands of libraries (billions of data points, millions of daily updates) and our plans for how we’re managing HBase in an environment that is equal parts end user facing and batch.
In this talk, Ian will table about Amazon Redshift, a managed petabyte scale data warehouse, give an overview of integration with Amazon Elastic MapReduce, a managed Hadoop environment, and cover some exciting new developments in the analytics space.
A comprehensive overview on the entire Hadoop operations and tools: cluster management, coordination, injection, streaming, formats, storage, resources, processing, workflow, analysis, search and visualization
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and SparkMichael Stack
Wei Li of Alibaba
Track 2: Ecology and Solutions
https://open.mi.com/conference/hbasecon-asia-2019
THE COMMUNITY EVENT FOR APACHE HBASE™
July 20th, 2019 - Sheraton Hotel, Beijing, China
https://hbase.apache.org/hbaseconasia-2019/
This document summarizes Gareth Llewellyn's experience redesigning the network architecture at DataSift to improve performance and scalability. The initial Cisco-based design suffered from issues like buffering, head of line blocking, and oversubscription of uplinks. Gareth considered moving to an Arista leaf-spine architecture with Arista 7050 core switches and 7048 top-of-rack switches, which would provide better redundancy, scalability, and throughput while reducing complexity compared to the mesh design. Questions are welcomed about the new design.
GPU databases - How to use them and what the future holdsArnon Shimoni
GPU databases are the hottest new thing, with about 7 different companies producing their own variant. In this session, we will discuss why they were created, how they are already disrupting the database world, and what the future of computing holds for them.
This presentation demonstrates how the power of NVIDIA GPUs can be leveraged to both accelerate speed to insight and to scale the amount of hot and warm data analyzed to meet the increasing demands of data scientists and business intelligence professionals alike, as well as to find tactical and strategic insights with greater speed on exponentially growing datasets.
Organizations commonly believe that they are advancing in analytical capabilities due to the rise in the data science profession and the myriad of technologies available for analytics, business intelligence, artificial intelligence and machine learning. However, if you do the math, they are actually falling behind as the increases in the rates of data collection volume far outpace the rate of increases in hot and warm data used for analytics. This is causing organizations to rely on an ever-decreasing percentage of their information assets for decision making.
We talk about why GPU databases were created and share what sets SQream apart from other GPU databases, MPP solutions, in memory and Hadoop based analytic alternatives.
We will also outline how an organization can use GPU databases to thrive in the information revolution by using a significantly greater percentage of its data for analytical purposes, obtaining insights that are desired today, and will remain cost-effective into the next few years when data lakes are expected to balloon from petabytes to exabytes.
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...Michael Stack
This document summarizes a presentation on scaling a 30 TB data lake using Apache HBase and Scala. It introduces Apache HBase and Spark as technologies for building fast data platforms. It then describes a case study where they were used to architect a retail analytics platform capable of processing 4.6 billion events weekly from 30 TB of historical data. Key aspects included using HBase for data deduplication and as a master data management system, and connecting Spark to HBase using a Scala DSL for efficient querying and updates at scale. Performance was improved 5x by reengineering the data pipeline to be highly concurrent and asynchronous.
A New “Sparkitecture” for Modernizing your Data Warehouse: Spark Summit East ...Spark Summit
Legacy enterprise data warehouse (EDW) architecture, geared toward day-to-day workloads associated with operational querying, reporting, and analytics, are often ill-equipped to handle the volume of data, traffic, and varied data types associated with a modern, ad-hoc analytics platform. Faced with challenges of increasing pipeline speed, aggregation, and visualization in a simplified, self-service fashion, organizations are increasingly turning to some combination of Spark, Hadoop, Kafka, and proven analytical databases like Vertica as key enabling technologies to optimize their EDW architecture. Join us to learn how successful organizations have developed real-time streaming solutions with these technologies for range of use cases, including IOT predictive maintenance.
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...Sumeet Singh
This document discusses lessons learned from building a scalable, self-serve, real-time, multi-tenant monitoring service at Yahoo. It describes transitioning from a classical architecture to one based on real-time big data technologies like Storm and Kafka. Key lessons include properly handling producer-consumer problems at scale, challenges of debugging skewed data, strategically managing multi-tenancy and resources, issues optimizing asynchronous systems, and not neglecting assumptions outside the application.
Data Freeway is a system developed by Facebook to handle large volumes of data in real-time at scale. It includes components like Scribe for distributed logging, Calligraphus for persisting logs to HDFS, and Puma for real-time analytics on the data. The system is designed to handle over 10GB/second of data reliably with low latency of less than 10 seconds for 99% of data. It provides a simple interface for applications to access real-time data streams through tools like ptail. The system is open source and used at Facebook to power applications like real-time search, spam detection, and metrics analysis.
The main topic of slides is building high availability high throughput system for receiveing and saving different kind of information with horizontal scalling possibility using HBase, Flume and Grizzly hosted on Amazon EC2 low cost instances. Talk describes HBase HA cluster setup process with useful hints and EC2 pitfalls, Flume setup process with providing comparasion between standalone and embedded Flume versions and show difference and usecases of both versions. A lot of attention payed to Flume2Hbase streaming features with tweaks and different approaches for speeding up this process.
The document summarizes Oracle Cloud services including Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and Software as a Service (SaaS). It provides an overview of Oracle database, middleware, and engineered systems available on Oracle Cloud. It also discusses how to create a database on Oracle Cloud using REST APIs and cURL, and how to perform RMAN backups to Oracle Cloud Storage. Finally, it covers connecting to databases on Oracle Cloud using Enterprise Manager and SQL Developer.
An updated version of the introduction to our gospel series "Growing Deep in the Gospel". In it we look at how Jesus, the Gospel, the Church and the Mission form the basis of the "Big Picture". If we understand the "Big Picture" we will be able to better answer questions like "What does God says life is all about?", "Why did God create me?", "What is the purpose of life?", "What is God's will for me?", and "How do I decide what is important in life?".
GNW01: In-Memory Processing for DatabasesTanel Poder
This document discusses in-memory execution for databases. It begins with introductions and background on the author. It then discusses how databases can offload data to memory to improve query performance 2-24x by analyzing storage use and access patterns. It covers concepts like how RAM access is now the performance bottleneck and how CPU cache-friendly data structures are needed. It shows examples measuring performance differences when scanning data in memory versus disk. Finally, it discusses future directions like more integrated storage and memory and new data formats optimized for CPU caches.
Top 10 Reasons Events Are the Best B2B Marketing Channel in the WorldDoubleDutch
Marketers spend a lot on events... sometimes as much as 50% of the B2B marketing budget is allocated to live events (sponsored and produced). Events impact every aspect of marketing and they can be the ultimate content distribution engine. See why events really are the best B2B marketing channel in the world!
Gluent New World #02 - SQL-on-Hadoop : A bit of History, Current State-of-the...Mark Rittman
Hadoop and NoSQL platforms initially focused on Java developers and slow but massively-scalable MapReduce jobs as an alternative to high-end but limited-scale analytics RDBMS engines. Apache Hive opened-up Hadoop to non-programmers by adding a SQL query engine and relational-style metadata layered over raw HDFS storage, and since then open-source initiatives such as Hive Stinger, Cloudera Impala and Apache Drill along with proprietary solutions from closed-source vendors have extended SQL-on-Hadoop’s capabilities into areas such as low-latency ad-hoc queries, ACID-compliant transactions and schema-less data discovery – at massive scale and with compelling economics.
In this session we’ll focus on technical foundations around SQL-on-Hadoop, first reviewing the basic platform Apache Hive provides and then looking in more detail at how ad-hoc querying, ACID-compliant transactions and data discovery engines work along with more specialised underlying storage that each now work best with – and we’ll take a look to the future to see how SQL querying, data integration and analytics are likely to come together in the next five years to make Hadoop the default platform running mixed old-world/new-world analytics workloads.
Learn why successful leaders are keeping a journal. See the direct benefits of journaling and how it can improve your life.
BONUS: Download this free Journaling Template:
https://lifeboarding.co/bonus-journaling
If you liked this presentation you can download it here:
https://lifeboarding.co/presentation-download-journaling
As humans, we never fail to think that we are highly intelligent beings, and that we are mentally superior than any other creatures found on Earth.
Well, that...... may be true.
However, we can be equally stupid and dumb too.
Worse still, we don't even realize it - in terms of how we can make erroneous judgments, decisions and choices, based on how our mind processes and filters information, as well as how our belief system works.
As intriguing and exciting this topic is to me, I find it difficult to illustrate the concepts involve, and that took me nearly 6 months to complete this work. (The Planning Fallacy in play?!) Throughout writing this deck, I've made a total of 8 major revisions before coming to this final piece.
I hope you'll find this deck both interesting and useful!
The Productivity Secret Of The Best LeadersOfficevibe
Content by Jacob Shriar & Kevin Kruse.
In this Officeviibe presentation, you'll see:
- 3 biggest problems leaders face and what you can do to fix them
- The secret to time management
- Examples from great leaders
- You'll find bonus content
The document provides an overview of Google Cloud's data platform and big data portfolio. It discusses Google Cloud Platform and its various data storage and database services like Cloud Storage, Cloud Bigtable, Cloud Datastore, Cloud SQL, Cloud Spanner, and BigQuery. It then summarizes each service's ideal use cases. The document also presents Google Cloud's big data reference architectures and data science reference architecture. It concludes by highlighting BigQuery's advantages over other data warehouse solutions and providing a link to a BigQuery hands-on lab.
Say goodbye to data silos! Analytics in a Day will simplify and accelerate your journey towards the modern data warehouse. Join CCG and Microsoft for a two-day virtual workshop, hosted by James McAuliffe.
Many organizations focus on the licensing cost of Hadoop when considering migrating to a cloud platform. But other costs should be considered, as well as the biggest impact, which is the benefit of having a modern analytics platform that can handle all of your use cases. This session will cover lessons learned in assisting hundreds of companies to migrate from Hadoop to Databricks.
The document discusses how organizations can leverage big data through Oracle's integrated big data solutions. It describes Oracle's offerings for acquiring and organizing big data from various sources using products like Oracle NoSQL Database and Hadoop. It then discusses how Oracle solutions allow users to analyze large datasets using R and visualize insights in BI dashboards. Finally, it provides an overview of Oracle's Exalytics and Big Data Appliance hardware and software platforms for processing and managing big data at scale.
Modernizing to a Cloud Data ArchitectureDatabricks
Organizations with on-premises Hadoop infrastructure are bogged down by system complexity, unscalable infrastructure, and the increasing burden on DevOps to manage legacy architectures. Costs and resource utilization continue to go up while innovation has flatlined. In this session, you will learn why, now more than ever, enterprises are looking for cloud alternatives to Hadoop and are migrating off of the architecture in large numbers. You will also learn how elastic compute models’ benefits help one customer scale their analytics and AI workloads and best practices from their experience on a successful migration of their data and workloads to the cloud.
Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...Lace Lofranco
Data orchestration is the lifeblood of any successful data analytics solution. Take a deep dive into Azure Data Factory's data movement and transformation activities, particularly its integration with Azure's Big Data PaaS offerings such as HDInsight, SQL Data warehouse, Data Lake, and AzureML. Participants will learn how to design, build and manage big data orchestration pipelines using Azure Data Factory and how it stacks up against similar Big Data orchestration tools such as Apache Oozie.
Video of presentation:
https://channel9.msdn.com/Events/Ignite/Australia-2017/DA332
Integrating Google Cloud Dataproc with Alluxio for faster performance in the ...Alluxio, Inc.
Google Dataproc is Google Cloud's fully managed Apache Spark and Apache Hadoop service. Alluxio is an open source data orchestration platform that can be used with Dataproc to accelerate analytics workloads. With a single initialization action, Alluxio can be installed on a Dataproc cluster to cache data from Cloud Storage for faster queries. Alluxio also enables "zero-copy bursting" of workloads to the cloud by allowing frameworks to access data directly from remote HDFS without needing to copy it. This provides elastic compute capacity while avoiding high network latency and bandwidth costs of copying large datasets.
The document discusses real-time analytics best practices using Google Cloud Platform technologies. It provides an overview of AllCloud, a cloud services company, and their experience. The presentation covers big data concepts, real-time analytics patterns, and using Google Cloud Dataflow for stream processing. Example architectures are presented for ingesting data, processing it in real-time, and analyzing the results.
The document discusses real-time analytics and big data best practices. It provides an overview of AllCloud, a company with 9 years of cloud experience. It then covers topics like big data characteristics of volume, velocity and variety. Different architectures for historical analytics versus real-time analytics are presented. Google Cloud Platform services like PubSub, DataFlow and BigQuery are discussed in the context of real-time data pipelines. Finally complex event processing engines and data processing technologies are compared.
This document discusses using Azure HDInsight for big data applications. It provides an overview of HDInsight and describes how it can be used for various big data scenarios like modern data warehousing, advanced analytics, and IoT. It also discusses the architecture and components of HDInsight, how to create and manage HDInsight clusters, and how HDInsight integrates with other Azure services for big data and analytics workloads.
IS-4082, Real-Time insight in Big Data – Even faster using HSA, by Norbert He...AMD Developer Central
This document discusses using heterogeneous system architecture (HSA) to provide real-time insights from big data more quickly. It describes ParStream's technical architecture, which uses a columnar database, in-memory technology, and distributed query processing to enable fast analytics on large datasets. The document explains how ParStream's execution engine breaks queries into modular operations that can be distributed across different processing units, allowing it to make use of HSA to accelerate query performance.
How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
Big Data Integration Webinar: Getting Started With Hadoop Big DataPentaho
This document discusses getting started with big data analytics using Hadoop and Pentaho. It provides an overview of installing and configuring Hadoop and Pentaho on a single machine or cluster. Dell's Crowbar tool is presented as a way to quickly deploy Hadoop clusters on Dell hardware in about two hours. The document also covers best practices like leveraging different technologies, starting with small datasets, and not overloading networks. A demo is given and contact information provided.
Analytics in a Day Ft. Synapse Virtual WorkshopCCG
Say goodbye to data silos! Analytics in a Day will simplify and accelerate your journey towards the modern data warehouse. Join CCG and Microsoft for a half-day virtual workshop, hosted by James McAuliffe.
Managing data analytics in a hybrid cloudKaran Singh
Managing Data Analytics in a Hybrid Cloud discusses challenges with traditional analytics approaches and proposes using shared data lakes with dynamic compute clusters. Common challenges include explosive analytics team growth leading to resource contention, and duplicating large datasets for each cluster. The proposed approach uses shared object storage to hold unified datasets accessed by multiple ephemeral analytics clusters provisioned on-demand. This allows teams independent resources while avoiding duplicate storage costs and improving agility. The document outlines example architectures and benefits of this shared data lake approach when implemented on a private or public cloud.
The document discusses Big Data on Azure and provides an overview of HDInsight, Microsoft's Apache Hadoop-based data platform on Azure. It describes HDInsight cluster types for Hadoop, HBase, Storm and Spark and how clusters can be automatically provisioned on Azure. Example applications and demos of Storm, HBase, Hive and Spark are also presented. The document highlights key aspects of using HDInsight including storage integration and tools for interactive analysis.
Developing high frequency indicators using real time tick data on apache supe...Zekeriya Besiroglu
This document summarizes the Central Bank of Turkey's project to develop high frequency market indicators using real-time tick data from the Thomson Reuters Enterprise Platform. It describes how they set up Apache Kafka, Druid, Spark and Superset on Hadoop to ingest, store, analyze and visualize the data. Their goal was to observe foreign exchange markets in real-time to detect risks and patterns. The architecture evolved over three phases from an initial test cluster to integrating Druid and Hive for improved querying and scaling to production. Work is ongoing to implement additional indicators and integrate historical data for enhanced analysis.
This document outlines 10 vital tips for optimizing Oracle Real Application Clusters (RAC) performance. The tips include: 1) properly sizing capacity and architecture based on hardware components and estimated database sizes; 2) tuning SQL and parallel query performance through techniques like partitioning, parallelism, and reducing full table scans; 3) additional tuning of the database, network, recovery processes, global cache, storage, and Clusterware can further optimize RAC performance.
Analysis insight about a Flyball dog competition team's performanceroli9797
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Aggregage
This webinar will explore cutting-edge, less familiar but powerful experimentation methodologies which address well-known limitations of standard A/B Testing. Designed for data and product leaders, this session aims to inspire the embrace of innovative approaches and provide insights into the frontiers of experimentation!
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeWalaa Eldin Moustafa
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
State of Artificial intelligence Report 2023kuntobimo2016
Artificial intelligence (AI) is a multidisciplinary field of science and engineering whose goal is to create intelligent machines.
We believe that AI will be a force multiplier on technological progress in our increasingly digital, data-driven world. This is because everything around us today, ranging from culture to consumer products, is a product of intelligence.
The State of AI Report is now in its sixth year. Consider this report as a compilation of the most interesting things we’ve seen with a goal of triggering an informed conversation about the state of AI and its implication for the future.
We consider the following key dimensions in our report:
Research: Technology breakthroughs and their capabilities.
Industry: Areas of commercial application for AI and its business impact.
Politics: Regulation of AI, its economic implications and the evolving geopolitics of AI.
Safety: Identifying and mitigating catastrophic risks that highly-capable future AI systems could pose to us.
Predictions: What we believe will happen in the next 12 months and a 2022 performance review to keep us honest.
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...sameer shah
"Join us for STATATHON, a dynamic 2-day event dedicated to exploring statistical knowledge and its real-world applications. From theory to practice, participants engage in intensive learning sessions, workshops, and challenges, fostering a deeper understanding of statistical methodologies and their significance in various fields."
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataKiwi Creative
Harness the power of AI-backed reports, benchmarking and data analysis to predict trends and detect anomalies in your marketing efforts.
Peter Caputa, CEO at Databox, reveals how you can discover the strategies and tools to increase your growth rate (and margins!).
From metrics to track to data habits to pick up, enhance your reporting for powerful insights to improve your B2B tech company's marketing.
- - -
This is the webinar recording from the June 2024 HubSpot User Group (HUG) for B2B Technology USA.
Watch the video recording at https://youtu.be/5vjwGfPN9lw
Sign up for future HUG events at https://events.hubspot.com/b2b-technology-usa/
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
1. ZEKERIYA BEŞIROĞLU
BILGINC IT ACADEMY
ORACLE CLOUD DAY
19-11-2015
TROUG-TURKISH ORACLE USER GROUP
BIG DATA : BIG PICTURE
2. ZEKERIYA BEŞIROĞLU
▸ +18 IT
▸ +15 ORACLE DB&DWH
▸ +3 BIG DATA
▸ Leader of TROUG
▸ Instructor&Consultant
▸ http://zekeriyabesiroglu.com
▸ @zbesiroglu
TROUG BIG DATA
BIG PICTURE
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
5. METIN
BIG DATA
Social networks
Banking and financial services
E-commerce services
Web-centric services
Internet search indexes
Scientific and document searches
Medical records
Web loggs
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
7. FIRMALAR ,
MÜŞTERILERININ DNA
SINI ANALIZ ETMEK
ZORUNDALAR.
Zekeriya Beşiroğlu
TROUG
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
8. TROUG
BIG DATADA HEDEF NEDİR? NASIL
YAPILMALI?
▸Big data teknolojilerini kullanarak business’a nasıl değer
katabilirim. Bir takım costları azaltabilirmiyim?
▸Big Data ile geleneksel database nasıl entegre edeceğim?
Structured,semi structured ve unstructured verileri
birleştirme
▸Analytics toolları ile sonuça ulaşma. Oracle Advance
Analytics,BI ve DW teknolojileri
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
9. TROUG
DATA
▸ Schema on Write yapıyoruz
▸ Schema on READ yapalım.
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
10. TROUG
BIG DATA PROJESI SAFHALARI
▸DATA ACQUISITION and Storage
▸DATA ACCESS and Processing
▸Data Unification and Analysis
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
11. DATA ACQUISITION AND STORAGE
HADOOP DISTRIBUTED FILE SYSTEM-HDFS
▸petabyte-scale distributed file system
▸linearly scalable on commodity hardware
▸Schema on Read
▸Cheaper
▸low security
▸write once,read many
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
12. DATA ACQUISITION AND STORAGE
HADOOP DISTRIBUTED FILE SYSTEM-HDFS
▸Basic file system operations
▸JSON log file HDFS yükleyebilirim. (hadoop fs -put)
13. DATA ACQUISITION AND STORAGE
WHAT IS FLUME?
▸Avro Source
▸Memory Channel
▸HDFS Sink
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
14. DATA ACQUISITION AND STORAGE
ORACLE NOSQL DATABASE
▸Key Value Database
▸Access by java Apı
▸Stores unstructured or semi structured data as byte arrays
▸Highly reliable
▸Scalable throughput and predictable latency
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
15. DATA ACQUISITION AND STORAGE
RDBMS & NOSQL
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
16. DATA ACQUISITION AND STORAGE
HDFS & NOSQL
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
17. DATA ACQUISITION AND STORAGE
APPLICATION DATABASE TECHNOLOGY
▸High Volume with Low value
▸Dynamic application schema
▸if answer yes NOSQL
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
18. DATA ACQUISITION AND STORAGE
NOSQL EXAMPLE
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
19. DATA ACCESS AND PROCESSING
MAP REDUCE
▸Write applications that process vast amounts of data , in
parallel on large cluster of commodity hardware in reliable
and fault tolerant.
▸Storing data in HDFS is low cost , fault tolerant and scalable.
▸Integrates with HDFS to provide parallel data processing
▸Batch-oriented
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
20. DATA ACCESS AND PROCESSING
MAP REDUCE ORNEK
map(String input_key, String input_value)
foreach word w in input_value:
emit(w, 1)
reduce(String output_key,
Iterator<int> intermediate_vals)
set count = 0
foreach v in intermediate_vals:
count += v
emit(output_key, count)
(1000,’Galatasaray sampiyon olur’)
(2000,’beşiktas sampiyon olur’)
(2200,’Galatasaray Türkiyedir’)
(3000,’fenerbahce sampiyon olur’)
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
21. DATA ACCESS AND PROCESSING
MAP REDUCE ORNEK
Output Mapper
(‘Galatasaray’, 1), (‘sampiyon’, 1), (‘olur’, 1), (‘beşiktas’, 1),
(‘sampiyon, 1), (‘olur’, 1), (‘Galatasaray’, 1), (‘Türkiyedir’, 1) (‘fenerbahce’, 1),
(‘sampiyon, 1), (‘olur’, 1)
Intermediate Data Reducer’a gönderilen
(‘Galatasaray’,[1,1])
(‘sampiyon’,[1,1,1])
(‘olur’,[1,1,1])
(‘beşiktas’,[1])
(‘fenerbahce’,[1])
(‘Türkiyedir’,[1])
Reducer’ın son cıktısı
(‘sampiyon’,3)
(‘olur’,3)
(‘Galatasaray’,2)
(‘fenerbahce’,1)
(‘beşiktas’,1)
(‘Türkiyedir’,1)
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
22. DATA ACCESS AND PROCESSING
HIVE
▸SQL to query HDFS by using Hive QL(SQL like language)
▸Hive transform HiveQL queries into standard Mapreduce
jobs
▸Schema on Read via InputFormat and SerDe
▸Not ideal for ad hoc(slow)
▸Immature optimizer
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
23. DATA ACCESS AND PROCESSING
HIVE
▸Log Processing
▸Text mining
▸Document Indexing
▸Business Analytics
▸Predictive Modeling
▸Not ideal for ad hoc query
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
24. DATA ACCESS AND PROCESSING
PIG
▸Open Source Data flow system
▸simple language for queries and data manipulation, which is
compiled into map-reduce jobs that are run on hadoop
▸Provides common operations like join,group,sort
▸Works on files in HDFS
▸Ad hoc queries across large data sets.
▸log analysis
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
25. DATA ACCESS AND PROCESSING
CLOUDERA IMPALA
▸DATABASE -LIKE SQL layer on top of Hadoop
▸Distributed,massively parallel processing database engine
▸SQL is the primary development language
▸Open Source,Impala process data in hadoop cluster
WITHOUT using MapReduce
▸Interactive analysis on data stored in HDFS and Hbase
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
26. DATA ACCESS AND PROCESSING
ORACELE XQUERY FOR HADOOP
▸Is a transform engine for semistructured data that is stored in
Apache Hadoop
▸Transform Xquery language translating them into series of
Mapreduce
▸load data efficiently into Oracle Database by using Oracle
Loader for Hadoop
▸Provides read and write support to Oracle NOSQL DB
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
27. DATA ACCESS AND PROCESSING
ORACELE XQUERY FOR HADOOP
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
28. DATA ACCESS AND PROCESSING
APACHE SPARK
▸Open Source parallel data processing
▸Develop Fast
▸Online Streaming
▸Interactive analytics
▸Machine Learning
▸Speed
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
29. DATA ACCESS AND PROCESSING
APACHE SPARK ÖRNEK
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
30. DATA UNIFICATION AND ANALYSIS
APACHE SQOOP
▸Batch Loading
▸Transfer bulk data between structured data stores and
Apache Hadoop
▸Data import and Export between external data stores and
Hadoop
▸Parallelizes data transfer for fast performance
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
31. DATA UNIFICATION AND ANALYSIS
ORACLE LOADER FOR HADOOP
▸Batch Loading
▸High performance loader for fast movement of data from
Hadoop into a table in Oracle Database
▸Loading using online and offline modes
▸offloading expensive data processing from the database
server to hadoop
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
32. DATA UNIFICATION AND ANALYSIS
COPY TO BDA
▸Batch Loading
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
33. DATA UNIFICATION AND ANALYSIS
ORACLE SQL CONNECTOR FOR
HADOOP
▸ Generate external table in database
pointing to HDFS data
▸ Load into database or query data in
place on HDFS
▸ Fine-grained control over type
mapping
▸ Parallel load with automatic load
balancing
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
34. DATA UNIFICATION AND ANALYSIS
ORACLE TECHNOLOGIES
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
35. DATA UNIFICATION AND ANALYSIS
ORACLE ADVANCED ANALYTICS
▸OAA=Oracle Data Mining+Oracle R enterprise
▸Performance
▸Predictive Analytics
▸Easy
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE
36. METIN
ORACLE BDA BENEFITS
▸ Ships with leading Hadoop
distribution(Cloudera)
▸ Hdfs,hbase,hive,flume,kafka,spark …
▸ Cloudera manager
▸ Ships with great connectivity to Oracle
Db
▸ Big Data SQL
▸ Big Data Connectors & ODI
TROUG @ZBESIROGLU BILGINC IT ACADEMY BIG DATA BIG
PICTURE