gsoc_mentor for Shivram Mani
•
0 likes•160 views
This certificate of appreciation was presented to Shivram Mani from the Apache Software Foundation for serving as a mentor during Google Summer of Code 2016 from April 22 to August 23, 2016. Jason Titus, VP of Engineering, recognized Shivram Mani's contributions as a mentor during the summer program.
Report
Share
Report
Share
Download to read offline
Recommended
PXF BDAM 2016
PXF is a unified access framework that provides a uniform SQL interface for heterogeneous data sources on HDFS. It exploits parallelism to efficiently access data across various storage formats and data sources. PXF uses a pluggable architecture with built-in connectors that allow it to access data in HDFS files, Hive tables, HBase tables, and other data sources. It provides a common developer view and allows writing queries against external data using various profile definitions and plugins.
Managing Apache HAWQ with Apache AMBARI
This document summarizes a presentation about managing Apache HAWQ, an open source massively parallel processing (MPP) database, using Apache Ambari. It discusses how Ambari integrates with HAWQ for installation, configuration, topology recommendations, high availability, alerts and more. Challenges in the integration are addressed as HAWQ is not part of the Hortonworks Data Platform stack. The presentation recommends future work for Ambari like supporting automated HAWQ upgrades and enabling dynamic configuration reloads without requiring a service restart.
Apache Zeppelin Meetup Christian Tzolov 1/21/16
Zeppelin Interpreters
PSQL (to became JDBC in 0.6.x)
Geode
SpringXD
Apache Ambari
Zeppelin Service
Geode, HAWQ and Spring XD services
Webpage Embedder View
Hawq Hcatalog Integration
1. HCatalog is a table and storage management layer for Hadoop that provides a relational view of data in HDFS and abstracts data formats and locations from users.
2. Previously, HAWQ accessed Hive tables through PXF using external tables, but this required specifying the schema, location, and format which was error prone and wouldn't detect metadata changes.
3. The new integration retrieves metadata from HCatalog and parses it into in-memory catalog tables to provide dynamic access to Hive tables from HAWQ without needing to specify schemas.
PXF HAWQ Unmanaged Data
PXF is an extension framework that facilitates parallel access of any distributed dataset on HDFS to a SQL engine
Apache HAWQ : An Introduction
HAWQ is an in-memory, distributed SQL query engine that runs as a Hadoop service. It provides two-way integration with HDFS, Hive, and HBase. HAWQ supports SQL transactions through commands like BEGIN, COMMIT, and ROLLBACK. External tables in HAWQ can be used to query data stored in HDFS files, Hive tables, and HBase tables.
HAWQ: a massively parallel processing SQL engine in hadoop
HAWQ, developed at Pivotal, is a massively parallel processing SQL engine sitting on top of HDFS. As a hybrid of MPP database and Hadoop, it inherits the merits from both parties. It adopts a layered architecture and relies on the distributed file system for data replication and fault tolerance. In addition, it is standard SQL compliant, and unlike other SQL engines on Hadoop, it is fully transactional. This paper presents the novel design of HAWQ, including query processing, the scalable software interconnect based on UDP protocol, transaction management, fault tolerance, read optimized storage, the extensible framework for supporting various popular Hadoop based data stores and formats, and various optimization choices we considered to enhance the query performance. The extensive performance study shows that HAWQ is about 40x faster than Stinger, which is reported 35x-45x faster than the original Hive.
Pivotal HAWQ - High Availability (2014)
The document discusses the new features in Pivotal HD 1.1, including improved high availability for HAWQ and Namenode, new UDF and diagnostic tools for HAWQ, upgraded Apache Hadoop components to version 2.0.5 and 2.0.6, improved Hive, HBase, and Oozie, Kerberos support for security, and new tools like the Unified Storage Service, Data Loader, and Command Center for easier administration.
Recommended
PXF BDAM 2016
PXF is a unified access framework that provides a uniform SQL interface for heterogeneous data sources on HDFS. It exploits parallelism to efficiently access data across various storage formats and data sources. PXF uses a pluggable architecture with built-in connectors that allow it to access data in HDFS files, Hive tables, HBase tables, and other data sources. It provides a common developer view and allows writing queries against external data using various profile definitions and plugins.
Managing Apache HAWQ with Apache AMBARI
This document summarizes a presentation about managing Apache HAWQ, an open source massively parallel processing (MPP) database, using Apache Ambari. It discusses how Ambari integrates with HAWQ for installation, configuration, topology recommendations, high availability, alerts and more. Challenges in the integration are addressed as HAWQ is not part of the Hortonworks Data Platform stack. The presentation recommends future work for Ambari like supporting automated HAWQ upgrades and enabling dynamic configuration reloads without requiring a service restart.
Apache Zeppelin Meetup Christian Tzolov 1/21/16
Zeppelin Interpreters
PSQL (to became JDBC in 0.6.x)
Geode
SpringXD
Apache Ambari
Zeppelin Service
Geode, HAWQ and Spring XD services
Webpage Embedder View
Hawq Hcatalog Integration
1. HCatalog is a table and storage management layer for Hadoop that provides a relational view of data in HDFS and abstracts data formats and locations from users.
2. Previously, HAWQ accessed Hive tables through PXF using external tables, but this required specifying the schema, location, and format which was error prone and wouldn't detect metadata changes.
3. The new integration retrieves metadata from HCatalog and parses it into in-memory catalog tables to provide dynamic access to Hive tables from HAWQ without needing to specify schemas.
PXF HAWQ Unmanaged Data
PXF is an extension framework that facilitates parallel access of any distributed dataset on HDFS to a SQL engine
Apache HAWQ : An Introduction
HAWQ is an in-memory, distributed SQL query engine that runs as a Hadoop service. It provides two-way integration with HDFS, Hive, and HBase. HAWQ supports SQL transactions through commands like BEGIN, COMMIT, and ROLLBACK. External tables in HAWQ can be used to query data stored in HDFS files, Hive tables, and HBase tables.
HAWQ: a massively parallel processing SQL engine in hadoop
HAWQ, developed at Pivotal, is a massively parallel processing SQL engine sitting on top of HDFS. As a hybrid of MPP database and Hadoop, it inherits the merits from both parties. It adopts a layered architecture and relies on the distributed file system for data replication and fault tolerance. In addition, it is standard SQL compliant, and unlike other SQL engines on Hadoop, it is fully transactional. This paper presents the novel design of HAWQ, including query processing, the scalable software interconnect based on UDP protocol, transaction management, fault tolerance, read optimized storage, the extensible framework for supporting various popular Hadoop based data stores and formats, and various optimization choices we considered to enhance the query performance. The extensive performance study shows that HAWQ is about 40x faster than Stinger, which is reported 35x-45x faster than the original Hive.
Pivotal HAWQ - High Availability (2014)
The document discusses the new features in Pivotal HD 1.1, including improved high availability for HAWQ and Namenode, new UDF and diagnostic tools for HAWQ, upgraded Apache Hadoop components to version 2.0.5 and 2.0.6, improved Hive, HBase, and Oozie, Kerberos support for security, and new tools like the Unified Storage Service, Data Loader, and Command Center for easier administration.
Pivotal Strata NYC 2015 Apache HAWQ Launch
Pivotal is a trusted partner for IT innovation and transformation. From the technology, to the people, to the way people interact with technology, Pivotal is transforming how the world builds software.
At Strata NYC 2015, Pivotal, announced it will Supercharge the Hadoop Ecosystem by contributing the HAWQ advanced SQL on Hadoop analytics and MADlib machine learning technologies to The Apache Software Foundation.
Build & test Apache Hawq
Requirement to build hawq
Setup build & test environment
Build and Test
Do everything with Docker
Demo & QA
Apache HAWQ and Apache MADlib: Journey to Apache
The document summarizes the journey of HAWQ and MADlib from being proprietary Pivotal technologies to becoming Apache open source projects. It provides an overview of HAWQ, including its key features like SQL compliance, performance advantages over other SQL-on-Hadoop systems, and flexible deployment options. It also summarizes MADlib, describing its machine learning functions and advantages of scalable in-database machine learning. Both projects are now available on open source platforms like Hadoop and aim to advance SQL and machine learning on big data through open collaboration.
Massively Parallel Processing with Procedural Python - Pivotal HAWQ
The document discusses massively parallel processing using procedural Python. It describes EMC Corporation and its subsidiaries which provide data storage, virtualization, security, and other software solutions. It also discusses Pivotal's open source contributions and the architecture of its HAWQ database which allows Python user-defined functions to perform parallel operations across clusters.
Phd tutorial hawq_v0.1
The document describes creating and loading sample data into HAWQ internal tables. First, the retail_demo schema and tables are dropped and recreated. Then, sample data files stored in HDFS are copied into the corresponding HAWQ tables using the COPY command in psql. The load is verified by running a script to check the row counts in each table. This demonstrates how to define and populate tables within HAWQ using sample data for testing and analysis.
SQL On Hadoop
This document provides an agenda and overview for a presentation on SQL on Hadoop. The presentation will cover various SQL on Hadoop technologies including Hive, HAWQ, Impala, SparkSQL, HBase with Phoenix, and Drill. It will also include an introduction, surveys to collect information from attendees, and discussions on networking and food. The hosts will provide background on their experience with big data and Hadoop.
Pivotal HAWQ 소개
HAWQ is an enterprise platform that provides the fewest barriers, lowest risk, and fastest way to perform big data analytics on Hadoop. It combines SQL with Hadoop by providing ANSI SQL capabilities on Hadoop for high performance analytics. HAWQ stores all data directly on HDFS and runs on various Hadoop distributions like Pivotal HD, HDP and IBM BigInsights.
Pivotal HAWQ and Hortonworks Data Platform: Modern Data Architecture for IT T...
Pivotal HAWQ, one of the world’s most advanced enterprise SQL on Hadoop technology, coupled with the Hortonworks Data Platform, the only 100% open source Apache Hadoop data platform, can turbocharge your analytic efforts. The slides from this technical webinar present a deep dive on this powerful modern data architecture for analytics and data science.
Learn more here: http://pivotal.io/big-data/pivotal-hawq
Pivotal hawq internals
The presentation delivered during Hadoop Kitchen in Moscow on 27.09.2014. It describes the technology that lies beneath Pivotal HAWQ technology
Webinar turbo charging_data_science_hawq_on_hdp_final
This document discusses how Hortonworks Data Platform (HDP) can enable enterprises to build a modern data architecture centered around Hadoop. It describes how HDP provides a centralized platform for managing all types of data at scale using technologies like YARN. Case studies are presented showing how companies have used HDP to optimize costs, develop new analytics applications, and work towards creating a unified "data lake". The document outlines the key components of HDP including its support for any application, any data, and deployment anywhere. It also highlights how partners extend HDP's capabilities and how Hortonworks provides enterprise-grade support.
How to manage Hortonworks HDB Resources with YARN
Part three in a five-part series, this webcast will be a demonstration of the integration of Hortonworks HDB and Apache Hadoop YARN.
Greenplum Architecture
This is the presentation I delivered on Hadoop User Group Ireland meetup in Dublin on Nov 28 2015. It covers at glance the architecture of GPDB and most important its features. Sorry for the colors - Slideshare is crappy with PDFs
Apache HAWQ Architecture
I gave this talk on the Highload++ conference 2015 in Moscow. Slides have been translated into English. They cover the Apache HAWQ components, its architecture, query processing logic, and also competitive information
MPP vs Hadoop
This is the presentation I made on the Hadoop User Group Ireland meetup in Dublin. It covers the main ideas of both MPP, Hadoop and the distributed systems in general, and also how to chose the best option for you
How to Use Apache Zeppelin with HWX HDB
Part five in a five-part series, this webcast will be a demonstration of the integration of Apache Zeppelin and Pivotal HDB. Apache Zeppelin is a web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more. This webinar will demonstrate the configuration of the psql interpreter and the basic operations of Apache Zeppelin when used in conjunction with Hortonworks HDB.
Data Exploration with Apache Drill: Day 1
Study after study shows that data scientists and analysts spend between 50% and 90% of their time preparing their data for analysis. Using Drill, you can dramatically reduce the time it takes to go from raw data to insight. This course will show you how.
The course material for this presentation are available at https://github.com/cgivre/data-exploration-with-apache-drill
SQL and Machine Learning on Hadoop using HAWQ
It is true to the extent it is almost considered rhetorical to say
“Many Enterprises have adopted HDFS as the foundational layer for their Data Lakes. HDFS provides the flexibility to store any kind of data and more importantly it’s infinitely scaleable on commodity hardware.”
But the conundrum till date is the solution for a low latency query engine for HDFS.
At Pivotal, we cracked that problem and the answer is HAWQ, which we intend to open source this year. During this event, we will present and demo HAWQ’s Architecture, it’s powerful ANSI SQL features and it’s ability to transcend traditional BI in the form of in-database analytics (or machine learning).
More Related Content
Viewers also liked
Pivotal Strata NYC 2015 Apache HAWQ Launch
Pivotal is a trusted partner for IT innovation and transformation. From the technology, to the people, to the way people interact with technology, Pivotal is transforming how the world builds software.
At Strata NYC 2015, Pivotal, announced it will Supercharge the Hadoop Ecosystem by contributing the HAWQ advanced SQL on Hadoop analytics and MADlib machine learning technologies to The Apache Software Foundation.
Build & test Apache Hawq
Requirement to build hawq
Setup build & test environment
Build and Test
Do everything with Docker
Demo & QA
Apache HAWQ and Apache MADlib: Journey to Apache
The document summarizes the journey of HAWQ and MADlib from being proprietary Pivotal technologies to becoming Apache open source projects. It provides an overview of HAWQ, including its key features like SQL compliance, performance advantages over other SQL-on-Hadoop systems, and flexible deployment options. It also summarizes MADlib, describing its machine learning functions and advantages of scalable in-database machine learning. Both projects are now available on open source platforms like Hadoop and aim to advance SQL and machine learning on big data through open collaboration.
Massively Parallel Processing with Procedural Python - Pivotal HAWQ
The document discusses massively parallel processing using procedural Python. It describes EMC Corporation and its subsidiaries which provide data storage, virtualization, security, and other software solutions. It also discusses Pivotal's open source contributions and the architecture of its HAWQ database which allows Python user-defined functions to perform parallel operations across clusters.
Phd tutorial hawq_v0.1
The document describes creating and loading sample data into HAWQ internal tables. First, the retail_demo schema and tables are dropped and recreated. Then, sample data files stored in HDFS are copied into the corresponding HAWQ tables using the COPY command in psql. The load is verified by running a script to check the row counts in each table. This demonstrates how to define and populate tables within HAWQ using sample data for testing and analysis.
SQL On Hadoop
This document provides an agenda and overview for a presentation on SQL on Hadoop. The presentation will cover various SQL on Hadoop technologies including Hive, HAWQ, Impala, SparkSQL, HBase with Phoenix, and Drill. It will also include an introduction, surveys to collect information from attendees, and discussions on networking and food. The hosts will provide background on their experience with big data and Hadoop.
Pivotal HAWQ 소개
HAWQ is an enterprise platform that provides the fewest barriers, lowest risk, and fastest way to perform big data analytics on Hadoop. It combines SQL with Hadoop by providing ANSI SQL capabilities on Hadoop for high performance analytics. HAWQ stores all data directly on HDFS and runs on various Hadoop distributions like Pivotal HD, HDP and IBM BigInsights.
Pivotal HAWQ and Hortonworks Data Platform: Modern Data Architecture for IT T...
Pivotal HAWQ, one of the world’s most advanced enterprise SQL on Hadoop technology, coupled with the Hortonworks Data Platform, the only 100% open source Apache Hadoop data platform, can turbocharge your analytic efforts. The slides from this technical webinar present a deep dive on this powerful modern data architecture for analytics and data science.
Learn more here: http://pivotal.io/big-data/pivotal-hawq
Pivotal hawq internals
The presentation delivered during Hadoop Kitchen in Moscow on 27.09.2014. It describes the technology that lies beneath Pivotal HAWQ technology
Webinar turbo charging_data_science_hawq_on_hdp_final
This document discusses how Hortonworks Data Platform (HDP) can enable enterprises to build a modern data architecture centered around Hadoop. It describes how HDP provides a centralized platform for managing all types of data at scale using technologies like YARN. Case studies are presented showing how companies have used HDP to optimize costs, develop new analytics applications, and work towards creating a unified "data lake". The document outlines the key components of HDP including its support for any application, any data, and deployment anywhere. It also highlights how partners extend HDP's capabilities and how Hortonworks provides enterprise-grade support.
How to manage Hortonworks HDB Resources with YARN
Part three in a five-part series, this webcast will be a demonstration of the integration of Hortonworks HDB and Apache Hadoop YARN.
Greenplum Architecture
This is the presentation I delivered on Hadoop User Group Ireland meetup in Dublin on Nov 28 2015. It covers at glance the architecture of GPDB and most important its features. Sorry for the colors - Slideshare is crappy with PDFs
Apache HAWQ Architecture
I gave this talk on the Highload++ conference 2015 in Moscow. Slides have been translated into English. They cover the Apache HAWQ components, its architecture, query processing logic, and also competitive information
MPP vs Hadoop
This is the presentation I made on the Hadoop User Group Ireland meetup in Dublin. It covers the main ideas of both MPP, Hadoop and the distributed systems in general, and also how to chose the best option for you
How to Use Apache Zeppelin with HWX HDB
Part five in a five-part series, this webcast will be a demonstration of the integration of Apache Zeppelin and Pivotal HDB. Apache Zeppelin is a web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more. This webinar will demonstrate the configuration of the psql interpreter and the basic operations of Apache Zeppelin when used in conjunction with Hortonworks HDB.
Data Exploration with Apache Drill: Day 1
Study after study shows that data scientists and analysts spend between 50% and 90% of their time preparing their data for analysis. Using Drill, you can dramatically reduce the time it takes to go from raw data to insight. This course will show you how.
The course material for this presentation are available at https://github.com/cgivre/data-exploration-with-apache-drill
SQL and Machine Learning on Hadoop using HAWQ
It is true to the extent it is almost considered rhetorical to say
“Many Enterprises have adopted HDFS as the foundational layer for their Data Lakes. HDFS provides the flexibility to store any kind of data and more importantly it’s infinitely scaleable on commodity hardware.”
But the conundrum till date is the solution for a low latency query engine for HDFS.
At Pivotal, we cracked that problem and the answer is HAWQ, which we intend to open source this year. During this event, we will present and demo HAWQ’s Architecture, it’s powerful ANSI SQL features and it’s ability to transcend traditional BI in the form of in-database analytics (or machine learning).
Viewers also liked (17)
Massively Parallel Processing with Procedural Python - Pivotal HAWQ
Massively Parallel Processing with Procedural Python - Pivotal HAWQ
Pivotal HAWQ and Hortonworks Data Platform: Modern Data Architecture for IT T...
Pivotal HAWQ and Hortonworks Data Platform: Modern Data Architecture for IT T...
Webinar turbo charging_data_science_hawq_on_hdp_final
Webinar turbo charging_data_science_hawq_on_hdp_final
gsoc_mentor for Shivram Mani
- 1. Certificate of Appreciation presented to Shivram Mani Apache Software Foundation for serving as a mentor during Google Summer of Code 2016 April 22 — August 23, 2016 Jason Titus VP, Engineering