Big Data: Working with Big SQL data from Spark Cynthia Saracco
Follow this hands-on lab to discover how Spark programmers can work with data managed by Big SQL, IBM's SQL interface for Hadoop. Examples use Scala and the Spark shell in a BigInsights 4.3 technical preview 2 environment.
Using your DB2 SQL Skills with Hadoop and SparkCynthia Saracco
Learn about Big SQL, IBM's SQL interface for Apache Hadoop based on DB2's query engine. We'll walk through some code example and discuss Spark integration for JDBC data sources (DB2 and Big SQL) using examples from a hands-on lab. Explore benchmark results comparing Big SQL and Spark SQL at 100TB. This presentation was created for the DB2 LUW TRIDEX Users Group meeting in NYC in June 2017.
Big Data: Big SQL web tooling (Data Server Manager) self-study labCynthia Saracco
This hands-on lab introduces you to Data Server Manager, a Web tool for querying and monitoring your Big SQL database. Data Server Manager (DSM) and Big SQL support select Apache Hadoop platforms.
Big Data: SQL query federation for Hadoop and RDBMS dataCynthia Saracco
Explore query federation capabilities in IBM Big SQL, which enables programmers to transparently join Hadoop data with relational database management (RDBMS) data.
Big Data: Working with Big SQL data from Spark Cynthia Saracco
Follow this hands-on lab to discover how Spark programmers can work with data managed by Big SQL, IBM's SQL interface for Hadoop. Examples use Scala and the Spark shell in a BigInsights 4.3 technical preview 2 environment.
Using your DB2 SQL Skills with Hadoop and SparkCynthia Saracco
Learn about Big SQL, IBM's SQL interface for Apache Hadoop based on DB2's query engine. We'll walk through some code example and discuss Spark integration for JDBC data sources (DB2 and Big SQL) using examples from a hands-on lab. Explore benchmark results comparing Big SQL and Spark SQL at 100TB. This presentation was created for the DB2 LUW TRIDEX Users Group meeting in NYC in June 2017.
Big Data: Big SQL web tooling (Data Server Manager) self-study labCynthia Saracco
This hands-on lab introduces you to Data Server Manager, a Web tool for querying and monitoring your Big SQL database. Data Server Manager (DSM) and Big SQL support select Apache Hadoop platforms.
Big Data: SQL query federation for Hadoop and RDBMS dataCynthia Saracco
Explore query federation capabilities in IBM Big SQL, which enables programmers to transparently join Hadoop data with relational database management (RDBMS) data.
Big Data: InterConnect 2016 Session on Getting Started with Big Data AnalyticsCynthia Saracco
Learn how to get started with Big Data using a platform based on Apache Hadoop, Apache Spark, and IBM BigInsights technologies. The emphasis here is on free or low-cost options that require modest technical skills.
Big Data: Getting off to a fast start with Big SQL (World of Watson 2016 sess...Cynthia Saracco
Got Big Data? Then check out what Big SQL can do for you . . . . Learn how IBM's industry-standard SQL interface enables you to leverage your existing SQL skills to query, analyze, and manipulate data managed in an Apache Hadoop environment on cloud or on premise. This quick technical tour is filled with practical examples designed to get you started working with Big SQL in no time. Specifically, you'll learn how to create Big SQL tables over Hadoop data in HDFS, Hive, or HBase; populate Big SQL tables with data from HDFS, a remote file system, or a remote RDBMS; execute simple and complex Big SQL queries; work with non-traditional data formats and more. These charts are for session ALB-3663 at the IBM World of Watson 2016 conference.
Big Data: Querying complex JSON data with BigInsights and HadoopCynthia Saracco
Explore how you can query complex JSON data using Big SQL, Hive, and BigInsights, IBM's Hadoop-based platform. Collect sample data from The Weather Company's service on Bluemix (a cloud platform) and learn different approaches for modeling and analyzing the data in a Hadoop environment.
Big Data: Explore Hadoop and BigInsights self-study labCynthia Saracco
Want a quick tour of Apache Hadoop and InfoSphere BigInsights (IBM's Hadoop distribution)? Follow this self-study lab to get hands-on experience with HDFS, MapReduce jobs, BigSheets, Big SQL, and more. This lab was tested against the free BigInsights Quick Start Edition 3.0 VMware image.
Big Data: Using free Bluemix Analytics Exchange Data with Big SQL Cynthia Saracco
Explains how to access free public data sets from IBM Analytics Exchange on the Bluemix cloud environment, transfer the data to BigInsights (a Hadoop-based platform), layer a Big SQL schema over the data, and query the data.
Big Data: Get started with SQL on Hadoop self-study lab Cynthia Saracco
Learn how to use SQL on Hadoop to query and analyze Big Data following this hands-on lab guide. Links in the lab explain where you can download a free VMware image of InfoSphere BigInsights 3.0 (IBM's Hadoop distribution) and sample data required for the lab. This lab focuses on Big SQL 3.0 technology released in June 2014.
Conduct data discovery or rapid BI prototyping without becoming a Hadoop expert by analyzing big data with standard BI tools, including Cognos. View the webinar video recording and download this deck: http://www.senturus.com/resources/running-cognos-on-hadoop/.
See a cost effective, scalable solution that does not have the barriers to entry common with big data applications. The webinar explains: 1) use cases for Hadoop, 2) pros and cons of different visualization tools and their integration with Hadoop and 3) a demonstration of BigInsights, IBM’s solution.
Senturus, a business analytics consulting firm, has a resource library with hundreds of free recorded webinars, trainings, demos and unbiased product reviews. Take a look and share them with your colleagues and friends: http://www.senturus.com/resources/.
Hadoop-DS: Which SQL-on-Hadoop Rules the HerdIBM Analytics
Originally Published on Oct 27, 2014
An overview of IBM's audited Hadoop-DS comparing IBM Big SQL, Cloudera Impala and Hortonworks Hive for performance and SQL compatibility. For more information, visit: http://www-01.ibm.com/software/data/infosphere/hadoop/
Adding Value to HBase with IBM InfoSphere BigInsights and BigSQLPiotr Pruski
This is the extended deck I used for my presentation at the Information On Demand 2013 conference for Session Number 1687 - Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL.
This presentation covers accessing HBase using Big SQL. It starts by going over general HBase concepts, than delves into how Big SQL adds an SQL layer on top of HBase (via HBase storage handler), secondary index support, queries, etc.
Getting started with Hadoop on the Cloud with BluemixNicolas Morales
Silicon Valley Code Camp -- October 11, 2014.
Session: Getting started with Hadoop on the Cloud.
Hadoop and Cloud is an almost perfect marriage. Hadoop is a distributed computing framework that leverages a cluster built on commodity hardware. The Cloud simplifies provisioning of machines and software. Getting started with Hadoop on the Cloud makes it simple to provision your environment quickly and actually get started using Hadoop. IBM Bluemix has democratized Hadoop for the masses! This session will provide a brief introduction to what Hadoop is, how does cloud work and will then focus on how to get started via a series of demos. We will conclude with a discussion around the tutorials and public datasets - all of the tools needed to get you started quickly.
Learn more about BigInsights for Hadoop: https://developer.ibm.com/hadoop/
Apache Hive is a rapidly evolving project, many people are loved by the big data ecosystem. Hive continues to expand support for analytics, reporting, and bilateral queries, and the community is striving to improve support along with many other aspects and use cases. In this lecture, we introduce the latest and greatest features and optimization that appeared in this project last year. This includes benchmarks covering LLAP, Apache Druid's materialized views and integration, workload management, ACID improvements, using Hive in the cloud, and performance improvements. I will also tell you a little about what you can expect in the future.
Big Data: InterConnect 2016 Session on Getting Started with Big Data AnalyticsCynthia Saracco
Learn how to get started with Big Data using a platform based on Apache Hadoop, Apache Spark, and IBM BigInsights technologies. The emphasis here is on free or low-cost options that require modest technical skills.
Big Data: Getting off to a fast start with Big SQL (World of Watson 2016 sess...Cynthia Saracco
Got Big Data? Then check out what Big SQL can do for you . . . . Learn how IBM's industry-standard SQL interface enables you to leverage your existing SQL skills to query, analyze, and manipulate data managed in an Apache Hadoop environment on cloud or on premise. This quick technical tour is filled with practical examples designed to get you started working with Big SQL in no time. Specifically, you'll learn how to create Big SQL tables over Hadoop data in HDFS, Hive, or HBase; populate Big SQL tables with data from HDFS, a remote file system, or a remote RDBMS; execute simple and complex Big SQL queries; work with non-traditional data formats and more. These charts are for session ALB-3663 at the IBM World of Watson 2016 conference.
Big Data: Querying complex JSON data with BigInsights and HadoopCynthia Saracco
Explore how you can query complex JSON data using Big SQL, Hive, and BigInsights, IBM's Hadoop-based platform. Collect sample data from The Weather Company's service on Bluemix (a cloud platform) and learn different approaches for modeling and analyzing the data in a Hadoop environment.
Big Data: Explore Hadoop and BigInsights self-study labCynthia Saracco
Want a quick tour of Apache Hadoop and InfoSphere BigInsights (IBM's Hadoop distribution)? Follow this self-study lab to get hands-on experience with HDFS, MapReduce jobs, BigSheets, Big SQL, and more. This lab was tested against the free BigInsights Quick Start Edition 3.0 VMware image.
Big Data: Using free Bluemix Analytics Exchange Data with Big SQL Cynthia Saracco
Explains how to access free public data sets from IBM Analytics Exchange on the Bluemix cloud environment, transfer the data to BigInsights (a Hadoop-based platform), layer a Big SQL schema over the data, and query the data.
Big Data: Get started with SQL on Hadoop self-study lab Cynthia Saracco
Learn how to use SQL on Hadoop to query and analyze Big Data following this hands-on lab guide. Links in the lab explain where you can download a free VMware image of InfoSphere BigInsights 3.0 (IBM's Hadoop distribution) and sample data required for the lab. This lab focuses on Big SQL 3.0 technology released in June 2014.
Conduct data discovery or rapid BI prototyping without becoming a Hadoop expert by analyzing big data with standard BI tools, including Cognos. View the webinar video recording and download this deck: http://www.senturus.com/resources/running-cognos-on-hadoop/.
See a cost effective, scalable solution that does not have the barriers to entry common with big data applications. The webinar explains: 1) use cases for Hadoop, 2) pros and cons of different visualization tools and their integration with Hadoop and 3) a demonstration of BigInsights, IBM’s solution.
Senturus, a business analytics consulting firm, has a resource library with hundreds of free recorded webinars, trainings, demos and unbiased product reviews. Take a look and share them with your colleagues and friends: http://www.senturus.com/resources/.
Hadoop-DS: Which SQL-on-Hadoop Rules the HerdIBM Analytics
Originally Published on Oct 27, 2014
An overview of IBM's audited Hadoop-DS comparing IBM Big SQL, Cloudera Impala and Hortonworks Hive for performance and SQL compatibility. For more information, visit: http://www-01.ibm.com/software/data/infosphere/hadoop/
Adding Value to HBase with IBM InfoSphere BigInsights and BigSQLPiotr Pruski
This is the extended deck I used for my presentation at the Information On Demand 2013 conference for Session Number 1687 - Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL.
This presentation covers accessing HBase using Big SQL. It starts by going over general HBase concepts, than delves into how Big SQL adds an SQL layer on top of HBase (via HBase storage handler), secondary index support, queries, etc.
Getting started with Hadoop on the Cloud with BluemixNicolas Morales
Silicon Valley Code Camp -- October 11, 2014.
Session: Getting started with Hadoop on the Cloud.
Hadoop and Cloud is an almost perfect marriage. Hadoop is a distributed computing framework that leverages a cluster built on commodity hardware. The Cloud simplifies provisioning of machines and software. Getting started with Hadoop on the Cloud makes it simple to provision your environment quickly and actually get started using Hadoop. IBM Bluemix has democratized Hadoop for the masses! This session will provide a brief introduction to what Hadoop is, how does cloud work and will then focus on how to get started via a series of demos. We will conclude with a discussion around the tutorials and public datasets - all of the tools needed to get you started quickly.
Learn more about BigInsights for Hadoop: https://developer.ibm.com/hadoop/
Apache Hive is a rapidly evolving project, many people are loved by the big data ecosystem. Hive continues to expand support for analytics, reporting, and bilateral queries, and the community is striving to improve support along with many other aspects and use cases. In this lecture, we introduce the latest and greatest features and optimization that appeared in this project last year. This includes benchmarks covering LLAP, Apache Druid's materialized views and integration, workload management, ACID improvements, using Hive in the cloud, and performance improvements. I will also tell you a little about what you can expect in the future.
Apache Hive is a rapidly evolving project, many people are loved by the big data ecosystem. Hive continues to expand support for analytics, reporting, and bilateral queries, and the community is striving to improve support along with many other aspects and use cases. In this lecture, we introduce the latest and greatest features and optimization that appeared in this project last year. This includes benchmarks covering LLAP, Apache Druid's materialized views and integration, workload management, ACID improvements, using Hive in the cloud, and performance improvements. I will also tell you a little about what you can expect in the future.
Overview of the architecture, and benefits of Dell HPC Storage with Intel EE Lustre in High Performance Computing and Big Science workloads.
Presented by Andrew Underwood at the Melbourne Big Data User Group - January 2016.
Lustre is a trademark of Seagate Technology.
Hadoop Demystified + MapReduce (Java and C#), Pig, and Hive DemosLester Martin
A walk-thru of core Hadoop, the ecosystem tools, and Hortonworks Data Platform (HDP) followed by code examples in MapReduce (Java and C#), Pig, and Hive.
Presented at the Atlanta .NET User Group meeting in July 2014.
Fundamentals of Big Data, Hadoop project design and case study or Use case
General planning consideration and most necessaries in Hadoop ecosystem and Hadoop projects
This will provide the basis for choosing the right Hadoop implementation, Hadoop technologies integration, adoption and creating an infrastructure.
Building applications using Apache Hadoop with a use-case of WI-FI log analysis has real life example.
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016MLconf
Big Data Processing Above and Beyond Hadoop: Data-intensive computing represents a new computing paradigm to address Big Data processing requirements using high-performance architectures supporting scalable parallel processing to allow government, commercial organizations, and research environments to process massive amounts of data and implement new applications previously thought to be impractical or infeasible. The fundamental challenges of data-intensive computing are managing and processing exponentially growing data volumes, significantly reducing associated data analysis cycles to support practical, timely applications, and developing new algorithms which can scale to search and process massive amounts of data. The open source HPCC (High-Performance Computing Cluster) Systems platform offers a unified approach to Big Data processing requirements: (1) a scalable, integrated computer systems hardware and software architecture designed for parallel processing of data-intensive computing applications, and (2) a new programming paradigm in the form of a high-level, declarative, data-centric programming language designed specifically for big data processing. This presentation explores the challenges of data-intensive computing from a programming perspective, and describes the ECL programming language and the HPCC architecture designed for data-intensive computing applications. HPCC is an alternative to the Hadoop platform, and ECL is compared to Pig Latin, a high-level language developed for the Hadoop MapReduce architecture.
Speed Up Your Queries with Hive LLAP Engine on Hadoop or in the Cloudgluent.
Hive was the first popular SQL layer built on Hadoop and has long been known as a heavyweight SQL engine suitable mainly for long-running batch jobs. This has greatly changed since Hive was announced to the world over 8 years ago. Hortonworks and the open source community have evolved Apache Hive into a fast, dynamic SQL on Hadoop engine capable of running highly concurrent query workloads over large datasets with sub-second response time.
The latest Hortonworks and Azure HDInsight platform versions fully support Hive with LLAP execution engine for production use. In this webinar, we will go through the architecture of Hive + LLAP engine and explain how it differs from previous Hive versions. We will then dive deeper and show how features like query vectorization and LLAP columnar caching bring further automatic performance improvements.
In the end, we will show how Gluent brings these new performance benefits to traditional enterprise database platforms via transparent data virtualization, allowing even your largest databases to benefit from all this without changing any application code. Join this webinar to learn about significant improvements in modern Hive architecture and how Gluent and Hive LLAP on Hortonworks or Azure HDInsight platforms can accelerate cloud migrations and greatly improve hybrid query performance!
Similar to Big SQL 3.0: Datawarehouse-grade Performance on Hadoop - At last! (20)
Benchmarking SQL-on-Hadoop Systems: TPC or not TPC?Nicolas Morales
Abstract. Benchmarks are important tools to evaluate systems, as long as their results are transparent, reproducible and they are conducted with due diligence. Today, many SQL-on-Hadoop vendors use the data generators and the queries of existing TPC benchmarks, but fail to adhere to the rules, producing results that are not transparent. As the SQL-on-Hadoop movement continues to gain more traction, it is important to bring some order to this \wild west" of benchmarking. First, new rules and policies should be dened to satisfy the demands of the new generation SQL systems. The new benchmark evaluation schemes should be inexpensive, eective and open enough to embrace the variety of SQL-on-Hadoop systems and their corresponding vendors. Second, adhering to the new standards requires industry commitment and collaboration. In this paper, we discuss the problems we observe in the current practices of benchmarking, and present our proposal for bringing standardization in the SQL-on-Hadoop space.
InfoSphere BigInsights for Hadoop @ IBM Insight 2014Nicolas Morales
InfoSphere BigInsights for Hadoop
Visit the IM Demo Room to learn more about Hadoop, InfoSphere BigInsights, Big SQL and more.
For more information:
- Hadoop and Big SQL, visit ibm.com/hadoop
- BigInsights Developer Community: https://developer.ibm.com/hadoop/
- IBM Insight 2014, visit ibm.com/software/events/insight
IBM Big SQL @ Insight 2014
Visit IBM Big SQL in the Information Management Demo room @ pedestal HD-01.
For more information:
- IBM Big SQL technology preview, visit http://ibm.biz/bigsqlpreview
- Hadoop and Big SQL, visit ibm.com/hadoop
- BigInsights Developer Community: https://developer.ibm.com/hadoop/
- IBM Insight 2014, visit ibm.com/software/events/insight
Challenges of Building a First Class SQL-on-Hadoop EngineNicolas Morales
Challenges of Building a First Class SQL-on-Hadoop Engine:
Why and what is Big SQL 3.0?
Overview of the challenges
How we solved (some of) them
Architecture and interaction with Hadoop
Query rewrite
Query optimization
Future challenges
BigInsights and Text Analytics.
As enterprises seek to gain operational efficiencies and competitive advantage through greater use of analytics, much of the new information they need to analyze is found in text documents and, increasingly, in a wide variety of social media sites and portals. A critical step in gaining insights from this information is extracting core data from huge volumes of text. That data is then available for downstream analytic, mining and machine learning tools. AQL (Annotator Query Language) is a powerful declarative, rule-based language for the extraction of information from text documents.
Social Data Analytics using IBM Big Data TechnologiesNicolas Morales
Distilling Insights from Social Media Using Big Data Technologies
Have you ever wondered what your customers are saying about you in Social media, and the impact it might be having on your business? This session will focus on how BigInsights and Big Data technologies can be used to glean useful and actionable insights from social media data.
You'll see how data can be ingested and prepped and do text analytics on social data in real time. Using Hadoop, we'll show you how you can store and analyze your large volume of historical social media data and reference data. This talk and demo will provide an introduction to text analytics and how it is used within the IBM Big Data platform for a social media solution.
The value of the fast growing class of big data technologies is the ability to handle high velocity and volumes of data. However, a lack of robust security and auditing capabilities are holding organizations back from fully using the potential of these systems. Learn how you can use Big Data technologies to help you meet this compliance and data protection challenge head on so you can return to innovating for competitive advantage.
Using InfoSphere Guardium and BigInsights, we'll show you how you can meet your Hadoop security, compliance and audit requirements.
Gain New Insights by Analyzing Machine Logs using Machine Data Analytics and BigInsights.
Half of Fortune 500 companies experience more than 80 hours of system down time annually. Spread evenly over a year, that amounts to approximately 13 minutes every day. As a consumer, the thought of online bank operations being inaccessible so frequently is disturbing. As a business owner, when systems go down, all processes come to a stop. Work in progress is destroyed and failure to meet SLA’s and contractual obligations can result in expensive fees, adverse publicity, and loss of current and potential future customers. Ultimately the inability to provide a reliable and stable system results in loss of $$$’s. While the failure of these systems is inevitable, the ability to timely predict failures and intercept them before they occur is now a requirement.
A possible solution to the problem can be found is in the huge volumes of diagnostic big data generated at hardware, firmware, middleware, application, storage and management layers indicating failures or errors. Machine analysis and understanding of this data is becoming an important part of debugging, performance analysis, root cause analysis and business analysis. In addition to preventing outages, machine data analysis can also provide insights for fraud detection, customer retention and other important use cases.
Utilocate offers a comprehensive solution for locate ticket management by automating and streamlining the entire process. By integrating with Geospatial Information Systems (GIS), it provides accurate mapping and visualization of utility locations, enhancing decision-making and reducing the risk of errors. The system's advanced data analytics tools help identify trends, predict potential issues, and optimize resource allocation, making the locate ticket management process smarter and more efficient. Additionally, automated ticket management ensures consistency and reduces human error, while real-time notifications keep all relevant personnel informed and ready to respond promptly.
The system's ability to streamline workflows and automate ticket routing significantly reduces the time taken to process each ticket, making the process faster and more efficient. Mobile access allows field technicians to update ticket information on the go, ensuring that the latest information is always available and accelerating the locate process. Overall, Utilocate not only enhances the efficiency and accuracy of locate ticket management but also improves safety by minimizing the risk of utility damage through precise and timely locates.
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Graspan: A Big Data System for Big Code AnalysisAftab Hussain
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeAftab Hussain
Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.
- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia
Understanding Nidhi Software Pricing: A Quick Guide 🌟
Choosing the right software is vital for Nidhi companies to streamline operations. Our latest presentation covers Nidhi software pricing, key factors, costs, and negotiation tips.
📊 What You’ll Learn:
Key factors influencing Nidhi software price
Understanding the true cost beyond the initial price
Tips for negotiating the best deal
Affordable and customizable pricing options with Vector Nidhi Software
🔗 Learn more at: www.vectornidhisoftware.com/software-for-nidhi-company/
#NidhiSoftwarePrice #NidhiSoftware #VectorNidhi
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Zoom is a comprehensive platform designed to connect individuals and teams efficiently. With its user-friendly interface and powerful features, Zoom has become a go-to solution for virtual communication and collaboration. It offers a range of tools, including virtual meetings, team chat, VoIP phone systems, online whiteboards, and AI companions, to streamline workflows and enhance productivity.
Mobile App Development Company In Noida | Drona InfotechDrona Infotech
Looking for a reliable mobile app development company in Noida? Look no further than Drona Infotech. We specialize in creating customized apps for your business needs.
Visit Us For : https://www.dronainfotech.com/mobile-application-development/
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Łukasz Chruściel
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Crescat
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
2. Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a
commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be incorporated
into any contract. The development, release, and timing of any future features or
functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM
benchmarks in a controlled environment. The actual throughput or performance
that any user will experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in the user’s job stream,
the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results
similar to those stated here.
3. Agenda
What is this Hadoop thing?
Why SQL on Hadoop?
What is Hive?
SQL-on-Hadoop landscape
Big SQL 3.0
• What is it?
• SQL capabilities
• Architecture
• Application portability and
integration
• Enterprise capabilities
• Performance
Conclusion
2
4. What is Hadoop?
Hadoop is not a piece of software, you can't install "hadoop"
It is an ecosystem of software that work together
• Hadoop Core (API's)
• HDFS (File system)
• MapReduce (Data processing framework)
• Hive (SQL access)
• HBase (NoSQL database)
• Sqoop (Data movement)
• Oozie (Job workflow)
• …. There are is a LOT of "Hadoop" software
However, there is one common component they all build on: HDFS…
• *Not exactly 100% true but 99.999% true
5. The Hadoop Filesystem (HDFS)
Driving principals
• Files are stored across the entire cluster
• Programs are brought to the data, not the data to the program
Distributed file system (DFS) stores blocks across the whole cluster
• Blocks of a single file are distributed across the cluster
• A given block is typically replicated as well for resiliency
• Just like a regular file system, the contents of a file is up to the application
10110100
10100100
11100111
11100101
00111010
01010010
11001001
01010011
00010100
10111010
11101011
11011011
01010110
10010101
00101010
10101110
01001101
01110100
Logical File
1
2
3
4
Blocks
1
Cluster
1
1
2
2
2
3
3
34
4
4
6. Data processing on Hadoop
Hadoop (HDFS) doesn't dictate file content/structure
• It is just a filesystem!
• It looks and smells almost exactly like the filesystem on your laptop
• Except, you can ask it "where does each block of my file live?"
The entire Hadoop ecosystem is built around that question!
• Parallelize work by sending your programs to the data
• Each copy processes a given block of the file
• Other nodes may be chosen to aggregate together computed results
10110100
10100100
11100111
11100101
00111010
01010010
11001001
01010011
00010100
10111010
11101011
11011011
01010110
10010101
1
2
3
Logical File
Splits
1
Cluster
23
App
(Read)
App
(Read)
App
(Read)
App
(Compute)
Result
7. Why SQL for Hadoop?
Hadoop is designed for any data
• Doesn't impose any structure
• Extremely flexible
At lowest levels is API based
• Requires strong programming
expertise
• Steep learning curve
• Even simple operations can be
tedious
Yet many, if not most, use cases deal with structured data!
• e.g. aging old warehouse data into queriable archive
Why not use SQL in places its strengths shine?
• Familiar widely used syntax
• Separation of what you want vs. how to get it
• Robust ecosystem of tools
Pre-Processing Hub Query-able Archive Exploratory Analysis
Information
Integration
Data Warehouse
Streams
Real-time
processing
BigInsights
Landing zone
for all data
Data Warehouse
BigInsights Can combine
with
unstructured
information
Data Warehouse
1 2 3
8. Then along comes Hive
Hive was the first SQL interface for Hadoop data
• Defacto standard for SQL on Hadoop
• Ships with all major Hadoop distributions
SQL queries are executed using MapReduce (today)
Hive introduced several important concepts/components…
• Most of which are shared by all SQL-on-Hadoop solutions
reduce
dept 1
reduce
dept 2
reduce
dept 3
1011010
0101001
0011100
1111110
0101001
1010111
0111010
1
1 map
2 map
2
1 map
employees
1011010
0101001
0011110
0111011
1
depts
9. Tables in Hive
A table describes a mapping between columns and the structure and
location of files (generally)
• For common file formats, there is special syntax to make mapping easy
• But you can define totally new storage formats yourself
create table users
(
id int,
office_id int
)
row format delimited
fields terminated by '|'
stored as textfile
location '/warehouse/sales.db/users'
create table users
(
id int,
office_id int
)
row format serde 'org.apache.hive…LazySimpleSerde'
with serdeproperties ( 'field.delim' = '|' )
inputformat 'org.apache.hadoop.mapred.TextInputFormat'
outputformat 'org.apache.hadoop.mapred.TextOutputFormat'
10. Hive MetaStore
Hive maintains a centralized database of metadata
Table definitions
• Location (directory on HDFS)
• Column names and types
• Partition information
• Classes used to read/write the table
• Etc.
Security
• Groups, roles, permissions
11. SQL-on-Hadoop landscape
The SQL-on-Hadoop landscape changes constantly!
Being relatively new to the SQL game, they have all generally
meant compromising one or more of….
• Speed
• Robust SQL
• Enterprise features
• Interoperability with the Hadoop ecosystem
Big SQL 3.0 is based upon tried and true IBM relational
technology, addressing all of these areas
12. Big SQL 3.0 – At a glance
Available for POWER Linux (Redhat) and Intel x64 Linux (Redhat/SUSE)
11-Apr-2014
13. Open processing
Big SQL is about applying SQL to your existing data
• No propriety storage format
A "table" is simply a view on your Hadoop data
Table definitions shared with Hive
• The Hive Metastore catalogs table definitions
• Reading/writing data logic is shared
with Hive
• Definitions can be shared across the
Hadoop ecosystem
Sometimes SQL isn't the answer!
• Use the right tool for the right job
Hive
Hive
Metastore
Hadoop
Cluster
Pig
Hive APIs
Sqoop
Hive APIs
Big SQL
Hive APIs
14. SQL capabilities
Leverage IBM's rich SQL heritage, expertise, and technology
• SQL standards compliant query support
• SQL bodied functions and stored procedures
– Encapsulate your business logic and security at the server
• DB2 compatible SQL PL support
– Cursors
– Anonymous blocks (batches of statements)
– Flow of control (if/then/else, error handling, prepared statements, etc.)
The same SQL you use on your data warehouse should run with
few or no modifications
15. SQL capability highlights
Full support for subqueries
• In SELECT, FROM, WHERE and
HAVING clauses
• Correlated and uncorrelated
• Equality, non-equality subqueries
• EXISTS, NOT EXISTS, IN, ANY,
SOME, etc.
All standard join operations
• Standard and ANSI join syntax
• Inner, outer, and full outer joins
• Equality, non-equality, cross join support
• Multi-value join
• UNION, INTERSECT, EXCEPT
SELECT
s_name,
count(*) AS numwait
FROM
supplier,
lineitem l1,
orders,
nation
WHERE
s_suppkey = l1.l_suppkey
AND o_orderkey = l1.l_orderkey
AND o_orderstatus = 'F'
AND l1.l_receiptdate > l1.l_commitdate
AND EXISTS (
SELECT
*
FROM
lineitem l2
WHERE
l2.l_orderkey = l1.l_orderkey
AND l2.l_suppkey <> l1.l_suppkey
)
AND NOT EXISTS (
SELECT
*
FROM
lineitem l3
WHERE
l3.l_orderkey = l1.l_orderkey
AND l3.l_suppkey <> l1.l_suppkey
AND l3.l_receiptdate >
l3.l_commitdate
)
AND s_nationkey = n_nationkey
AND n_name = ':1'
GROUP BY
s_name
ORDER BY
numwait desc,
s_name;
SELECT
s_name,
count(*) AS numwait
FROM
supplier,
lineitem l1,
orders,
nation
WHERE
s_suppkey = l1.l_suppkey
AND o_orderkey = l1.l_orderkey
AND o_orderstatus = 'F'
AND l1.l_receiptdate > l1.l_commitdate
AND EXISTS (
SELECT
*
FROM
lineitem l2
WHERE
l2.l_orderkey = l1.l_orderkey
AND l2.l_suppkey <> l1.l_suppkey
)
AND NOT EXISTS (
SELECT
*
FROM
lineitem l3
WHERE
l3.l_orderkey = l1.l_orderkey
AND l3.l_suppkey <> l1.l_suppkey
AND l3.l_receiptdate >
l3.l_commitdate
)
AND s_nationkey = n_nationkey
AND n_name = ':1'
GROUP BY
s_name
ORDER BY
numwait desc,
s_name;
17. Architected for performance
Architected from the ground up for low latency and high throughput
MapReduce replaced with a modern MPP architecture
• Compiler and runtime are native code (not java)
• Big SQL worker daemons live directly on cluster
– Continuously running (no startup latency)
– Processing happens locally at the data
• Message passing allows data to flow directly
between nodes
Operations occur in memory with the ability
to spill to disk
• Supports aggregations and sorts larger than
available RAM
Head Node
Big SQL
Head Node
Hive
Metastore
Compute Node
Task
Tracker
Data
Node
Big
SQL
Compute Node
Task
Tracker
Data
Node
Big
SQL
Compute Node
Task
Tracker
Data
Node
Big
SQL
Compute Node
Task
Tracker
Data
Node
Big
SQL
HDFS/GPFS
18. Extreme parallelism
Massively parallel SQL engine that replaces MR
Shared-nothing architecture that eliminates scalability and networking
issues
Engine pushes processing out to data nodes to maximize data locality.
Hadoop data accessed natively via C++ and Java readers and writers.
Inter- and intra-node parallelism where work is distributed to multiple
worker nodes and on each node multiple worker threads collaborate on
the I/O and data processing (scale out horizontally and scale up
vertically)
Intelligent data partition elimination based on SQL predicates
Fault tolerance through active health monitoring and management of
parallel data and worker nodes
20. Big SQL 3.0 – Architecture (cont.)
Big SQL's runtime execution engine is all native code
For common table formats a native I/O engine is utilized
• e.g. delimited, RC, SEQ, Parquet, …
For all others, a java I/O engine is used
• Maximizes compatibility with existing tables
• Allows for custom file formats and SerDe's
All Big SQL built-in functions are native code
Customer built UDx's can be developed in C++ or Java
Maximize performance without sacrificing
extensibility
Mgmt Node
Big SQL
Compute Node
Task
Tracker
Data
Node
Big
SQL
Big SQL Worker
Native I/O
Engine
Java I/O
Engine
SerDe I/O Fmt
Runtime
Java UDFs
Native UDFs
19
21. Resource management
Big SQL doesn't run in isolation
Nodes tend to be shared with a variety of Hadoop services
• Task tracker
• Data node
• HBase region servers
• MapReduce jobs
• etc.
Big SQL can be constrained to limit its footprint on the cluster
• % of CPU utilization
• % of memory utilization
Resources are automatically adjusted based upon workload
• Always fitting within constraints
• Self-tuning memory manager that re-distributes resources across
components dynamically
• default WLM concurrency control for heavy queries
Compute Node
Task
Tracker
Data
Node
Big
SQL
HBase
MR
Task
MR
Task
MR
Task
22. Performance
Query rewrites
• Exhaustive query rewrite capabilities
• Leverages additional metadata such as constraints and nullability
Optimization
• Statistics and heuristic driven query optimization
• Query optimizer based upon decades of IBM RDBMS experience
Tools and metrics
• Highly detailed explain plans and query diagnostic tools
• Extensive number of available performance metrics
SELECT ITEM_DESC, SUM(QUANTITY_SOLD),
AVG(PRICE), AVG(COST)
FROM PERIOD, DAILY_SALES, PRODUCT,
STORE
WHERE
PERIOD.PERKEY=DAILY_SALES.PERKEY AND
PRODUCT.PRODKEY=DAILY_SALES.PRODKE
Y AND
STORE.STOREKEY=DAILY_SALES.STOREKEY
AND
CALENDAR_DATE BETWEEN AND
'01/01/2012' AND '04/28/2012' AND
STORE_NUMBER='03' AND
CATEGORY=72
GROUP BY ITEM_DESC
Access plan generationQuery transformation
Dozens of query
transformations
Hundreds or thousands
of access plan options
Store
Product
Product Store
NLJOIN
Daily SalesNLJOIN
Period
NLJOIN
Product
NLJOIN
Daily Sales
NLJOIN
Period
NLJOIN
Store
HSJOIN
Daily Sales
HSJOIN
Period
HSJOIN
Product
StoreZZJOIN
Daily Sales
HSJOIN
Period
23. Application portability and integration
Big SQL 3.0 adopts IBM's standard Data Server Client Drivers
• Robust, standards compliant ODBC, JDBC, and .NET drivers
• Same driver used for DB2 LUW, DB2/z and Informix
• Expands support to numerous languages (Python, Ruby, Perl, etc.)
Putting the story together….
• Big SQL shares a common SQL dialect with DB2
• Big SQL shares the same client drivers with DB2
• Data warehouse augmentation just got significantly easier
Compatible
SQL
Compatible
SQL
Compatible
Drivers
Compatible
Drivers
Portable
Application
Portable
Application
24. Application portability and integration (cont.)
This compatibility extends beyond your own applications
Open integration across Business Analytic Tools
• IBM Optim Data Studio performance tool portfolio
• Superior enablement for IBM Software – e.g. Cognos
• Enhanced support by 3rd party software – e.g. Microstrategy
25. Query federation
Data never lives in isolation
• Either as a landing zone or a queryable archive it is desirable to
query data across Hadoop and active data warehouses
Big SQL provides the ability to query heterogeneous systems
• Join Hadoop to other relational databases
• Query optimizer understands capabilities of external system
– Including available statistics
• As much work as possible is pushed to each system to process
Head Node
Big SQL
Compute Node
Task
Tracker
Data
Node
Big
SQL
Compute Node
Task
Tracker
Data
Node
Big
SQL
Compute Node
Task
Tracker
Data
Node
Big
SQL
Compute Node
Task
Tracker
Data
Node
Big
SQL
26. Enterprise security
Users may be authenticated via
• Operating system
• Lightweight directory access protocol (LDAP)
• Kerberos
User authorization mechanisms include
• Full GRANT/REVOKE based security
• Group and role based hierarchical security
• Object level, column level, or row level (fine-grained) access controls
Auditing
• You may define audit policies and track user activity
Transport layer security (TLS)
• Protect integrity and confidentiality of data between the client and Big SQL
27. Monitoring
Comprehensive runtime monitoring infrastructure that helps
answer the question: what is going on in my system?
• SQL interfaces to the monitoring data via table functions
• Ability to drill down into more granular metrics for problem determination and/ or
detailed performance analysis
• Runtime statistics collected during the execution of the section for a (SQL) access
plan
• Support for event monitors to track specific types of operations and activities
• Protect against and discover unknown or unacceptable behaviors by monitoring
data access via Audit facility.
Reporting Level
(Example: Service Class)
Big SQL 3.0
Worker Threads
Connection
Control Blocks
Worker Threads Collect Locally
Push Up Data Incrementally
Extract Data Directly From
Reporting level
Monitor Query
28. 27
Comparing Big SQL and Hive 0.12 for Ad-Hoc Queries
Big SQL is upto 41x faster
than Hive 0.12
Big SQL is upto 41x faster
than Hive 0.12
*Based on IBM internal tests comparing IBM Infosphere Biginsights 3.0 Big SQL with Hive 0.12 executing the "1TB Classic
BI Workload" in a controlled laboratory environment. The 1TB Classic BI Workload is a workload derived from the TPC-H
Benchmark Standard, running at 1TB scale factor. It is materially equivalent with the exception that no update functions are
performed. TPC Benchmark and TPC-H are trademarks of the Transaction Processing Performance Council (TPC).
Configuration: Cluster of 9 System x3650HD servers, each with 64GB RAM and 9x2TB HDDs running Redhat Linux 6.3.
Results may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in
a production environment. Results as of April 22, 2014
29. Big SQL is 10x
faster than Hive 0.12
(total workload
elapsed time)
Big SQL is 10x
faster than Hive 0.12
(total workload
elapsed time)
28
Comparing Big SQL and Hive 0.12
for Decision Support Queries
* Based on IBM internal tests comparing IBM Infosphere Biginsights 3.0 Big SQL with Hive 0.12 executing the "1TB Modern BI
Workload" in a controlled laboratory environment. The 1TB Modern BI Workload is a workload derived from the TPC-DS Benchmark
Standard, running at 1TB scale factor. It is materially equivalent with the exception that no updates are performed, and only 43 out of
99 queries are executed. The test measured sequential query execution of all 43 queries for which Hive syntax was publically
available. TPC Benchmark and TPC-DS are trademarks of the Transaction Processing Performance Council (TPC).
Configuration: Cluster of 9 System x3650HD servers, each with 64GB RAM and 9x2TB HDDs running Redhat Linux 6.3. Results
may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in a production
environment. Results as of April 22, 2014
30. How many times faster is Big SQL than Hive 0.12?
* Based on IBM internal tests comparing IBM Infosphere Biginsights 3.0 Big SQL with Hive 0.12 executing the "1TB Modern BI
Workload" in a controlled laboratory environment. The 1TB Modern BI Workload is a workload derived from the TPC-DS Benchmark
Standard, running at 1TB scale factor. It is materially equivalent with the exception that no updats are performed, and only 43 out of
99 queries are executed. The test measured sequential query execution of all 43 queries for which Hive syntax was publically
available. TPC Benchmark and TPC-DS are trademarks of the Transaction Processing Performance Council (TPC).
Configuration: Cluster of 9 System x3650HD servers, each with 64GB RAM and 9x2TB HDDs running Redhat Linux 6.3. Results
may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in a production
environment. Results as of April 22, 2014
Max
Speedup
of 74x
Max
Speedup
of 74x
29
Queries sorted by speed up ratio (worst to best)
Avg
Speedup
of 20x
Avg
Speedup
of 20x
31. Power of Standard SQL
Everyone loves performance numbers, but that's not the whole story
• How much work do you have to do to achieve those numbers?
A portion of our internal performance numbers are based upon industry
standard benchmarks
Big SQL is capable of executing
• All 22 TPC-H queries without modification
• All 99 TPC-DS queries without modification
SELECT s_name, count(*) AS numwait
FROM supplier, lineitem l1, orders, nation
WHERE s_suppkey = l1.l_suppkey
AND o_orderkey = l1.l_orderkey
AND o_orderstatus = 'F'
AND l1.l_receiptdate > l1.l_commitdate
AND EXISTS (
SELECT *
FROM lineitem l2
WHERE l2.l_orderkey = l1.l_orderkey
AND l2.l_suppkey <> l1.l_suppkey)
AND NOT EXISTS (
SELECT *
FROM lineitem l3
WHERE l3.l_orderkey = l1.l_orderkey
AND l3.l_suppkey <> l1.l_suppkey
AND l3.l_receiptdate > l3.l_commitdate)
AND s_nationkey = n_nationkey
AND n_name = ':1'
GROUP BY s_name
ORDER BY numwait desc, s_name
SELECT s_name, count(*) AS numwait
FROM supplier, lineitem l1, orders, nation
WHERE s_suppkey = l1.l_suppkey
AND o_orderkey = l1.l_orderkey
AND o_orderstatus = 'F'
AND l1.l_receiptdate > l1.l_commitdate
AND EXISTS (
SELECT *
FROM lineitem l2
WHERE l2.l_orderkey = l1.l_orderkey
AND l2.l_suppkey <> l1.l_suppkey)
AND NOT EXISTS (
SELECT *
FROM lineitem l3
WHERE l3.l_orderkey = l1.l_orderkey
AND l3.l_suppkey <> l1.l_suppkey
AND l3.l_receiptdate > l3.l_commitdate)
AND s_nationkey = n_nationkey
AND n_name = ':1'
GROUP BY s_name
ORDER BY numwait desc, s_name
JOIN
(SELECT s_name, l_orderkey, l_suppkey
FROM orders o
JOIN
(SELECT s_name, l_orderkey, l_suppkey
FROM nation n
JOIN supplier s
ON s.s_nationkey = n.n_nationkey
AND n.n_name = 'INDONESIA'
JOIN lineitem l
ON s.s_suppkey = l.l_suppkey
WHERE l.l_receiptdate > l.l_commitdate) l1
ON o.o_orderkey = l1.l_orderkey
AND o.o_orderstatus = 'F') l2
ON l2.l_orderkey = t1.l_orderkey) a
WHERE (count_suppkey > 1) or ((count_suppkey=1)
AND (l_suppkey <> max_suppkey))) l3
ON l3.l_orderkey = t2.l_orderkey) b
WHERE (count_suppkey is null)
OR ((count_suppkey=1) AND (l_suppkey = max_suppkey))) c
GROUP BY s_name
ORDER BY numwait DESC, s_name
JOIN
(SELECT s_name, l_orderkey, l_suppkey
FROM orders o
JOIN
(SELECT s_name, l_orderkey, l_suppkey
FROM nation n
JOIN supplier s
ON s.s_nationkey = n.n_nationkey
AND n.n_name = 'INDONESIA'
JOIN lineitem l
ON s.s_suppkey = l.l_suppkey
WHERE l.l_receiptdate > l.l_commitdate) l1
ON o.o_orderkey = l1.l_orderkey
AND o.o_orderstatus = 'F') l2
ON l2.l_orderkey = t1.l_orderkey) a
WHERE (count_suppkey > 1) or ((count_suppkey=1)
AND (l_suppkey <> max_suppkey))) l3
ON l3.l_orderkey = t2.l_orderkey) b
WHERE (count_suppkey is null)
OR ((count_suppkey=1) AND (l_suppkey = max_suppkey))) c
GROUP BY s_name
ORDER BY numwait DESC, s_name
SELECT s_name, count(1) AS numwait
FROM
(SELECT s_name FROM
(SELECT s_name, t2.l_orderkey, l_suppkey,
count_suppkey, max_suppkey
FROM
(SELECT l_orderkey,
count(distinct l_suppkey) as count_suppkey,
max(l_suppkey) as max_suppkey
FROM lineitem
WHERE l_receiptdate > l_commitdate
GROUP BY l_orderkey) t2
RIGHT OUTER JOIN
(SELECT s_name, l_orderkey, l_suppkey
FROM
(SELECT s_name, t1.l_orderkey, l_suppkey,
count_suppkey, max_suppkey
FROM
(SELECT l_orderkey,
count(distinct l_suppkey) as count_suppkey,
max(l_suppkey) as max_suppkey
FROM lineitem
GROUP BY l_orderkey) t1
SELECT s_name, count(1) AS numwait
FROM
(SELECT s_name FROM
(SELECT s_name, t2.l_orderkey, l_suppkey,
count_suppkey, max_suppkey
FROM
(SELECT l_orderkey,
count(distinct l_suppkey) as count_suppkey,
max(l_suppkey) as max_suppkey
FROM lineitem
WHERE l_receiptdate > l_commitdate
GROUP BY l_orderkey) t2
RIGHT OUTER JOIN
(SELECT s_name, l_orderkey, l_suppkey
FROM
(SELECT s_name, t1.l_orderkey, l_suppkey,
count_suppkey, max_suppkey
FROM
(SELECT l_orderkey,
count(distinct l_suppkey) as count_suppkey,
max(l_suppkey) as max_suppkey
FROM lineitem
GROUP BY l_orderkey) t1
Original Query
Re-written for Hive
32. Conclusion
Today, it seems, performance numbers are the name of the game
But in reality there is so much more…
• How rich is the SQL?
• How difficult is it to (re-)use your existing SQL?
• How secure is your data?
• Is your data still open for other uses on Hadoop?
• Can your queries span your enterprise?
• Can other Hadoop workloads co-exist in harmony?
• …
With Big SQL 3.0 performance doesn't mean compromise
34. We Value Your Feedback
Don’t forget to submit your Impact session and speaker
feedback! Your feedback is very important to us – we use it to
continually improve the conference.
Use the Conference Mobile App or the online Agenda Builder to
quickly submit your survey
• Navigate to “Surveys” to see a view of surveys for sessions
you’ve attended
33