Presto Bangalore Meetup1 Event Listeners@qubole

•

1 like•197 views

This document discusses event listeners in Presto, which allow notifications at trigger points like query creation and completion. It describes setting up an event listener to analyze Presto query performance using a TPC DS dataset on an AWS cluster. Potential use cases include resource accounting, configuring alarms, and cluster configuration analysis. Limitations are that events only occur at boundaries and don't provide an overall system state view.

Analysing presto setup using
Event Listener
Oct 13th, 2018
Ankit Dixit

00Copyright 2017 © Qubole
Goal
● What is an event listener
● How to leverage it for analysis

00Copyright 2017 © Qubole
Analysis time, Source

00Copyright 2017 © Qubole
Cluster presto usage, Query distribution

00Copyright 2017 © Qubole
What is an event listener
● Mechanism to get notified at trigger points: events
● Events:
● Query creation
● Query completion
● Split completion
● Query/Task stats, session info

00Copyright 2017 © Qubole
Code
Event Listener github

00Copyright 2017 © Qubole
Setup
● TPC DS scale 1000
● Master r3.2xlarge
● 3 Slave r3.4xlarge
● 19 perf queries ran multiple times

00Copyright 2017 © Qubole
Use Cases
● Resource Accounting
● Configure alarms on system state
○ Query completion rate
○ Queue size
○ Average data read
● Cluster configuration analysis
● Post execution hooks

00Copyright 2017 © Qubole
Limitations
● Triggers only at event boundaries
● Does not provide with a overall system state
● Not suitable for proactive analysis

MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business. This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented. In addition, we'll discuss future plans and opportunities and offer ample Q&A time with the engineers on the project.

MongoDB .local Houston 2019: MongoDB Atlas Data Lake Technical Deep Dive

MongoDB

Thomas likes to push the limits of what relational open-source databases are capable of. Having worked at the intersection of business and technology in venture financed start-ups in Norway, Germany and the US, Thomas enjoys bringing customer requirements and great technologies together. With Intel’s push of the Field Programmable Gate Array as an accelerator into the server market, Swarm64 is joining in and teaching established databases new tricks. Thomas has a background in managing IT projects for Lufthansa Systems. After his MBA from Lancaster, UK, he journeyed into the start-up world and held various C-level roles before founding Swarm64.

IPTC News Exchange Working Group 2013 Autumn Meeting

Stuart Myles

Iot data aggrigation

dokechin

The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka

All Things Open

Botconf 2013 - DNS-based Botnet C2 Server Detection

SensePost

Key considerations in productionizing streaming applications

KafkaZone

Ridwan Fadjar Septian PyCon ID 2021 Regular Talk - django application monitor...

Ridwan Fadjar

Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...

Severalnines

What if … - Traditional, labour-intensive backup and archive practices for your MySQL, MariaDB, MongoDB and PostgreSQL databases were a thing of the past? - You could have one backup management solution for all your business data? - You could ensure integrity of all your backups? - You could leverage the competitive pricing and almost limitless capacity of cloud-based backup while meeting cost, manageability, and compliance requirements from the business. Welcome to our webinar on Backup Management with ClusterControl. ClusterControl’s centralized backup management for open source databases provides you with hot backups of large datasets, point in time recovery in a couple of clicks, at-rest and in-transit data encryption, data integrity via automatic restore verification, cloud backups (AWS, Google and Azure) for Disaster Recovery, retention policies to ensure compliance, and automated alerts and reporting. Whether you are looking at rebuilding your existing backup infrastructure, or updating it, this webinar is for you! AGENDA - Backup and recovery management of local or remote databases - Logical or physical backups - Full or Incremental backups - Position or time-based Point in Time Recovery (for MySQL and PostgreSQL) - Upload to the cloud (Amazon S3, Google Cloud Storage, Azure Storage) - Encryption of backup data - Compression of backup data - One centralized backup system for your open source databases (Demo) - Schedule, manage and operate backups - Define backup policies, retention, history - Validation - Automatic restore verification - Backup reporting SPEAKER Bartlomiej Oles, Senior Support Engineer at Severalnines, is a MySQL and Oracle DBA, with over 15 years experience in managing highly available production systems at IBM, Nordea Bank, Acxiom, Lufthansa, and other Fortune 500 companies. In the past five years, his focus has been on building and applying automation tools to manage multi-datacenter database environments.

New Relic

Gene Chuang

Big Query Basics

Ido Green

Cost Effective Presto on AWS with Spot Nodes - Strata SF 2019

Shubham Tagra

OpenTelemetry For Operators

Kevin Brockhoff

Presto Summit 2018 - 10 - Qubole

kbajda

Security Monitoring for big Infrastructures without a Million Dollar budget

Juan Berner

Nowadays in an increasingly more complex and dynamic network its not enough to be a regex ninja and storing only the logs you think you might need. From network traffic to custom logs you won't know which logs will be crucial to stop the next attacker, and if you are not planning to spend a half of your security budget in a commercial solution we will show you a way to building you own SIEM with open source. The talk will go from how to build a powerful logging environment for your organization to scaling on the cloud and storing everything forever. We will walk through how to build such a system with open source solutions as Elasticsearch and Hadoop, and creating your own custom monitoring rules to monitor everything you need. The talk will also include how to secure the environment and allow restricted access to other teams as well as avoiding common pitfalls and ensuring compliance standards.

Log Files

Heinrich Hartmann

Night owl by Boyd Meyer of PROS Mark Kerzner

20180503 kube con eu kubernetes metrics deep dive

Bob Cotton

Kubernetes generates a wealth of metrics. Some explicitly within the Kubernetes API server, the Kublet, and cAdvisor or implicitly by observing events such as the kube-state-metrics project. A subset of these metrics are used within Kubernetes itself to make scheduling decisions, however, other metrics can be used to determine the overall health of the system or for capacity planning purposes. Kubernetes exposes metrics from several places, some available internally, others through add-on projects. In this session you will learn about: - Node level metrics, as exposed from the node_exporter - Kublet metrics - API server metrics - etcd metrics - cAdvisor metrics - Metrics exposed from kube-state-metrics

Netflix Open Source: Building a Distributed and Automated Open Source Program

aspyker

Netflix has been using and contributing to open source for several years. Over the years, Netflix has released over one hundred Netflix Open Source (aka NetflixOSS) libraries, servers, and technologies. Netflix engineers benefit by accepting contributions and gathering feedback with key collaborators around the world. Users of NetflixOSS from many industries benefit from our solutions including Big Data, Build and Delivery Tools, Runtime Services and Libraries, Data Persistence, Insight, Reliability and Performance, Security and User Interface. With such a large and mature open source program, Netflix has worked on approaches and tools that help manage and improve the NetflixOSS source offerings and communities. Netflix has taken a different approach to building support for open source as compared to other Internet scale companies. Come to this session to learn about the unique approaches Netflix has taken to both distribute and automate the responsibilities of building a world-class open source program.

Building a Distributed & Automated Open Source Program at Netflix

All Things Open

What's hot

Collecting Endpoint Security Logs Through Big Data Technology - Dedi Dwianto

idsecconf

Graph Computing with Apache TinkerPop

Jason Plurad

Graphite, an introduction

jamesrwu

DN 2017 | The King is Dead, Long Live the King | Thomas Richter | Swarm64

Dataconomy Media

IPTC News Exchange Working Group 2013 Autumn Meeting

Stuart Myles

Iot data aggrigation

dokechin

The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka

All Things Open

Botconf 2013 - DNS-based Botnet C2 Server Detection

SensePost

What's hot (8)

Collecting Endpoint Security Logs Through Big Data Technology - Dedi Dwianto

Graph Computing with Apache TinkerPop

Graphite, an introduction

DN 2017 | The King is Dead, Long Live the King | Thomas Richter | Swarm64

IPTC News Exchange Working Group 2013 Autumn Meeting

Iot data aggrigation

The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka

Botconf 2013 - DNS-based Botnet C2 Server Detection

Similar to Presto Bangalore Meetup1 Event Listeners@qubole

Key considerations in productionizing streaming applications

KafkaZone

Ridwan Fadjar Septian PyCon ID 2021 Regular Talk - django application monitor...

Ridwan Fadjar

Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...

Severalnines

New Relic

Gene Chuang

Big Query Basics

Ido Green

Cost Effective Presto on AWS with Spot Nodes - Strata SF 2019

Shubham Tagra

OpenTelemetry For Operators

Kevin Brockhoff

Presto Summit 2018 - 10 - Qubole

kbajda

Security Monitoring for big Infrastructures without a Million Dollar budget

Juan Berner

Log Files

Heinrich Hartmann

Night owl by Boyd Meyer of PROS Mark Kerzner

20180503 kube con eu kubernetes metrics deep dive

Bob Cotton

Netflix Open Source: Building a Distributed and Automated Open Source Program

aspyker

Building a Distributed & Automated Open Source Program at Netflix

All Things Open

Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...

HostedbyConfluent

Apache Kafka is used as the primary message bus for propagating events and logs across Uber. In particular, it pairs with Apache Pinot, a real-time distributed OLAP datastore, to deliver real-time insights seconds after the messages produced to Kafka. One challenge we faced was to update existing data in Pinot with the changelog in Kafka, and deliver an accurate view in the real-time analytical results. For example, the financial dashboard can report gross booking with the corrected Ride fares. And restaurant owners can analyze the UberEats orders with their latest delivery status. Implementing upserts in an immutable real-time OLAP store like Pinot is nontrivial. We need to make architectural changes in how data is distributed via Kafka amongst the server nodes, how it's indexed and queried in a distributed fashion. In this talk I will discuss how we leveraged Kafka's partition-by-key feature to this end and how we added this ability in Pinot without any performance degradation.

Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019

VMware Tanzu

Eko10 workshop - OPEN SOURCE DATABASE MONITORING

Pablo Garbossa

Most database products have their own auditing functionalities or plugins but they always involve overhead which means they end up having them turned off or with the bare minimum enabled. In this workshop we will show how to get reliable logging for mysql and mongodb servers in a scalable and non intrusive way, its drawbacks and how we can build our own open source tools to achieve results similar to most commercial products. Tools to sniff, process and act upon queries will be shared and we will show how simple is to set up and monitor a database environment so it can be replicated and grow horizontally. All the code needed will be published.

Google Cloud Platform Special Training

Simon Su

Ceilometer presentation ODS Grizzly.pdf

OpenStack Foundation

Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl

Severalnines

Running PostgreSQL in production comes with the responsibility for a business critical environment; this includes high availability, disaster recovery, and performance. Ops staff worry whether databases are up and running, if backups are taken and tested for integrity, whether there are performance problems that might affect end user experience, if failover will work properly in case of server failure without breaking applications, and the list goes on. ClusterControl can be used to operationalize your PostgreSQL footprint across your enterprise. It offers a standard way of deploying high-availability replication setups with auto-failover, integrated with load balancers offering a single endpoint to applications. It provides constant health and performance monitoring through rich dashboards, as well as backup management and point-in-time recovery See how much time and effort can be saved, as well as risks mitigated, with the help of a unified management platform over the more traditional, manual methods. We’ve seen a 152% increase in ClusterControl installations by PostgreSQL users last year, so make sure you don’t miss out on the trend! AGENDA - Managing PostgreSQL “the old way”: - Common challenges - Important tasks to perform - Tools that are available to help - PostgreSQL automation and management with ClusterControl: - Deployment - Backup and recovery - HA setups - Failover - Monitoring - Live Demo SPEAKER Sebastian Insausti, Support Engineer at Severalnines, has loved technology since his childhood, when he did his first computer course (Windows 3.11). And from that moment he was decided on what his profession would be. He has since built up experience with MySQL, PostgreSQL, HAProxy, WAF (ModSecurity), Linux (RedHat, CentOS, OL, Ubuntu server), Monitoring (Nagios), Networking and Virtualization (VMWare, Proxmox, Hyper-V, RHEV). Prior to joining Severalnines, Sebastian worked as a consultant to state companies in security, database replication and high availability scenarios. He’s also a speaker and has given a few talks locally on InnoDB Cluster and MySQL Enterprise together with an Oracle team. Previous to that, he worked for a Mexican company as chief of sysadmin department as well as for a local ISP (Internet Service Provider), where he managed customers' servers and connectivity.

Similar to Presto Bangalore Meetup1 Event Listeners@qubole (20)

Key considerations in productionizing streaming applications

Ridwan Fadjar Septian PyCon ID 2021 Regular Talk - django application monitor...

Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...

New Relic

Big Query Basics

Cost Effective Presto on AWS with Spot Nodes - Strata SF 2019

OpenTelemetry For Operators

Presto Summit 2018 - 10 - Qubole

Security Monitoring for big Infrastructures without a Million Dollar budget

Log Files

Night owl by Boyd Meyer of PROS

20180503 kube con eu kubernetes metrics deep dive

Netflix Open Source: Building a Distributed and Automated Open Source Program

Building a Distributed & Automated Open Source Program at Netflix

Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...

Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019

Eko10 workshop - OPEN SOURCE DATABASE MONITORING

Google Cloud Platform Special Training

Ceilometer presentation ODS Grizzly.pdf

Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl

Recently uploaded

OpenMetadata Community Meeting - 5th June 2024

OpenMetadata

The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features. * How to run your own data quality framework * What is the performance impact of running data quality frameworks * How to run the test cases in your own ETL pipelines * How the Incident Manager is integrated * Get notified with alerts when test cases fail Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E

Enterprise Resource Planning System in Telangana

NYGGS Automation Suite

Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics. To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/

Lecture 1 Introduction to games development

abdulrafaychaudhry

BoxLang: Review our Visionary Licenses of 2024

Ortus Solutions, Corp

Graspan: A Big Data System for Big Code Analysis

Aftab Hussain

We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs. We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations. These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18. - Accepted in ASPLOS ‘17, Xi’an, China. - Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17. - Invited for presentation at SoCal PLS ‘16. - Invited for poster presentation at PLDI SRC ‘16.

Quarkus Hidden and Forbidden Extensions

Max Andersen

Vitthal Shirke Java Microservices Resume.pdf

Vitthal Shirke

In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...

Juraj Vysvader

openEuler Case Study - The Journey to Supply Chain Security

Shane Coughlan

Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better

XfilesPro

Globus Connect Server Deep Dive - GlobusWorld 2024

Globus

Globus Compute wth IRI Workflows - GlobusWorld 2024

Globus

As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.

Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx

ShamsuddeenMuhammadA

Need for Speed: Removing speed bumps from your Symfony projects ⚡️

Łukasz Chruściel

No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception. In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed. We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

Mind IT Systems

May Marketo Masterclass, London MUG May 22 2024.pdf

Adele Miller

First Steps with Globus Compute Multi-User Endpoints

Globus

In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.

Introduction to Pygame (Lecture 7 Python Game Development)

abdulrafaychaudhry

Large Language Models and the End of Programming

Matt Welsh

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...

Crescat

Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry. Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events. With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use. Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements. If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io

Recently uploaded (20)

OpenMetadata Community Meeting - 5th June 2024

Enterprise Resource Planning System in Telangana

Lecture 1 Introduction to games development

BoxLang: Review our Visionary Licenses of 2024

Graspan: A Big Data System for Big Code Analysis

Quarkus Hidden and Forbidden Extensions

Vitthal Shirke Java Microservices Resume.pdf

In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...

openEuler Case Study - The Journey to Supply Chain Security

Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better

Globus Connect Server Deep Dive - GlobusWorld 2024

Globus Compute wth IRI Workflows - GlobusWorld 2024

Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx

Need for Speed: Removing speed bumps from your Symfony projects ⚡️

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

May Marketo Masterclass, London MUG May 22 2024.pdf

First Steps with Globus Compute Multi-User Endpoints

Introduction to Pygame (Lecture 7 Python Game Development)

Large Language Models and the End of Programming

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...