Monitoring anomalies in experimentation platform.

•Download as PPTX, PDF•

0 likes•144 views

This document discusses monitoring anomalies in eBay's experimentation (A/B testing) platform. It provides an overview of A/B testing and how eBay uses it to test different experiences on over 150 million users. It then describes how eBay processes experimentation data using Hadoop and builds 200+ daily metrics. It also discusses the types of anomalies that can occur during experiments and how eBay uses Grafana and InfluxDB to visualize anomalies identified in Teradata/Hive to help monitor the experimentation platform.

Engineering

Monitoring Anomalies in
Experimentation Platform
Deepak Vasthimal – MTS @ eBay
Connect Me - https://www.linkedin.com/in/whatisdeepak
Available on eBay Tech Blog - https://goo.gl/6bUbE9

Overview of Experimentation (A/B Test) Platform
• A/B Testing is comparing different experiences and
measure their performance.
• Variance could be in UI, Components, Algorithms etc.
• Measure Bankable metrics and non-bankable (activity
click rates).
• Enables data driven decisions.
• Avoid making releases features on intuitions to over 150
million users.
Monitoring Anomalies in the Experimentation Platform 2

Experimentation Reporting
• 1500+ experiments
• Migrated from Teradata/SQL to Hadoop using Scala.
• Process 100s of TBs of data daily on Hadoop cluster with
around 400 M/R jobs.
• 200+ metrics generated daily using batch system.
• Built using a mix of open source technologies like Scala,
Scoobi, Hive, Hadoop and proprietary tech like Teradata
and MicroStrategy.
Monitoring Anomalies in the Experimentation Platform 3

High Level Flow
Monitoring Anomalies in the Experimentation Platform 4

Anomalies with Experiments
• Traffic corruption – Traffic between test and control is
skewed by UID/GUID.
• Tag corruption – Data loss/corrupted during logging &
transfer to HDFS.
• GUID Reset – Browser cookie.
• Cache refresh - eBay application servers maintain
caches of experiment configurations. A software or
hardware glitch can cause corruption of cache.
Monitoring Anomalies in the Experimentation Platform 5

Monitoring Anomalies
• Identify & Categorize anomalies within
experiments using Teradata/Hive.
• Store identified anomalies in HDFS and route
to InfluxDB (TSDB)
•Visualize using Grafana.
•I was introduced to Grafana through SpaceX
tweet by Torkel.
Monitoring Anomalies in the Experimentation Platform 6

Reason we chose Grafana
• Visually pleasing graphs.
• Easy setup.
• In Built Query Editor (SQL & UI)
• Instantly change dashboards using duplicate
dashboards/panels feature.
• InfluxDB for its ability to be setup in minutes
when compared to Graphite/Prometheus .
• Entire pipeline took couple of days.
Monitoring Anomalies in the Experimentation Platform 7

Home Page
Monitoring Anomalies in the Experimentation Platform 8

DrillDown
Monitoring Anomalies in the Experimentation Platform 9

Search
Monitoring Anomalies in the Experimentation Platform 10

Scale
• InfluxDB (v 0.11-1) is installed on a single node with
45 GB of memory.
• Grafana (v 3.0.2) is installed on a single node with 45
GB of memory.
• 2000 points are ingested daily which is minuscule.
• Currently have around 10 months of historical data.
Monitoring Anomalies in the Experimentation Platform 11

Grafana <3 is spreading.
• Performance analysis of MapReduce jobs by
Experimentation platform with job counters.
• Monitor Elastic Search Cluster (10k+ points per
second).
• Anomaly detection in Tracking data (10k+ points
per minute).
• Each use case stores data in InfluxDB.
Monitoring Anomalies in the Experimentation Platform 12

Thank You
Monitoring Anomalies in the Experimentation Platform 13

This document discusses running an Elasticsearch server inside a Spark cluster for development purposes. It presents the problem of Elasticsearch running outside of Spark making development cumbersome. The solution is to start an Elasticsearch server inside the Spark cluster that can be accessed natively from Spark jobs. Data can then be written to and read from this local Elasticsearch server. Additionally, snapshots of the Elasticsearch indices can be stored in S3 for backup and restoration. A demo application is shown that indexes tweets in real-time to the local Elasticsearch server running inside the Spark cluster.

Production Readiness Testing At Salesforce Using Spark MLlib

Spark Summit

The document discusses Salesforce's production readiness testing process called Hammer and how it was improved using Apache Spark. Hammer executes customer unit tests on two different versions of the platform to identify bugs. It was challenging to setup infrastructure to run Hammer on multiple versions and compare large volumes of test results. The new system uses Spark SQL to prepare test execution data and Spark ML to cluster test failures into groups representing potential bugs, enabling faster analysis. This improved the speed of analyzing hundreds of thousands of test records from hours to minutes, reducing analysis time by over 97% on average.

Stream Processing use cases and applications with Apache Apex by Thomas Weise

Big Data Spain

19 features you will miss if you leave Oracle Database

Franck Pachot

Today, the choice of the database technology is in the hands of the developers, and Open Source alternatives look appealing. Many talks and papers show the growing features of MySQL or PostgreSQL, which competes with the commercial RDBMS. But do not forget the core features that are there for decades. They may not be there, or less efficient, in those alternative solutions. Let’s show those basic features, with demos: in-place updates, cursor sharing, guaranteed disk writes, advanced materialized views, IOT, wait events, online operations,… Many were already in Oracle 30 years ago and we take them for granted. We may not realize what we will miss when going to another RDBMS.

Spark Uber Development Kit

Jen Aman

This document summarizes a presentation given at Spark Summit 2016 about tools and techniques used at Uber for Spark development and jobs. It introduces SCBuilder for encapsulating cluster environments, Kafka dispersal for publishing RDD results to Kafka, and SparkPlug for kickstarting job development with templates. It also discusses SparkChamber for distributed log debugging and future work including analytics, machine learning, and resource usage auditing.

Oracle to PostgreSQL migration

strikr .

StrikrSystemsLLP investigated various freeware tools for migrating databases ranging in size from 10Mb to 80Gb from Oracle to PostgreSQL and MariaDB. The goal was to migrate the schema, tablespace, table, view, index and some constraints with high fidelity. CSV dumps of data from Oracle were successful. Tables, views and partial indexes were migrated but stored procedures and user-defined functions did not migrate. Challenges included different data type storage sizes and encoding between databases. After customizations, schema, tables, views and partial indexes were successfully migrated but stored procedures and functions require parsing SQL grammar and object representations. Going forward it is recommended to use PostgreSQL for new projects, refactor existing applications before migration, and

Spark Summit EU talk by Yiannis Gkoufas

Spark Summit

This document introduces SparkOscope, a tool that enables optimization of Apache Spark through cross-stack monitoring and visualization. SparkOscope extends Spark's monitoring framework by collecting operating system-level metrics using Hyperic Sigar library. It enhances Spark's web UI to plot all available metrics, including the new OS metrics. SparkOscope provides an easy way to monitor Spark jobs and identify bottlenecks by associating metrics with source code locations. The author demonstrates SparkOscope and discusses plans to improve its storage mechanism and enable infrastructure recommendations based on job resource usage patterns.

Stream data from Apache Kafka for processing with Apache Apex

Apache Apex

Meetup presentation: How Apache Apex consumes from Kafka topics for real-time time processing and analytics. Learn about features of the Apex Kafka Connector, which is one of the most popular operators in the Apex Malhar operator library, and powers several production use cases. We explain the advanced features this operator provides for high throughput, low latency ingest and how it enables fault tolerant topologies with exactly once processing semantics.

The team at InfluxData needed to review benchmark data on InfluxDB during their development cycle to ensure any new changes continue to improve performance. However, using the existing benchmarking framework, InfluxDB-comparison, was manual and time consuming and not used consistently. To change this, InfluxData asked the team at Bonitoo to enhance the benchmark framework to easily incorporate new use cases, add new versions of the software quickly and easily and provide benchmark data on a cadence that works for the development (daily, weekly, monthly) cycle. In this webinar the team from Bonitoo will share how they were able to accomplish this as well as build automation into the existing framework. In addition, they will share the benchmark results generated from the framework that highlights how performant a time series database like InfluxDB is compared to the latest versions of products like MongoDB, Cassandra, Elasticsearch, and OpenTSDB.

Deployment Checkup: How to Regularly Tune Your Cloud Environment - RightScale...

RightScale

The document discusses the importance of regularly tuning cloud environments through deployment checkups. It highlights key areas to focus on during checkups, including cost optimization by identifying unused resources, ensuring optimal server utilization, implementing high availability and disaster recovery strategies, addressing security issues, and following best practices. Regular checkups help avoid inefficiencies that can arise over time and ensure deployments are optimized for cost, performance, availability and security.

Spark Summit EU talk by Jim Dowling

Spark Summit

This document discusses Hopsworks, a Spark-as-a-Service platform built on Hops Hadoop. It provides: - Secure multi-tenant Spark and Kafka clusters hosted on-premise using YARN. - Project-based access control and quotas for storage and compute. - Simplified development of secure Spark Streaming applications with Kafka using automatically distributed certificates. - Support for Zeppelin notebooks, automated installation, and tools like DrElephant for job monitoring.

Sysml 2019 demo_paper

strange_loop

This document introduces a horizontally scalable machine learning (ML) pipeline platform called Hopsworks that features an integrated Feature Store. The Feature Store provides APIs for data engineers to produce features and data scientists to select features for models, simplifying pipelines. The end-to-end ML pipelines on Hopsworks can scale out to hundreds of servers using Apache Spark, Hive, and HopsFS. The demonstration will showcase a full pipeline running on Hopsworks, including feature engineering, publishing to the Feature Store, training, and model deployment. Hopsworks is also used as an interactive teaching platform for university courses on deep learning and big data.

OOW13 Exadata and ODI with Parallel

Kellyn Pot'Vin-Gorman

This document summarizes a case study on using Exadata, Oracle Data Integrator (ODI), and parallel data loading to improve ETL performance. It describes how the environment had performance issues with ODI loads due to temporary tablespace usage. Tuning included setting dynamic sampling to 0, adding rollup tables and materialized views to pre-aggregate data, and indexing to reduce sorting and aggregation during loads. These changes led to significant improvements in elapsed time.

DataStax & O'Reilly Media: Large Scale Data Analytics with Spark and Cassandr...

DataStax Academy

In this in-depth workshop you will gain hands on experience with using Spark and Cassandra inside the DataStax Enterprise Platform. The focus of the workshop will be working through data analytics exercises to understand the major developer developer considerations. You will also gain an understanding of the internals behind the integration that allow for large scale data loading and analysis. It will also review some of the major machine learning libraries in Spark as an example of data analysis. The workshop will start with a review the basics of how Spark and Cassandra are integrated. Then we will work through a series of exercises that will show how to perform large scale Data Analytics with Spark and Cassandra. A major part of the workshop will be to understand effective data modeling techniques in Cassandra that allow for fast parallel loading of the data into Spark to perform large scale analytics on that data. The exercises will also look at how to how to use the open source Spark Notebook to run interactive data analytics with the DataStax Enterprise Platform.

Apache Apex Meetup at Cask

Apache Apex

This document provides an overview of Apache Apex, an open source unified streaming and fast batching platform. It discusses key aspects of Apex including its application programming model using operators and directed acyclic graphs, native Hadoop integration using YARN and HDFS, partitioning and scaling operators for high throughput, windowing support, fault tolerance, and data locality features. Examples of building a data processing pipeline and its logical and physical plans are also presented.

Apache Apex connector with Kafka 0.9 consumer API

Apache Apex

Scaling Machine Learning To Billions Of Parameters

Jen Aman

This document summarizes scaling machine learning to billions of parameters using Spark and a parameter server architecture. It describes the requirements for supporting both batch and sequential optimization at web scale. It then outlines the Spark + Parameter server approach, leveraging Spark for distributed processing and the parameter server for synchronizing model updates. Examples of distributed L-BFGS and Word2Vec training are presented to illustrate batch and sequential optimization respectively.

Highlights and Challenges from Running Spark on Mesos in Production by Morri ...

Spark Summit

This document discusses AppsFlyer's experience running Spark on Mesos in production for retention data processing and analytics. Key points include: - AppsFlyer processes over 30 million installs and 5 billion sessions daily for retention reporting across 18 dimensions using Spark, Mesos, and S3. - Challenges included timeouts and errors when using Spark's S3 connectors due to the eventual consistency of S3, which was addressed by using more robust connectors and configuration options. - A coarse-grained Mesos scheduling approach was found to be more stable than fine-grained, though it has limitations like static core allocation that future Mesos improvements may address. - Tuning jobs for coarse-

Apache Apex Kafka Input Operator

Apache Apex

Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...

Databricks

Dr. Elephant helps improve Spark and Hadoop developer productivity and increase cluster efficiency by making clear recommendations on how to tune workloads and configurations. Originally developed by LinkedIn, Dr. Elephant is now in use at multiple sites. This session will explore how Dr. Elephant works, the data it collects from Spark environments and the customizable heuristics that generate tuning recommendations. Learn how Dr. Elephant can be used to improve production cluster operations, help developers avoid common issues, and green light applications for use on production clusters.

How to Improve Performance Testing Using InfluxDB and Apache JMeter

InfluxData

The document describes an end-to-end performance testing framework that uses InfluxDB and Grafana for real-time monitoring and analytics. It includes a replay framework that eliminates data loss during performance tests by writing transactions to a CSV file when the database is down and replaying from the CSV when the database comes back up. The framework provides benefits like increased efficiency, accuracy, and scalability for performance and load testing.

Introduction to Apache Apex

Apache Apex

Extending The Yahoo Streaming Benchmark to Apache Apex

Apache Apex

From Pipelines to Refineries: scaling big data applications with Tim Hunter

Databricks

Big data tools are challenging to combine into a larger application: ironically, big data applications themselves do not tend to scale very well. These issues of integration and data management are only magnified by increasingly large volumes of data. Apache Spark provides strong building blocks for batch processes, streams and ad-hoc interactive analysis. However, users face challenges when putting together a single coherent pipeline that could involve hundreds of transformation steps, especially when confronted by the need of rapid iterations. This talk explores these issues through the lens of functional programming. It presents an experimental framework that provides full-pipeline guarantees by introducing more laziness to Apache Spark. This framework allows transformations to be seamlessly composed and alleviates common issues, thanks to whole program checks, auto-caching, and aggressive computation parallelization and reuse.

Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations

Apache Apex

Architectual Comparison of Apache Apex and Spark Streaming

Apache Apex

This presentation discusses architectural differences between Apache Apex features with Spark Streaming. It discusses how these differences effect use cases like ingestion, fast real-time analytics, data movement, ETL, fast batch, very low latency SLA, high throughput and large scale ingestion. Also, it will cover fault tolerance, low latency, connectors to sources/destinations, smart partitioning, processing guarantees, computation and scheduling model, state management and dynamic changes. Further, it will discuss how these features affect time to market and total cost of ownership.

Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache Apex

Apache Apex

Stream data processing is becoming increasingly important to support business needs for faster time to insight and action with growing volume of information from more sources. Apache Apex (http://apex.apache.org/) is a unified big data in motion processing platform for the Apache Hadoop ecosystem. Apex supports demanding use cases with: * Architecture for high throughput, low latency and exactly-once processing semantics. * Comprehensive library of building blocks including connectors for Kafka, Files, Cassandra, HBase and many more * Java based with unobtrusive API to build real-time and batch applications and implement custom business logic. * Advanced engine features for auto-scaling, dynamic changes, compute locality. Apex was developed since 2012 and is used in production in various industries like online advertising, Internet of Things (IoT) and financial services.

Smart Partitioning with Apache Apex (Webinar)

Apache Apex

Processing big data often requires running the same computations parallelly in multiple processes or threads, called partitions, with each partition handling a subset of the data. This becomes all the more necessary when processing live data streams where maintaining SLA is paramount. Furthermore, multiple different computations make up an application and each of them may have different partitioning needs. Partitioning also needs to adapt to changing data rates, input sources and other application requirements like SLA. In this talk, we will introduce how Apache Apex, a distributed stream processing platform on Hadoop, handles partitioning. We will look at different partitioning schemes provided by Apex some of which are unique in this space. We will also look at how Apex does dynamic partitioning, a feature unique to and pioneered by Apex to handle varying data needs with examples. We will also talk about the different utilities and libraries that Apex provides for users to be able to affect their own custom partitioning.

Razvoj novca u Republici Hrvatskoj

Marko Božac

Tablas

bryanmedina00101

What's hot

How the Automation of a Benchmark Famework Keeps Pace with the Dev Cycle at I...

DevOps.com

Deployment Checkup: How to Regularly Tune Your Cloud Environment - RightScale...

RightScale

Spark Summit EU talk by Jim Dowling

Spark Summit

Sysml 2019 demo_paper

strange_loop

OOW13 Exadata and ODI with Parallel

Kellyn Pot'Vin-Gorman

DataStax & O'Reilly Media: Large Scale Data Analytics with Spark and Cassandr...

DataStax Academy

Apache Apex Meetup at Cask

Apache Apex

Apache Apex connector with Kafka 0.9 consumer API

Apache Apex

Scaling Machine Learning To Billions Of Parameters

Jen Aman

Highlights and Challenges from Running Spark on Mesos in Production by Morri ...

Spark Summit

Apache Apex Kafka Input Operator

Apache Apex

Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...

Databricks

How to Improve Performance Testing Using InfluxDB and Apache JMeter

InfluxData

Introduction to Apache Apex

Apache Apex

Extending The Yahoo Streaming Benchmark to Apache Apex

Apache Apex

From Pipelines to Refineries: scaling big data applications with Tim Hunter

Databricks

Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations

Apache Apex

Architectual Comparison of Apache Apex and Spark Streaming

Apache Apex

Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache Apex

Apache Apex

Smart Partitioning with Apache Apex (Webinar)

Apache Apex

What's hot (20)

How the Automation of a Benchmark Famework Keeps Pace with the Dev Cycle at I...

Deployment Checkup: How to Regularly Tune Your Cloud Environment - RightScale...

Spark Summit EU talk by Jim Dowling

Sysml 2019 demo_paper

OOW13 Exadata and ODI with Parallel

DataStax & O'Reilly Media: Large Scale Data Analytics with Spark and Cassandr...

Apache Apex Meetup at Cask

Apache Apex connector with Kafka 0.9 consumer API

Scaling Machine Learning To Billions Of Parameters

Highlights and Challenges from Running Spark on Mesos in Production by Morri ...

Apache Apex Kafka Input Operator

Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop with Carl ...

How to Improve Performance Testing Using InfluxDB and Apache JMeter

Introduction to Apache Apex

Extending The Yahoo Streaming Benchmark to Apache Apex

From Pipelines to Refineries: scaling big data applications with Tim Hunter

Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations

Architectual Comparison of Apache Apex and Spark Streaming

Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache Apex

Smart Partitioning with Apache Apex (Webinar)

Viewers also liked

Razvoj novca u Republici Hrvatskoj

Marko Božac

Tablas

bryanmedina00101

Godišnje izvješće Financijskog kluba u mandatu 2014./2015.

Marko Božac

When ego gets in the way of marketing

weinmark

When clients complain about fees, it is often due to ego getting in the way of effective marketing. To address client complaints, professionals should realize that (1) clients will always question fees to get the best deal, (2) fees are less likely to be questioned if they are competitive and the value is clear, and (3) fee complaints may disguise real complaints. Professionals should educate clients on value, improve the client experience, and consider alternative fee structures beyond hourly billing to address complaints in a way that serves clients.

Dietética

Blanckys Gonzalez

Muhammad_Hassan_Karachi

Syed Muhammad Hassan Zaidi

Muhammad Hassan has over 15 years of experience in IT roles including systems administration, network administration, database administration, and project management. He has worked in various industries and educational institutions, and has expertise in technologies such as Windows Server, Active Directory, virtualization, networking, databases, and SAP. He provides his contact information and includes details of his work history, areas of expertise, education, and certifications.

Demanda sap y otros

Adalberto Cervantes Rodriguez

Curso adm 471 aplicacion de flujo de caja

Procasecapacita

Este documento presenta un curso sobre la aplicación de flujo de caja. El curso busca enseñar a planificar los flujos de caja para determinar las necesidades de efectivo a corto y mediano plazo, mejorar los procesos de cobranza y pagos, e implementar un sistema de control de gestión. El curso dura 30 horas y cubre temas como la disponibilidad de efectivo, determinación de flujos, indicadores financieros, estados de flujo de efectivo y el control de la disponibilidad de fondos.

trabajo final de tecnologia

yuraniscope

El documento resume las biografías y logros científicos de tres importantes figuras históricas: Arquímedes, Leonardo da Vinci y Nicolás Copérnico. Arquímedes fue un destacado matemático y físico de la antigua Grecia que realizó importantes descubrimientos en geometría y mecánica. Leonardo da Vinci fue un polifacético genio del Renacimiento italiano que destacó como pintor, científico e inventor. Nicolás Copérnico fue un astrónomo polaco que formuló

Nesvrstani

Милан Ристић

Finding Product Market Fit - Peter Reinhardt at SaaStock 16

SaaStock

Slide smau firenze 08.07.2016

Claudio Morelli

Mantenimiento de estructuras

Antonio Mogollon

Energias alternas o alternativas

yuraniscope

Las energías alternativas o renovables son fuentes de energía planteadas como alternativa a los combustibles fósiles tradicionales como el carbón, gas y petróleo. Algunas de estas energías alternativas incluyen la energía solar, eólica, geotérmica e hidroeléctrica. Aunque también existen debates sobre si la energía nuclear debería considerarse una fuente de energía alternativa. Las energías alternativas sirven para proveer energía de una manera más sostenible y con menores emisiones de carbono que los combustibles fósiles

Viewers also liked (14)

Razvoj novca u Republici Hrvatskoj

Tablas

Godišnje izvješće Financijskog kluba u mandatu 2014./2015.

When ego gets in the way of marketing

Dietética

Muhammad_Hassan_Karachi

Demanda sap y otros

Curso adm 471 aplicacion de flujo de caja

trabajo final de tecnologia

Nesvrstani

Finding Product Market Fit - Peter Reinhardt at SaaStock 16

Slide smau firenze 08.07.2016

Mantenimiento de estructuras

Energias alternas o alternativas

Similar to Monitoring anomalies in experimentation platform.

Cerberus : Framework for Manual and Automated Testing (Web Application)

CIVEL Benoit

Cerberus_Presentation1

CIVEL Benoit

This document describes Cerberus, an open source test automation tool developed by La Redoute. Cerberus allows centralized management of test cases across multiple technologies like web, mobile, and APIs. It supports features like a step library, test automation, execution reporting, and integration with other tools. The document also provides examples of how Cerberus is used at La Redoute for regression testing websites in multiple languages and environments. It maintains over 3,500 regression tests that execute twice daily. Cerberus can also be used for functional monitoring of websites by regularly executing test cases and monitoring performance metrics.

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

ssuserd3a367

1) StumbleUpon uses open source tools like Kafka, HBase, Hive and Pig to build a scalable big data infrastructure to process large amounts of data from its services in real-time and batch. 2) Data is collected from various services using Kafka and stored in HBase for real-time analytics. Batch processing is done using Pig and data is loaded into Hive for ad-hoc querying. 3) The infrastructure powers various applications like recommendations, ads and business intelligence dashboards.

Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...

Spark Summit

The document discusses Conviva's Unified Framework (CUF) for analyzing video streaming data in real-time, near real-time, and offline using Spark and Databricks. It summarizes Conviva's platform for measuring video quality of experience across devices and networks. The framework unifies the three analysis stacks onto Spark to share code and insights. Using Databricks improves the offline analysis speed and enables data scientists to independently explore large datasets and build machine learning models.

Intro to Apache Apex - Next Gen Platform for Ingest and Transform

Apache Apex

Introduction to Apache Apex - The next generation native Hadoop platform. This talk will cover details about how Apache Apex can be used as a powerful and versatile platform for big data processing. Common usage of Apache Apex includes big data ingestion, streaming analytics, ETL, fast batch alerts, real-time actions, threat detection, etc. Bio: Pramod Immaneni is Apache Apex PMC member and senior architect at DataTorrent, where he works on Apache Apex and specializes in big data platform and applications. Prior to DataTorrent, he was a co-founder and CTO of Leaf Networks LLC, eventually acquired by Netgear Inc, where he built products in core networking space and was granted patents in peer-to-peer VPNs.

Stream Computing (The Engineer's Perspective)

Ilya Ganelin

This document discusses stream computing from an engineer's perspective. It begins by contrasting batch and stream processing, noting that stream processing handles data one record at a time with an emphasis on latency over throughput. The document then explores how to achieve scalability, performance, durability and availability in stream processing systems. It notes the tradeoffs between these goals and discusses challenges like handling failures. Specific open-source stream processing systems like Storm, Flink and Apex are then analyzed in terms of how they work, strengths, weaknesses and failure handling. The document concludes by discussing using distributed databases for state management in stream processing applications.

PEARC17: Live Integrated Visualization Environment: An Experiment in General...

moneyjh

Tale of two streaming frameworks- Apace Storm & Apache Flink

Karthik Deivasigamani

This document compares the Apache Storm and Apache Flink streaming frameworks. Both frameworks can be used to process streaming data in real-time. Storm uses spouts and bolts to define topologies while Flink uses data sources, transformations, and sinks. Flink focuses on exactly-once processing semantics using checkpointing while Storm provides at-least-once guarantees. The document discusses using each framework for catalog and pricing use cases at a large retailer and highlights challenges faced and lessons learned. Overall, both frameworks are mature but Flink is seen as faster with richer APIs and better support for features like windowing, joins, and stateful stream processing.

Tale of two streaming frameworks (Karthik D - Walmart)

KafkaZone

SQL on Hadoop benchmarks using TPC-DS query set

Kognitio

IoT Ingestion & Analytics using Apache Apex - A Native Hadoop Platform

Apache Apex

Internet of Things (IoT) devices are becoming more ubiquitous in consumer, business and industrial landscapes. They are being widely used in applications ranging from home automation to the industrial internet. They pose a unique challenge in terms of the volume of data they produce, and the velocity with which they produce it, and the variety of sources they need to handle. The challenge is to ingest and process this data at the speed at which it is being produced in a real-time and fault tolerant fashion. Apache Apex is an industrial grade, scalable and fault tolerant big data processing platform that runs natively on Hadoop. In this deck, you will see how Apex is being used in IoT applications and also see how the enterprise features such as dimensional analytics, real-time dashboards and monitoring play a key role. Presented by Pramod Immaneni, Principal Architect at DataTorrent and PPMC member Apache Apex, on BrightTALK webinar on Apr 6th, 2016

Bay Area Impala User Group Meetup (Sept 16 2014)

Cloudera, Inc.

The document discusses Impala releases and roadmaps. It outlines key features released in different Impala versions, including SQL capabilities, performance improvements, and support for additional file formats and data types. It also describes Impala's performance advantages compared to other SQL-on-Hadoop systems and how its approach is expected to increasingly favor performance gains. Lastly, it encourages trying out Impala and engaging with their community.

Testistanbul 2016 - Keynote: "Performance Testing of Big Data" by Roland Leusden

Turkish Testing Board

Agile, Continous Intergration, DevOps, Big data are not longer buzzwords but part of the day today process of everyone working in software development and delivery. To cope with applications that need to be deployed in production almost the same moment they were created, software development has changed, impacting the way of working for everyone in the team. In this talk, Roland will discuss the challenges performance testers face with Big Data applications and how Architecture, Agile, Continous Intergration and DevOps come together to create solutions.

Nonfunctional Testing: Examine the Other Side of the Coin

TechWell

Creating a highly available, scalable, and high-performing system requires a substantial amount of what we call nonfunctional testing. Developing nonfunctional testing skills is a must for many of today’s quality engineers (QEs). For the past several years, Balaji Arunachalam’s quality team for Intuit Core Services has experienced several highly available and disaster recovery buildup and testing challenges. Their journey includes the evolution of functional QEs into hybrid QEs who are capable of doing both functional and nonfunctional testing. Nonfunctional testing includes capacity, stability, benchmarking, FMEA/RAS, datacenter failover, and scalability testing. Balaji shares nonfunctional testing best practices, learnings, and mistakes they encountered on this journey. If you or your team is ready flip the coin and take a serious look at nonfunctional testing methods, opportunities, challenges, and solutions, this session is for you.

New Performance Benchmarks: Apache Impala (incubating) Leads Traditional Anal...

Cloudera, Inc.

Recording Link: http://bit.ly/LSImpala Author: Greg Rahn, Cloudera Director of Product Management In this session, we'll review the recent set of benchmark tests the Apache Impala (incubating) performance team completed that compare Apache Impala to a traditional analytic database (Greenplum), as well as to other SQL-on-Hadoop engines (Hive LLAP, Spark SQL, and Presto). We'll go over the methodology and results, and we'll also discuss some of the performance features and best practices that make this performance possible in Impala. Lastly, we'll look at some recent advancements in in Impala over the past few releases.

Using Machine Learning to Understand Kafka Runtime Behavior (Shivanath Babu, ...

confluent

Apache Kafka is now nearly ubiquitous in modern data pipelines and use cases. While the Kafka development model is elegantly simple, operating Kafka clusters in production environments is a challenge. It’s hard to troubleshoot misbehaving Kafka clusters, especially when there are potentially hundreds or thousands of topics, producers and consumers and billions of messages. The root cause of why real-time applications is lag may be due to an application problem – like poor data partitioning or load imbalance – or due to a Kafka problem – like resource exhaustion or suboptimal configuration. Therefore getting the best performance, predictability, and reliability for Kafka-based applications can be difficult. In the end, the operation of your Kafka powered analytics pipelines could themselves benefit from machine learning (ML).

SCAPE - Scalable Preservation Environments

SCAPE Project

Ross King, Project Director of SCAPE, gave a short presentation of the EU funded project SCAPE, including descriptions of tools for planning and monitoring digital preservation, scalable computation and repositories, SCAPE Testbeds and where to learn more. The presentation was given at the workshop ‘Preservation at Scale’ http://bit.ly/17ppAln in connection with the iPres2013 conference in Lissabon, Portugal, in September 2013.

Tech talk microservices debugging

Andrey Kolodnitsky

- Debugging microservices presents key challenges due to their distributed nature across multiple processes. Observability techniques like logging, monitoring and tracing are important to gain visibility. - Telepresence allows debugging services locally by intercepting requests to emulate the environment without needing to deploy to the cluster. Telepresence v1 swaps the deployment entirely for local debugging, while v2 intercepts specific ports/requests. - Choosing between Telepresence v1 and v2 depends on use cases - v1 is better for consuming messages while v2 is better for intercepting specific ports/requests without a full deployment swap. Both provide useful debugging capabilities for microservices running in Kubernetes.

Debugging Microservices - key challenges and techniques - Microservices Odesa...

Lohika_Odessa_TechTalks

Microservice architecture is widespread our days. It comes with a lot of benefits and challenges to solve. Main goal of this talk is to go through troubleshooting and debugging in the distributed micro-service world. Topic would cover: main aspects of the logging,  monitoring, distributed tracing, debugging services on the cluster. About speaker: Andrеy Kolodnitskiy is Staff engineer in the Lohika and his primary focus is around distributed systems, microservices and JVM based languages. Majority of time engineers spend debugging and fixing the issues. This talk will be dedicated to best practicies and tools Andrеys team uses on its project which do help to find issues more efficiently.

Monitoring federation open stack infrastructure

Fernando Lopez Aguilar

Similar to Monitoring anomalies in experimentation platform. (20)

Cerberus : Framework for Manual and Automated Testing (Web Application)

Cerberus_Presentation1

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...

Intro to Apache Apex - Next Gen Platform for Ingest and Transform

Stream Computing (The Engineer's Perspective)

PEARC17: Live Integrated Visualization Environment: An Experiment in General...

Tale of two streaming frameworks- Apace Storm & Apache Flink

Tale of two streaming frameworks (Karthik D - Walmart)

SQL on Hadoop benchmarks using TPC-DS query set

IoT Ingestion & Analytics using Apache Apex - A Native Hadoop Platform

Bay Area Impala User Group Meetup (Sept 16 2014)

Testistanbul 2016 - Keynote: "Performance Testing of Big Data" by Roland Leusden

Nonfunctional Testing: Examine the Other Side of the Coin

New Performance Benchmarks: Apache Impala (incubating) Leads Traditional Anal...

Using Machine Learning to Understand Kafka Runtime Behavior (Shivanath Babu, ...

SCAPE - Scalable Preservation Environments

Tech talk microservices debugging

Debugging Microservices - key challenges and techniques - Microservices Odesa...

Monitoring federation open stack infrastructure

Recently uploaded

Introduction to Computer Networks & OSI MODEL.ppt

Dwarkadas J Sanghvi College of Engineering

AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...

Paris Salesforce Developer Group

Prediction of Electrical Energy Efficiency Using Information on Consumer's Ac...

PriyankaKilaniya

Energy efficiency has been important since the latter part of the last century. The main object of this survey is to determine the energy efficiency knowledge among consumers. Two separate districts in Bangladesh are selected to conduct the survey on households and showrooms about the energy and seller also. The survey uses the data to find some regression equations from which it is easy to predict energy efficiency knowledge. The data is analyzed and calculated based on five important criteria. The initial target was to find some factors that help predict a person's energy efficiency knowledge. From the survey, it is found that the energy efficiency awareness among the people of our country is very low. Relationships between household energy use behaviors are estimated using a unique dataset of about 40 households and 20 showrooms in Bangladesh's Chapainawabganj and Bagerhat districts. Knowledge of energy consumption and energy efficiency technology options is found to be associated with household use of energy conservation practices. Household characteristics also influence household energy use behavior. Younger household cohorts are more likely to adopt energy-efficient technologies and energy conservation practices and place primary importance on energy saving for environmental reasons. Education also influences attitudes toward energy conservation in Bangladesh. Low-education households indicate they primarily save electricity for the environment while high-education households indicate they are motivated by environmental concerns.

Ericsson LTE Throughput Troubleshooting Techniques.ppt

wafawafa52

一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理

sydezfe

原版一模一样【微信：741003700 】【(uoft毕业证书)加拿大多伦多大学毕业证成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原。 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 【主营项目】一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！二.真实使馆公证(即留学回国人员证明,不成功不收费) 三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度) 如果您处于以下几种情况： ◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】 ◇面对父母的压力，希望尽快拿到； ◇不清楚认证流程以及材料该如何准备； ◇回国时间很长，忘记办理； ◇回国马上就要找工作，办给用人单位看； ◇企事业单位必须要求办理的 ◇需要报考公务员、购买免税车、落转户口 ◇申请留学生创业基金留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才办理(uoft毕业证书)加拿大多伦多大学毕业证【微信：741003700 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。办理(uoft毕业证书)加拿大多伦多大学毕业证【微信：741003700 】格式相对统一，各专业都有相应的模板。通常包括以下部分：校徽：象征着学校的荣誉和传承。校名:学校英文全称授予学位：本部分将注明获得的具体学位名称。毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。办理(uoft毕业证书)加拿大多伦多大学毕业证【微信：741003700 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。综上所述，办理(uoft毕业证书)加拿大多伦多大学毕业证【微信：741003700 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。

ITSM Integration with MuleSoft.pptx

VANDANAMOHANGOUDA

Open Channel Flow: fluid flow with a free surface

Indrajeet sahu

Open Channel Flow: This topic focuses on fluid flow with a free surface, such as in rivers, canals, and drainage ditches. Key concepts include the classification of flow types (steady vs. unsteady, uniform vs. non-uniform), hydraulic radius, flow resistance, Manning's equation, critical flow conditions, and energy and momentum principles. It also covers flow measurement techniques, gradually varied flow analysis, and the design of open channels. Understanding these principles is vital for effective water resource management and engineering applications.

3rd International Conference on Artificial Intelligence Advances (AIAD 2024)

GiselleginaGloria

3rd International Conference on Artificial Intelligence Advances (AIAD 2024) will act as a major forum for the presentation of innovative ideas, approaches, developments, and research projects in the area advanced Artificial Intelligence. It will also serve to facilitate the exchange of information between researchers and industry professionals to discuss the latest issues and advancement in the research area. Core areas of AI and advanced multi-disciplinary and its applications will be covered during the conferences.

paper relate Chozhavendhan et al. 2020.pdf

ShurooqTaib

A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...

DharmaBanothu

The Network on Chip (NoC) has emerged as an effective solution for intercommunication infrastructure within System on Chip (SoC) designs, overcoming the limitations of traditional methods that face significant bottlenecks. However, the complexity of NoC design presents numerous challenges related to performance metrics such as scalability, latency, power consumption, and signal integrity. This project addresses the issues within the router's memory unit and proposes an enhanced memory structure. To achieve efficient data transfer, FIFO buffers are implemented in distributed RAM and virtual channels for FPGA-based NoC. The project introduces advanced FIFO-based memory units within the NoC router, assessing their performance in a Bi-directional NoC (Bi-NoC) configuration. The primary objective is to reduce the router's workload while enhancing the FIFO internal structure. To further improve data transfer speed, a Bi-NoC with a self-configurable intercommunication channel is suggested. Simulation and synthesis results demonstrate guaranteed throughput, predictable latency, and equitable network access, showing significant improvement over previous designs

This study Examines the Effectiveness of Talent Procurement through the Imple...

DharmaBanothu

In the world with high technology and fast forward mindset recruiters are walking/showing interest towards E-Recruitment. Present most of the HRs of many companies are choosing E-Recruitment as the best choice for recruitment. E-Recruitment is being done through many online platforms like Linkedin, Naukri, Instagram , Facebook etc. Now with high technology E- Recruitment has gone through next level by using Artificial Intelligence too. Key Words : Talent Management, Talent Acquisition , E- Recruitment , Artificial Intelligence Introduction Effectiveness of Talent Acquisition through E- Recruitment in this topic we will discuss about 4important and interlinked topics which are

SELENIUM CONF -PALLAVI SHARMA - 2024.pdf

Pallavi Sharma

UNIT-III- DATA CONVERTERS ANALOG TO DIGITAL CONVERTER

vmspraneeth

Butterfly Valves Manufacturer (LBF Series).pdf

Lubi Valves

Blood finder application project report (1).pdf

Kamal Acharya

Blood Finder is an emergency time app where a user can search for the blood banks as well as the registered blood donors around Mumbai. This application also provide an opportunity for the user of this application to become a registered donor for this user have to enroll for the donor request from the application itself. If the admin wish to make user a registered donor, with some of the formalities with the organization it can be done. Specialization of this application is that the user will not have to register on sign-in for searching the blood banks and blood donors it can be just done by installing the application to the mobile. The purpose of making this application is to save the user’s time for searching blood of needed blood group during the time of the emergency. This is an android application developed in Java and XML with the connectivity of SQLite database. This application will provide most of basic functionality required for an emergency time application. All the details of Blood banks and Blood donors are stored in the database i.e. SQLite. This application allowed the user to get all the information regarding blood banks and blood donors such as Name, Number, Address, Blood Group, rather than searching it on the different websites and wasting the precious time. This application is effective and user friendly.

Beckhoff Programmable Logic Control Overview Presentation

VanTuDuong1

ELS: 2.4.1 POWER ELECTRONICS Course objectives: This course will enable stude...

Kuvempu University

Introduction - Applications of Power Electronics, Power Semiconductor Devices, Control Characteristics of Power Devices, types of Power Electronic Circuits. Power Transistors: Power BJTs: Steady state characteristics. Power MOSFETs: device operation, switching characteristics, IGBTs: device operation, output and transfer characteristics. Thyristors - Introduction, Principle of Operation of SCR, Static Anode- Cathode Characteristics of SCR, Two transistor model of SCR, Gate Characteristics of SCR, Turn-ON Methods, Turn-OFF Mechanism, Turn-OFF Methods: Natural and Forced Commutation – Class A and Class B types, Gate Trigger Circuit: Resistance Firing Circuit, Resistance capacitance firing circuit.

309475979-Creativity-Innovation-notes-IV-Sem-2016-pdf.pdf

Sou Tibon

Particle Swarm Optimization–Long Short-Term Memory based Channel Estimation w...

IJCNCJournal

Paper Title Particle Swarm Optimization–Long Short-Term Memory based Channel Estimation with Hybrid Beam Forming Power Transfer in WSN-IoT Applications Authors Reginald Jude Sixtus J and Tamilarasi Muthu, Puducherry Technological University, India Abstract Non-Orthogonal Multiple Access (NOMA) helps to overcome various difficulties in future technology wireless communications. NOMA, when utilized with millimeter wave multiple-input multiple-output (MIMO) systems, channel estimation becomes extremely difficult. For reaping the benefits of the NOMA and mm-Wave combination, effective channel estimation is required. In this paper, we propose an enhanced particle swarm optimization based long short-term memory estimator network (PSOLSTMEstNet), which is a neural network model that can be employed to forecast the bandwidth required in the mm-Wave MIMO network. The prime advantage of the LSTM is that it has the capability of dynamically adapting to the functioning pattern of fluctuating channel state. The LSTM stage with adaptive coding and modulation enhances the BER.PSO algorithm is employed to optimize input weights of LSTM network. The modified algorithm splits the power by channel condition of every single user. Participants will be first sorted into distinct groups depending upon respective channel conditions, using a hybrid beamforming approach. The network characteristics are fine-estimated using PSO-LSTMEstNet after a rough approximation of channels parameters derived from the received data. Keywords Signal to Noise Ratio (SNR), Bit Error Rate (BER), mm-Wave, MIMO, NOMA, deep learning, optimization. Volume URL: https://airccse.org/journal/ijc2022.html Abstract URL:https://aircconline.com/abstract/ijcnc/v14n5/14522cnc05.html Pdf URL: https://aircconline.com/ijcnc/V14N5/14522cnc05.pdf #scopuspublication #scopusindexed #callforpapers #researchpapers #cfp #researchers #phdstudent #researchScholar #journalpaper #submission #journalsubmission #WBAN #requirements #tailoredtreatment #MACstrategy #enhancedefficiency #protrcal #computing #analysis #wirelessbodyareanetworks #wirelessnetworks #adhocnetwork #VANETs #OLSRrouting #routing #MPR #nderesidualenergy #korea #cognitiveradionetworks #radionetworks #rendezvoussequence Here's where you can reach us : ijcnc@airccse.org or ijcnc@aircconline.com

一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理

upoux

原版一模一样【微信：741003700 】【(uofo毕业证书)美国俄勒冈大学毕业证成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原。 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 【主营项目】一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！二.真实使馆公证(即留学回国人员证明,不成功不收费) 三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度) 如果您处于以下几种情况： ◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】 ◇面对父母的压力，希望尽快拿到； ◇不清楚认证流程以及材料该如何准备； ◇回国时间很长，忘记办理； ◇回国马上就要找工作，办给用人单位看； ◇企事业单位必须要求办理的 ◇需要报考公务员、购买免税车、落转户口 ◇申请留学生创业基金留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才办理(uofo毕业证书)美国俄勒冈大学毕业证【微信：741003700 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。办理(uofo毕业证书)美国俄勒冈大学毕业证【微信：741003700 】格式相对统一，各专业都有相应的模板。通常包括以下部分：校徽：象征着学校的荣誉和传承。校名:学校英文全称授予学位：本部分将注明获得的具体学位名称。毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。办理(uofo毕业证书)美国俄勒冈大学毕业证【微信：741003700 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。综上所述，办理(uofo毕业证书)美国俄勒冈大学毕业证【微信：741003700 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。

Recently uploaded (20)

Introduction to Computer Networks & OSI MODEL.ppt

AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...

Prediction of Electrical Energy Efficiency Using Information on Consumer's Ac...

Ericsson LTE Throughput Troubleshooting Techniques.ppt

一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理

ITSM Integration with MuleSoft.pptx

Open Channel Flow: fluid flow with a free surface

3rd International Conference on Artificial Intelligence Advances (AIAD 2024)

paper relate Chozhavendhan et al. 2020.pdf

A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...

This study Examines the Effectiveness of Talent Procurement through the Imple...

SELENIUM CONF -PALLAVI SHARMA - 2024.pdf

UNIT-III- DATA CONVERTERS ANALOG TO DIGITAL CONVERTER

Butterfly Valves Manufacturer (LBF Series).pdf

Blood finder application project report (1).pdf

Beckhoff Programmable Logic Control Overview Presentation

ELS: 2.4.1 POWER ELECTRONICS Course objectives: This course will enable stude...

309475979-Creativity-Innovation-notes-IV-Sem-2016-pdf.pdf

Particle Swarm Optimization–Long Short-Term Memory based Channel Estimation w...

一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理

Monitoring anomalies in experimentation platform.

1. Monitoring Anomalies in Experimentation Platform Deepak Vasthimal – MTS @ eBay Connect Me - https://www.linkedin.com/in/whatisdeepak Available on eBay Tech Blog - https://goo.gl/6bUbE9

2. Overview of Experimentation (A/B Test) Platform • A/B Testing is comparing different experiences and measure their performance. • Variance could be in UI, Components, Algorithms etc. • Measure Bankable metrics and non-bankable (activity click rates). • Enables data driven decisions. • Avoid making releases features on intuitions to over 150 million users. Monitoring Anomalies in the Experimentation Platform 2

3. Experimentation Reporting • 1500+ experiments • Migrated from Teradata/SQL to Hadoop using Scala. • Process 100s of TBs of data daily on Hadoop cluster with around 400 M/R jobs. • 200+ metrics generated daily using batch system. • Built using a mix of open source technologies like Scala, Scoobi, Hive, Hadoop and proprietary tech like Teradata and MicroStrategy. Monitoring Anomalies in the Experimentation Platform 3

4. High Level Flow Monitoring Anomalies in the Experimentation Platform 4

5. Anomalies with Experiments • Traffic corruption – Traffic between test and control is skewed by UID/GUID. • Tag corruption – Data loss/corrupted during logging & transfer to HDFS. • GUID Reset – Browser cookie. • Cache refresh - eBay application servers maintain caches of experiment configurations. A software or hardware glitch can cause corruption of cache. Monitoring Anomalies in the Experimentation Platform 5

6. Monitoring Anomalies • Identify & Categorize anomalies within experiments using Teradata/Hive. • Store identified anomalies in HDFS and route to InfluxDB (TSDB) •Visualize using Grafana. •I was introduced to Grafana through SpaceX tweet by Torkel. Monitoring Anomalies in the Experimentation Platform 6

7. Reason we chose Grafana • Visually pleasing graphs. • Easy setup. • In Built Query Editor (SQL & UI) • Instantly change dashboards using duplicate dashboards/panels feature. • InfluxDB for its ability to be setup in minutes when compared to Graphite/Prometheus . • Entire pipeline took couple of days. Monitoring Anomalies in the Experimentation Platform 7

8. Home Page Monitoring Anomalies in the Experimentation Platform 8

9. DrillDown Monitoring Anomalies in the Experimentation Platform 9

10. Search Monitoring Anomalies in the Experimentation Platform 10

11. Scale • InfluxDB (v 0.11-1) is installed on a single node with 45 GB of memory. • Grafana (v 3.0.2) is installed on a single node with 45 GB of memory. • 2000 points are ingested daily which is minuscule. • Currently have around 10 months of historical data. Monitoring Anomalies in the Experimentation Platform 11

12. Grafana <3 is spreading. • Performance analysis of MapReduce jobs by Experimentation platform with job counters. • Monitor Elastic Search Cluster (10k+ points per second). • Anomaly detection in Tracking data (10k+ points per minute). • Each use case stores data in InfluxDB. Monitoring Anomalies in the Experimentation Platform 12

13. Thank You Monitoring Anomalies in the Experimentation Platform 13

Monitoring anomalies in experimentation platform.

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to Monitoring anomalies in experimentation platform.

Similar to Monitoring anomalies in experimentation platform. (20)

Recently uploaded

Recently uploaded (20)

Monitoring anomalies in experimentation platform.