Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity

•

2 likes•1,465 views

**NOTE: Catch the demo part of the presentation on the Lightbend YouTube channel: https://youtu.be/upjvaDQ8N6s** In this webinar with Lightbend Principal Engineer, David Brinegar, we review how OpsClarity is purpose-built to intelligently monitor streaming applications and help Ops teams resolve issues quickly. Included in the video is a 20-minute demo showing out of the box operational metrics, end-to-end visibility into data pipelines and the ability to correlate problems and issues across various streaming components.

Agenda
1. Fast Data & Streaming Applications
2. The Challenges of Monitoring Fast Data Applications
3. What To Look For In a Fast Data Application Monitoring Solution
4. Intelligent End-To-End Monitoring from Lightbend
5. Live Demo
6. Questions

reactivemanifesto.org
Reactive Underpinnings: Fast Data and streaming applications
often incorporate, or are based on, Reactive principles

• Rapidly Evolving Ecosystem
• Understanding the Data Pipeline
• Dynamic Architectures
• Intricately Interconnected
• Distributed And Clustered
The Challenges of Monitoring Fast Data Applications

Apache Spark, As An Illustrative Example
The Challenges of Monitoring Fast Data Applications
Concern Questions To Ask
Data Health (for a
given application)
• Throughput: is data processing occurring at the expected rate?
• Latency: is data processing occurring within the expected timeframe?
• Error/quality: are there problems with the data being produced?
• Input data: are input data streams flowing into Spark behaving normally? For instance, what are the
throughput rates for Kafka topics feeding into the Spark job?
Dependency Health • Are the systems feeding input into the spark job (such as Kafka) healthy?
• Are the systems that the application is dependent on, such as Memcache or other API endpoints,
healthy?
Service Health • Is the Spark master operating normally? If not, engineering will be unable to re-balance workloads or
restart jobs.
Application Health • Are the application KPIs within normal operating parameters?
Topology Health • Are there resources assigned to the given Spark topology?
• • Are the Spark tasks and executors well-distributed amongst the Spark cluster?
• • Are the performance counters (emitted, failed, latency, etc.) for the given Spark topology normal?
Node System Health • Are the key system metrics (load, CPU, memory, net-i/o, disk-i/o, disk free) operating normally?

Modern monitoring for modern applications

Why traditional monitoring tools won’t help you
• Built to monitor monolithic
applications
• Can only be used to extract
metrics and trace information
based on a synchronous flow
• Not built for asynchronous
flows (i.e. in Fast Data and
streaming applications)
• Cannot easily handle streaming
systems running on distributed
clusters

• Automatic Telemetry
• Visual Maps
• Intelligent, Rapid
Troubleshooting
What users need to effectively monitor Fast Data and
streaming applications

• Deep Telemetry
• Domain Expertise
• Intelligent Anomaly
Detection
• Fine-Grained
Visibility, with Drill-
Down Capabilities
Data-Science Driven Anomaly Detection

• Automated Topology
Discovery
• Automatic Metric
Collection
• Real-Time Topology
Visualization
Automated Discovery, Configuration & Topology
Visualization

• Single Pane of Glass
Visibility
• Rapid Root Cause
Analysis
• Reduced Mean-Time-
To-Repair (MTTR)
Intelligent, Rapid Troubleshooting

• Dramatically reduce the time and cost to identify and remediate issues across
application life-cycle. 
• Create happier, more satisfied customers – and lower churn
• Lower HW/infrastructure costs and reduce concerns about chargebacks & SLA
penalties
• Deliver rapid time to value because everything you need for monitoring is packaged
into an easy-to-use solution
Benefits for your business

If you’re serious about having end-to-end monitoring
for your Fast Data and streaming applications,  
let’s chat!
SET UP A 20-MIN DEMO

Apache Spark 2.0 introduced Structured Streaming which allows users to continually and incrementally update your view of the world as new data arrives while still using the same familiar Spark SQL abstractions. Michael Armbrust from Databricks talks about the progress made since the release of Spark 2.0 on robustness, latency, expressiveness and observability, using examples of production end-to-end continuous applications. Speaker: Michael Armbrust Video: http://go.databricks.com/videos/spark-summit-east-2017/using-structured-streaming-apache-spark This talk was originally presented at Spark Summit East 2017.

Case Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud

Rick Bilodeau

Blind spots in big data erez koren @ forter

Ido Shilon

Anomaly Detection using Spark MLlib and Spark Streaming

Keira Zhou

Databus - LinkedIn's Change Data Capture Pipeline

Sunil Nagaraj

Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models

Anyscale

Apache Spark has rapidly become a key tool for data scientists to explore, understand and transform massive datasets and to build and train advanced machine learning models. The question then becomes, how do I deploy these model to a production environment? How do I embed what I have learned into customer facing data applications? In this webinar, we will discuss best practices from Databricks on how our customers productionize machine learning models do a deep dive with actual customer case studies, show live tutorials of a few example architectures and code in Python, Scala, Java and SQL.

Data Infrastructure at LinkedIn

Amy W. Tang

Using Spark Mllib Models in a Production Training and Serving Platform: Exper...

Databricks

Overview Uber's Michelangelo is a machine learning platform that supports training and serving thousands of models in production. Most Michelangelo customer models are based on Spark Mllib. In this talk, we will describe Michelangelo's experiences with and evolving use of Spark Mllib, particularly in the areas of model persistence and online serving. Extended Description Michelangelo [https://eng.uber.com/michelangelo/] was originally developed to support scalable machine learning for production models. Its end-to-end support for scheduled Spark-based data ingestion and model training, along with model evaluation and deployment for batch and online model serving, has gained wide acceptance across Uber. More recently, Michelangelo is evolving to handle more use cases, including evaluating and serving models trained outside of core Michelangelo, e.g., on a distributed tensorflow platform providing Horovod [https://eng.uber.com/horovod/] or using PySpark in a Jupyter notebook on Data Science Workbench [https://eng.uber.com/dsw/] To support evaluation and serving of models trained outside of Michelangelo, Michelangelo's use of Spark Mllib needed updating, to generalize its mechanisms for model persistence and online serving. In this talk, we will describe these mechanisms and explore possible avenues for open-sourcing them. Speakers: Anne Holler, Michael Mui

"We all know the unprecedented potential impact for Machine Learning. But how do you take advantage of the myriad of data and ML tools now available? How do you streamline processes, speed up discovery, share knowledge, and scale up implementations for real-life scenarios? In this talk, we'll cover some of the latest innovations brought into the Databricks Unified Analytics Platform for Machine Learning. In particular we will show you how to: - Get started quickly using the Databricks Runtime for Machine Learning, that provides pre-configured Databricks clusters including the most popular ML frameworks and libraries, Conda support, performance optimizations, and more. - Get started with most popular Deep Learning frameworks within a few minutes and go deep with state of the art model DL diagnostics tools. - Scale up Deep Learning training workloads from a single machine to large clusters for the most demanding applications using the new HorovodRunner with ease. - How all of these ML frameworks get exposed to large and distributed data using Databricks Runtime for Machine Learning."

KFServing, Model Monitoring with Apache Spark and a Feature Store

Databricks

In recent years, MLOps has emerged to bring DevOps processes to the machine learning (ML) development process, aiming at more automation in the execution of repetitive tasks and at smoother interoperability between tools. Among the different stages in the ML lifecycle, model monitoring involves the supervision of model performance over time, involving the combination of techniques in four categories: outlier detection, data drift detection, explainability and adversarial attacks. Most existing model monitoring tools follow a scheduled batch processing approach or analyse model performance using isolated subsets of the inference data. However, for the continuous monitoring of models, stream processing platforms show several advantages, including support for continuous data analytics, scalable processing of large amounts of data and first-class support for window-based aggregations useful for concept drift detection. In this talk, we present an open-source platform for serving and monitoring models at scale based on Kubeflow’s model serving framework, KFServing, the Hopsworks Online Feature Store for enriching feature vectors with transformer in KFServing, and Spark and Spark Streaming as general purpose frameworks for monitoring models in production. We also show how Spark Streaming can use the Hopsworks Feature Store to implement continuous data drift detection, where the Feature Store provides statistics on the distribution of feature values in training, and Spark Streaming computes the statistics on live traffic to the model, alerting if the live traffic differs significantly from the training data. We will include a live demonstration of the platform in action.

Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...

Databricks

This talk describes migrating a large random forest classifier from scikit-learn to Spark's MLlib. We cut training time from 2 days to 2 hours, reduced failed runs, and track experiments better with MLflow. Kount provides certainty in digital interactions like online credit card transactions. One of our scores uses a random forest classifier with 250 trees and 100,000 nodes per tree. We used scikit-learn to train using 60 million samples that each contained over 150 features. The in-memory requirements exceeded 750 GB, took 2 days, and were not robust to disruption in our database or training execution. To migrate workflow to Spark, we built a 6-node cluster with HDFS. This provides 1.35 TB of RAM and 484 cores. Using MLlib and parallelization, the training time for our random forests are now less than 2 hours. Training data stays in our production environment, which used to require a deploy cycle to move locally-developed code onto our training server. The new implementation uses Jupyter notebooks for remote development with server-side execution. MLflow tracks all input parameters, code, and git revision number, while the performance and model itself are retained as experiment artifacts. The new workflow is robust to service disruption. Our training pipeline begins by pulling from a Vertica database. Originally, this single connection took over 8 hours to complete with any problem causing a restart. Using sqoop and multiple connections, we pull the data in 45 minutes. The old technique used volatile storage and required the data for each experiment. Now, we pull the data from Vertica one time and then reload much faster from HDFS. While a significant undertaking, moving to the Spark ecosystem converted an ad hoc and hands-on training process into a fully repeatable pipeline that meets regulatory and business goals for traceability and speed. Speaker: Josh Johnston

MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle

Databricks

Successfully building a machine learning model is hard enough. Reproducing your results at scale — enabling others to reproduce pipelines, comparing results from other versions, moving models into production, redeploying and rolling out updated models — is exponentially harder. To address these challenges and accelerate innovation, many companies are building custom “ML platforms” to automate the end-to-end ML lifecycle. Watch a replay of this MLOps Virtual Event to hear more about the latest developments and best practices for managing the full ML lifecycle on Databricks with MLflow. We covered a checklist of capabilities you’ll need, common pitfalls, technological and organizational challenges, and how to overcome them. https://www.youtube.com/playlist?list=PLTPXxbhUt-YUFNBwBsSIlknoNbS7GExZw

Insights Without Tradeoffs Using Structured Streaming keynote by Michael Armb...

Spark Summit

Lessons From HPE: From Batch To Streaming For 20 Billion Sensors With Lightbe...

Lightbend

Revolution R Enterprise 7.4 - Presentation by Bill Jacobs 11Jun15

Revolution Analytics

R at Microsoft

Revolution Analytics

Challenges of Operationalising Data Science in Production

iguazio

The presentation topic for this meet-up was covered in two sections without any breaks in-between Section 1: Business Aspects (20 mins) Speaker: Rasmi Mohapatra, Product Owner, Experian https://www.linkedin.com/in/rasmi-m-428b3a46/ Once your data science application is in the production, there are many typical data science operational challenges experienced today - across business domains - we will cover a few challenges with example scenarios Section 2: Tech Aspects (40 mins, slides & demo, Q&A ) Speaker: Santanu Dey, Solution Architect, Iguazio https://www.linkedin.com/in/santanu/ In this part of the talk, we will cover how these operational challenges can be overcome e.g. automating data collection & preparation, making ML models portable & deploying in production, monitoring and scaling, etc. with relevant demos.

Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...

Databricks

The BigDL framework scales deep learning for large data sets using Apache Spark. However there is significant scheduling overhead from Spark when running BigDL at large scale. In this talk we propose a new parameter manager implementation that along with coarse-grained scheduling can provide significant speedups for deep learning models like Inception, VGG etc. Aggregation functions like reduce or treeReduce that are used for parameter aggregation in Apache Spark (and the original MapReduce) are slow as the centralized scheduling and driver network bandwidth become a bottleneck especially in large clusters. To reduce the overhead of parameter aggregation and allow for near-linear scaling, we introduce a new AllReduce operation, a part of the parameter manager in BigDL which is built directly on top of the BlockManager in Apache Spark. AllReduce in BigDL uses a peer-to-peer mechanism to synchronize and aggregate parameters. During parameter synchronization and aggregation, all nodes in the cluster play the same role and driver’s overhead is eliminated thus enabling near-linear scaling. To address the scheduling overhead we use Drizzle, a recently proposed scheduling framework for Apache Spark. Currently, Spark uses a BSP computation model, and notifies the scheduler at the end of each task. Invoking the scheduler at the end of each task adds overheads and results in decreased throughput and increased latency. Drizzle introduces group scheduling, where multiple iterations (or a group) of iterations are scheduled at once. This helps decouple the granularity of task execution from scheduling and amortizes the costs of task serialization and launch. Finally we will present results from using the new AllReduce operation and Drizzle on a number of common deep learning models including VGG and Inception. Our benchmarks run on Amazon EC2 and Google DataProc will show the speedups and scalability of our implementation.

Streaming Data Ingest and Processing with Apache Kafka

Attunity

Apache™ Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system. It offers higher throughput, reliability and replication. To manage growing data volumes, many companies are leveraging Kafka for streaming data ingest and processing. Join experts from Confluent, the creators of Apache™ Kafka, and the experts at Attunity, a leader in data integration software, for a live webinar where you will learn how to: -Realize the value of streaming data ingest with Kafka -Turn databases into live feeds for streaming ingest and processing -Accelerate data delivery to enable real-time analytics -Reduce skill and training requirements for data ingest The recorded webinar on slide 32 includes a demo using automation software (Attunity Replicate) to stream live changes from a database into Kafka and also includes a Q&A with our experts. For more information, please go to www.attunity.com/kafka.

Fighting Fraud with Apache Spark

Miklos Christine

Spark ML Pipeline serving

Stepan Pushkarev

My talk at Data Science Labs conference in Odessa. Training a model in Apache Spark while having it automatically available for real-time serving is an essential feature for end-to-end solutions. There is an option to export the model into PMML and then import it into a separated scoring engine. The idea of interoperability is great but it has multiple challenges, such as code duplication, limited extensibility, inconsistency, extra moving parts. In this talk we discussed an alternative solution that does not introduce custom model formats and new standards, not based on export/import workflow and shares Apache Spark API.

How to Productionize Your Machine Learning Models Using Apache Spark MLlib 2....

Databricks

Apache Spark has rapidly become a key tool for data scientists to explore, understand and transform massive datasets and to build and train advanced machine learning models. The question then becomes, how do you deploy these ML model to a production environment? How do you embed what you’ve learned into customer facing data applications? In this talk I will discuss best practices on how data scientists productionize machine learning models, do a deep dive with actual case studies, and show live tutorials of a few example architectures and code in Python, Scala, Java and SQL.

Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Todd Fritz

Using PySpark to Process Boat Loads of Data

Robert Dempsey

Learn how to use PySpark for processing massive amounts of data. Combined with the GitHub repo - https://github.com/rdempsey/pyspark-for-data-processing - this presentation will help you gain familiarity with processing data using Python and Spark. If you're thinking about machine learning and not sure if it can help improve your business, but want to find out, set up a free 20-minute consultation with us: https://calendly.com/robertwdempsey/free-consultation

MLeap: Productionize Data Science Workflows Using Spark

Jen Aman

Innovation in the Enterprise Rent-A-Car Data Warehouse

DataWorks Summit

Big Data adoption is a journey. Depending on the business the process can take weeks, months, or even years. With any transformative technology the challenges have less to do with the technology and more to do with how a company adapts itself to a new way of thinking about data. Building a Center of Excellence is one way for IT to help drive success. This talk will explore Enterprise Holdings Inc. (which operates the Enterprise Rent-A-Car, National Car Rental and Alamo Rent A Car) and their experience with Big Data. EHI’s journey started in 2013 with Hadoop as a POC and today are working to create the next generation data warehouse in Microsoft’s Azure cloud utilizing a lambda architecture. We’ll discuss the Center of Excellence, the roles in the new world, share the things which worked well, and rant about those which didn’t. No deep Hadoop knowledge is necessary, architect or executive level.

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

Spark Summit

Here we present a general supervised framework for record deduplication and author-disambiguation via Spark. This work differentiates itself by – Application of Databricks and AWS makes this a scalable implementation. Compute resources are comparably lower than traditional legacy technology using big boxes 24/7. Scalability is crucial as Elsevier’s Scopus data, the biggest scientific abstract repository, covers roughly 250 million authorships from 70 million abstracts covering a few hundred years. – We create a fingerprint for each content by deep learning and/or word2vec algorithms to expedite pairwise similarity calculation. These encoders substantially reduce compute time while maintaining semantic similarity (unlike traditional TFIDF or predefined taxonomies). We will briefly discuss how to optimize word2vec training with high parallelization. Moreover, we show how these encoders can be used to derive a standard representation for all our entities namely such as documents, authors, users, journals, etc. This standard representation can simplify the recommendation problem into a pairwise similarity search and hence it can offer a basic recommender for cross-product applications where we may not have a dedicate recommender engine designed. – Traditional author-disambiguation or record deduplication algorithms are batch-processing with small to no training data. However, we have roughly 25 million authorships that are manually curated or corrected upon user feedback. Hence, it is crucial to maintain historical profiles and hence we have developed a machine learning implementation to deal with data streams and process them in mini batches or one document at a time. We will discuss how to measure the accuracy of such a system, how to tune it and how to process the raw data of pairwise similarity function into final clusters. Lessons learned from this talk can help all sort of companies where they want to integrate their data or deduplicate their user/customer/product databases.

1200x630 1

MonicaEscobar55

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

Yahoo Developer Network

Spark and SQL-on-Hadoop have made it easier than ever for enterprises to create or migrate apps to the big data stack. Thousands of apps are being generated every day in the form of ETL and modeling pipelines, business intelligence and data cubes, deep machine learning, graph analytics, and real-time data streaming. However, the task of reliably operationalizing these big data apps involves many painpoints. Developers may not have the experience in distributed systems to tune apps for efficiency and performance. Diagnosing failures or unpredictable performance of apps can be a laborious process that involves multiple people. Apps may get stuck or steal resources and cause mission-critical apps to miss SLAs. This talk with introduce the audience to these problems and their common causes. We will also demonstrate how to find and fix these problems quickly, as well as prevent such problems from happening in the first place. Speakers: Dr. Shivnath Babu is a Co-founder and CTO of Unravel and Associate Professor of Computer Science at Duke University. With more than a decade of experience researching the ease of use and manageability of data-intensive systems, he leads the Starfish project at Duke, which pioneered the automation of Hadoop application tuning, problem diagnosis, and resource management. Shivnath has more than 80 peer-reviewed publications to his credit and has received the U.S. National Science Foundation CAREER Award, the HP Labs Innovation Award, and three IBM Faculty Awards.

Using Machine Learning to Understand Kafka Runtime Behavior (Shivanath Babu, ...

confluent

Apache Kafka is now nearly ubiquitous in modern data pipelines and use cases. While the Kafka development model is elegantly simple, operating Kafka clusters in production environments is a challenge. It’s hard to troubleshoot misbehaving Kafka clusters, especially when there are potentially hundreds or thousands of topics, producers and consumers and billions of messages. The root cause of why real-time applications is lag may be due to an application problem – like poor data partitioning or load imbalance – or due to a Kafka problem – like resource exhaustion or suboptimal configuration. Therefore getting the best performance, predictability, and reliability for Kafka-based applications can be difficult. In the end, the operation of your Kafka powered analytics pipelines could themselves benefit from machine learning (ML).

What's hot

Accelerating Machine Learning on Databricks Runtime

Databricks

KFServing, Model Monitoring with Apache Spark and a Feature Store

Databricks

Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...

Databricks

MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle

Databricks

Insights Without Tradeoffs Using Structured Streaming keynote by Michael Armb...

Spark Summit

Lessons From HPE: From Batch To Streaming For 20 Billion Sensors With Lightbe...

Lightbend

Revolution R Enterprise 7.4 - Presentation by Bill Jacobs 11Jun15

Revolution Analytics

R at Microsoft

Revolution Analytics

Challenges of Operationalising Data Science in Production

iguazio

Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...

Databricks

Streaming Data Ingest and Processing with Apache Kafka

Attunity

Fighting Fraud with Apache Spark

Miklos Christine

Spark ML Pipeline serving

Stepan Pushkarev

How to Productionize Your Machine Learning Models Using Apache Spark MLlib 2....

Databricks

Apache Spark has rapidly become a key tool for data scientists to explore, understand and transform massive datasets and to build and train advanced machine learning models. The question then becomes, how do you deploy these ML model to a production environment? How do you embed what you’ve learned into customer facing data applications? In this talk I will discuss best practices on how data scientists productionize machine learning models, do a deep dive with actual case studies, and show live tutorials of a few example architectures and code in Python, Scala, Java and SQL.

Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Todd Fritz

Using PySpark to Process Boat Loads of Data

Robert Dempsey

MLeap: Productionize Data Science Workflows Using Spark

Jen Aman

Innovation in the Enterprise Rent-A-Car Data Warehouse

DataWorks Summit

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

Spark Summit

1200x630 1

MonicaEscobar55

What's hot (20)

Accelerating Machine Learning on Databricks Runtime

KFServing, Model Monitoring with Apache Spark and a Feature Store

Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...

MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle

Insights Without Tradeoffs Using Structured Streaming keynote by Michael Armb...

Lessons From HPE: From Batch To Streaming For 20 Billion Sensors With Lightbe...

Revolution R Enterprise 7.4 - Presentation by Bill Jacobs 11Jun15

R at Microsoft

Challenges of Operationalising Data Science in Production

Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...

Streaming Data Ingest and Processing with Apache Kafka

Fighting Fraud with Apache Spark

Spark ML Pipeline serving

How to Productionize Your Machine Learning Models Using Apache Spark MLlib 2....

Reactive Fast Data & the Data Lake with Akka, Kafka, Spark

Using PySpark to Process Boat Loads of Data

MLeap: Productionize Data Science Workflows Using Spark

Innovation in the Enterprise Rent-A-Car Data Warehouse

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

1200x630 1

Similar to Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

Yahoo Developer Network

Using Machine Learning to Understand Kafka Runtime Behavior (Shivanath Babu, ...

confluent

I Heart Log: Real-time Data and Apache Kafka

Jay Kreps

Data Streaming For Big Data

Seval Çapraz

Monitoring microservices lightning ddd north 20171014

Sean Farmar

Hands-on Machine Learning Using Healthcare

Health Catalyst

Levi Thatcher, Health Catalyst Director of Data Science and his team provide a live demonstration using healthcare.ai to implement a healthcare-specific machine learning model from data source to patient impact. Levi goes through a hands-on coding example while sharing his insights on the value of predictive analytics, the best path towards implementation, and avoiding common pitfalls. Frequently asked questions are answered during the session. During the webinar, we will: Describe and install healthcare.ai Build and evaluate a machine learning model Deploy interpretable predictions to SQL Server Discuss the process of deploying into a live analytics environment. If you’d like to follow along, you should download and install R and RStudio prior to the event. We look forward to you joining us!

Observability in real time at scale

Balvinder Hira

The rise of cloud and containers has led to systems that are much more distributed and dynamic in nature. Highly elastic microservice and serverless architectures mean containers spin up on demand and scale to zero when that demand goes away. This generates a continous stream of infrastructure data. On the business side, we have started storing lot of data and this data contains enormours information, specially when married with infrastructure data this gives holistic health information of the entire platform. We will talk about how to achieve this kind of fine-grained observability at scale in real-time.

Data Engineering.pdf

Datacademy.ai

Data Engineering is the process of collecting, transforming, and loading data into a database or data warehouse for analysis and reporting. It involves designing, building, and maintaining the infrastructure necessary to store, process, and analyze large and complex datasets. This can involve tasks such as data extraction, data cleansing, data transformation, data loading, data management, and data security. The goal of data engineering is to create a reliable and efficient data pipeline that can be used by data scientists, business intelligence teams, and other stakeholders to make informed decisions. Visit by :- https://www.datacademy.ai/what-is-data-engineering-data-engineering-data-e/

Automating the process of continuously prioritising data, updating and deploy...

Ola Spjuth

Presentation at Data Innovation Summit 2019 in Stockholm, Sweden. ABSTRACT Microscopes are capable of producing vast amounts of data, and when used in automated laboratories both the number and size of images present many challenges for storing, categorizing, analyzing, annotating, and transforming the data into actionable information that can used for decision making; either by humans or machines. In this presentation I will describe the informatics system we have established at the Department of Pharmaceutical Biosciences at Uppsala University, which consists of computational hardware (CPUs, GPUs, storage), middleware (Kubernetes), imaging database (OMERO), and workflow system (Pachyderm) to perform online prioritization of new data, as well as the continuous analytics system to automate the process from captured images to continuously updated and deployed AI models. The AI methodologies include Deep Learning models trained on image data, and conventional machine learning models trained on features extracted from images or chemical structures. Due to the microservice architecture the system is scalable and can be expanded using hybrid-architectures with cloud computing resources. The informatics system serves a robotized cell profiling setup with incubators, liquid handling and high-content microscopy. The lab is quite young and is targeting applications primarily in drug screening and toxicity assessment, with the aim to improve research using AI and intelligent design of experiments.

Medidata AMUG Meeting / Presentation 2013

Brock Heinz

Lecture 01 Evolution of Decision Support Systems

phanleson

Apache kafka

Zeeshan Khan

Data munging and analysisRaminder Singh

The UK National Chemical Database Service – an integration of commercial and ...

US Environmental Protection Agency (EPA), Center for Computational Toxicology and Exposure

At a time when the data explosion has simply been redefined as “Big”, the hurdles associated with building a subject-specific data repository for chemistry are daunting. Combining a multitude of non-standard data formats for chemicals, related properties, reactions, spectra etc., together with the confusion of licensing and embargoing, and providing for data exchange and integration with services and platforms external to the repository, the challenge is significant. This all at a time when semantic technologies are touted as the fundamental technology to enhance integration and discoverability. Funding agencies are demanding change, especially a change towards access to open data to parallel their expectations around Open Access publishing. The Royal Society of Chemistry has been funded by the Engineering and Physical Science Research of the UK to deliver a “chemical database service” for UK scientists. This presentation will provide an overview of the challenges associated with this project and our progress in delivering a chemistry repository capable of handling the complex data types associated with chemistry. The benefits of such a repository in terms of providing data to develop prediction models to further enable scientific discovery will be discussed and the potential impact on the future of scientific publishing will also be examined.

Performance Testing

Anu Shaji

Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassan...

Lightbend

It’s become clear to many business that the ability to extract real-time actionable insights from data is not only a source of competitive advantage, but also a way to defend their existing business models from disruption. So while legacy models such as nightly batch jobs aren’t disappearing, an era of fast, streaming data (aka “Fast Data”) is upon us, and represents the state of the art for gaining real-time perishable insights that can then be used to serve existing customers better, acquiring new markets and keep the competition at bay. That said, distributed, Fast Data architectures are much harder to build, and carry their own set of challenges. Enterprises looking to move quickly are presented with a growing ecosystem of technologies, which often delays fast decisions and provides its own set of risks: * With so many choices, what tools should you use? * How do you avoid making rookie mistakes? * What are the best patterns and practices for streaming applications? In this webinar with Sean Glover, Senior Consultant at Lightbend and industry veteran, we examine the rise of streaming systems built around Spark, Mesos, Akka, Cassandra and Kafka, their role in handling endless streams of data to gain real-time insights. Sean then reviews how the Lightbend Fast Data Platform (FDP) brings them together in a comprehensive, easy to use, integrated platform, which includes installation, integration, and monitoring tools tuned for various deployment scenarios, plus sample applications.

Providence: rapid vulnerability prevention

Salesforce Engineering

Talk given by Max Feldman, Product Security Engineer at Salesforce, at AppSec USA. One challenging aspect of achieving software security is the struggle to catch up with the speed of development and deployment. We built Providence with the goal of preventing obvious bugs from ever being deployed into production. Providence is a lightweight and scalable tool which finds bugs and anti-patterns of varying complexity from code commits, and we’ve used it to prevent vulnerabilities ranging from XSS, to access control issues, to XXE. It works by continuously monitoring and pulling commits from version control systems and scanning them for bugs with rules defined in plugins. Additional plugins are easy to create and deploy, which has allowed for quick reaction to new bugs or problems as they are discovered. Providence is easily integrated with SDLC workflows or bug-tracking tools, and we will discuss how we have integrated it in-house in an unobtrusive manner. This model of addressing issues also provides relative immediacy of resolution; on average, potential problems found by Providence are resolved more quickly than other vulnerabilities because developers are presented the issues right after they commit the code, instead of weeks to months later. We are currently in the process of open-sourcing Providence in order to share the tool with the DevOps/security community (or any interested parties). This talk will cover the internals of Providence, its engine and plugin architecture (including examples of plugins and their ease of creation), as well as its integration with our SDLC and the faster and more efficient responses we’ve achieved as a result. We’re continuing to build new plugins and features, and we’re excited see what ideas others may have in mind!

Nicola Pagni - Anomaly Detection in Elasticsearch

MeetupDataScienceRoma

Developing high frequency indicators using real time tick data on apache supe...

Zekeriya Besiroglu

Observing Intraday Indicators Using Real-Time Tick Data on Apache Superset an...

DataWorks Summit

The Central Bank of the Republic of Turkey is primarily responsible for steering the monetary and exchange rate policies in Turkey. One of the major core functions of the Bank is market operations. In this context, analyzing and interpreting real-time tick data related to money market instruments has become not only a requirement but also a challenge. For this use case, an API provided by one of the financial data vendors has been used to gather real-time tick data and data routing has been orchestrated by Apache NiFi. Gathered data is being transferred to Kafka topics and then handed off to Druid for real-time indexing tasks. Indicators such as effective cost, bid-ask spread, price impact measures, return reversal are calculated using Apache Storm and finally visualized by means of Apache Superset in order to provide decision-makers with a new set of tools.

Similar to Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity (20)

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

Using Machine Learning to Understand Kafka Runtime Behavior (Shivanath Babu, ...

I Heart Log: Real-time Data and Apache Kafka

Data Streaming For Big Data

Monitoring microservices lightning ddd north 20171014

Hands-on Machine Learning Using Healthcare

Observability in real time at scale

Data Engineering.pdf

Automating the process of continuously prioritising data, updating and deploy...

Medidata AMUG Meeting / Presentation 2013

Lecture 01 Evolution of Decision Support Systems

Apache kafka

Data munging and analysis

The UK National Chemical Database Service – an integration of commercial and ...

Performance Testing

Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassan...

Providence: rapid vulnerability prevention

Nicola Pagni - Anomaly Detection in Elasticsearch

Developing high frequency indicators using real time tick data on apache supe...

Observing Intraday Indicators Using Real-Time Tick Data on Apache Superset an...

More from Lightbend

IoT 'Megaservices' - High Throughput Microservices with Akka

Lightbend

********** Watch this presentation on-demand! https://info.lightbend.com/iot-megaservices-high-throughput-microservices-with-akka-register.html ********** In this interactive presentation by Hugh McKee, Developer Advocate at Lightbend, we’ll share our experiences helping our clients create a system architecture that can support high throughput microservices (aka "Megaservices"). We’ll do that using IoT demo applications designed to push cloud service providers like Amazon and Google to their limits. Using sample code that you can later run on your own machine, we’ll look at: * Modeling real-life digital twins for hundreds of thousands of IoT devices in the field, looking into how these megaservices are implemented in Akka. * Visualizing Akka Actors–which represent IoT digital twins–in a “crop circle” formation that represents a complete distributed Reactive application, and watching at messages are processed across Akka Cluster nodes using cluster sharding. * Some code behind the whole set up, which is built using OSS like Akka, Java, JavaScript, and Kubernetes. Follow us on social: TW: https://twitter.com/lightbend LI: https://www.linkedin.com/company/lightbend-inc-/ FB: https://www.facebook.com/lightbendOfficial/ For more about Lightbend: Blog: https://www.lightbend.com/blog  Newsletter: https://www.lightbend.com/newsletter 

How Akka Cluster Works: Actors Living in a Cluster

Lightbend

The Reactive Principles: Eight Tenets For Building Cloud Native Applications

Lightbend

Putting the 'I' in IoT - Building Digital Twins with Akka Microservices

Lightbend

Akka at Enterprise Scale: Performance Tuning Distributed Applications

Lightbend

Organizations like Starbucks, HPE, and PayPal (see our customers) have selected the Akka toolkit for their enterprise scale distributed applications; and when it comes to squeezing out the best possible performance, the secret is using two particular modules in tandem: Akka Cluster and Akka Streams. In this webinar by Nolan Grace, Senior Solution Architect at Lightbend, we look at these two Akka modules and discuss the features that will push your application architecture to the next tier of performance. For the full blog post, including the video, visit: https://www.lightbend.com/blog/akka-at-enterprise-scale-performance-tuning-distributed-applications

Digital Transformation with Kubernetes, Containers, and Microservices

Lightbend

Detecting Real-Time Financial Fraud with Cloudflow on Kubernetes

Lightbend

Deploying a robust streaming data pipeline can be a daunting task when your company’s financial information is at risk. For starters, how do you ensure proper provisioning of resources? How do you preserve end-to-end application and data consistency? How do you make all of this work in the cloud with Kubernetes and avoid YAML hell? Answer: Cloudflow, a new open-source toolkit for simplifying the development, deployment, and operation of streaming data pipelines.

Cloudstate - Towards Stateful Serverless

Lightbend

Digital Transformation from Monoliths to Microservices to Serverless and Beyond

Lightbend

Join this highly-visual presentation by Hugh McKee, Developer Advocate at Lightbend, to learn more about the ramifications and opportunities along the evolution from monolithic systems, to microservices architectures, to serverless (FaaS). See the video presentation on the Lightbend blog at: https://www.lightbend.com/blog/digital-transformation-from-monoliths-to-microservices-to-serverless-and-beyond

Akka Anti-Patterns, Goodbye: Six Features of Akka 2.6

Lightbend

How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...

Lightbend

Microservices, Kubernetes, and Application Modernization Done Right

Lightbend

In this talk by David Ogren, Enterprise Architect at Lightbend, we draw from experiences helping our clients successfully create, migrate to, and manage cloud-native system architectures. We look at some of the common pitfalls and anti-patterns of modernization efforts, and some of the best practices for taking an incremental approach to transforming legacy systems. See the full post with video on the Lightbend blog: https://www.lightbend.com/blog/microservices-kubernetes-application-modernization

Full Stack Reactive In Practice

Lightbend

Akka and Kubernetes: A Symbiotic Love Story

Lightbend

Scala 3 Is Coming: Martin Odersky Shares What To Know

Lightbend

Migrating From Java EE To Cloud-Native Reactive Systems

Lightbend

A lot of businesses that never before considered themselves as “technology companies” are now faced with digital modernization imperatives that force them to rethink their application and infrastructure architecture. On the path to becoming a digital, on-demand provider, development speed is the ultimate competitive advantage. This presents challenges to many organizations that have huge investments in legacy Java EE infrastructure, where technical debt and monolithic system architectures require modernization in order to confront various business risks. Usually, changes need to be made within existing frameworks to keep pace with new web-scale organizations. If your legacy monolith is no longer serving the expanding needs of your business, then join Markus Eisele, Director of Developer Advocacy at Lightbend, to learn what you can do to migrate from Java EE to cloud-native, Reactive systems—as defined by the Reactive Manifesto.

Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications

Lightbend

In this talk by Sean Glover, Principal Engineer at Lightbend, we will review how the Strimzi Kafka Operator, a supported technology in Lightbend Platform, makes many operational tasks in Kafka easy, such as the initial deployment and updates of a Kafka and ZooKeeper cluster. See the blog post containing the YouTube video here: https://www.lightbend.com/blog/running-kafka-on-kubernetes-with-strimzi-for-real-time-streaming-applications

Designing Events-First Microservices For A Cloud Native World

Lightbend

Scala Security: Eliminate 200+ Code-Level Threats With Fortify SCA For Scala

Lightbend

How To Build, Integrate, and Deploy Real-Time Streaming Pipelines On Kubernetes

Lightbend

More from Lightbend (20)

IoT 'Megaservices' - High Throughput Microservices with Akka

How Akka Cluster Works: Actors Living in a Cluster

The Reactive Principles: Eight Tenets For Building Cloud Native Applications

Putting the 'I' in IoT - Building Digital Twins with Akka Microservices

Akka at Enterprise Scale: Performance Tuning Distributed Applications

Digital Transformation with Kubernetes, Containers, and Microservices

Detecting Real-Time Financial Fraud with Cloudflow on Kubernetes

Cloudstate - Towards Stateful Serverless

Digital Transformation from Monoliths to Microservices to Serverless and Beyond

Akka Anti-Patterns, Goodbye: Six Features of Akka 2.6

How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...

Microservices, Kubernetes, and Application Modernization Done Right

Full Stack Reactive In Practice

Akka and Kubernetes: A Symbiotic Love Story

Scala 3 Is Coming: Martin Odersky Shares What To Know

Migrating From Java EE To Cloud-Native Reactive Systems

Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications

Designing Events-First Microservices For A Cloud Native World

Scala Security: Eliminate 200+ Code-Level Threats With Fortify SCA For Scala

How To Build, Integrate, and Deploy Real-Time Streaming Pipelines On Kubernetes

Recently uploaded

Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx

rickgrimesss22

Mobile App Development Company In Noida | Drona Infotech

Drona Infotech

Lecture 1 Introduction to games development

abdulrafaychaudhry

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

Aftab Hussain

Understanding variable roles in code has been found to be helpful by students in learning programming -- could variable roles help deep neural models in performing coding tasks? We do an exploratory study. - These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite

Google

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-pilot-review/ AI Pilot Review: Key Features ✅Deploy AI expert bots in Any Niche With Just A Click ✅With one keyword, generate complete funnels, websites, landing pages, and more. ✅More than 85 AI features are included in the AI pilot. ✅No setup or configuration; use your voice (like Siri) to do whatever you want. ✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It… ✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again. ✅ZERO Limits On Features Or Usages ✅Use Our AI-powered Traffic To Get Hundreds Of Customers ✅No Complicated Setup: Get Up And Running In 2 Minutes ✅99.99% Up-Time Guaranteed ✅30 Days Money-Back Guarantee ✅ZERO Upfront Cost See My Other Reviews Article: (1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review

Globus Compute Introduction - GlobusWorld 2024

Globus

Globus Connect Server Deep Dive - GlobusWorld 2024

Globus

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Google

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-fusion-buddy-review AI Fusion Buddy Review: Key Features ✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini ✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique! ✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs! ✅Fully automated AI articles bulk generation! ✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more. ✅With one keyword or URL, generate complete websites, landing pages, and more… ✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7. ✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches. ✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all! ✅Save over $5000 per year and kick out dependency on third parties completely! ✅Brand New App: Not available anywhere else! ✅ Beginner-friendly! ✅ZERO upfront cost or any extra expenses ✅Risk-Free: 30-Day Money-Back Guarantee! ✅Commercial License included! See My Other Reviews Article: (1) AI Genie Review: https://sumonreview.com/ai-genie-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review #AIFusionBuddyReview, #AIFusionBuddyFeatures, #AIFusionBuddyPricing, #AIFusionBuddyProsandCons, #AIFusionBuddyTutorial, #AIFusionBuddyUserExperience #AIFusionBuddyforBeginners, #AIFusionBuddyBenefits, #AIFusionBuddyComparison, #AIFusionBuddyInstallation, #AIFusionBuddyRefundPolicy, #AIFusionBuddyDemo, #AIFusionBuddyMaintenanceFees, #AIFusionBuddyNewbieFriendly, #WhatIsAIFusionBuddy?, #HowDoesAIFusionBuddyWorks

Atelier - Innover avec l’IA Générative et les graphes de connaissances

Neo4j

Atelier - Innover avec l’IA Générative et les graphes de connaissances Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement. Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...

Crescat

Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry. Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events. With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use. Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements. If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io

E-commerce Application Development Company.pdf

Hornet Dynamics

Navigating the Metaverse: A Journey into Virtual Evolution"

Donna Lenk

May Marketo Masterclass, London MUG May 22 2024.pdf

Adele Miller

Enhancing Research Orchestration Capabilities at ORNL.pdf

Globus

Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.

Large Language Models and the End of Programming

Matt Welsh

Pro Unity Game Development with C-sharp Book

abdulrafaychaudhry

Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...

Globus

Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

Mind IT Systems

BoxLang: Review our Visionary Licenses of 2024

Ortus Solutions, Corp

Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better

XfilesPro

Recently uploaded (20)

Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx

Mobile App Development Company In Noida | Drona Infotech

Lecture 1 Introduction to games development

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite

Globus Compute Introduction - GlobusWorld 2024

Globus Connect Server Deep Dive - GlobusWorld 2024

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Atelier - Innover avec l’IA Générative et les graphes de connaissances

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...

E-commerce Application Development Company.pdf

Navigating the Metaverse: A Journey into Virtual Evolution"

May Marketo Masterclass, London MUG May 22 2024.pdf

Enhancing Research Orchestration Capabilities at ORNL.pdf

Large Language Models and the End of Programming

Pro Unity Game Development with C-sharp Book

Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

BoxLang: Review our Visionary Licenses of 2024

Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better

Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity

2. Agenda 1. Fast Data & Streaming Applications 2. The Challenges of Monitoring Fast Data Applications 3. What To Look For In a Fast Data Application Monitoring Solution 4. Intelligent End-To-End Monitoring from Lightbend 5. Live Demo 6. Questions

5. reactivemanifesto.org Reactive Underpinnings: Fast Data and streaming applications often incorporate, or are based on, Reactive principles

6. • Rapidly Evolving Ecosystem • Understanding the Data Pipeline • Dynamic Architectures • Intricately Interconnected • Distributed And Clustered The Challenges of Monitoring Fast Data Applications

7. Apache Spark, As An Illustrative Example The Challenges of Monitoring Fast Data Applications Concern Questions To Ask Data Health (for a given application) • Throughput: is data processing occurring at the expected rate? • Latency: is data processing occurring within the expected timeframe? • Error/quality: are there problems with the data being produced? • Input data: are input data streams flowing into Spark behaving normally? For instance, what are the throughput rates for Kafka topics feeding into the Spark job? Dependency Health • Are the systems feeding input into the spark job (such as Kafka) healthy? • Are the systems that the application is dependent on, such as Memcache or other API endpoints, healthy? Service Health • Is the Spark master operating normally? If not, engineering will be unable to re-balance workloads or restart jobs. Application Health • Are the application KPIs within normal operating parameters? Topology Health • Are there resources assigned to the given Spark topology? • • Are the Spark tasks and executors well-distributed amongst the Spark cluster? • • Are the performance counters (emitted, failed, latency, etc.) for the given Spark topology normal? Node System Health • Are the key system metrics (load, CPU, memory, net-i/o, disk-i/o, disk free) operating normally?

8. Modern monitoring for modern applications

9. Why traditional monitoring tools won’t help you • Built to monitor monolithic applications • Can only be used to extract metrics and trace information based on a synchronous flow • Not built for asynchronous flows (i.e. in Fast Data and streaming applications) • Cannot easily handle streaming systems running on distributed clusters

10. • Automatic Telemetry • Visual Maps • Intelligent, Rapid Troubleshooting What users need to effectively monitor Fast Data and streaming applications

11. • Deep Telemetry • Domain Expertise • Intelligent Anomaly Detection • Fine-Grained Visibility, with Drill- Down Capabilities Data-Science Driven Anomaly Detection

12. • Automated Topology Discovery • Automatic Metric Collection • Real-Time Topology Visualization Automated Discovery, Configuration & Topology Visualization

13. • Single Pane of Glass Visibility • Rapid Root Cause Analysis • Reduced Mean-Time- To-Repair (MTTR) Intelligent, Rapid Troubleshooting

14. • Dramatically reduce the time and cost to identify and remediate issues across application life-cycle.  • Create happier, more satisfied customers – and lower churn • Lower HW/infrastructure costs and reduce concerns about chargebacks & SLA penalties • Deliver rapid time to value because everything you need for monitoring is packaged into an easy-to-use solution Benefits for your business

15. On to the demo…

16. lightbend.com/learn

17. If you’re serious about having end-to-end monitoring for your Fast Data and streaming applications,   let’s chat! SET UP A 20-MIN DEMO

Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity

Similar to Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity (20)

More from Lightbend

More from Lightbend (20)

Recently uploaded

Recently uploaded (20)

Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity