Oracle 12c Analytics New Features
•
1 like•1,445 views
The document discusses new features in Oracle 12c for analytics including: 1. Advanced analytics capabilities such as Oracle Data Mining, Oracle R Enterprise, and the ability to analyze large datasets of up to 10 billion rows in a week. 2. Performance improvements for partitioning, such as asynchronous global index maintenance and online partition movement. 3. Enhancements to the cost-based optimizer including dynamic sampling, cardinality feedback, and concurrent statistics gathering for faster query optimization. 4. Advanced analytics features like new data mining algorithms, predictive queries, and improved integration with Oracle Business Intelligence.
Report
Share
Report
Share
Download to read offline
Recommended
Efficiently Building Machine Learning Models for Predictive Maintenance in th...
For each drilling site, there are thousands of different equipment operating simultaneously 24/7. For the oil & gas industry, the downtime can cost millions of dollars daily. As current standard practice, the majority of the equipment are on scheduled maintenance with standby units to reduce the downtime.
The Critical Missing Component in the Production ML Stack
The day the ML application is deployed to production and begins facing the real world is the best and the worst day in the life of the model builder. The joy of seeing accurate predictions is quickly overshadowed by a myriad of operational challenges. Debugging, troubleshooting & monitoring takes over the majority of their day, leaving little time for model building. In DevOps, software operations are taken to a level of an art. Sophisticated tools enable engineers to quickly identify and resolve issues, continuously improving software stability and robustness. In the ML world, operations are still largely a manual process that involves Jupyter notebooks and shell scripts. One of the cornerstones of the DevOps toolchain is logging. Traces and metrics are built on top of logs enabling monitoring and feedback loops. What does logging look like in an ML system?
In this talk we will demonstrate how to enable data logging for an AI application using MLflow in a matter of minutes. We will discuss how something so simple enables testing, monitoring and debugging in an AI application that handles TBs of data and runs in real-time. Attendees will leave the talk equipped with tools and best practices to supercharge MLOps in their team.
Advanced SQL For Data Scientists
I will begin with a brief overview of SQL. Then the five major topics a data scientist should understand when working with relational databases: basic statistics in SQL, data preparation in SQL, advanced filtering and data aggregation, window functions, and preparing data for use with analytics tools.
Unified MLOps: Feature Stores & Model Deployment
If you’ve brought two or more ML models into production, you know the struggle that comes from managing multiple data sets, feature engineering pipelines, and models. This talk will propose a whole new approach to MLOps that allows you to successfully scale your models, without increasing latency, by merging a database, a feature store, and machine learning.
Splice Machine is a hybrid (HTAP) database built upon HBase and Spark. The database powers a one of a kind single-engine feature store, as well as the deployment of ML models as tables inside the database. A simple JDBC connection means Splice Machine can be used with any model ops environment, such as Databricks.
The HBase side allows us to serve features to deployed ML models, and generate ML predictions, in milliseconds. Our unique Spark engine allows us to generate complex training sets, as well as ML predictions on petabytes of data.
In this talk, Monte will discuss how his experience running the AI lab at NASA, and as CEO of Red Pepper, Blue Martini Software and Rocket Fuel, led him to create Splice Machine. Jack will give a quick demonstration of how it all works.
Oct 2011 CHADNUG Presentation on Hadoop
Josh Patterson gave a presentation on Hadoop and how it has been used. He discussed his background working on Hadoop projects including for the Tennessee Valley Authority. He outlined what Hadoop is, how it works, and examples of use cases. This includes how Hadoop was used to store and analyze large amounts of smart grid sensor data for the openPDC project. He discussed integrating Hadoop with existing enterprise systems and tools for working with Hadoop like Pig and Hive.
Enhancements on Spark SQL optimizer by Min Qiu
This document summarizes enhancements made to Spark SQL's optimizer including rule-based optimizations and cost-based optimizations. Rule-based optimizations included join condition push down through predicate rewrite and join order adjustment, as well as data volume reduction through column pruning enhancements. Cost-based optimizations leveraged statistics and histograms to select optimal join types, partitions, and join orders. Future work focuses on enumerating the full space of query plans and improving estimation accuracy.
Willump: Optimizing Feature Computation in ML Inference
Systems for performing ML inference are increasingly important, but are far slower than they could be because they use techniques designed for conventional data serving workloads, neglecting the statistical nature of ML inference. As an alternative, this talk presents Willump, an optimizer for ML inference.
Lazy Join Optimizations Without Upfront Statistics with Matteo Interlandi
Modern Data-Intensive Scalable Computing (DISC) systems such as Apache Spark do not support sophisticated cost-based query optimizers because they are specifically designed to process data that resides in external storage systems (e.g. HDFS), or they lack the necessary data statistics. Consequently, many crucial optimizations, such as join order and plan selection, are presently out-of-scope in these DISC system optimizers. Yet, join order is one of the most important decisions a cost-optimizer can make because wrong orders can result in a query response time that can become more than an order-of-magnitude slower compared to the better order.
Recommended
Efficiently Building Machine Learning Models for Predictive Maintenance in th...
For each drilling site, there are thousands of different equipment operating simultaneously 24/7. For the oil & gas industry, the downtime can cost millions of dollars daily. As current standard practice, the majority of the equipment are on scheduled maintenance with standby units to reduce the downtime.
The Critical Missing Component in the Production ML Stack
The day the ML application is deployed to production and begins facing the real world is the best and the worst day in the life of the model builder. The joy of seeing accurate predictions is quickly overshadowed by a myriad of operational challenges. Debugging, troubleshooting & monitoring takes over the majority of their day, leaving little time for model building. In DevOps, software operations are taken to a level of an art. Sophisticated tools enable engineers to quickly identify and resolve issues, continuously improving software stability and robustness. In the ML world, operations are still largely a manual process that involves Jupyter notebooks and shell scripts. One of the cornerstones of the DevOps toolchain is logging. Traces and metrics are built on top of logs enabling monitoring and feedback loops. What does logging look like in an ML system?
In this talk we will demonstrate how to enable data logging for an AI application using MLflow in a matter of minutes. We will discuss how something so simple enables testing, monitoring and debugging in an AI application that handles TBs of data and runs in real-time. Attendees will leave the talk equipped with tools and best practices to supercharge MLOps in their team.
Advanced SQL For Data Scientists
I will begin with a brief overview of SQL. Then the five major topics a data scientist should understand when working with relational databases: basic statistics in SQL, data preparation in SQL, advanced filtering and data aggregation, window functions, and preparing data for use with analytics tools.
Unified MLOps: Feature Stores & Model Deployment
If you’ve brought two or more ML models into production, you know the struggle that comes from managing multiple data sets, feature engineering pipelines, and models. This talk will propose a whole new approach to MLOps that allows you to successfully scale your models, without increasing latency, by merging a database, a feature store, and machine learning.
Splice Machine is a hybrid (HTAP) database built upon HBase and Spark. The database powers a one of a kind single-engine feature store, as well as the deployment of ML models as tables inside the database. A simple JDBC connection means Splice Machine can be used with any model ops environment, such as Databricks.
The HBase side allows us to serve features to deployed ML models, and generate ML predictions, in milliseconds. Our unique Spark engine allows us to generate complex training sets, as well as ML predictions on petabytes of data.
In this talk, Monte will discuss how his experience running the AI lab at NASA, and as CEO of Red Pepper, Blue Martini Software and Rocket Fuel, led him to create Splice Machine. Jack will give a quick demonstration of how it all works.
Oct 2011 CHADNUG Presentation on Hadoop
Josh Patterson gave a presentation on Hadoop and how it has been used. He discussed his background working on Hadoop projects including for the Tennessee Valley Authority. He outlined what Hadoop is, how it works, and examples of use cases. This includes how Hadoop was used to store and analyze large amounts of smart grid sensor data for the openPDC project. He discussed integrating Hadoop with existing enterprise systems and tools for working with Hadoop like Pig and Hive.
Enhancements on Spark SQL optimizer by Min Qiu
This document summarizes enhancements made to Spark SQL's optimizer including rule-based optimizations and cost-based optimizations. Rule-based optimizations included join condition push down through predicate rewrite and join order adjustment, as well as data volume reduction through column pruning enhancements. Cost-based optimizations leveraged statistics and histograms to select optimal join types, partitions, and join orders. Future work focuses on enumerating the full space of query plans and improving estimation accuracy.
Willump: Optimizing Feature Computation in ML Inference
Systems for performing ML inference are increasingly important, but are far slower than they could be because they use techniques designed for conventional data serving workloads, neglecting the statistical nature of ML inference. As an alternative, this talk presents Willump, an optimizer for ML inference.
Lazy Join Optimizations Without Upfront Statistics with Matteo Interlandi
Modern Data-Intensive Scalable Computing (DISC) systems such as Apache Spark do not support sophisticated cost-based query optimizers because they are specifically designed to process data that resides in external storage systems (e.g. HDFS), or they lack the necessary data statistics. Consequently, many crucial optimizations, such as join order and plan selection, are presently out-of-scope in these DISC system optimizers. Yet, join order is one of the most important decisions a cost-optimizer can make because wrong orders can result in a query response time that can become more than an order-of-magnitude slower compared to the better order.
Prácticas recomendadas en materia de arquitectura y errores que debes evitar
Crece con confianza. Desde la implementación de un pequeño nodo de desarrollo para la búsqueda de aplicaciones hasta la gestión de una gran implementación de cientos de nodos, nuestros expertos en Elastic te contarán todo lo que necesites saber.
Operationalizing Big Data Pipelines At Scale
Running a global, world-class business with data-driven decision making requires ingesting and processing diverse sets of data at tremendous scale. How does a company achieve this while ensuring quality and honoring their commitment as responsible stewards of data? This session will detail how Starbucks has embraced big data, building robust, high-quality pipelines for faster insights to drive world-class customer experiences.
Forecasting time series powerful and simple
Time series are a sequence of data points positioned in order of time. Time series forecasting has two main purposes - to understand the mechanisms that lead to rise or fall, and to predict future values. Very often it analyses trends, cyclical events, seasonality and has unique importance in Economics and Business. The quality of predictions can be evaluated only in future due to temporal dependencies on previous data points and there are many model types for approximation. In this session we are going to talk about challenges, ways of improvement and technology stack like ML.NET, ARIMA, Python, Azure ML, Regression and FB Prophet
From Batch to Streaming ET(L) with Apache Apex
Stream data processing is increasingly required to support business needs for faster actionable insight with growing volume of information from more sources. Apache Apex is a true stream processing framework for low-latency, high-throughput and reliable processing of complex analytics pipelines on clusters. Apex is designed for quick time-to-production, and is used in production by large companies for real-time and batch processing at scale.
This session will use an Apex production use case to walk through the incremental transition from a batch pipeline with hours of latency to an end-to-end streaming architecture with billions of events per day which are processed to deliver real-time analytical reports. The example is representative for many similar extract-transform-load (ETL) use cases with other data sets that can use a common library of building blocks. The transform (or analytics) piece of such pipelines varies in complexity and often involves business logic specific, custom components.
Topics include:
* Pipeline functionality from event source through queryable state for real-time insights.
* API for application development and development process.
* Library of building blocks including connectors for sources and sinks such as Kafka, JMS, Cassandra, HBase, JDBC and how they enable end-to-end exactly-once results.
* Stateful processing with event time windowing.
* Fault tolerance with exactly-once result semantics, checkpointing, incremental recovery
* Scalability and low-latency, high-throughput processing with advanced engine features for auto-scaling, dynamic changes, compute locality.
* Who is using Apex in production, and roadmap.
Following the session attendees will have a high level understanding of Apex and how it can be applied to use cases at their own organizations.
Time Series Databases for IoT (On-premises and Azure)
This document discusses choosing the right time series database for IoT data. It compares InfluxDB to SQL Server and other databases.
Some key points made:
- InfluxDB outperforms SQL Server for writes by 40x and queries by 59x for time series data due to its optimized design.
- InfluxDB uses 19x-26x less disk storage than SQL Server for the same data.
- InfluxDB also outperforms MongoDB, Elasticsearch, OpenTSDB, and Cassandra for time series workloads.
- Azure Stream Insights is a managed service but has limited capabilities and can be pricey for high volumes of data.
- InfluxDB is open source, has no dependencies, and
Big Data, Bigger Analytics
This document discusses using Apache Spark for big data analytics at an insurance pricing and customer analytics company called Earnix. It summarizes Earnix's business problems modeling large customer behavior data, how Spark helps address performance issues with their existing 10GB datasets, and improvements made to Spark's MLlib machine learning library. These include adding statistical functionality like covariance estimation to logistic regression models and optimizing algorithms to run efficiently on Spark. Benchmark results show Spark providing scalability by reducing algorithm run times as more nodes are added.
Continuous Evaluation of Deployed Models in Production Many high-tech industr...
Many high-tech industries rely on machine-learning systems in production environments to automatically classify and respond to vast amounts of incoming data. Despite their critical roles, these systems are often not actively monitored. When a problem first arises, it may go unnoticed for some time. Once it is noticed, investigating its underlying cause is a time-consuming, manual process. Wouldn’t it be great if the model’s output were automatically monitored? If they could be visualized, sliced by different dimensions? If the system could automatically detect performance degradation and trigger alerts? In this presentation, we describe our experience from building such a core machine-learning services: Model Evaluation.
Our service provides automated, continuous evaluation of the performance of a deployed model over commonly-used metrics like the area-under-the-curve (AUC), root-mean-square-error (RMSE) etc. In addition, summary statistics about the model’s output, their distributions are also computed. The service also provides a dashboard to visualize the performance metrics, summary statistics and distributions of a model over time along with REST APIs to retrieve these metrics programmatically.
These metrics can be sliced by input features (e.g. Geography, Product type) to provide insights into model performance over different segments. The talk will describe various components that are required in building such a service and metrics of interest. Our system has a backend component built with spark on Azure Databricks. The backend can scale to analyze TBs of data to generate model evaluation metrics.
We will talk about how we modified Spark MLLib for computing AUC sliced by different dimensions and other optimizations in Spark to improve compute and performance. Our front-end and middle-tier, built with Docker and Azure Webapp provides visuals and REST APIs to retrieve the above metrics. This talk will cover various aspects of building, deploying and using the above system.
Assessing Graph Solutions for Apache Spark
Users have several options for running graph algorithms with Apache Spark. To support a graph data architecture on top of its linear-oriented DataFrames, the Spark platform offers GraphFrames. However, due to the fact that GraphFrames are immutable and not a native graph, there are cases where it might not offer the features or performance needed for certain use cases. Another option is to connect Spark to a real-time, scalable and distributed native graph database such as TigerGraph.
In this session, we compare three options — GraphX, Cypher for Apache Spark, and TigerGraph — for different types of workload requirements and data sizes, to help users select the right solution for their needs. We also look at the data transfer and loading time for TigerGraph.
Predicting Optimal Parallelism for Data Analytics
A key benefit of serverless computing is that resources can be allocated on demand, but the quantity of resources to request, and allocate, for a job can profoundly impact its running time and cost. For a job that has not yet run, how can we provide users with an estimate of how the job’s performance changes with provisioned resources, so that users can make an informed choice upfront about cost-performance tradeoffs?
This talk will describe several related research efforts at Microsoft to address this question. We focus on optimizing the amount of computational resources that control a data analytics query’s achieved intra-parallelism. These use machine learning models on query characteristics to predict the run time or Performance Characteristic Curve (PCC) as a function of the maximum parallelism that the query will be allowed to exploit.
The AutoToken project uses models to predict the peak number of tokens (resource units) that is determined by the maximum parallelism that the recurring SCOPE job can ever exploit while running in Cosmos, an Exascale Big Data analytics platform at Microsoft. AutoToken_vNext, or TASQ, predicts the PCC as a function of the number of allocated tokens (limited parallelism). The AutoExecutor project uses models to predict the PCC for Apache Spark SQL queries as a function of the number of executors. The AutoDOP project uses models to predict the run time for SQL Server analytics queries, running on a single machine, as a function of their maximum allowed Degree Of Parallelism (DOP).
We will present our approaches and prediction results for these scenarios, discuss some common challenges that we handled, and outline some open research questions in this space.
26 Trillion App Recomendations using 100 Lines of Spark Code - Ayman Farahat
This document summarizes an approach to build an app recommendation engine using collaborative filtering on data from 53,793 apps and 496 million users. It implements collaborative filtering using 100 lines of Spark code to generate low-dimensional user and app features and compute 26 trillion user-app ratings. Key aspects included using data frames, caching joined data, broadcasting app features for efficient BLAS-3 matrix multiplication, and evaluating recommendations.
Superworkflow of Graph Neural Networks with K8S and Fugue
This document introduces a superworkflow for running Node2Vec on graphs using the Fugue framework on Kubernetes. It describes the Node2Vec algorithm and different steps in the superworkflow, including graph creation and indexing, random walks, Word2Vec preprocessing, and embedding training. The superworkflow provides advantages like parallelizing steps, efficient resource usage through auto-persist and checkpointing. Benchmark results show the superworkflow reduces runtime significantly compared to Spark MLlib, such as reducing a 100M node graph embedding from 6,800 CPU hours to 100 CPU hours and 16 GPU hours. Open source links for the Node2Vec on Fugue project are also provided.
Case Study Real Time Olap Cubes
Are OLAP cubes "large monters" that deliver quick data retrieval at the expense of long upload time? This presentation shows one way to kill this myth.
Stock Analyzer Hadoop MapReduce Implementation
This document presents a Hadoop project to analyze stock prices using MapReduce and Hive. The project aims to find the adjusted closing price for stock days without dividends by joining daily stock price and dividend datasets. The technical architecture uses Eclipse, Hadoop, and AWS EC2 for development and clustering. Pseudo code is provided for MapReduce jobs to parse input data and calculate adjusted closing prices for output. The results found adjusted closing prices for 44 records out of 75 total records by removing headers and dividend-only days.
A Production Quality Sketching Library for the Analysis of Big Data
In the analysis of big data there are often problem queries that don’t scale because they require huge compute resources to generate exact results, or don’t parallelize well.
Is This Thing On? A Well State Model for the People
The document discusses using machine learning models to determine well production state (on vs off) from sensor data. It presents an existing data architecture and issues with data quality. A supervised learning model is proposed using a decision tree trained on labeled rod pump production data. The modeling workflow includes data preprocessing, feature engineering, hyperparameter tuning and grid search. Decision trees are chosen for their interpretability but the document notes larger models may perform better. Overall production state modeling could help optimize operations and outperform existing controllers.
Koalas: How Well Does Koalas Work?
Koalas is an open source project that provides pandas APIs on top of Apache Spark. Pandas is the standard tool for data science and it is typically the first step to explore and manipulate a data set, but pandas does not scale well to big data. Koalas fills the gap by providing pandas equivalent APIs that work on Apache Spark.
There are also many libraries trying to scale pandas APIs, such as Vaex, Modin, and so on. Dask is one of them and very popular among pandas users, and also works on its own cluster similar to Koalas which is on top of Spark cluster. In this talk, we will introduce Koalas and its current status, and the comparison between Koalas and Dask, including benchmarking.
Large-Scaled Telematics Analytics in Apache Spark with Wayne Zhang and Neil P...
The increasing availability of mobile phones with embedded GPS devices and sensors has spurred the use of vehicle telematics in recent years. Telematics provides detailed and continuous information of a vehicle such as the location, speed, and movement. Vehicle telematics can be further linked with other spatial data to provide context to understand driving behaviors. The collection of high-frequency telematics data results in huge volumes of data that must be processed efficiently. We present a solution that uses Apache Spark to load and transform large-scaled telematics data. We then present how to use machine learning on telematics data to derive insights about driving safety.
MemSQL 201: Advanced Tips and Tricks Webcast
This document summarizes a webinar on advanced tips and tricks for MemSQL. It discusses the differences between rowstore and columnstore storage models and when each is best used. It also covers data ingestion using MemSQL Pipelines for real-time loading, data sharding and query tuning techniques like using reference tables. Additionally, it discusses monitoring memory usage, workload management using management views, and query optimization tools like analyzing and optimizing tables.
Cost-Based Optimizer in Apache Spark 2.2
The document discusses Apache Spark's new cost-based optimizer (CBO) in version 2.2. It describes how the CBO works in two key steps:
1. It collects and propagates statistics about tables and columns to estimate the cardinality of operations like filters, joins and aggregates.
2. It calculates the estimated cost of different execution plans and selects the most optimal plan based on minimizing the estimated cost. This allows it to pick more efficient join orders and join algorithms.
The document provides examples of how the CBO improves queries on TPC-DS benchmarks by producing smaller intermediate results and faster execution times compared to the previous rule-based optimizer in Spark 2.1.
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
The document discusses optimizations made to Spark SQL performance when working with Parquet files at ByteDance. It describes how Spark originally reads Parquet files and identifies two main areas for optimization: Parquet filter pushdown and the Parquet reader. For filter pushdown, sorting columns improved statistics and reduced data reads by 30%. For the reader, splitting it to first filter then read other columns prevented loading unnecessary data. These changes improved Spark SQL performance at ByteDance without changing jobs.
Best Practices for Supercharging Cloud Analytics on Amazon Redshift
In this webinar, we discuss how the secret sauce to your business analytics strategy remains rooted on your approached, methodologies and the amount of data incorporated into this critical exercise. We also address best practices to supercharge your cloud analytics initiatives, and tips and tricks on designing the right information architecture, data models and other tactical optimizations.
To learn more, visit: http://www.snaplogic.com/redshift-trial
Enhancements that will make your sql database roar sp1 edition sql bits 2017
This document provides information about various SQL Server features and editions. It includes a list of features available in each edition like row-level security, dynamic data masking, and in-memory OLTP. It also includes memory limits, MAXDOP settings, and pushdown capabilities for different editions. The document discusses lightweight query profiling improvements in SQL Server 2016 SP1 and provides details on predicate pushdown indicators in showplans.
More Related Content
What's hot
Prácticas recomendadas en materia de arquitectura y errores que debes evitar
Crece con confianza. Desde la implementación de un pequeño nodo de desarrollo para la búsqueda de aplicaciones hasta la gestión de una gran implementación de cientos de nodos, nuestros expertos en Elastic te contarán todo lo que necesites saber.
Operationalizing Big Data Pipelines At Scale
Running a global, world-class business with data-driven decision making requires ingesting and processing diverse sets of data at tremendous scale. How does a company achieve this while ensuring quality and honoring their commitment as responsible stewards of data? This session will detail how Starbucks has embraced big data, building robust, high-quality pipelines for faster insights to drive world-class customer experiences.
Forecasting time series powerful and simple
Time series are a sequence of data points positioned in order of time. Time series forecasting has two main purposes - to understand the mechanisms that lead to rise or fall, and to predict future values. Very often it analyses trends, cyclical events, seasonality and has unique importance in Economics and Business. The quality of predictions can be evaluated only in future due to temporal dependencies on previous data points and there are many model types for approximation. In this session we are going to talk about challenges, ways of improvement and technology stack like ML.NET, ARIMA, Python, Azure ML, Regression and FB Prophet
From Batch to Streaming ET(L) with Apache Apex
Stream data processing is increasingly required to support business needs for faster actionable insight with growing volume of information from more sources. Apache Apex is a true stream processing framework for low-latency, high-throughput and reliable processing of complex analytics pipelines on clusters. Apex is designed for quick time-to-production, and is used in production by large companies for real-time and batch processing at scale.
This session will use an Apex production use case to walk through the incremental transition from a batch pipeline with hours of latency to an end-to-end streaming architecture with billions of events per day which are processed to deliver real-time analytical reports. The example is representative for many similar extract-transform-load (ETL) use cases with other data sets that can use a common library of building blocks. The transform (or analytics) piece of such pipelines varies in complexity and often involves business logic specific, custom components.
Topics include:
* Pipeline functionality from event source through queryable state for real-time insights.
* API for application development and development process.
* Library of building blocks including connectors for sources and sinks such as Kafka, JMS, Cassandra, HBase, JDBC and how they enable end-to-end exactly-once results.
* Stateful processing with event time windowing.
* Fault tolerance with exactly-once result semantics, checkpointing, incremental recovery
* Scalability and low-latency, high-throughput processing with advanced engine features for auto-scaling, dynamic changes, compute locality.
* Who is using Apex in production, and roadmap.
Following the session attendees will have a high level understanding of Apex and how it can be applied to use cases at their own organizations.
Time Series Databases for IoT (On-premises and Azure)
This document discusses choosing the right time series database for IoT data. It compares InfluxDB to SQL Server and other databases.
Some key points made:
- InfluxDB outperforms SQL Server for writes by 40x and queries by 59x for time series data due to its optimized design.
- InfluxDB uses 19x-26x less disk storage than SQL Server for the same data.
- InfluxDB also outperforms MongoDB, Elasticsearch, OpenTSDB, and Cassandra for time series workloads.
- Azure Stream Insights is a managed service but has limited capabilities and can be pricey for high volumes of data.
- InfluxDB is open source, has no dependencies, and
Big Data, Bigger Analytics
This document discusses using Apache Spark for big data analytics at an insurance pricing and customer analytics company called Earnix. It summarizes Earnix's business problems modeling large customer behavior data, how Spark helps address performance issues with their existing 10GB datasets, and improvements made to Spark's MLlib machine learning library. These include adding statistical functionality like covariance estimation to logistic regression models and optimizing algorithms to run efficiently on Spark. Benchmark results show Spark providing scalability by reducing algorithm run times as more nodes are added.
Continuous Evaluation of Deployed Models in Production Many high-tech industr...
Many high-tech industries rely on machine-learning systems in production environments to automatically classify and respond to vast amounts of incoming data. Despite their critical roles, these systems are often not actively monitored. When a problem first arises, it may go unnoticed for some time. Once it is noticed, investigating its underlying cause is a time-consuming, manual process. Wouldn’t it be great if the model’s output were automatically monitored? If they could be visualized, sliced by different dimensions? If the system could automatically detect performance degradation and trigger alerts? In this presentation, we describe our experience from building such a core machine-learning services: Model Evaluation.
Our service provides automated, continuous evaluation of the performance of a deployed model over commonly-used metrics like the area-under-the-curve (AUC), root-mean-square-error (RMSE) etc. In addition, summary statistics about the model’s output, their distributions are also computed. The service also provides a dashboard to visualize the performance metrics, summary statistics and distributions of a model over time along with REST APIs to retrieve these metrics programmatically.
These metrics can be sliced by input features (e.g. Geography, Product type) to provide insights into model performance over different segments. The talk will describe various components that are required in building such a service and metrics of interest. Our system has a backend component built with spark on Azure Databricks. The backend can scale to analyze TBs of data to generate model evaluation metrics.
We will talk about how we modified Spark MLLib for computing AUC sliced by different dimensions and other optimizations in Spark to improve compute and performance. Our front-end and middle-tier, built with Docker and Azure Webapp provides visuals and REST APIs to retrieve the above metrics. This talk will cover various aspects of building, deploying and using the above system.
Assessing Graph Solutions for Apache Spark
Users have several options for running graph algorithms with Apache Spark. To support a graph data architecture on top of its linear-oriented DataFrames, the Spark platform offers GraphFrames. However, due to the fact that GraphFrames are immutable and not a native graph, there are cases where it might not offer the features or performance needed for certain use cases. Another option is to connect Spark to a real-time, scalable and distributed native graph database such as TigerGraph.
In this session, we compare three options — GraphX, Cypher for Apache Spark, and TigerGraph — for different types of workload requirements and data sizes, to help users select the right solution for their needs. We also look at the data transfer and loading time for TigerGraph.
Predicting Optimal Parallelism for Data Analytics
A key benefit of serverless computing is that resources can be allocated on demand, but the quantity of resources to request, and allocate, for a job can profoundly impact its running time and cost. For a job that has not yet run, how can we provide users with an estimate of how the job’s performance changes with provisioned resources, so that users can make an informed choice upfront about cost-performance tradeoffs?
This talk will describe several related research efforts at Microsoft to address this question. We focus on optimizing the amount of computational resources that control a data analytics query’s achieved intra-parallelism. These use machine learning models on query characteristics to predict the run time or Performance Characteristic Curve (PCC) as a function of the maximum parallelism that the query will be allowed to exploit.
The AutoToken project uses models to predict the peak number of tokens (resource units) that is determined by the maximum parallelism that the recurring SCOPE job can ever exploit while running in Cosmos, an Exascale Big Data analytics platform at Microsoft. AutoToken_vNext, or TASQ, predicts the PCC as a function of the number of allocated tokens (limited parallelism). The AutoExecutor project uses models to predict the PCC for Apache Spark SQL queries as a function of the number of executors. The AutoDOP project uses models to predict the run time for SQL Server analytics queries, running on a single machine, as a function of their maximum allowed Degree Of Parallelism (DOP).
We will present our approaches and prediction results for these scenarios, discuss some common challenges that we handled, and outline some open research questions in this space.
26 Trillion App Recomendations using 100 Lines of Spark Code - Ayman Farahat
This document summarizes an approach to build an app recommendation engine using collaborative filtering on data from 53,793 apps and 496 million users. It implements collaborative filtering using 100 lines of Spark code to generate low-dimensional user and app features and compute 26 trillion user-app ratings. Key aspects included using data frames, caching joined data, broadcasting app features for efficient BLAS-3 matrix multiplication, and evaluating recommendations.
Superworkflow of Graph Neural Networks with K8S and Fugue
This document introduces a superworkflow for running Node2Vec on graphs using the Fugue framework on Kubernetes. It describes the Node2Vec algorithm and different steps in the superworkflow, including graph creation and indexing, random walks, Word2Vec preprocessing, and embedding training. The superworkflow provides advantages like parallelizing steps, efficient resource usage through auto-persist and checkpointing. Benchmark results show the superworkflow reduces runtime significantly compared to Spark MLlib, such as reducing a 100M node graph embedding from 6,800 CPU hours to 100 CPU hours and 16 GPU hours. Open source links for the Node2Vec on Fugue project are also provided.
Case Study Real Time Olap Cubes
Are OLAP cubes "large monters" that deliver quick data retrieval at the expense of long upload time? This presentation shows one way to kill this myth.
Stock Analyzer Hadoop MapReduce Implementation
This document presents a Hadoop project to analyze stock prices using MapReduce and Hive. The project aims to find the adjusted closing price for stock days without dividends by joining daily stock price and dividend datasets. The technical architecture uses Eclipse, Hadoop, and AWS EC2 for development and clustering. Pseudo code is provided for MapReduce jobs to parse input data and calculate adjusted closing prices for output. The results found adjusted closing prices for 44 records out of 75 total records by removing headers and dividend-only days.
A Production Quality Sketching Library for the Analysis of Big Data
In the analysis of big data there are often problem queries that don’t scale because they require huge compute resources to generate exact results, or don’t parallelize well.
Is This Thing On? A Well State Model for the People
The document discusses using machine learning models to determine well production state (on vs off) from sensor data. It presents an existing data architecture and issues with data quality. A supervised learning model is proposed using a decision tree trained on labeled rod pump production data. The modeling workflow includes data preprocessing, feature engineering, hyperparameter tuning and grid search. Decision trees are chosen for their interpretability but the document notes larger models may perform better. Overall production state modeling could help optimize operations and outperform existing controllers.
Koalas: How Well Does Koalas Work?
Koalas is an open source project that provides pandas APIs on top of Apache Spark. Pandas is the standard tool for data science and it is typically the first step to explore and manipulate a data set, but pandas does not scale well to big data. Koalas fills the gap by providing pandas equivalent APIs that work on Apache Spark.
There are also many libraries trying to scale pandas APIs, such as Vaex, Modin, and so on. Dask is one of them and very popular among pandas users, and also works on its own cluster similar to Koalas which is on top of Spark cluster. In this talk, we will introduce Koalas and its current status, and the comparison between Koalas and Dask, including benchmarking.
Large-Scaled Telematics Analytics in Apache Spark with Wayne Zhang and Neil P...
The increasing availability of mobile phones with embedded GPS devices and sensors has spurred the use of vehicle telematics in recent years. Telematics provides detailed and continuous information of a vehicle such as the location, speed, and movement. Vehicle telematics can be further linked with other spatial data to provide context to understand driving behaviors. The collection of high-frequency telematics data results in huge volumes of data that must be processed efficiently. We present a solution that uses Apache Spark to load and transform large-scaled telematics data. We then present how to use machine learning on telematics data to derive insights about driving safety.
MemSQL 201: Advanced Tips and Tricks Webcast
This document summarizes a webinar on advanced tips and tricks for MemSQL. It discusses the differences between rowstore and columnstore storage models and when each is best used. It also covers data ingestion using MemSQL Pipelines for real-time loading, data sharding and query tuning techniques like using reference tables. Additionally, it discusses monitoring memory usage, workload management using management views, and query optimization tools like analyzing and optimizing tables.
Cost-Based Optimizer in Apache Spark 2.2
The document discusses Apache Spark's new cost-based optimizer (CBO) in version 2.2. It describes how the CBO works in two key steps:
1. It collects and propagates statistics about tables and columns to estimate the cardinality of operations like filters, joins and aggregates.
2. It calculates the estimated cost of different execution plans and selects the most optimal plan based on minimizing the estimated cost. This allows it to pick more efficient join orders and join algorithms.
The document provides examples of how the CBO improves queries on TPC-DS benchmarks by producing smaller intermediate results and faster execution times compared to the previous rule-based optimizer in Spark 2.1.
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
The document discusses optimizations made to Spark SQL performance when working with Parquet files at ByteDance. It describes how Spark originally reads Parquet files and identifies two main areas for optimization: Parquet filter pushdown and the Parquet reader. For filter pushdown, sorting columns improved statistics and reduced data reads by 30%. For the reader, splitting it to first filter then read other columns prevented loading unnecessary data. These changes improved Spark SQL performance at ByteDance without changing jobs.
What's hot (20)
Prácticas recomendadas en materia de arquitectura y errores que debes evitar
Prácticas recomendadas en materia de arquitectura y errores que debes evitar
Time Series Databases for IoT (On-premises and Azure)
Time Series Databases for IoT (On-premises and Azure)
Continuous Evaluation of Deployed Models in Production Many high-tech industr...
Continuous Evaluation of Deployed Models in Production Many high-tech industr...
26 Trillion App Recomendations using 100 Lines of Spark Code - Ayman Farahat
26 Trillion App Recomendations using 100 Lines of Spark Code - Ayman Farahat
Superworkflow of Graph Neural Networks with K8S and Fugue
Superworkflow of Graph Neural Networks with K8S and Fugue
A Production Quality Sketching Library for the Analysis of Big Data
A Production Quality Sketching Library for the Analysis of Big Data
Is This Thing On? A Well State Model for the People
Is This Thing On? A Well State Model for the People
Large-Scaled Telematics Analytics in Apache Spark with Wayne Zhang and Neil P...
Large-Scaled Telematics Analytics in Apache Spark with Wayne Zhang and Neil P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Similar to Oracle 12c Analytics New Features
Best Practices for Supercharging Cloud Analytics on Amazon Redshift
In this webinar, we discuss how the secret sauce to your business analytics strategy remains rooted on your approached, methodologies and the amount of data incorporated into this critical exercise. We also address best practices to supercharge your cloud analytics initiatives, and tips and tricks on designing the right information architecture, data models and other tactical optimizations.
To learn more, visit: http://www.snaplogic.com/redshift-trial
Enhancements that will make your sql database roar sp1 edition sql bits 2017
This document provides information about various SQL Server features and editions. It includes a list of features available in each edition like row-level security, dynamic data masking, and in-memory OLTP. It also includes memory limits, MAXDOP settings, and pushdown capabilities for different editions. The document discusses lightweight query profiling improvements in SQL Server 2016 SP1 and provides details on predicate pushdown indicators in showplans.
Oracle Query Optimizer - An Introduction
An attempt to present Oracle Query Optimizer concepts together with some related saucy details / real-life situations.
Self-serve analytics journey at Celtra: Snowflake, Spark, and Databricks
Celtra provides a platform for streamlined ad creation and campaign management used by customers including Porsche, Taco Bell, and Fox to create, track, and analyze their digital display advertising. Celtra’s platform processes billions of ad events daily to give analysts fast and easy access to reports and ad hoc analytics. Celtra’s Grega Kešpret leads a technical dive into Celtra’s data-pipeline challenges and explains how it solved them by combining Snowflake’s cloud data warehouse with Spark to get the best of both.
Topics include:
- Why Celtra changed its pipeline, materializing session representations to eliminate the need to rerun its pipeline
- How and why it decided to use Snowflake rather than an alternative data warehouse or a home-grown custom solution
- How Snowflake complemented the existing Spark environment with the ability to store and analyze deeply nested data with full consistency
- How Snowflake + Spark enables production and ad hoc analytics on a single repository of data
Optimizing applications and database performance
This document discusses various ways to optimize application and database performance, including:
1. Optimizing database design, application design, memory usage, I/O, and reducing database contention.
2. Specific techniques mentioned include de-normalization, materialized views, bind variables, partitioning, parallel execution, and optimizing storage structures.
3. Monitoring tools are also discussed to detect issues like lock and latch contention that could be optimized.
How KeyBank Used Elastic to Build an Enterprise Monitoring Solution
KeyBank is using an iterative design approach to scale their end-to-end enterprise monitoring system with Kafka and Elasticsearch at its core. See how they did it and the lessons learned along the way.
Presentación Oracle Database Migración consideraciones 10g/11g/12c
Antes de migrar de 10g a 11g o 12c, tome en cuenta las siguientes consideraciones. No es tan sencillo como simplemente cambiar de motor de base de datos, se necesita hacer consideraciones a nivel del aplicativo.
SnappyData, the Spark Database. A unified cluster for streaming, transactions...
Apache Spark 2.0 offers many enhancements that make continuous analytics quite simple. In this talk, we will discuss many other things that you can do with your Apache Spark cluster. We explain how a deep integration of Apache Spark 2.0 and in-memory databases can bring you the best of both worlds! In particular, we discuss how to manage mutable data in Apache Spark, run consistent transactions at the same speed as state-the-art in-memory grids, build and use indexes for point lookups, and run 100x more analytics queries at in-memory speeds. No need to bridge multiple products or manage, tune multiple clusters. We explain how one can take regulation Apache Spark SQL OLAP workloads and speed them up by up to 20x using optimizations in SnappyData.
We then walk through several use-case examples, including IoT scenarios, where one has to ingest streams from many sources, cleanse it, manage the deluge by pre-aggregating and tracking metrics per minute, store all recent data in a in-memory store along with history in a data lake and permit interactive analytic queries at this constantly growing data. Rather than stitching together multiple clusters as proposed in Lambda, we walk through a design where everything is achieved in a single, horizontally scalable Apache Spark 2.0 cluster. A design that is simpler, a lot more efficient, and let’s you do everything from Machine Learning and Data Science to Transactions and Visual Analytics all in one single cluster.
SnappyData at Spark Summit 2017
Why does big data always have to go through a pipeline? multiple data copies, slow, complex and stale analytics? We present a unified analytics platform that brings streaming, transactions and adhoc OLAP style interactive analytics in a single in-memory cluster based on Spark.
Evolution of Performance Management: Oracle 12c adaptive optimizations - ukou...
On version 12c Oracle introduced new features to allow Adaptive optimizations: Adaptive Plans and Adaptive Statistics. After a quick presentation of concepts, this session will explore the interaction of these features with other performance management techniques using examples, like SPM and SQL profiles. Attendees will get an updated picture of tools available to troubleshoot performance issues, and how to get the most of these new features.
SnappyData Ad Analytics Use Case -- BDAM Meetup Sept 14th
Presented at the BDAM meetup in Palo Alto on Sept 14th. Jags Ramnarayan, CTO, SnappyData, discusses an ad analytics use case running on SnappyData.
Introduction to Apache Apex by Thomas Weise
https://www.bigdataspain.org/2016/program/thu-introduction-apache-apex.html
https://www.youtube.com/watch?v=93mWU2k8AaU&index=41&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&t=199s
Triple C - Centralize, Cloudify and Consolidate Dozens of Oracle Databases (O...
Dozens of Oracle Databases - each health center location has one on its local server with the same data model and the same set of applications. These databases have to be centralized and cloudified and also be consolidated into one or as few databases as possible. To lower costs, ease operations and enable innovation. Each location can access only its own data, applications do not have to be changed and different locations can run different versions of applications and database objects. This is the story of a critical migration. About the cloud ready analysis, the Proofs of Concept with Oracle Database features VPD and Edition Based Redefinition, the scalability investigation, the redesign of change management, rollout and operational management processes and the careful modernization of a 25 year old platform on the latest database release and a shiny new, fully automated cloud platform. This is the story of an organization that had state of the art systems in the mid-90s. And they have these same systems today - no longer state of the art. They can keep the systems alive, but barely, and at increasing cost. In the Fall of 2020, we started an investigation into the feasibility of bringing the 100s of databases from each of the locations together, in a central location, in the cloud and finally: consolidated into one or at least as few database instances as possible. Using Oracle Database Virtual Private Database and Edition Based Redefinition, a smart database connection configuration in each site and a limited reimplementation of non-cloud/non-consolidated mechanisms (interaction with local file system for example) we have designed and proven a working new design and migration approach.
Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...
In this presentation, we are going to talk about the state of the art infrastructure we have established at Walmart Labs for the Search product using Spark Streaming and DataFrames. First, we have been able to successfully use multiple micro batch spark streaming pipelines to update and process information like product availability, pick up today etc. along with updating our product catalog information in our search index to up to 10,000 kafka events per sec in near real-time. Earlier, all the product catalog changes in the index had a 24 hour delay, using Spark Streaming we have made it possible to see these changes in near real-time. This addition has provided a great boost to the business by giving the end-costumers instant access to features likes availability of a product, store pick up, etc.
Second, we have built a scalable anomaly detection framework purely using Spark Data Frames that is being used by our data pipelines to detect abnormality in search data. Anomaly detection is an important problem not only in the search domain but also many domains such as performance monitoring, fraud detection, etc. During this, we realized that not only are Spark DataFrames able to process information faster but also are more flexible to work with. One could write hive like queries, pig like code, UDFs, UDAFs, python like code etc. all at the same place very easily and can build DataFrame template which can be used and reused by multiple teams effectively. We believe that if implemented correctly Spark Data Frames can potentially replace hive/pig in big data space and have the potential of becoming unified data language.
We conclude that Spark Streaming and Data Frames are the key to processing extremely large streams of data in real-time with ease of use.
Machine Learning and AI at Oracle
This document discusses Oracle's machine learning capabilities. It provides an overview of the types of machine learning algorithms available in Oracle such as classification, clustering, regression, and time series analysis. It also describes how machine learning can be used directly from SQL and integrated with Oracle Autonomous Database and Oracle Database to build and deploy models. New algorithms like XGBoost and features for enhanced prediction are highlighted.
Get the most out of your AWS Redshift investment while keeping cost down
Amazon Redshift offers many powerful features. Yet, there are many instances where customers encounter sloppy performance and cost upheavals beyond control.
Scaling AWS Redshift clusters to meet the increasing compute and reporting needs, while ensuring optimal cost, performance and security standards is quite a challenge for many organizations.
This webinar covered the following,
• Understand key design/architectural considerations of AWS Redshift
• Tips & Tricks to optimize Cost & Performance
• How Agilisium helped clients reduce AWS Redshift run cost up to 40%
Presented by:
Jay Palaniappan - CTO & Head of Innovation Labs || Smitha Basavaraju - Big Data Architect || Arun Chinnadurai - Associate Director – BD
Optimizing industrial operations using the big data ecosystem
GE Digital is undertaking a journey to optimize the reliability, availability, and efficiency of assets in the industrial sector and converge IT and OT. To do so, GE Digital is building cloud-based products that enable customers to analyze the asset data, detect anomalies, and provide recommendations for operating plants efficiently while increasing productivity. In a energy sector such as oil and gas, power, or renewables, a single plant comprises multiple complex assets, such as steam turbines, gas turbines, and compressors, to generate power. Each system contains various sensors to detect the operating conditions of the assets, generating large volumes of variety of data. A highly scalable distributed environment is required to analyze such a large volume of data and provide operating insights in near real time.
In this session I will share the challenges encountered when analyzing the large volumes of data, in-stream data analysis and how we standardized the industrial data based on data frames, and performance tuning.
From Data to Services at the Speed of Business
From Data to Services at the Speed of Business: Applying cloud-native paradigm to combine fast data analytics with microservices architecture for hybrid workloads.
Lessons learned from embedding Cassandra in xPatterns
The document discusses lessons learned from embedding Cassandra in the xPatterns big data analytics platform. It provides an agenda that includes discussing Cassandra usage in xPatterns, the necessary developments like data modeling optimizations, robust REST APIs, geo-replication, and a demo of exporting to NoSQL APIs. Key lessons learned since Cassandra versions 0.6 to 2.0.6 are also summarized, such as the need for consistent clocks, reducing column families, and monitoring.
The Developer Data Scientist – Creating New Analytics Driven Applications usi...
The Developer Data Scientist – Creating New Analytics Driven Applications usi...Microsoft Tech Community
The developer world is changing as we create and generate new data patterns and handling processes within our applications. Additionally, with the massive interest in machine learning and advanced analytics how can we as developers build intelligence directly into our applications that can integrate with the data and data paths we are creating? The answer is Azure Databricks and by attending this session you will be able to confidently develop smarter and more intelligent applications and solutions which can be continuously built upon and that can scale with the growing demands of a modern application estate. Similar to Oracle 12c Analytics New Features (20)
Best Practices for Supercharging Cloud Analytics on Amazon Redshift
Best Practices for Supercharging Cloud Analytics on Amazon Redshift
Enhancements that will make your sql database roar sp1 edition sql bits 2017
Enhancements that will make your sql database roar sp1 edition sql bits 2017
Self-serve analytics journey at Celtra: Snowflake, Spark, and Databricks
Self-serve analytics journey at Celtra: Snowflake, Spark, and Databricks
How KeyBank Used Elastic to Build an Enterprise Monitoring Solution
How KeyBank Used Elastic to Build an Enterprise Monitoring Solution
Presentación Oracle Database Migración consideraciones 10g/11g/12c
Presentación Oracle Database Migración consideraciones 10g/11g/12c
SnappyData, the Spark Database. A unified cluster for streaming, transactions...
SnappyData, the Spark Database. A unified cluster for streaming, transactions...
Evolution of Performance Management: Oracle 12c adaptive optimizations - ukou...
Evolution of Performance Management: Oracle 12c adaptive optimizations - ukou...
SnappyData Ad Analytics Use Case -- BDAM Meetup Sept 14th
SnappyData Ad Analytics Use Case -- BDAM Meetup Sept 14th
Triple C - Centralize, Cloudify and Consolidate Dozens of Oracle Databases (O...
Triple C - Centralize, Cloudify and Consolidate Dozens of Oracle Databases (O...
Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...
Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...
Get the most out of your AWS Redshift investment while keeping cost down
Get the most out of your AWS Redshift investment while keeping cost down
Optimizing industrial operations using the big data ecosystem
Optimizing industrial operations using the big data ecosystem
Lessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatterns
The Developer Data Scientist – Creating New Analytics Driven Applications usi...
The Developer Data Scientist – Creating New Analytics Driven Applications usi...
Recently uploaded
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
dbms calicut university B. sc Cs 4th sem.pdf
Its a seminar ppt on database management system using sql
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Recently uploaded (20)
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Oracle 12c Analytics New Features
- 1. Oracle 12c Analytics New Features Husnu Sensoy Global Maksimum Data & Information Technologies
- 2. Global Maksimum Data & Information Technologies • Complex Event Processing • Oracle CEP • Making hundred of different business decisions for millions of events in a second • Advanced Analytics • Oracle Data Mining • Oracle R Enterprise • Large scale data analytics • Ten billion rows in a week • Data Visualisation • State of the art data visualisation • DIY BI
- 3. New Features for Analytics • Partitioning • CBO • Advanced Analytics • Data Management
- 4. Partitioning • Core Functionality • • Interval-REF Partitioning Performance • • • Partition Maintenance on multiple partitions Partial local and global indexes Manageability • Asynchronous global index maintenance for DROP/TRUNCATE • Online partition MOVE • Cascading TRUNCATE/EXCHANGE
- 5. Cost Based Optimiser • Adaptive Statistics • Dynamic Sampling (LEVEL=11) • Cardinality Feedback Enhancement • Re-optimisation • Histograms • Better and Faster Statistics Gathering • • Session private statistics on GTT • • STATS_ON_LOAD: For CTAS and IAS on empty tables Concurrent statistics gathering Adaptive Plans • Join methods • Parallel distribution methods
- 6. Adaptive Query Optimisation SELECT product_name FROM order_items o, product_information p WHERE o.unit_price = 15 AND o.quantity > 1 AND p.product_id = o.product_id NESTED LOOP HASH JOIN threshold Stats collector Table scan order_items Index scan prod_info_idx Table scan product_information
- 7. Adaptive Query Optimisation • Join method decision deferred until runtime • • Alternate sub-plans are pre-computed and stored in the cursor • • Default plan is computed using available statistics Statistic collectors are inserted at key points in the plan Data distribution method can also be changed during execution.
- 8. Advanced Analytics • Oracle Advanced Analytics 12c • New Data Mining Algorithms • • SVD - PCA • • EM (Expectation Maximisation) Predictive Queries Oracle Data Miner/SQL Developer 4.0 • • SQL Query Node • • New Graph Node (box,scatter, bar,histogram) R Script Node Oracle Advanced Analytics/ORE 1.3 • Neural Networks • Improved integration with OBIEE
- 9. Predictive Queries SELECT cust_income_level, cust_id, Round(prob_anom, 2) prob_anom, Round(pctrank, 3) * 100 pct_rank FROM (SELECT cust_id, cust_income_level, prob_anom, Percent_rank() over( PARTITION BY cust_income_level ORDER BY prob_anom DESC) AS pct_rank FROM (SELECT cust_id, cust_income_level, Prediction_probability(OF ANOMALY,0 using *) over( PARTITION BY cust_income_level) prob_anom FROM customers)) WHERE pct_rank <= .05 ORDER BY cust_income_level, prob_anom desc
- 10. SQL Pattern Matching X YWZ first_x 1 9 9 13 1 last_z 19 13 19 SELECT first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY name ORDER BY time MEASURES FIRST(x.time) AS first_x LAST(z.time) AS last_z ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)) Y AS (price > PREV(price)) W AS (price < PREV(price)) Z AS (price > PREV(price))
- 12. Automatic Data Optimisation (ADO) Policy Active/Hot Frequently Accessed Occasional Access Dormant ALTER TABLE sales ILM ADD row store compress advanced row after 2 DAYS OF NO_MODIFICATION; ALTER TABLE sales ILM ADD compress for query low after 7 DAYS OF NO_MODIFICATION; ALTER TABLE sales ILM ADD TIER TO sata_tbs AFTER 1 MONTH OF NO ACCESS; ALTER TABLE sales ILM ADD compress for archive high AFTER 7 MONTHS OF NO ACCESS;
- 13. Q& A