Realistic Synthetic Generation Allows Secure Development
•
0 likes•535 views
Ted Dunning is the Chief Applications Architect at MapR Technologies. He has authored short books on real-world Hadoop, time series databases, anomaly detection, and recommendations. The document discusses how to generate realistic simulated data for testing anomaly detection and fraud analytics systems while keeping the actual customer data private. It describes parametric simulation techniques to match key performance indicators and failure modes rather than precise data distributions in order to emulate complex behaviors and test detection algorithms. Various statistical and database methods are presented for generating different types of numeric, symbolic, and behavioral data.
Report
Share
Report
Share
Recommended
Dealing with an Upside Down Internet With High Performance Time Series Database
The document discusses time series data storage and analysis. It begins with an overview of how the internet is changing with more data being generated at the edge by sensors and IoT devices. It then discusses challenges with storing and analyzing high volumes of time series data using traditional databases. The document proposes using NoSQL databases like MapR to build time series databases that can ingest data at faster rates like 100 million data points per second. It also discusses techniques for bulk loading and retrieving large ranges of time series data efficiently.
Dealing with an Upside Down Internet
From the Hadoop Summit 2015 Session with Ted Dunning:
Just when we thought the last mile problem was solved, the Internet of Things is turning the last mile problem of the consumer internet into the first mile problem of the industrial internet. This inversion impacts every aspect of the design of networked applications. I will show how to use existing Hadoop ecosystem tools, such as Spark, Drill and others, to deal successfully with this inversion. I will present real examples of how data from things leads to real business benefits and describe real techniques for how these examples work.
How the Internet of Things are Turning the Internet Upside Down
- The document discusses how time series data from sensors can be ingested and analyzed at large scales. It describes how traditional internet architecture concentrates resources at the core while sensors and devices reside at the edge, producing large amounts of time series data.
- It then summarizes techniques for ingesting and analyzing time series data at rates of millions to hundreds of millions of data points per second using technologies like OpenTSDB, HBase, and MapR databases. This involves batching data at the edge and optimized storage designs.
- The document concludes by discussing the advantages of MapR for time series use cases due to its high ingestion rates and integration with query engines like Drill for flexible analysis of large time series datasets
Building HBase Applications - Ted Dunning
Ted Dunning, Chief Application Architect at MapR presents at Big Data Everywhere
Tel Aviv, June 2014
Apache HBase applications can turn in some pretty impressive performance and scalability numbers, but designing these applications is not the same as designing relational databases. There are numerous opportunities for a design to make use of special characteristics of HBase and related systems like MapR's tables.
We will talk about the why and the what. Why is it that HBase allows these opportunities and requires special design? What specific strategies can the HBase application designer use to take advantage of the virtues and avoid the pitfalls.
To do this, we will examine several real applications including a time series database and a data cube drill down.
Practice of large Hadoop cluster in China Mobile
China Mobile Limited is the leading telecommunications services provider in China, with more than 800 million active users. In China Mobile, distributed big data clusters are built by branch companies in each province for their unique requirements. Meanwhile, we have built a centralized Hadoop cluster with scale more than 1600 nodes, on which we collect data from dozens of distributed clusters and make analysis for our business.
In this session, we will introduce the architecture of the centralized Hadoop cluster and experience of constructing and tuning this large scale Hadoop cluster. Key points are as follows:
1. About Ambari: we improve Ambari with features like supporting HDFS Federation and Ambari HA , improving its performance and enabling it to support up to 1600 nodes.
2. About HDFS: we build a large HDFS cluster with data up to 60PB, using federation, ViewFS, FairCallQueue. Our best practice of cluster operation and management will also be included.
3. About Flume: We use the reformed Flume to collect data as much as 200TB per day.
Speakers
Yuxuan Pan, Software Engineer, China Mobile Software Technology
Duan Yunfeng, Chief Designer of China Mobile's big data system, China Mobile Communications Corporation
Apache Hadoop 3.0 What's new in YARN and MapReduce
This document summarizes a presentation about new features in Apache Hadoop 3.0 related to YARN and MapReduce. It discusses major evolutions like the re-architecture of the YARN Timeline Service (ATS) to address scalability, usability, and reliability limitations. Other evolutions mentioned include improved support for long-running native services in YARN, simplified REST APIs, service discovery via DNS, scheduling enhancements, and making YARN more cloud-friendly with features like dynamic resource configuration and container resizing. The presentation estimates the timeline for Apache Hadoop 3.0 releases with alpha, beta, and general availability targeted throughout 2017.
What's new in Hadoop Common and HDFS
The document discusses new features in Apache Hadoop Common and HDFS for version 3.0. Key updates include upgrading the minimum Java version to Java 8, improving dependency management, adding a new Azure Data Lake Storage connector, and introducing erasure coding in HDFS to improve storage efficiency. Erasure coding in HDFS phase 1 allows for striping of small blocks and parallel writes/reads while trading off higher network usage compared to replication.
Analyzing Hadoop Using Hadoop
Sheetal Dolas, a principal architect at Hortonworks, gave a presentation on analyzing Hadoop using Hadoop. She discussed the need for operational insights from Hadoop metrics to manage system reliability, uptime, and performance. While Hadoop generates a lot of metrics, understanding and analyzing them is challenging due to the large volume and lack of tools. Shem showed how datasets like MapReduce job histories, HDFS lsr reports, and audit logs can be analyzed using Hadoop to generate operational reports on topics like resource utilization, workload distributions, security checks, and more.
Recommended
Dealing with an Upside Down Internet With High Performance Time Series Database
The document discusses time series data storage and analysis. It begins with an overview of how the internet is changing with more data being generated at the edge by sensors and IoT devices. It then discusses challenges with storing and analyzing high volumes of time series data using traditional databases. The document proposes using NoSQL databases like MapR to build time series databases that can ingest data at faster rates like 100 million data points per second. It also discusses techniques for bulk loading and retrieving large ranges of time series data efficiently.
Dealing with an Upside Down Internet
From the Hadoop Summit 2015 Session with Ted Dunning:
Just when we thought the last mile problem was solved, the Internet of Things is turning the last mile problem of the consumer internet into the first mile problem of the industrial internet. This inversion impacts every aspect of the design of networked applications. I will show how to use existing Hadoop ecosystem tools, such as Spark, Drill and others, to deal successfully with this inversion. I will present real examples of how data from things leads to real business benefits and describe real techniques for how these examples work.
How the Internet of Things are Turning the Internet Upside Down
- The document discusses how time series data from sensors can be ingested and analyzed at large scales. It describes how traditional internet architecture concentrates resources at the core while sensors and devices reside at the edge, producing large amounts of time series data.
- It then summarizes techniques for ingesting and analyzing time series data at rates of millions to hundreds of millions of data points per second using technologies like OpenTSDB, HBase, and MapR databases. This involves batching data at the edge and optimized storage designs.
- The document concludes by discussing the advantages of MapR for time series use cases due to its high ingestion rates and integration with query engines like Drill for flexible analysis of large time series datasets
Building HBase Applications - Ted Dunning
Ted Dunning, Chief Application Architect at MapR presents at Big Data Everywhere
Tel Aviv, June 2014
Apache HBase applications can turn in some pretty impressive performance and scalability numbers, but designing these applications is not the same as designing relational databases. There are numerous opportunities for a design to make use of special characteristics of HBase and related systems like MapR's tables.
We will talk about the why and the what. Why is it that HBase allows these opportunities and requires special design? What specific strategies can the HBase application designer use to take advantage of the virtues and avoid the pitfalls.
To do this, we will examine several real applications including a time series database and a data cube drill down.
Practice of large Hadoop cluster in China Mobile
China Mobile Limited is the leading telecommunications services provider in China, with more than 800 million active users. In China Mobile, distributed big data clusters are built by branch companies in each province for their unique requirements. Meanwhile, we have built a centralized Hadoop cluster with scale more than 1600 nodes, on which we collect data from dozens of distributed clusters and make analysis for our business.
In this session, we will introduce the architecture of the centralized Hadoop cluster and experience of constructing and tuning this large scale Hadoop cluster. Key points are as follows:
1. About Ambari: we improve Ambari with features like supporting HDFS Federation and Ambari HA , improving its performance and enabling it to support up to 1600 nodes.
2. About HDFS: we build a large HDFS cluster with data up to 60PB, using federation, ViewFS, FairCallQueue. Our best practice of cluster operation and management will also be included.
3. About Flume: We use the reformed Flume to collect data as much as 200TB per day.
Speakers
Yuxuan Pan, Software Engineer, China Mobile Software Technology
Duan Yunfeng, Chief Designer of China Mobile's big data system, China Mobile Communications Corporation
Apache Hadoop 3.0 What's new in YARN and MapReduce
This document summarizes a presentation about new features in Apache Hadoop 3.0 related to YARN and MapReduce. It discusses major evolutions like the re-architecture of the YARN Timeline Service (ATS) to address scalability, usability, and reliability limitations. Other evolutions mentioned include improved support for long-running native services in YARN, simplified REST APIs, service discovery via DNS, scheduling enhancements, and making YARN more cloud-friendly with features like dynamic resource configuration and container resizing. The presentation estimates the timeline for Apache Hadoop 3.0 releases with alpha, beta, and general availability targeted throughout 2017.
What's new in Hadoop Common and HDFS
The document discusses new features in Apache Hadoop Common and HDFS for version 3.0. Key updates include upgrading the minimum Java version to Java 8, improving dependency management, adding a new Azure Data Lake Storage connector, and introducing erasure coding in HDFS to improve storage efficiency. Erasure coding in HDFS phase 1 allows for striping of small blocks and parallel writes/reads while trading off higher network usage compared to replication.
Analyzing Hadoop Using Hadoop
Sheetal Dolas, a principal architect at Hortonworks, gave a presentation on analyzing Hadoop using Hadoop. She discussed the need for operational insights from Hadoop metrics to manage system reliability, uptime, and performance. While Hadoop generates a lot of metrics, understanding and analyzing them is challenging due to the large volume and lack of tools. Shem showed how datasets like MapReduce job histories, HDFS lsr reports, and audit logs can be analyzed using Hadoop to generate operational reports on topics like resource utilization, workload distributions, security checks, and more.
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...DataWorks Summit/Hadoop Summit
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark based Lambda Architecture
Apache Eagle - Monitor Hadoop in Real Time
Apache Eagle is a distributed real-time monitoring and alerting engine for Hadoop created by eBay to address limitations of existing tools in handling large volumes of metrics and logs from Hadoop clusters. It provides data activity monitoring, job performance monitoring, and unified monitoring. Eagle detects anomalies using machine learning algorithms and notifies users through alerts. It has been deployed across multiple eBay clusters with over 10,000 nodes and processes hundreds of thousands of events per day.
Achieving 100k Queries per Hour on Hive on Tez
Hive on Tez with LLAP (Late Loading Application) can achieve query processing speeds of over 100,000 queries per hour. Tuning various Hive and YARN parameters such as increasing the number of executor and I/O threads, memory allocation, and disabling consistent splits between LLAP daemons and data nodes was needed to reach this performance level on a test cluster of 45 nodes. Future work includes adding a web UI for monitoring LLAP clusters and implementing column-level access controls while allowing other frameworks like Spark to still access data through HiveServer2 and prevent direct access to HDFS for security reasons.
To The Cloud and Back: A Look At Hybrid Analytics
This document discusses hybrid analytics and the movement of data between on-premise and cloud environments. It notes that IT infrastructures now require both traditional and cloud-native approaches. Moving forward, hybrid analytics using both on-premise and cloud resources will become more common. Specifically, automatically moving data between on-premise storage and the cloud based on policies will help break down data silos and enable greater analysis. The document also explores using Dell EMC's OneFS software to enable this type of policy-based data movement to and from the cloud.
MapR M7: Providing an enterprise quality Apache HBase API
The document provides an overview of MapR M7, an integrated system for structured and unstructured data. M7 combines aspects of LSM trees and B-trees to provide faster reads and writes compared to Apache HBase. It achieves instant recovery from failures through its use of micro write-ahead logs and parallel region recovery. Benchmark results show MapR M7 providing 5-11x faster performance than HBase for common operations like reads, updates, and scans.
The Future of Apache Storm
This document summarizes updates to Apache Storm presented by P. Taylor Goetz of Hortonworks at Hadoop Summit 2016. Some key points include: Storm 0.9.x added high availability features and expanded integration capabilities. Storm 1.0 focused on maturity and improved performance. New features in Storm 1.0 include Pacemaker replacing Zookeeper, distributed caching, high availability Nimbus, native streaming windows, and state management with automatic checkpointing. Storm usability was also improved with features like dynamic log levels, tuple sampling for debugging, and distributed log searching. Future integrations and performance optimizations were also discussed.
Spark vstez
This is slides from our recent HadoopIsrael meetup. It is dedicated to comparison Spark and Tez frameworks.
In the end of the meetup there is small update about our ImpalaToGo project.
Yahoo's Experience Running Pig on Tez at Scale
Yahoo migrated most of its Pig workload from MapReduce to Tez to achieve significant performance improvements and resource utilization gains. Some key challenges in the migration included addressing misconfigurations, bad programming practices, and behavioral changes between the frameworks. Yahoo was able to run very large and complex Pig on Tez jobs involving hundreds of vertices and terabytes of data smoothly at scale. Further optimizations are still needed around speculative execution and container reuse to improve utilization even more. The migration to Tez resulted in up to 30% reduction in runtime, memory, and CPU usage for Yahoo's Pig workload.
Fast SQL on Hadoop, Really?
The document discusses Apache Hive and Apache Druid for fast SQL on big data. It provides performance benchmarks showing Hive LLAP is faster than Presto and Spark SQL for TPC-DS queries. It describes features of Hive LLAP including in-memory caching, query result caching, and metadata caching. It also discusses new Hive 3 features like materialized views and optimizer improvements. The document then provides an overview of Apache Druid's capabilities for real-time ingestion and querying of streaming data before discussing how Hive and Druid can work together, with Hive able to push down queries to Druid.
LLAP Nov Meetup
The document discusses Live Long and Process (LLAP), a new capability in Apache Hive that enables sub-second query performance. LLAP achieves this through caching the hottest data in RAM on each Hadoop node and running queries against this cache via lightweight long-running daemon processes. It allows for 100% SQL compatibility while integrating with existing security and tools. LLAP provides benefits like failure tolerance, concurrency, ACID transactions, and elastic scaling. Performance tests on TPC-DS queries demonstrated sub-second latency for queries even at large data scales and high concurrency levels.
Ingest and Stream Processing - What will you choose?
This document discusses streaming data ingestion and processing options. It provides an overview of common streaming architectures including Kafka as an ingestion hub and various streaming engines. Spark Streaming is highlighted as a popular and full-featured option for processing streaming data due to its support for SQL, machine learning, and ease of transition from batch workflows. The document also briefly profiles StreamSets Data Collector as a higher-level tool for building streaming data pipelines.
Architectural Overview of MapR's Apache Hadoop Distribution
Describes the thinking behind MapR's architecture. MapR"s Hadoop achieves better reliability on commodity hardware compared to anything on the planet, including custom, proprietary hardware from other vendors. Apache HDFS and Cassandra replication is also discussed, as are SAN and NAS storage systems like Netapp and EMC.
Stinger Initiative - Deep Dive
The document discusses the Stinger Initiative from Hortonworks to improve the performance and capabilities of interactive queries in Hive. The initiative takes a two-pronged approach, focusing on improvements to the query engine and the introduction of a new optimized column store file format called ORCFile. A new Tez execution engine is also introduced to avoid bottlenecks in MapReduce and enable lower latency queries. The goal is to extend Hive's ability to handle interactive queries with response times measured in seconds rather than minutes.
YARN Federation
This document discusses YARN federation, which allows multiple YARN clusters to be connected together. It summarizes:
- YARN is used at Microsoft for resource management but faces challenges of large scale and diverse workloads. Federation aims to address this.
- The federation architecture connects multiple independent YARN clusters through centralized services for routing, policies, and state. Applications are unaware and can seamlessly run across clusters.
- Federation policies determine how work is routed and scheduled across clusters, balancing objectives like load balancing, scaling, fairness, and isolation. A spectrum of policy options is discussed from full partitioning to full replication to dynamic partial replication.
- A demo is presented showing a job running across
LLAP: Sub-Second Analytical Queries in Hive
The document discusses LLAP (Live Long and Process), a new execution layer for Hive that enables sub-second analytical queries. LLAP uses daemons running on worker nodes to cache data in memory and keep query fragments executing between queries for faster performance. It allows for highly concurrent queries without specialized YARN queues. Benchmarks show LLAP providing up to 90% faster performance over Hive for queries against large datasets. LLAP also aims to serve as a unified data access layer for other systems like Spark SQL.
Elastic Scaling of a High-Throughput Content-Based Publish/Subscribe Engine
Publish/subscribe (pub/sub) infrastructures, running as a service in cloud environments, offer simplicity and flexibility for composing distributed applications. Provisioning them appropriately is, however, challenging. The amount of stored subscriptions and incoming publications varies over time, and the computational cost depends on the nature of the applications and in particular on the filtering operations they require, e.g., content-based vs. topic-based, encrypted vs. non-encrypted filtering. The ability to elastically adapt the amount of resources required to sustain given throughput and delay requirements is key to achieving cost-effectiveness for a pub/sub service running in a cloud environment. In this paper, we present the design and evaluation of an elastic content-based pub/sub system: eStreamHub. Specific contributions of this paper include: (1) a mechanism for dynamic scaling, both in and out, of stateful and stateless pub/sub operators, (2) a local and global elasticity policy enforcer maintaining high system utilization and stable end-to-end latencies, and (3) an evaluation using real-world tick workload from the Frankfurt Stock Exchange and encrypted content-based filtering.
Real-time Hadoop: The Ideal Messaging System for Hadoop
Ted Dunning presents on streaming architectures and MapR Technologies' streaming capabilities. He discusses MapR Streams, which implements the Kafka API for high performance and scale. MapR provides a converged data platform with files, tables, and streams managed under common security and permissions. Dunning reviews several use cases and lessons learned around real-time data processing, microservices, and global data management requirements.
LLAP: Sub-Second Analytical Queries in Hive
The document discusses LLAP (Live Long and Process), a new execution engine in Apache Hive 2.0 that enables sub-second analytical queries. LLAP keeps a small subset of frequently accessed data in memory to enable faster query processing times compared to traditional Hive architectures that rely on disk access. It works by running Hive query fragments simultaneously in both YARN containers and long-running daemon processes that cache data in memory. This allows for highly concurrent query execution without specialized YARN configurations. The document provides details on how LLAP is implemented and evaluates its performance benefits based on benchmarks and customer case studies.
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
This document discusses using Spark as an execution engine for Hive queries. It begins by explaining that Hive and Spark are both commonly used in the big data space, and that Hive on Spark uses the Hive optimizer with the Spark query engine, while Spark with a Hive context uses both the Catalyst optimizer and Spark engine. The document then covers challenges in deploying Hive on Spark, such as using a custom Spark JAR without Hive dependencies. It shows how the Hive EXPLAIN command works the same on Spark, and how the execution plan and stages differ between MapReduce and Spark. Overall, the document provides a high-level overview of using Spark as a query engine for Hive.
Resource Aware Scheduling in Apache Storm
This document provides an overview of resource aware scheduling in Apache Storm. It discusses the challenges of scheduling Storm topologies at Yahoo scale, including increasing heterogeneous clusters, low cluster utilization, and unbalanced resource usage. It then introduces the Resource Aware Scheduler (RAS) built for Storm, which allows fine-grained resource control and isolation for topologies through APIs and cgroups. Key features of RAS include pluggable scheduling strategies, per user resource guarantees, and topology priorities. Experimental results from Yahoo Storm clusters show significant improvements to throughput and resource utilization with RAS. Future work may include improved scheduling strategies and real-time resource monitoring.
One Click Hadoop Clusters - Anywhere (Using Docker)
This document provides an overview of Hortonworks' one-click Hadoop clusters solution that uses Docker and Apache Ambari to provision Hadoop clusters on any infrastructure. It discusses the goals of automating and unifying the Hadoop cluster provisioning process. The technology stack uses Docker for containerization, Swarm for clustering and orchestration, Consul for service discovery and registration, and Ambari for provisioning and management. Cloudbreak is introduced as a cloud-agnostic API that abstracts Hadoop cluster provisioning using these tools, while Periscope provides heuristic scheduling and autoscaling of clusters based on service level agreements.
Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...
This document discusses integrating an existing HPC infrastructure with the Hadoop ecosystem and YARN. It proposes building a custom YARN Application Master that acts as a "valve" between YARN and the HPC scheduler, allowing HPC applications to run on the shared infrastructure while reusing existing hardware. The advantages are better resource utilization and allowing new systems to leverage existing infrastructure. Potential drawbacks include added complexity from the HPC Application Master and slower performance from gradual changes.
More Related Content
What's hot
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...DataWorks Summit/Hadoop Summit
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark based Lambda Architecture
Apache Eagle - Monitor Hadoop in Real Time
Apache Eagle is a distributed real-time monitoring and alerting engine for Hadoop created by eBay to address limitations of existing tools in handling large volumes of metrics and logs from Hadoop clusters. It provides data activity monitoring, job performance monitoring, and unified monitoring. Eagle detects anomalies using machine learning algorithms and notifies users through alerts. It has been deployed across multiple eBay clusters with over 10,000 nodes and processes hundreds of thousands of events per day.
Achieving 100k Queries per Hour on Hive on Tez
Hive on Tez with LLAP (Late Loading Application) can achieve query processing speeds of over 100,000 queries per hour. Tuning various Hive and YARN parameters such as increasing the number of executor and I/O threads, memory allocation, and disabling consistent splits between LLAP daemons and data nodes was needed to reach this performance level on a test cluster of 45 nodes. Future work includes adding a web UI for monitoring LLAP clusters and implementing column-level access controls while allowing other frameworks like Spark to still access data through HiveServer2 and prevent direct access to HDFS for security reasons.
To The Cloud and Back: A Look At Hybrid Analytics
This document discusses hybrid analytics and the movement of data between on-premise and cloud environments. It notes that IT infrastructures now require both traditional and cloud-native approaches. Moving forward, hybrid analytics using both on-premise and cloud resources will become more common. Specifically, automatically moving data between on-premise storage and the cloud based on policies will help break down data silos and enable greater analysis. The document also explores using Dell EMC's OneFS software to enable this type of policy-based data movement to and from the cloud.
MapR M7: Providing an enterprise quality Apache HBase API
The document provides an overview of MapR M7, an integrated system for structured and unstructured data. M7 combines aspects of LSM trees and B-trees to provide faster reads and writes compared to Apache HBase. It achieves instant recovery from failures through its use of micro write-ahead logs and parallel region recovery. Benchmark results show MapR M7 providing 5-11x faster performance than HBase for common operations like reads, updates, and scans.
The Future of Apache Storm
This document summarizes updates to Apache Storm presented by P. Taylor Goetz of Hortonworks at Hadoop Summit 2016. Some key points include: Storm 0.9.x added high availability features and expanded integration capabilities. Storm 1.0 focused on maturity and improved performance. New features in Storm 1.0 include Pacemaker replacing Zookeeper, distributed caching, high availability Nimbus, native streaming windows, and state management with automatic checkpointing. Storm usability was also improved with features like dynamic log levels, tuple sampling for debugging, and distributed log searching. Future integrations and performance optimizations were also discussed.
Spark vstez
This is slides from our recent HadoopIsrael meetup. It is dedicated to comparison Spark and Tez frameworks.
In the end of the meetup there is small update about our ImpalaToGo project.
Yahoo's Experience Running Pig on Tez at Scale
Yahoo migrated most of its Pig workload from MapReduce to Tez to achieve significant performance improvements and resource utilization gains. Some key challenges in the migration included addressing misconfigurations, bad programming practices, and behavioral changes between the frameworks. Yahoo was able to run very large and complex Pig on Tez jobs involving hundreds of vertices and terabytes of data smoothly at scale. Further optimizations are still needed around speculative execution and container reuse to improve utilization even more. The migration to Tez resulted in up to 30% reduction in runtime, memory, and CPU usage for Yahoo's Pig workload.
Fast SQL on Hadoop, Really?
The document discusses Apache Hive and Apache Druid for fast SQL on big data. It provides performance benchmarks showing Hive LLAP is faster than Presto and Spark SQL for TPC-DS queries. It describes features of Hive LLAP including in-memory caching, query result caching, and metadata caching. It also discusses new Hive 3 features like materialized views and optimizer improvements. The document then provides an overview of Apache Druid's capabilities for real-time ingestion and querying of streaming data before discussing how Hive and Druid can work together, with Hive able to push down queries to Druid.
LLAP Nov Meetup
The document discusses Live Long and Process (LLAP), a new capability in Apache Hive that enables sub-second query performance. LLAP achieves this through caching the hottest data in RAM on each Hadoop node and running queries against this cache via lightweight long-running daemon processes. It allows for 100% SQL compatibility while integrating with existing security and tools. LLAP provides benefits like failure tolerance, concurrency, ACID transactions, and elastic scaling. Performance tests on TPC-DS queries demonstrated sub-second latency for queries even at large data scales and high concurrency levels.
Ingest and Stream Processing - What will you choose?
This document discusses streaming data ingestion and processing options. It provides an overview of common streaming architectures including Kafka as an ingestion hub and various streaming engines. Spark Streaming is highlighted as a popular and full-featured option for processing streaming data due to its support for SQL, machine learning, and ease of transition from batch workflows. The document also briefly profiles StreamSets Data Collector as a higher-level tool for building streaming data pipelines.
Architectural Overview of MapR's Apache Hadoop Distribution
Describes the thinking behind MapR's architecture. MapR"s Hadoop achieves better reliability on commodity hardware compared to anything on the planet, including custom, proprietary hardware from other vendors. Apache HDFS and Cassandra replication is also discussed, as are SAN and NAS storage systems like Netapp and EMC.
Stinger Initiative - Deep Dive
The document discusses the Stinger Initiative from Hortonworks to improve the performance and capabilities of interactive queries in Hive. The initiative takes a two-pronged approach, focusing on improvements to the query engine and the introduction of a new optimized column store file format called ORCFile. A new Tez execution engine is also introduced to avoid bottlenecks in MapReduce and enable lower latency queries. The goal is to extend Hive's ability to handle interactive queries with response times measured in seconds rather than minutes.
YARN Federation
This document discusses YARN federation, which allows multiple YARN clusters to be connected together. It summarizes:
- YARN is used at Microsoft for resource management but faces challenges of large scale and diverse workloads. Federation aims to address this.
- The federation architecture connects multiple independent YARN clusters through centralized services for routing, policies, and state. Applications are unaware and can seamlessly run across clusters.
- Federation policies determine how work is routed and scheduled across clusters, balancing objectives like load balancing, scaling, fairness, and isolation. A spectrum of policy options is discussed from full partitioning to full replication to dynamic partial replication.
- A demo is presented showing a job running across
LLAP: Sub-Second Analytical Queries in Hive
The document discusses LLAP (Live Long and Process), a new execution layer for Hive that enables sub-second analytical queries. LLAP uses daemons running on worker nodes to cache data in memory and keep query fragments executing between queries for faster performance. It allows for highly concurrent queries without specialized YARN queues. Benchmarks show LLAP providing up to 90% faster performance over Hive for queries against large datasets. LLAP also aims to serve as a unified data access layer for other systems like Spark SQL.
Elastic Scaling of a High-Throughput Content-Based Publish/Subscribe Engine
Publish/subscribe (pub/sub) infrastructures, running as a service in cloud environments, offer simplicity and flexibility for composing distributed applications. Provisioning them appropriately is, however, challenging. The amount of stored subscriptions and incoming publications varies over time, and the computational cost depends on the nature of the applications and in particular on the filtering operations they require, e.g., content-based vs. topic-based, encrypted vs. non-encrypted filtering. The ability to elastically adapt the amount of resources required to sustain given throughput and delay requirements is key to achieving cost-effectiveness for a pub/sub service running in a cloud environment. In this paper, we present the design and evaluation of an elastic content-based pub/sub system: eStreamHub. Specific contributions of this paper include: (1) a mechanism for dynamic scaling, both in and out, of stateful and stateless pub/sub operators, (2) a local and global elasticity policy enforcer maintaining high system utilization and stable end-to-end latencies, and (3) an evaluation using real-world tick workload from the Frankfurt Stock Exchange and encrypted content-based filtering.
Real-time Hadoop: The Ideal Messaging System for Hadoop
Ted Dunning presents on streaming architectures and MapR Technologies' streaming capabilities. He discusses MapR Streams, which implements the Kafka API for high performance and scale. MapR provides a converged data platform with files, tables, and streams managed under common security and permissions. Dunning reviews several use cases and lessons learned around real-time data processing, microservices, and global data management requirements.
LLAP: Sub-Second Analytical Queries in Hive
The document discusses LLAP (Live Long and Process), a new execution engine in Apache Hive 2.0 that enables sub-second analytical queries. LLAP keeps a small subset of frequently accessed data in memory to enable faster query processing times compared to traditional Hive architectures that rely on disk access. It works by running Hive query fragments simultaneously in both YARN containers and long-running daemon processes that cache data in memory. This allows for highly concurrent query execution without specialized YARN configurations. The document provides details on how LLAP is implemented and evaluates its performance benefits based on benchmarks and customer case studies.
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
This document discusses using Spark as an execution engine for Hive queries. It begins by explaining that Hive and Spark are both commonly used in the big data space, and that Hive on Spark uses the Hive optimizer with the Spark query engine, while Spark with a Hive context uses both the Catalyst optimizer and Spark engine. The document then covers challenges in deploying Hive on Spark, such as using a custom Spark JAR without Hive dependencies. It shows how the Hive EXPLAIN command works the same on Spark, and how the execution plan and stages differ between MapReduce and Spark. Overall, the document provides a high-level overview of using Spark as a query engine for Hive.
Resource Aware Scheduling in Apache Storm
This document provides an overview of resource aware scheduling in Apache Storm. It discusses the challenges of scheduling Storm topologies at Yahoo scale, including increasing heterogeneous clusters, low cluster utilization, and unbalanced resource usage. It then introduces the Resource Aware Scheduler (RAS) built for Storm, which allows fine-grained resource control and isolation for topologies through APIs and cgroups. Key features of RAS include pluggable scheduling strategies, per user resource guarantees, and topology priorities. Experimental results from Yahoo Storm clusters show significant improvements to throughput and resource utilization with RAS. Future work may include improved scheduling strategies and real-time resource monitoring.
What's hot (20)
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
MapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase API
Ingest and Stream Processing - What will you choose?
Ingest and Stream Processing - What will you choose?
Architectural Overview of MapR's Apache Hadoop Distribution
Architectural Overview of MapR's Apache Hadoop Distribution
Elastic Scaling of a High-Throughput Content-Based Publish/Subscribe Engine
Elastic Scaling of a High-Throughput Content-Based Publish/Subscribe Engine
Real-time Hadoop: The Ideal Messaging System for Hadoop
Real-time Hadoop: The Ideal Messaging System for Hadoop
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
Viewers also liked
One Click Hadoop Clusters - Anywhere (Using Docker)
This document provides an overview of Hortonworks' one-click Hadoop clusters solution that uses Docker and Apache Ambari to provision Hadoop clusters on any infrastructure. It discusses the goals of automating and unifying the Hadoop cluster provisioning process. The technology stack uses Docker for containerization, Swarm for clustering and orchestration, Consul for service discovery and registration, and Ambari for provisioning and management. Cloudbreak is introduced as a cloud-agnostic API that abstracts Hadoop cluster provisioning using these tools, while Periscope provides heuristic scheduling and autoscaling of clusters based on service level agreements.
Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...
This document discusses integrating an existing HPC infrastructure with the Hadoop ecosystem and YARN. It proposes building a custom YARN Application Master that acts as a "valve" between YARN and the HPC scheduler, allowing HPC applications to run on the shared infrastructure while reusing existing hardware. The advantages are better resource utilization and allowing new systems to leverage existing infrastructure. Potential drawbacks include added complexity from the HPC Application Master and slower performance from gradual changes.
Inspiring Travel at Airbnb [WIP]
This document discusses Airbnb's discovery product and experiments. It provides details on location recommendations, attributes, and A/B testing experiments to measure the impact of discovery features on user behavior like search, contact, and bookings. The goal is to increase bookings by testing different discovery integrations and content on the homepage and weekend pages. Tests showed reduced bounce rates and increased bookings for visitors who engaged with discovery content.
Hadoop in Validated Environment - Data Governance Initiative
The document discusses Merck's data governance initiative to implement Hadoop and establish data governance policies and processes. It outlines their 90 day and 6 month roadmaps to set up a data lake, implement data cataloging and metadata management, and establish access controls and data lineage tracking to enable self-service analytics while ensuring data governance and security. Key components include Hadoop, Hive, HDFS, Ranger, Falcon, and metadata stored in the Hive metastore.
The Most Valuable Customer on Earth-1298: Comic Book Analysis with Oracel's B...
This document contains slides from a presentation given by Dan McClary of Oracle about using big data tools to analyze comic books. It discusses how Oracle uses its own Hadoop cluster to analyze hardware failure data and improve customer service. It then presents Oracle's big data product suite for Hadoop, including tools for data integration, SQL querying, and graph analysis. Examples are given of customers using Oracle's Hadoop solutions for tasks like modernizing data models and answering questions faster. The document concludes by posing questions about determining the most important Marvel hero that could be answered using different big data techniques.
Running Spark and MapReduce together in Production
This document discusses running Spark and MapReduce together in production environments and some of the challenges that arise. It addresses issues like resource conflicts that can occur when running interactive frameworks like Spark alongside batch frameworks like MapReduce. Specific solutions discussed include increasing container sizes, reserving nodes for interactive workloads, and using features like node labels and gang scheduling to avoid fragmentation. Ongoing challenges of security, web serving, resource attribution, and balancing stability vs agility are also covered. The document advocates for using Hadoop as a service rather than managing it yourself to avoid spending time on cluster maintenance.
Karta an ETL Framework to process high volume datasets
Karta is an ETL framework developed by eBay to process large, complex datasets on Hadoop. It uses a XML-based configuration to define ETL flows that can perform tasks like copying, transforming, joining and versioning data. Karta allows users with limited Hadoop skills to build ETL jobs and has been used to load over 1.3 billion records from hundreds of files onto Hadoop in a single load. It provides consistent performance and has helped eBay process daily loads of 400 million records within the required SLAs.
50 Shades of SQL
The document discusses Seagate's plans to integrate hard disk drives (HDDs) with flash storage, systems, services, and consumer devices to deliver unique hybrid solutions for customers. It notes Seagate's annual revenue, employees, manufacturing plants, and design centers. It also discusses Seagate exploring the use of big data analytics and Hadoop across various potential use cases and outlines Seagate's high-level plans for Hadoop implementation.
Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...
The document discusses re-platforming existing enterprise business intelligence and analytic workloads from platforms like Oracle, Teradata, SAP and IBM to the Hadoop platform. It notes that many existing analytic workloads are struggling with increasing data volumes and are too costly. Hadoop offers a modern distributed platform that can address these issues through the use of a production-grade SQL database like VectorH on Hadoop. The document provides guidelines for re-platforming workloads and notes potential benefits such as improved performance, reduced costs and leveraging the Hadoop ecosystem.
Hadoop for Genomics__HadoopSummit2010
This document discusses using Hadoop for genomics applications. It provides an overview of genomics and DNA sequencing. Traditional DNA sequencing is expensive and produces large datasets that are challenging to analyze. Hadoop is well-suited for genomics applications because it can handle large datasets in a distributed, parallel manner. The document describes how Hadoop can be used to align DNA reads to a reference genome and call consensus sequences from aligned reads. It also discusses existing tools that use Hadoop for genomics and opportunities to develop new Hadoop-based tools and pipelines.
HBase and Drill: How loosley typed SQL is ideal for NoSQL
The document discusses how complex data structures can be modeled in a database using an extended relational model. It begins with an agenda that includes discussing loose typing, examples of what can be done, and looking at a real database with 10-20x fewer tables. It then contrasts the traditional relational model with HBase and discusses how structuring allows complex objects in fields and references between objects. Examples are given of modeling time-series data and music metadata in fewer tables using these techniques. Apache Drill is presented as a way to perform SQL queries over these complex data structures.
Carpe Datum: Building Big Data Analytical Applications with HP Haven
This document discusses HP's Haven platform for building big data analytical applications. Key points include:
- Haven includes HP Vertica for SQL analytics on Hadoop, IDOL for analyzing human data, and Distributed R for predictive analytics leveraging Hadoop clusters.
- Vertica provides fast SQL queries on Hadoop data and integration with Hadoop systems like Ambari and Kerberos.
- Distributed R allows building predictive models on large datasets using familiar R tools, vastly improving performance over single-node R.
- The full Haven platform is presented as providing 100% of data harnessing, extreme scale and speed in analytics to seamlessly deploy applications anywhere using Hadoop, private/public clouds, and advanced analytics
Big Data Simplified - Is all about Ab'strakSHeN
This document discusses designing a new big data platform to replace an existing complex and outdated one. It analyzes challenges with the current platform, including inability to keep up with business needs. The proposed new platform called Dredge would use abstraction layers to integrate big data tools in a loosely coupled and scalable way. This would simplify development and maintenance while supporting business goals. Key aspects of Dredge include declarative configuration, logical workflows, and plug-and-play integration of tools like HDFS, Hive, HBase, Kafka and Spark in a reusable and event-driven manner. The new platform aims to improve scalability, reduce costs and better support analytics needs over time.
Practical Distributed Machine Learning Pipelines on Hadoop
This document summarizes machine learning pipelines in Apache Spark using MLlib. It introduces Spark DataFrames for structured data manipulation and Apache Spark MLlib for building machine learning workflows. An example text classification pipeline is presented to demonstrate loading data, feature extraction, training a logistic regression model, and evaluating performance. Parameter tuning is discussed as an important part of the machine learning process.
DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARN
DeathStar is a system that runs HBase on YARN to provide easy, dynamic multi-tenant HBase clusters via YARN. It allows different applications to run HBase in separate application-specific clusters on a shared HDFS and YARN infrastructure. This provides strict isolation between applications and enables dynamic scaling of clusters as needed. Some key benefits are improved cluster utilization, easier capacity planning and configuration, and the ability to start new clusters on demand without lengthy provisioning times.
Open Source SQL for Hadoop: Where are we and Where are we Going?
Teradata has acquired Hadapt and the Teradata Center for Hadoop now has 40 developers working on open source SQL technologies like Presto. Teradata is committing resources to advancing Presto's open source codebase through contributions and plans to offer the first commercial support for Presto. Presto is an open source distributed SQL query engine that is optimized for interactive queries across data platforms.
Spark Application Development Made Easy
The document discusses Spark and common problems that can occur when using Spark. It notes that failures, wrong results, poor performance, poor scalability, and application, data/storage, Spark, and resource problems can all occur. It asks how application developers currently detect and fix these issues using logs, but notes logs are spread out, incomplete and difficult to understand. It proposes a better approach would be to visualize all relevant data in one place, optimize by analyzing the data to provide diagnoses and fixes, and help strategize to prevent problems and meet goals. It lists some existing tools for Hadoop and Spark that provide some level of visualization, optimization and strategization capabilities.
NoSQL Needs SomeSQL
This document summarizes key differences between SQL and NoSQL databases. It discusses the history and advantages of SQL, including its ubiquitous use, standardization, and ability to optimize queries based on schema. However, SQL has limitations for sparse and complex data. The document then introduces NoSQL databases and how they are designed for scale-out. It uses HBase as an example to illustrate how NoSQL databases can solve SQL's distributed join challenges by co-locating related data without schemas. While this improves performance, it also has disadvantages for modeling relationships and exploring data.
Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters
This document discusses the Cloud and Information Services Lab (CISL) and its vision of having one cluster that can run all workloads. It describes CISL's research into improving resource management in shared clusters through projects like Mercury, which aims to improve cluster utilization by opportunistically using otherwise idle resources. The document outlines Mercury's architecture and how it implements a hybrid of centralized and distributed scheduling to better schedule tasks with short durations.
Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...
Spark Streaming provides fault-tolerant stream processing capabilities to Spark. To achieve fault-tolerance and exactly-once processing semantics in production, Spark Streaming uses checkpointing to recover from driver failures and write-ahead logging to recover processed data from executor failures. The key aspects required are configuring automatic driver restart, periodically saving streaming application state to a fault-tolerant storage system using checkpointing, and synchronously writing received data batches to storage using write-ahead logging to allow recovery after failures.
Viewers also liked (20)
One Click Hadoop Clusters - Anywhere (Using Docker)
One Click Hadoop Clusters - Anywhere (Using Docker)
Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...
Coexistence and Migration of Vendor HPC based infrastructure to Hadoop Ecosys...
Hadoop in Validated Environment - Data Governance Initiative
Hadoop in Validated Environment - Data Governance Initiative
The Most Valuable Customer on Earth-1298: Comic Book Analysis with Oracel's B...
The Most Valuable Customer on Earth-1298: Comic Book Analysis with Oracel's B...
Running Spark and MapReduce together in Production
Running Spark and MapReduce together in Production
Karta an ETL Framework to process high volume datasets
Karta an ETL Framework to process high volume datasets
Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...
Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...
HBase and Drill: How loosley typed SQL is ideal for NoSQL
HBase and Drill: How loosley typed SQL is ideal for NoSQL
Carpe Datum: Building Big Data Analytical Applications with HP Haven
Carpe Datum: Building Big Data Analytical Applications with HP Haven
Practical Distributed Machine Learning Pipelines on Hadoop
Practical Distributed Machine Learning Pipelines on Hadoop
DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARN
DeathStar: Easy, Dynamic, Multi-Tenant HBase via YARN
Open Source SQL for Hadoop: Where are we and Where are we Going?
Open Source SQL for Hadoop: Where are we and Where are we Going?
Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters
Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters
Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...
Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...
Similar to Realistic Synthetic Generation Allows Secure Development
Anomaly Detection - New York Machine Learning
Anomaly detection is the art of finding what you don't know how to ask for. In this talk, I walk through the why and how of building probabilistic models for a variety of problems including continuous signals and web traffic. This talk blends theory and practice in a highly approachable way.
Deep Learning for Fraud Detection
This document summarizes a presentation on deep learning and fraud detection. The presentation explores the state of the art in deep learning and fraud detection, provides guidance on getting results, and includes experiments. The agenda includes discussing motivation for advanced modeling in fraud detection, explaining neural networks and deep learning, and exploring sample fraud detection features and challenges. Examples of applying clustering and autoencoders to time series anomaly detection and card velocity fraud detection are also summarized.
Predictive Analytics with Hadoop
This document discusses predictive analytics using Hadoop. It provides examples of recommendation and classification using big data. It describes obtaining large training datasets through crowdsourcing and implicit feedback. It also discusses operational considerations for predictive models, including snapshotting data, leveraging NFS for ingestion, and ensuring high availability. The document concludes with a question and answer section.
How to find what you didn't know to look for, oractical anomaly detection
This document summarizes Ted Dunning's presentation on anomaly detection, time-series databases, and deep learning. The presentation discusses what anomaly detection is, provides examples of detecting anomalies in time-series data, and explains how techniques like sparse coding, autoencoders, and deep learning can be used to build probabilistic models of normal behavior and detect anomalies. It also discusses how these techniques can be applied to build time-series databases that enable semantic search and anomaly detection in real-time and long-term data.
How to Determine which Algorithms Really Matter
Ted Dunning presents on algorithms that really matter for deploying machine learning systems. The most important advances are often not the algorithms but how they are implemented, including making them deployable, robust, transparent, and with the proper skillsets. Clever prototypes don't matter if they can't be standardized. Sketches that produce many weighted centroids can enable online clustering at scale. Recursive search and recommendations, where one implements the other, can also be important.
Practical Computing with Chaos
Ted Dunning gives a presentation on practical machine learning techniques. He discusses randomized geo-coding to embed spatial data in a total order. He demonstrates Thompson sampling, a Bayesian approach to exploration-exploitation tradeoffs. He also covers dithering in signals and generating synthetic data when real data cannot be shared due to privacy issues.
Practical Computing With Chaos
Ted Dunning presents on practical machine learning techniques. He discusses randomized geo-coding to derive keys that preserve locality. He demonstrates Thompson sampling, a Bayesian approach to exploration versus exploitation tradeoffs. He also covers using dithering to add noise that improves signals and synthetic data generation without privacy violations by sharing only statistical summaries.
Cheap learning-dunning-9-18-2015
This talk focuses on how larger data sets are not only enabling advanced techniques, but also increasing the number of problems within reach of relatively simple techniques, that is "cheap learning".
Ted Dunning, Chief Application Architect, MapR at MLconf ATL - 9/18/15
Complement Deep Learning with Cheap Learning: Recent results of deep learning on hard problems has set the data world all a titter and made deep learning the fashion of the time.
But it is very important to remember that as data expands, the learning problems that are encountered are often nearly green field problems and it is often possible to solve these problems using remarkably simple techniques. Indeed, on many problems these simple techniques will give results as good as more complex ones, not because they are profound, but because many problems become simpler at scale.
That said, it isn’t always obvious how to do this. I will describe some of these techniques and show how they can be applied in practice.
Hadoop and R Go to the Movies
This document discusses using machine learning algorithms to model energy use. It describes how evolutionary algorithms can be used to model energy use in homes and offices, which can help identify problems and save energy. Specifically, it discusses using a meta-evolutionary algorithm that adapts the mutation rate over time based on what has worked best. This approach converges faster than traditional evolutionary algorithms. The document then discusses challenges with visualizing high-dimensional models and solutions for diagnostic visualization and video generation to help understand the machine learning process and solutions.
Sharing Sensitive Data Securely
These are the slides from my talk at FAR Con in Minneapolis recently. The topics are the implications of buried treasure hoards on data security, horror stories and new, simpler and provably secure methods for public data disclosure.
Anomaly Detection: How to find what you didn’t know to look for
A description of a range of basic anomaly detection methods that can be applied to practical problems.
How to tell which algorithms really matter
This document discusses how to determine which algorithms and techniques really matter for practical applications.
It provides examples of techniques that had a big impact on recommendations, such as result dithering which improved user experience more than advances in recommendation algorithms. It also discusses how techniques like Bayesian bandits and online clustering with sketches can enable effective exploration and handle large, streaming data. Additionally, it describes how search and recommendations can recursively improve each other through indicator matrices.
MapR & Skytree:
Predicting failure in power networks, detecting fraudulent activities in payment card transactions, and identifying next logical products targeted at the right customer at the right time all require machine learning around massive data sets. This form of artificial intelligence requires complex self-learning algorithms, rapid data iteration for advanced analytics and a robust big data architecture that’s up to the task.
Learn how you can quickly exploit your existing IT infrastructure and scale operations in line with your budget to enjoy advanced data modeling, without having to invest in a large data science team.
Deep Learning vs. Cheap Learning
The document discusses how big data has enabled new opportunities by changing scaling laws and problem landscapes. Specifically, linearly scaling costs with big data now make it feasible to process large amounts of data, opening up many problems that were previously impossible or too difficult. This has created many "green field" opportunities where simple approaches can solve important problems. Two examples discussed are using log analysis to detect security threats and using transaction histories to find a common point of compromise for a data breach.
HUG_Ireland_Streaming_Ted_Dunning
A presentation by Ted Dunning of MapR on why Streaming Matters made to the Hadoop User Group (HUG) Ireland at Hadoop Summit on April 12th 2016. This presentation covers streaming and why it is so important in any big data solution
Recommendation Techn
Ted Dunning discusses recommendation systems and Apache Mahout. He explains how recommendation works using user-item interaction data to identify patterns and make predictions. Recommendation engines can suggest additional items like movies, music, or restaurants based on a user's preferences and behaviors. Dunning outlines the process of building recommendation models with Mahout including transforming data into user-item matrices and using techniques like log likelihood ratios to identify meaningful relationships.
Ted Dunning - Keynote: How Can We Take Flink Forward?
http://flink-forward.org/kb_sessions/keynote-tba/
Apache Flink has come a long way from its academic beginnings. It is now one of the most technically advanced solutions for streaming computation. And batch computation, too. Flink has serious technical advantages when compared with nearly every alternative system.
This success ironically means that Apache Flink is right on the cusp of a critical moment. Over the next few months it will be decided whether Flink is the Next Big Thing or if it is a fine technology with limited impact.
Right now, what you and I do can make a huge difference. But as business people like to say, what got Flink here isn’t what’s going to get it there. The challenges the Flink community faces now are different from the technical challenges it has met so far.
I will talk about what I think will help and how we can all pitch in to take Flink forward.
Cognitive computing with big data, high tech and low tech approaches
I explain some very approachable methods for analyzing big data via a detour through clipper ships and the 19th century open source scene.
Note that I mixed up the route of the Flying Cloud record in this talk. The Flying Cloud's record was actually from New York to San Francisco and was even more impressive than what I said. The usual time had been about 180 days. With Maury's charts, the time was reduced to about 135 days. The Flying Cloud's time was 89 days.
Thanks to Chen Kung for noticing my error.
Architecting R into Storm Application Development Process
This document discusses combining R and Storm to perform real-time analytics on streaming data. R is a programming language for advanced statistics while Storm is a framework for processing streaming data. The document proposes running R code inside Storm bolts to leverage R's statistical capabilities for online change point detection on streaming data. As a demonstration, it detects change points in Oakland A's game score differences during their 2002 20-game winning streak, but does not find any, as it is not using the optimal data. Integrating further with data modeling teams is suggested. Combining R and Storm provides benefits like independent development timelines while enabling real-time statistical analysis on data streams.
Similar to Realistic Synthetic Generation Allows Secure Development (20)
How to find what you didn't know to look for, oractical anomaly detection
How to find what you didn't know to look for, oractical anomaly detection
Ted Dunning, Chief Application Architect, MapR at MLconf ATL - 9/18/15
Ted Dunning, Chief Application Architect, MapR at MLconf ATL - 9/18/15
Anomaly Detection: How to find what you didn’t know to look for
Anomaly Detection: How to find what you didn’t know to look for
Ted Dunning - Keynote: How Can We Take Flink Forward?
Ted Dunning - Keynote: How Can We Take Flink Forward?
Cognitive computing with big data, high tech and low tech approaches
Cognitive computing with big data, high tech and low tech approaches
Architecting R into Storm Application Development Process
Architecting R into Storm Application Development Process
More from DataWorks Summit
Data Science Crash Course
Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL).
Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW).
Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models.
Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.
Floating on a RAFT: HBase Durability with Apache Ratis
In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort.
This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase.
Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs.
Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables.
Resources:
https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html
https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html
https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html
HBase Tales From the Trenches - Short stories about most common HBase operati...
Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.
Managing the Dewey Decimal System
OCLC has been using HBase since 2012 to enable single-search-box access to over a billion items from your library and the world’s library collection. This talk will provide an overview of how HBase is structured to provide this information and some of the challenges they have encountered to scale to support the world catalog and how they have overcome them.
Practical NoSQL: Accumulo's dirlist Example
Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL.
Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist).
In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.
HBase Global Indexing to support large-scale data ingestion at Uber
Danny Chen presented on Uber's use of HBase for global indexing to support large-scale data ingestion. Uber uses HBase to provide a global view of datasets ingested from Kafka and other data sources. To generate indexes, Spark jobs are used to transform data into HFiles, which are loaded into HBase tables. Given the large volumes of data, techniques like throttling HBase access and explicit serialization are used. The global indexing solution supports requirements for high throughput, strong consistency and horizontal scalability across Uber's data lake.
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions.
These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
This document discusses using Apache NiFi to build a high-speed cyber security data pipeline. It outlines the challenges of ingesting, transforming, and routing large volumes of security data from various sources to stakeholders like security operations centers, data scientists, and executives. It proposes using NiFi as a centralized data gateway to ingest data from multiple sources using a single entry point, transform the data according to destination needs, and reliably deliver the data while avoiding issues like network traffic and data duplication. The document provides an example NiFi flow and discusses metrics from processing over 20 billion events through 100+ production flows and 1000+ transformations.
Supporting Apache HBase : Troubleshooting and Supportability Improvements
This document discusses supporting Apache HBase and improving troubleshooting and supportability. It introduces two Cloudera employees who work on HBase support and provides an overview of typical troubleshooting scenarios for HBase like performance degradation, process crashes, and inconsistencies. The agenda covers using existing tools like logs and metrics to troubleshoot HBase performance issues with a general approach, and introduces htop as a real-time monitoring tool for HBase.
Security Framework for Multitenant Architecture
In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.
With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.
Extending Twitter's Data Platform to Google Cloud
Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.
Computer Vision: Coming to a Store Near You
Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as:
● Optimizing merchandising execution, in-stocks and sell-thru
● Enhancing operational efficiencies, enable real-time customer engagement
● Enhancing loss prevention capabilities, response time
● Creating frictionless experiences for shoppers
Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry.
We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey.
Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables.
We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance.
We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing.
Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems.
By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.
More from DataWorks Summit (20)
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
HBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at Uber
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything Engine
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Recently uploaded
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Webinar: Designing a schema for a Data Warehouse
Are you new to data warehouses (DWH)? Do you need to check whether your data warehouse follows the best practices for a good design? In both cases, this webinar is for you.
A data warehouse is a central relational database that contains all measurements about a business or an organisation. This data comes from a variety of heterogeneous data sources, which includes databases of any type that back the applications used by the company, data files exported by some applications, or APIs provided by internal or external services.
But designing a data warehouse correctly is a hard task, which requires gathering information about the business processes that need to be analysed in the first place. These processes must be translated into so-called star schemas, which means, denormalised databases where each table represents a dimension or facts.
We will discuss these topics:
- How to gather information about a business;
- Understanding dictionaries and how to identify business entities;
- Dimensions and facts;
- Setting a table granularity;
- Types of facts;
- Types of dimensions;
- Snowflakes and how to avoid them;
- Expanding existing dimensions and facts.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Recently uploaded (20)
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Realistic Synthetic Generation Allows Secure Development
- 1. © 2014 MapR Technologies 1© 2014 MapR Technologies
- 2. © 2014 MapR Technologies 2 Who am I? Ted Dunning, Chief Applications Architect MapR Technologies Email tdunning@mapr.com tdunning@apache.org Twitter @Ted_Dunning
- 3. © 2014 MapR Technologies 3 Short Books by Ted Dunning & Ellen Friedman • Published by O’Reilly in 2014 and 2015 • For sale from Amazon or O’Reilly • Free e-books currently available courtesy of MapR http://bit.ly/ebook-real- world-hadoop http://bit.ly/mapr-tsdb- ebook http://bit.ly/ebook- anomaly http://bit.ly/recommend ation-ebook
- 4. © 2014 MapR Technologies 4
- 5. © 2014 MapR Technologies 5 The basic idea
- 6. © 2014 MapR Technologies 6 Anomaly Detection and Fraud Analytics • Financial customer wants to identify zero-day attacks • And advanced persistent threats • By sophisticated adversaries who don’t use known vectors • Must keep logs and other data secret – But must also collaborate on detection algorithms
- 7. © 2014 MapR Technologies 7 Secure Development is Hard System knowledge Observed data Training algorithm Model New measurements Model Anomaly scores Model deployment
- 8. © 2014 MapR Technologies 8 Secure Development is Hard System knowledge Observed data Training algorithm Model New measurements Model Anomaly scores Model deployment Outside collaborators are outside the security perimeter They can’t see the data and they can’t tune new algorithms to fit reality
- 9. © 2014 MapR Technologies 9 How To Make Realistic Data System under test Live data Failure signatures Fake data Failure signatures
- 10. © 2014 MapR Technologies 10 Parametric Simulation Match here Live data System under test Failure signatures Fake data Failure signatures Fake data System under test Failure signatures Parametric matching of failure signatures allows emulation of complex data properties Matching on KPI’s and failure modes guarantees practical fidelity
- 11. © 2014 MapR Technologies 11 Do’s and Don’ts • Do match the KPI’s and failure modes – Speed – Score distribution – False positive rates versus score • Don’t try to match the actual data distribution precisely – Good enough is good enough and we want to imitate failures, not create new life forms – Probably impossible to do precisely – Even if possible, it is vastly harder to match distributions
- 12. © 2014 MapR Technologies 12 Methods for Generating Numbers • Well-known distributions – Uniform, normal, gamma, Poisson – Truncations • Cumulations – Random walk v1 • Mixture distributions • Hyper-parameters – Random walk v2
- 13. © 2014 MapR Technologies 13 Normal data = data.frame(x=rnorm(10000), y=rnorm(10000))
- 14. © 2014 MapR Technologies 14 Mixture of Normals
- 15. © 2014 MapR Technologies 15 Random Walk y = cumsum(rnorm(10000))
- 16. © 2014 MapR Technologies 16 Pick Mean from Multinomial
- 17. © 2014 MapR Technologies 17 Random Walk with Variable Standard Deviation y = cumsum(rt(10000, df=0.9))
- 18. © 2014 MapR Technologies 18 Methods for Generating Symbols • Symbols are really just integers with a dictionary • Well-known distributions – Multinomial – Dirichlet processes – Rich-get-richer, Pittman-Yor • Mixture distributions • Hyper-parameters • Lookup tables!!! – Simple tables – Data table joins for correlated components
- 19. © 2014 MapR Technologies 19 Skewed Integers 207 3 203 0 198 7 196 4 195 12 193 10 189 2 187 1 185 13 179 6 178 9 177 5 177 25 174 21 173 8 173 14 170 18 [ {"name":"x", "class":"int", "skew":1} ]
- 20. © 2014 MapR Technologies 20 Methods for Generating Behaviors • Use structured data! – Generate user meta-data – Generate list of transactions • Only flatten if necessary • See Apache Drill for post-processing
- 21. © 2014 MapR Technologies 21 Methods for Generating Databases • Use integers (see previous) as foreign keys • Normalized form implies (approximate) independence of tables
- 22. © 2014 MapR Technologies 22
- 23. © 2014 MapR Technologies 23 Go get log-synth https://github.com/tdunning/log-synth
- 24. © 2014 MapR Technologies 24 A worked example...
- 25. © 2014 MapR Technologies 25 Simulation Setup 0 20 40 60 80 100 0100300500 day count Compromise period Exploit period compromises frauds
- 26. © 2014 MapR Technologies 26
- 27. © 2014 MapR Technologies 27
- 28. © 2014 MapR Technologies 28 Questions?
- 29. © 2014 MapR Technologies 29 Thank You @mapr maprtech tdunning@mapr.com tdunning@apache.org Ted Dunning, ChiefApplicationArchitect MapRTechnologies maprtech mapr-technologies