Deep Learning 101
•
2 likes•305 views
In this talk, we will give a high-level overview of what is Deep Learning (DL) is and provide a lot of visual examples for an intuitive take-aways from the session – and no worries, no math here! The session will: - Illustrate what neural nets are via interactive examples and mini-demos - Show applications of DL to solve specific real-world problems with State-of-the-Art (SOTA) results - Demonstrate how latest research in DL can be applied today to areas of vision, language, recommender systems, and tabular data - Provide pro-tips on how to get started Speaker: Robert Hryniewicz, AI Evangelist, Hortonworks
Report
Share
Report
Share
Download to read offline
Recommended
Big Traffic, Big Trouble: Big Data - Tokyo
With the rise of IoT and complexity of applications, clouds, networks and infrastructure, it is becoming more difficult to protect data and infrastructure from attackers. When groups of bad actors collaborate, share information, provide unauthorized access, and do botnet as a service, attacks in terabit units also start easily. On the other hand, it is also difficult to find enough security analysts to deal with and defend against such attacks.
Here is the emergence of community cooperation like Apache Metron and efforts to open source. Metron provides a comprehensive framework for applications, networks and security built on Apache Hadoop and open source streaming analysis (eg Apache Nifi, Apache Kafka) tools in scalable data management and processing stacks. Extensions such as profiling, machine learning, and visualization work and real-time streaming detection make SOC analysts more efficient, while intrinsic scalability of open source gives data scientists security insight from data laboratories So that it can be quickly incorporated into production.
This section explains how real-world businesses and managed service providers use Apache Metron, identify and resolve security threats on a large scale, and explain methods and ideas for adapting the platform to your security architecture · I will demonstrate.
Big Traffic, Big Trouble: Big Data Security Analytics
With the rise of IoT and the increasing complexity of applications, clouds, networks and infrastructure, the battle to keep your data and your infrastructure safe from attackers is getting harder. As groups of bad actors collaborate, sharing information and offering illegal access, and botnets as a service, terabits of attack can be launched cheaply. Meanwhile, it’s hard to find enough security analysts to catch and prevent these attacks.
This is where community collaboration and open source efforts like Apache Metron come in. Metron presents a comprehensive framework for application and network, security built on Apache Hadoop and open source Streaming Analytics(ie Apache Nifi, Apache Kafka) tool’s highly scalable data management and processing stacks. Advanced features like profiling, machine learning, and visualization work with real-time streaming detection to make your SOC analysts more efficient, while the intrinsic extensibility of open source helps your data scientists get security insights out of the lab and into production fast.
We will discuss and demonstrate how some real-world businesses and managed service providers are using Apache Metron to identify and solve security threats at scale, and some approaches and ideas for how the platform can fit into your security architecture.
Running Enterprise Workloads with an open source Hybrid Cloud Data Architecture
The document discusses Hortonworks DataPlane Service (DPS), a platform that provides consistent security, governance, and management of data across hybrid cloud environments. Key capabilities of DPS include data lifecycle management using Data Lifecycle Manager (DLM), data discovery and profiling through Data Steward Studio (DSS), and self-service analytics with Data Analytics Studio (DAS). DPS provides a global data fabric to address challenges of securing, governing, and delivering data across multiple data sources and locations.
Fast SQL on Hadoop, really?
How is it that one system can query terabytes of data, yet still provide interactive query support? This talk will discuss two of the underlying technologies that allow Apache Hive to support fast query response, both on-premise in HDFS and in cloud object stores such as S3 and WASB.
LLAP was introduced in Hive 2.6. It provides standing processes that securely cache Hive’s columnar data and can do query processing without ever needing to start tasks in Hadoop. We will cover LLAP’s architecture, intended uses cases, and performance numbers for both on-premise and in the cloud.
The second technology is the integration of Hive with Apache Druid. Druid excels at low-latency, interactive queries over streaming data. Its method of storing data makes it very well suited for OLAP style queries. We will cover how Hive can be integrated with Druid to support real-time streaming of data from Kafka and OLAP queries.
10 Lessons Learned from Meeting with 150 Banks Across the Globe
This document summarizes 10 practical lessons learned from companies about their big data and analytics journeys:
1. There are clear leaders in each market who are gaining substantial benefits from using big data and machine learning, widening the gap with other companies.
2. Real transformation requires buy-in from top executives, as reflected by new innovation centers, roles, and organizations.
3. Projects should have clear revenue impact objectives and be selected based on estimated return, with pre- and post-implementation measurements.
4. While cost reduction brings the fastest ROI, new revenue opportunities can transform a business more lastingly if the projects address real customer and business needs.
The Car of the Future - Autonomous, Connected, and Data Centric
incredibly data intensive endeavor. Traditional data management approaches are straining to cope with the demands imposed by autonomous driving research.
This session investigates the role of data in teaching cars to drive and the data management challenges that automakers must overcome in achieving this objective. Finally, a modern data architecture, leveraging the latest advances in data management technologies is proposed to facilitate the promise of a self-driving future.
Speaker: Robert Hryniewicz, AI Evangelist, Hortonworks
Curing the Kafka Blindness – Streams Messaging Manager
Companies who use Kafka today struggle with monitoring and managing Kafka clusters. Kafka is a key backbone of IoT streaming analytics applications. The challenge is understanding what is going on overall in the Kafka cluster including performance, issues and message flows. No open source tool caters to the needs of different users that work with Kafka: DevOps/developers, platform team, and security/governance teams. See how the new Hortonworks Streams Messaging Manager enables users to visualize their entire Kafka environment end-to-end and simplifies Kafka operations.
In this session learn how SMM visualizes the intricate details of how Apache Kafka functions in real time while simultaneously surfacing every nuance of tuning, optimizing, and measuring input and output. SMM will assist users to quickly understand and operate Kafka while providing the much-needed transparency that sophisticated and experienced users need to avoid all the pitfalls of running a Kafka cluster.
Speaker: Andrew Psaltis, Principal Solution Engineer, Hortonworks
Using Spark Streaming and NiFi for the next generation of ETL in the enterprise
In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world so that nearly every streaming framework now supports higher level relational operations.
On paper, combining Apache NiFi, Kafka, and Spark Streaming provides a compelling architecture option for building your next generation ETL data pipeline in near real time. What does this look like in an enterprise production environment to deploy and operationalized?
The newer Spark Structured Streaming provides fast, scalable, fault-tolerant, end-to-end exactly-once stream processing with elegant code samples, but is that the whole story?
We discuss the drivers and expected benefits of changing the existing event processing systems. In presenting the integrated solution, we will explore the key components of using NiFi, Kafka, and Spark, then share the good, the bad, and the ugly when trying to adopt these technologies into the enterprise. This session is targeted toward architects and other senior IT staff looking to continue their adoption of open source technology and modernize ingest/ETL processing. Attendees will take away lessons learned and experience in deploying these technologies to make their journey easier.
Recommended
Big Traffic, Big Trouble: Big Data - Tokyo
With the rise of IoT and complexity of applications, clouds, networks and infrastructure, it is becoming more difficult to protect data and infrastructure from attackers. When groups of bad actors collaborate, share information, provide unauthorized access, and do botnet as a service, attacks in terabit units also start easily. On the other hand, it is also difficult to find enough security analysts to deal with and defend against such attacks.
Here is the emergence of community cooperation like Apache Metron and efforts to open source. Metron provides a comprehensive framework for applications, networks and security built on Apache Hadoop and open source streaming analysis (eg Apache Nifi, Apache Kafka) tools in scalable data management and processing stacks. Extensions such as profiling, machine learning, and visualization work and real-time streaming detection make SOC analysts more efficient, while intrinsic scalability of open source gives data scientists security insight from data laboratories So that it can be quickly incorporated into production.
This section explains how real-world businesses and managed service providers use Apache Metron, identify and resolve security threats on a large scale, and explain methods and ideas for adapting the platform to your security architecture · I will demonstrate.
Big Traffic, Big Trouble: Big Data Security Analytics
With the rise of IoT and the increasing complexity of applications, clouds, networks and infrastructure, the battle to keep your data and your infrastructure safe from attackers is getting harder. As groups of bad actors collaborate, sharing information and offering illegal access, and botnets as a service, terabits of attack can be launched cheaply. Meanwhile, it’s hard to find enough security analysts to catch and prevent these attacks.
This is where community collaboration and open source efforts like Apache Metron come in. Metron presents a comprehensive framework for application and network, security built on Apache Hadoop and open source Streaming Analytics(ie Apache Nifi, Apache Kafka) tool’s highly scalable data management and processing stacks. Advanced features like profiling, machine learning, and visualization work with real-time streaming detection to make your SOC analysts more efficient, while the intrinsic extensibility of open source helps your data scientists get security insights out of the lab and into production fast.
We will discuss and demonstrate how some real-world businesses and managed service providers are using Apache Metron to identify and solve security threats at scale, and some approaches and ideas for how the platform can fit into your security architecture.
Running Enterprise Workloads with an open source Hybrid Cloud Data Architecture
The document discusses Hortonworks DataPlane Service (DPS), a platform that provides consistent security, governance, and management of data across hybrid cloud environments. Key capabilities of DPS include data lifecycle management using Data Lifecycle Manager (DLM), data discovery and profiling through Data Steward Studio (DSS), and self-service analytics with Data Analytics Studio (DAS). DPS provides a global data fabric to address challenges of securing, governing, and delivering data across multiple data sources and locations.
Fast SQL on Hadoop, really?
How is it that one system can query terabytes of data, yet still provide interactive query support? This talk will discuss two of the underlying technologies that allow Apache Hive to support fast query response, both on-premise in HDFS and in cloud object stores such as S3 and WASB.
LLAP was introduced in Hive 2.6. It provides standing processes that securely cache Hive’s columnar data and can do query processing without ever needing to start tasks in Hadoop. We will cover LLAP’s architecture, intended uses cases, and performance numbers for both on-premise and in the cloud.
The second technology is the integration of Hive with Apache Druid. Druid excels at low-latency, interactive queries over streaming data. Its method of storing data makes it very well suited for OLAP style queries. We will cover how Hive can be integrated with Druid to support real-time streaming of data from Kafka and OLAP queries.
10 Lessons Learned from Meeting with 150 Banks Across the Globe
This document summarizes 10 practical lessons learned from companies about their big data and analytics journeys:
1. There are clear leaders in each market who are gaining substantial benefits from using big data and machine learning, widening the gap with other companies.
2. Real transformation requires buy-in from top executives, as reflected by new innovation centers, roles, and organizations.
3. Projects should have clear revenue impact objectives and be selected based on estimated return, with pre- and post-implementation measurements.
4. While cost reduction brings the fastest ROI, new revenue opportunities can transform a business more lastingly if the projects address real customer and business needs.
The Car of the Future - Autonomous, Connected, and Data Centric
incredibly data intensive endeavor. Traditional data management approaches are straining to cope with the demands imposed by autonomous driving research.
This session investigates the role of data in teaching cars to drive and the data management challenges that automakers must overcome in achieving this objective. Finally, a modern data architecture, leveraging the latest advances in data management technologies is proposed to facilitate the promise of a self-driving future.
Speaker: Robert Hryniewicz, AI Evangelist, Hortonworks
Curing the Kafka Blindness – Streams Messaging Manager
Companies who use Kafka today struggle with monitoring and managing Kafka clusters. Kafka is a key backbone of IoT streaming analytics applications. The challenge is understanding what is going on overall in the Kafka cluster including performance, issues and message flows. No open source tool caters to the needs of different users that work with Kafka: DevOps/developers, platform team, and security/governance teams. See how the new Hortonworks Streams Messaging Manager enables users to visualize their entire Kafka environment end-to-end and simplifies Kafka operations.
In this session learn how SMM visualizes the intricate details of how Apache Kafka functions in real time while simultaneously surfacing every nuance of tuning, optimizing, and measuring input and output. SMM will assist users to quickly understand and operate Kafka while providing the much-needed transparency that sophisticated and experienced users need to avoid all the pitfalls of running a Kafka cluster.
Speaker: Andrew Psaltis, Principal Solution Engineer, Hortonworks
Using Spark Streaming and NiFi for the next generation of ETL in the enterprise
In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world so that nearly every streaming framework now supports higher level relational operations.
On paper, combining Apache NiFi, Kafka, and Spark Streaming provides a compelling architecture option for building your next generation ETL data pipeline in near real time. What does this look like in an enterprise production environment to deploy and operationalized?
The newer Spark Structured Streaming provides fast, scalable, fault-tolerant, end-to-end exactly-once stream processing with elegant code samples, but is that the whole story?
We discuss the drivers and expected benefits of changing the existing event processing systems. In presenting the integrated solution, we will explore the key components of using NiFi, Kafka, and Spark, then share the good, the bad, and the ugly when trying to adopt these technologies into the enterprise. This session is targeted toward architects and other senior IT staff looking to continue their adoption of open source technology and modernize ingest/ETL processing. Attendees will take away lessons learned and experience in deploying these technologies to make their journey easier.
Data in the Cloud Crash Course
Introduction
This workshop is a hands-on session to quickly deploy Hadoop and Streaming on AWS / Azure / Google Cloud.
Cloudbreak simplifies the deployment of Hadoop in cloud environments. It enables the enterprise to quickly run big data workloads in the cloud while optimizing the use of cloud resources.
Format
A short introductory lecture about Cloudbreak. This is followed by a walk through and lab leveraging Hadoop and Streaming in the Cloud with Cloudbreak.
Objective
To provide a quick and short hands-on introduction to Hadoop on the cloud. Review key benefits of cluster deployment automation.
This lab will use Cloudbreak to quickly and effortlessly stand up Hadoop and Streaming clusters in a cloud provider of your choice. The lab shows the use of Ambari blueprints that are your declarative definitions of your Hadoop or Streaming clusters. Steps to dynamically change these blueprints and use external databases and external authentication sources and in essence showing a way to provide Shared Authentication, Authorization and Audit across ephemeral and long-lasting clusters. However it is not limited to only custom blueprints, the lab also shows how Cloudbreak provides easy to use custom scripts called recipes that can be executed before or after Ambari start or after cluster installation.
Deep learning 101
In this talk, we will give an overview of the deep learning space starting with a brief history. We will distinguish between deep learning hype vs practical real-world applications, cover how deep learning differs from other machine learning algorithms, go over sample neural net architectures, and provide a step-by-step guide on how to get started.
Specifically, we will cover what type of training data is required and how to prepare it with Apache Spark, followed by how to choose a correct neural net architecture, train, and deploy a deep learning model with TensorFlow on Apache Hadoop 3.1.
Finally, we will wrap-up with deep learning challenges and shortcomings, and offer short- and long-term recommendations to successfully train and deploy deep learning models within your organization to maximize return on investment.
Running Enterprise Workloads with an Open Source Hybrid Cloud Data Architecture
Cloud is turbocharging the Enterprise IT landscape with agility and flexibility. And now, discussions of cloud architecture dominate Enterprise IT. Cloud is enabling many ephemeral on-demand use cases which is a game-changing opportunity for analytic workloads. But all of this comes with the challenges of running enterprise workloads in the cloud securely and with ease.
With the convergence of cloud, IoT, and big data technologies, enterprises increasingly have their data spread across multiple data lakes on-prem and in cloud data lake stores in many geographies and across multiple public cloud vendor platforms, for example, due to regulatory and compliance mandates that limit cross-border data transfer. With the proliferation of data types and sources in this complex landscape, the process of discovery, provisioning, and running relevant workloads on this data to gather insights has become more complex. Additionally, gaining global visibility into the business context, usage, and trustworthiness of data requires a centralized view of all data and metadata, security controls, data access, and monitoring.
All of these challenges create a significant chasm between initial data capture and subsequent data insights generation to drive value creation. Therefore, enterprises now require a “global insight fabric” that can find a happy medium between adequate rules and policies of data governance while providing a trusted environment for users to collaborate and share data responsibly in order to create value.
In this talk, we will outline how Hortonworks DataPlane Service(DPS) can help customers build a global insight fabric that can span storing and analyzing data within data centers to implementing an open source hybrid architecture that takes advantage of cloud's elasticity and new use cases. We will get a personal view of the challenges faced in safely moving data from on-premises data centers into multiple public clouds, safeguarding it through replication, and then applying consistent security and governance policies across diverse environments to deliver trusted data and insights to the business. We will highlight how DataPlane Service can help enterprises with this hybrid architectural journey, and how open source architectures are enabling this transformation across enterprises.
Speaker: Alan Gates, Co-Founder, Hortonworks
Overcoming the AI hype — and what enterprises should really focus on
Deep learning for all its hype is brittle, non-generalizeable, and its learnings are not readily transferable from one application to another. Since we are unlikely to see anything close to artificial general intelligence in the next few decades., we should instead focus on how enterprises can capitalize on the state of the art in machine learning and re-implement successful algorithms and follow the data science lifecycles that generate highest ROI.
This talk will cover the current state of the art in AI, its limits vs. hype, and discuss concrete steps that enterprises can take to achieve desired ROI by re-implementing production-grade-ready machine learning algorithms, that have been hardened and demonstrated to work very well in specific, constrained domains.
By the end of this talk, attendees should have a better grasp on how to avoid costly and unnecessary investments into yet unproven technologies, be better equipped to navigate the complex space of AI, and understand where to best focus their resources to maximize ROI. ROBERT HRYNIEWICZ, Technical Evangelist, Hortonworks
The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi
Apache NiFi MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately.
Abstract: Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love.
Expected prior knowledge / intended audience: developers and data flow managers should have a passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi.
Takeaways: Attendees will learn about opportunities to bring their data flow and capture closer to the "edge" -- sources of data like IoT devices, vehicles, machinery, etc. They will understand the possibilities to prioritize, filter, secure, and manipulate this data earlier in the data lifecycle to enhance their data visibility and performance.
Speaker: Andy LoPresto, Sr. Member of Technical Staff, Hortonworks
Data in the Cloud Crash Course
Santhosh B Gowda presents on Cloudbreak, a tool for provisioning Hadoop clusters on cloud infrastructure. Cloudbreak allows for simplified cluster provisioning through prescriptive setups and automation. It supports declarative workload provisioning across multiple cloud providers with flexible topologies and security configuration options. Cloudbreak also enables features like auto-scaling, recipes to customize clusters, and shared services data lakes to provide common metadata and access management across ephemeral clusters. Demonstrations of launching HDP and HDF clusters from the Cloudbreak UI and CLI are also provided.
What is New in Apache Hive 3.0?
Apache Hive is a rapidly evolving project which continues to enjoy great adoption in the big data ecosystem. As Hive continues to grow its support for analytics, reporting, and interactive query, the community is hard at work in improving it along with many different dimensions and use cases. This talk will provide an overview of the latest and greatest features and optimizations which have landed in the project over the last year. Materialized views, the extension of ACID semantics to non-ORC data, and workload management are some noteworthy new features.
We will discuss optimizations which provide major performance gains, including significantly improved performance for ACID tables. The talk will also provide a glimpse of what is expected to come in the near future.
Speaker: Alan Gates, Co-Founder, Hortonworks
Running Enterprise Workloads with an open source Hybrid Cloud Data Architectu...
Cloud accelerates corporate IT landscapes with agility and flexibility. Today, discussion of cloud architecture dominates corporate IT. The cloud enables a number of temporary on-demand use cases that revolutionize analytical workload opportunities. But all of this involves the task of running corporate workloads safely and easily in the cloud.
With the convergence of cloud, IoT, and big data technology, enterprises are increasingly using multiple on-premises Data Lake and multiple Public on different geographies, for example due to regulations and compliance requirements restricting cross- It now distributes data to the cloud Data Lake store of the cloud vendor platform. Diffusion of data types and sources in this complex landscape makes the discovery process, provisioning, and getting insight by performing the appropriate workload on this data more complicated. In addition, to obtain business context, usage, and visibility of data trustworthiness worldwide, it is necessary to display all data and metadata, security management, data access, and monitoring in a centralized way .
All these problems create cracks during the creation of data insights to promote initial data capture and subsequent value creation. As a result, companies now look for compromises between appropriate rules and data control policies while providing a trusted environment that allows them to share data and partner with users responsibly to create value We need "Global Insight Fabric".
In this talk, how the Hortonworks DataPlane Service (DPS) analyzes the data in the data center to expand the storage, implement the open source hybrid architecture utilizing cloud flexibility and new use cases, global in Describes how site fabrics can help customers create. Securely migrate data from on-premises data centers to multiple public clouds, protect the data with replication, then apply consistent safety and governance policies to a wide variety of environments to ensure trustworthy data and inn We provide personal views on the challenges we face in providing the site to the business. I will explain how the DetaPlane service can be useful for traveling to this hybrid architecture and how the open source architecture enables the transformation of the entire enterprise.
The Implacable advance of the data
Data proliferation from 7+ billion humans and 20+ billion devices from every walk of life has been the focus in the last decade. With the velocity, variety and volume of data, every data organization’s goal shifted to protecting and monetizing data from rapidly growing network of IOT embedded objects and sensors.
One of the true and tried business continuity methodology of storing and retrieving vast amount of data has been through replication of Hadoop systems on hybrid clouds and in geographically distributed data centers. Replication is similar to Blockchain using autonomous smart contracts instantiated on the metadata and data so that the replicated data follows a single source of truth.
Replicas can be maintained across geographically distributed data centers giving greater risk tolerance capabilities to the businesses continuity plan for the data-sets. With intelligent predictive analytics based on usage patterns, dynamic tiering policies can be triggered on the data sets to provide true value-add to the data. The temperature of the data is used to move data between hot/warm/cold/archival storage based on configurable policies leading to greater reduction in total cost of ownership.
Users in 2018 and beyond demand absolute availability of data as and when they desire. The dynamic data access management is fundamental concept to satisfy the business continuity plan. Seamless enterprise-grade disaster recovery to support business continuity use case has significant challenges around replicating security and governance on data-sets. In this talk we will discuss how the above challenge can be addressed for supporting seamless replication and disaster recovery for Hadoop-scale data. NIRU ANISETI, Product Manager, Hortonworks
What's New in Apache Hive 3.0?
Apache Hive is a rapidly evolving project, many people are loved by the big data ecosystem. Hive continues to expand support for analytics, reporting, and bilateral queries, and the community is striving to improve support along with many other aspects and use cases. In this lecture, we introduce the latest and greatest features and optimization that appeared in this project last year. This includes benchmarks covering LLAP, Apache Druid's materialized views and integration, workload management, ACID improvements, using Hive in the cloud, and performance improvements. I will also tell you a little about what you can expect in the future.
Containers and Big Data
As containerization continues to gain momentum and become a de facto standard for application deployment, challenges around containerization of big data workloads are coming to light. Great strides have been made within the open source communities towards running big data workloads in containers, but much is left to be done.
Apache Hadoop YARN is the modern distributed operating system for big data applications. It has morphed the Hadoop compute layer into a common resource-management platform that can host a wide variety of applications. At its core, YARN has a very powerful scheduler which enforces global cluster level invariants and helps sites manage user and operator expectations of elastic sharing, resource usage limits, SLAs, and more. YARN recently increased its support for Docker containerization and added a YARN service framework supporting long-running services.
In this session we will explore the emerging patterns and challenges related to containers and big data workloads, including running applications such as Apache Spark, Apache HBase, and Kubernetes in containers on YARN.
Unlock Value from Big Data with Apache NiFi and Streaming CDC
The document discusses Apache NiFi and streaming change data capture (CDC) with Attunity Replicate. It provides an overview of NiFi's capabilities for dataflow management and visualization. It then demonstrates how Attunity Replicate can be used for real-time CDC to capture changes from source databases and deliver them to NiFi for further processing, enabling use cases across multiple industries. Examples of source systems include SAP, Oracle, SQL Server, and file data, with targets including Hadoop, data warehouses, and cloud data stores.
Containers and Big Data
As containerization continues to gain momentum and become a de facto standard for application deployment, challenges around containerization of big data workloads are coming to light. Great strides have been made within the open source communities towards running big data workloads in containers, but much is left to be done.
Apache Hadoop YARN is the modern distributed operating system for big data applications. It has morphed the Hadoop compute layer into a common resource-management platform that can host a wide variety of applications. At its core, YARN has a very powerful scheduler which enforces global cluster level invariants and helps sites manage user and operator expectations of elastic sharing, resource usage limits, SLAs, and more. YARN recently increased its support for Docker containerization and added a YARN service framework supporting long-running services.
In this session we will explore the emerging patterns and challenges related to containers and big data workloads, including running applications such as Apache Spark, Apache HBase, and Kubernetes in containers on YARN.
Speaker: Sanjay Radia, Chief Architect, Founder, Hortonworks
Intelligently Collecting Data at the Edge – Intro to Apache MiNiFi
The document appears to be a slide deck presentation about Apache NiFi and Apache MiNiFi. Some key points:
- Apache NiFi is a dataflow solution for automating the movement of data between systems. MiNiFi is a lighter-weight version designed for edge computing environments.
- The presentation discusses the challenges of IoT data collection and processing. MiNiFi is presented as a solution to run close to data sources on devices with limited resources.
- Examples are given showing MiNiFi running on a Raspberry Pi to collect credit card transaction data and sensor data from a Sense HAT, sending it to NiFi for further processing.
Keynote
1. Sanjay Radia discusses designing data systems for data everywhere, including on-premises and in the cloud.
2. He advocates for applying a consistent set of tools, managing data in one place, and sharing security and governance across systems.
3. Radia also notes the importance of leveraging innovation from open source communities to continually improve the systems.
Spark Summit EMEA - Arun Murthy's Keynote
Spark and Hadoop are perfectly together. Spark is a key tool in Hadoop's toolbox that provides elegant developer APIs and accelerates data science and machine learning. It can process streaming data in real-time for applications like web analytics and insurance claims processing. The future of Spark and Hadoop includes innovating the core technologies, providing seamless data access across data platforms, and further accelerating data science tools and libraries.
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Cloudbreak, a part of Hortonworks Data Platform (HDP), simplifies the provisioning and cluster management within any cloud environment to help your business toward its path to a hybrid cloud architecture.
https://hortonworks.com/webinar/getting-data-cloud-cloudbreak-live-demo/
Ozone and HDFS’s evolution
HDFS has several strengths: horizontally scale its IO bandwidth and scale its storage to petabytes of storage. Further, it provides very low latency metadata operations and scales to over 60K concurrent clients. Hadoop 3.0 recently added Erasure Coding. One of HDFS’s limitations is scaling a number of files and blocks in the system. We describe a radical change to Hadoop’s storage infrastructure with the upcoming Ozone technology. It allows Hadoop to scale to tens of billions of files and blocks and, in the future, to every larger number of smaller objects. Ozone fundamentally separates the namespace layer and the block layer allowing new namespace layers to be added in the future. Further, the use of RAFT protocol has allowed the storage layer to be self-consistent. We show how this technology helps a Hadoop user and also what it means for evolving HDFS in the future. We will also cover the technical details of Ozone.
Hortonworks Open Connected Data Platforms for IoT and Predictive Big Data Ana...
The energy industry is well known to be laggard adopters of new technology. However, industry challenges such as aging assets & workforce, increased regulatory scrutiny, renewable energy sources, depressed commodity prices, changing customer expectations, and growing data volumes are pushing companies to explore new technologies to help solve these problems. Learn how energy companies are leveraging Hortonworks Open and Connected Data Platforms to provide the predictive analysis and data insights to optimize performance for the energy industry.
Speaker
Kenneth Smith, General Manager, Energy, Hortonworks
Scalable and adaptable typosquatting detection in Apache Metron
Typosquatting is a form of cybersquatting (e.g. registering a domain that is similar to an existing domain) where a domain that is a common misspelling of another domain is registered and used for possibly malicious ends. It is an attack that has affected everyone from Google to the televangelist Jerry Falwell and has regulations against it encoded in US law. Even so, it remains extremely popular and particularly heinous and effective when used by advanced malicious actors who may rely on typosquatted domains to gull unwitting users to be more likely to click on a malicious URL in a spearphishing attack.
Detecting typosquatting attacks in realtime can be challenging as there are as many ways to typosquat as there are to make typos. Often, intrusion detection systems will generate the typosquatted domains and store them in a database for comparison. However, given the number of domains possible, this is a daunting task storage-wise. Furthermore, this approach can become out of date quickly.
We will talk about using sketching data structures in Metron to detect typosquatted domains scalably and adaptably. Furthermore, we will discuss how to ensure that the set of typosquatted domains is kept current with the domains actually seen in an organization's network. CASEY STELLA, Principal Software Engineer, Hortonworks and MICHAEL MIKLAVCIC, Staff Engineer, Hortonworks
[Hortonworks] Future Of Data: Madrid - HDF & Data in motion
First meetup event Future Of Data event where we introduced Hortonworks DataFlow (HDF).
The slides describe what HDF is, and we presented a very simple demo about sentiment analysis of tweets using Apache OpenNLP as the NLP framework to do so.
Apache Metron in the Real World
Dave Russell - Presented at Big Data Vilnius 27-29th November 2018 https://www.bigdataconference.lt/
More Related Content
What's hot
Data in the Cloud Crash Course
Introduction
This workshop is a hands-on session to quickly deploy Hadoop and Streaming on AWS / Azure / Google Cloud.
Cloudbreak simplifies the deployment of Hadoop in cloud environments. It enables the enterprise to quickly run big data workloads in the cloud while optimizing the use of cloud resources.
Format
A short introductory lecture about Cloudbreak. This is followed by a walk through and lab leveraging Hadoop and Streaming in the Cloud with Cloudbreak.
Objective
To provide a quick and short hands-on introduction to Hadoop on the cloud. Review key benefits of cluster deployment automation.
This lab will use Cloudbreak to quickly and effortlessly stand up Hadoop and Streaming clusters in a cloud provider of your choice. The lab shows the use of Ambari blueprints that are your declarative definitions of your Hadoop or Streaming clusters. Steps to dynamically change these blueprints and use external databases and external authentication sources and in essence showing a way to provide Shared Authentication, Authorization and Audit across ephemeral and long-lasting clusters. However it is not limited to only custom blueprints, the lab also shows how Cloudbreak provides easy to use custom scripts called recipes that can be executed before or after Ambari start or after cluster installation.
Deep learning 101
In this talk, we will give an overview of the deep learning space starting with a brief history. We will distinguish between deep learning hype vs practical real-world applications, cover how deep learning differs from other machine learning algorithms, go over sample neural net architectures, and provide a step-by-step guide on how to get started.
Specifically, we will cover what type of training data is required and how to prepare it with Apache Spark, followed by how to choose a correct neural net architecture, train, and deploy a deep learning model with TensorFlow on Apache Hadoop 3.1.
Finally, we will wrap-up with deep learning challenges and shortcomings, and offer short- and long-term recommendations to successfully train and deploy deep learning models within your organization to maximize return on investment.
Running Enterprise Workloads with an Open Source Hybrid Cloud Data Architecture
Cloud is turbocharging the Enterprise IT landscape with agility and flexibility. And now, discussions of cloud architecture dominate Enterprise IT. Cloud is enabling many ephemeral on-demand use cases which is a game-changing opportunity for analytic workloads. But all of this comes with the challenges of running enterprise workloads in the cloud securely and with ease.
With the convergence of cloud, IoT, and big data technologies, enterprises increasingly have their data spread across multiple data lakes on-prem and in cloud data lake stores in many geographies and across multiple public cloud vendor platforms, for example, due to regulatory and compliance mandates that limit cross-border data transfer. With the proliferation of data types and sources in this complex landscape, the process of discovery, provisioning, and running relevant workloads on this data to gather insights has become more complex. Additionally, gaining global visibility into the business context, usage, and trustworthiness of data requires a centralized view of all data and metadata, security controls, data access, and monitoring.
All of these challenges create a significant chasm between initial data capture and subsequent data insights generation to drive value creation. Therefore, enterprises now require a “global insight fabric” that can find a happy medium between adequate rules and policies of data governance while providing a trusted environment for users to collaborate and share data responsibly in order to create value.
In this talk, we will outline how Hortonworks DataPlane Service(DPS) can help customers build a global insight fabric that can span storing and analyzing data within data centers to implementing an open source hybrid architecture that takes advantage of cloud's elasticity and new use cases. We will get a personal view of the challenges faced in safely moving data from on-premises data centers into multiple public clouds, safeguarding it through replication, and then applying consistent security and governance policies across diverse environments to deliver trusted data and insights to the business. We will highlight how DataPlane Service can help enterprises with this hybrid architectural journey, and how open source architectures are enabling this transformation across enterprises.
Speaker: Alan Gates, Co-Founder, Hortonworks
Overcoming the AI hype — and what enterprises should really focus on
Deep learning for all its hype is brittle, non-generalizeable, and its learnings are not readily transferable from one application to another. Since we are unlikely to see anything close to artificial general intelligence in the next few decades., we should instead focus on how enterprises can capitalize on the state of the art in machine learning and re-implement successful algorithms and follow the data science lifecycles that generate highest ROI.
This talk will cover the current state of the art in AI, its limits vs. hype, and discuss concrete steps that enterprises can take to achieve desired ROI by re-implementing production-grade-ready machine learning algorithms, that have been hardened and demonstrated to work very well in specific, constrained domains.
By the end of this talk, attendees should have a better grasp on how to avoid costly and unnecessary investments into yet unproven technologies, be better equipped to navigate the complex space of AI, and understand where to best focus their resources to maximize ROI. ROBERT HRYNIEWICZ, Technical Evangelist, Hortonworks
The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi
Apache NiFi MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately.
Abstract: Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love.
Expected prior knowledge / intended audience: developers and data flow managers should have a passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi.
Takeaways: Attendees will learn about opportunities to bring their data flow and capture closer to the "edge" -- sources of data like IoT devices, vehicles, machinery, etc. They will understand the possibilities to prioritize, filter, secure, and manipulate this data earlier in the data lifecycle to enhance their data visibility and performance.
Speaker: Andy LoPresto, Sr. Member of Technical Staff, Hortonworks
Data in the Cloud Crash Course
Santhosh B Gowda presents on Cloudbreak, a tool for provisioning Hadoop clusters on cloud infrastructure. Cloudbreak allows for simplified cluster provisioning through prescriptive setups and automation. It supports declarative workload provisioning across multiple cloud providers with flexible topologies and security configuration options. Cloudbreak also enables features like auto-scaling, recipes to customize clusters, and shared services data lakes to provide common metadata and access management across ephemeral clusters. Demonstrations of launching HDP and HDF clusters from the Cloudbreak UI and CLI are also provided.
What is New in Apache Hive 3.0?
Apache Hive is a rapidly evolving project which continues to enjoy great adoption in the big data ecosystem. As Hive continues to grow its support for analytics, reporting, and interactive query, the community is hard at work in improving it along with many different dimensions and use cases. This talk will provide an overview of the latest and greatest features and optimizations which have landed in the project over the last year. Materialized views, the extension of ACID semantics to non-ORC data, and workload management are some noteworthy new features.
We will discuss optimizations which provide major performance gains, including significantly improved performance for ACID tables. The talk will also provide a glimpse of what is expected to come in the near future.
Speaker: Alan Gates, Co-Founder, Hortonworks
Running Enterprise Workloads with an open source Hybrid Cloud Data Architectu...
Cloud accelerates corporate IT landscapes with agility and flexibility. Today, discussion of cloud architecture dominates corporate IT. The cloud enables a number of temporary on-demand use cases that revolutionize analytical workload opportunities. But all of this involves the task of running corporate workloads safely and easily in the cloud.
With the convergence of cloud, IoT, and big data technology, enterprises are increasingly using multiple on-premises Data Lake and multiple Public on different geographies, for example due to regulations and compliance requirements restricting cross- It now distributes data to the cloud Data Lake store of the cloud vendor platform. Diffusion of data types and sources in this complex landscape makes the discovery process, provisioning, and getting insight by performing the appropriate workload on this data more complicated. In addition, to obtain business context, usage, and visibility of data trustworthiness worldwide, it is necessary to display all data and metadata, security management, data access, and monitoring in a centralized way .
All these problems create cracks during the creation of data insights to promote initial data capture and subsequent value creation. As a result, companies now look for compromises between appropriate rules and data control policies while providing a trusted environment that allows them to share data and partner with users responsibly to create value We need "Global Insight Fabric".
In this talk, how the Hortonworks DataPlane Service (DPS) analyzes the data in the data center to expand the storage, implement the open source hybrid architecture utilizing cloud flexibility and new use cases, global in Describes how site fabrics can help customers create. Securely migrate data from on-premises data centers to multiple public clouds, protect the data with replication, then apply consistent safety and governance policies to a wide variety of environments to ensure trustworthy data and inn We provide personal views on the challenges we face in providing the site to the business. I will explain how the DetaPlane service can be useful for traveling to this hybrid architecture and how the open source architecture enables the transformation of the entire enterprise.
The Implacable advance of the data
Data proliferation from 7+ billion humans and 20+ billion devices from every walk of life has been the focus in the last decade. With the velocity, variety and volume of data, every data organization’s goal shifted to protecting and monetizing data from rapidly growing network of IOT embedded objects and sensors.
One of the true and tried business continuity methodology of storing and retrieving vast amount of data has been through replication of Hadoop systems on hybrid clouds and in geographically distributed data centers. Replication is similar to Blockchain using autonomous smart contracts instantiated on the metadata and data so that the replicated data follows a single source of truth.
Replicas can be maintained across geographically distributed data centers giving greater risk tolerance capabilities to the businesses continuity plan for the data-sets. With intelligent predictive analytics based on usage patterns, dynamic tiering policies can be triggered on the data sets to provide true value-add to the data. The temperature of the data is used to move data between hot/warm/cold/archival storage based on configurable policies leading to greater reduction in total cost of ownership.
Users in 2018 and beyond demand absolute availability of data as and when they desire. The dynamic data access management is fundamental concept to satisfy the business continuity plan. Seamless enterprise-grade disaster recovery to support business continuity use case has significant challenges around replicating security and governance on data-sets. In this talk we will discuss how the above challenge can be addressed for supporting seamless replication and disaster recovery for Hadoop-scale data. NIRU ANISETI, Product Manager, Hortonworks
What's New in Apache Hive 3.0?
Apache Hive is a rapidly evolving project, many people are loved by the big data ecosystem. Hive continues to expand support for analytics, reporting, and bilateral queries, and the community is striving to improve support along with many other aspects and use cases. In this lecture, we introduce the latest and greatest features and optimization that appeared in this project last year. This includes benchmarks covering LLAP, Apache Druid's materialized views and integration, workload management, ACID improvements, using Hive in the cloud, and performance improvements. I will also tell you a little about what you can expect in the future.
Containers and Big Data
As containerization continues to gain momentum and become a de facto standard for application deployment, challenges around containerization of big data workloads are coming to light. Great strides have been made within the open source communities towards running big data workloads in containers, but much is left to be done.
Apache Hadoop YARN is the modern distributed operating system for big data applications. It has morphed the Hadoop compute layer into a common resource-management platform that can host a wide variety of applications. At its core, YARN has a very powerful scheduler which enforces global cluster level invariants and helps sites manage user and operator expectations of elastic sharing, resource usage limits, SLAs, and more. YARN recently increased its support for Docker containerization and added a YARN service framework supporting long-running services.
In this session we will explore the emerging patterns and challenges related to containers and big data workloads, including running applications such as Apache Spark, Apache HBase, and Kubernetes in containers on YARN.
Unlock Value from Big Data with Apache NiFi and Streaming CDC
The document discusses Apache NiFi and streaming change data capture (CDC) with Attunity Replicate. It provides an overview of NiFi's capabilities for dataflow management and visualization. It then demonstrates how Attunity Replicate can be used for real-time CDC to capture changes from source databases and deliver them to NiFi for further processing, enabling use cases across multiple industries. Examples of source systems include SAP, Oracle, SQL Server, and file data, with targets including Hadoop, data warehouses, and cloud data stores.
Containers and Big Data
As containerization continues to gain momentum and become a de facto standard for application deployment, challenges around containerization of big data workloads are coming to light. Great strides have been made within the open source communities towards running big data workloads in containers, but much is left to be done.
Apache Hadoop YARN is the modern distributed operating system for big data applications. It has morphed the Hadoop compute layer into a common resource-management platform that can host a wide variety of applications. At its core, YARN has a very powerful scheduler which enforces global cluster level invariants and helps sites manage user and operator expectations of elastic sharing, resource usage limits, SLAs, and more. YARN recently increased its support for Docker containerization and added a YARN service framework supporting long-running services.
In this session we will explore the emerging patterns and challenges related to containers and big data workloads, including running applications such as Apache Spark, Apache HBase, and Kubernetes in containers on YARN.
Speaker: Sanjay Radia, Chief Architect, Founder, Hortonworks
Intelligently Collecting Data at the Edge – Intro to Apache MiNiFi
The document appears to be a slide deck presentation about Apache NiFi and Apache MiNiFi. Some key points:
- Apache NiFi is a dataflow solution for automating the movement of data between systems. MiNiFi is a lighter-weight version designed for edge computing environments.
- The presentation discusses the challenges of IoT data collection and processing. MiNiFi is presented as a solution to run close to data sources on devices with limited resources.
- Examples are given showing MiNiFi running on a Raspberry Pi to collect credit card transaction data and sensor data from a Sense HAT, sending it to NiFi for further processing.
Keynote
1. Sanjay Radia discusses designing data systems for data everywhere, including on-premises and in the cloud.
2. He advocates for applying a consistent set of tools, managing data in one place, and sharing security and governance across systems.
3. Radia also notes the importance of leveraging innovation from open source communities to continually improve the systems.
Spark Summit EMEA - Arun Murthy's Keynote
Spark and Hadoop are perfectly together. Spark is a key tool in Hadoop's toolbox that provides elegant developer APIs and accelerates data science and machine learning. It can process streaming data in real-time for applications like web analytics and insurance claims processing. The future of Spark and Hadoop includes innovating the core technologies, providing seamless data access across data platforms, and further accelerating data science tools and libraries.
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Cloudbreak, a part of Hortonworks Data Platform (HDP), simplifies the provisioning and cluster management within any cloud environment to help your business toward its path to a hybrid cloud architecture.
https://hortonworks.com/webinar/getting-data-cloud-cloudbreak-live-demo/
Ozone and HDFS’s evolution
HDFS has several strengths: horizontally scale its IO bandwidth and scale its storage to petabytes of storage. Further, it provides very low latency metadata operations and scales to over 60K concurrent clients. Hadoop 3.0 recently added Erasure Coding. One of HDFS’s limitations is scaling a number of files and blocks in the system. We describe a radical change to Hadoop’s storage infrastructure with the upcoming Ozone technology. It allows Hadoop to scale to tens of billions of files and blocks and, in the future, to every larger number of smaller objects. Ozone fundamentally separates the namespace layer and the block layer allowing new namespace layers to be added in the future. Further, the use of RAFT protocol has allowed the storage layer to be self-consistent. We show how this technology helps a Hadoop user and also what it means for evolving HDFS in the future. We will also cover the technical details of Ozone.
Hortonworks Open Connected Data Platforms for IoT and Predictive Big Data Ana...
The energy industry is well known to be laggard adopters of new technology. However, industry challenges such as aging assets & workforce, increased regulatory scrutiny, renewable energy sources, depressed commodity prices, changing customer expectations, and growing data volumes are pushing companies to explore new technologies to help solve these problems. Learn how energy companies are leveraging Hortonworks Open and Connected Data Platforms to provide the predictive analysis and data insights to optimize performance for the energy industry.
Speaker
Kenneth Smith, General Manager, Energy, Hortonworks
Scalable and adaptable typosquatting detection in Apache Metron
Typosquatting is a form of cybersquatting (e.g. registering a domain that is similar to an existing domain) where a domain that is a common misspelling of another domain is registered and used for possibly malicious ends. It is an attack that has affected everyone from Google to the televangelist Jerry Falwell and has regulations against it encoded in US law. Even so, it remains extremely popular and particularly heinous and effective when used by advanced malicious actors who may rely on typosquatted domains to gull unwitting users to be more likely to click on a malicious URL in a spearphishing attack.
Detecting typosquatting attacks in realtime can be challenging as there are as many ways to typosquat as there are to make typos. Often, intrusion detection systems will generate the typosquatted domains and store them in a database for comparison. However, given the number of domains possible, this is a daunting task storage-wise. Furthermore, this approach can become out of date quickly.
We will talk about using sketching data structures in Metron to detect typosquatted domains scalably and adaptably. Furthermore, we will discuss how to ensure that the set of typosquatted domains is kept current with the domains actually seen in an organization's network. CASEY STELLA, Principal Software Engineer, Hortonworks and MICHAEL MIKLAVCIC, Staff Engineer, Hortonworks
What's hot (20)
Running Enterprise Workloads with an Open Source Hybrid Cloud Data Architecture
Running Enterprise Workloads with an Open Source Hybrid Cloud Data Architecture
Overcoming the AI hype — and what enterprises should really focus on
Overcoming the AI hype — and what enterprises should really focus on
The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi
The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi
Running Enterprise Workloads with an open source Hybrid Cloud Data Architectu...
Running Enterprise Workloads with an open source Hybrid Cloud Data Architectu...
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Intelligently Collecting Data at the Edge – Intro to Apache MiNiFi
Intelligently Collecting Data at the Edge – Intro to Apache MiNiFi
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Hortonworks Open Connected Data Platforms for IoT and Predictive Big Data Ana...
Hortonworks Open Connected Data Platforms for IoT and Predictive Big Data Ana...
Scalable and adaptable typosquatting detection in Apache Metron
Scalable and adaptable typosquatting detection in Apache Metron
Similar to Deep Learning 101
[Hortonworks] Future Of Data: Madrid - HDF & Data in motion
First meetup event Future Of Data event where we introduced Hortonworks DataFlow (HDF).
The slides describe what HDF is, and we presented a very simple demo about sentiment analysis of tweets using Apache OpenNLP as the NLP framework to do so.
Apache Metron in the Real World
Dave Russell - Presented at Big Data Vilnius 27-29th November 2018 https://www.bigdataconference.lt/
Johns Hopkins - Using Hadoop to Secure Access Log Events
In this webinar, we talk with experts from Johns Hopkins as they share techniques and lessons learned in real-world Apache Hadoop implementation.
https://hortonworks.com/webinar/johns-hopkins-using-hadoop-securely-access-log-events/
Introduction to Streaming Analytics Manager
This document introduces Streaming Analytics Manager (SAM), an open source project led by Hortonworks to simplify building streaming analytics applications. SAM aims to provide the same easy experience for streaming analytics as NiFi does for flow management applications. It allows users to create a streaming analytics application in 10 minutes and supports prescriptive, predictive, and descriptive analytics functions including routing, filtering, predictive modeling, and real-time dashboards. SAM applications are scalable through one-click deployment on distributed streaming platforms.
Apache Metron in the Real World
In this session we'll be looking at a number of different organisations who are on their big data cybersecurity journey with Apache Metron, we will take a look at the different usecases they are investigating, the data sources they used, the analytics they performed and in some cases the results they were able to find.
We'll also spend some time talking about the common themes in these projects, there are some common approaches to using Apache Metron as a phased project in a project, we'll review some of the common pitfalls and give some concrete suggestions about the things you should (and shouldn't) do when you're getting started.
Finally we'll try and tackle some of the key FAQ's that come up when people are first investigating the potential usage of Apache Metron in the real world based on over a year of interacting with customers and prospects as they look deeper into Apache Metron to see how it fits in to their cybersecurity portfolio.
Speaker
Dave Russell, Principal Solutions Engineer, Hortonworks
Hortonworks hdp i_mplementation_with_apache_ambari
Apache Ambari is an open-source platform for provisioning, managing, monitoring and securing Hadoop clusters. It provides an intuitive web UI and REST API. Ambari allows for centralized operational control of Hadoop clusters, regardless of their size.
Using Apache® NiFi to Empower Self-Organising Teams
The document discusses how Apache NiFi can empower self-organizing teams by allowing them to control their own data movement pipelines. It describes NiFi as a data-movement production line that enables teams to build and change pipelines quickly. This helps reduce time-to-production and feedback loops by moving responsibility for data integration to individual teams rather than having changes controlled in a centralized manner.
Enterprise IIoT Edge Processing with Apache NiFi
April 5, 2018 IoT Fusion 2018 Conference in Philadelphia, PA hosted by Chariot Solutions. This talk is about Apache NiFi, MiniFi, Python, Deep Learning, NVidia Jetson TX1, Raspberry Pi, Apache MXNet, TensorFlow and how to run things at the edge and process in your big data center. http://iotfusion.net/session/ https://github.com/tspannhw/IoTFusion2018Talk
そのデータフロー NiFiで楽にしてあげましょう
Future of Data: Apache NiFi 勉強会 〜データフローの自動化〜
http://futureofdata.connpass.com/event/35428/
で発表したスライドです。皆さん是非ダウンロードしてNiFi, HDFを使ってみてください!
Design a Dataflow in 7 minutes with Apache NiFi/HDF
The document describes how to create a live dataflow in Hortonworks DataFlow in 7 minutes. It involves dragging and dropping two processors onto a canvas - one for data intake and one for data output - configuring each processor, connecting them, and starting the flow. The dataflow can then be dynamically adjusted and tuned in real-time. Hortonworks DataFlow also allows viewing data provenance to trace the lineage and changes of data as it flows through the system.
Apache NiFi MiNiFi C++ and the tale of edgey stuff
In this presentation we discuss IoT and Edge use cases with a focus on how MiNiFi C++ has been able to tackle both while clearly living within the constraints dictated by these use cases.
Building a Smarter Home with Apache NiFi and Spark
This document discusses using Apache NiFi and Spark to build a smarter home. It describes using NiFi on a Raspberry Pi and EC2 to collect sensor data from smart home devices and transmit it to an HDP cluster for storage and analysis with Pig and Spark. It outlines the architecture as a hub-and-spoke model and shows the evolution of the NiFi flows from sequential blocking writes to attribute-based routing. Key discoveries include privacy issues, using MAC addresses to predict arrivals, and motion sensors being less useful alone. Challenges involved Oracle vs OpenJDK, backpressure, and site-to-site configuration.
Analyzing Hadoop Using Hadoop
Sheetal Dolas, a principal architect at Hortonworks, gave a presentation on analyzing Hadoop using Hadoop. She discussed the need for operational insights from Hadoop metrics to manage system reliability, uptime, and performance. While Hadoop generates a lot of metrics, understanding and analyzing them is challenging due to the large volume and lack of tools. Shem showed how datasets like MapReduce job histories, HDFS lsr reports, and audit logs can be analyzed using Hadoop to generate operational reports on topics like resource utilization, workload distributions, security checks, and more.
The Great Enterprise Hadoop Analyst Debate
A panel discussion was moderated by Shaun Connolly from Hortonworks on the state of enterprise Hadoop adoption. The panelists - Tony Baer from Ovum, Mark Madsen from Third Nature, and Noel Yuhanna from Forrester - discussed topics like Hadoop's drivers and inhibitors, how SQL enables access to Hadoop data, potential killer apps for Hadoop, ensuring Hadoop data lakes do not become swamps, the role of Internet of Things and cloud computing with Hadoop, and how data wrangling differs from traditional ETL processes.
Hadoop Summit Tokyo Apache NiFi Crash Course
This document provides an overview of Apache NiFi and data flow fundamentals. It begins with an introduction to Apache NiFi and outlines the agenda. It then discusses data flow and streaming fundamentals, including challenges in moving data effectively. The document introduces Apache NiFi's architecture and capabilities for addressing these challenges. It also previews a live demo of NiFi and discusses the NiFi community.
Joseph Witt
The document discusses the challenges faced by data-driven organizations in achieving closed-loop processes to enable real-time insights at scale. It outlines how such organizations want all available data in real-time to better inform decisions, but face issues such as limited bandwidth, evolving data formats, and ensuring compliance. The document proposes that a closed-loop "OODA" (Observe, Orient, Decide, Act) system architecture is needed to address these challenges, with distributed regional infrastructures feeding into a low-latency core infrastructure to facilitate real-time command and control and end-to-end data provenance.
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
For years, the healthcare industry has had problems of data scarcity and latency. Clearsense solved the problem by building an open-source Hortonworks Data Platform (HDP) solution while providing decades worth of clinical expertise. Clearsense is delivering smart, real-time streaming data, to its healthcare customers enabling mission-critical data to feed clinical decisions.
https://hortonworks.com/webinar/delivering-smart-real-time-streaming-data-healthcare-customers-clearsense/
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit
The document discusses bringing multi-tenancy to Apache Zeppelin through the use of Apache Livy. Livy is an open-source REST interface that allows interacting with Spark from anywhere and enables features like multi-user sessions and security. It improves on previous versions of interactive analysis in Zeppelin by allowing custom user sessions through Livy and improving security and isolation between users through mechanisms like SPNEGO and impersonation. The integration of Livy provides multi-tenancy, security, and isolation for interactive analysis in Zeppelin.Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi
This document provides an overview of a presentation about taking dataflow management to the edge with Apache NiFi and MiniFi. The presentation discusses the problem of moving data between systems with different formats, protocols, and security requirements. It introduces Apache NiFi as a solution for dataflow management and introduces Apache MiniFi for managing dataflows at the edge. The presentation includes a demo and time for Q&A.
Apache Spark in Cloud and Hybrid: Why Security and Governance Become More Imp...
An Increasing number of Apache Spark deployments are in Cloud and hybrid environments. This often means that Spark workloads are ephemeral but the data exists in a durable storage either in cloud and on-prem. The data also moves between cloud storage and on-prem. With this architecture in place, security and governance have become paramount to run Spark workloads across on-prem and cloud. In this keynote, we will walk through several issues and highlight a Spark workload running in an ephemeral cluster with security and governance across Cloud/On-Prem and how the same security and governance is shared with other workloads.
Similar to Deep Learning 101 (20)
[Hortonworks] Future Of Data: Madrid - HDF & Data in motion
[Hortonworks] Future Of Data: Madrid - HDF & Data in motion
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
Hortonworks hdp i_mplementation_with_apache_ambari
Hortonworks hdp i_mplementation_with_apache_ambari
Using Apache® NiFi to Empower Self-Organising Teams
Using Apache® NiFi to Empower Self-Organising Teams
Design a Dataflow in 7 minutes with Apache NiFi/HDF
Design a Dataflow in 7 minutes with Apache NiFi/HDF
Apache NiFi MiNiFi C++ and the tale of edgey stuff
Apache NiFi MiNiFi C++ and the tale of edgey stuff
Building a Smarter Home with Apache NiFi and Spark
Building a Smarter Home with Apache NiFi and Spark
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data Analysis
Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi
Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi
Apache Spark in Cloud and Hybrid: Why Security and Governance Become More Imp...
Apache Spark in Cloud and Hybrid: Why Security and Governance Become More Imp...
More from DataWorks Summit
Data Science Crash Course
Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL).
Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW).
Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models.
Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.
Floating on a RAFT: HBase Durability with Apache Ratis
In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort.
This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase.
Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs.
Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables.
Resources:
https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html
https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html
https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html
HBase Tales From the Trenches - Short stories about most common HBase operati...
Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.
Managing the Dewey Decimal System
OCLC has been using HBase since 2012 to enable single-search-box access to over a billion items from your library and the world’s library collection. This talk will provide an overview of how HBase is structured to provide this information and some of the challenges they have encountered to scale to support the world catalog and how they have overcome them.
Practical NoSQL: Accumulo's dirlist Example
Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL.
Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist).
In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.
HBase Global Indexing to support large-scale data ingestion at Uber
Danny Chen presented on Uber's use of HBase for global indexing to support large-scale data ingestion. Uber uses HBase to provide a global view of datasets ingested from Kafka and other data sources. To generate indexes, Spark jobs are used to transform data into HFiles, which are loaded into HBase tables. Given the large volumes of data, techniques like throttling HBase access and explicit serialization are used. The global indexing solution supports requirements for high throughput, strong consistency and horizontal scalability across Uber's data lake.
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions.
These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
This document discusses using Apache NiFi to build a high-speed cyber security data pipeline. It outlines the challenges of ingesting, transforming, and routing large volumes of security data from various sources to stakeholders like security operations centers, data scientists, and executives. It proposes using NiFi as a centralized data gateway to ingest data from multiple sources using a single entry point, transform the data according to destination needs, and reliably deliver the data while avoiding issues like network traffic and data duplication. The document provides an example NiFi flow and discusses metrics from processing over 20 billion events through 100+ production flows and 1000+ transformations.
Supporting Apache HBase : Troubleshooting and Supportability Improvements
This document discusses supporting Apache HBase and improving troubleshooting and supportability. It introduces two Cloudera employees who work on HBase support and provides an overview of typical troubleshooting scenarios for HBase like performance degradation, process crashes, and inconsistencies. The agenda covers using existing tools like logs and metrics to troubleshoot HBase performance issues with a general approach, and introduces htop as a real-time monitoring tool for HBase.
Security Framework for Multitenant Architecture
In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.
With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.
Extending Twitter's Data Platform to Google Cloud
Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.
Computer Vision: Coming to a Store Near You
Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as:
● Optimizing merchandising execution, in-stocks and sell-thru
● Enhancing operational efficiencies, enable real-time customer engagement
● Enhancing loss prevention capabilities, response time
● Creating frictionless experiences for shoppers
Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry.
We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey.
Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables.
We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance.
We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing.
Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems.
By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.
More from DataWorks Summit (20)
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
HBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at Uber
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything Engine
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Recently uploaded
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Project Management Semester Long Project - Acuity
Acuity is an innovative learning app designed to transform the way you engage with knowledge. Powered by AI technology, Acuity takes complex topics and distills them into concise, interactive summaries that are easy to read & understand. Whether you're exploring the depths of quantum mechanics or seeking insight into historical events, Acuity provides the key information you need without the burden of lengthy texts.
Webinar: Designing a schema for a Data Warehouse
Are you new to data warehouses (DWH)? Do you need to check whether your data warehouse follows the best practices for a good design? In both cases, this webinar is for you.
A data warehouse is a central relational database that contains all measurements about a business or an organisation. This data comes from a variety of heterogeneous data sources, which includes databases of any type that back the applications used by the company, data files exported by some applications, or APIs provided by internal or external services.
But designing a data warehouse correctly is a hard task, which requires gathering information about the business processes that need to be analysed in the first place. These processes must be translated into so-called star schemas, which means, denormalised databases where each table represents a dimension or facts.
We will discuss these topics:
- How to gather information about a business;
- Understanding dictionaries and how to identify business entities;
- Dimensions and facts;
- Setting a table granularity;
- Types of facts;
- Types of dimensions;
- Snowflakes and how to avoid them;
- Expanding existing dimensions and facts.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Fueling AI with Great Data with Airbyte Webinar
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Recently uploaded (20)
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Deep Learning 101
- 1. 1 © Hortonworks Inc. 2011–2018. All rights reserved DWS Melbourne, Australia 2019 Robert Hryniewicz Deep Learning 101
- 2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
- 3. 3 © Hortonworks Inc. 2011–2018. All rights reserved Deep Learning in the News
- 4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
- 5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
- 6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
- 7. 7 © Hortonworks Inc. 2011–2018. All rights reserved arxiv.org/pdf/1809.01726.pdf
- 8. 8 © Hortonworks Inc. 2011–2018. All rights reserved 8 source: youtube.com/watch?v=kSLJriaOumA
- 9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
- 10. 10 © Hortonworks Inc. 2011–2018. All rights reserved What is Deep Learning? 10
- 11. 11 © Hortonworks Inc. 2011–2018. All rights reserved • Convolutional Neural Nets (CNNs) • Recurrent Neural Nets (RNNs) • Long Short-Term Memory (LSTM) Popular Neural Net Architectures ß Images ß Text / Language (NLP) & Time Series
- 12. 12 © Hortonworks Inc. 2011–2018. All rights reserved Number Probability 0 0.03 1 0.01 2 0.04 3 0.08 4 0.05 5 0.08 6 0.07 7 0.02 8 0.54 9 0.08
- 13. 13 © Hortonworks Inc. 2011–2018. All rights reserved Natural Language Processing (NLP) & Time Series Recurrent Neural Nets (RNNs)
- 14. 14 © Hortonworks Inc. 2011–2018. All rights reserved
- 15. 15 © Hortonworks Inc. 2011–2018. All rights reserved J. Howard, Exponential Finance 2014
- 16. 16 © Hortonworks Inc. 2011–2018. All rights reserved Stock Prediction source: heartbeat.fritz.ai/using-a-keras-long-shortterm-memory-lstm-model-to-predict-stock-prices-a08c9f69aa74
- 17. 17 © Hortonworks Inc. 2011–2018. All rights reserved Tabular Data
- 18. 18 © Hortonworks Inc. 2011–2018. All rights reserved Predicting income over $50,000/year
- 19. 19 © Hortonworks Inc. 2011–2018. All rights reserved Recommender Systems
- 20. 20 © Hortonworks Inc. 2011–2018. All rights reserved User Movie/Show Rating Will Adama Battlestar Galactica 8.7 Robert Hryniewicz The Expanse 8.4 James Malkovich The Hitchhikers Guide to the Galaxy 6.8 Humphrey Smith Casablanca 8.5 Hugh Greene Love Actually 7.6
- 21. 21 © Hortonworks Inc. 2011–2018. All rights reserved GPU vs CPU Performance
- 22. 22 © Hortonworks Inc. 2011–2018. All rights reserved GPU vs CPU Inference Performance 22 azure.microsoft.com/en-us/blog/gpus-vs-cpus-for-deployment-of-deep-learning-models/ • 1 GPU is 392% faster than 35 pod 5 CPU cluster • 3 GPUs are 804% faster than 35 pod 5 CPU cluster at similar cost for MobileNetV2 GPUs è K80s CPU è D4 v2 (8 cores)
- 23. 23 © Hortonworks Inc. 2011–2018. All rights reserved Questions?
- 24. 24 © Hortonworks Inc. 2011–2018. All rights reserved Thanks! Robert Hryniewicz