Big Traffic, Big Trouble: Big Data - Tokyo

•

1 like•231 views

With the rise of IoT and complexity of applications, clouds, networks and infrastructure, it is becoming more difficult to protect data and infrastructure from attackers. When groups of bad actors collaborate, share information, provide unauthorized access, and do botnet as a service, attacks in terabit units also start easily. On the other hand, it is also difficult to find enough security analysts to deal with and defend against such attacks. Here is the emergence of community cooperation like Apache Metron and efforts to open source. Metron provides a comprehensive framework for applications, networks and security built on Apache Hadoop and open source streaming analysis (eg Apache Nifi, Apache Kafka) tools in scalable data management and processing stacks. Extensions such as profiling, machine learning, and visualization work and real-time streaming detection make SOC analysts more efficient, while intrinsic scalability of open source gives data scientists security insight from data laboratories So that it can be quickly incorporated into production. This section explains how real-world businesses and managed service providers use Apache Metron, identify and resolve security threats on a large scale, and explain methods and ideas for adapting the platform to your security architecture · I will demonstrate.

Technology

1 © Hortonworks Inc. 2011–2018. All rights reserved.
Big Traffic, Big Trouble: Big Data
How Cyber Security became about scale

2 © Hortonworks Inc. 2011–2018. All rights reserved.
Simon Elliston Ball
• Product Manager (Hortonworks)
• Data Scientist
• Elephant herder
• @sireb

3 © Hortonworks Inc. 2011–2018. All rights reserved.
Big Business
• $tn market
• Sharing networks
Sources: BT and KPMG Report, Taking the Offensive

4 © Hortonworks Inc. 2011–2018. All rights reserved.
Criminals as a Service
DDoS
attacks
attackers $5
per hour
defenders
~$40k
Stolen
Accounts
$1.6bn fine
(Facebook)
100m
Linkedin
accounts: 5

5 © Hortonworks Inc. 2011–2018. All rights reserved.
Too Much Data

6 © Hortonworks Inc. 2011–2018. All rights reserved.
Not Enough People
! 300k
unfilled job
openings " 3,000,000
professionals
needed
(IBM)
# 350,000
shortage by 2022
Sources: https://www.cyberseek.org/heatmap.html
https://www.business-standard.com/article/companies/india-needs-3-million-cyber-security-professionals-right-now-ibm-118051300153_1.html
https://www.ey.com/sg/en/newsroom/news-releases/news-ey-singapore-companies-confident-of-predicting-and-resisting-cyber-attacks
https://www.computerweekly.com/news/450420193/Europe-faces-shortage-of-350000-cyber-security-professionals-by-2022
https://researchcenter.paloaltonetworks.com/2017/05/cso-japan-aiming-close-cybersecurity-skills-gap-tokyo-2020/
$ 80% of
companies report
insufficient skills
%132,060
shortage and
growing

7 © Hortonworks Inc. 2011–2018. All rights reserved.
Scaling for Quantity Scaling for Speed
t = 1 t = 2 t = 3 t = n
Sketch
Period: 0<t<1
Sketch
Period: 1<t<2
Sketch
Period: 2<t<3
Sketch
Period: n-1<t<n
Behavior profiler

8 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache Metron: Open Source, Open Platform, Extensible
for consistent cyber security data
REAL-TIME PROCESSING CYBER SECURITY ENGINE
Cyber Security Stream Processing Pipeline
Telemetry Data
Sources
Telemetry Data
Collectors
Telemetry
Parsers Enrichment
Threat
Intel Profiler
Alert
Triage
Indexers
and
Writers
SecurityEndPoint
Devices
(Fireye,PaloAlto,
BlueCoat,etc.)
Machine
GeneratedLogs
(AD,App/Web
Server,firewall,
VPN,etc.)
IDS
(Suricata,Snort,
etc.)
NetworkData
PCAP,Netflow,Bro,
etc.)
ThreatIntelligence
Feeds
(Soltra,OpenTaxi
third-partyfeeds)
Performance
NetworkIngest
Probes
Real-Time
Enrich/Threat
IntelStreams
/Other…
DataVault
Real-TimeSearch
EvidentiaryStore
ThreatIntelligence
Platform
ModelasaService
CommunityModels
DataScience
Workbench
PCAPForensics
Modules
Data Services
& Integration
Layer
Telemetry
Ingest Buffer

9 © Hortonworks Inc. 2011–2018. All rights reserved.
IoT: the cloud
for bot nets

10 © Hortonworks Inc. 2011–2018. All rights reserved.
Scaling the people: humans + data science
Collect Visualize Hypothesis Test Automate

11 © Hortonworks Inc. 2011–2018. All rights reserved.
Botnet Demo Dashboard

12 © Hortonworks Inc. 2011–2018. All rights reserved.
Catching a Hacker
• Authentication data from a Red
Team exercise at Los Alamos
National Labs
• Beyond the visualization
• Automation
• Instant response
That one!

13 © Hortonworks Inc. 2011–2018. All rights reserved.
Visualization Demo

14 © Hortonworks Inc. 2011–2018. All rights reserved.
Turning Insight into Profiles
t = 1 t = 2 t = 3 t = n
Sketch
Period: 0<t<1
Sketch
Period: 1<t<2
Sketch
Period: 2<t<3
Sketch
Period: n-1<t<n

15 © Hortonworks Inc. 2011–2018. All rights reserved.
Alerts UI Demo

16 © Hortonworks Inc. 2011–2018. All rights reserved.
Packaged Insights: Community and Ecosystem
Data Pipeline
and Analytics
in Motion ü Visualization and Exploration
ü Reporting and Compliance
ü Turnkey Hardware
&
&
ü Real-time Security Data Enrichment,
Analytics, and Management

17 © Hortonworks Inc. 2011–2018. All rights reserved.
Thank you.

18 © Hortonworks Inc. 2011–2018. All rights reserved.
Any questions?
• Apache Metron: http://metron.apache.org
• Twitter: @sireb

What's hot

Cloud accelerates corporate IT landscapes with agility and flexibility. Today, discussion of cloud architecture dominates corporate IT. The cloud enables a number of temporary on-demand use cases that revolutionize analytical workload opportunities. But all of this involves the task of running corporate workloads safely and easily in the cloud. With the convergence of cloud, IoT, and big data technology, enterprises are increasingly using multiple on-premises Data Lake and multiple Public on different geographies, for example due to regulations and compliance requirements restricting cross- It now distributes data to the cloud Data Lake store of the cloud vendor platform. Diffusion of data types and sources in this complex landscape makes the discovery process, provisioning, and getting insight by performing the appropriate workload on this data more complicated. In addition, to obtain business context, usage, and visibility of data trustworthiness worldwide, it is necessary to display all data and metadata, security management, data access, and monitoring in a centralized way . All these problems create cracks during the creation of data insights to promote initial data capture and subsequent value creation. As a result, companies now look for compromises between appropriate rules and data control policies while providing a trusted environment that allows them to share data and partner with users responsibly to create value We need "Global Insight Fabric". In this talk, how the Hortonworks DataPlane Service (DPS) analyzes the data in the data center to expand the storage, implement the open source hybrid architecture utilizing cloud flexibility and new use cases, global in Describes how site fabrics can help customers create. Securely migrate data from on-premises data centers to multiple public clouds, protect the data with replication, then apply consistent safety and governance policies to a wide variety of environments to ensure trustworthy data and inn We provide personal views on the challenges we face in providing the site to the business. I will explain how the DetaPlane service can be useful for traveling to this hybrid architecture and how the open source architecture enables the transformation of the entire enterprise.

Running Enterprise Workloads with an open source Hybrid Cloud Data Architectu...

DataWorks Summit

Deep learning for all its hype is brittle, non-generalizeable, and its learnings are not readily transferable from one application to another. Since we are unlikely to see anything close to artificial general intelligence in the next few decades., we should instead focus on how enterprises can capitalize on the state of the art in machine learning and re-implement successful algorithms and follow the data science lifecycles that generate highest ROI. This talk will cover the current state of the art in AI, its limits vs. hype, and discuss concrete steps that enterprises can take to achieve desired ROI by re-implementing production-grade-ready machine learning algorithms, that have been hardened and demonstrated to work very well in specific, constrained domains. By the end of this talk, attendees should have a better grasp on how to avoid costly and unnecessary investments into yet unproven technologies, be better equipped to navigate the complex space of AI, and understand where to best focus their resources to maximize ROI. ROBERT HRYNIEWICZ, Technical Evangelist, Hortonworks

Overcoming the AI hype — and what enterprises should really focus on

DataWorks Summit

Introduction This workshop is a hands-on session to quickly deploy Hadoop and Streaming on AWS / Azure / Google Cloud. Cloudbreak simplifies the deployment of Hadoop in cloud environments. It enables the enterprise to quickly run big data workloads in the cloud while optimizing the use of cloud resources. Format A short introductory lecture about Cloudbreak. This is followed by a walk through and lab leveraging Hadoop and Streaming in the Cloud with Cloudbreak. Objective To provide a quick and short hands-on introduction to Hadoop on the cloud. Review key benefits of cluster deployment automation. This lab will use Cloudbreak to quickly and effortlessly stand up Hadoop and Streaming clusters in a cloud provider of your choice. The lab shows the use of Ambari blueprints that are your declarative definitions of your Hadoop or Streaming clusters. Steps to dynamically change these blueprints and use external databases and external authentication sources and in essence showing a way to provide Shared Authentication, Authorization and Audit across ephemeral and long-lasting clusters. However it is not limited to only custom blueprints, the lab also shows how Cloudbreak provides easy to use custom scripts called recipes that can be executed before or after Ambari start or after cluster installation.

Data in the Cloud Crash Course

DataWorks Summit

As containerization continues to gain momentum and become a de facto standard for application deployment, challenges around containerization of big data workloads are coming to light. Great strides have been made within the open source communities towards running big data workloads in containers, but much is left to be done. Apache Hadoop YARN is the modern distributed operating system for big data applications. It has morphed the Hadoop compute layer into a common resource-management platform that can host a wide variety of applications. At its core, YARN has a very powerful scheduler which enforces global cluster level invariants and helps sites manage user and operator expectations of elastic sharing, resource usage limits, SLAs, and more. YARN recently increased its support for Docker containerization and added a YARN service framework supporting long-running services. In this session we will explore the emerging patterns and challenges related to containers and big data workloads, including running applications such as Apache Spark, Apache HBase, and Kubernetes in containers on YARN.

Containers and Big Data

DataWorks Summit

Continuously improving factory operations is of critical importance to manufacturers. Consider the facts: the total cost of poor quality amounts to a staggering 20% of sales (American Society of Quality), and unplanned downtime costs plants approximately $50 billion per year (Deloitte). The most pressing questions are: which process variables effect quality and yield and which process variables predict equipment failure? Getting to those answers is providing forward thinking manufacturers a leg up over competitors. The speakers address the data management challenges facing today's manufacturers, including proprietary systems and siloed data sources, as well as an inability to make sensor-based data usable. Integrating enterprise data from ERP, MES, maintenance systems, and other sources with real-time operations data from sensors, PLCs, SCADA systems, and historians represents a major first step. But how to get started? What is the value of a data lake? How are AI/ML being applied to enable real time action? Join us for this educational session, which includes a view into a roadmap for an open source industrial IoT data management platform. Key Takeaways: • Understand key use cases commonly undertaken by manufacturing enterprises • Understand the value of using multivariate manufacturing data sources, as opposed to a single sensor on a piece of equipment • Understand advances in big data management and streaming analytics that are paving the way to next-generation factory performance Speakers Michael Ger, General Manager Manufacturing and Automotive, Hortonworks Wade Salazar, Solutions Engineer, Hortonworks

Achieving a 360-degree view of manufacturing via open source industrial data ...

DataWorks Summit

Companies who use Kafka today struggle with monitoring and managing Kafka clusters. Kafka is a key backbone of IoT streaming analytics applications. The challenge is understanding what is going on overall in the Kafka cluster including performance, issues and message flows. No open source tool caters to the needs of different users that work with Kafka: DevOps/developers, platform team, and security/governance teams. See how the new Hortonworks Streams Messaging Manager enables users to visualize their entire Kafka environment end-to-end and simplifies Kafka operations. In this session learn how SMM visualizes the intricate details of how Apache Kafka functions in real time while simultaneously surfacing every nuance of tuning, optimizing, and measuring input and output. SMM will assist users to quickly understand and operate Kafka while providing the much-needed transparency that sophisticated and experienced users need to avoid all the pitfalls of running a Kafka cluster. Speaker: Andrew Psaltis, Principal Solution Engineer, Hortonworks

Curing the Kafka Blindness – Streams Messaging Manager

DataWorks Summit

In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world so that nearly every streaming framework now supports higher level relational operations. On paper, combining Apache NiFi, Kafka, and Spark Streaming provides a compelling architecture option for building your next generation ETL data pipeline in near real time. What does this look like in an enterprise production environment to deploy and operationalized? The newer Spark Structured Streaming provides fast, scalable, fault-tolerant, end-to-end exactly-once stream processing with elegant code samples, but is that the whole story? We discuss the drivers and expected benefits of changing the existing event processing systems. In presenting the integrated solution, we will explore the key components of using NiFi, Kafka, and Spark, then share the good, the bad, and the ugly when trying to adopt these technologies into the enterprise. This session is targeted toward architects and other senior IT staff looking to continue their adoption of open source technology and modernize ingest/ETL processing. Attendees will take away lessons learned and experience in deploying these technologies to make their journey easier.

Using Spark Streaming and NiFi for the next generation of ETL in the enterprise

DataWorks Summit

Apache NiFi MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Abstract: Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love. Expected prior knowledge / intended audience: developers and data flow managers should have a passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi. Takeaways: Attendees will learn about opportunities to bring their data flow and capture closer to the "edge" -- sources of data like IoT devices, vehicles, machinery, etc. They will understand the possibilities to prioritize, filter, secure, and manipulate this data earlier in the data lifecycle to enhance their data visibility and performance. Speaker: Andy LoPresto, Sr. Member of Technical Staff, Hortonworks

The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi

DataWorks Summit

Apache Hive is a rapidly evolving project which continues to enjoy great adoption in the big data ecosystem. As Hive continues to grow its support for analytics, reporting, and interactive query, the community is hard at work in improving it along with many different dimensions and use cases. This talk will provide an overview of the latest and greatest features and optimizations which have landed in the project over the last year. Materialized views, the extension of ACID semantics to non-ORC data, and workload management are some noteworthy new features. We will discuss optimizations which provide major performance gains, including significantly improved performance for ACID tables. The talk will also provide a glimpse of what is expected to come in the near future. Speaker: Alan Gates, Co-Founder, Hortonworks

What is New in Apache Hive 3.0?

DataWorks Summit

Hortonworks DataFlow & Apache Nifi @Oslo Hadoop Big Data

Mats Johansson

Apache Spark 2.0 set the architectural foundations of structure in Spark, unified high-level APIs, structured streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community has continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2. Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. Likewise, Apache Spark 2.4 will have many JIRA issues resolved over 1100. In this talk, I want to skim and go through those notable features and changes.

What’s new in Apache Spark 2.3 and Spark 2.4

DataWorks Summit

HDFS has several strengths: horizontally scale its IO bandwidth and scale its storage to petabytes of storage. Further, it provides very low latency metadata operations and scales to over 60K concurrent clients. Hadoop 3.0 recently added Erasure Coding. One of HDFS’s limitations is scaling a number of files and blocks in the system. We describe a radical change to Hadoop’s storage infrastructure with the upcoming Ozone technology. It allows Hadoop to scale to tens of billions of files and blocks and, in the future, to every larger number of smaller objects. Ozone fundamentally separates the namespace layer and the block layer allowing new namespace layers to be added in the future. Further, the use of RAFT protocol has allowed the storage layer to be self-consistent. We show how this technology helps a Hadoop user and also what it means for evolving HDFS in the future. We will also cover the technical details of Ozone.

Ozone and HDFS’s evolution

DataWorks Summit

With the rise of IoT and the increasing complexity of applications, clouds, networks and infrastructure, the battle to keep your data and your infrastructure safe from attackers is getting harder. As groups of bad actors collaborate, sharing information and offering illegal access, and botnets as a service, terabits of attack can be launched cheaply. Meanwhile, it’s hard to find enough security analysts to catch and prevent these attacks. This is where community collaboration and open source efforts like Apache Metron come in. Metron presents a comprehensive framework for application and network, security built on Apache Hadoop and open source Streaming Analytics(ie Apache Nifi, Apache Kafka) tool’s highly scalable data management and processing stacks. Advanced features like profiling, machine learning, and visualization work with real-time streaming detection to make your SOC analysts more efficient, while the intrinsic extensibility of open source helps your data scientists get security insights out of the lab and into production fast. We will discuss and demonstrate how some real-world businesses and managed service providers are using Apache Metron to identify and solve security threats at scale, and some approaches and ideas for how the platform can fit into your security architecture. Speaker: Laurence Da Luz, Senior Solutions Architect, Hortonworks

Solving Cybersecurity at Scale

DataWorks Summit

Typosquatting is a form of cybersquatting (e.g. registering a domain that is similar to an existing domain) where a domain that is a common misspelling of another domain is registered and used for possibly malicious ends. It is an attack that has affected everyone from Google to the televangelist Jerry Falwell and has regulations against it encoded in US law. Even so, it remains extremely popular and particularly heinous and effective when used by advanced malicious actors who may rely on typosquatted domains to gull unwitting users to be more likely to click on a malicious URL in a spearphishing attack. Detecting typosquatting attacks in realtime can be challenging as there are as many ways to typosquat as there are to make typos. Often, intrusion detection systems will generate the typosquatted domains and store them in a database for comparison. However, given the number of domains possible, this is a daunting task storage-wise. Furthermore, this approach can become out of date quickly. We will talk about using sketching data structures in Metron to detect typosquatted domains scalably and adaptably. Furthermore, we will discuss how to ensure that the set of typosquatted domains is kept current with the domains actually seen in an organization's network. CASEY STELLA, Principal Software Engineer, Hortonworks and MICHAEL MIKLAVCIC, Staff Engineer, Hortonworks

Scalable and adaptable typosquatting detection in Apache Metron

DataWorks Summit

Spark Summit EMEA - Arun Murthy's Keynote

Hortonworks

Meetup oslo hortonworks HDP

Alexander Bakos Leirvåg

As enterprises around the world bring more of their sensitive data into Hadoop data lakes, balancing the need for democratization of access to data without sacrificing strong security principles becomes paramount. In this webinar, Srikanth Venkat, director of product management for security & governance will demonstrate two new data protection capabilities in Apache Ranger – dynamic column masking and row level filtering of data stored in Apache Hive. These features have been introduced as part of HDP 2.5 platform release.

Dynamic Column Masking and Row-Level Filtering in HDP

Hortonworks

Containers and Big Data

DataWorks Summit

Trading in financial markets today is dominated by automated trading across most asset classes, but current programs are implemented using structured programming approaches which are static and represent a snapshot of the authors ideas, biases, and shortcomings at the time of implementation. Building automated trading bots that can learn from experience and can adapt to changing market conditions is changing the landscape and will deeply change trading as we know it. In this presentation we will explore the history of automated trading, the environment in which these programs operate, current state, and challenges of the current approach. We will explore how a machine learning approach can be applied to automated trading and the forces driving this transformation. Analysis, which used to take hours or days, can now be done in seconds, back-testing over a larger length of time with fuller data now possible, and more data sources are available that can be used to build richer more accurate models. Speaker Diego Baez, GM Financial Services, Hortonworks

Machine Learning trading bots

DataWorks Summit

Innovative mobile operators need to mine the vast troves of unstructured data now available to them to help develop compelling customer experiences and uncover new revenue opportunities. In this webinar, you’ll learn how HDB’s in-database analytics enable advanced use cases in network operations, customer care, and marketing for better customer experience. Join us, and get started on your advanced analytics journey today!

Pivotal - Advanced Analytics for Telecommunications

Hortonworks

What's hot (20)

Running Enterprise Workloads with an open source Hybrid Cloud Data Architectu...

Overcoming the AI hype — and what enterprises should really focus on

Data in the Cloud Crash Course

Containers and Big Data

Achieving a 360-degree view of manufacturing via open source industrial data ...

Curing the Kafka Blindness – Streams Messaging Manager

Using Spark Streaming and NiFi for the next generation of ETL in the enterprise

The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi

What is New in Apache Hive 3.0?

Hortonworks DataFlow & Apache Nifi @Oslo Hadoop Big Data

What’s new in Apache Spark 2.3 and Spark 2.4

Ozone and HDFS’s evolution

Solving Cybersecurity at Scale

Scalable and adaptable typosquatting detection in Apache Metron

Spark Summit EMEA - Arun Murthy's Keynote

Meetup oslo hortonworks HDP

Dynamic Column Masking and Row-Level Filtering in HDP

Containers and Big Data

Machine Learning trading bots

Pivotal - Advanced Analytics for Telecommunications

Similar to Big Traffic, Big Trouble: Big Data - Tokyo

Apache Metron in the Real World

Dave Russell

Cybersecurity today is a big data problem. There’s a ton of data landing on you faster than you can load, let alone search it. In order to make sense of it, we need to act on data-in-motion, use both machine learning, and the most advanced pattern recognition system on the planet: your SOC analysts. Advanced visualization makes your analysts more efficient, helps them find the hidden gems, or bombs in masses of logs and packets. https://hortonworks.com/webinar/catch-hacker-real-time-live-visuals-bots-bad-guys/

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

Hortonworks

Hortonworks - IBM Cognitive - The Future of Data Science

Thiago Santiago

The electric grid has evolved from linear generation and delivery to a complex mix of renewables, prosumer-generated electricity, and electric vehicles (EVs). Smart meters are generating loads of data. As a result, traditional forecasting models and technologies can no longer adequately predict supply and demand. Extreme weather, an aging infrastructure, and the burgeoning worldwide population are also contributing to increased outage frequency. In oil and gas, commodity pricing pressures, resulting workforce reductions, and the need to reduce failures, automate workflows, and increase operational efficiencies are driving operators to shift analytics initiatives to advanced data-driven applications to complement physics-based tools. While sensored equipment and legacy surveillance applications are generating massive amounts of data, just 2% is understood and being leveraged. Operationalizing it along with external datasets enables a shift from time-based to condition-based maintenance, better forecasting and dramatic reductions in unplanned downtime. The session includes plenty of real-world anecdotes. For example, how an electric power holding company reduced the time it took to investigate energy theft from six months to less than one hour, producing theft leads in minutes and an expected multi-million dollar ROI. How a global offshore contract drilling services provider implemented an open source IIoT solution across its fleet of assets in less than a year, enabling remote monitoring, predictive analytics and maintenance. Key takeaways: • How are new processes for data collection, storage and democratization making it accessible and usable at scale? • Beyond time series data, what other data types are important to assess? • What advantage are open source technologies providing to enterprises deploying IIoT? • Why is collaboration important across industrial verticals to increase IIoT open source adoption? Speaker Kenneth Smith, General Manager, Energy, Hortonworks

IIoT + Predictive Analytics: Solving for Disruption in Oil & Gas and Energy &...

DataWorks Summit

Forrester forecasts* that direct spending on the Internet of Things (IoT) will exceed $400 Billion by 2023. From manufacturing and utilities, to oil & gas and transportation, IoT improves visibility, reduces downtime, and creates opportunities for entirely new business models. But successful IoT implementations require far more than simply connecting sensors to a network. The data generated by these devices must be collected, aggregated, cleaned, processed, interpreted, understood, and used. Data-driven decisions and actions must be taken, without which an IoT implementation is bound to fail. https://hortonworks.com/webinar/iot-predictions-2019-beyond-data-heart-iot-strategy/

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

Hortonworks

Real-time Analytics in Financial

Yifeng Jiang

The energy industry is well known to be laggard adopters of new technology. However, industry challenges such as aging assets & workforce, increased regulatory scrutiny, renewable energy sources, depressed commodity prices, changing customer expectations, and growing data volumes are pushing companies to explore new technologies to help solve these problems. Learn how energy companies are leveraging Hortonworks Open and Connected Data Platforms to provide the predictive analysis and data insights to optimize performance for the energy industry. Speaker Kenneth Smith, General Manager, Energy, Hortonworks

Hortonworks Open Connected Data Platforms for IoT and Predictive Big Data Ana...

DataWorks Summit

Real-time Analytics in Financial: Use Case, Architecture and Challenges

DataWorks Summit/Hadoop Summit

Make Streaming IoT Analytics Work for You

Hortonworks

Johns Hopkins - Using Hadoop to Secure Access Log Events

Hortonworks

Data Science Crash Course

DataWorks Summit/Hadoop Summit

Watch Dinesh's keynote presentation from Fast Data Strategy Virtual Summit here: https://goo.gl/3Pa8np Leaders are re-inventing their data management strategies through the effective use of IoT, Big Data, and data science to boost their customer experience. Yet, they struggle to modernize their data architecture due to lack of global data management processes and technologies. Attend this session to hear from the Big Data pioneer, Hortonworks: • Why big data and data virtualization should be core technology components of your digital transformation. • How to manage, govern, and secure your global data footprint across a hybrid multi-cloud landscape. • Learn about key global data management strategies and use cases that drive leading digital enterprises.

Reinvent Your Data Management Strategy for Successful Digital Transformation

Denodo

Enterprise IIoT Edge Processing with Apache NiFi

Timothy Spann

Hortonworks - How Hadoop makes the successful Retailer.

Mats Johansson

In this talk, Edgar Orendain walks through a modern real-time streaming application serving as a reference framework for developing a big data pipeline, complete with a broad range of use cases and powerful reusable core components. Modern applications can ingest data and leverage analytics in real-time. These analytics are based on machine learning models typically built using historical big data. This reference application provides examples of connecting data-in-motion analytics to your application based on Big Data. We review code, best practices and considerations involved when integrating different components into a complete data platform. From IoT sensor data collection, to flow management, real-time stream processing and analytics, through to machine learning and prediction, this reference project aims to help developers seed their own open source solutions – fast.

Building a modern end-to-end open source Big Data reference application

DataWorks Summit

Watch live session on-demand here: https://goo.gl/QccnZQ Big data optimization is a daunting task. The optimizer is one of the most complex parts of any big data engine; it must ensure that the execution engine is performing at its best, so understanding how it works is crucial. Watch this Denodo DataFest 2017 session to discover: • Challenges in working with Big Data Workloads. • What is under the hood of the data virtualization optimization engine. • When, how, and why to use different optimization options.

Denodo DataFest 2017: Edge Computing: Collecting vs. Connecting to Streaming ...

Denodo

HDF 3.2 - What's New

Hortonworks

Apache Spark Crash Course

DataWorks Summit/Hadoop Summit

In this session we'll be looking at a number of different organisations who are on their big data cybersecurity journey with Apache Metron, we will take a look at the different usecases they are investigating, the data sources they used, the analytics they performed and in some cases the results they were able to find. We'll also spend some time talking about the common themes in these projects, there are some common approaches to using Apache Metron as a phased project in a project, we'll review some of the common pitfalls and give some concrete suggestions about the things you should (and shouldn't) do when you're getting started. Finally we'll try and tackle some of the key FAQ's that come up when people are first investigating the potential usage of Apache Metron in the real world based on over a year of interacting with customers and prospects as they look deeper into Apache Metron to see how it fits in to their cybersecurity portfolio. Speaker Dave Russell, Principal Solutions Engineer, Hortonworks

Apache Metron in the Real World

DataWorks Summit

Cloud is turbocharging the Enterprise IT landscape with agility and flexibility. And now, discussions of cloud architecture dominate Enterprise IT. Cloud is enabling many ephemeral on-demand use cases which is a game-changing opportunity for analytic workloads. But all of this comes with the challenges of running enterprise workloads in the cloud securely and with ease. With the convergence of cloud, IoT, and big data technologies, enterprises increasingly have their data spread across multiple data lakes on-prem and in cloud data lake stores in many geographies and across multiple public cloud vendor platforms, for example, due to regulatory and compliance mandates that limit cross-border data transfer. With the proliferation of data types and sources in this complex landscape, the process of discovery, provisioning, and running relevant workloads on this data to gather insights has become more complex. Additionally, gaining global visibility into the business context, usage, and trustworthiness of data requires a centralized view of all data and metadata, security controls, data access, and monitoring. All of these challenges create a significant chasm between initial data capture and subsequent data insights generation to drive value creation. Therefore, enterprises now require a “global insight fabric” that can find a happy medium between adequate rules and policies of data governance while providing a trusted environment for users to collaborate and share data responsibly in order to create value. In this talk, we will outline how Hortonworks DataPlane Service(DPS) can help customers build a global insight fabric that can span storing and analyzing data within data centers to implementing an open source hybrid architecture that takes advantage of cloud's elasticity and new use cases. We will get a personal view of the challenges faced in safely moving data from on-premises data centers into multiple public clouds, safeguarding it through replication, and then applying consistent security and governance policies across diverse environments to deliver trusted data and insights to the business. We will highlight how DataPlane Service can help enterprises with this hybrid architectural journey, and how open source architectures are enabling this transformation across enterprises. Speaker: Alan Gates, Co-Founder, Hortonworks

Running Enterprise Workloads with an Open Source Hybrid Cloud Data Architecture

DataWorks Summit

Similar to Big Traffic, Big Trouble: Big Data - Tokyo (20)

Apache Metron in the Real World

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

Hortonworks - IBM Cognitive - The Future of Data Science

IIoT + Predictive Analytics: Solving for Disruption in Oil & Gas and Energy &...

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

Real-time Analytics in Financial

Hortonworks Open Connected Data Platforms for IoT and Predictive Big Data Ana...

Real-time Analytics in Financial: Use Case, Architecture and Challenges

Make Streaming IoT Analytics Work for You

Johns Hopkins - Using Hadoop to Secure Access Log Events

Data Science Crash Course

Reinvent Your Data Management Strategy for Successful Digital Transformation

Enterprise IIoT Edge Processing with Apache NiFi

Hortonworks - How Hadoop makes the successful Retailer.

Building a modern end-to-end open source Big Data reference application

Denodo DataFest 2017: Edge Computing: Collecting vs. Connecting to Streaming ...

HDF 3.2 - What's New

Apache Spark Crash Course

Apache Metron in the Real World

Running Enterprise Workloads with an Open Source Hybrid Cloud Data Architecture

More from DataWorks Summit

Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL). Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW). Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models. Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.

Data Science Crash Course

DataWorks Summit

In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort. This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.

Floating on a RAFT: HBase Durability with Apache Ratis

DataWorks Summit

Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase. Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs. Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables. Resources: https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

DataWorks Summit

Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.

HBase Tales From the Trenches - Short stories about most common HBase operati...

DataWorks Summit

LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

DataWorks Summit

Managing the Dewey Decimal System

DataWorks Summit

Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL. Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist). In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.

Practical NoSQL: Accumulo's dirlist Example

DataWorks Summit

Data serves as the platform for decision-making at Uber. To facilitate data driven decisions, many datasets at Uber are ingested in a Hadoop Data Lake and exposed to querying via Hive. Analytical queries joining various datasets are run to better understand business data at Uber. Data ingestion, at its most basic form, is about organizing data to balance efficient reading and writing of newer data. Data organization for efficient reading involves factoring in query patterns to partition data to ensure read amplification is low. Data organization for efficient writing involves factoring the nature of input data - whether it is append only or updatable. At Uber we ingest terabytes of many critical tables such as trips that are updatable. These tables are fundamental part of Uber's data-driven solutions, and act as the source-of-truth for all the analytical use-cases across the entire company. Datasets such as trips constantly receive updates to the data apart from inserts. To ingest such datasets we need a critical component that is responsible for bookkeeping information of the data layout, and annotates each incoming change with the location in HDFS where this data should be written. This component is called as Global Indexing. Without this component, all records get treated as inserts and get re-written to HDFS instead of being updated. This leads to duplication of data, breaking data correctness and user queries. This component is key to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. This component will need to have strong consistency and provide large throughputs for index writes and reads. At Uber, we have chosen HBase to be the backing store for the Global Indexing component and is a critical component in allowing us to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. In this talk, we will discuss data@Uber and expound more on why we built the global index using Apache Hbase and how this helps to scale out our cluster usage. We’ll give details on why we chose HBase over other storage systems, how and why we came up with a creative solution to automatically load Hfiles directly to the backend circumventing the normal write path when bootstrapping our ingestion tables to avoid QPS constraints, as well as other learnings we had bringing this system up in production at the scale of data that Uber encounters daily.

HBase Global Indexing to support large-scale data ingestion at Uber

DataWorks Summit

Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions. These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

DataWorks Summit

Cybersecurity requires an organization to collect data, analyze it, and alert on cyber anomalies in near real-time. This is a challenging endeavor when considering the variety of data sources which need to be collected and analyzed. Everything from application logs, network events, authentications systems, IOT devices, business events, cloud service logs, and more need to be taken into consideration. In addition, multiple data formats need to be transformed and conformed to be understood by both humans and ML/AI algorithms. To solve this problem, the Aetna Global Security team developed the Unified Data Platform based on Apache NiFi, which allows them to remain agile and adapt to new security threats and the onboarding of new technologies in the Aetna environment. The platform currently has over 60 different data flows with 95% doing real-time ETL and handles over 20 billion events per day. In this session learn from Aetna’s experience building an edge to AI high-speed data pipeline with Apache NiFi.

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

DataWorks Summit

Supporting Apache HBase : Troubleshooting and Supportability Improvements

DataWorks Summit

In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”

Security Framework for Multitenant Architecture

DataWorks Summit

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.

Presto: Optimizing Performance of SQL-on-Anything Engine

DataWorks Summit

Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

DataWorks Summit

Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.

Extending Twitter's Data Platform to Google Cloud

DataWorks Summit

At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

DataWorks Summit

Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

DataWorks Summit

Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

DataWorks Summit

Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as: ● Optimizing merchandising execution, in-stocks and sell-thru ● Enhancing operational efficiencies, enable real-time customer engagement ● Enhancing loss prevention capabilities, response time ● Creating frictionless experiences for shoppers Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry. We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey. Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables. We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance. We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing. Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems. By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.

Computer Vision: Coming to a Store Near You

DataWorks Summit

Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

DataWorks Summit

More from DataWorks Summit (20)

Data Science Crash Course

Floating on a RAFT: HBase Durability with Apache Ratis

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

HBase Tales From the Trenches - Short stories about most common HBase operati...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Managing the Dewey Decimal System

Practical NoSQL: Accumulo's dirlist Example

HBase Global Indexing to support large-scale data ingestion at Uber

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Security Framework for Multitenant Architecture

Presto: Optimizing Performance of SQL-on-Anything Engine

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Extending Twitter's Data Platform to Google Cloud

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Computer Vision: Coming to a Store Near You

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Recently uploaded

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

DBX First Quarter 2024 Investor Presentation

Dropbox

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

ICT role in 21st century education and its challenges

rafiqahmad00786416

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

A Beginners Guide to Building a RAG App Using Open Source Milvus

Zilliz

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Recently uploaded (20)

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

DBX First Quarter 2024 Investor Presentation

Strategies for Landing an Oracle DBA Job as a Fresher

Automating Google Workspace (GWS) & more with Apps Script

A Year of the Servo Reboot: Where Are We Now?

presentation ICT roal in 21st century education

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Exploring the Future Potential of AI-Enabled Smartphone Processors

ICT role in 21st century education and its challenges

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Boost Fertility New Invention Ups Success Rates.pdf

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

How to Troubleshoot Apps for the Modern Connected Worker

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

A Beginners Guide to Building a RAG App Using Open Source Milvus

Ransomware_Q4_2023. The report. [EN].pdf

Data Cloud, More than a CDP by Matt Robison

Big Traffic, Big Trouble: Big Data - Tokyo

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved. Not Enough People ! 300k unfilled job openings " 3,000,000 professionals needed (IBM) # 350,000 shortage by 2022 Sources: https://www.cyberseek.org/heatmap.html https://www.business-standard.com/article/companies/india-needs-3-million-cyber-security-professionals-right-now-ibm-118051300153_1.html https://www.ey.com/sg/en/newsroom/news-releases/news-ey-singapore-companies-confident-of-predicting-and-resisting-cyber-attacks https://www.computerweekly.com/news/450420193/Europe-faces-shortage-of-350000-cyber-security-professionals-by-2022 https://researchcenter.paloaltonetworks.com/2017/05/cso-japan-aiming-close-cybersecurity-skills-gap-tokyo-2020/ $ 80% of companies report insufficient skills %132,060 shortage and growing

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved. Scaling for Quantity Scaling for Speed t = 1 t = 2 t = 3 t = n Sketch Period: 0<t<1 Sketch Period: 1<t<2 Sketch Period: 2<t<3 Sketch Period: n-1<t<n Behavior profiler

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved. Apache Metron: Open Source, Open Platform, Extensible for consistent cyber security data REAL-TIME PROCESSING CYBER SECURITY ENGINE Cyber Security Stream Processing Pipeline Telemetry Data Sources Telemetry Data Collectors Telemetry Parsers Enrichment Threat Intel Profiler Alert Triage Indexers and Writers SecurityEndPoint Devices (Fireye,PaloAlto, BlueCoat,etc.) Machine GeneratedLogs (AD,App/Web Server,firewall, VPN,etc.) IDS (Suricata,Snort, etc.) NetworkData PCAP,Netflow,Bro, etc.) ThreatIntelligence Feeds (Soltra,OpenTaxi third-partyfeeds) Performance NetworkIngest Probes Real-Time Enrich/Threat IntelStreams /Other… DataVault Real-TimeSearch EvidentiaryStore ThreatIntelligence Platform ModelasaService CommunityModels DataScience Workbench PCAPForensics Modules Data Services & Integration Layer Telemetry Ingest Buffer

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved. Catching a Hacker • Authentication data from a Red Team exercise at Los Alamos National Labs • Beyond the visualization • Automation • Instant response That one!

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved. Packaged Insights: Community and Ecosystem Data Pipeline and Analytics in Motion ü Visualization and Exploration ü Reporting and Compliance ü Turnkey Hardware & & ü Real-time Security Data Enrichment, Analytics, and Management

Big Traffic, Big Trouble: Big Data - Tokyo

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Big Traffic, Big Trouble: Big Data - Tokyo

Similar to Big Traffic, Big Trouble: Big Data - Tokyo (20)

More from DataWorks Summit

More from DataWorks Summit (20)

Recently uploaded

Recently uploaded (20)

Big Traffic, Big Trouble: Big Data - Tokyo