Beyond Big Data: Data Science and AI
•
4 likes•1,188 views
This document discusses how data science and AI are fueling new business models driven by data. It summarizes that (1) connected devices, customers, and sensors are generating massive amounts of data across manufacturing, distribution, marketing, sales, and service; (2) technologies like cloud computing, streaming data, IoT, and machine learning are enabling new ways to harness this data; and (3) a modern data architecture is needed to encompass all data sources, enable analytics and machine learning, and power actionable intelligence across edge, cloud, and on-premises environments.
Report
Share
Report
Share
Download to read offline
Recommended
Pouring the Foundation: Data Management in the Energy Industry
At CenterPoint Energy, both structured and unstructured data are continuing to grow at a rapid pace. This growth presents many opportunities to deliver business value and many challenges to control costs. To maximize the value of this data while controlling costs, CenterPoint Energy created a data lake using SAP HANA and Hadoop. During this presentation, CenterPoint will discuss their journey of moving smart meter data to Hadoop, how Hadoop is allowing CenterPoint to derive value from big data and their future use case road map.
Use Cases from Batch to Streaming, MapReduce to Spark, Mainframe to Cloud: To...
This document discusses how Syncsort helps companies access and integrate data from various sources to power analytics. It provides examples of how Syncsort has helped companies in insurance, media, and hotels easily onboard and integrate both historical and streaming data from multiple sources like mainframes, databases, and IoT devices. This allows for faster insights, increased productivity, cost savings, and helps future proof applications.
Data Process Systems, connecting everything
This document summarizes Patrick de Vries' presentation on connecting everything at the Hadoop Summit 2016. The presentation discusses KPN's use of Hadoop to manage increasing data and network capacity needs. It outlines KPN's data flow process from source systems to Hadoop for processing and generating reports. The presentation also covers lessons learned in implementing Hadoop including having strong executive support, addressing cultural challenges around data ownership, and leveraging existing investments. Finally, it promotes joining a new TELCO Hadoop community for telecommunications providers to share use cases and lessons.
Rob Bearden Keynote Hadoop Summit San Jose
Rob Bearden argues that data will transform everything as data has become a valuable corporate asset. Enterprises must change their business models to take advantage of all available data from billions of connected devices and consumer access to data or become irrelevant. Traditional constraints of using only structured data in silos and centralized data centers limit insights and the ability to gain actionable intelligence. However, open connected data platforms can break down these limitations by integrating all types of data from multiple sources. This will allow enterprises to gain deep historical analysis, real-time stream analytics at the edge, and machine learning capabilities to transform their industries.
Hybrid Cloud Strategy for Big Data and Analytics
Digital transformations require a new hybrid cloud—one that’s open by design, and frees clients to choose and change environments, data and services as needed. This approach allows cloud apps and services to be rapidly composed using the best relevant data and insights available, while maintaining clear visibility, control and security—everywhere. How do you decide where to put data on a hybrid cloud and how to use it? What’s the best hybrid cloud strategy in terms of data and workload? How should you leverage a 50/50 rule or a 80/20 rule and user interaction to evaluate which data/workload to move to the cloud and which data/workload to keep on-premise? Hybrid cloud provides an open platform for innovation, including cognitive computing. Organizations are looking for taking shadow IT out of the shadows by providing a self-service way to the information and a hybrid cloud strategy is allowing that. Also, how to use hybrid cloud for better manage data sovereignty & compliance?
Spark meets Smart Meters
This document discusses how Spark and big data analytics can power Australia's energy transformation as the country transitions to smart meters and renewable energy sources like solar and batteries. Key points:
- Smart meters are generating much more granular energy usage data that needs to be stored, processed and analyzed at large scale.
- Hadoop and Spark are well-suited for handling the exponentially growing volumes of time series energy data from smart meters.
- Advanced analytics on smart meter data can provide insights into customer energy usage patterns and behaviors to improve demand forecasting, marketing and grid management.
- Integrating distributed energy resources like rooftop solar and batteries with smart appliances and electric vehicles will require real-time coordination and optimization across the
It Takes a Village: Organizational Alignment to Deliver Big Data Value in Hea...
The business and technology teams within a health insurer must align the company’s central data platform with its data strategy. That requires substantial organizational alignment. Hear the firsthand perspective from Health Care Service Corporation (HCSC), the largest customer-owned health insurance company in the United States. The speaker will cover how they integrated membership information, regulatory compliance, and the general ledger, to improve overall healthcare management. At HCSC, the strong alignment between executive leadership, business portfolio direction, architectural strategy, technology delivery, and program management have helped create leading-edge capabilities which help the company respond nimbly to a quickly evolving healthcare industry.
How Market Intelligence From Hadoop on Azure Shows Trucking Companies a Clear...
TMW Systems (a Trimble Company) has been in the business of long-haul trucking, logistics operations and fleet management for more than thirty years, but we wanted more data, so we turned to our customer community. Now, we turn that data into market intelligence, which we then provide back to our customers. To do this, we invested heavily in Hortonworks Data Platform running on Microsoft Azure in the cloud. In our talk, we’ll share our strategy for capturing operational, maintenance, financial and mobile communications information and how we provide that back to our customer base. Our approach enables advanced analytics by leveraging Big Data technologies to find new relationships in data that may have been previously overlooked. Survey responses capture business performance metrics, strategy and emerging trends from 150 businesses, representing more than 31 billion dollars in freight movement. Learn how we combine that survey data with other sources like machine and sensor data to help guide our customers to profitability.
Recommended
Pouring the Foundation: Data Management in the Energy Industry
At CenterPoint Energy, both structured and unstructured data are continuing to grow at a rapid pace. This growth presents many opportunities to deliver business value and many challenges to control costs. To maximize the value of this data while controlling costs, CenterPoint Energy created a data lake using SAP HANA and Hadoop. During this presentation, CenterPoint will discuss their journey of moving smart meter data to Hadoop, how Hadoop is allowing CenterPoint to derive value from big data and their future use case road map.
Use Cases from Batch to Streaming, MapReduce to Spark, Mainframe to Cloud: To...
This document discusses how Syncsort helps companies access and integrate data from various sources to power analytics. It provides examples of how Syncsort has helped companies in insurance, media, and hotels easily onboard and integrate both historical and streaming data from multiple sources like mainframes, databases, and IoT devices. This allows for faster insights, increased productivity, cost savings, and helps future proof applications.
Data Process Systems, connecting everything
This document summarizes Patrick de Vries' presentation on connecting everything at the Hadoop Summit 2016. The presentation discusses KPN's use of Hadoop to manage increasing data and network capacity needs. It outlines KPN's data flow process from source systems to Hadoop for processing and generating reports. The presentation also covers lessons learned in implementing Hadoop including having strong executive support, addressing cultural challenges around data ownership, and leveraging existing investments. Finally, it promotes joining a new TELCO Hadoop community for telecommunications providers to share use cases and lessons.
Rob Bearden Keynote Hadoop Summit San Jose
Rob Bearden argues that data will transform everything as data has become a valuable corporate asset. Enterprises must change their business models to take advantage of all available data from billions of connected devices and consumer access to data or become irrelevant. Traditional constraints of using only structured data in silos and centralized data centers limit insights and the ability to gain actionable intelligence. However, open connected data platforms can break down these limitations by integrating all types of data from multiple sources. This will allow enterprises to gain deep historical analysis, real-time stream analytics at the edge, and machine learning capabilities to transform their industries.
Hybrid Cloud Strategy for Big Data and Analytics
Digital transformations require a new hybrid cloud—one that’s open by design, and frees clients to choose and change environments, data and services as needed. This approach allows cloud apps and services to be rapidly composed using the best relevant data and insights available, while maintaining clear visibility, control and security—everywhere. How do you decide where to put data on a hybrid cloud and how to use it? What’s the best hybrid cloud strategy in terms of data and workload? How should you leverage a 50/50 rule or a 80/20 rule and user interaction to evaluate which data/workload to move to the cloud and which data/workload to keep on-premise? Hybrid cloud provides an open platform for innovation, including cognitive computing. Organizations are looking for taking shadow IT out of the shadows by providing a self-service way to the information and a hybrid cloud strategy is allowing that. Also, how to use hybrid cloud for better manage data sovereignty & compliance?
Spark meets Smart Meters
This document discusses how Spark and big data analytics can power Australia's energy transformation as the country transitions to smart meters and renewable energy sources like solar and batteries. Key points:
- Smart meters are generating much more granular energy usage data that needs to be stored, processed and analyzed at large scale.
- Hadoop and Spark are well-suited for handling the exponentially growing volumes of time series energy data from smart meters.
- Advanced analytics on smart meter data can provide insights into customer energy usage patterns and behaviors to improve demand forecasting, marketing and grid management.
- Integrating distributed energy resources like rooftop solar and batteries with smart appliances and electric vehicles will require real-time coordination and optimization across the
It Takes a Village: Organizational Alignment to Deliver Big Data Value in Hea...
The business and technology teams within a health insurer must align the company’s central data platform with its data strategy. That requires substantial organizational alignment. Hear the firsthand perspective from Health Care Service Corporation (HCSC), the largest customer-owned health insurance company in the United States. The speaker will cover how they integrated membership information, regulatory compliance, and the general ledger, to improve overall healthcare management. At HCSC, the strong alignment between executive leadership, business portfolio direction, architectural strategy, technology delivery, and program management have helped create leading-edge capabilities which help the company respond nimbly to a quickly evolving healthcare industry.
How Market Intelligence From Hadoop on Azure Shows Trucking Companies a Clear...
TMW Systems (a Trimble Company) has been in the business of long-haul trucking, logistics operations and fleet management for more than thirty years, but we wanted more data, so we turned to our customer community. Now, we turn that data into market intelligence, which we then provide back to our customers. To do this, we invested heavily in Hortonworks Data Platform running on Microsoft Azure in the cloud. In our talk, we’ll share our strategy for capturing operational, maintenance, financial and mobile communications information and how we provide that back to our customer base. Our approach enables advanced analytics by leveraging Big Data technologies to find new relationships in data that may have been previously overlooked. Survey responses capture business performance metrics, strategy and emerging trends from 150 businesses, representing more than 31 billion dollars in freight movement. Learn how we combine that survey data with other sources like machine and sensor data to help guide our customers to profitability.
Depositing Value from Transactional Data at Danske Bank
Processing transactions is at the core of any bank’s business. Danske Bank’s journey started with recognising the value that could be gleaned from generating insights from the data to improve customer behaviour analytics. Today, the company streams large volumes of transactional data in near-real time onto its Hortonworks data Platform to improve fraud detection and customer marketing. In this session, Nadeem will outline the bank’s vision, how it was socialised across the executive board team and the resulting sponsorship, the technological path, challenges overcome and the results that have not only improved the customer experience but quantifiable metrics fraud and opening new revenue streams. Furthermore, Nadeem will cover future use cases around maintenance and operations.
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, Cloudera
Bernard Doering, Senior Slaes Director DACH, Cloudera.
Hadoop and the Future of Data Management. As Hadoop takes the data management market by storm, organisations are evolving the role it plays in the modern data centre. Explore how this disruptive technology is quickly transforming an industry and how you can leverage it today, in combination with MongoDB, to drive meaningful change in your business.
Democratizing Data Science on Kubernetes
Presented at the 2018 Rice Data Science Conference - ExxonMobil discusses how they leverage Openshift for Data Science team needs
Open Source in the Energy Industry - Creating a New Operational Model for Dat...
Centrica supplies energy to 28 million customers globally. It is developing integrated energy solutions for commercial and industrial customers through its Distributed Energy & Power division. Centrica created Io-Tahoe to provide a new operational model for data management that empowers businesses and IT to innovate using data. Io-Tahoe ingests diverse data sources into Centrica's data lake and uses smart data discovery and metadata management to create a known data model. This allows Centrica to extract more value from data through data science and gain business insights.
The Ecosystem is too damn big
The ecosystem for big data and analytics has become too large and complex, with too many vendors, distributions, engines, projects, and iterations. This leads to problems like analysis paralysis during platform decisions, solutions becoming obsolete too quickly, and constant stress over choosing the right engine for each job. The document suggests that industries, vendors, investors, analysts, technologists, and customers all need to take steps to reduce complexity and focus on standards and merit-based evaluations of options.
Extending Data Lake using the Lambda Architecture June 2015
The document discusses using a Lambda architecture to extend a data lake with real-time capabilities. It describes considerations for choosing a real-time architecture and common use cases. Specific examples discussed include using real-time architectures for patient critical care in healthcare and customer engagement in marketing.
Gov & Private Sector Regulatory Compliance: Using Hadoop to Address Requirements
This presentation discusses forward-looking statements that are subject to risks and uncertainties. It addresses issues around who owns data, who has access to data, and what type of data analysis can be done. It provides details on government-to-government, bank-to-government, and regional data exchanges. It discusses Rante's divisions and approach to unique experiences. Rante aims to anticipate industry trends and push boundaries through research and technology innovations.
Practical advice to build a data driven company
1) The document provides practical advice for building a data-driven company, including making data easily accessible, ensuring business awareness of opportunities to use algorithms, and having data science projects with the lowest time to market.
2) It emphasizes focusing on usability over architecture when building a data lake, and having feature teams to deliver code ready for production through a message broker that allows for reuse of data flows.
3) The document advises businesses to be aware of algorithm opportunities, mixing business and data science teams for collaboration, and spending time together through activities like pair programming.
Making Bank Predictive and Real-Time
This document discusses making banks more predictive and real-time using Hadoop. It describes the challenges of siloed data and batch processing at banks. The author details his bank's journey from setting up a small "play area" Hadoop cluster to experiment, to building a secure predictive analytics lab, to plans for a production system. Key challenges discussed include securing Hadoop, hardware limitations, and rapid tool innovation. Real-time analytics require tools beyond Hadoop like Storm or Spark. Business cases around marketing, spending forecasts and risk are highlighted. The author concludes Hadoop can save costs and accelerate predictive analytics when driven by business cases.
Using Hadoop for Cognitive Analytics
Using Hadoop for Cognitive Analytics discusses using Hadoop and external data sources for cognitive analytics. The document outlines solution architectures that integrate external and customer-specific metrics to improve decision making. Microservices are used for data ingestion and curation from various sources into Hadoop for storage and analytics. This allows combining business metrics with hyperlocal data at precise locations to provide insights.
Smart data for a predictive bank
ING Bank has developed a data lake architecture to centralize and govern all of its data. The data lake will serve as the "memory" of the bank, holding all data relevant for reporting, analytics, and data exchanges. ING formed an international data community to collaborate on Hadoop implementations and identify common patterns for file storage, deep data analytics, and real-time usage. Key challenges included the complexity of Hadoop, difficulty of large-scale collaboration, and ensuring analytic data received proper security protections. Future steps include standardizing building blocks, defining analytical model production, and embedding analytics in governance for privacy compliance.
Big Data Application Architectures - IoT
The document discusses reference architectures for building big data applications with Internet of Things (IoT) technologies. It describes an IoT reference architecture that includes components for device connectivity, data processing/analytics, and business connectivity. It provides examples of device types, connectivity options, and how to use Azure services for device identity/registry, stream processing, analytics, and presentation. Guiding principles are also outlined for building scalable, secure, and flexible IoT solutions.
Use dependency injection to get Hadoop *out* of your application code
Hadoop MapReduce provides transparent parallelization but often results in specialized code bases that interact with low-level data formats. We present a means of using dependency injection to manage data flows in MapReduce which in turn supports reusable, Hadoop-agnostic application code that interacts with high-level business domain objects. An example is provided that applies Dependency Injection to the Hadoop WordCount example and shows how the same code invoked from the WordCount MapReduce job can be reused in a real-time context. We then discuss Opower’s application of this pattern to employ the same core calculations in both batch processing and in servicing real-time requests from end users. This topic will be of interest to those interested in reusing core batch calculations in real-time contexts. It also provides a means forward for organizations moving to Hadoop that have existing code components that they would like to employ in batch MapReduce computations.
Benefits of Transferring Real-Time Data to Hadoop at Scale
Today’s Big Data teams demand solutions designed for Big Data that are optimized, secure, and adaptable to changing workload requirements. Working together, Hortonworks, IBM, and Attunity have designed an integrated solution that transfers large volumes of data to a platform that can handle rapid ingest, processing and analysis of data of all types from all sources, at scale.
https://hortonworks.com/webinar/benefits-transferring-real-time-data-hadoop-scale-ibm-hortonworks-attunity/
SplunkSummit 2015 - Real World Big Data Architecture
This document discusses big data architectures using Splunk, Hadoop, and relational databases. It begins with an overview of Splunk's scalability and real-time analytics capabilities. It then discusses Hunk, an analytics platform for Hadoop that provides self-service analytics. The document also examines using structured data in Splunk and connecting to relational databases. A case study examines challenges with the open source Hadoop ecosystem. Finally, it outlines a real-world customer architecture that uses Splunk for machine data, Hadoop for storage, Hunk for analytics, and connects to relational databases.
Hadoop dev 01
This document discusses big data analysis and Hadoop. It begins by describing different stages of data analysis and roles of various personnel. It then discusses challenges of analyzing big data using traditional tools and how Hadoop addresses these challenges through its distributed architecture and MapReduce programming model. Several case studies are presented where companies have used Hadoop to perform large-scale data analysis. Key components of Hadoop like MapReduce, Pig, Hive and Mahout are also introduced.
Oil and gas big data edition
The document discusses Informatica's data integration platform and its capabilities for big data and analytics projects. Some key points:
- Informatica is a leading data integration vendor with over 5,000 customers including over 70% of the Global 500.
- The Informatica platform provides capabilities across the entire data lifecycle from ingestion to delivery including data quality, master data management, integration, and analytics.
- It supports a variety of data sources including structured, unstructured, cloud, and big data and can run on-premises or in the cloud.
- Customers report the Informatica platform improves agility, scalability, and operational confidence for data integration projects compared to
Transforming GE Healthcare with Data Platform Strategy
Data and Analytics is foundational to the success of GE Healthcare’s digital transformation and market competitiveness. This use case focuses on a heavy platform transformation that GE Healthcare drove in the last year to move from an On prem legacy data platforming strategy to a cloud native and completely services oriented strategy. This was a huge effort for an 18Bn company and executed in the middle of the pandemic. It enables GE Healthcare to leap frog in the enterprise data analytics strategy.
Solving Big Data Problems using Hortonworks
Hortonworks provides an open source Apache Hadoop data platform for managing large volumes of data. It was founded in 2011 and went public in 2014. Hortonworks has over 800 employees across 17 countries and partners with over 1,350 technology companies. Hortonworks' Data Platform is a collection of Apache projects that provides data management, access, governance, integration, operations and security capabilities. It supports batch, interactive and real-time processing on a shared infrastructure using the YARN resource management system.
Building intelligent applications, experimental ML with Uber’s Data Science W...
In this talk, we will explore how Uber enables rapid experimentation of machine learning models and optimization algorithms through Uber’s Data Science Workbench (DSW). DSW covers a series of stages in data scientists’ workflows including data exploration, feature engineering, machine learning model training, testing, and production deployment. DSW provides interactive notebooks for multiple languages with on-demand resource allocation and the ability to share their works through community features. It also has support for notebooks and intelligent applications backed by Spark job servers. Deep learning applications based on TensorFlow and Torch can be brought into DSW smoothly where resources management is taken care of by the system. The environment in DSW is customizable where users can bring their own libraries and frameworks. Moreover, DSW provides support for Shiny and Python dashboards as well as many other in-house visualization and mapping tools.
In the second part of this talk, we will explore the use cases where custom machine learning models developed in DSW are productionized within the platform. Uber applies machine learning extensively to solve some hard problems. Some use cases include calculating the right price for a ride for over 600 cities and applying NLP technologies to customer feedbacks to offer safe rides and reduce support costs. We will look at various options evaluated for productionizing custom models (server based and serverless). We will also look at how DSW integrates into the larger Uber’s ML ecosystem, model/feature stores, and other ML tools to realize the vision of a complete ML platform for Uber.
Speakers
Adam Hudson, Uber, Senior Software Engineer
Atul Gupte, Uber, Product Manager
Performance Update: When Apache ORC Met Apache Spark
Apache Spark 1.4 introduced support for Apache ORC. However, initially it did not take advantage of the full power of ORC. For instance, it was slow because ORC vectorization was not used and push-down predicate wa s also not supported on DATE types. Recently the Apache Spark community has started to use the latest Apache ORC which include new enhancements to address these limitations. In this talk, we show the result of integrating the latest Apache ORC and Apache Spark. We will also review the latest enhancements and roadmap.
Speakers:
Owen O'Malley, Co-founder & Technical Fellow, Hortonworks
Dongjoon Hyun, Staff Software Engineer, Hortonworks
MaaS (Model as a Service): Modern Streaming Data Science with Apache Metron
Apache Metron is a streaming cybersecurity application
built on Apache Storm and Hadoop. One of its core missions is to enable advanced analytics through machine learning and data science to the users. Because of the relative immaturity of data science platform infrastructure integrated into Hadoop that is oriented to streaming analytics applications, we have been forced to create the requisite platform components out of necessity, utilizing many of the pieces of the Hadoop ecosystem.
In this talk, we will speak about the Metron analytics architecture and how it utilizes a custom data science model deployment and autodiscovery service that is tightly integrated with Hadoop via Yarn and Zookeeper. We will discuss how we interact with the models deployed there via a custom domain specific language that can query models as data streams past. We will generally discuss the full-stack data science tooling that has been created to enable data science at scale on an advanced analytics
streaming application.
Speaker:
Casey Stella, Principal Software Engineer/Data Scientist, Hortonworks
More Related Content
What's hot
Depositing Value from Transactional Data at Danske Bank
Processing transactions is at the core of any bank’s business. Danske Bank’s journey started with recognising the value that could be gleaned from generating insights from the data to improve customer behaviour analytics. Today, the company streams large volumes of transactional data in near-real time onto its Hortonworks data Platform to improve fraud detection and customer marketing. In this session, Nadeem will outline the bank’s vision, how it was socialised across the executive board team and the resulting sponsorship, the technological path, challenges overcome and the results that have not only improved the customer experience but quantifiable metrics fraud and opening new revenue streams. Furthermore, Nadeem will cover future use cases around maintenance and operations.
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, Cloudera
Bernard Doering, Senior Slaes Director DACH, Cloudera.
Hadoop and the Future of Data Management. As Hadoop takes the data management market by storm, organisations are evolving the role it plays in the modern data centre. Explore how this disruptive technology is quickly transforming an industry and how you can leverage it today, in combination with MongoDB, to drive meaningful change in your business.
Democratizing Data Science on Kubernetes
Presented at the 2018 Rice Data Science Conference - ExxonMobil discusses how they leverage Openshift for Data Science team needs
Open Source in the Energy Industry - Creating a New Operational Model for Dat...
Centrica supplies energy to 28 million customers globally. It is developing integrated energy solutions for commercial and industrial customers through its Distributed Energy & Power division. Centrica created Io-Tahoe to provide a new operational model for data management that empowers businesses and IT to innovate using data. Io-Tahoe ingests diverse data sources into Centrica's data lake and uses smart data discovery and metadata management to create a known data model. This allows Centrica to extract more value from data through data science and gain business insights.
The Ecosystem is too damn big
The ecosystem for big data and analytics has become too large and complex, with too many vendors, distributions, engines, projects, and iterations. This leads to problems like analysis paralysis during platform decisions, solutions becoming obsolete too quickly, and constant stress over choosing the right engine for each job. The document suggests that industries, vendors, investors, analysts, technologists, and customers all need to take steps to reduce complexity and focus on standards and merit-based evaluations of options.
Extending Data Lake using the Lambda Architecture June 2015
The document discusses using a Lambda architecture to extend a data lake with real-time capabilities. It describes considerations for choosing a real-time architecture and common use cases. Specific examples discussed include using real-time architectures for patient critical care in healthcare and customer engagement in marketing.
Gov & Private Sector Regulatory Compliance: Using Hadoop to Address Requirements
This presentation discusses forward-looking statements that are subject to risks and uncertainties. It addresses issues around who owns data, who has access to data, and what type of data analysis can be done. It provides details on government-to-government, bank-to-government, and regional data exchanges. It discusses Rante's divisions and approach to unique experiences. Rante aims to anticipate industry trends and push boundaries through research and technology innovations.
Practical advice to build a data driven company
1) The document provides practical advice for building a data-driven company, including making data easily accessible, ensuring business awareness of opportunities to use algorithms, and having data science projects with the lowest time to market.
2) It emphasizes focusing on usability over architecture when building a data lake, and having feature teams to deliver code ready for production through a message broker that allows for reuse of data flows.
3) The document advises businesses to be aware of algorithm opportunities, mixing business and data science teams for collaboration, and spending time together through activities like pair programming.
Making Bank Predictive and Real-Time
This document discusses making banks more predictive and real-time using Hadoop. It describes the challenges of siloed data and batch processing at banks. The author details his bank's journey from setting up a small "play area" Hadoop cluster to experiment, to building a secure predictive analytics lab, to plans for a production system. Key challenges discussed include securing Hadoop, hardware limitations, and rapid tool innovation. Real-time analytics require tools beyond Hadoop like Storm or Spark. Business cases around marketing, spending forecasts and risk are highlighted. The author concludes Hadoop can save costs and accelerate predictive analytics when driven by business cases.
Using Hadoop for Cognitive Analytics
Using Hadoop for Cognitive Analytics discusses using Hadoop and external data sources for cognitive analytics. The document outlines solution architectures that integrate external and customer-specific metrics to improve decision making. Microservices are used for data ingestion and curation from various sources into Hadoop for storage and analytics. This allows combining business metrics with hyperlocal data at precise locations to provide insights.
Smart data for a predictive bank
ING Bank has developed a data lake architecture to centralize and govern all of its data. The data lake will serve as the "memory" of the bank, holding all data relevant for reporting, analytics, and data exchanges. ING formed an international data community to collaborate on Hadoop implementations and identify common patterns for file storage, deep data analytics, and real-time usage. Key challenges included the complexity of Hadoop, difficulty of large-scale collaboration, and ensuring analytic data received proper security protections. Future steps include standardizing building blocks, defining analytical model production, and embedding analytics in governance for privacy compliance.
Big Data Application Architectures - IoT
The document discusses reference architectures for building big data applications with Internet of Things (IoT) technologies. It describes an IoT reference architecture that includes components for device connectivity, data processing/analytics, and business connectivity. It provides examples of device types, connectivity options, and how to use Azure services for device identity/registry, stream processing, analytics, and presentation. Guiding principles are also outlined for building scalable, secure, and flexible IoT solutions.
Use dependency injection to get Hadoop *out* of your application code
Hadoop MapReduce provides transparent parallelization but often results in specialized code bases that interact with low-level data formats. We present a means of using dependency injection to manage data flows in MapReduce which in turn supports reusable, Hadoop-agnostic application code that interacts with high-level business domain objects. An example is provided that applies Dependency Injection to the Hadoop WordCount example and shows how the same code invoked from the WordCount MapReduce job can be reused in a real-time context. We then discuss Opower’s application of this pattern to employ the same core calculations in both batch processing and in servicing real-time requests from end users. This topic will be of interest to those interested in reusing core batch calculations in real-time contexts. It also provides a means forward for organizations moving to Hadoop that have existing code components that they would like to employ in batch MapReduce computations.
Benefits of Transferring Real-Time Data to Hadoop at Scale
Today’s Big Data teams demand solutions designed for Big Data that are optimized, secure, and adaptable to changing workload requirements. Working together, Hortonworks, IBM, and Attunity have designed an integrated solution that transfers large volumes of data to a platform that can handle rapid ingest, processing and analysis of data of all types from all sources, at scale.
https://hortonworks.com/webinar/benefits-transferring-real-time-data-hadoop-scale-ibm-hortonworks-attunity/
SplunkSummit 2015 - Real World Big Data Architecture
This document discusses big data architectures using Splunk, Hadoop, and relational databases. It begins with an overview of Splunk's scalability and real-time analytics capabilities. It then discusses Hunk, an analytics platform for Hadoop that provides self-service analytics. The document also examines using structured data in Splunk and connecting to relational databases. A case study examines challenges with the open source Hadoop ecosystem. Finally, it outlines a real-world customer architecture that uses Splunk for machine data, Hadoop for storage, Hunk for analytics, and connects to relational databases.
Hadoop dev 01
This document discusses big data analysis and Hadoop. It begins by describing different stages of data analysis and roles of various personnel. It then discusses challenges of analyzing big data using traditional tools and how Hadoop addresses these challenges through its distributed architecture and MapReduce programming model. Several case studies are presented where companies have used Hadoop to perform large-scale data analysis. Key components of Hadoop like MapReduce, Pig, Hive and Mahout are also introduced.
Oil and gas big data edition
The document discusses Informatica's data integration platform and its capabilities for big data and analytics projects. Some key points:
- Informatica is a leading data integration vendor with over 5,000 customers including over 70% of the Global 500.
- The Informatica platform provides capabilities across the entire data lifecycle from ingestion to delivery including data quality, master data management, integration, and analytics.
- It supports a variety of data sources including structured, unstructured, cloud, and big data and can run on-premises or in the cloud.
- Customers report the Informatica platform improves agility, scalability, and operational confidence for data integration projects compared to
Transforming GE Healthcare with Data Platform Strategy
Data and Analytics is foundational to the success of GE Healthcare’s digital transformation and market competitiveness. This use case focuses on a heavy platform transformation that GE Healthcare drove in the last year to move from an On prem legacy data platforming strategy to a cloud native and completely services oriented strategy. This was a huge effort for an 18Bn company and executed in the middle of the pandemic. It enables GE Healthcare to leap frog in the enterprise data analytics strategy.
Solving Big Data Problems using Hortonworks
Hortonworks provides an open source Apache Hadoop data platform for managing large volumes of data. It was founded in 2011 and went public in 2014. Hortonworks has over 800 employees across 17 countries and partners with over 1,350 technology companies. Hortonworks' Data Platform is a collection of Apache projects that provides data management, access, governance, integration, operations and security capabilities. It supports batch, interactive and real-time processing on a shared infrastructure using the YARN resource management system.
Building intelligent applications, experimental ML with Uber’s Data Science W...
In this talk, we will explore how Uber enables rapid experimentation of machine learning models and optimization algorithms through Uber’s Data Science Workbench (DSW). DSW covers a series of stages in data scientists’ workflows including data exploration, feature engineering, machine learning model training, testing, and production deployment. DSW provides interactive notebooks for multiple languages with on-demand resource allocation and the ability to share their works through community features. It also has support for notebooks and intelligent applications backed by Spark job servers. Deep learning applications based on TensorFlow and Torch can be brought into DSW smoothly where resources management is taken care of by the system. The environment in DSW is customizable where users can bring their own libraries and frameworks. Moreover, DSW provides support for Shiny and Python dashboards as well as many other in-house visualization and mapping tools.
In the second part of this talk, we will explore the use cases where custom machine learning models developed in DSW are productionized within the platform. Uber applies machine learning extensively to solve some hard problems. Some use cases include calculating the right price for a ride for over 600 cities and applying NLP technologies to customer feedbacks to offer safe rides and reduce support costs. We will look at various options evaluated for productionizing custom models (server based and serverless). We will also look at how DSW integrates into the larger Uber’s ML ecosystem, model/feature stores, and other ML tools to realize the vision of a complete ML platform for Uber.
Speakers
Adam Hudson, Uber, Senior Software Engineer
Atul Gupte, Uber, Product Manager
What's hot (20)
Depositing Value from Transactional Data at Danske Bank
Depositing Value from Transactional Data at Danske Bank
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, Cloudera
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, Cloudera
Open Source in the Energy Industry - Creating a New Operational Model for Dat...
Open Source in the Energy Industry - Creating a New Operational Model for Dat...
Extending Data Lake using the Lambda Architecture June 2015
Extending Data Lake using the Lambda Architecture June 2015
Gov & Private Sector Regulatory Compliance: Using Hadoop to Address Requirements
Gov & Private Sector Regulatory Compliance: Using Hadoop to Address Requirements
Use dependency injection to get Hadoop *out* of your application code
Use dependency injection to get Hadoop *out* of your application code
Benefits of Transferring Real-Time Data to Hadoop at Scale
Benefits of Transferring Real-Time Data to Hadoop at Scale
SplunkSummit 2015 - Real World Big Data Architecture
SplunkSummit 2015 - Real World Big Data Architecture
Transforming GE Healthcare with Data Platform Strategy
Transforming GE Healthcare with Data Platform Strategy
Building intelligent applications, experimental ML with Uber’s Data Science W...
Building intelligent applications, experimental ML with Uber’s Data Science W...
Viewers also liked
Performance Update: When Apache ORC Met Apache Spark
Apache Spark 1.4 introduced support for Apache ORC. However, initially it did not take advantage of the full power of ORC. For instance, it was slow because ORC vectorization was not used and push-down predicate wa s also not supported on DATE types. Recently the Apache Spark community has started to use the latest Apache ORC which include new enhancements to address these limitations. In this talk, we show the result of integrating the latest Apache ORC and Apache Spark. We will also review the latest enhancements and roadmap.
Speakers:
Owen O'Malley, Co-founder & Technical Fellow, Hortonworks
Dongjoon Hyun, Staff Software Engineer, Hortonworks
MaaS (Model as a Service): Modern Streaming Data Science with Apache Metron
Apache Metron is a streaming cybersecurity application
built on Apache Storm and Hadoop. One of its core missions is to enable advanced analytics through machine learning and data science to the users. Because of the relative immaturity of data science platform infrastructure integrated into Hadoop that is oriented to streaming analytics applications, we have been forced to create the requisite platform components out of necessity, utilizing many of the pieces of the Hadoop ecosystem.
In this talk, we will speak about the Metron analytics architecture and how it utilizes a custom data science model deployment and autodiscovery service that is tightly integrated with Hadoop via Yarn and Zookeeper. We will discuss how we interact with the models deployed there via a custom domain specific language that can query models as data streams past. We will generally discuss the full-stack data science tooling that has been created to enable data science at scale on an advanced analytics
streaming application.
Speaker:
Casey Stella, Principal Software Engineer/Data Scientist, Hortonworks
Apache Hadoop Crash Course
Introduction: This workshop will provide a hands on introduction to Apache Hadoop using the HDP Sandbox on students’ personal machines.
Format: A short introductory lecture about Apache Hadoop and a few key additional Apache projects in the extended ecosystem used in the lab followed by a demo, lab exercises and a Q&A session.
Objective: To provide a quick and short hands-on introduction to Hadoop. This lab will use the following Hadoop components: HDFS, YARN, Apache Pig, Apache Hive, Apache Spark, and Apache Ambari User Views. You will learn how to move data into HDFS, explore the data, clean the data, issue SQL queries and then build a report with Apache Zeppelin.
Pre-requisites: Registrants must bring a laptop and have the Hortonworks Sandbox installed.
Speaker:
Rafael Coss, Data Community Developer Advocate, Hortonworks
Next Generation Execution for Apache Storm
With its large install base in production, the Storm 1.x line has proven itself as a stable and reliable workhorse that scales well horizontally. Much has been learnt from evolving the 1.x line that we can now leverage to build the next generation execution engine. Under the STORM-2284 umbrella, we are working hard to bring you this new engine which is being redesigned at a fundamental level for Storm 2.0. The goal is to dramatically improve performance and enhance Storm's abilities without breaking compatibility.
This improved vertical scaling will help meet the needs of the growing user base by delivering more performance with less hardware.
In this talk, we will take an in-depth look at the existing and proposed designs for Storm's threading model and the messaging subsystem. We will also do a quick run-down of the major proposed improvements and share some early results from the work in progress.
Speaker
Roshan Naik, Senior MTS, Hortonworks
The Apache Way
The Apache Way describes the community patterns and style of governance that all projects at the Apache Software Foundation are guided by. With a span of more than 20 years, and now more than 300 projects, the Apache Way has helped to establish long lasting, diverse communities of volunteers who collaborate to build software used by millions of users worldwide.
In this talk, I’ll outline the underlying principles of the Apache Way, what this means for projects and their ecosystems, and how the Apache Software Foundation is structured to support such a large number of projects.
Speaker
Brett Porter, Director, Apache Software Foundation
Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...
DataWorks Summit 2017 - Sydney
Alejandro Tesch, Cloud Evangelist, Asia Pacific and Japan, HPE
Big Data is a hot topic today for most organisations today as they race to convert vast amounts of data into useful information that can be leveraged to make critical decisions and recommendations in a very limited time windows. Today, there is a widely accepted talent gap when it comes to creating and managing Hadoop cluster, even for the experts – it can take hours (or days) to get a fully functional hadoop farm up and running. The HDP Ambari plugin for Sahara is looking to address most of this challenges by facilitating the deployment of Hortonworks Hadoop clusters and provide a set of open API to facilitate data analytics tasks in your own cloud. In this presentation we will cover why it makes sense to run your data analytics cluster in your cloud and we will demonstrate basic Sahara / Ambari functionality.
Data Guarantees and Fault Tolerance in Streaming Systems
Does your big data streaming pipeline have a hole in its pocket ? Streaming involves gathering data, processing it and delivering the results to the intended destinations in real time. Glitches at any stage can cause data loss unless the products employed in the pipeline provide the necessary guarantees and are configured properly to deliver on those guarantees.
Realtime stream processing brings unique challenges with respect to data handling guarantees and fault tolerance. Each streaming product comes with a unique approach to tackle these problems. When assembling a streaming pipeline, it is important to understand this critical topic for proper selection and configuration of the individual components of the pipeline. The exercise to determine if you are missing some records in your data lake can be expensive, but it can be extremely difficult to track down the cause to prevent it from recurring.
To help you build reliable streaming pipelines, this talk will give you a better understanding of the problems involved in realtime streaming, the kinds of guarantees involved and how they are handled in popular open source products such as Storm, Flink, Kafka, Hive Streaming APIs and Flume.
Speaker
Roshan Naik, Senior MTS, Hortonworks
The Power of Intelligent Flows: Real-Time IoT Botnet Classification with Apac...
The last 5 years have been marked by an explosion of Internet-connected devices. From cars to solar power, from TVs to juice makers, modern life is filled with interconnected smart devices.
But while those ubiquitous devices enhance the interaction with the technology that surrounds us, the lifecycle management of IoT firmware and poor security design choices still present a significant threat to our daily lives.
Despite the ascent of threats like the Mirai botnet, the amount of published research around how to programmatically detect new IoTs in the wild has been somewhat limited.
In this presentation we introduce Data Engineering in the context of cyber security, discuss why it is important to move away from the view that security log pipelines are enrichment and indicator matching tools, and push the boundaries of “Simple Event Processing” to demonstrate how Apache NiFi and Apache MiNiFi’s feature rich dataflows can be used to dynamically identify new IoT botnet activities in the wild.
Speakers
Andre Fucs De Miranda, Independent Consultant, Fluenda
Andy LoPresto, Sr. Member of Technical Staff, Hortonworks
Data-In-Motion Unleashed
The document discusses Hortonworks DataFlow (HDF), which is a platform for data in motion. HDF allows users to collect data at the edge, route and process streaming data with Apache NiFi and Kafka, and analyze, visualize, predict and prescribe outcomes from the data using HDF platform services. The HDF platform provides scalable stream processing, security, data provenance, and management capabilities for data in motion applications across the enterprise.
Data Science Crash Course
Introduction: This workshop will provide a hands on introduction to basic Machine Learning techniques with Spark ML using a Sandbox on students’ personal machines.
Format: A short introductory lecture on a select important supervised and unsupervised Machine Learning techniques followed by a demo, lab exercises and a Q&A session. The lecture will be followed by lab time to work through the lab exercises and ask questions.
Objective: To provide a quick and short hands-on introduction to Machine Learning with Spark ML. In the lab, you will use the following components: Apache Zeppelin (a “Modern Data Science Toolbox”) and Apache Spark. You will learn how to analyze the data, structure the data, train Machine Learning models and apply them to answer real-world questions.
Pre-requisites: Registrants must bring a laptop that can run the Hortonworks Data Cloud.
Speaker:
Robert Hryniewicz, Developer Advocate, Hortonworks
How Big Data and Deep Learning are Revolutionizing AML and Financial Crime De...
Banks, Payment Providers and capital markets firms are under intense regulatory mandate to process huge amounts of transaction-related data from both traditional and non-traditional sources. Compliance teams need to constantly analyze data-in-motion (wires, fund transfers, banking transactions) and data-at-rest (years worth of historical data) for actionable intelligence required for Suspicious Activity Reports—to discover illegal activity and provide detailed reporting to authorities. Annual estimates of global money laundering flows ranging anywhere from $ 1 trillion to 2 trillion – almost 5% of global GDP. Almost all of this is laundered via Retail & Merchant Banks, Payment Networks, Securities & Futures firms, Casino Services & Clubs etc – which explains why annual AML related fines on Banking organizations run into the billions and are increasing every year. However, the number of SARs (Suspicious Activity Reports) filed by banking institutions are much higher as a category as compared to the numbers filed by these other businesses. In this presentation we will discuss the business imperatives, value drivers and the woeful inadequacy of current technology architectures and approaches in tackling AML. We will then pivot to a deepdive around Big Data and Predictive Analytics in how they can ease and solve these vexing challenges that Banking executives are grappling with globally.
Speaker
Sanjay Kumar, GM Industry Solutions - Telecom and FS, Hortonworks
SparkR Best Practices for R Data Scientists
Spark plays an important role on data scientists to solve all kinds of problems, especially the release of SparkR which provide very friendly APIs for traditional data scientists. However, processing various data size, data format and models will lead to different application patterns compared with traditional R. In this talk, we will illustrate the practical experience that using SparkR to solve some typical data science problems, such as the performance improvement for SparkR and native R interoperation, how to load data from HBase which is a very common data source efficiently, how to schedule a large scale machine learning job with multiple single R machine learning jobs, how to tuning performance for jobs triggered by many different users, how to use SparkR in the cloud-based environment, etc. At last, we will shortly introduce the community efforts in progress on SparkR in the coming releases.
Speakers:
Yanbo Liang, Software Engineer, Hortonworks
Casey Stella, Principal Software Engineer/Data Scientist, Hortonworks
Delivering Data Science to the Business
DataWorks Summit 2017 - Sydney Keynote
Madhu Kochar, Vice President, Analytics Product Development and Client Success, IBM
Data science holds the promise of transforming businesses and disrupting entire industries. However, many organizations struggle to deploy and scale key technologies such as machine learning and deep learning. IBM will share how it is making data science accessible to all by simplifying the use of a range of open source technologies and data sources, including high performing and open architectures geared for cognitive workloads.
Apache Spark Crash Course
Introduction: This workshop will provide a hands-on introduction to Apache Spark using the HDP Sandbox on students’ personal machines.
Format: A short introductory lecture about Apache Spark components used in the lab followed by a demo, lab exercises and a Q&A session. The lecture will be followed by lab time to work through the lab exercises and ask questions.
Objective: To provide a quick and short hands-on introduction to Apache Spark. This lab will use the following Spark and Apache Hadoop components: Spark, Spark SQL, Apache Hadoop HDFS, Apache Hadoop YARN, Apache ORC, and Apache Ambari User Views. You will learn how to move data into HDFS using Spark APIs, create Apache Hive tables, explore the data with Spark and Spark SQL, transform the data and then issue some SQL queries.
Pre-requisites: Registrants must bring a laptop that can run the Hortonworks Data Cloud.
Speaker:
Robert Hryniewicz, Developer Advocate, Hortonworks
The Future of Data in Telecom and the Rise of Connected Communities
The Telecom Industry has been at the heart of the new Digital Transformation Era with the focal points of the smart phone, digital content, and social media allowing us to interact with the new world. This digital consumer era is overlapping with the connected world of smart cities, connected car, smart homes, connected medical devices, and the industrial internet of things. This is giving rise to the new connected communities, member networks or marketplaces sharing data and insights across these new functional ecosystems. Service providers are being challenged to better manage the relationship and the experiences of their customers, build and leverage smart networks, and drive or facilitate these new digital services and connected communities.
Join Sanjay Kumar, General Manager of the Telecom Industry at Hortonworks as he facilitates an interactive session with your peers and other executives from service providers and operators in discussing the future of data in telecom and the rise of connected communities. Share your experiences and hear from other telecom service providers on their journey with big data and the transformation to becoming a data-drive organization.
During this interactive session, areas of discussion will include:
- Big data and real-time analytics for enhancing customer experience and target advertising
- Analytics driven next generation OSS and the Smart Network
- Connected communities that leveraging blockchain with big data
In the age of the connected world, hear from industry leaders about the role of the service provider and their relationship with their customers and connected communities across this new digital ecosystem.
Speaker
Sanjay Kumar, General Manager, Telecom & FS, Hortonworks
Running Zeppelin in Enterprise
Apache Zeppelin has become a popular way to unlock the value of data lake due to its user interface and appeal to business users. These business users ask their IT department for access to Zeppelin. Enterprise IT department want to help their business users but they have several enterprise concerns such as enterprise security, integration with their corporate LDAP/AD, scalability and multi-user environment, integration with Ranger and Kerberos. This session will walk through enterprise concerns and how these concerns can be handled with Zeppelin.
Speaker
Simon Elliston Ball, Director Product Management, Cyber Security, Hortonworks
An Apache Hive Based Data Warehouse
Using Apache Hadoop and related technologies as a data warehouse has been an area of interest since the early days of Hadoop. In recent years Hive has made great strides towards enabling data warehousing by expanding its SQL coverage, adding transactions, and enabling sub-second queries with LLAP. But data warehousing requires more than a full powered SQL engine. Security, governance, data movement, workload management, monitoring, and user tools are required as well. These functions are being addressed by other Apache projects such as Ranger, Atlas, Falcon, Ambari, and Zeppelin. This talk will examine how these projects can be assembled to build a data warehousing solution. It will also discuss features and performance work going on in Hive and the other projects that will enable more data warehousing use cases. These include use cases like data ingestion using merge, support for OLAP cubing queries via Hive’s integration with Druid, expanded SQL coverage, replication of data between data warehouses, advanced access control options, data discovery, and user tools to manage, monitor, and query the warehouse.
Speaker
Alan Gates, Co-founder, Hortonworks
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love.
Expected prior knowledge / intended audience: developers and data flow managers should have passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi.
Speaker
Andy LoPresto, Sr. Member of Technical Staff, Hortonworks
Viewers also liked (18)
Performance Update: When Apache ORC Met Apache Spark
Performance Update: When Apache ORC Met Apache Spark
MaaS (Model as a Service): Modern Streaming Data Science with Apache Metron
MaaS (Model as a Service): Modern Streaming Data Science with Apache Metron
Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...
Driving in the Desert - Running Your HDP Cluster with Helion, Openstack, and ...
Data Guarantees and Fault Tolerance in Streaming Systems
Data Guarantees and Fault Tolerance in Streaming Systems
The Power of Intelligent Flows: Real-Time IoT Botnet Classification with Apac...
The Power of Intelligent Flows: Real-Time IoT Botnet Classification with Apac...
How Big Data and Deep Learning are Revolutionizing AML and Financial Crime De...
How Big Data and Deep Learning are Revolutionizing AML and Financial Crime De...
The Future of Data in Telecom and the Rise of Connected Communities
The Future of Data in Telecom and the Rise of Connected Communities
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
Similar to Beyond Big Data: Data Science and AI
Driving Digital Transformation Through Global Data Management
Using your data smarter and faster than your peers could be the difference between dominating your market and merely surviving. Organizations are investing in IoT, big data, and data science to drive better customer experience and create new products, yet these projects often stall in ideation phase to a lack of global data management processes and technologies. Your new data architecture may be taking shape around you, but your goal of globally managing, governing, and securing your data across a hybrid, multi-cloud landscape can remain elusive. Learn how industry leaders are developing their global data management strategy to drive innovation and ROI.
Presented at Gartner Data and Analytics Summit
Speaker:
Dinesh Chandrasekhar
Director of Product Marketing, Hortonworks
Driving Digital Transformation through Global Data Management by Hortonworks
This is a talk I gave at Gartner Data & Analytics Summit in Dallas, TX on March 8th, 2018. This walks you through how enterprises are adapting into Modern Data Architectures involuntarily due to the technology trends and consumption patterns that are directly impacting innovations. It also explains how data exists as Data-at-Rest and Data-in-Motion on-premises as well as across multi-cloud environments. So, data management challenges are abound in today's enterprises. This talk then examines how Global Data Management addresses such challenges and enables enterprises to thrive at Digital Transformation. It explains how Hortonworks offers a comprehensive Global Data Management offering that allows you to manage, govern and secure your global data footprint.
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise
The document discusses the need for a "Smart Enterprise Big Data Bus" to efficiently synchronize data in real-time between various enterprise applications and Hadoop, as existing integration approaches are unable to meet the demands of real-time analytics. It proposes using a streaming analytics platform called StreamAnalytix to create a high-speed data bus that can ingest, transform, analyze, and synchronize data across the enterprise. Examples of implementing such a solution for a healthcare company are also provided.
Reinvent Your Data Management Strategy for Successful Digital Transformation
Watch Dinesh's keynote presentation from Fast Data Strategy Virtual Summit here: https://goo.gl/3Pa8np
Leaders are re-inventing their data management strategies through the effective use of IoT, Big Data, and data science to boost their customer experience. Yet, they struggle to modernize their data architecture due to lack of global data management processes and technologies.
Attend this session to hear from the Big Data pioneer, Hortonworks:
• Why big data and data virtualization should be core technology components of your digital transformation.
• How to manage, govern, and secure your global data footprint across a hybrid multi-cloud landscape.
• Learn about key global data management strategies and use cases that drive leading digital enterprises.
Harnessing Big Data_UCLA
This document discusses methods for harnessing big data. It describes how sensors collect Internet of Things (IoT) data and how Volvo applies analytics. It also summarizes three methods: 1) The US Air Force uses an integrated data warehouse and geospatial analysis to track assets globally. 2) Siemens uses data discovery processes to predict train failures by analyzing sensor and failure report data. 3) Yahoo uses Hadoop as a data lake to store and analyze large amounts of user data from various sources like social media and clickstreams. The document emphasizes that no single technology is a silver bullet for big data.
Denodo DataFest 2017: Edge Computing: Collecting vs. Connecting to Streaming ...
This document discusses connected data and edge computing. It summarizes that connected devices, customers, vehicles, and assets are fueling new business models powered by streaming data, artificial intelligence, cloud computing, and the internet of things. It then describes Hortonworks' data platforms for managing both data at rest and in motion across cloud, on-premises and hybrid environments to enable analytics and power the modern data architecture.
Hortonworks - IBM Cognitive - The Future of Data Science
The document discusses Hortonworks and IBM's partnership around data management and analytics. It highlights how their combined platforms can power the modern data architecture with solutions for data at rest and in motion. Examples are provided of how customers like Merck and JPMC have leveraged Hortonworks' technologies to gain insights from their data and drive business outcomes. Industries that are investing in data science are also listed.
Roger Woodward, Managing Director, EMEA - Tridium
1) The document discusses how connecting buildings and devices to the Internet of Things (IoT) can change businesses and create new opportunities through services like remote maintenance, energy reporting, and facility management.
2) It notes that while products may already be "smart," connecting them allows accessing data to provide services, conduct analytics, and create new revenue streams. This "servitization" transforms businesses from product sales to solutions.
3) The talk encourages collaborating with other companies to take advantage of opportunities in connected buildings and smart cities, as no single company can address this transformation alone. Mobility, analytics, security and other technologies will be crucial enablers.
Global Data Management – a practical framework to rethinking enterprise, oper...
Global data management is not a newly coined term. However, what it stands for is actually widening in scope particularly around data-in-motion and data-at-rest. Significant technology trends such as IoT, cloud, AI/ML, blockchain, and streaming data have given rise to excessive data volumes and also innovative use cases. The scope for global data management now extends all the way from ingestion, processing, storage, governance, security to analysis. With a good number of endpoints served through the cloud and major application footprints remaining on-premisess, it is pertinent to have a global data management strategy that supports hybrid models and more specifically, a multi-cloud model.
Many modern businesses struggle to balance the demands of rapidly innovating through new technologies like machine learning with the need to keep data safe and secure, all while responding to a constantly changing regulatory landscape. This puts data stewards, data engineers, architects, data scientists, and analysts under intense pressure as they must contend with existing and new applications, multiple logical and physical data stores and sources, diverse data types, and data spread across several deployment environments.
Attend this session led by Matt Aslett, Research Director at 451 Research and Dinesh Chandrasekhar, Director, Hortonworks to learn more about creating a framework for your enterprise that offers guidance on how to think about global data management—priorities, responsibilities, key stakeholders, compliance, and growth.
Speakers
Dinesh Chandrasekhar, Hortonworks, Director Product Marketing
Matt Aslett, 451 Research, Research Director, Data platforms and Analytics
Digital innovations
Digital innovationsMadCann Alberta Inc. —Digital Strategy—Digital Innovation—Digital Roadmap—Digital Technologies
Digital technologies are exerting strong influence on oil and gas. Sensors generate data, artificial intelligence reads the data, robots use the data, cloud stores the data, blockchain provides the trust and confidence in the systems. Big Data Solutions on Cloud – The Way Forward by Kiththi Perera SLT
ITU-TRCSL Symposium on Cloud Computing 2015 Colombo
Session 04: Big Data Strategy in the Cloud and Applications
Speaker's PPT by K. A. Kiththi Perera, Chief Enterprise and Wholesale Officer, Sri Lanka Telecom
Big data solutions on cloud – the way forward
Big data solutions on the cloud are the future according to the author. Big data analytics and cloud computing both offer organizations valuable insights, innovations, and revenue opportunities while also enhancing agility, efficiency, and reducing costs. Hadoop is commonly used for big data but Spark is emerging as more flexible. The cloud is becoming a popular option for big data due to elastic infrastructure and ability to process data close to its source. Common big data use cases include optimizing sales funnels, behavioral analytics, customer segmentation, predictive maintenance, and market analysis.
MapR Streams and MapR Converged Data Platform
The document discusses MapR Streams, a global publish/subscribe event streaming system. It provides converged, continuous, and global capabilities. MapR Streams allows producers to publish billions of messages per second to topics, and guarantees immediate and reliable delivery to consumers. It also enables tying together geo-dispersed clusters globally. The document demonstrates MapR Streams capabilities with a live demo and discusses use cases for event streaming across various industries.
Big data Introduction by Mohan
This document provides an introduction to big data and Hadoop. It discusses the three V's of big data: volume, variety, and velocity. Examples are given of the large amounts of data generated daily from various sources. The growth and market opportunity for big data technologies is also discussed. Common use cases for big data in different industries are outlined. The document then covers Hadoop components and how Hadoop HDFS and MapReduce work. Other Hadoop technologies like Hive, Pig, and Zookeeper are introduced. Benefits of Hadoop and commercial Hadoop distributions are summarized. Finally, technologies alternative to Hadoop like HPCC and SAP HANA are briefly described.
Accelerate Digital Transformation with Data Virtualization in Banking, Financ...
Watch full webinar here: https://bit.ly/38uCCUB
Banking, Financial Services and Insurance (BFSI) organizations are globally accelerating their digital journey, making rapid strides with their digitization efforts, and adding key capabilities to adapt and innovate in the new normal.
Many companies find digital transformation challenging as they rely on established systems that are often not only poorly integrated, but also highly resistant to modernization without downtime. Hear how the BFSI industry is leveraging data virtualization that facilitates digital transformation via a modern data integration / data delivery approach to gain greater agility, flexibility, and efficiency.
In this joint live webinar session from Denodo and Wipro, you will learn:
- Industry key trends and challenges driving the digital transformation mandate and platform modernization initiatives
- Key concepts of Data Virtualization, and how it can enable BFSI customers to develop critical capabilities for real-time / near real-time data integration
- Success Stories on organizations who already use data virtualization to differentiate themselves from the competition
- Wipro’s role in helping enterprises define the business case, end-to-end services and operating model for the successful data virtualization implementations
Schedule a Discovery Session to learn more about Wipro and Denodo joint solutions for Banking, Financial Services, and Insurance.
Cloud 2020: taking your customers into the future - Peter Schwartz Avanxo Clo...
This document summarizes a keynote presentation by Peter Schwartz on emerging technologies and their impact. It discusses how technological changes are transforming human relationships and driving adoption of cloud computing. It outlines several trends, including ubiquitous mobile access; new user interfaces using facial/gesture recognition and augmented reality; a social and collaborative revolution; and a connected and contextual future with predictive analytics and real-time engagement powered by billions of connected devices and sensors. Emerging technologies discussed include in-memory computing, fog/edge computing, and neuromorphic processors that could enable more human-like artificial intelligence.
What Is Rain
Rain provides services and solutions to help clients rapidly transform their businesses through innovation. It connects clients to participants from diverse ecosystems to power this transformation. Some key benefits Rain delivers include up to 40% reduction in implementation time, faster solution building through prototyping and leveraging assets, and a rapid, measurable, and innovative path to value.
Wireless communication in big data era vfinal upload
Delivered by Venkata Rangarajan, Orzota India Development Center
This was delivered as a open talk at an academic institution as a walk through for the students of Communication Engineering on Big Data Applications in Wireless Communication Industry. The Talk has derived its contents from a variety of research articles and technical papers to put forth the potential areas for deployment.
Author duly thanks the authors and publishers of the articles and papers from which some of the artifacts have been derived or republished.
The contents of this document are solely that of the author and does not represent the views or opinions of the organization he is representing.
Kinexions 2014 Breakout_28Oct2014
The document discusses how the planning process is being transformed by incorporating data from the Internet of Things (IoT). It outlines how IoT connects products, machines, infrastructure and people, generating vast amounts of new data. This data from across the entire value chain can improve planning accuracy by removing latency and allowing events to be predicted. The reference architecture shows how IoT data on logistics, transportation, weather and more can enhance planning effectiveness. The key takeaway is that tomorrow's planning must leverage IoT to adapt in real-time and remain agile in increasingly complex operating environments.
Neo4j GraphTalks - Einführung in Graphdatenbanken
The document announces a Neo4j GraphTalks event in October 2016 in Berlin. It includes an agenda with presentations on ADAMA's use of Neo4j for data sharing and knowledge management, and their experiences implementing and demoing Neo4j. There will also be an open networking session with NeoTechnology and PRODYNA representatives.
Similar to Beyond Big Data: Data Science and AI (20)
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation through Global Data Management by Hortonworks
Driving Digital Transformation through Global Data Management by Hortonworks
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise
Reinvent Your Data Management Strategy for Successful Digital Transformation
Reinvent Your Data Management Strategy for Successful Digital Transformation
Denodo DataFest 2017: Edge Computing: Collecting vs. Connecting to Streaming ...
Denodo DataFest 2017: Edge Computing: Collecting vs. Connecting to Streaming ...
Hortonworks - IBM Cognitive - The Future of Data Science
Hortonworks - IBM Cognitive - The Future of Data Science
Global Data Management – a practical framework to rethinking enterprise, oper...
Global Data Management – a practical framework to rethinking enterprise, oper...
Big Data Solutions on Cloud – The Way Forward by Kiththi Perera SLT
Big Data Solutions on Cloud – The Way Forward by Kiththi Perera SLT
Accelerate Digital Transformation with Data Virtualization in Banking, Financ...
Accelerate Digital Transformation with Data Virtualization in Banking, Financ...
Cloud 2020: taking your customers into the future - Peter Schwartz Avanxo Clo...
Cloud 2020: taking your customers into the future - Peter Schwartz Avanxo Clo...
Wireless communication in big data era vfinal upload
Wireless communication in big data era vfinal upload
More from DataWorks Summit
Data Science Crash Course
Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL).
Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW).
Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models.
Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.
Floating on a RAFT: HBase Durability with Apache Ratis
In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort.
This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase.
Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs.
Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables.
Resources:
https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html
https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html
https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html
HBase Tales From the Trenches - Short stories about most common HBase operati...
Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.
Managing the Dewey Decimal System
OCLC has been using HBase since 2012 to enable single-search-box access to over a billion items from your library and the world’s library collection. This talk will provide an overview of how HBase is structured to provide this information and some of the challenges they have encountered to scale to support the world catalog and how they have overcome them.
Practical NoSQL: Accumulo's dirlist Example
Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL.
Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist).
In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.
HBase Global Indexing to support large-scale data ingestion at Uber
Danny Chen presented on Uber's use of HBase for global indexing to support large-scale data ingestion. Uber uses HBase to provide a global view of datasets ingested from Kafka and other data sources. To generate indexes, Spark jobs are used to transform data into HFiles, which are loaded into HBase tables. Given the large volumes of data, techniques like throttling HBase access and explicit serialization are used. The global indexing solution supports requirements for high throughput, strong consistency and horizontal scalability across Uber's data lake.
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions.
These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
This document discusses using Apache NiFi to build a high-speed cyber security data pipeline. It outlines the challenges of ingesting, transforming, and routing large volumes of security data from various sources to stakeholders like security operations centers, data scientists, and executives. It proposes using NiFi as a centralized data gateway to ingest data from multiple sources using a single entry point, transform the data according to destination needs, and reliably deliver the data while avoiding issues like network traffic and data duplication. The document provides an example NiFi flow and discusses metrics from processing over 20 billion events through 100+ production flows and 1000+ transformations.
Supporting Apache HBase : Troubleshooting and Supportability Improvements
This document discusses supporting Apache HBase and improving troubleshooting and supportability. It introduces two Cloudera employees who work on HBase support and provides an overview of typical troubleshooting scenarios for HBase like performance degradation, process crashes, and inconsistencies. The agenda covers using existing tools like logs and metrics to troubleshoot HBase performance issues with a general approach, and introduces htop as a real-time monitoring tool for HBase.
Security Framework for Multitenant Architecture
In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.
With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.
Extending Twitter's Data Platform to Google Cloud
Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.
Computer Vision: Coming to a Store Near You
Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as:
● Optimizing merchandising execution, in-stocks and sell-thru
● Enhancing operational efficiencies, enable real-time customer engagement
● Enhancing loss prevention capabilities, response time
● Creating frictionless experiences for shoppers
Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry.
We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey.
Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables.
We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance.
We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing.
Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems.
By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.
More from DataWorks Summit (20)
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
HBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at Uber
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything Engine
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Recently uploaded
Azure API Management to expose backend services securely
How to use Azure API Management to expose backend service securely
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
When it comes to unit testing in the .NET ecosystem, developers have a wide range of options available. Among the most popular choices are NUnit, XUnit, and MSTest. These unit testing frameworks provide essential tools and features to help ensure the quality and reliability of code. However, understanding the differences between these frameworks is crucial for selecting the most suitable one for your projects.
A Comprehensive Guide to DeFi Development Services in 2024
DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum.
In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance.
In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape.
At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology.
Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI.
With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Recently uploaded (20)
Azure API Management to expose backend services securely
Azure API Management to expose backend services securely
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Beyond Big Data: Data Science and AI
- 2. The New Way of Business Is Fueled by Data • Connected customers, vehicles, devices • Socially crowd-sourced requirements • Digital design and analysis • Digital prototypes and tests • Connected factories, sensors, devices • Human-robotic interaction • 3D-printing on demand • Connected trucks, inventory • Location, traffic, weather-aware distribution • Real-time inventory visibility • Dynamic rerouting • Connected customers, devices • Omni- channel demand sensing • Real-time recommendations • Connected assets • Remote service monitoring & delivery • Predictive maintenance • OTA updates MANUFACTURING DISTRIBUTION MARKETING/SALES SERVICEDEVELOPMENT
- 4. INTERNET OF THINGS DATA SCIENCE/ MACHINE LEARNING CLOUD COMPUTING STREAMING DATA ~$380B $210B~$1300B ~$19B Sources: Public Cloud Services Market size, $383B by 2020, Gartner 2017 WW Public Cloud Services market. Big Data & Business Analytics revenues forecast to be $210B by 2020, IDC 2017. IoT Spending forecast to be ~1.31T by 2020, IDC 2017 Worldwide IoT Spending Guide. AI intelligence market size to reach $19,478 million by 2022, growing at a CAGR of 45.4% from 2016 to 2022, Allied Market Research. The Perfect Storm: Tech Trends Fueling Business
- 6. Exception-Based Monitoring 360 View of Operations, Equipment Failure Analytics, etc. Deep Historical Analysis DATA C E NT E R Stream Analytics Cyber Security & Threat Detection Telemetry – Connected Devices Machine Learning C LO UD Sensors, SCADA, Control Systems Edge Analytics Time Series Historian Modern Data Architecture
- 8. Powering the Modern Data Architecture DATA AT RESTDATA IN MOTION ACTIONABLE INTELLIGENCE COMPLETE DATA LIFECYCLE MANAGEMENT RUN CONTAINERIZED APPLICATIONS CONCURRENTLY EDGECLOUD H O L I S T I C M A N A G E M E N T, G O V E R N A N C E A N D S E C U R I T Y ON-PREMISES MULTI-WORKLOADS MULTI-TYPE MULTI-TIER
- 10. 10 T H A N K Y O U
Editor's Notes
- The future of the enterprise is becoming clearer as organizations begin to realize the strategic value of data. Today I want to walk you through the drivers and look at what the high level architecture will look like as enterprises realize that value.
- * Business across all industries are undergoing a digital transformation of massive scale. * Establishing a world where they are connecting everything to everything else. People, devices, vehicles
- We started our journey by making Hadoop ready for the enterprise. Established a data platform for structured data AND the new paradigm data from streams and social platforms open community open ecosystem. Multi tenancy and integrated security and governance Data in motion: manage data through its entire lifecycle from inception to where it lands at rest. With security, governance, lineage across that entire journey. On Prem and cloud Now serving ML, DL and AI
- Over the course of the last 5 years the 4 mega trends have driven and even accelerated the need to transform into a modern data architecture. * These trends are powering and enabling these transformations * Driving with it tremendous rewards for winners and losers in each industry
- Hortonworks open and Connected Platforms enable this transformation and are the core of the modern data architecture. Real time decisions on data in motion and data at rest—to the edge.
- So, what’s really happening? There is a an entire new world being created by combining lots of data with break through tools. Data could be on-premises and in the cloud Data is moving from sensors in real time across our data fabric and giving us precise instrumentation of what happened just before an event as well as after the event. This is true for customers buying on the web as well as products that might fail. We can run our machine learning and deep learning on these vast repositories of data And we can push these models down to the edges to automate decision
- • Data Science is the next key driver to transformational business execution • Companies need a strategic approach to turn data into value and create a competitive advantage We need three things today to succeed: • Lots of data makes the models very accurate • Lots of compute makes the models run fast • Data science as a team sport, new tools enable collaboration and make the models easy to deploy.
- We are not done innovating – the journey will continue We have solved the need for Hadoop to become enterprise ready We saw the need to manage data from inception to rest with our data in motion platform And we saw the need for driving the consumability of these platforms via the cloud Like we did for Hadoop, we are now working on making our the modern Data Architecture enterprise ready and usable