Data Science: Driving Smarter Finance and Workforce Decsions for the Enterprise
•
2 likes•930 views
The document discusses different levels of analytics maturity from reactive operational reporting to prescriptive analytics. It provides examples of analytics applications including predicting top talent retention and identifying abnormal patterns in organizational structures. The second half of the document focuses on building a state-of-the-art analytics system, outlining key components like data integration, machine learning pipelines for feature extraction, model training and evaluation, and publishing results.
Report
Share
Report
Share
Recommended
Designing Data Pipelines for Automous and Trusted Analytics
This document discusses designing data pipelines for autonomous analytics. It notes that up to 80% of analyst time is spent on data preparation and that big data is difficult to adopt, process, and trust. It then presents the need for speed, quality, agility and autonomy in big data projects. The solution proposed is to design for autonomous analytics by automating data discovery, preparation, security, documentation and recommending best actions using machine learning to deliver trusted and timely data.
IoT: How Data Science Driven Software is Eating the Connected World
The document discusses how data science can be used to improve operations in the oil and gas industry through the Internet of Things. Large amounts of sensor data are generated during drilling operations that can be used to build predictive models to optimize drilling and predict equipment failures. Examples of opportunities include using models to predict drill rate of penetration to lower costs and failure prediction to allow for early warning and reduce downtime. The challenges of working with large sensor datasets and building accurate models at scale are also covered.
A Mayo Clinic Big Data Implementation
Brian Brownlow is an experienced senior analyst programmer for Mayo Clinic. He is made a workshop presentation at the 2014 BDPA Technology Conference on the topic, 'Big Data Implementation - Mayo Clinic Case Study'. This presentation will show part of the Mayo Clinic story on the embarking of an exploration of the application of `Big Data' technologies. `Big Data' is seen as one set of tools that can be used to enhance medical research, medical education and practice management. Mayo Clinic is always searching for better, faster and cheaper ways to use its data to improve patient care and sustain financial outcomes in a challenging reimbursement environment. Our approach uses several components that are open source and combines them with data from various sources to provide information to decision makers in near real time. We have created a center of `Big Data' excellence using in-house staff and vendor engagements. `Big Data' is one element of our Enterprise Data Trust framework.
Hadoop in the cloud – The what, why and how from the experts
The document discusses Hadoop in the cloud and its benefits. It summarizes that Hadoop in the cloud provides distributed storage, automated failover, hyper-scaling, distributed computing, and extensibility. It also discusses deploying Hadoop clusters in Azure HDInsight and options for customizing clusters and integrating them.
IT @ Intel: Preparing the Future Enterprise with the Internet of Things
The Internet of Things (IoT) is the concept of diverse machines, devices, and technologies connecting, interacting, and negotiating with each other to help improve and enrich our lives. No longer is this limited to just computer or smart phone technology. Everyday items such as household appliance, cars and even toys can connect to the internet to integrate with other computing things, processes and services. This new paradigm is changing how data is used and collected, and introducing new challenges for enterprises.
Optimizing your Hadoop Infastructure: An Industry Panel Presentation
This document introduces the panel speakers for a discussion on modernizing Hadoop infrastructures. It provides brief biographies of five speakers: Armando Acosta of Dell, who has 15 years of experience in IT solutions and big data; Brandon Draeger of Intel, who manages partnerships between Intel, Cloudera, and their shared ecosystems; TJ Laher of Cloudera, who helps organizations implement Hadoop; Vin Sharma of Intel; and Mark Muncy of Syncsort, who leads technical marketing for big data and has experience in data architecture. The panel will discuss how to evolve Hadoop capabilities to meet emerging customer needs.
Hadoop Summit San Jose 2015: YARN - Past, Present and Future
The document discusses the past, present, and future of Apache Hadoop YARN. It describes how YARN was created to address limitations in MapReduce and provide a more flexible resource management framework. The presentation outlines major releases of YARN from 2010 to 2015, focusing on new features like rolling upgrades, long-running services, node labels, and improved usability tools. It envisions future enhancements such as per-queue scheduling policies, reservations, containerized applications, and improved network and disk isolation.
Moving Health Care Analytics to Hadoop to Build a Better Predictive Model
This document discusses Dignity Health's move to using Hadoop for healthcare analytics to build better predictive models. It outlines their goals of saving costs and lives by leveraging over 30 TB of clinical data using Hadoop and SAS technologies on their Dignity Health Insights platform. The presentation agenda covers Dignity Health, healthcare analytics challenges, their big data ecosystem architecture featuring Hadoop, and how they are using this infrastructure for applications like sepsis surveillance analytics.
Recommended
Designing Data Pipelines for Automous and Trusted Analytics
This document discusses designing data pipelines for autonomous analytics. It notes that up to 80% of analyst time is spent on data preparation and that big data is difficult to adopt, process, and trust. It then presents the need for speed, quality, agility and autonomy in big data projects. The solution proposed is to design for autonomous analytics by automating data discovery, preparation, security, documentation and recommending best actions using machine learning to deliver trusted and timely data.
IoT: How Data Science Driven Software is Eating the Connected World
The document discusses how data science can be used to improve operations in the oil and gas industry through the Internet of Things. Large amounts of sensor data are generated during drilling operations that can be used to build predictive models to optimize drilling and predict equipment failures. Examples of opportunities include using models to predict drill rate of penetration to lower costs and failure prediction to allow for early warning and reduce downtime. The challenges of working with large sensor datasets and building accurate models at scale are also covered.
A Mayo Clinic Big Data Implementation
Brian Brownlow is an experienced senior analyst programmer for Mayo Clinic. He is made a workshop presentation at the 2014 BDPA Technology Conference on the topic, 'Big Data Implementation - Mayo Clinic Case Study'. This presentation will show part of the Mayo Clinic story on the embarking of an exploration of the application of `Big Data' technologies. `Big Data' is seen as one set of tools that can be used to enhance medical research, medical education and practice management. Mayo Clinic is always searching for better, faster and cheaper ways to use its data to improve patient care and sustain financial outcomes in a challenging reimbursement environment. Our approach uses several components that are open source and combines them with data from various sources to provide information to decision makers in near real time. We have created a center of `Big Data' excellence using in-house staff and vendor engagements. `Big Data' is one element of our Enterprise Data Trust framework.
Hadoop in the cloud – The what, why and how from the experts
The document discusses Hadoop in the cloud and its benefits. It summarizes that Hadoop in the cloud provides distributed storage, automated failover, hyper-scaling, distributed computing, and extensibility. It also discusses deploying Hadoop clusters in Azure HDInsight and options for customizing clusters and integrating them.
IT @ Intel: Preparing the Future Enterprise with the Internet of Things
The Internet of Things (IoT) is the concept of diverse machines, devices, and technologies connecting, interacting, and negotiating with each other to help improve and enrich our lives. No longer is this limited to just computer or smart phone technology. Everyday items such as household appliance, cars and even toys can connect to the internet to integrate with other computing things, processes and services. This new paradigm is changing how data is used and collected, and introducing new challenges for enterprises.
Optimizing your Hadoop Infastructure: An Industry Panel Presentation
This document introduces the panel speakers for a discussion on modernizing Hadoop infrastructures. It provides brief biographies of five speakers: Armando Acosta of Dell, who has 15 years of experience in IT solutions and big data; Brandon Draeger of Intel, who manages partnerships between Intel, Cloudera, and their shared ecosystems; TJ Laher of Cloudera, who helps organizations implement Hadoop; Vin Sharma of Intel; and Mark Muncy of Syncsort, who leads technical marketing for big data and has experience in data architecture. The panel will discuss how to evolve Hadoop capabilities to meet emerging customer needs.
Hadoop Summit San Jose 2015: YARN - Past, Present and Future
The document discusses the past, present, and future of Apache Hadoop YARN. It describes how YARN was created to address limitations in MapReduce and provide a more flexible resource management framework. The presentation outlines major releases of YARN from 2010 to 2015, focusing on new features like rolling upgrades, long-running services, node labels, and improved usability tools. It envisions future enhancements such as per-queue scheduling policies, reservations, containerized applications, and improved network and disk isolation.
Moving Health Care Analytics to Hadoop to Build a Better Predictive Model
This document discusses Dignity Health's move to using Hadoop for healthcare analytics to build better predictive models. It outlines their goals of saving costs and lives by leveraging over 30 TB of clinical data using Hadoop and SAS technologies on their Dignity Health Insights platform. The presentation agenda covers Dignity Health, healthcare analytics challenges, their big data ecosystem architecture featuring Hadoop, and how they are using this infrastructure for applications like sepsis surveillance analytics.
Evolution of Big Data at Intel - Crawl, Walk and Run Approach
Intel's big data journey began in 2011 with an evaluation of Hadoop. Since then, Intel has expanded its use of Hadoop and Cloudera across multiple environments. Intel's 3-year roadmap focuses on evolving its Hadoop platform to support more advanced analytics, real-time capabilities, and integrating with traditional BI tools. Key strategies include designing for scalability, following an iterative approach to understand data, and leveraging open source technologies.
Pouring the Foundation: Data Management in the Energy Industry
At CenterPoint Energy, both structured and unstructured data are continuing to grow at a rapid pace. This growth presents many opportunities to deliver business value and many challenges to control costs. To maximize the value of this data while controlling costs, CenterPoint Energy created a data lake using SAP HANA and Hadoop. During this presentation, CenterPoint will discuss their journey of moving smart meter data to Hadoop, how Hadoop is allowing CenterPoint to derive value from big data and their future use case road map.
Driving Enterprise Adoption: Tragedies, Triumphs and Our NEXT
Standard Bank is a leading South African bank with a vision to be the leading financial services organization in and for Africa. We will share our vision, greatest challenges, and most valuable lessons learned on our journey towards enterprise adoption of a big data strategy.
This includes our implementation of: a multi-tenant enterprise data lake, a real time streaming capability, appropriate data management and governance principles, a data science workbench, and a process for model productionisation to support data science teams across the Group and across Africa and Europe.
Speakers
Zakeera Mahomen, Standard Bank, Big Data Practice Lead
Kristel Sampson, Standard Bank, Platform Lead
Extending Data Lake using the Lambda Architecture June 2015
The document discusses using a Lambda architecture to extend a data lake with real-time capabilities. It describes considerations for choosing a real-time architecture and common use cases. Specific examples discussed include using real-time architectures for patient critical care in healthcare and customer engagement in marketing.
Building intelligent applications, experimental ML with Uber’s Data Science W...
In this talk, we will explore how Uber enables rapid experimentation of machine learning models and optimization algorithms through Uber’s Data Science Workbench (DSW). DSW covers a series of stages in data scientists’ workflows including data exploration, feature engineering, machine learning model training, testing, and production deployment. DSW provides interactive notebooks for multiple languages with on-demand resource allocation and the ability to share their works through community features. It also has support for notebooks and intelligent applications backed by Spark job servers. Deep learning applications based on TensorFlow and Torch can be brought into DSW smoothly where resources management is taken care of by the system. The environment in DSW is customizable where users can bring their own libraries and frameworks. Moreover, DSW provides support for Shiny and Python dashboards as well as many other in-house visualization and mapping tools.
In the second part of this talk, we will explore the use cases where custom machine learning models developed in DSW are productionized within the platform. Uber applies machine learning extensively to solve some hard problems. Some use cases include calculating the right price for a ride for over 600 cities and applying NLP technologies to customer feedbacks to offer safe rides and reduce support costs. We will look at various options evaluated for productionizing custom models (server based and serverless). We will also look at how DSW integrates into the larger Uber’s ML ecosystem, model/feature stores, and other ML tools to realize the vision of a complete ML platform for Uber.
Speakers
Adam Hudson, Uber, Senior Software Engineer
Atul Gupte, Uber, Product Manager
Hadoop Trends
1) The document discusses trends in the usage of Apache Hadoop, including the growing adoption of Hadoop to process large amounts of data and enable new data-driven business strategies.
2) Key drivers of Hadoop adoption are the need to analyze more data to gain business insights, cost advantages of Hadoop's ability to use commodity hardware, and Hadoop's ability to handle diverse and rapidly growing unstructured data sources.
3) Emerging trends include keeping raw data for long periods to enable flexible analytics, data-driven development approaches, specialization of data systems with Hadoop handling analytics and reporting, and more organizations piloting Hadoop projects.
Data Science at Speed. At Scale.
Every business is looking for a game-changer in data science, machine learning, and AI. Most organizations are also looking for ways to tap into open-source and commercial data science tools such as Python, RStudio, Apache Spark, Jupyter, and Zeppelin notebooks, to accelerate predictive and machine learning model building and deployment while leveraging the scale, security and governance of the Hortonworks Data Platform and other commercial platforms.
Ana Maria Echeverri will demonstrate how to accelerate data science, machine learning, and deep learning workflows by using IBM Watson Studio, an integrated environment for data scientists, application developers, and subject matter experts. This suite of tools allows to collaboratively connect to data, wrangle that data and use it to build, train and deploy models at scale while using Open Source skills (i.e.: Python) and expanding into cognitive capabilities through access to Watson APIs to build AI-powered applications. If you love Python and want to tap into the power of IBM Watson, this is the session for you.
Software engineering practices for the data science and machine learning life...
With the advent of newer frameworks and toolkits, data scientists are now more productive than ever and starting to prove indispensable to enterprises. Typical organizations have large teams of data scientists who build out key analytics assets that are used on a daily basis and an integral part of live transactions. However, there is also quite a lot of chaos and complexities that get introduced because of the state of the industry. Many packages used by data scientists are from open source, and even if they are well curated, there is a growing tendency to pick out the cutting-edge or unstable packages and frameworks to accelerate analytics. Different data scientists may use different versions of runtimes, different Python or R versions, or even different versions of the same packages. Predominantly data scientists work on their laptops and it becomes difficult to reproduce their environments for use by others. Since data science is now a team sport across multiple personas, involving non-practitioners, traditional application developers, execs, and IT operators, how does an enterprise create a platform for productive cross-role collaboration?
Enterprises need a very reliable and repeatable process, especially when it results in something that affects their production environments. They also require a well managed approach that enables the graduation of an asset from development through a testing and staging process to production. Given the pace of businesses nowadays, the process needs to be quite agile and flexible too—even enabling an easy path to reversing a change. Compliance and audit processes require clear lineage and history as well as approval chains.
In the traditional software engineering world, this lifecycle has been well understood and best practices have been followed for ages. But what does it mean when you have non-programmers or users who are not really trained in software engineering philosophies or who perceive all of this as "big process" roadblocks in their daily work ? How do you we engage them in a productive manner and yet support enterprise requirements for reliability, tracking, and a clear continuous integration and delivery practice? The presenters, in this session, will bring up interesting techniques based on their user research, real life customer interviews, and productized best practices. The presenters also invite the audience to share their stories and best practices to make this a lively conversation.
Speaker
Sriram Srinivasan, Senior Technical Staff Member, Analytics Platform Architect, IBM
How Big Data and Hadoop Integrated into BMC ControlM at CARFAX
Learn how CARFAX utilized the power of Control-M to help drive big data processing via Cloudera. See why it was a no-brainer to choose Control-M to help manage workflows through Hadoop, some of the challenges faced, and the benefits the business received by using an existing, enterprise-wide workload management system instead of choosing “yet another tool.”
Evolving Hadoop into an Operational Platform with Data Applications
The document discusses Cask Data Application Platform (CDAP), an open source platform for building data applications on Hadoop. It provides an overview of CDAP's key components including datasets, programs, and applications. Datasets are standardized containers that encapsulate data access patterns and data models through reusable APIs. Programs are containers for different processing paradigms like batch and real-time. Applications in CDAP compose multiple datasets and programs.
Build Big Data Enterprise Solutions Faster on Azure HDInsight
Hadoop and Spark are big data frameworks used to extract useful span a variety of scenarios from ingestion, data prep, data management, processing, analyzing and visualizing data. Each step requires specialized toolsets to be productive. In this talk I will share solution examples in the Big Data ecosystem such as Cask, StreamSets, Datameer, AtScale, Dataiku on Microsoft’s Azure HDInsight that simplify your Big Data solutions. Azure HDInsight is a cloud Spark and Hadoop service for the enterprise and take advantage of all the benefits of HDInsight giving you the best of both worlds. Join this session for practical information that will enable faster time to insights for you and your business.
The rise of big data governance: insight on this emerging trend from active o...
Each of today’s most forward-thinking enterprises have been forced to face similar data challenges: the reliance on real-time data to better serve their customers and, subsequently, the requirement of complying with regulations to protect that data – one example being the General Data Protection Regulation (GDPR).
The solution to this emerging challenge is a tricky one – for companies like ING, this data governance challenge has been met with metadata, a consistent view across a large heterogeneous ecosystem and collaboration with an active open source community.
This joint presentation, John Mertic – Director of ODPi – and Ferd Scheepers – Global Chief Information Architect of ING – will address the benefits of a vendor-neutral approach to data governance, the need for an open metadata standard, along with insight around how companies ING, IBM, Hortonworks and more are delivering solutions to this challenge as an open source initiative.
Audience Takeaways include:
Understand the role of metadata;
Understand the need for a cross technology view on metadata;
Understand the role of Apache Atlas as a reference implementation; and
Understand the role of ODPi in offering value-added services including certification.
Speaker
John Mertic, Director of Program Management for ODPi, R Consortium, and Open Mainframe Project, The Linux Foundation
Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...
The document discusses re-platforming existing enterprise business intelligence and analytic workloads from platforms like Oracle, Teradata, SAP and IBM to the Hadoop platform. It notes that many existing analytic workloads are struggling with increasing data volumes and are too costly. Hadoop offers a modern distributed platform that can address these issues through the use of a production-grade SQL database like VectorH on Hadoop. The document provides guidelines for re-platforming workloads and notes potential benefits such as improved performance, reduced costs and leveraging the Hadoop ecosystem.
The Practice of Big Data - The Hadoop ecosystem explained with usage scenarios
What's the origin of Big Data? What are the real life usage scenarios where Hadoop has been successfully adopted? How do you get started within your organizations?
BI on Big Data with instant response times at Verizon
As one of the world's largest communications companies, Verizon faced the challenge of analyzing all the video viewership data from its 6+ Million viewers to assess a variety of metrics like viewing patterns, popular channels, popular programs, prime time viewership, VOD / DVR consumption etc. Queries would often take hours or even days to run and more to evaluate, resulting in the company only running the analysis periodically. This led to lag times in Verizon's ability to respond to network, content, customer and company problems. Using Kyvos, Verizon was able to build a BI Consumption Layer that helped them analyze this massive data in its entirety, with the ability to slice and dice and drill down to the lowest levels of granularity, with instantaneous response times. Analysis across mediums, viewers, geographies and diagnostics have enabled Verizon to get faster, deeper insights at a more granular level, and optimize their engagement with partners and customers. ARUN JINDE, Technical Architect, Verizon and AJAY ANAND, VP Products, Kyvos Insights
InfoSphere BigInsights
InfoSphere BigInsights is IBM's distribution of Hadoop that:
- Enhances ease of use and usability for both technical and non-technical users.
- Includes additional tools, technologies, and accelerators to simplify developing and running analytics on Hadoop.
- Aims to help users gain business insights from their data more quickly through an integrated platform.
Create a Smarter Data Lake with HP Haven and Apache Hadoop
An organization’s information is spread across multiple repositories, on-premise and in the cloud, with limited ability to correlate information and derive insights. The Smart Content Hub solution from HP and Hortonworks enables a shared content infrastructure that transparently synchronizes information with existing systems and offers an open standards-based platform for deep analysis and data monetization.
- Leverage 100% of your data: Text, images, audio, video, and many more data types can be automatically consumed and enriched using HP Haven (powered by HP IDOL and HP Vertica), making it possible to integrate this valuable content and insights into various line of business applications.
- Democratize and enable multi-dimensional content analysis: - Empower your analysts, business users, and data scientists to search and analyze Hadoop data with ease, using the 100% open source Hortonworks Data Platform.
- Extend the enterprise data warehouse: Synchronize and manage content from content management systems, and crack open the files in whatever format they happen to be in.
- Dramatically reduce complexity with enterprise-ready SQL engine: Tap into the richest analytics that support JOINs, complex data types, and other capabilities only available with HP Vertica SQL on the Hortonworks Data Platform.
Speakers:
- Ajay Singh, Director, Technical Channels, Hortonworks
- Will Gardella, Product Management, HP Big Data
Leveraging advanced technologies to support critical applications in a secure...
The Sherlock Cloud platform at the San Diego Supercomputer Center (SDSC) has been deploying solutions to support critical applications including, but not limited to, cancer research, decoding human immune system, detecting and preventing medical fraud, enterprise risk management and medical device data management. These applications have relied on technologies ranging from the traditional data warehouse to the more recent big data platforms leveraging Hadoop, Software as a Service (SaaS) capability leveraging container technology, business intelligence and analytics solutions and IOT/streaming solutions. Operating these applications and the underlying platforms requires organizations to be agile and visionary in this arena to grow with the ever-changing technological, regulatory and unique customer requirements, while simultaneously ensuring data security and privacy. As such, during our presentation, we will speak to some key aspects of hosting these applications, data and technology platforms within the highly secure and HIPAA-compliant boundaries of our hybrid cloud that spans an on-premise cloud running at UC San Diego, and another operating in AWS Cloud. We will further identify the challenges and lessons learned from deploying, and operating these capabilities across a hybrid cloud setup.
Speaker
Sandeep Chandra, San Diego Supercomputer Center, Division Director
Breakout: Operational Analytics with Hadoop
Operationalizing models and responding to large volumes of data, fast, requires bolt on systems that can struggle with processing (transforming the data), consistency (always responding to data), and scalability (processing and responding to large volumes of data). If the data volume become too large, these traditional systems fail to deliver their responses resulting in significant losses to organizations. Join this breakout to learn how to overcome the roadblocks.
Designing data pipelines for analytics and machine learning in industrial set...
Machine learning has made it possible for technologists to do amazing things with data. Its arrival coincides with the evolution of networked manufacturing systems driven by IoT. In this presentation we’ll examine the rise of IoT and ML from a practitioners perspective to better understand how applications of AI can be built in industrial settings. We'll walk through a case study that combines multiple IoT and ML technologies to monitor and optimize an industrial heating and cooling HVAC system. Through this instructive example you'll see how the following components can be put into action:
1. A StreamSets data pipeline that sources from MQTT and persists to OpenTSDB
2. A TensorFlow model that predicts anomalies in streaming sensor data
3. A Spark application that derives new event streams for real-time alerts
4. A Grafana dashboard that displays factory sensors and alerts in an interactive view
By walking through this solution step-by-step, you'll learn how to build the fundamental capabilities needed in order to handle endless streams of IoT data and derive ML insights from that data:
1. How to transport IoT data through scalable publish/subscribe event streams
2. How to process data streams with transformations and filters
3. How to persist data streams with the timeliness required for interactive dashboards
4. How to collect labeled datasets for training machine learning models
At the end of this presentation you will have learned how a variety of tools can be used together to build ML enhanced applications and data products for instrumented manufacturing systems.
Speakers
Ian Downard, Sr. Developer Evangelist, MapR
William Ochandarena, Senior Director of Product Management, MapR
Succession planning and potential appraisal
Succession planning is a process to identify, develop, and transition potential successors for key roles aligned with employees' talents and ambitions. It is important for organizational continuity and mission fulfillment if key staff leave. Potential challenges to succession planning include small organization size, lack of resources, funding nature, retaining top employees, poor communication, and inadequate training. Benefits include reduced dependency, readiness for changes, and knowledge sharing. Potential appraisal identifies hidden talents and evaluates employees' potential for higher roles. It informs career development, enhances opportunities, and enables succession programs. The process involves role descriptions, required qualities, indicators, and feedback systems.
Employee Burnout And How to Avoid It
Employee burnout is a state of constant stress. It's not an excuse for not working. It's a real problem that affects employees, leaders and, on a whole, companies.
We live in a world and culture where things, like work, must be done quickly, efficiently, and with little regard to our health. Side effects of such environment are seen in statistics: 72% of people are stressed, 67% consider switching careers and 85% feel like work intrudes their personal life.
There are a lot of ways for managers and leaders to reduce stress levels at work and to help their employees stay happy and productive. Check them out!
More Related Content
What's hot
Evolution of Big Data at Intel - Crawl, Walk and Run Approach
Intel's big data journey began in 2011 with an evaluation of Hadoop. Since then, Intel has expanded its use of Hadoop and Cloudera across multiple environments. Intel's 3-year roadmap focuses on evolving its Hadoop platform to support more advanced analytics, real-time capabilities, and integrating with traditional BI tools. Key strategies include designing for scalability, following an iterative approach to understand data, and leveraging open source technologies.
Pouring the Foundation: Data Management in the Energy Industry
At CenterPoint Energy, both structured and unstructured data are continuing to grow at a rapid pace. This growth presents many opportunities to deliver business value and many challenges to control costs. To maximize the value of this data while controlling costs, CenterPoint Energy created a data lake using SAP HANA and Hadoop. During this presentation, CenterPoint will discuss their journey of moving smart meter data to Hadoop, how Hadoop is allowing CenterPoint to derive value from big data and their future use case road map.
Driving Enterprise Adoption: Tragedies, Triumphs and Our NEXT
Standard Bank is a leading South African bank with a vision to be the leading financial services organization in and for Africa. We will share our vision, greatest challenges, and most valuable lessons learned on our journey towards enterprise adoption of a big data strategy.
This includes our implementation of: a multi-tenant enterprise data lake, a real time streaming capability, appropriate data management and governance principles, a data science workbench, and a process for model productionisation to support data science teams across the Group and across Africa and Europe.
Speakers
Zakeera Mahomen, Standard Bank, Big Data Practice Lead
Kristel Sampson, Standard Bank, Platform Lead
Extending Data Lake using the Lambda Architecture June 2015
The document discusses using a Lambda architecture to extend a data lake with real-time capabilities. It describes considerations for choosing a real-time architecture and common use cases. Specific examples discussed include using real-time architectures for patient critical care in healthcare and customer engagement in marketing.
Building intelligent applications, experimental ML with Uber’s Data Science W...
In this talk, we will explore how Uber enables rapid experimentation of machine learning models and optimization algorithms through Uber’s Data Science Workbench (DSW). DSW covers a series of stages in data scientists’ workflows including data exploration, feature engineering, machine learning model training, testing, and production deployment. DSW provides interactive notebooks for multiple languages with on-demand resource allocation and the ability to share their works through community features. It also has support for notebooks and intelligent applications backed by Spark job servers. Deep learning applications based on TensorFlow and Torch can be brought into DSW smoothly where resources management is taken care of by the system. The environment in DSW is customizable where users can bring their own libraries and frameworks. Moreover, DSW provides support for Shiny and Python dashboards as well as many other in-house visualization and mapping tools.
In the second part of this talk, we will explore the use cases where custom machine learning models developed in DSW are productionized within the platform. Uber applies machine learning extensively to solve some hard problems. Some use cases include calculating the right price for a ride for over 600 cities and applying NLP technologies to customer feedbacks to offer safe rides and reduce support costs. We will look at various options evaluated for productionizing custom models (server based and serverless). We will also look at how DSW integrates into the larger Uber’s ML ecosystem, model/feature stores, and other ML tools to realize the vision of a complete ML platform for Uber.
Speakers
Adam Hudson, Uber, Senior Software Engineer
Atul Gupte, Uber, Product Manager
Hadoop Trends
1) The document discusses trends in the usage of Apache Hadoop, including the growing adoption of Hadoop to process large amounts of data and enable new data-driven business strategies.
2) Key drivers of Hadoop adoption are the need to analyze more data to gain business insights, cost advantages of Hadoop's ability to use commodity hardware, and Hadoop's ability to handle diverse and rapidly growing unstructured data sources.
3) Emerging trends include keeping raw data for long periods to enable flexible analytics, data-driven development approaches, specialization of data systems with Hadoop handling analytics and reporting, and more organizations piloting Hadoop projects.
Data Science at Speed. At Scale.
Every business is looking for a game-changer in data science, machine learning, and AI. Most organizations are also looking for ways to tap into open-source and commercial data science tools such as Python, RStudio, Apache Spark, Jupyter, and Zeppelin notebooks, to accelerate predictive and machine learning model building and deployment while leveraging the scale, security and governance of the Hortonworks Data Platform and other commercial platforms.
Ana Maria Echeverri will demonstrate how to accelerate data science, machine learning, and deep learning workflows by using IBM Watson Studio, an integrated environment for data scientists, application developers, and subject matter experts. This suite of tools allows to collaboratively connect to data, wrangle that data and use it to build, train and deploy models at scale while using Open Source skills (i.e.: Python) and expanding into cognitive capabilities through access to Watson APIs to build AI-powered applications. If you love Python and want to tap into the power of IBM Watson, this is the session for you.
Software engineering practices for the data science and machine learning life...
With the advent of newer frameworks and toolkits, data scientists are now more productive than ever and starting to prove indispensable to enterprises. Typical organizations have large teams of data scientists who build out key analytics assets that are used on a daily basis and an integral part of live transactions. However, there is also quite a lot of chaos and complexities that get introduced because of the state of the industry. Many packages used by data scientists are from open source, and even if they are well curated, there is a growing tendency to pick out the cutting-edge or unstable packages and frameworks to accelerate analytics. Different data scientists may use different versions of runtimes, different Python or R versions, or even different versions of the same packages. Predominantly data scientists work on their laptops and it becomes difficult to reproduce their environments for use by others. Since data science is now a team sport across multiple personas, involving non-practitioners, traditional application developers, execs, and IT operators, how does an enterprise create a platform for productive cross-role collaboration?
Enterprises need a very reliable and repeatable process, especially when it results in something that affects their production environments. They also require a well managed approach that enables the graduation of an asset from development through a testing and staging process to production. Given the pace of businesses nowadays, the process needs to be quite agile and flexible too—even enabling an easy path to reversing a change. Compliance and audit processes require clear lineage and history as well as approval chains.
In the traditional software engineering world, this lifecycle has been well understood and best practices have been followed for ages. But what does it mean when you have non-programmers or users who are not really trained in software engineering philosophies or who perceive all of this as "big process" roadblocks in their daily work ? How do you we engage them in a productive manner and yet support enterprise requirements for reliability, tracking, and a clear continuous integration and delivery practice? The presenters, in this session, will bring up interesting techniques based on their user research, real life customer interviews, and productized best practices. The presenters also invite the audience to share their stories and best practices to make this a lively conversation.
Speaker
Sriram Srinivasan, Senior Technical Staff Member, Analytics Platform Architect, IBM
How Big Data and Hadoop Integrated into BMC ControlM at CARFAX
Learn how CARFAX utilized the power of Control-M to help drive big data processing via Cloudera. See why it was a no-brainer to choose Control-M to help manage workflows through Hadoop, some of the challenges faced, and the benefits the business received by using an existing, enterprise-wide workload management system instead of choosing “yet another tool.”
Evolving Hadoop into an Operational Platform with Data Applications
The document discusses Cask Data Application Platform (CDAP), an open source platform for building data applications on Hadoop. It provides an overview of CDAP's key components including datasets, programs, and applications. Datasets are standardized containers that encapsulate data access patterns and data models through reusable APIs. Programs are containers for different processing paradigms like batch and real-time. Applications in CDAP compose multiple datasets and programs.
Build Big Data Enterprise Solutions Faster on Azure HDInsight
Hadoop and Spark are big data frameworks used to extract useful span a variety of scenarios from ingestion, data prep, data management, processing, analyzing and visualizing data. Each step requires specialized toolsets to be productive. In this talk I will share solution examples in the Big Data ecosystem such as Cask, StreamSets, Datameer, AtScale, Dataiku on Microsoft’s Azure HDInsight that simplify your Big Data solutions. Azure HDInsight is a cloud Spark and Hadoop service for the enterprise and take advantage of all the benefits of HDInsight giving you the best of both worlds. Join this session for practical information that will enable faster time to insights for you and your business.
The rise of big data governance: insight on this emerging trend from active o...
Each of today’s most forward-thinking enterprises have been forced to face similar data challenges: the reliance on real-time data to better serve their customers and, subsequently, the requirement of complying with regulations to protect that data – one example being the General Data Protection Regulation (GDPR).
The solution to this emerging challenge is a tricky one – for companies like ING, this data governance challenge has been met with metadata, a consistent view across a large heterogeneous ecosystem and collaboration with an active open source community.
This joint presentation, John Mertic – Director of ODPi – and Ferd Scheepers – Global Chief Information Architect of ING – will address the benefits of a vendor-neutral approach to data governance, the need for an open metadata standard, along with insight around how companies ING, IBM, Hortonworks and more are delivering solutions to this challenge as an open source initiative.
Audience Takeaways include:
Understand the role of metadata;
Understand the need for a cross technology view on metadata;
Understand the role of Apache Atlas as a reference implementation; and
Understand the role of ODPi in offering value-added services including certification.
Speaker
John Mertic, Director of Program Management for ODPi, R Consortium, and Open Mainframe Project, The Linux Foundation
Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...
The document discusses re-platforming existing enterprise business intelligence and analytic workloads from platforms like Oracle, Teradata, SAP and IBM to the Hadoop platform. It notes that many existing analytic workloads are struggling with increasing data volumes and are too costly. Hadoop offers a modern distributed platform that can address these issues through the use of a production-grade SQL database like VectorH on Hadoop. The document provides guidelines for re-platforming workloads and notes potential benefits such as improved performance, reduced costs and leveraging the Hadoop ecosystem.
The Practice of Big Data - The Hadoop ecosystem explained with usage scenarios
What's the origin of Big Data? What are the real life usage scenarios where Hadoop has been successfully adopted? How do you get started within your organizations?
BI on Big Data with instant response times at Verizon
As one of the world's largest communications companies, Verizon faced the challenge of analyzing all the video viewership data from its 6+ Million viewers to assess a variety of metrics like viewing patterns, popular channels, popular programs, prime time viewership, VOD / DVR consumption etc. Queries would often take hours or even days to run and more to evaluate, resulting in the company only running the analysis periodically. This led to lag times in Verizon's ability to respond to network, content, customer and company problems. Using Kyvos, Verizon was able to build a BI Consumption Layer that helped them analyze this massive data in its entirety, with the ability to slice and dice and drill down to the lowest levels of granularity, with instantaneous response times. Analysis across mediums, viewers, geographies and diagnostics have enabled Verizon to get faster, deeper insights at a more granular level, and optimize their engagement with partners and customers. ARUN JINDE, Technical Architect, Verizon and AJAY ANAND, VP Products, Kyvos Insights
InfoSphere BigInsights
InfoSphere BigInsights is IBM's distribution of Hadoop that:
- Enhances ease of use and usability for both technical and non-technical users.
- Includes additional tools, technologies, and accelerators to simplify developing and running analytics on Hadoop.
- Aims to help users gain business insights from their data more quickly through an integrated platform.
Create a Smarter Data Lake with HP Haven and Apache Hadoop
An organization’s information is spread across multiple repositories, on-premise and in the cloud, with limited ability to correlate information and derive insights. The Smart Content Hub solution from HP and Hortonworks enables a shared content infrastructure that transparently synchronizes information with existing systems and offers an open standards-based platform for deep analysis and data monetization.
- Leverage 100% of your data: Text, images, audio, video, and many more data types can be automatically consumed and enriched using HP Haven (powered by HP IDOL and HP Vertica), making it possible to integrate this valuable content and insights into various line of business applications.
- Democratize and enable multi-dimensional content analysis: - Empower your analysts, business users, and data scientists to search and analyze Hadoop data with ease, using the 100% open source Hortonworks Data Platform.
- Extend the enterprise data warehouse: Synchronize and manage content from content management systems, and crack open the files in whatever format they happen to be in.
- Dramatically reduce complexity with enterprise-ready SQL engine: Tap into the richest analytics that support JOINs, complex data types, and other capabilities only available with HP Vertica SQL on the Hortonworks Data Platform.
Speakers:
- Ajay Singh, Director, Technical Channels, Hortonworks
- Will Gardella, Product Management, HP Big Data
Leveraging advanced technologies to support critical applications in a secure...
The Sherlock Cloud platform at the San Diego Supercomputer Center (SDSC) has been deploying solutions to support critical applications including, but not limited to, cancer research, decoding human immune system, detecting and preventing medical fraud, enterprise risk management and medical device data management. These applications have relied on technologies ranging from the traditional data warehouse to the more recent big data platforms leveraging Hadoop, Software as a Service (SaaS) capability leveraging container technology, business intelligence and analytics solutions and IOT/streaming solutions. Operating these applications and the underlying platforms requires organizations to be agile and visionary in this arena to grow with the ever-changing technological, regulatory and unique customer requirements, while simultaneously ensuring data security and privacy. As such, during our presentation, we will speak to some key aspects of hosting these applications, data and technology platforms within the highly secure and HIPAA-compliant boundaries of our hybrid cloud that spans an on-premise cloud running at UC San Diego, and another operating in AWS Cloud. We will further identify the challenges and lessons learned from deploying, and operating these capabilities across a hybrid cloud setup.
Speaker
Sandeep Chandra, San Diego Supercomputer Center, Division Director
Breakout: Operational Analytics with Hadoop
Operationalizing models and responding to large volumes of data, fast, requires bolt on systems that can struggle with processing (transforming the data), consistency (always responding to data), and scalability (processing and responding to large volumes of data). If the data volume become too large, these traditional systems fail to deliver their responses resulting in significant losses to organizations. Join this breakout to learn how to overcome the roadblocks.
Designing data pipelines for analytics and machine learning in industrial set...
Machine learning has made it possible for technologists to do amazing things with data. Its arrival coincides with the evolution of networked manufacturing systems driven by IoT. In this presentation we’ll examine the rise of IoT and ML from a practitioners perspective to better understand how applications of AI can be built in industrial settings. We'll walk through a case study that combines multiple IoT and ML technologies to monitor and optimize an industrial heating and cooling HVAC system. Through this instructive example you'll see how the following components can be put into action:
1. A StreamSets data pipeline that sources from MQTT and persists to OpenTSDB
2. A TensorFlow model that predicts anomalies in streaming sensor data
3. A Spark application that derives new event streams for real-time alerts
4. A Grafana dashboard that displays factory sensors and alerts in an interactive view
By walking through this solution step-by-step, you'll learn how to build the fundamental capabilities needed in order to handle endless streams of IoT data and derive ML insights from that data:
1. How to transport IoT data through scalable publish/subscribe event streams
2. How to process data streams with transformations and filters
3. How to persist data streams with the timeliness required for interactive dashboards
4. How to collect labeled datasets for training machine learning models
At the end of this presentation you will have learned how a variety of tools can be used together to build ML enhanced applications and data products for instrumented manufacturing systems.
Speakers
Ian Downard, Sr. Developer Evangelist, MapR
William Ochandarena, Senior Director of Product Management, MapR
What's hot (20)
Evolution of Big Data at Intel - Crawl, Walk and Run Approach
Evolution of Big Data at Intel - Crawl, Walk and Run Approach
Pouring the Foundation: Data Management in the Energy Industry
Pouring the Foundation: Data Management in the Energy Industry
Driving Enterprise Adoption: Tragedies, Triumphs and Our NEXT
Driving Enterprise Adoption: Tragedies, Triumphs and Our NEXT
Extending Data Lake using the Lambda Architecture June 2015
Extending Data Lake using the Lambda Architecture June 2015
Building intelligent applications, experimental ML with Uber’s Data Science W...
Building intelligent applications, experimental ML with Uber’s Data Science W...
Software engineering practices for the data science and machine learning life...
Software engineering practices for the data science and machine learning life...
How Big Data and Hadoop Integrated into BMC ControlM at CARFAX
How Big Data and Hadoop Integrated into BMC ControlM at CARFAX
Evolving Hadoop into an Operational Platform with Data Applications
Evolving Hadoop into an Operational Platform with Data Applications
Build Big Data Enterprise Solutions Faster on Azure HDInsight
Build Big Data Enterprise Solutions Faster on Azure HDInsight
The rise of big data governance: insight on this emerging trend from active o...
The rise of big data governance: insight on this emerging trend from active o...
Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...
Can you Re-Platform your Teradata, Oracle, Netezza and SQL Server Analytic Wo...
The Practice of Big Data - The Hadoop ecosystem explained with usage scenarios
The Practice of Big Data - The Hadoop ecosystem explained with usage scenarios
BI on Big Data with instant response times at Verizon
BI on Big Data with instant response times at Verizon
Create a Smarter Data Lake with HP Haven and Apache Hadoop
Create a Smarter Data Lake with HP Haven and Apache Hadoop
Leveraging advanced technologies to support critical applications in a secure...
Leveraging advanced technologies to support critical applications in a secure...
Designing data pipelines for analytics and machine learning in industrial set...
Designing data pipelines for analytics and machine learning in industrial set...
Viewers also liked
Succession planning and potential appraisal
Succession planning is a process to identify, develop, and transition potential successors for key roles aligned with employees' talents and ambitions. It is important for organizational continuity and mission fulfillment if key staff leave. Potential challenges to succession planning include small organization size, lack of resources, funding nature, retaining top employees, poor communication, and inadequate training. Benefits include reduced dependency, readiness for changes, and knowledge sharing. Potential appraisal identifies hidden talents and evaluates employees' potential for higher roles. It informs career development, enhances opportunities, and enables succession programs. The process involves role descriptions, required qualities, indicators, and feedback systems.
Employee Burnout And How to Avoid It
Employee burnout is a state of constant stress. It's not an excuse for not working. It's a real problem that affects employees, leaders and, on a whole, companies.
We live in a world and culture where things, like work, must be done quickly, efficiently, and with little regard to our health. Side effects of such environment are seen in statistics: 72% of people are stressed, 67% consider switching careers and 85% feel like work intrudes their personal life.
There are a lot of ways for managers and leaders to reduce stress levels at work and to help their employees stay happy and productive. Check them out!
Trends In Attracting and Retaining Top Talent
The document discusses turnover costs for different types of employees. It finds that replacing low-wage, high-turnover employees costs 16% of their annual salary on average. Replacing mid-range salary employees between $30,000-$50,000 costs 20% of their annual salary. Highly educated executive replacements can cost up to 213% of their $100k annual salary.
Breaking Burnout: Understanding & Preventing Employee Burnout
This document discusses burnout, including risk factors, impacts, identification, prevention and mitigation. It notes that burnout is characterized by exhaustion, lack of motivation and reduced work effectiveness. Jobs with high burnout rates include physicians, nurses, social workers, teachers and police officers. The document outlines causes of burnout like unclear requirements and high stress without breaks. It provides steps to prevent and combat burnout on individual and organizational levels through coping strategies and addressing work conditions.
Social Talent Management Research
The document discusses social talent and the use of social networks for professional purposes. It provides statistics on InnoCentive such as the number of registered solvers and solution submissions. It also discusses how companies can leverage external talent and social networks to source innovation. Additionally, it presents data on employee engagement, how well managers understand their employees' skills and careers, and challenges in performance management and retaining top talent.
Succession Planning
The document discusses succession planning for a NAPA auto parts store owner. It recommends starting succession planning early, developing a strategy that may involve transferring the business to a family member, key employee, or outside buyer. The owner should prepare the business for transition by keeping it profitable and maintaining strong financial records to demonstrate growth over time. Various equity plans are described that can help develop a key employee as a potential successor or new owner through restricted stock, stock appreciation rights, or phantom equity agreements.
Business Case For Succession Planning
Ensuring the whole management team of a firm identify the impact of properly crafted succession plan: The effects of its presence and absence on business continuity and competitiveness.
Common EMAIL MARKETING Mistakes One Should Avoid
This document outlines common mistakes to avoid in email marketing. It discusses not having a long-term strategy, creating campaigns that come across as spam, not testing emails before sending them to recipients, sending unsolicited messages, underestimating the importance of the subject and sender, mishandling unsubscribes, not optimizing for mobile, broken dynamic content, a one-size-fits-all approach, buying email lists, and not sending emails at all. Avoiding these pitfalls can help improve email marketing effectiveness and deliverability.
Talent Management
Request Talent Management Demo http://www.PeopleStreme.com
Know your people, find your people, keep your people. PeopleStreme's Talent Management Generation3 is innovative talent management software that enables organisations to successfully manage their talent, set up career development plans, improve the succession planning process and drive employee retention rather than respond to unexpected resignations.
Hay group -_how_to_stop_your_talent_taking_flight_guide
This document discusses how organizations can retain their talent as job markets improve globally. It identifies five key factors that drive employee commitment: 1) Confidence in leadership and direction, 2) Room for growth, 3) A fair exchange of compensation for contributions, 4) An enabling environment for success, and 5) Employee authority and influence. The document provides analysis showing global employee turnover is expected to increase sharply starting in 2014 as economies strengthen. It then offers specific strategies organizations can implement to address each of the five retention factors and secure the long-term commitment of their workforce.
EMPLOYEE BURNOUT:A VIEW INTO EDUCATION SECTOR
This paper was accepted for inabsentia presentation and subsequent publication as conference proceedings bearing ISBN, in SIMSARC 12 (Symbiosis Institute of Management Studies Annual Research Conference) held during 13-14 December, 2012.
About Empire 2011
The document provides an overview of Empire Office, a furniture dealership company. It discusses Empire Office's 64-year history, the services it offers to clients, its project management and installation processes, employee locations across the US and internationally, completed projects for national clients, its network of vendors, organizational chart, environmental policy, and references.
Talent Management and Succession Planning Models
This document discusses integrating assessment data from ProfileXT and CheckPoint into succession planning and talent management. It includes examples of development grids that classify employees based on their potential and performance, and how to provide different types of development and feedback to employees in each classification. The goal is to actively develop high potential employees for future leadership roles and close any gaps in the leadership pipeline.
Talent Pipeline: Ensuring the Right Flow of Talent for Your Organization | Da...
When it comes to creating a sustainable Talent pipeline, ask yourself this: do you have the right tools you need to ensure that your organisation has the right flow of Talent to support and grow your business?
If you’re still scrambling to fill key positions quickly when an employee leaves or retires, a lack of preparation could significantly impact your bottom-line in the long-run.
Download the free eBook to find out how you can always stay on top of your recruitment needs: https://hrboss.com/whitepapers/talent-pipeline-ensuring-right-flow-talent-your-organisation
Employee Burnout - Symptoms, Causes, and Preventive & Counter Measures
Employee Burnout is an issue which is now affecting more than 90% of the businesses across the world. It is very difficult to understand and measure the depth of its affect as there are so many kinds of industries. But, slowly and steadily it has managed to penetrate every industry and show its symptoms. Here in this presentation, I have listed down the Symptoms, Causes and Preventive / Counter measures of Employee Burnout. Hope it helps you to identify and prevent burnouts in your organization.
Retention: Stop Talent From Taking Flight
Is your organisation struggling with the following issues?
1. High employee turnover rates
2. Poor business performance due to loss of key employees
3. Increased hiring and training costs
4. Sudden manpower crunch caused by a significant number of employees leaving
Download the report to find out how to keep Talent from walking out the door and nip the root of the problem before it's too late: https://hrboss.com/whitepapers/talent-retention-how-keep-talent-walking-out-door
Succession planning in HRP
This document discusses various aspects of human resource planning such as career planning and development, succession planning, and potential appraisal and development. It provides details on the meaning and process of career planning, management, and development. It also outlines the need for career planning, steps involved in establishing a career development system, advantages for individuals and organizations, and limitations. Succession planning and potential appraisal are also defined along with best practices.
Succession Planning
The document provides an overview of a webinar from CLC Human Resources on best practices for high-impact succession management, including frameworks for assessing vacancy risk, readiness risk, transition risk, and portfolio risk. It summarizes CLC's research on managing the four succession risks and translating them into strategic imperatives to safeguard critical capabilities, accelerate development, overcome executive derailment, and maximize talent leverage. The webinar outlines CLC's approach to moving from traditional succession planning to strategic executive talent management.
Burnout Presentation
The presentation discusses burnout in athletic trainers and students. It defines burnout as a reaction to chronic stress involving negative interactions between environmental and personal factors. Burnout is characterized by three responses: depersonalization, emotional exhaustion, and lack of personal accomplishment. The presentation outlines common stressors for athletic trainers and students and stages of burnout. It also reports on studies finding rates of burnout among these groups and effective coping strategies to avoid burnout.
9 box model with titles
The document discusses the 9 box model for classifying employees based on their performance and potential. The 9 box model assesses people on two dimensions: demonstrated performance on their current role and their long-term potential. It places employees into 9 categories within a grid based on these dimensions to help identify high potentials, develop employees, and ensure they are placed in roles that suit their skills. Each category is described in terms of the typical employee attributes and recommended actions for development or placement. The overall purpose is to accurately assess individuals to match their development plans to maximize their contribution to the organization.
Viewers also liked (20)
Breaking Burnout: Understanding & Preventing Employee Burnout
Breaking Burnout: Understanding & Preventing Employee Burnout
Hay group -_how_to_stop_your_talent_taking_flight_guide
Hay group -_how_to_stop_your_talent_taking_flight_guide
Talent Pipeline: Ensuring the Right Flow of Talent for Your Organization | Da...
Talent Pipeline: Ensuring the Right Flow of Talent for Your Organization | Da...
Employee Burnout - Symptoms, Causes, and Preventive & Counter Measures
Employee Burnout - Symptoms, Causes, and Preventive & Counter Measures
Similar to Data Science: Driving Smarter Finance and Workforce Decsions for the Enterprise
Situation Awareness In A Complex World
The document discusses situation awareness and knowledge management approaches for complex and dynamic systems. It proposes modeling different elements of the system and information processing requirements. This includes entity, activity, representation, measurement, observation and event models. It also discusses roles, tasks, outcomes and guidance for stakeholders involved in situation awareness through a Situation Awareness Unified Process. Design decisions and views of the proposed situation awareness system architecture are also covered at a high level.
Mohammed AL Madhani
Setting up a Big Data Team requires best practices including building a team with diverse skills in areas like math, computer science, statistics and domain expertise. Data scientists fulfill key roles like generating prototypes to demonstrate ideas, decomposing problems, and communicating with stakeholders. Effective teams require data preparation, using tools like statistical systems and data management systems. Certification can increase a team's maturity, while frameworks like CRISP-DM and Six Sigma's DMAIC provide processes to optimize the data workflow. Talent management is also important to support the team over time.
Operational Risk Management Data Validation Architecture
This describes a structured approach to validating data used to construct and use an operational risk model. It details an integrated approach to operational risk data involving three components:
1. Using the Open Group FAIR (Factor Analysis of Information Risk) risk taxonomy to create a risk data model that reflects the required data needed to assess operational risk
2. Using the DMBOK model to define a risk data capability framework to assess the quality and accuracy of risk data
3. Applying standard fault analysis approaches - Fault Tree Analysis (FTA) and Failure Mode and Effect Analysis (FMEA) - to the risk data capability framework to understand the possible causes of risk data failures within the risk model definition, operation and use
Machine Learning in Autonomous Data Warehouse
Machine Learning in Autonomous Data Warehouse: One can use Oracle Autonomous Data Warehouse for machine learning. There are several ways to do this. This presentation explores these different but related options for performing machine learning. Each of these options enables people with different backgrounds to engage with building machine learning solutions on their data. At the end of the session, you will know which option will work best for you
This is from the Bay area Cloud Computing event https://www.meetup.com/All-Things-Cloud-Computing-Bay-Area/events/271017950/
Big data and Process Safety
Presented at the conference for Big Data and Process Safety in Frankfurt am Main organized by CCPS and EPSC.
Introduction to Machine Learning and Data Science using the Autonomous databa...
This session will focus on basics of what Machine Learning is, different types of Machine Learning and Neural Networks, supervised and unsupervised machine learning, AutoML for training models and this ends with an example of how to predict workloads using Average Active sessions and different algorithms as an example and also how to predict maintenance windows for your databases. We will also use many examples from the ADW Oracle Autonomous Database offering, Oracle Machine Learning library to make this a session with lots of code examples in addition to the theory of Machine Learning and you will walk out having a definitive path to being a data scientist
Introduction to Machine Learning and Data Science using Autonomous Database ...
This session will focus on basics of what Machine Learning is , different types of Machine Learning and Neural Networks , supervised and unsupervised machine learning , autoML for training models and this ends with an example of how to predict workloads using Average Active sessions and different algorithms as an example and also how to predict maintenance windows for your databases. We will also use different open source frameworks as well as some of the tools in the Autonomous Database cloud to do this. If you are a DBA and want to learn something about machine learning and use the tools to perform your tasks more efficiently and automaticall
Applications of Semantic Technology in the Real World Today
Amit Sheth, "Applications of Semantic Technology in the Real World Today," talk given at Semantic Technology Conference, San Jose, CA, March 2005.
This talk reviews real-world applications mainly deployed in financial services industry developed over Semagix Freedom platform described in http://knoesis.org/library/resource.php?id=810 . Technology is based on this patent: "Semantic web and its applications in browsing, searching, profiling, personalization and advertising", http://knoesis.org/library/resource.php?id=843 .
Amit Sheth founded Taalee in 1999, which merged with Voquette in 2002, and then with Semagix in 2004.
Sage Solutions Brief.Mjo
The document discusses a national continuity solutions platform that provides:
1) Continuous assessment and monitoring of organizational policies, assets, configurations, controls, risks, and other elements through a purpose-built GRC platform.
2) Integrated applications that support compliance reporting, policy management, risk monitoring, and other functions through a common data model and open connector architecture.
3) Customization and integration capabilities to help customers in sectors like defense, intelligence, and federal agencies manage compliance and risk through automated and continuous processes rather than manual ones.
Is Your Staff Big Data Ready? 5 Things to Know About What It Will Take to Suc...
This document discusses the importance of big data readiness for organizations and assessing an organization's current data skills and capabilities. It finds that most executives agree that harnessing all organizational data would strengthen their business. It then provides a framework for data mapping and discusses that effective data management requires a team approach. It also outlines the variety of skills needed, from IT to business skills, and that organizations plan to meet skills needs through training, hiring and contracting. There is also a projected shortage of big data workers due to increased demand.
Procedural Risk Management
The most common consequences of ineffective or no procedural risk management are the excessive time spent by managers in dealing with unanticipated difficulties.
Bio IT World 2019 - AI For Healthcare - Simon Taylor, Lucidworks
1) An AI system implemented at Johns Hopkins Hospital helped optimize hospital operations and bed assignment. It allowed beds to be assigned 30% faster.
2) This reduced the need to keep surgery patients in recovery rooms longer than necessary by 80% and cut wait times for ER patients to receive beds by 20%.
3) The efficiencies also allowed the hospital to accept 60% more transfer patients from other hospitals.
MLOps - Build pipelines with Tensor Flow Extended & Kubeflow
Introduction to Machine Learning Operations (MLOps) - Build pipelines with Tensor Flow Extended & Kubeflow
-linkedin
This document summarizes a capstone project on automated data science. It discusses the data science pipeline, which includes preparation, analysis, and integration phases. Activities within the preparation phase like data cleansing, dimension reduction, correlation analysis, and feature synthesis have varying levels of automation maturity. Algorithms and techniques for automating tasks in each phase are presented. The document also examines challenges of incorporating automated data science like change management and skills transition. Finally, it discusses vendor capabilities in different phases and factors to consider for in-house vs outsourced integration solutions.
Presentation at Bio IT World West: To AI or Not to AI, Presented by Simon Tay...
Breakout presentation in Machine Learning Track at Bio IT World West 2019 / Molecular Med Tri-Conference. Presented by Lucidworks VP Global Partners and Alliances, Simon Taylor.
Intro to ai application emeritus uob-final
This document provides an overview of an introductory webinar on AI concepts and terminologies. The webinar agenda includes discussing concepts of AI learning algorithms, understanding the 5 spokes framework of AI, and a team exercise. Key concepts covered are the 7 stages of building intelligence from data, examples of common learning algorithms like neural networks, and the 5 components of the 5 spokes AI framework: sensing, perception, communication, decision making, and interaction. Real-world applications of each component are also examined, such as computer vision, natural language processing, time series analysis, and reinforcement learning.
Machine Learning: Addressing the Disillusionment to Bring Actual Business Ben...
'Machine learning’ is one of those cringy phrases, almost (if not already) taboo in the world of high-tech SaaS. Applying true machine learning to an organization’s product(s), however, can have real benefit for the business, its clients, and the industry as a whole. From credit card fraud investigations to the way that a car is built, machine learning has permeated our everyday life without a common understanding of what it is and how to implement it.
Bigdataanalytics
This document provides an overview of big data analytics and discusses related concepts and tools. It describes challenges of big data such as increased data volume, velocity and variety. It introduces the Hadoop platform and tools like HDFS, Hive and Spark for storing and analyzing large datasets. Different types of analytics including descriptive, predictive and sentiment analysis are covered. The document also outlines the analytics lifecycle and provides an example use case of sentiment analysis on Twitter data.
KIT-601 Lecture Notes-UNIT-1.pdf
This document provides an introduction to the concepts of data analytics and the data analytics lifecycle. It discusses big data in terms of the 4Vs - volume, velocity, variety and veracity. It also discusses other characteristics of big data like volatility, validity, variability and value. The document then discusses various concepts in data analytics like traditional business intelligence, data mining, statistical applications, predictive analysis, and data modeling. It explains how these concepts are used to analyze large datasets and derive value from big data. The goal of data analytics is to gain insights and a competitive advantage through analyzing large and diverse datasets.
Similar to Data Science: Driving Smarter Finance and Workforce Decsions for the Enterprise (20)
Operational Risk Management Data Validation Architecture
Operational Risk Management Data Validation Architecture
Introduction to Machine Learning and Data Science using the Autonomous databa...
Introduction to Machine Learning and Data Science using the Autonomous databa...
Introduction to Machine Learning and Data Science using Autonomous Database ...
Introduction to Machine Learning and Data Science using Autonomous Database ...
Applications of Semantic Technology in the Real World Today
Applications of Semantic Technology in the Real World Today
Is Your Staff Big Data Ready? 5 Things to Know About What It Will Take to Suc...
Is Your Staff Big Data Ready? 5 Things to Know About What It Will Take to Suc...
Bio IT World 2019 - AI For Healthcare - Simon Taylor, Lucidworks
Bio IT World 2019 - AI For Healthcare - Simon Taylor, Lucidworks
MLOps - Build pipelines with Tensor Flow Extended & Kubeflow
MLOps - Build pipelines with Tensor Flow Extended & Kubeflow
Presentation at Bio IT World West: To AI or Not to AI, Presented by Simon Tay...
Presentation at Bio IT World West: To AI or Not to AI, Presented by Simon Tay...
Machine Learning: Addressing the Disillusionment to Bring Actual Business Ben...
Machine Learning: Addressing the Disillusionment to Bring Actual Business Ben...
More from DataWorks Summit
Data Science Crash Course
Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL).
Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW).
Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models.
Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.
Floating on a RAFT: HBase Durability with Apache Ratis
In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort.
This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase.
Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs.
Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables.
Resources:
https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html
https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html
https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html
HBase Tales From the Trenches - Short stories about most common HBase operati...
Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.
Managing the Dewey Decimal System
OCLC has been using HBase since 2012 to enable single-search-box access to over a billion items from your library and the world’s library collection. This talk will provide an overview of how HBase is structured to provide this information and some of the challenges they have encountered to scale to support the world catalog and how they have overcome them.
Practical NoSQL: Accumulo's dirlist Example
Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL.
Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist).
In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.
HBase Global Indexing to support large-scale data ingestion at Uber
Danny Chen presented on Uber's use of HBase for global indexing to support large-scale data ingestion. Uber uses HBase to provide a global view of datasets ingested from Kafka and other data sources. To generate indexes, Spark jobs are used to transform data into HFiles, which are loaded into HBase tables. Given the large volumes of data, techniques like throttling HBase access and explicit serialization are used. The global indexing solution supports requirements for high throughput, strong consistency and horizontal scalability across Uber's data lake.
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions.
These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
This document discusses using Apache NiFi to build a high-speed cyber security data pipeline. It outlines the challenges of ingesting, transforming, and routing large volumes of security data from various sources to stakeholders like security operations centers, data scientists, and executives. It proposes using NiFi as a centralized data gateway to ingest data from multiple sources using a single entry point, transform the data according to destination needs, and reliably deliver the data while avoiding issues like network traffic and data duplication. The document provides an example NiFi flow and discusses metrics from processing over 20 billion events through 100+ production flows and 1000+ transformations.
Supporting Apache HBase : Troubleshooting and Supportability Improvements
This document discusses supporting Apache HBase and improving troubleshooting and supportability. It introduces two Cloudera employees who work on HBase support and provides an overview of typical troubleshooting scenarios for HBase like performance degradation, process crashes, and inconsistencies. The agenda covers using existing tools like logs and metrics to troubleshoot HBase performance issues with a general approach, and introduces htop as a real-time monitoring tool for HBase.
Security Framework for Multitenant Architecture
In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.
With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.
Extending Twitter's Data Platform to Google Cloud
Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.
Computer Vision: Coming to a Store Near You
Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as:
● Optimizing merchandising execution, in-stocks and sell-thru
● Enhancing operational efficiencies, enable real-time customer engagement
● Enhancing loss prevention capabilities, response time
● Creating frictionless experiences for shoppers
Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry.
We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey.
Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables.
We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance.
We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing.
Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems.
By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.
More from DataWorks Summit (20)
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
HBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at Uber
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything Engine
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Recently uploaded
Northern Engraving | Nameplate Manufacturing Process - 2024
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Gursev Pirge, PhD
Senior Data Scientist - JohnSnowLabs
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
Session 1 - Intro to Robotic Process Automation.pdf
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
Mutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Java is for many years one of the most popular programming languages, but it used to have hard times in the Serverless community. Java is known for its high cold start times and high memory footprint, comparing to other programming languages like Node.js and Python. In this talk I'll look at the general best practices and techniques we can use to decrease memory consumption, cold start times for Java Serverless development on AWS including GraalVM (Native Image) and AWS own offering SnapStart based on Firecracker microVM snapshot and restore and CRaC (Coordinated Restore at Checkpoint) runtime hooks. I'll also provide a lot of benchmarking on Lambda functions trying out various deployment package sizes, Lambda memory settings, Java compilation options and HTTP (a)synchronous clients and measure their impact on cold and warm start times.
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Pitangent Analytics & Technology Solutions Pvt. Ltd
Discover top-tier mobile app development services, offering innovative solutions for iOS and Android. Enhance your business with custom, user-friendly mobile applications.Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.Must Know Postgres Extension for DBA and Developer during Migration
Mydbops Opensource Database Meetup 16
Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration
Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting
Date & Time: 8th June | 10 AM - 1 PM IST
Venue: Bangalore International Centre, Bangalore
Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle.
Key Takeaways:
* Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities.
* Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom.
* Discover how these key extensions can empower both developers and DBAs during the migration process.
* Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: info@mydbops.com
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
For more details and updates, please follow up the below links.
Meetup Page : https://www.meetup.com/mydbops-databa...
Twitter: https://twitter.com/mydbopsofficial
Blogs: https://www.mydbops.com/blog/
Facebook(Meta): https://www.facebook.com/mydbops/
A Deep Dive into ScyllaDB's Architecture
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
The Microsoft 365 Migration Tutorial For Beginner.pptx
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
Recently uploaded (20)
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
Data Science: Driving Smarter Finance and Workforce Decsions for the Enterprise
- 1. Smarter Decisions Using Data Adeyemi Ajao Vlad Giverts
- 2. Reactive – Operational Reporting Operational Reporting for Measurement of Efficiency and Compliance, Data Exploration and Integration, Development of Data Dictionary Level 1 Predictive Analytics Development of Predictive Models, Scenario Planning, Risk Analysis and Mitigation, Integration with Strategic Planning Level 4 Strategic Analysis Segmentation, Statistical Analysis, Development of “People Models”, Analysis of Dimensions to Understand Cause & Delivery of Actionable Solutions Level 3 Proactive – Advanced Reporting Operational Reporting for Benchmarking and Decision-making, Multidimensional Analysis and Dashboards Level 2 Prescriptive Analytics Machine Learning, Prescribe Recommended Actions, Specify Interrelated Effects of Decisions Level 5 Evolution of Analytics Source: Bersin & AssociatesSource: Bersin & Associates
- 4. State of the Art Data Analytics System Ad-Hoc initial analysis Expensive Software Taxonomy-Building & Semantic Grouping Unsupervised Learning Clustering Supervised Learning Optimization Deep Learning Algorithms Natural Language Processing Developers Predictive Applications xxx xxx Hardware Upgrades Prescriptive Applications Versioning Machine Learning Algorithms Data Cleansing/ Transformation Multi-Source Data Integration Expensive Software Data Scientists Creating a State of the Art Analytics System VERY HARD AND EXPENSIVE! How hard can it be? Data Aggregation Data Classification Implementation of Algorithms Application Creation Continuous Machine Learning Application UpdatesMaintenance
- 5. Taxonomy-Building & Semantic Grouping Ad-Hoc initial analysis Unsupervised Learning Clustering Prescriptive Applications Versioning Data Cleansing/ Transformation Multi-Source Data Integration Maintenance Expensive Software Supervised Learning Optimization Deep Learning Algorithms Natural Language Processing Developers Predictive Applications xxx xxx Hardware UpgradesMachine Learning Algorithms Expensive Software Data Scientists Application Updates Data Aggregation Data Classification Application Creation Continuous Machine Learning Implementation of AlgorithmsInsight Apps
- 7. HCM Example: Top Performer Retention Business Question Addressed: Which of my top performers are at risk of leaving? To help VPs of HR and lower-level managers retain top talent by: • Building insights from performance metrics • Integrating with Recommended Candidates to ensure prospective talent has strong potential • Suggesting improvements to reduce talent flight risk Key Objectives Performance Reviews Identify top performers With high flight risk Take actions to keep performers Reduce Flight Risk Cycle for Maintaining Top Talent
- 8. 8 SYMAN and Industry Trees Example: Too many “nurses”
- 9. HCM Example: Top Performer Retention Normal Department Abnormal Department
- 10. ▪ Model was 40% more accurate than the manager ▪ With 25% overlap between the manager and the model Tangibly more Accurate Predictions High Risk (By Model) Terminated High Risk (By Manager) 55 215 137 550
- 11. Retention Prediction Quality WORKDAY CONFIDENTIAL Period Total Random Prediction Model Prediction Termed (Apr – June) 300 6 110 (38%) Termed (July-Nov) 585 22 (3.7%) 169 (30%)
- 12. Workday Analytics Connect Any type – structured and unstructured Any size, including Big Data Inform Intuitive, interactive visualizations Contextual Predictive and Prescriptive Act Same global platform Same security model Accessible anywhere
- 13. Retention Risk: Tech Stack
- 14. Retention Risk: Tech Stack Application
- 15. Retention Risk: Tech Stack Application Spark HDFS Elastic Search YARN ML Pipeline Kafka Indexing
- 16. Indexing & Information Retrieval Application Elastic Search Kafka Indexing
- 17. Indexing & Information Retrieval Application Elastic Search Kafka Indexing
- 24. Data and “Features” Tenure Time in Current Function Pay Range Penetration Manager Attrition Rate Num Promotions Avg Time Between Promotions
- 28. Training and Validation Barry Raise: $1,000 2014 2016 Ravi Left :( John Left :( Albert Promoted! Yury Hired Tejas Changed Teams
- 29. Training and Validation Q1 2014 Q2 2014 Q3 2014 Q4 2014 Q1 2015 Q2 2015
- 30. Training and Validation Q1 2014 Q2 2014 Q3 2014 Q4 2014 Q1 2015 Q2 2015 TRAINING VALIDATION
- 33. Evaluation Q1 2014 Q2 2014 Q3 2014 Q4 2014 Q1 2015 Q2 2015
- 34. Evaluation Q3 2015 Q4 2015 Q1 2014 Q2 2014 Q3 2014 Q4 2014 Q1 2015 Q2 2015
- 35. Evaluation PREDICTION Q3 2015 Q4 2015 Q1 2014 Q2 2014 Q3 2014 Q4 2014 Q1 2015 Q2 2015
- 41. Thank You! Q & A