The document discusses modern data applications and architectures. It introduces Apache Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. Hadoop provides massive scalability and easy data access for applications. The document outlines the key components of Hadoop, including its distributed storage, processing framework, and ecosystem of tools for data access, management, analytics and more. It argues that Hadoop enables organizations to innovate with all types and sources of data at lower costs.
Tuomas Autio's and Mikko Mattila's presentation from Hadoop & Azure Marketplace - digitalisaation tekijät Breakfast seminar on the 26th April. Find our blogs about Hadoop: http://www.bilot.fi/en/explore/?cat=blog&tag=hadoop
Pasi Vuorela's presentation from the Hadoop ja Azure Marketplace - digitalisaation tekijät - event. Vuorela works as Nordic Sales Manager @ Hortonworks
Digital transformations require a new hybrid cloud—one that’s open by design, and frees clients to choose and change environments, data and services as needed. This approach allows cloud apps and services to be rapidly composed using the best relevant data and insights available, while maintaining clear visibility, control and security—everywhere. How do you decide where to put data on a hybrid cloud and how to use it? What’s the best hybrid cloud strategy in terms of data and workload? How should you leverage a 50/50 rule or a 80/20 rule and user interaction to evaluate which data/workload to move to the cloud and which data/workload to keep on-premise? Hybrid cloud provides an open platform for innovation, including cognitive computing. Organizations are looking for taking shadow IT out of the shadows by providing a self-service way to the information and a hybrid cloud strategy is allowing that. Also, how to use hybrid cloud for better manage data sovereignty & compliance?
Tuomas Autio's and Mikko Mattila's presentation from Hadoop & Azure Marketplace - digitalisaation tekijät Breakfast seminar on the 26th April. Find our blogs about Hadoop: http://www.bilot.fi/en/explore/?cat=blog&tag=hadoop
Pasi Vuorela's presentation from the Hadoop ja Azure Marketplace - digitalisaation tekijät - event. Vuorela works as Nordic Sales Manager @ Hortonworks
Digital transformations require a new hybrid cloud—one that’s open by design, and frees clients to choose and change environments, data and services as needed. This approach allows cloud apps and services to be rapidly composed using the best relevant data and insights available, while maintaining clear visibility, control and security—everywhere. How do you decide where to put data on a hybrid cloud and how to use it? What’s the best hybrid cloud strategy in terms of data and workload? How should you leverage a 50/50 rule or a 80/20 rule and user interaction to evaluate which data/workload to move to the cloud and which data/workload to keep on-premise? Hybrid cloud provides an open platform for innovation, including cognitive computing. Organizations are looking for taking shadow IT out of the shadows by providing a self-service way to the information and a hybrid cloud strategy is allowing that. Also, how to use hybrid cloud for better manage data sovereignty & compliance?
Introduction: This workshop will provide a hands on introduction to basic Machine Learning techniques with Spark ML using a Sandbox on students’ personal machines.
Format: A short introductory lecture on a select important supervised and unsupervised Machine Learning techniques followed by a demo, lab exercises and a Q&A session. The lecture will be followed by lab time to work through the lab exercises and ask questions.
Objective: To provide a quick and short hands-on introduction to Machine Learning with Spark ML. In the lab, you will use the following components: Apache Zeppelin (a “Modern Data Science Toolbox”) and Apache Spark. You will learn how to analyze the data, structure the data, train Machine Learning models and apply them to answer real-world questions.
Pre-requisites: Registrants must bring a laptop that can run the Hortonworks Data Cloud.
Speaker:
Robert Hryniewicz, Developer Advocate, Hortonworks
General Data Protection Regulation (GDPR) which will be in effect in 2018, brings newer requirements for managing personal and sensitive data of European Union subjects. The recently enacted Privacy Shield directive from 2016 now regulates the movement of data between EU and the US. Together, both regulations are impacting how CXOs are thinking about procuring, storing and processing personal and sensitive data.
Over the last few years, open-source projects such as Apache Ranger and Apache Atlas have been driving comprehensive security and governance within Hadoop and the big data ecosystem. Solution vendors such as Privacera are leveraging the power of Hadoop and Apache projects such as Atlas, Ranger to help security and compliance teams within enterprises easily identify and protect data that are subject to the privacy regulations and monitor the use of such data.
This talk will walk through the current regulatory climate in Europe and how it can impact big data implementations. We will specifically walk through a business framework that enterprises can use to build a strategy to manage GDPR, Privacy Shield, and other regulations. We will use a live demonstration to show how projects such as Apache Ranger, Apache Atlas and solutions such as Privacera can be used effectively to address specific requirements of these regulations.
Data proliferation from 7+ billion humans and 20+ billion devices from every walk of life has been the focus in the last decade. With the velocity, variety and volume of data, every data organization’s goal shifted to protecting and monetizing data from rapidly growing network of IOT embedded objects and sensors.
One of the true and tried business continuity methodology of storing and retrieving vast amount of data has been through replication of Hadoop systems on hybrid clouds and in geographically distributed data centers. Replication is similar to Blockchain using autonomous smart contracts instantiated on the metadata and data so that the replicated data follows a single source of truth.
Replicas can be maintained across geographically distributed data centers giving greater risk tolerance capabilities to the businesses continuity plan for the data-sets. With intelligent predictive analytics based on usage patterns, dynamic tiering policies can be triggered on the data sets to provide true value-add to the data. The temperature of the data is used to move data between hot/warm/cold/archival storage based on configurable policies leading to greater reduction in total cost of ownership.
Users in 2018 and beyond demand absolute availability of data as and when they desire. The dynamic data access management is fundamental concept to satisfy the business continuity plan. Seamless enterprise-grade disaster recovery to support business continuity use case has significant challenges around replicating security and governance on data-sets. In this talk we will discuss how the above challenge can be addressed for supporting seamless replication and disaster recovery for Hadoop-scale data. NIRU ANISETI, Product Manager, Hortonworks
Introduction: This workshop will provide a hands on introduction to Apache Hadoop using the HDP Sandbox on students’ personal machines.
Format: A short introductory lecture about Apache Hadoop and a few key additional Apache projects in the extended ecosystem used in the lab followed by a demo, lab exercises and a Q&A session.
Objective: To provide a quick and short hands-on introduction to Hadoop. This lab will use the following Hadoop components: HDFS, YARN, Apache Pig, Apache Hive, Apache Spark, and Apache Ambari User Views. You will learn how to move data into HDFS, explore the data, clean the data, issue SQL queries and then build a report with Apache Zeppelin.
Pre-requisites: Registrants must bring a laptop and have the Hortonworks Sandbox installed.
Speaker:
Rafael Coss, Data Community Developer Advocate, Hortonworks
Eliminating the Challenges of Big Data Management Inside HadoopHortonworks
Your Big Data strategy is only as good as the quality of your data. Today, deriving business value from data depends on how well your company can capture, cleanse, integrate and manage data. During this webinar, we discussed how to eliminate the challenges to Big Data management inside Hadoop.
Go over these slides to learn:
· How to use the scalability and flexibility of Hadoop to drive faster access to usable information across the enterprise.
· Why a pure-YARN implementation for data integration, quality and management delivers competitive advantage.
· How to use the flexibility of RedPoint and Hortonworks to create an enterprise data lake where data is captured, cleansed, linked and structured in a consistent way.
The Power of your Data Achieved - Next Gen ModernizationHortonworks
Fueled by ever-changing customer behaviors and an increasing number of industry disruptions, the modern enterprise requires analytics to stay ahead of the game. Today’s data warehouse needs continuous enhancements to address new requirements for advanced analytics, real-time streaming data, Big Data, and unstructured data. The focus should be on developing a forward-looking, future-proof view and holistically addressing the combination of forces that are impacting the existing operational model.
Insurance companies of all sizes are challenged to keep up with emerging technologies that deliver a competitive advantage. Recording: https://www.brighttalk.com/webcast/9573/192877
Big data holds the key to greater customer insight and stronger customer relationships. But risk of sensitive data exposure — and compliance violations — keeps many insurers from pursuing big data initiatives and reaping the rewards of business-driven analytics. Join Dataguise and Hortonworks for this live webinar to learn how you can free your organization from traditional information security constraints and unlock the power of your most valuable business assets.
• What do you need to know about PII/PHI privacy before embarking on big data initiatives?
• Why do so many big data initiatives fail before they’ve even begun—and what can you do about it?
• How can IT security organizations help data scientists extract more business value from their data?
• How are leading insurance companies leveraging big data to gain competitive advantage?
see the recording: http://youtu.be/qdhF1sfef10
Ofer Medelvitch, Director of Data Science of Hortonworks and Michael Zeller, Founder and CEO of Zementis present key learnings as to what drives successful implementations of big data analytics projects. Their knowledge comes from working with dozens of companies from small cloud-based start-ups to some of the largest companies in the world.
Make Streaming IoT Analytics Work for YouHortonworks
Download Hortonworks DataFlow (HDF™) here - http://hortonworks.com/downloads/#dataflow. Making Streaming IoT Analytics Work For You With Apache NiFi, Storm, Raspberry Pi and more.
3 CTOs Discuss the Shift to Next-Gen Analytic EcosystemsHortonworks
Wow! When have you ever sat in on a Big Data analytics discussion by three of the most influential CTOs in the industry? What do they talk about among themselves?
Join Teradata's Stephen Brobst, Informatica's Sanjay Krishnamurthi, and Hortonworks' Scott Gnau as they provide a framework and best practices for maximizing value for data assets deployed within a Big Data & Analytics Architecture.
Slides from the joint webinar. Learn how Pivotal HAWQ, one of the world’s most advanced enterprise SQL on Hadoop technology, coupled with the Hortonworks Data Platform, the only 100% open source Apache Hadoop data platform, can turbocharge your Data Science efforts.
Together, Pivotal HAWQ and the Hortonworks Data Platform provide businesses with a Modern Data Architecture for IT transformation.
Global Data Management – a practical framework to rethinking enterprise, oper...DataWorks Summit
Global data management is not a newly coined term. However, what it stands for is actually widening in scope particularly around data-in-motion and data-at-rest. Significant technology trends such as IoT, cloud, AI/ML, blockchain, and streaming data have given rise to excessive data volumes and also innovative use cases. The scope for global data management now extends all the way from ingestion, processing, storage, governance, security to analysis. With a good number of endpoints served through the cloud and major application footprints remaining on-premisess, it is pertinent to have a global data management strategy that supports hybrid models and more specifically, a multi-cloud model.
Many modern businesses struggle to balance the demands of rapidly innovating through new technologies like machine learning with the need to keep data safe and secure, all while responding to a constantly changing regulatory landscape. This puts data stewards, data engineers, architects, data scientists, and analysts under intense pressure as they must contend with existing and new applications, multiple logical and physical data stores and sources, diverse data types, and data spread across several deployment environments.
Attend this session led by Matt Aslett, Research Director at 451 Research and Dinesh Chandrasekhar, Director, Hortonworks to learn more about creating a framework for your enterprise that offers guidance on how to think about global data management—priorities, responsibilities, key stakeholders, compliance, and growth.
Speakers
Dinesh Chandrasekhar, Hortonworks, Director Product Marketing
Matt Aslett, 451 Research, Research Director, Data platforms and Analytics
Making Enterprise Big Data Small with EaseHortonworks
Every division in an organization builds its own database to keep track of its business. When the organization becomes big, those individual databases grow as well. The data from each database may become silo-ed and have no idea about the data in the other database.
https://hortonworks.com/webinar/making-enterprise-big-data-small-ease/
Introduction: This workshop will provide a hands on introduction to basic Machine Learning techniques with Spark ML using a Sandbox on students’ personal machines.
Format: A short introductory lecture on a select important supervised and unsupervised Machine Learning techniques followed by a demo, lab exercises and a Q&A session. The lecture will be followed by lab time to work through the lab exercises and ask questions.
Objective: To provide a quick and short hands-on introduction to Machine Learning with Spark ML. In the lab, you will use the following components: Apache Zeppelin (a “Modern Data Science Toolbox”) and Apache Spark. You will learn how to analyze the data, structure the data, train Machine Learning models and apply them to answer real-world questions.
Pre-requisites: Registrants must bring a laptop that can run the Hortonworks Data Cloud.
Speaker:
Robert Hryniewicz, Developer Advocate, Hortonworks
General Data Protection Regulation (GDPR) which will be in effect in 2018, brings newer requirements for managing personal and sensitive data of European Union subjects. The recently enacted Privacy Shield directive from 2016 now regulates the movement of data between EU and the US. Together, both regulations are impacting how CXOs are thinking about procuring, storing and processing personal and sensitive data.
Over the last few years, open-source projects such as Apache Ranger and Apache Atlas have been driving comprehensive security and governance within Hadoop and the big data ecosystem. Solution vendors such as Privacera are leveraging the power of Hadoop and Apache projects such as Atlas, Ranger to help security and compliance teams within enterprises easily identify and protect data that are subject to the privacy regulations and monitor the use of such data.
This talk will walk through the current regulatory climate in Europe and how it can impact big data implementations. We will specifically walk through a business framework that enterprises can use to build a strategy to manage GDPR, Privacy Shield, and other regulations. We will use a live demonstration to show how projects such as Apache Ranger, Apache Atlas and solutions such as Privacera can be used effectively to address specific requirements of these regulations.
Data proliferation from 7+ billion humans and 20+ billion devices from every walk of life has been the focus in the last decade. With the velocity, variety and volume of data, every data organization’s goal shifted to protecting and monetizing data from rapidly growing network of IOT embedded objects and sensors.
One of the true and tried business continuity methodology of storing and retrieving vast amount of data has been through replication of Hadoop systems on hybrid clouds and in geographically distributed data centers. Replication is similar to Blockchain using autonomous smart contracts instantiated on the metadata and data so that the replicated data follows a single source of truth.
Replicas can be maintained across geographically distributed data centers giving greater risk tolerance capabilities to the businesses continuity plan for the data-sets. With intelligent predictive analytics based on usage patterns, dynamic tiering policies can be triggered on the data sets to provide true value-add to the data. The temperature of the data is used to move data between hot/warm/cold/archival storage based on configurable policies leading to greater reduction in total cost of ownership.
Users in 2018 and beyond demand absolute availability of data as and when they desire. The dynamic data access management is fundamental concept to satisfy the business continuity plan. Seamless enterprise-grade disaster recovery to support business continuity use case has significant challenges around replicating security and governance on data-sets. In this talk we will discuss how the above challenge can be addressed for supporting seamless replication and disaster recovery for Hadoop-scale data. NIRU ANISETI, Product Manager, Hortonworks
Introduction: This workshop will provide a hands on introduction to Apache Hadoop using the HDP Sandbox on students’ personal machines.
Format: A short introductory lecture about Apache Hadoop and a few key additional Apache projects in the extended ecosystem used in the lab followed by a demo, lab exercises and a Q&A session.
Objective: To provide a quick and short hands-on introduction to Hadoop. This lab will use the following Hadoop components: HDFS, YARN, Apache Pig, Apache Hive, Apache Spark, and Apache Ambari User Views. You will learn how to move data into HDFS, explore the data, clean the data, issue SQL queries and then build a report with Apache Zeppelin.
Pre-requisites: Registrants must bring a laptop and have the Hortonworks Sandbox installed.
Speaker:
Rafael Coss, Data Community Developer Advocate, Hortonworks
Eliminating the Challenges of Big Data Management Inside HadoopHortonworks
Your Big Data strategy is only as good as the quality of your data. Today, deriving business value from data depends on how well your company can capture, cleanse, integrate and manage data. During this webinar, we discussed how to eliminate the challenges to Big Data management inside Hadoop.
Go over these slides to learn:
· How to use the scalability and flexibility of Hadoop to drive faster access to usable information across the enterprise.
· Why a pure-YARN implementation for data integration, quality and management delivers competitive advantage.
· How to use the flexibility of RedPoint and Hortonworks to create an enterprise data lake where data is captured, cleansed, linked and structured in a consistent way.
The Power of your Data Achieved - Next Gen ModernizationHortonworks
Fueled by ever-changing customer behaviors and an increasing number of industry disruptions, the modern enterprise requires analytics to stay ahead of the game. Today’s data warehouse needs continuous enhancements to address new requirements for advanced analytics, real-time streaming data, Big Data, and unstructured data. The focus should be on developing a forward-looking, future-proof view and holistically addressing the combination of forces that are impacting the existing operational model.
Insurance companies of all sizes are challenged to keep up with emerging technologies that deliver a competitive advantage. Recording: https://www.brighttalk.com/webcast/9573/192877
Big data holds the key to greater customer insight and stronger customer relationships. But risk of sensitive data exposure — and compliance violations — keeps many insurers from pursuing big data initiatives and reaping the rewards of business-driven analytics. Join Dataguise and Hortonworks for this live webinar to learn how you can free your organization from traditional information security constraints and unlock the power of your most valuable business assets.
• What do you need to know about PII/PHI privacy before embarking on big data initiatives?
• Why do so many big data initiatives fail before they’ve even begun—and what can you do about it?
• How can IT security organizations help data scientists extract more business value from their data?
• How are leading insurance companies leveraging big data to gain competitive advantage?
see the recording: http://youtu.be/qdhF1sfef10
Ofer Medelvitch, Director of Data Science of Hortonworks and Michael Zeller, Founder and CEO of Zementis present key learnings as to what drives successful implementations of big data analytics projects. Their knowledge comes from working with dozens of companies from small cloud-based start-ups to some of the largest companies in the world.
Make Streaming IoT Analytics Work for YouHortonworks
Download Hortonworks DataFlow (HDF™) here - http://hortonworks.com/downloads/#dataflow. Making Streaming IoT Analytics Work For You With Apache NiFi, Storm, Raspberry Pi and more.
3 CTOs Discuss the Shift to Next-Gen Analytic EcosystemsHortonworks
Wow! When have you ever sat in on a Big Data analytics discussion by three of the most influential CTOs in the industry? What do they talk about among themselves?
Join Teradata's Stephen Brobst, Informatica's Sanjay Krishnamurthi, and Hortonworks' Scott Gnau as they provide a framework and best practices for maximizing value for data assets deployed within a Big Data & Analytics Architecture.
Slides from the joint webinar. Learn how Pivotal HAWQ, one of the world’s most advanced enterprise SQL on Hadoop technology, coupled with the Hortonworks Data Platform, the only 100% open source Apache Hadoop data platform, can turbocharge your Data Science efforts.
Together, Pivotal HAWQ and the Hortonworks Data Platform provide businesses with a Modern Data Architecture for IT transformation.
Global Data Management – a practical framework to rethinking enterprise, oper...DataWorks Summit
Global data management is not a newly coined term. However, what it stands for is actually widening in scope particularly around data-in-motion and data-at-rest. Significant technology trends such as IoT, cloud, AI/ML, blockchain, and streaming data have given rise to excessive data volumes and also innovative use cases. The scope for global data management now extends all the way from ingestion, processing, storage, governance, security to analysis. With a good number of endpoints served through the cloud and major application footprints remaining on-premisess, it is pertinent to have a global data management strategy that supports hybrid models and more specifically, a multi-cloud model.
Many modern businesses struggle to balance the demands of rapidly innovating through new technologies like machine learning with the need to keep data safe and secure, all while responding to a constantly changing regulatory landscape. This puts data stewards, data engineers, architects, data scientists, and analysts under intense pressure as they must contend with existing and new applications, multiple logical and physical data stores and sources, diverse data types, and data spread across several deployment environments.
Attend this session led by Matt Aslett, Research Director at 451 Research and Dinesh Chandrasekhar, Director, Hortonworks to learn more about creating a framework for your enterprise that offers guidance on how to think about global data management—priorities, responsibilities, key stakeholders, compliance, and growth.
Speakers
Dinesh Chandrasekhar, Hortonworks, Director Product Marketing
Matt Aslett, 451 Research, Research Director, Data platforms and Analytics
Making Enterprise Big Data Small with EaseHortonworks
Every division in an organization builds its own database to keep track of its business. When the organization becomes big, those individual databases grow as well. The data from each database may become silo-ed and have no idea about the data in the other database.
https://hortonworks.com/webinar/making-enterprise-big-data-small-ease/
Data in Motion - Data at Rest - Hortonworks a Modern ArchitectureMats Johansson
Presentation at Data Innovation Summit 2016 in Stockholm
How to build a modern data architecture supporting data in motion and data at rest with Hortonworks Data Flow and Data Platform.
Learn how when an organizations combine HP and Vertica Analytics Platform and Hortonworks, they can quickly explore and analyze broad variety of data types to transform to actionable information that allows them to better understand how their customers and site visitors interact with their business, offline and online.
Hortonworks Open Connected Data Platforms for IoT and Predictive Big Data Ana...DataWorks Summit
The energy industry is well known to be laggard adopters of new technology. However, industry challenges such as aging assets & workforce, increased regulatory scrutiny, renewable energy sources, depressed commodity prices, changing customer expectations, and growing data volumes are pushing companies to explore new technologies to help solve these problems. Learn how energy companies are leveraging Hortonworks Open and Connected Data Platforms to provide the predictive analysis and data insights to optimize performance for the energy industry.
Speaker
Kenneth Smith, General Manager, Energy, Hortonworks
Hortonworks Data In Motion Series Part 4Hortonworks
How real-world enterprises leverage Hortonworks DataFlow/Apache NiFi to to create real-time data flows in record time to enable new business opportunities, improve customer retention, accelerate big data projects from months to minutes through increased efficiency and reduced costs.
On-Demand webinar: http://hortonworks.com/webinar/paradigm-shift-business-usual-real-time-dataflows-record-time/
Hortonworks and Red Hat Webinar_Sept.3rd_Part 1Hortonworks
As the enterprise's big data program matures and Apache Hadoop becomes more deeply embedded in critical operations, the ability to support and operate it efficiently and reliably becomes increasingly important. To aid enterprise in operating modern data architecture at scale, Red hat and Hortonworks have collaborated to integrate Hortonworks Data Platform with Red Hat's proven platform technologies. Join us in this interactive 3-part webinar series, as we'll demonstrate how Red Hat JBoss Data Virtualization can integrate with Hadoop through Hive and provide users easy access to data.
Enabling the Real Time Analytical EnterpriseHortonworks
Combining IOT, Customer Experience and Real-Time Enterprise Data within Hadoop. What if you could derive real-time insights using ALL of your data? Join us for this webinar and learn how companies are combining “new” real-time data sources (i.e. IOT, Social, Web Logs) with continuously updated enterprise data from SAP and other enterprise transactional systems, providing deep and up-to-the-second analytical insights. This presentation will include a demonstration of how this can be achieved quickly, easily and affordably by utilizing a joint solution from Attunity and Hortonworks.
Achieving a 360-degree view of manufacturing via open source industrial data ...DataWorks Summit
Continuously improving factory operations is of critical importance to manufacturers. Consider the facts: the total cost of poor quality amounts to a staggering 20% of sales (American Society of Quality), and unplanned downtime costs plants approximately $50 billion per year (Deloitte).
The most pressing questions are: which process variables effect quality and yield and which process variables predict equipment failure? Getting to those answers is providing forward thinking manufacturers a leg up over competitors.
The speakers address the data management challenges facing today's manufacturers, including proprietary systems and siloed data sources, as well as an inability to make sensor-based data usable.
Integrating enterprise data from ERP, MES, maintenance systems, and other sources with real-time operations data from sensors, PLCs, SCADA systems, and historians represents a major first step. But how to get started? What is the value of a data lake? How are AI/ML being applied to enable real time action?
Join us for this educational session, which includes a view into a roadmap for an open source industrial IoT data management platform.
Key Takeaways:
• Understand key use cases commonly undertaken by manufacturing enterprises
• Understand the value of using multivariate manufacturing data sources, as opposed to a single sensor on a piece of equipment
• Understand advances in big data management and streaming analytics that are paving the way to next-generation factory performance
Speakers
Michael Ger, General Manager Manufacturing and Automotive, Hortonworks
Wade Salazar, Solutions Engineer, Hortonworks
Reinvent Your Data Management Strategy for Successful Digital TransformationDenodo
Watch Dinesh's keynote presentation from Fast Data Strategy Virtual Summit here: https://goo.gl/3Pa8np
Leaders are re-inventing their data management strategies through the effective use of IoT, Big Data, and data science to boost their customer experience. Yet, they struggle to modernize their data architecture due to lack of global data management processes and technologies.
Attend this session to hear from the Big Data pioneer, Hortonworks:
• Why big data and data virtualization should be core technology components of your digital transformation.
• How to manage, govern, and secure your global data footprint across a hybrid multi-cloud landscape.
• Learn about key global data management strategies and use cases that drive leading digital enterprises.
Continuously improving factory operations is of critical importance to manufacturers. Consider the facts: the total cost of poor quality amounts to a staggering 20% of sales (American Society of Quality) and unplanned downtime costs plants approximately $50 billion per year (Deloitte).
The most pressing questions are: which process variables effect quality and yield and which process variables predict equipment failure? Getting to those answers is providing forward thinking manufacturers a leg up over competitors.
The speakers address the data management challenges facing today's manufacturers, including proprietary systems and silo'ed data sources, as well as an inability to make sensor-based data usable.
Integrating enterprise data from ERP, MES, maintenance systems and other sources with real time operations data from sensors, PLCs, SCADA systems and historians represents a major first step. But how to get started? What is the value of a data lake? How are AI/ML being applied to enable real time action?
Join us for this educational session, which includes a rare view from one of our SWAT team experts into our roadmap for an open source industrial IoT data management platform.
Key Takeaways:
• How to choose an initial project from which to quickly demonstrate high value returns
• Understand the value of multivariate data sources, as opposed to a single sensor on a piece of equipment
• Understand advances in big data management and streaming analytics that are paving the way to next-generation factory performance
MICHAEL GER, General Manager, Manufacturing and Automotive, Hortonworks and RYAN TEMPLETON, Senior Solutions Engineer, Hortonworks
HP Software Performance Tour 2014 - Vincere i Big Data con HP HAVEnHP Enterprise Italia
During the HP Software Performance Tour 2014 Danilo Piatti, Regional Manager EMEA South – Autonomy, HP Software, talked about how to "win big data with HAVEn".
Human Information is made up of ideas, is diverse, and has context.
Ideas don’t exactly match like data does; they have distance.
Human Information is not static – it’s dynamic and lives everywhere.
Details on applications
HAVEn is integrated to costumers architecture through other n Apps
HP has started modifying our existing application portfolio to use HAVEn
And HP is building new applications that leverage power of HAVEn
Many customers are already building applications that use multiple HAVEn
Many Organizations are currently processing various types of data and in different formats. Most often this data will be in free form, As the consumers of this data growing it’s imperative that this free-flowing data needs to adhere to a schema. It will help data consumers to have an expectation of about the type of data they are getting and also they will be able to avoid immediate impact if the upstream source changes its format. Having a uniform schema representation also gives the Data Pipeline a really easy way to integrate and support various systems that use different data formats.
SchemaRegistry is a central repository for storing, evolving schemas. It provides an API & tooling to help developers and users to register a schema and consume that schema without having any impact if the schema changed. Users can tag different schemas and versions, register for notifications of schema changes with versions etc.
In this talk, we will go through the need for a schema registry and schema evolution and showcase the integration with Apache NiFi, Apache Kafka, Apache Storm.
There is increasing need for large-scale recommendation systems. Typical solutions rely on periodically retrained batch algorithms, but for massive amounts of data, training a new model could take hours. This is a problem when the model needs to be more up-to-date. For example, when recommending TV programs while they are being transmitted the model should take into consideration users who watch a program at that time.
The promise of online recommendation systems is fast adaptation to changes, but methods of online machine learning from streams is commonly believed to be more restricted and hence less accurate than batch trained models. Combining batch and online learning could lead to a quickly adapting recommendation system with increased accuracy. However, designing a scalable data system for uniting batch and online recommendation algorithms is a challenging task. In this talk we present our experiences in creating such a recommendation engine with Apache Flink and Apache Spark.
DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.
QE automation for large systems is a great step forward in increasing system reliability. In the big-data world, multiple components have to come together to provide end-users with business outcomes. This means, that QE Automations scenarios need to be detailed around actual use cases, cross-cutting components. The system tests potentially generate large amounts of data on a recurring basis, verifying which is a tedious job. Given the multiple levels of indirection, the false positives of actual defects are higher, and are generally wasteful.
At Hortonworks, we’ve designed and implemented Automated Log Analysis System - Mool, using Statistical Data Science and ML. Currently the work in progress has a batch data pipeline with a following ensemble ML pipeline which feeds into the recommendation engine. The system identifies the root cause of test failures, by correlating the failing test cases, with current and historical error records, to identify root cause of errors across multiple components. The system works in unsupervised mode with no perfect model/stable builds/source-code version to refer to. In addition the system provides limited recommendations to file/open past tickets and compares run-profiles with past runs.
Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together.
This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear:
• How and why the business and IT requirements originated
• How we leverage the platform to fulfill security and production requirements
• How we organize a community to:
o Guard all the players, no one gets left on the ground!
o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead)
• What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community
We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match!
DETAILS
This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.
HBase hast established itself as the backend for many operational and interactive use-cases, powering well-known services that support millions of users and thousands of concurrent requests. In terms of features HBase has come a long way, overing advanced options such as multi-level caching on- and off-heap, pluggable request handling, fast recovery options such as region replicas, table snapshots for data governance, tuneable write-ahead logging and so on. This talk is based on the research for the an upcoming second release of the speakers HBase book, correlated with the practical experience in medium to large HBase projects around the world. You will learn how to plan for HBase, starting with the selection of the matching use-cases, to determining the number of servers needed, leading into performance tuning options. There is no reason to be afraid of using HBase, but knowing its basic premises and technical choices will make using it much more successful. You will also learn about many of the new features of HBase up to version 1.3, and where they are applicable.
There has been an explosion of data digitising our physical world – from cameras, environmental sensors and embedded devices, right down to the phones in our pockets. Which means that, now, companies have new ways to transform their businesses – both operationally, and through their products and services – by leveraging this data and applying fresh analytical techniques to make sense of it. But are they ready? The answer is “no” in most cases.
In this session, we’ll be discussing the challenges facing companies trying to embrace the Analytics of Things, and how Teradata has helped customers work through and turn those challenges to their advantage.
In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.
In high-risk manufacturing industries, regulatory bodies stipulate continuous monitoring and documentation of critical product attributes and process parameters. On the other hand, sensor data coming from production processes can be used to gain deeper insights into optimization potentials. By establishing a central production data lake based on Hadoop and using Talend Data Fabric as a basis for a unified architecture, the German pharmaceutical company HERMES Arzneimittel was able to cater to compliance requirements as well as unlock new business opportunities, enabling use cases like predictive maintenance, predictive quality assurance or open world analytics. Learn how the Talend Data Fabric enabled HERMES Arzneimittel to become data-driven and transform Big Data projects from challenging, hard to maintain hand-coding jobs to repeatable, future-proof integration designs.
Talend Data Fabric combines Talend products into a common set of powerful, easy-to-use tools for any integration style: real-time or batch, big data or master data management, on-premises or in the cloud.
While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.
Hadoop Distributed File System (HDFS) evolves from a MapReduce-centric storage system to a generic, cost-effective storage infrastructure where HDFS stores all data of inside the organizations. The new use case presents a new sets of challenges to the original HDFS architecture. One challenge is to scale the storage management of HDFS - the centralized scheme within NameNode becomes a main bottleneck which limits the total number of files stored. Although a typical large HDFS cluster is able to store several hundred petabytes of data, it is inefficient to handle large amounts of small files under the current architecture.
In this talk, we introduce our new design and in-progress work that re-architects HDFS to attack this limitation. The storage management is enhanced to a distributed scheme. A new concept of storage container is introduced for storing objects. HDFS blocks are stored and managed as objects in the storage containers instead of being tracked only by NameNode. Storage containers are replicated across DataNodes using a newly-developed high-throughput protocol based on the Raft consensus algorithm. Our current prototype shows that under the new architecture the storage management of HDFS scales 10x better, demonstrating that HDFS is capable of storing billions of files.
How to optimize Hortonworks Apache Spark ML workloads on Power - POWER 8/9 architecture is the latest offering from IBM and OpenPower foundation. It is the perfect platform for optimizing Hortonworks Spark's performance. During this presentation we will walk the audience through steps required to optimize YARN, HDFS, and Spark on a Power cluster.
Step required:
1) Classify workload into CPU, Memory, IO or mixed (CPU, memory, IO) intensive
2) Characterize "out-of-box" Hortonworks spark workload to understand CPU, Memory, IO and Network performance characteristics
3) Floor Plan cluster resources
4) Tune "out-of-box" workload to navigate "Roofline" Performance space in the above named dimensions
5) If workload is Memory / IO/ Network intensive bound then tune SPARK to increase operational intensity operations/byte as much as possible to make it CPU bound
6) Divide search space into regions and perform exhaustive search.
7) Identify Performance bottlenecks by resource monitoring and tune the System, JVM or application layer by profiling application and hardware counters if required.
The challenge of computing big data for evolving digital business processes demands variety of computation techniques and engines (SQL, OLAP, time-series, graph, document store), but working in unified framework. A simple architecture of data transformations while ensuring the security, governance, and operational administration are the necessary critical components for enterprise production environments supporting day-to-day business processes. In this session, you will learn about best practices & critical components to ensure business value from latest production deployments. Hear how existing customers are using SAP Vora and the value they have achieved so far with this in-memory engine for distributed data processing. The session provides you with a clear understanding how SAP Vora and open source components like Apache Hadoop and Apache Spark offer an architecture that supports a wide variety of use cases and industries. You will also receive very useful insight where to find development resources, test drive demos, and general documentation.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
78. Data Discovery Lab
• Elefante Wine Company has a fleet of over 100 trucks.
• The geolocation data collected from the trucks contains events generated while the truck drivers are
driving.
• The company’s goal with Hadoop is to Mitigate Risk:
o Understand correlations between miles driven and events
o Compute the risk factor for each driver based on mileage & events
o Lab Env
o Sandbox 2.5
o Lab Doc
o URL: http://tinyurl.com/hello-hdp
o Load Data
o Query Data
o Process Data
79. Elefante Wine Current Challenges
The Company
Elefante Wine is a boutique wine fulfillment company with a large fleet of trucks. It delivers wine
in a highly-regulated industry with stringent transportation requirements.
The Situation
Recently a number of driver violations led to fines and increased insurance rates
The Challenges
• Rising Operational Costs
• Driver Safety
• Risk Management
• Logistics Optimization
81. Elefante Wine Risk and Driver Safety Challenges
Trucks outfitted with new sensors generating large
volumes of new data:
• Location
• Speed
• Driver Violations
Need to be integrate real-time & historical data
Increase safety and reduce liabilities
Anticipate driver violations BEFORE they
happen and take precautionary actions
Find predictive correlations in driver behavior over
large volumes of real-time data
Difficult to deliver timely insights to the right
people and systems to take action
Data Discovery
Uncover new
findings
Predictive Analytics
Identify your next best
action
Better Understanding
of the Past
Better Prediction
of the Future