Hadoop for the disillusioned
•Download as PPT, PDF•
1 like•1,308 views
This document discusses Hadoop and its use for managing large and growing amounts of data beyond what traditional systems can handle. It outlines the different layers, technologies, and distributions that make up Hadoop platforms today. It notes that while many organizations initially adopt Hadoop to save money on data management and queries, they then face the challenge of determining the next steps. It recommends asking domain experts what unanswered questions they have and finding ways to obtain the necessary data to answer those questions, either from within or outside the organization. Building data products is also presented as a way for organizations to explore their data assets. A few examples of real-world Hadoop uses are briefly described.
Report
Share
Report
Share
Recommended
Hadoop file systems
This document discusses how Hadoop's pluggable file system architecture allows analytics to be performed on data in various storage systems without needing to move the data. It provides examples of configuring Hadoop to access data stored in HDFS, GlusterFS, Swift, CassandraFS and other file systems. The key point is that Hadoop applications like MapReduce can access data through a common file system interface while the underlying storage implementation varies. This flexibility enables analyzing large datasets without replicating the data into a separate repository.
4 hadoop for-the-disillusioned
This document discusses Hadoop and its use for managing large and growing amounts of data beyond what can typically be handled. It provides an overview of the different technological layers that make up Hadoop platforms, including computational runtimes, file systems, infrastructures, and popular distributions. It also discusses how many early adopters of Hadoop may feel disillusioned once their initial problems are solved and provides suggestions for continuing to innovate with Hadoop, such as asking domain experts what new questions can be answered with available or obtainable data in order to develop new data products and services.
Data Orchestration for AI, Big Data, and Cloud
IFA+ Summit 2019
Sept 9, 2019
Keynote by Haoyuan Li, Alluxio
Founder, Chairman, CTO
For more events: https://www.alluxio.io/events/
Data Orchestration for AI, Big Data, and Cloud
This document discusses the need for data orchestration across fragmented data environments. As more data is generated and stored across different storage systems and clouds, data silos have become inevitable. A data orchestration solution like Alluxio can abstract and orchestrate data across these silos, making data locally accessible to compute frameworks regardless of where the data is stored. Alluxio provides a unified view of data locations, enables data access from any application, and allows data to be burst elastically across clouds for compute. Many large companies are adopting data orchestration to improve data access, reduce costs, and gain more flexibility in their cloud environments.
Meetup at AI NextCon 2019: In-Stream data process, Data Orchestration & More
Alluxio - Data Orchestration for Analytics and AI in the Cloud
Oct 8, 2019
Speakers:
Haoyuan Li & Bin Fan, Alluxio
Visit https://www.alluxio.io/events/ for more Alluxio events.
Cloud computing and Hadoop introduction
This document introduces BioCloud, a tool for using cloud computing platforms like Hadoop to process large biological datasets in parallel. It discusses how biology applications are becoming more resource-intensive and how cloud platforms can provide scalable computing resources at a lower cost than local hardware. It provides an overview of Hadoop and MapReduce as an framework for processing vast amounts of data across clusters of machines. Examples of companies using Hadoop include Google, Yahoo, and Facebook for applications involving terabytes of data.
Extending your Hadoop Implementation to the Cloud
This document discusses strategies for extending Hadoop implementations to the cloud. It describes using infrastructure as a service (IaaS) to run Hadoop clusters in the cloud, leveraging cloud storage for flexibility, and using Hadoop as a service offerings. Hybrid scenarios involving both on-premises and cloud deployments are also covered. Demonstrations show deploying Hadoop clusters to Microsoft Azure and using Azure HDInsight.
Data streaming at VRT
To capture and perform analysis the data stream generated by the VRT NU and VRT NWS platforms, we built a high-performance and real-time data streaming platform based on Apache Kafka.
These slides contain the lessons learned and a brief introduction to our open-sourced architecture
Recommended
Hadoop file systems
This document discusses how Hadoop's pluggable file system architecture allows analytics to be performed on data in various storage systems without needing to move the data. It provides examples of configuring Hadoop to access data stored in HDFS, GlusterFS, Swift, CassandraFS and other file systems. The key point is that Hadoop applications like MapReduce can access data through a common file system interface while the underlying storage implementation varies. This flexibility enables analyzing large datasets without replicating the data into a separate repository.
4 hadoop for-the-disillusioned
This document discusses Hadoop and its use for managing large and growing amounts of data beyond what can typically be handled. It provides an overview of the different technological layers that make up Hadoop platforms, including computational runtimes, file systems, infrastructures, and popular distributions. It also discusses how many early adopters of Hadoop may feel disillusioned once their initial problems are solved and provides suggestions for continuing to innovate with Hadoop, such as asking domain experts what new questions can be answered with available or obtainable data in order to develop new data products and services.
Data Orchestration for AI, Big Data, and Cloud
IFA+ Summit 2019
Sept 9, 2019
Keynote by Haoyuan Li, Alluxio
Founder, Chairman, CTO
For more events: https://www.alluxio.io/events/
Data Orchestration for AI, Big Data, and Cloud
This document discusses the need for data orchestration across fragmented data environments. As more data is generated and stored across different storage systems and clouds, data silos have become inevitable. A data orchestration solution like Alluxio can abstract and orchestrate data across these silos, making data locally accessible to compute frameworks regardless of where the data is stored. Alluxio provides a unified view of data locations, enables data access from any application, and allows data to be burst elastically across clouds for compute. Many large companies are adopting data orchestration to improve data access, reduce costs, and gain more flexibility in their cloud environments.
Meetup at AI NextCon 2019: In-Stream data process, Data Orchestration & More
Alluxio - Data Orchestration for Analytics and AI in the Cloud
Oct 8, 2019
Speakers:
Haoyuan Li & Bin Fan, Alluxio
Visit https://www.alluxio.io/events/ for more Alluxio events.
Cloud computing and Hadoop introduction
This document introduces BioCloud, a tool for using cloud computing platforms like Hadoop to process large biological datasets in parallel. It discusses how biology applications are becoming more resource-intensive and how cloud platforms can provide scalable computing resources at a lower cost than local hardware. It provides an overview of Hadoop and MapReduce as an framework for processing vast amounts of data across clusters of machines. Examples of companies using Hadoop include Google, Yahoo, and Facebook for applications involving terabytes of data.
Extending your Hadoop Implementation to the Cloud
This document discusses strategies for extending Hadoop implementations to the cloud. It describes using infrastructure as a service (IaaS) to run Hadoop clusters in the cloud, leveraging cloud storage for flexibility, and using Hadoop as a service offerings. Hybrid scenarios involving both on-premises and cloud deployments are also covered. Demonstrations show deploying Hadoop clusters to Microsoft Azure and using Azure HDInsight.
Data streaming at VRT
To capture and perform analysis the data stream generated by the VRT NU and VRT NWS platforms, we built a high-performance and real-time data streaming platform based on Apache Kafka.
These slides contain the lessons learned and a brief introduction to our open-sourced architecture
Enabling Apache Spark for Hybrid Cloud
1) Alluxio provides a solution for accessing data across hybrid cloud environments by serving as an abstraction layer between applications and underlying storage systems.
2) It allows compute resources and data storage to be separated and scaled independently through its unified namespace and ability to access data locally through intelligent data tiering even when stored remotely.
3) Use cases include bursting big data workloads between on-premise and cloud environments, accelerating popular big data frameworks on object storage, and enabling data orchestration for agility across multiple clouds and storage systems.
Final deck
The document provides an introduction to big data concepts including what big data is, how issues with processing large amounts of data were addressed, and what Apache Hadoop is. It then discusses two use case scenarios involving using Hadoop and Apache Cassandra to solve challenges around product recommendations and managing large volumes of email data at scale.
Build Your Own Data Beast : Greenplum + Dell
Looking for a software and hardware stack that can get you results with big data now? Greenplum Database plus Dell hardware will help you construct a data science monster.
Hadoop Summit Amsterdam 2013 - Making Hadoop Ready for Prime Time - Syncsort ...
Lightening talk from the Hadoop Summit 2013 in Amsterdam covering how Syncsort is helping make Hadoop Ready for Prime Time. It includes the pluggable sort contribution - the impact on sort, join, aggregation, merge, filter in hadoopand Syncsort's ability to move mainframe data to hadoop - Big Iron to Big Data.
Get Results, Build Your Own Big Data Beast : Greenplum + Dell
Build your own big data cluster using a software package that enables your current DBAs and Analysts to step up their data science game. Dell hardware and Pivotal Greenplum Database are a perfect match.
Developing high frequency indicators using real time tick data on apache supe...
This document summarizes the Central Bank of Turkey's project to develop high frequency market indicators using real-time tick data from the Thomson Reuters Enterprise Platform. It describes how they set up Apache Kafka, Druid, Spark and Superset on Hadoop to ingest, store, analyze and visualize the data. Their goal was to observe foreign exchange markets in real-time to detect risks and patterns. The architecture evolved over three phases from an initial test cluster to integrating Druid and Hive for improved querying and scaling to production. Work is ongoing to implement additional indicators and integrate historical data for enhanced analysis.
Realtime
Distributed Analysis
of Datastreams
This document discusses real-time distributed data analysis and frameworks like Storm. It introduces the lambda architecture for working with big data, which has a speed layer, serving layer, and batch layer. The batch layer stores immutable data and precomputes batch views using Hadoop. The serving layer provides random access to these views using systems like Impala. The speed layer compensates for high-latency batch views using fast, incremental algorithms like Storm. It also describes how Twitter evolved from queues and workers to using Storm for scalable, fault-tolerant real-time analytics.
Data Tools and the Data Scientist Shortage
Wes McKinney discusses the shortage of data scientists and analysts. There is a shortage of 140,000-190,000 people with analytics expertise and 1.5 million managers/analysts with skills to understand and make decisions based on big data analysis in the United States alone. This shortage can be addressed through improved education, tools, and a cultural shift. New approaches and tools are needed to make data science accessible to more people and bring analytics capabilities to various industries.
Lambda Architecture - Storm, Trident, SummingBird ... - Architecture and Over...
This document discusses the Lambda architecture, which is a design pattern for building data processing systems that require both batch and real-time processing. It describes the key components of a Lambda architecture, including batch and real-time data pipelines, serving layers, and a speed layer for low-latency queries. It also covers some of the main tools and frameworks used to implement Lambda architectures, such as Storm, Trident, Redis, and Summingbird, which provides a common API for both batch and real-time processing.
Introduction to Hivemall
This document discusses a tech talk given by Makoto Yui at Treasure Data on May 14, 2015. It includes an introduction to Hivemall, an open source machine learning library built on Apache Hive. The talk covers how to use Hivemall for tasks like data preparation, feature engineering, model training, and prediction. It also discusses doing real-time prediction by training models offline on Hadoop and performing online predictions using the models on a relational database management system.
Building Open Data Lakes on AWS with Debezium and Apache Hudi
Build a simple open data lake on AWS using a combination of open-source software (OSS), including Red Hat’s Debezium, Apache Kafka, and Kafka Connect for change data capture (CDC), and Apache Hive, Apache Spark, Apache Hudi, and Hudi’s DeltaStreamer for managing our data lake. We will use fully-managed AWS services to host the open data lake components, including Amazon RDS, Amazon MKS, Amazon EKS, and EMR.
Link to the blog post and video: https://garystafford.medium.com/building-open-data-lakes-with-debezium-and-apache-hudi-c3370d3f86fb
R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk ...
The central premise of DataXu is to apply data science to better marketing. At its core, is the Real Time Bidding Platform that processes 2 Petabytes of data per day and responds to ad auctions at a rate of 2.1 million requests per second across 5 different continents. Serving on top of this platform is Dataxu’s analytics engine that gives their clients insightful analytics reports addressed towards client marketing business questions. Some common requirements for both these platforms are the ability to do real-time processing, scalable machine learning, and ad-hoc analytics. This talk will showcase DataXu’s successful use-cases of using the Apache Spark framework and Databricks to address all of the above challenges while maintaining its agility and rapid prototyping strengths to take a product from initial R&D phase to full production. The team will share their best practices and highlight the steps of large scale Spark ETL processing, model testing, all the way through to interactive analytics.
"Einstürzenden Neudaten: Building an Analytics Engine from Scratch", Tobias J...
The document discusses Valo, a big data analytics engine built from scratch focusing on simplicity and distributed capabilities. It describes Valo's architecture including time-series and semi-structured data repositories, REST API, and execution engine. It also discusses challenges of building distributed systems including cluster failures, data distribution, algorithms, and more.
Powers of Ten Redux
One of the first problems a developer encounters when evaluating a graph database is how to construct a graph efficiently. Recognizing this need in 2014, TinkerPop's Stephen Mallette penned a series of blog posts titled "Powers of Ten" which addressed several bulkload techniques for Titan. Since then Titan has gone away, and the open source graph database landscape has evolved significantly. Do the same approaches stand the test of time? In this session, we will take a deep dive into strategies for loading data of various sizes into modern Apache TinkerPop graph systems. We will discuss bulkloading with JanusGraph, the scalable graph database forked from Titan, to better understand how its architecture can be optimized for ingestion. Presented at Data Day Texas on January 27, 2018.
Graph Computing with JanusGraph
Graph Computing with JanusGraph. Presented at Cleveland Big Data Mega Meetup on September 11, 2017. https://www.meetup.com/Cleveland-Hadoop/events/241553826/
Data in Motion vs Data at Rest
In general, data can be broken into two categories – data in motion vs data at rest. Learn the difference between these two types of data and the best infrastructure options to get optimal performance.
Dataiku Flow and dctc - Berlin Buzzwords
This document discusses Dataiku Flow and DCTC. Dataiku Flow is a data-driven orchestration framework for complex data pipelines that manages data dependencies and parallelization. It allows defining datasets and tasks to transform data. DCTC is a tool that can manipulate files across different storage systems like S3, GCS, HDFS to perform operations like copying, synchronizing, dispatching files. It aims to simplify common data transfer pains. The presentation concludes with contacting information for Dataiku executives.
Snaplogic Live: Big Data in Motion
Watch this recorded demonstration of SnapLogic from our team of experts who answer your hybrid cloud and big data integration questions.
demo, ipaas, elastic integration, cloud data, app integration, data integration, hybrid could integration, big data, big data integration
Iceberg + Alluxio for Fast Data Analytics
Alluxio Day VIII
December 14, 2021
https://www.alluxio.io/alluxio-day/
Speakers:
Shouwei Chen & Beinan Wang, Alluxio
Accelerating analytics in a new era of data
Organizations today produce exponentially more data than they did just a few years ago, but their databases weren’t built to handle these new volumes. As a result, reporting takes way too long, and some complex analytics simply cannot be done. The Era of Massive Data is upon us, and a new approach is required to overcome the limitations of traditional CPU-based data stores.
KEY TAKEAWAYS
- Flexible data exploration with minimal preparation
- Unrestricted access to your organization’s full scope of data
- Access to previously unobtainable insights, for smarter business decisions
Steve Watt, Chief Architect, Hadoop and Big Data, Red Hat - 21st BDL meetup
"Hadoop Analytics on your data in place"
Steve Watt leads engineering for the Hadoop and Big Data program at Red Hat. Most recently Steve has been focusing on Hadoop Interoperability and better enabling Hadoop support for alternative filesystems. Prior to Red Hat, Steve spent 2 years at Hewlett-Packard, first co-founding the Hadoop business and then leading engineering as the Hadoop CTO. Prior to HP, Steve was at IBM for 10 years where he created IBMs first Hadoop Distribution and was part of the team that built BigSheets, the first spreadsheet interface for Hadoop.
Architecting Virtualized Infrastructure for Big Data
This document discusses architecting virtualized infrastructure for big data. It notes that data is growing exponentially and that the value of data now exceeds hardware costs. It advocates using virtualization to simplify and optimize big data infrastructure, enabling flexible provisioning of workloads like Hadoop, SQL, and NoSQL clusters on a unified analytics cloud platform. This platform leverages both shared and local storage to optimize performance while reducing costs.
More Related Content
What's hot
Enabling Apache Spark for Hybrid Cloud
1) Alluxio provides a solution for accessing data across hybrid cloud environments by serving as an abstraction layer between applications and underlying storage systems.
2) It allows compute resources and data storage to be separated and scaled independently through its unified namespace and ability to access data locally through intelligent data tiering even when stored remotely.
3) Use cases include bursting big data workloads between on-premise and cloud environments, accelerating popular big data frameworks on object storage, and enabling data orchestration for agility across multiple clouds and storage systems.
Final deck
The document provides an introduction to big data concepts including what big data is, how issues with processing large amounts of data were addressed, and what Apache Hadoop is. It then discusses two use case scenarios involving using Hadoop and Apache Cassandra to solve challenges around product recommendations and managing large volumes of email data at scale.
Build Your Own Data Beast : Greenplum + Dell
Looking for a software and hardware stack that can get you results with big data now? Greenplum Database plus Dell hardware will help you construct a data science monster.
Hadoop Summit Amsterdam 2013 - Making Hadoop Ready for Prime Time - Syncsort ...
Lightening talk from the Hadoop Summit 2013 in Amsterdam covering how Syncsort is helping make Hadoop Ready for Prime Time. It includes the pluggable sort contribution - the impact on sort, join, aggregation, merge, filter in hadoopand Syncsort's ability to move mainframe data to hadoop - Big Iron to Big Data.
Get Results, Build Your Own Big Data Beast : Greenplum + Dell
Build your own big data cluster using a software package that enables your current DBAs and Analysts to step up their data science game. Dell hardware and Pivotal Greenplum Database are a perfect match.
Developing high frequency indicators using real time tick data on apache supe...
This document summarizes the Central Bank of Turkey's project to develop high frequency market indicators using real-time tick data from the Thomson Reuters Enterprise Platform. It describes how they set up Apache Kafka, Druid, Spark and Superset on Hadoop to ingest, store, analyze and visualize the data. Their goal was to observe foreign exchange markets in real-time to detect risks and patterns. The architecture evolved over three phases from an initial test cluster to integrating Druid and Hive for improved querying and scaling to production. Work is ongoing to implement additional indicators and integrate historical data for enhanced analysis.
Realtime
Distributed Analysis
of Datastreams
This document discusses real-time distributed data analysis and frameworks like Storm. It introduces the lambda architecture for working with big data, which has a speed layer, serving layer, and batch layer. The batch layer stores immutable data and precomputes batch views using Hadoop. The serving layer provides random access to these views using systems like Impala. The speed layer compensates for high-latency batch views using fast, incremental algorithms like Storm. It also describes how Twitter evolved from queues and workers to using Storm for scalable, fault-tolerant real-time analytics.
Data Tools and the Data Scientist Shortage
Wes McKinney discusses the shortage of data scientists and analysts. There is a shortage of 140,000-190,000 people with analytics expertise and 1.5 million managers/analysts with skills to understand and make decisions based on big data analysis in the United States alone. This shortage can be addressed through improved education, tools, and a cultural shift. New approaches and tools are needed to make data science accessible to more people and bring analytics capabilities to various industries.
Lambda Architecture - Storm, Trident, SummingBird ... - Architecture and Over...
This document discusses the Lambda architecture, which is a design pattern for building data processing systems that require both batch and real-time processing. It describes the key components of a Lambda architecture, including batch and real-time data pipelines, serving layers, and a speed layer for low-latency queries. It also covers some of the main tools and frameworks used to implement Lambda architectures, such as Storm, Trident, Redis, and Summingbird, which provides a common API for both batch and real-time processing.
Introduction to Hivemall
This document discusses a tech talk given by Makoto Yui at Treasure Data on May 14, 2015. It includes an introduction to Hivemall, an open source machine learning library built on Apache Hive. The talk covers how to use Hivemall for tasks like data preparation, feature engineering, model training, and prediction. It also discusses doing real-time prediction by training models offline on Hadoop and performing online predictions using the models on a relational database management system.
Building Open Data Lakes on AWS with Debezium and Apache Hudi
Build a simple open data lake on AWS using a combination of open-source software (OSS), including Red Hat’s Debezium, Apache Kafka, and Kafka Connect for change data capture (CDC), and Apache Hive, Apache Spark, Apache Hudi, and Hudi’s DeltaStreamer for managing our data lake. We will use fully-managed AWS services to host the open data lake components, including Amazon RDS, Amazon MKS, Amazon EKS, and EMR.
Link to the blog post and video: https://garystafford.medium.com/building-open-data-lakes-with-debezium-and-apache-hudi-c3370d3f86fb
R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk ...
The central premise of DataXu is to apply data science to better marketing. At its core, is the Real Time Bidding Platform that processes 2 Petabytes of data per day and responds to ad auctions at a rate of 2.1 million requests per second across 5 different continents. Serving on top of this platform is Dataxu’s analytics engine that gives their clients insightful analytics reports addressed towards client marketing business questions. Some common requirements for both these platforms are the ability to do real-time processing, scalable machine learning, and ad-hoc analytics. This talk will showcase DataXu’s successful use-cases of using the Apache Spark framework and Databricks to address all of the above challenges while maintaining its agility and rapid prototyping strengths to take a product from initial R&D phase to full production. The team will share their best practices and highlight the steps of large scale Spark ETL processing, model testing, all the way through to interactive analytics.
"Einstürzenden Neudaten: Building an Analytics Engine from Scratch", Tobias J...
The document discusses Valo, a big data analytics engine built from scratch focusing on simplicity and distributed capabilities. It describes Valo's architecture including time-series and semi-structured data repositories, REST API, and execution engine. It also discusses challenges of building distributed systems including cluster failures, data distribution, algorithms, and more.
Powers of Ten Redux
One of the first problems a developer encounters when evaluating a graph database is how to construct a graph efficiently. Recognizing this need in 2014, TinkerPop's Stephen Mallette penned a series of blog posts titled "Powers of Ten" which addressed several bulkload techniques for Titan. Since then Titan has gone away, and the open source graph database landscape has evolved significantly. Do the same approaches stand the test of time? In this session, we will take a deep dive into strategies for loading data of various sizes into modern Apache TinkerPop graph systems. We will discuss bulkloading with JanusGraph, the scalable graph database forked from Titan, to better understand how its architecture can be optimized for ingestion. Presented at Data Day Texas on January 27, 2018.
Graph Computing with JanusGraph
Graph Computing with JanusGraph. Presented at Cleveland Big Data Mega Meetup on September 11, 2017. https://www.meetup.com/Cleveland-Hadoop/events/241553826/
Data in Motion vs Data at Rest
In general, data can be broken into two categories – data in motion vs data at rest. Learn the difference between these two types of data and the best infrastructure options to get optimal performance.
Dataiku Flow and dctc - Berlin Buzzwords
This document discusses Dataiku Flow and DCTC. Dataiku Flow is a data-driven orchestration framework for complex data pipelines that manages data dependencies and parallelization. It allows defining datasets and tasks to transform data. DCTC is a tool that can manipulate files across different storage systems like S3, GCS, HDFS to perform operations like copying, synchronizing, dispatching files. It aims to simplify common data transfer pains. The presentation concludes with contacting information for Dataiku executives.
Snaplogic Live: Big Data in Motion
Watch this recorded demonstration of SnapLogic from our team of experts who answer your hybrid cloud and big data integration questions.
demo, ipaas, elastic integration, cloud data, app integration, data integration, hybrid could integration, big data, big data integration
Iceberg + Alluxio for Fast Data Analytics
Alluxio Day VIII
December 14, 2021
https://www.alluxio.io/alluxio-day/
Speakers:
Shouwei Chen & Beinan Wang, Alluxio
Accelerating analytics in a new era of data
Organizations today produce exponentially more data than they did just a few years ago, but their databases weren’t built to handle these new volumes. As a result, reporting takes way too long, and some complex analytics simply cannot be done. The Era of Massive Data is upon us, and a new approach is required to overcome the limitations of traditional CPU-based data stores.
KEY TAKEAWAYS
- Flexible data exploration with minimal preparation
- Unrestricted access to your organization’s full scope of data
- Access to previously unobtainable insights, for smarter business decisions
What's hot (20)
Hadoop Summit Amsterdam 2013 - Making Hadoop Ready for Prime Time - Syncsort ...
Hadoop Summit Amsterdam 2013 - Making Hadoop Ready for Prime Time - Syncsort ...
Get Results, Build Your Own Big Data Beast : Greenplum + Dell
Get Results, Build Your Own Big Data Beast : Greenplum + Dell
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Lambda Architecture - Storm, Trident, SummingBird ... - Architecture and Over...
Lambda Architecture - Storm, Trident, SummingBird ... - Architecture and Over...
Building Open Data Lakes on AWS with Debezium and Apache Hudi
Building Open Data Lakes on AWS with Debezium and Apache Hudi
R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk ...
R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk ...
"Einstürzenden Neudaten: Building an Analytics Engine from Scratch", Tobias J...
"Einstürzenden Neudaten: Building an Analytics Engine from Scratch", Tobias J...
Similar to Hadoop for the disillusioned
Steve Watt, Chief Architect, Hadoop and Big Data, Red Hat - 21st BDL meetup
"Hadoop Analytics on your data in place"
Steve Watt leads engineering for the Hadoop and Big Data program at Red Hat. Most recently Steve has been focusing on Hadoop Interoperability and better enabling Hadoop support for alternative filesystems. Prior to Red Hat, Steve spent 2 years at Hewlett-Packard, first co-founding the Hadoop business and then leading engineering as the Hadoop CTO. Prior to HP, Steve was at IBM for 10 years where he created IBMs first Hadoop Distribution and was part of the team that built BigSheets, the first spreadsheet interface for Hadoop.
Architecting Virtualized Infrastructure for Big Data
This document discusses architecting virtualized infrastructure for big data. It notes that data is growing exponentially and that the value of data now exceeds hardware costs. It advocates using virtualization to simplify and optimize big data infrastructure, enabling flexible provisioning of workloads like Hadoop, SQL, and NoSQL clusters on a unified analytics cloud platform. This platform leverages both shared and local storage to optimize performance while reducing costs.
Presentation architecting virtualized infrastructure for big data
The document discusses how virtualization can help simplify big data infrastructure and analytics. Key points include:
1) Virtualization can help simplify big data infrastructure by providing a unified analytics cloud platform that allows different data frameworks and workloads to easily share resources.
2) Hadoop performance on virtualization has been proven with studies showing little performance overhead from virtualization.
3) A unified analytics cloud platform using virtualization can provide benefits like better utilization, faster provisioning of elastic resources, and multi-tenancy for secure isolation of analytics workloads.
Presentation architecting virtualized infrastructure for big data
Download & Share Technology
Presentations http://goo.gl/k80oY0
Student Guide & Best http://goo.gl/6OkI77
Architecting virtualized infrastructure for big data presentation
This document discusses architecting virtualized infrastructure for big data. It summarizes that big data is growing exponentially and new frameworks like Hadoop are enabling analysis of large, diverse data sets. Virtualization can simplify and optimize big data platforms by providing a unified analytics cloud that elastically provisions various data systems like Hadoop and SQL clusters on shared hardware infrastructure. This improves utilization and makes big data platforms faster and easier to deploy and manage.
Azure Cafe Marketplace with Hortonworks March 31 2016
Azure Big Data: “Got Data? Go Modern and Monetize”.
In this session you will learn how to architected, developed, and build completely in the open, Hortonworks Data Platform (HDP) that provides an enterprise ready data platform to adopt a Modern Data Architecture.
Data analytics using the cloud challenges and opportunities for india
- Data analytics is transitioning from traditional paradigms like SAS and SPSS to newer paradigms using open source tools like R and Python, and distributed frameworks like Hadoop.
- Cloud computing provides on-demand access to computing resources and is enabling data analytics through services like IaaS, PaaS and SaaS. However, most cloud infrastructure is based in the US raising privacy and access concerns.
- India has an opportunity to leverage its engineering talent and build domestic cloud infrastructure to ensure data sovereignty, but needs to develop strong data privacy regulations and address gaps in domain expertise and entrepreneurial ecosystems.
(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR ...
We will explore the strengths and limitations of Hadoop for analyzing large data sets and review the growing ecosystem of tools for augmenting, extending, or replacing Hadoop MapReduce. We will introduce the Amazon Elastic MapReduce (EMR) platform as the big data foundation for Hadoop and beyond by providing specific examples of running Machine Learning (Mahout), Graph Analytics (Giraph), and Statistical Analysis (R) on EMR. We will discuss also big data analytics and visualization of results with Amazon Redshift + third party business intelligence tools, as well as typical end-to-end Big Data workflow on AWS.
We will conclude with real-world examples from ICAO of Big Data analytics for aviation safety data on AWS. The integrated Safety Trend Analysis and Reporting System (iSTARS) is a web based system linking a collection of safety datasets and related web application to perform online safety and risk analysis. It uses AWS EC2, S3, EMR and related partner tools for continuous data aggregation and filtering.
Hadoop world overview trends and topics
Valentyn Kropov, Big Data Solutions Architect has recently attended "Hadoop World / Strata" – biggest and coolest Big Data conference in a World, and he can't wait to share fresh trends and topics straight from New-York. Come and learn how Hadoop cluster will help NASA to explore Mars, how Netflix build 10PB platform, what are the latest trends in Spark, to learn about newest, just announced storage engine from Cloudera called Kudu and many many more interesting stuff.
The Hive Think Tank: "Stream Processing Systems" by M.C. Srivas of MapR
M.C. Shivas's presentation was part of a panel discussion on Stream Processing Systems on January 20th, 2016 led by Ben Lorica (O'Reilly Media) with panelists: Jay Kreps (Confluent), Karthik Ramasamy (Twitter), Nikita Shamgunov (MemSQL), Ram Sriharsha (Hortonworks)
20150314 sahara intro and the future plan for open stack meetup
Sahara is an OpenStack service that allows users to easily provision and manage Hadoop clusters in OpenStack. It currently supports plugins for Hortonworks, Cloudera, and MapR distributions of Hadoop. The Cloudera plugin integrates Cloudera Manager to provision CDH services. Sahara aims to provide analytics as a service and allow data processing directly in OpenStack clusters using technologies like HDFS, Swift, and Hadoop frameworks. Performance overhead compared to bare metal Hadoop clusters is a current limitation being addressed.
Solving the Really Big Tech Problems with IoT
The Briefing Room with Dr. Robin Bloor and HPE Security
The Internet of Things brings new technological problems: sensor communications are bi-directional, the scale of data generation points has no precedent and, in this new world, security, privacy and data protection need to go out to the edge. Likely, most of that data lands in Hadoop and Big Data platforms. With the need for rapid analytics never greater, companies try to seize opportunities in tighter time windows. Yet, cyber-threats are at an all-time high, targeting the most valuable of assets—the data.
Register for this episode of The Briefing Room to hear Analyst Dr. Robin Bloor explain the implications of today's divergent data forces. He’ll be briefed by Reiner Kappenberger of HPE, who will discuss how a recent innovation -- NiFi -- is revolutionizing the big data ecosystem. He’ll explain how this technology dramatically simplifies data flow design, enabling a new era of business-driven analysis, while also protecting sensitive data.
GSJUG: Mastering Data Streaming Pipelines 09May2023
GSJUG: Mastering Data Streaming Pipelines 09May2023
https://www.meetup.com/futureofdata-princeton/events/293233881/
This is a repost from the Garden State Java Users Group Event.
Join me at
https://www.meetup.com/garden-state-java-user-group/events/293229660/
See: https://www.eventbrite.com/e/mastering-data-streaming-pipelines-tickets-627677218457?_ga=2.253257801.1787151623.1682868226-741104479.1678110925
Please note that registration via EventBrite is required to attend either in-person or online.
We are happy to announce that Tim Spann will be our special guest for the May 9, 2023 meeting!
Abstract:
In this session, Tim will show you some best practices that he has discovered over the last seven years in building data streaming applications including IoT, CDC, Logs, and more.
In his modern approach, we utilize several Apache frameworks to maximize the best features of all. We often start with Apache NiFi as the orchestrator of streams flowing into Apache Kafka. From there we build streaming ETL with Apache Flink, enhance events with NiFi enrichment. We build continuous queries against our topics with Flink SQL.
We will show where Java fits in as sources, enrichments, NiFi processors and sinks.
We hope to see you on May 9!
Speaker
Timothy Spann
Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming.
Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.
In this session, Tim will show you some best practices that he has discovered over the last seven years in building data streaming applications, including IoT, CDC, Logs, and more.
In his modern approach, we utilize several Apache frameworks to maximize the best features of all. We often start with Apache NiFi as the orchestrator of streams flowing into Apache Kafka. From there, we build streaming ETL with Apache Flink, enhance events with NiFi enrichment. We build continuous queries against our topics with Flink SQL.
We will show where Java fits in as sources, enrichments, NiFi processors, and sinks.
https://www.eventbrite.com/e/mastering-data-streaming-pipelines-tickets-627677218457?_ga=2.253257801.178
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
https://www.youtube.com/watch?v=Yeua8NlzQ3Y
https://www.conf42.com/Large_Language_Models_LLMs_2024_Tim_Spann_generative_ai_streaming
Adding Generative AI to Real-Time Streaming Pipelines
Abstract
Let’s build streaming pipelines that convert streaming events into prompts, call LLMs, and process the results.
Summary
Tim Spann: My talk is adding generative AI to real time streaming pipelines. I'm going to discuss a couple of different open source technologies. We'll touch on Kafka, Nifi, Flink, Python, Iceberg. All the slides, all the code and GitHub are out there.
Llm, if you didn't know, is rapidly evolving. There's a lot of different ways to interact with models. That enrichment, transformation, processing really needs tools. The amount of models and projects and software that are available is massive.
Nifi supports hundreds of different inputs and can convert them on the fly. Great way to distribute your data quickly to whoever needs it without duplication, without tight coupling. Fun to find new things to integrate into.
So what we can do is, well, I want to get a meetup chat going. I have a processor here that just listens for events as they come from slack. And then I'm going to clean it up, add a couple fields and push that out to slack. Every model is a little bit of different tweaking.
Nifi acts as a whole website. And as you see here, it can be get, post, put, whatever you want. We send that response back to flink and it shows up here. Thank you for attending this talk. I'm going to be speaking at some other events very shortly.
Transcript
This transcript was autogenerated. To make changes, submit a PR.
Hi, Tim Spann here. My talk is adding generative AI to real time streaming pipelines, and we're here for the large language model conference at Comp 42, which is always a nice one, great place to be. I'm going to discuss a couple of different open source technologies that work together to enable you to build real time pipelines using large language models. So we'll touch on Kafka, Nifi, Flink, Python, Iceberg, and I'll show you a little bit of each one in the demos. I've been working with data machine learning, streaming IoT, some other things for a number of years, and you could contact me at any of these places, whether Twitter or whatever it's called, some different blogs, or in person at my meetups and at different conferences around the world. I do a weekly newsletter, cover streaming ML, a lot of LLM, open source, Python, Java, all kinds of fun stuff, as I mentioned, do a bunch of different meetups. They are not just in the east coast of the US, they are available virtually live, and I also put them on YouTube, and if you need them somewhere else, let me know. We publish all the slides, all the code and GitHub. Everything you need is out there. Let's get into the talk. Llm, if you didn't know, is rapidly evolving. While you're typing down the things that you use, it
Big Data, Simple and Fast: Addressing the Shortcomings of Hadoop
In this webinar
This talk identifies several shortcomings of Apache Hadoop and presents an alternative approach for building simple and flexible Big Data software stacks quickly, based on next generation computing paradigms, such as in-memory data/compute grids. The focus of the talk is on software architectures, but several code examples using Hazelcast will be provided to illustrate the concepts discussed.
We’ll cover these topics:
-Briefly explain why Hadoop is not a universal, or inexpensive, Big Data solution – despite the hype
-Lay out technical requirements for a flexible Big/Fast Data processing stack
-Present solutions thought to be alternatives to Hadoop
-Argue why In-Memory Data/Compute Grids are so attractive in creating future-proof Big/Fast Data applications
-Discuss how well Hazelcast meets the Big/Fast Data requirements vs Hadoop
-Present several code examples using Java and Hazelcast to illustrate concepts discussed
-Live Q&A Session
Presenter:
Jacek Kruszelnicki, President of Numatica Corporation
The Hadoop Path by Subash DSouza of Archangel Technology Consultants, LLC.
Hadoop is moving towards improved security and real-time analytics. For security, Hadoop vendors have made acquisitions and implemented features like Kerberos authentication and Apache Sentry authorization. For real-time analytics, tools are focusing on real-time streaming (like Storm, Spark Streaming, and Samza) and real-time querying of data (like Hive on Tez, Impala, Drill, and Spark). The right tool depends on use cases, and enterprises should choose what is easiest and most scalable.
Big Data/Hadoop Infrastructure Considerations
1) Big data is growing exponentially and new frameworks like Hadoop are needed to analyze large, unstructured datasets.
2) Hadoop uses distributed computing and storage across commodity servers to provide scalable and cost-effective analytics. It leverages local disks on each node for temporary data to improve performance.
3) Virtualizing Hadoop simplifies operations, enables mixed workloads, and provides high availability through features like vMotion and HA. It also allows for elastic scaling of compute and storage resources.
Wasp2 - IoT and Streaming Platform
WASP is a framework to develop big data pipelines, working with streaming analytics, multi model storages and machine learning models. Everything is in real time.
Is your cloud ready for Big Data? Strata NY 2013
1) Cloud platforms can support big data workloads through virtualization which provides agility, isolation, lower costs, and operational efficiency.
2) Modern networks with spine-leaf architectures are well-suited for big data by providing uniform high bandwidth connectivity. This allows for new converged and separated storage models.
3) New distributed storage solutions like HDFS, Ceph, and scale-out NAS provide much higher capacity at lower costs than traditional SAN/NAS. They also offer features like erasure coding, snapshots, cloning and geo-replication.
Scaling up with Cisco Big Data: Data + Science = Data Science
Discover how IoT data is ingested with real-time technologies such as Apache Spark and analysed using Data Science techniques to glean insights.
Similar to Hadoop for the disillusioned (20)
Steve Watt, Chief Architect, Hadoop and Big Data, Red Hat - 21st BDL meetup
Steve Watt, Chief Architect, Hadoop and Big Data, Red Hat - 21st BDL meetup
Architecting Virtualized Infrastructure for Big Data
Architecting Virtualized Infrastructure for Big Data
Presentation architecting virtualized infrastructure for big data
Presentation architecting virtualized infrastructure for big data
Presentation architecting virtualized infrastructure for big data
Presentation architecting virtualized infrastructure for big data
Architecting virtualized infrastructure for big data presentation
Architecting virtualized infrastructure for big data presentation
Azure Cafe Marketplace with Hortonworks March 31 2016
Azure Cafe Marketplace with Hortonworks March 31 2016
Data analytics using the cloud challenges and opportunities for india
Data analytics using the cloud challenges and opportunities for india
(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR ...
(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR ...
The Hive Think Tank: "Stream Processing Systems" by M.C. Srivas of MapR
The Hive Think Tank: "Stream Processing Systems" by M.C. Srivas of MapR
20150314 sahara intro and the future plan for open stack meetup
20150314 sahara intro and the future plan for open stack meetup
GSJUG: Mastering Data Streaming Pipelines 09May2023
GSJUG: Mastering Data Streaming Pipelines 09May2023
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Big Data, Simple and Fast: Addressing the Shortcomings of Hadoop
Big Data, Simple and Fast: Addressing the Shortcomings of Hadoop
The Hadoop Path by Subash DSouza of Archangel Technology Consultants, LLC.
The Hadoop Path by Subash DSouza of Archangel Technology Consultants, LLC.
Scaling up with Cisco Big Data: Data + Science = Data Science
Scaling up with Cisco Big Data: Data + Science = Data Science
More from Steve Watt
Building Clustered Applications with Kubernetes and Docker
August 2015 - Presented at LinuxCon and ContainerCon
Demos:
1) NGINX Web Cluster with Local Storage
2) Hot Upgrade/Deploy of an NGINX Web Cluster with Shared Storage (GlusterFS)
3) MySQL with Block Storage (Ceph RBD)
4) Apache Spark in Kubernetes with Shared Storage
Building Clustered Applications with Kubernetes and Docker
This document discusses building clustered applications with Kubernetes and Docker. It provides an overview of Kubernetes, including its architecture and components. It then demonstrates how to install Kubernetes, define and deploy pods, add replication controllers and services. It discusses using volumes for persistence, including different volume types like GlusterFS. Finally, it touches on debugging and provides contact information for following up.
Apache con 2013-hadoop
This document summarizes the results of profiling a Hadoop cluster to analyze infrastructure needs. Key findings include:
- The I/O and network subsystems were underutilized, with I/O at 10% capacity and network below 25% capacity.
- CPU utilization was high with low I/O wait, indicating optimal usage. Memory usage showed a high percentage of caching to reduce I/O waits.
- Initial HBase testing showed CPU utilization could not be fully driven by two workloads. Performance was good when data fit in cache but dropped when exceeding cache size.
Apache con 2012 taking the guesswork out of your hadoop infrastructure
This document discusses profiling a Hadoop cluster to determine infrastructure needs. It describes instrumenting a cluster running a 10TB TeraSort workload using the SAR tool to collect CPU, memory, I/O, and network metrics from each node. The results show the I/O subsystem was underutilized at 10% while CPU utilization was high, indicating the workload was not I/O bound. Memory metrics showed a high percentage of cached data, meaning the CPUs were not waiting on memory. Profiling workloads in this way helps right-size Hadoop infrastructure.
Mining the Web for Information using Hadoop
This document discusses using Apache Nutch to crawl and gather data from websites and store it in HDFS files. The data is then processed using Pig Latin scripts to structure the raw data and extract fields like company name, location, funding amounts and dates. The structured data is analyzed using bar charts to visualize total investments by year and sector. Additional visualizations show investments by zip code and county to identify geographic trends. The process requires careful error checking as crowd-sourced data contains many errors that can influence results.
Tech4Africa - Opportunities around Big Data
The document discusses big data and techniques for gathering, storing, processing, and delivering large amounts of data at scale. It covers using Apache Nutch to crawl web data, storing data in Apache Hadoop's distributed file system and processing it using MapReduce. For low-latency queries, it recommends column stores like Apache HBase or Apache Cassandra. The document also discusses using machine learning on historical data to build models for real-time decision making, and challenges of processing unstructured data like prose.
Bridging Structured and Unstructred Data with Apache Hadoop and Vertica
This document discusses bridging unstructured and structured data with Hadoop and Vertica. It describes using Hadoop to extract and structure unstructured investment data from the web. Then it uses Pig to add zip code data and store the results in Vertica. Finally, it explains how Vertica can be used for reporting and data visualization of the structured data for analysis.
Web Crawling and Data Gathering with Apache Nutch
Apache Nutch is an open source web crawler built on Hadoop. It crawls websites, indexes the downloaded content using Lucene, and supports querying the index via Solr. The crawl process involves seeding, filtering, fetching pages, indexing content, and merging results. Nutch can crawl websites in a single process or distributed mode using Hadoop. It provides tools to inject URLs, read crawl segments from HDFS, and demonstrate the crawl lifecycle.
Introduction to Apache Hadoop
Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of commodity servers. It was designed to scale up from single servers to thousands of machines, with very high fault tolerance. Hadoop features two main components - the Hadoop Distributed File System (HDFS) for storage, and MapReduce for distributed processing of large datasets in a parallel and distributed manner. Hadoop saw widespread adoption for applications such as log analysis, data mining, and large-scale graph processing.
Extractiv
Extractiv is a natural language processing service that extracts structured semantic data from unstructured web content by crawling millions of pages, applying entity extraction tools, and producing marked-up files for users. It is suited for those needing high-volume text extraction, additional entity types beyond OpenCalais, or who do not want to pay high costs. The service automatically builds extraction applications, packages entities into data models, and distributes work across a large heterogeneous global network of computers to rapidly process documents.
More from Steve Watt (10)
Building Clustered Applications with Kubernetes and Docker
Building Clustered Applications with Kubernetes and Docker
Building Clustered Applications with Kubernetes and Docker
Building Clustered Applications with Kubernetes and Docker
Apache con 2012 taking the guesswork out of your hadoop infrastructure
Apache con 2012 taking the guesswork out of your hadoop infrastructure
Bridging Structured and Unstructred Data with Apache Hadoop and Vertica
Bridging Structured and Unstructred Data with Apache Hadoop and Vertica
Recently uploaded
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market.
Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/how-axelera-ai-uses-digital-compute-in-memory-to-deliver-fast-and-energy-efficient-computer-vision-a-presentation-from-axelera-ai/
Bram Verhoef, Head of Machine Learning at Axelera AI, presents the “How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-efficient Computer Vision” tutorial at the May 2024 Embedded Vision Summit.
As artificial intelligence inference transitions from cloud environments to edge locations, computer vision applications achieve heightened responsiveness, reliability and privacy. This migration, however, introduces the challenge of operating within the stringent confines of resource constraints typical at the edge, including small form factors, low energy budgets and diminished memory and computational capacities. Axelera AI addresses these challenges through an innovative approach of performing digital computations within memory itself. This technique facilitates the realization of high-performance, energy-efficient and cost-effective computer vision capabilities at the thin and thick edge, extending the frontier of what is achievable with current technologies.
In this presentation, Verhoef unveils his company’s pioneering chip technology and demonstrates its capacity to deliver exceptional frames-per-second performance across a range of standard computer vision networks typical of applications in security, surveillance and the industrial sector. This shows that advanced computer vision can be accessible and efficient, even at the very edge of our technological ecosystem.Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Gursev Pirge, PhD
Senior Data Scientist - JohnSnowLabs
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Apps Break Data
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
GNSS spoofing via SDR (Criptored Talks 2024)
In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security.
This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing.
The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Recently uploaded (20)
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Hadoop for the disillusioned
- 1. Hadoop for the disillusioned Steve Watt, Red Hat CC flickr rubenswieringa @wattsteve
- 2. @wattsteve
- 3. Wired Magazine - July 2008 @wattsteve
- 4. Hadoop in 2013 Platform Layers Technologies Computational Runtimes YARN, GiRAPH, MapReduce, HBase, Phoenix, Spark/BDAS, Drill, Impala, Stinger & more FileSystems Azure, CassandraFS, CephFS, CleverSafe, GlusterFS, GridGain, HDFS, Lustre MapR FS, S3, SWIFT, Quantcast FS, Symantec VCFS & more Infrastructures System on a Chip, x86, Virtualization and Cloud Distributions Cloudera, Hortonworks, IBM, Intel, MapR, WanDisco CC flickr lowfatbrains @wattsteve
- 5. Source: Gartner Hype Cycle @wattsteve
- 6. Your data is growing beyond your ability to manage & query it CC flickr kakadu @wattsteve
- 7. Save money when asking the same questions of your data CC flickr martijnsnels @wattsteve
- 8. Hadoop Customer, “Great, but now what?” Innovators Early Adopters Early Majority Late Majority Laggards CHASM Geoffrey Moore’s Technology Adoption Lifecycle @wattsteve
- 9. new and build data products CC flickr cbcastro @wattsteve
- 10. Ask your domain experts and LOB folks what unanswered questions they have Where can you get the data you need to answer that question? (domain experts should know where to get it) Some of this data may be outside your organization (Social Media, Sensor Data, Data brokerages/Marketplaces, Web Pages) and some of it may be inside. If the data for the query doesn’t exist, figure out how to instrument or gather it. Pair your domain experts with your data engineers so they can work out how to obtain and massage the data given the types of queries desired CC flickr birdwatcher63 @wattsteve
- 11. • Building data products is a similar exercise except that it involves typical product planning, such as identifying a market. • This is also a great way for an organization to explore what assets they have within their data CC flickr syume @wattsteve
- 12. Mapping the night sky CC flickr bobfamiliar @wattsteve
- 13. Analyzing farm soil content to predict human conflict CC flickr oxfam @wattsteve
- 14. Crisis Management for the Chilean Earthquake CC flickr flodigrip @wattsteve
- 15. Thanks for listening Steve Watt swatt@redhat.com @wattsteve
Editor's Notes
- Hadoop is not new - NY Time Source: http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/
- Wired Source: http://www.wired.com/wired/issue/16-07
- Source: Gartner Hype Cycle - http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp “Big Data is a fad”, “Its just BI 2.0”, “This is all just hype”, “We can’t figure out how to use it”, “There’s nothing new here”, “It’s not ready”, “Too few support options”, “Its too hard”
- - You’re sharding your RDBMS infrastructure and its becoming brittle and a nightmare to maintain. - Twitter has a good quote where they stated it used to take them 2 weeks to run an alter table statement
- Using Hadoop for ETL to save money by displacing ETL vendors Using Hive to offload datasets and their corresponding queries from your EDW and lower your EDW bill
- A great way to competitively differentiate with arbitrarily structured data
- Hadoop’s power is in its single storage repository and its support for arbitrary data structures. You have the technology to ask any question if you just have the data.
- http://escience.washington.edu/get-help-now/astronomical-image-processing-hadoop
- http://strataconf.com/stratany2013/public/schedule/detail/30810
- http://vimeo.com/16861296